Chapter 134 OOM and Fault-Injection Testing

Chapter Orientation

Chapter 133 presented OOM testing as one of the four weapons; this chapter magnifies it into a full discipline — behind 185 _oom files stands a rigorous methodology. Four blocks: per-allocation-point rotation (the looping enumeration of "the Nth allocation fails" — "every allocation failing" is verified individually, not "some allocation fails"); the three failing-allocator forms (xrtMemDebugFailAfter counting / hand-written quota (Limited+Remaining — the map OOM test's shape) / fail-always (test_oom's FailAllocator) — each with its use); the error system's survival under exhaustion (error reporting in OOM allocates no memory — when xrtErrorCreate fails, the slot keeps XERR_MEMORY: reporting "no memory" itself needs no memory); the debug heap weaponized (FailAfter combined with the event stream — injected failures are auditable; the quarantine catches double-free/UAF — fault shapes go beyond "allocation failure"). One goal: "no resources" goes from a runtime assumption to an exhaustively enumerated fact.

134.1引入

One derivation shows the necessity of per-allocation-point rotation: a function makes 5 allocations — the cleanup path for the 3rd failing differs from the 5th (the former rolls back two full side effects, the latter four and a half); testing only "the 1st fails" (or not at all) leaves 4 cleanup paths never executed — any half-initialized leak/state corruption among them goes straight to production. The rotation method: the failure point sweeps from 1 to 5 (the whole flow rerun per round) — every cleanup path executed, every failure state asserted. The 185 files are exactly this sweep unrolled across all modules — the number itself is proof of discipline.

The error system's survival deserves separate emphasis: if reporting an error required allocation, then in OOM you would "fail to report the failure" — the troubleshooter sees a silent crash instead of a structured error. XRT's error object (Chapter 4) avoided this by design (immutable objects with minimal allocation, degrading to the static XERR_MEMORY category on failure) — and test_oom is exactly the machine verification of that promise: "verify that OOM error reporting itself allocates no memory" is its first-line comment.

134.2概念

Per-allocation-point rotation: the shape of enumeration

The strong "state unchanged" assertion has two tiers: still usable (OOM is temporary — after memory frees, the same object works as before, e.g. a map whose insert failed still queries); safely destructible (at least the Destroy path doesn't crash a second time — half-initialized objects clean up completely). Both tiers are contracts: the first is quality, the second the floor.

The three failing-allocator forms

The three forms amount to "treating the memory budget as a controllable knob" — testing moves from the binary world of "is there memory" to the precise world of "which allocation runs out".

The error system's survival under exhaustion

test_oom's assertion chain is the template: xrtMalloc(32)==NULL + error category XERR_MEMORY + domain xrt.memory; then xrtErrorCreate(...)==NULL (the error object's own allocation fails) + the slot still keeps XERR_MEMORY — "the failure to create an error does not overwrite the OOM fact". This verifies Chapter 4's deep "minimal dependencies" promise: the error channel stays usable under resource exhaustion. The all-OOM degradation semantics (recurring in contracts): "if OOM has made even error objects unallocatable, degrade to static XERR_MEMORY" — not every failure can carry rich context, but every failure can be seen.

Extending fault shapes: beyond allocation failure

The debug heap's (Chapter 6) quarantine extends fault injection to the free side: double free, use-after-free (UAF) — the quarantine's delayed reuse makes such errors necessarily visible in tests (instead of passing silently). Combining _mutation tests with OOM (mutation + point-by-point failure) is the extreme form of stress: malformed input × resource exhaustion — the parser's deepest corners are lit too. The same discipline in the extensions: xhttp's per-allocation-point rollback, the OOM gate of xws server routing — every extension library carries its own _oom matrix (the same gate after Chapter 130's manifest stacking).

134.3示例

First complete program: the error system surviving exhaustion

The program below is from tests/core/test_oom.c — this is the test itself (not an example), showing the core survival assertion:

#include "../test_allocator.h"



/* 验证 OOM 错误报告本身不申请内存。 */
int main(void)
{
	testRequire(testInstallFailAllocator(), "failure allocator install failed");
	testRequire(xrtMalloc(32) == NULL, "allocation should fail");
	testRequire(xrtErrorKind(xrtGetError()) == XERR_MEMORY, "allocation failure must report memory error");
	testRequire(strcmp(xrtErrorDomain(xrtGetError()), "xrt.memory") == 0, "OOM domain mismatch");

	xrtClearError();
	testRequire(xrtErrorCreate(XERR_IO, "test", 1, "cannot allocate") == NULL, "error allocation should fail");
	testRequire(xrtErrorKind(xrtGetError()) == XERR_MEMORY, "error allocation failure must preserve OOM");
	xrtClearError();
	printf("[PASS] oom\n");
	return 0;
}

What just happened. (1) testInstallFailAllocator() — the fail-always form installed (an always-NULL allocator takes over library memory). (2) First assertion group: allocation failure + category + domain — the error-reporting trio holds in the minimal scenario. (3) The second group is this chapter's soul: the very call creating the error object fails (returns NULL) — and the error slot still keeps XERR_MEMORY — "the channel reporting failures doesn't go mute because of a failure". (4) Behind the single line [PASS] oom lies the floor guarantee for all resource-exhaustion paths. This file is barely a dozen lines — because it is the foundation (complex modules' _oom stack business assertions on top).

Second complete program: the application face of injection

The second program is from examples/memory/fail_inject — the feel of the three-state knob:

 */

#include <stdio.h>
#include <xrt.h>

/* 事件访问器：只打印前几个 alloc 事件的名字。 */
static bool printEvent(const xmemdebugevent* pEvent, ptr pUserData)
{
	size_t* pCount = (size_t*)pUserData;

	if ( *pCount < 1u && pEvent->Kind == XMEMDEBUG_ALLOC ) {
		printf("event[%zu]: %s\n", *pCount,
			xrtMemDebugEventName(pEvent->Kind));
	}
	(*pCount)++;
	return true;
}

int main(void)
{
	ptr pFirst;
	ptr pSecond;
	ptr pThird;
	size_t iEvents = 0;

	(void)xrtMemDebugReset();
	(void)xrtMemDebugEnable(true);
	printf("enabled=%d\n", xrtMemDebugEnabled() ? 1 : 0);

	/* 注入：第 2 次分配失败。 */
	(void)xrtMemDebugFailAfter(1u);
	pFirst = xrtMalloc(16);
	pSecond = xrtMalloc(16);
	printf("fail-after-2: 1st=%s 2nd=%s triggered=%d\n",
		pFirst != NULL ? "ok" : "NULL",
		pSecond != NULL ? "ok" : "NULL",
		xrtMemDebugFailTriggered() ? 1 : 0);
	xrtFree(pFirst);

	/* 事件流：最早的 alloc 事件可按名访问。 */
	(void)xrtMemDebugVisit(printEvent, &iEvents);

	/* 清除未触发注入后再分配恢复成功；已触发的标志保持。 */
	xrtMemDebugFailClear();
	pThird = xrtMalloc(16);
	printf("cleared: 3rd=%s triggered-still=%d\n",
		pThird != NULL ? "ok" : "NULL",
		xrtMemDebugFailTriggered() ? 1 : 0);
	xrtFree(pThird);

	(void)xrtMemDebugEnable(false);
	printf("reset=%d\n", xrtMemDebugReset() ? 1 : 0);
	return 0;
}

What just happened. (1) Read alongside Chapter 133: there the point was the "four weapons" taxonomy; here the focus is precise control of injection — FailAfter(1)'s boundary semantics (1st=ok 2nd=NULL) visible field by field in the output. (2) The triggered flag's independence: queryable after triggering, reset only by Clear — the injector's own state is auditable (the alloc event appears in the event stream — a failure is an event too). (3) Real tests write rotation with these entrances: an outer for loop increments N, each round Resetting the injector state and the tested object — this sample is a single-step cross-section of that loop.

Why C libraries' OOM discipline is stricter than managed languages'

The contrast shows why this investment is necessary. In managed languages (Java/Go/C#) allocation failure is a Throwable — the runtime unwinds the stack, the GC guarantees reclamation — the platform takes over most of the application's cleanup duty. C has no such backstop: allocation failure is just a NULL return, cleanup is 100% the caller's responsibility — every field of a half-initialized structure, every block of a partially built container must be returned by someone in the right order. XRT's 185 _oom files are the point-by-point acceptance ledger of that "return duty". Few libraries of this scale reach this density — most C libraries' OOM handling stops at "the critical path checks NULL" — exactly the root of "occasional memory leaks finally OOM-freezing" in embedded and long-running services: not some unchecked NULL, but a cleanup path no one ever walked. Rotational enumeration turns "every cleanup path walked" into a machine guarantee — that is its entire value.

134.4契约

• Rotational enumeration: the failure-point sweep covers every allocation — every cleanup path executed and asserted.
• Threefold assertion: failure return / state unchanged (still usable or safely destructible) / correct error (category + domain).
• Reconciliation discipline: zero leaks per round (debug heap Reset/Snapshot cross-checked).
• Three allocator forms: counting / quota / fail-always — the knob precise to "which one".
• Error survival: error reporting in OOM allocates no memory; on error-creation failure the slot keeps XERR_MEMORY.
• Degradation semantics: when error objects are unallocatable, degrade to static XERR_MEMORY — failures are always visible.
• Free-side injection: the quarantine catches double-free/UAF — fault shapes cover release paths.
• Combined stress: mutation × point-by-point failure — the product space of malformed input and resource exhaustion.
• Extensions, same law: every extension library carries its own _oom matrix — stacking manifests stacks the gate.

Common Mistakes

for ( N = 1; N < 100; N++ ) {
	fail_after(N);
	run_flow();        /* previous round's trigger bit/quota residue - everything fails for N>1 */
}

for ( N = 1; N < 100; N++ ) {
	reset_injector();   /* Clear the trigger bit / refill the quota */
	rebuild_target();  /* rebuild the tested object to a consistent start */
	fail_after(N);
	run_flow();
	assert_three();    /* threefold assertion */
}

testRequire(op(&Obj) == false, "oom");
/* return directly - Destroy has never executed in the failed state */

testRequire(op(&Obj) == false, "oom");
testRequire(destroy(&Obj), "失败态可安全销毁");  /* the cleanup path executes */

testRequire(xrtErrorKind(xrtGetError()) == XERR_MEMORY, "kind");
/* domain unchecked: MEMORY of the xrt.io domain also passes */

testRequire(xrtErrorKind(xrtGetError()) == XERR_MEMORY, "kind");
testRequire(strcmp(xrtErrorDomain(xrtGetError()),
	"xrt.memory") == 0, "domain");   /* category + domain double assertion */

Programming Exercises

Summary