Chapter 133 Testing, Fault Injection, and Fuzzing

Chapter Orientation

Every API of the preceding chapters is guarded by a crowd of tests — this chapter covers those tests' morphology. XRT's test system (hung on the manifest's tests field, run by build.py on both tracks) has four weapons: positive tests (the testRequire(条件, "消息") (condition, "message") assertion family — acceptance terse enough for one line); thread tests (the _threads suffix — concurrency semantics measured for real); negative and mutation tests (the _negative/_mutation suffixes — robustness on malformed input: full-byte mutation round trips, boundary seeds); per-allocation-point OOM tests (the _oom suffix — 185 files! A failing allocator injected point by point, verifying "failure breaks no state and the error reports correctly"). Fuzzing (fuzz/): LibFuzzer-style entry points plus in-repo structured fuzzing (deterministic shift-register noise with syntax seeds first — the shape of test_http_auth_fuzz). The four weapons cover the four quadrants "right behavior, behavior under concurrency, behavior on wrong input, behavior without resources".

133.1引入

Why do OOM tests number 185 files? Because memory exhaustion is the most easily skipped path: developers write code assuming allocation succeeds (in normal environments it does), and tests run on memory-rich machines — so "cleanup after allocation failure" becomes a purely untested zone. Yet that path hides the most leaks and state corruption (half-initialized structures, partially built containers). XRT's discipline is per-allocation-point verification: a failing allocator (like testMapOomAlloc — "forward to the underlying allocator while allowed; return NULL once the quota runs out") makes each allocation of the code under test fail in turn — one round per the Nth failure, looping until every point has taken its turn — "allocation failure breaks no state" becomes an exhaustively enumerated fact, not an assumption.

Mutation testing shares the motive but a different object: parser robustness. HTTP headers, percent encodings, TLS messages — the real network delivers every byte sequence; dozens of hand-made cases are never enough. Mutation testing flips every byte of legal inputs one by one, then runs whole rounds of random noise — "arbitrary input never crashes; failure goes through structured errors" is machine-proven. Fuzzing adds one more layer: syntax seeds (real protocol shapes) + deterministic random variation — running combinations in CI that humans wouldn't think of.

133.2概念

The assertion family and the test-shape matrix

testRequire is the library-wide unified assertion (the test framework is minimal: on failure print message + line, exit non-zero — the quality gate trusts only exit codes). Shape suffixes are a naming convention — the manifest shows each module's covered quadrants at a glance (queue's card: test_queue/test_queue_spsc_threads/test_queue_spsc_oom — three quadrants present).

Per-allocation-point OOM: the failing-allocator pattern

Chapter 6 (memory debugging) covered the failing allocator's tool face (xrtMemDebugFailAfter); the test system uses it plus hand-written quota allocators to cover every module. "Error reporting allocates no memory" is a recurring assertion (xrtErrorCreate on failure keeps the OOM category in the error slot — the error system stays alive under resource exhaustion): the test face of Chapter 4's "minimal dependencies" promise.

Mutation testing: round trips and stability

Two shapes: full-byte mutation (a legal vector → each byte individually ±1/flipped → re-parse/re-decode — rejection or success both fine, but no crashes or leaks, and successes semantically match the original); noise round trips (deterministic random input — e.g. percent's 6000 rounds of "any bytes + case-mixed escaping round-trip stably"). Determinism (fixed seeds) is the key: failures reproduce — a fuzz-found issue replays by reporting the seed.

Fuzzing: seeds + noise + invariants

test_http_auth_fuzz's structure is the template: (1) syntax seeds first (a string array of real protocol shapes — empty string / standard Basic/Bearer/Digest with complex parameters — known boundaries pass first); (2) deterministic noise (shift-register variable-length random input — exploring combinations humans wouldn't think of); (3) invariant assertions (parsing either succeeds or fails structurally — FuzzAbort reports contract violations with line numbers, "avoiding losing diagnostics after the optimizer merges cold branches" — even compiler optimizations are accounted for). The fuzz/ directory (xhttp's four LibFuzzer-style entries: http_auth/http_route/http_router/http_sse): persistent entries for external fuzzers — CI quickly runs the built-in fuzzing; offline deep runs use external fuzzers, same invariants.

133.3示例

First complete program: the everyday face of fault injection

The program below is from examples/memory/fail_inject — the application-level posture of a failing allocator:

 */

#include <stdio.h>
#include <xrt.h>

/* 事件访问器：只打印前几个 alloc 事件的名字。 */
static bool printEvent(const xmemdebugevent* pEvent, ptr pUserData)
{
	size_t* pCount = (size_t*)pUserData;

	if ( *pCount < 1u && pEvent->Kind == XMEMDEBUG_ALLOC ) {
		printf("event[%zu]: %s\n", *pCount,
			xrtMemDebugEventName(pEvent->Kind));
	}
	(*pCount)++;
	return true;
}

int main(void)
{
	ptr pFirst;
	ptr pSecond;
	ptr pThird;
	size_t iEvents = 0;

	(void)xrtMemDebugReset();
	(void)xrtMemDebugEnable(true);
	printf("enabled=%d\n", xrtMemDebugEnabled() ? 1 : 0);

	/* 注入：第 2 次分配失败。 */
	(void)xrtMemDebugFailAfter(1u);
	pFirst = xrtMalloc(16);
	pSecond = xrtMalloc(16);
	printf("fail-after-2: 1st=%s 2nd=%s triggered=%d\n",
		pFirst != NULL ? "ok" : "NULL",
		pSecond != NULL ? "ok" : "NULL",
		xrtMemDebugFailTriggered() ? 1 : 0);
	xrtFree(pFirst);

	/* 事件流：最早的 alloc 事件可按名访问。 */
	(void)xrtMemDebugVisit(printEvent, &iEvents);

	/* 清除未触发注入后再分配恢复成功；已触发的标志保持。 */
	xrtMemDebugFailClear();
	pThird = xrtMalloc(16);
	printf("cleared: 3rd=%s triggered-still=%d\n",
		pThird != NULL ? "ok" : "NULL",
		xrtMemDebugFailTriggered() ? 1 : 0);
	xrtFree(pThird);

	(void)xrtMemDebugEnable(false);
	printf("reset=%d\n", xrtMemDebugReset() ? 1 : 0);
	return 0;
}

What just happened. (1) xrtMemDebugFailAfter(N)'s measured semantics: the first N succeed, the N+1st fails — the output's 1st=ok 2nd=NULL is exactly FailAfter(1). (2) The event stream (event[0]: alloc) — ordered event access on the debug heap: the injected failure also enters the event stream (complete audit). (3) FailClear also resets the "already triggered" flag (after clearing, 3rd=ok and triggered-still=0) — every _oom test variant is built on this set of entrances (header comment verbatim: tests and applications share one implementation — another "no second implementation").

Second complete program: locating a leak

The second program is from examples/memory/debug — the Reset/Snapshot/VisitLive trio:

 */

#include <stdio.h>

#include <xrt.h>






 */
static bool printAllocation(const xmemdebugallocation* pAllocation, ptr pUserData)
{
	(void)pUserData;
	printf("live address=%p size=%zu site=%s:%u\n",
		pAllocation->Address,
		pAllocation->Size,
		pAllocation->File != NULL ? pAllocation->File : "unknown",
		(unsigned int)pAllocation->Line);
	return true;
}



int main(void)
{
	xmemdebugsnapshot tSnapshot;
	unsigned char* pMemory;

	/* 起点清零：本范例观察到的所有事件都来自接下来的代码。 */
	if ( !xrtMemDebugReset() ) {
		return 1;
	}
	pMemory = (unsigned char*)xrtMalloc(64);
	if ( pMemory == NULL ) {
		return 2;
	}
	pMemory[0] = 0x5A;

	/* 泄漏现场：64 字节仍在活动列表，分配点精确到本文件第 30 行。 */
	xrtMemDebugSnapshot(&tSnapshot);
	printf("live_count=%zu live_bytes=%zu peak_bytes=%zu\n",
		tSnapshot.LiveCount,
		tSnapshot.LiveBytes,
		tSnapshot.PeakBytes);
	(void)xrtMemDebugVisitLive(printAllocation, NULL);

	/* 释放后再拍快照：Live 归零，累计计数器对账（1 alloc + 1 free）。 */
	xrtFree(pMemory);
	xrtMemDebugSnapshot(&tSnapshot);
	printf("alloc_count=%llu free_count=%llu events=%zu\n",
		(unsigned long long)tSnapshot.AllocCount,
		(unsigned long long)tSnapshot.FreeCount,
		tSnapshot.EventCount);

	/* 收尾复位，不影响后续程序段的调试观察。 */
	return xrtMemDebugReset() ? 0 : 3;
}

What just happened. (1) The standard posture for locating a leak (header comment): Reset before the suspect segment, Snapshot after — a LiveCount that won't return to zero plus the site printed by VisitLive (precise to file:line) is the leak point. (2) The output shows alloc_count=1 free_count=1 but live_count=1 — a live block after one alloc and one free? That's the example's deliberate leftover (to demonstrate that VisitLive sees it) — the count-reconciliation mindset: three counters (alloc/free/live) cross-check allocation discipline. (3) The stronger capability (quarantine catching double-free/UAF) comes in the JSON report form (debug_report sample) — tests and operations share the same data.

The test pyramid's third layer: the mutation/fuzz divide

Picture XRT's tests as a pyramid: the base is a mass of positive assertions (dozens of testRequire per module — behavior nailed down item by item); the middle is quadrant expansion (threads/oom — one group for each of the two costly quadrants); the top is mutation and fuzzing — mechanical exploration of the input space. Remember the divide: mutation anchors on known legal inputs (flip every byte — guaranteeing "the illegal near the legal" all passes); fuzzing anchors on syntax templates (randomly assembled realistic input — exploring the "structurally plausible but never seen" region). The former guards against regressions (old boundaries broken); the latter against the unknown (unwritten boundaries hiding). Both live on determinism as their lifeline — also the precondition for entering CI (minutes to run, failures always reproducible); true-random deep exploration is left to offline external fuzzers.

133.4契约

• Shape naming: _threads/_negative/_mutation/_oom/_fuzz suffixes — covered quadrants visible in the manifest.
• Minimal assertions: the testRequire family — failure prints + non-zero exit; the quality gate trusts only exit codes.
• OOM point by point: the failing allocator rotates point by point; assert failure return + unchanged state + XERR_MEMORY + correct domain.
• Error system alive: error reporting in OOM allocates no memory — the error slot keeps the OOM category.
• Mutation determinism: fixed seeds — failures reproduce (the seed is the replay credential).
• Invariants first: fuzz asserts the either/or "succeed or fail structurally" — crashes/leaks are contract violations.
• Seeds first: syntax boundary seeds precede random noise — known boundaries must pass.
• Dual-mode fuzzing: built-in deterministic (CI) + external fuzzer entries (offline deep runs) — the same invariants.
• Examples are testware: the fail_inject/debug samples share entrances with _oom tests — no second implementation.

Common Mistakes

testRequire(parse(str) == OK, "parse");
/* concurrency? malformed? OOM? - all untested */

/* at least one group per quadrant: positive + threads + mutation/negative + oom */
testRequire(parse(str) == OK, "parse");
/* tests/xxx/test_xxx_threads.c / _mutation.c / _oom.c attached to the same module in the manifest */

srand(time(NULL));            /* different every time - a failure is lost */
run_fuzz(random_input());

uint32 State = UINT32_C(0x9E3779B9);  /* fixed seed */
run_fuzz(noise(&State));               /* a failing seed replays */

testRequire(map_insert(...) == false, "oom fails");
/* state? error? - unchecked */

testRequire(map_insert(...) == false, "oom fails");
testRequire(xrtErrorKind(xrtGetError()) == XERR_MEMORY, "kind");
testRequire(map_still_consistent(&Map), "state intact");  /* state unchanged */
testRequire(map_destroy_path_clean(&Map), "可安全清理");   /* teardown does not crash twice */

Programming Exercises

Summary