Chapter 130 The Build System, End to End

Chapter Orientation

Volume 12 starts with the toolchain. XRT's build is not a hand-written Makefile but a Python toolchain with config/modules.json as its single source of truth: the manifest declares every module's headers/sources/tests/examples/docs (including the dependency closure), and tools/build.py drives two verification tracks from it — modular compilation (selecting sources per module, caching objects by closure fingerprint) and the single-header regression (tools/amalgamate.py synthesizes single/xrt.h, then recompiles all tests). This chapter makes four things clear: the manifest structure (schema/dependencies/three test-asset classes: tests/single_tests/examples); build.py's CLI face (suite selection/test filtering/parallel jobs/cflag injection/manifest stacking — extension libraries share the same toolchain); why two tracks (modular proves "this trim compiles", single-header proves "the merge conflicts nowhere" — each track is a release gate); fingerprint caching (objects recompile only when the dependency-closure fingerprint changes — the correctness root of incremental builds).

130.1引入

C-library build systems suffer three chronic ills: manifest drift (CMakeLists and the real source files tell different stories — a file added, the list forgotten, or the reverse); fake trimming (claiming configurability but never verifying that trim combinations compile — feature-macro combinations never tested); single-header desynchronization (the single header maintained by hand, a module changed and the merge forgotten). XRT's answer: one JSON manifest as the source of everything — build (what to compile), amalgamate (what to merge), package (what to pack), measure (what to benchmark) all start from the same modules.json. "The manifest drives everything" eliminates drift by construction: a source not in the manifest doesn't compile; a test not in the manifest doesn't enter the regression.

The dual track, expanded: the modular track takes the dependency closure of your chosen suite (say core,queue) and compiles only those sources — verifying "this trim compiles and runs"; the single-header track first merges all selected modules into one single/xrt.h (amalgamate concatenates in topological order, deduplicates includes, inlines local headers), then compiles the same tests as one translation unit with XRT_IMPLEMENTATION — verifying "the merged single header has no symbol clashes or macro leaks". Every PR must pass both tracks before merging — this is the implementation face of the SPEC's "check + embedded examples compiled into CI".

130.2概念

The manifest: each module's declaration card

The manifest is simultaneously the documentation index (the docs field) and the benchmark registry (the benchmarks field — the queue module's card carries all of dev/bench/queue's scripts and reports) — the full meaning of "source of truth": compile, test, package, measure, document lists — five faces, one card.

build.py's CLI face

Dual-track builds and fingerprint caching

Modular track: suite→closure→compile objects per module (-DXRT_FEATURE_* generated from the manifest's feature fields)→link→run tests in manifest order; examples compile and run one by one (the tutorial's embedded examples are verified here — the SPEC gate's "embedded examples compile"). Single-header track: amalgamate produces single/xrt.h (plus the declarations-only xrt_decl.h)→single_tests compile with #define XRT_IMPLEMENTATION + include of the single header→run. Fingerprint caching: objects cache by "dependency-closure fingerprint" — any source/header inside the closure changes, the fingerprint changes, only then recompile; --rebuild invalidates explicitly. Architecture matrix: native/x86/x64 × gcc/tcc/cl — the release gates cover 32-bit TinyCC (the stand-in for resource-constrained targets).

Extension libraries share the toolchain

The five extension libraries of Chapters 110–129 each carry a config/modules.json — with --manifest extlibs/xhttp/config/modules.json stacked, the same build.py compiles and tests the extension closure (including "the core trim closure the extension requires"). xruntime's README shows the full gate sequence: the build regression, the amalgamate consistency check (--check), the release-maturity check, package verification — library and extensions, one toolchain, one discipline.

130.3示例

First complete program: a manifest-declared example

The program below is from examples/core/version_limits — a member of the manifest's examples array (both build tracks compile it):

 */

#include <stdio.h>
#include <xrt.h>

int main(void)
{
	xrtresourcelimits Limits;

	printf("version=%s\n", xrtVersion());
	xrtResourceLimitsInit(&Limits);
	printf("limits: depth=%u entries=%llu input=%llu\n",
		(unsigned)Limits.iMaxDepth,
		(unsigned long long)Limits.iMaxEntries,
		(unsigned long long)Limits.iMaxInputBytes);
	return 0;
}

What just happened. (1) This file appears in the core module's examples array — python tools/build.py --suite core compiles and runs it; every tutorial-embedded example since Chapter 3 works the same way (tutorial code is gate code — where the SPEC's red lines are machine-executed). (2) xrtVersion() and xrtResourceLimitsInit are APIs declared in core's public headers — the manifest's public_headers field maps one-to-one onto the docs/api contract cards. (3) The default resource bounds (depth 128 / 100k entries / 256 MiB input) are every parser's shared defense line — Chapter 133's testing system comes back to build DoS tests on these bounds.

Second complete program: the allocator-tour example

The second program is from examples/core/allocator_tour — integration verification for a custom allocator:

 */

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <xrt.h>

/* 计数分配器：仍路由系统堆，便于验证替换生效。 */
typedef struct examplecount {
	size_t iAllocs;
	size_t iFrees;
} examplecount;

static ptr exampleAlloc(ptr pContext, size_t iSize)
{
	++((examplecount*)pContext)->iAllocs;
	return (ptr)malloc(iSize);
}

static ptr exampleRealloc(ptr pContext, ptr pMemory, size_t iSize)
{
	(void)pContext;
	return (ptr)realloc(pMemory, iSize);
}

static void exampleFree(ptr pContext, ptr pMemory)
{
	++((examplecount*)pContext)->iFrees;
	free(pMemory);
}

int main(void)
{
	xallocator Default;
	xallocator Custom;
	examplecount Count;
	uint8* pBlock;
	uint8* pDup;
	int iResult = 1;

	/* ---- 必须在任何 XRT 分配前完成替换。 ---- */
	xrtGetAllocator(&Default);
	memset(&Count, 0, sizeof(Count));
	Custom.Context = (ptr)&Count;
	Custom.Alloc = exampleAlloc;
	Custom.Realloc = exampleRealloc;
	Custom.Free = exampleFree;
	if ( !xrtSetAllocator(&Custom) ) {
		goto Cleanup;
	}

	/* ---- At 族五接口：位置参数不影响堆行为。 ---- */
	pBlock = (uint8*)xrtMallocAt(4u, __FILE__, __LINE__);
	if ( (pBlock == NULL) ||
		((pDup = (uint8*)xrtMemDupAt("hi", 2u, __FILE__,
			__LINE__)) == NULL) ||
		(memcmp(pDup, "hi", 2u) != 0) ||
		((pBlock = (uint8*)xrtReallocAt(pBlock, 16u, __FILE__,
			__LINE__)) == NULL) ||
		(xrtCallocAt(2u, 4u, __FILE__, __LINE__) ==
			NULL) ) {
		goto Cleanup;
	}
	/* Calloc 清零验证单独做——保留指针用于释放。 */
	{
		uint8* pZero = (uint8*)xrtCallocAt(2u, 4u, __FILE__,
			__LINE__);

		if ( (pZero == NULL) || (pZero[0] != 0u) ||
			(pZero[7] != 0u) ) {
			goto Cleanup;
		}
		xrtFreeAt(pZero, __FILE__, __LINE__);
	}
	xrtFreeAt(pDup, __FILE__, __LINE__);
	xrtFreeAt(pBlock, __FILE__, __LINE__);
	printf("allocator: swap -> at-family alloc/calloc/dup/realloc ok\n");

	
	 * 且 GetAllocator 仍报告已安装的计数分配器。 ---- */
	
	 * 分配才经过自定义 Alloc；计数只需 >= 1 即证明替换生效。 */
	if ( xrtSetAllocator(&Default) ||
		(Count.iAllocs < 1u) ) {
		goto Cleanup;
	}
	{
		xallocator Current;

		xrtGetAllocator(&Current);
		if ( (Current.Alloc != Custom.Alloc) ||
			(Current.Free != Custom.Free) ) {
			goto Cleanup;
		}
	}
	printf("allocator: locked after first alloc -> swap rejected ok\n");
	iResult = 0;

Cleanup:
	return iResult;
}

What just happened. (1) After xrtSetAllocator swaps in a custom allocator, the at-family four entrances run (Chapter 5's memory face, regressed at the example level). (2) The locked-after-first-allocation semantics verified: a swapped allocator cannot be swapped again — the stability guarantee for embedding hosts (Chapter 135). (3) The two output lines are themselves products of the dual-track build: manifest-driven, gate-executed — every tutorial example you have seen passed through this chain.

130.4契约

• Single source of truth: modules.json drives build/amalgamate/package/measure/doc — add the manifest before adding sources; keep both synchronous.
• Three asset classes: tests (modular) / single_tests (single header) / examples (tutorial + gate) — each with its track.
• Closure semantics: a suite expands to its dependency closure; compiling/trimming/packing all follow the closure.
• Dual-track gate: modular proves the trim compiles, single-header proves the merge conflicts nowhere — both green before merge.
• Deterministic order: tests run in manifest order (jobs parallelize compilation only, never tests).
• Fingerprint caching: objects cache by closure fingerprint; --rebuild invalidates explicitly.
• Architecture matrix: native/x86/x64 × gcc/tcc/cl — TinyCC 32-bit is the resource-constrained gate.
• Checkpoint restart: start-test/start-single-test resume from the breakpoint — fix one, run the rest.
• Extension stacking: --manifest stacks extension manifests — library and extensions, one toolchain.
• Examples are gates: examples compile and run in both tracks — tutorial-embedded code with zero extra maintenance.

Common Mistakes

/* create src/foo/bar.c and commit directly - the manifest doesn't have it */
/* CI: the modular track's closure is incomplete - link failure */

/* add it to the foo module's sources array in modules.json in the same change */
/* the PR diff shows manifest entry and source file together - reviewable consistency */

python tools/build.py --suite my_module --no-single && merge
/* skipping the single-header track before merging - clashes blow up in the user's hands */

/* before merging a PR: both tracks green - --no-single is only for fast mid-development iteration */
python tools/build.py --suite my_module   /* includes the single-header track */

/* assuming --jobs 8 runs tests in parallel - tests sharing resources occasionally interfere */
/* fact: --jobs parallelizes compilation only; tests strictly follow manifest order */

/* want speed: run a subset, not in parallel */
python tools/build.py --suite my_module --test test_queue_oom
/* checkpoint restart: --start-test continues from the failure */

Programming Exercises

Summary