Chapter 136 Performance Analysis Methods: Benchmark Design and Measurement

Chapter Orientation

The first step of tuning is not changing code but knowing how to measure — this chapter is performance work's methodology (Chapter 137 is the practice). XRT's measurement infrastructure provides the template: profiled benchmarks (config/performance_profiles.json: each profile a benchmark source + arguments + sampling policy — "what to run" is a declaration, not improvisation); environment baselines (fourteen elements: platform/CPU/affinity/power policy/compiler/optimization level/scheduling policy — numbers compare only within one baseline); statistical discipline (repeats sampling + MAD (median absolute deviation) and central-range dual dispersion gates — noisy measurements are rejected, not averaged away); regression thresholds (directional higher 10%/lower 20% — slow gets strict, fast gets suspicion); sweep experiments (a systematic traversal of parameter space, not single-point spot checks). Plus the questioning discipline: a benchmark answers the question you asked — with the wrong question, the most accurate numbers are useless.

136.1引入

Three failed openings of performance work: baseline-free measurement ("this feels slower" — by how much? Compared to what? On which machine?); cross-environment comparison (mixing dev-machine and CI-machine numbers — CPU generation/power policy/background load all differ); noise taken as signal (treating a 3% single-sample fluctuation as regression — a wild-goose chase). The common root: treating measurement as "run it and look at the number" instead of experiment design. XRT's measurement toolchain (measure_performance/measure_size + profiles + report schema) freezes experiment design in place: the profile declares what and how to run, the baseline records where, the gate decides whether it counts — humans are responsible only for asking the right question.

The questioning discipline, expanded: before benchmarking, write one sentence — "I suspect X is slower/faster than Z under Y"? X is the operation (say SPSC single enqueue), Y the scenario (capacity 4096, two threads), Z the comparison (the previous version / another data structure). If you can't write that sentence, it isn't time to measure. Chapter 62's "run with defaults first, observe, tune by data" takes this shape here: run the default profile for a baseline → look at dispersion (the noise level) → design the experiment → numbers answer the question.

136.2概念

Profiled benchmarks: declarative experiments

The two speeds of smoke and full: smoke_args is CI's fast form (seconds to verify "the benchmark still runs"); full args are formal sampling (minutes to statistics). Benchmark sources live in dev/bench/ (hung on the module manifest's benchmarks field — yet another consumption of Chapter 130's manifest) — benchmarks are manifest-managed assets too.

The environment baseline: fourteen elements

The two most-ignored elements: power_policy (a laptop's power-saver versus plugged-in performance can differ 2–3× — reports must record it); affinity (pinning changes cache and scheduling behavior — the queue latency benchmark has a pinned variant specifically for contrast). Comparison rule: trend conclusions hold only within one baseline; across baselines, suspect the environment before the code (Chapter 131's seven size elements, performance edition).

Statistical discipline: dispersion gates

max_relative_mad: 0.2 (median absolute deviation relative to the median ≤20%) and max_relative_central_range: 0.3 (central range ≤30%) — dual gates: either exceeded = this round of measurement is too noisy — the result is "rejected", not "averaged and carried on". This is the watershed with amateur measurement: amateurs average noise away (a very stable mean that means nothing — it lands somewhere different each time); discipline exposes noise (reject → investigate → interference source / more repeats / different policy). Median first: reports take the median (robust to outliers); the sample series is recorded in full (transparent, re-examinable).

Regression thresholds: directional dual thresholds

regression: {higher: 0.1, lower: 0.2} — 10% slower alarms, 20% faster also alarms. Why watch speedups: the measurement environment changed (a fake improvement), or it was paid for elsewhere (this profile faster, another slower — the full profile set must run). The directional asymmetry is pragmatic: slow is direct harm (strict gate), fast is an indirect signal (loose gate).

Sweep experiments: sweep

A single-point benchmark answers "how fast is this configuration"; sweep answers "what does the configuration space look like" — queue's capacity/batch two-dimensional sweep (run_queue_bench_sweep): a matrix of capacity 64..65536 × batch 1..128 — inflection points (returns saturate past batch 32) and plateaus (SPSC degrades at small capacity) at a glance. Sweeps are where selection evidence comes from: Chapter 21's "how to choose capacity" and Chapter 68's "how big a batch" — the answers come from sweeps, not gut feeling. Reports (QUEUE_BATCH_SWEEP et al.) are archived under dev/bench — experiment results are first-class assets.

136.3示例

First complete program: a memory-view micro-benchmark

The program below is from examples/memory/stats — stats readings as the lightest performance observation:

 */

#include <stdio.h>

#include <xrt.h>



int main(void)
{
	xmemstats tStats;
	ptr pSmall;
	ptr pLarge;

	/* 开关可查询：先确认当前状态，再打开并清零。 */
	printf("before=%d\n", xrtMemStatsEnabled() ? 1 : 0);
	xrtMemStatsEnable(true);
	printf("after=%d\n", xrtMemStatsEnabled() ? 1 : 0);
	xrtMemStatsReset();

	/* 一次小块（走池）+ 一次大块（直通）。 */
	pSmall = xrtMalloc(48);
	pLarge = xrtMalloc(4096);
	if ( (pSmall == NULL) || (pLarge == NULL) ) {
		xrtFree(pSmall);
		xrtFree(pLarge);
		return 1;
	}
	xrtFree(pSmall);
	xrtFree(pLarge);

	/* 读取窗口内计数（字段语义见文件头读数解释）。 */
	xrtMemStatsGet(&tStats);
	printf("malloc_calls=%llu pooled=%llu direct=%llu backing=%llu\n",
		(unsigned long long)tStats.MallocCalls,
		(unsigned long long)tStats.PooledAllocCalls,
		(unsigned long long)tStats.DirectAllocCalls,
		(unsigned long long)tStats.BackingAllocCalls);

	/* 用完关闭，回到零开销模式。 */
	xrtMemStatsEnable(false);
	return 0;
}

What just happened. (1) Performance analysis's first question is often "how much got allocated" — stats' four readings answer it: here 48B hits the pool (zero system calls) while 4096B passes through — the pool boundary measured live (faster than reading docs and always consistent with the current build). (2) This is micro-observation (counter level) — time-dimension measurement goes through the bench toolchain; but many "performance problems" are answered at the counter layer (backing abnormally high = pool configuration mismatching business sizes). (3) Methodological position: stats is the zero-cost always-on observation (Chapter 135's health panel) and also the first stop of performance investigation — look at counters before timing.

Second complete program: a self-check benchmark with deterministic output

The second program is from examples/core/version_limits — the trustworthy form of a benchmark:

 */

#include <stdio.h>
#include <xrt.h>

int main(void)
{
	xrtresourcelimits Limits;

	printf("version=%s\n", xrtVersion());
	xrtResourceLimitsInit(&Limits);
	printf("limits: depth=%u entries=%llu input=%llu\n",
		(unsigned)Limits.iMaxDepth,
		(unsigned long long)Limits.iMaxEntries,
		(unsigned long long)Limits.iMaxInputBytes);
	return 0;
}

What just happened. (1) This sample demonstrates deterministic output — contrasted with timing benchmarks' randomness: in benchmark engineering "reproducible" splits in two (deterministic output = same value every time; statistical output = stable distribution). (2) The default resource bounds (128/100000/256MiB) are the performance profiles' hidden parameters — parser benchmarks' input sizes sit behind these gates (oversized-input tests need explicit relaxation — profile cflags can inject). (3) The real benchmarks of dev/bench (queue/coroutine/task families) are bigger in shape (multi-threaded sampling + statistical output), but methodology's atoms are all present in this sample: fixed input, explicit output, comparable.

Anatomy of a benchmark report (QUEUE_BENCH as the template)

dev/bench/QUEUE_BENCH_20260728_XRT2.md's structure is the template: Sources (benchmark source + scripts + build shape — the reproducible entry); Environment (OS/CPU/compiler/build flags/scheduling policy — the baseline self-reported); Matrix (matrix parameters: items per producer / capacity / producer-consumer counts / batch / sample counts — the experiment design visible); Results (per-profile sample series + median — transparent raw data); Interpretation (reading — the connection between conclusions and numbers: the single-item path in the expected band, batch reservation amortizing effectively, batch as the preferred high-throughput path). Five sections, each with its duty: without Sources it's irreproducible, without Environment incomparable, without Matrix unevaluable, without Results untrustworthy, without Interpretation unusable. Self-check a report against these five sections — this is also the acceptance format of Chapter 136's exercises and Chapter 137's challenge.

136.4契约

• Profile declarations: benchmark source + args + smoke two-speeds + policy — what runs is never improvised.
• Fourteen-element baseline: platform/CPU/affinity/power/build/policy — trends compare only within one baseline.
• Dual dispersion gates: MAD ≤20% + central range ≤30% — over the limit means rejection, not averaging.
• Median reports: sample series recorded in full; the median is outlier-robust.
• Directional thresholds: 10% slower strict / 20% faster loose — fast is a signal too.
• Sweep experiments: systematic traversal of parameter space — where selection evidence is born; reports archived under dev/bench.
• smoke/full two speeds: CI seconds-level verification + formal minutes-level sampling.
• Benchmarks are assets: benchmarks hang on the manifest — managed with the same lifecycle as sources.
• Questions first: if you can't write "suspect X over Z under Y", it isn't time to measure.

Common Mistakes

run_bench_once();      /* single sample */
if ( faster_by_8pct() ) { merge(); }  /* concluding inside noise */

/* profiled sampling: repeats=5 + dispersion gates; median compared against baseline */
measure_profile("queue", /*repeats*/ 5, /*gates*/ true);
if ( regression_significant(&New, &Base) ) { conclude(); }

/* local 9M, CI 11M items/s */
if ( ci_faster() ) { trust_ci(); }   /* different baselines: meaningless */

/* same-machine same-policy A/B: environment identical, only code varies */
measure_ab(same_machine(), build_a(), build_b());
/* across machines compare only sweep curve shapes (structural conclusions) */

/* measuring only the single-item path */
bench_single_enqueue();      /* optimize it */
/* production load is 90% batch calls - the optimization goes unfelt */

/* stats first: production behavior counted -> batch dominates */
/* the benchmark includes batch profiles + sweep fixes the batch - the question comes from reality */
bench_batch_sweep();

Programming Exercises

Summary