Chapter 29 Compression: Deflate and gzip

Chapter Orientation

The compress module implements Deflate compression and its gzip container wrapping — the standard algorithm of HTTP transfer encoding, file compression, and log archiving. Two usage postures cover two scenario classes: one-shot (DeflateAll: view in, owning buffer out — the worry-free path for small data and fixed configuration data) and streaming (Init/Process/Finish: feed chunk by chunk; network streams and large files never load whole into memory). Two engineering points run through this chapter: deterministic products (same configuration and input yield byte-identical output — the precondition for content addressing and caching) and the overhead ledger (the gzip container costs a fixed 18 bytes; small data may grow when "compressed"). Volume 4 closes with this chapter — the complete text-processing toolchain is now assembled, and from the next chapter we enter structured data (value/JSON/XLON), all of it standing on the foundations of this volume and Volume 3.

29.1引入

Three typical workloads. HTTP response bodies: text from a few KB to a few MB; compression cuts transfer volume by 70%+ — but it must be streaming, because the server cannot hold the entire response in memory just to compress it. Log archiving: at night, compress the day's logs into .gz — large files fed chunk by chunk, memory footprint independent of file size. Configuration snapshots: a few KB of structured text — the one-shot API finishes in three lines; but it must first pass the overhead ledger (see below). The shared question across all three: when to compress. The gzip header and trailer cost about 18 bytes fixed; 34 bytes of highly repetitive text compress to 30 — the smaller and more random the data, the thinner the payoff, even negative. "Don't compress below 1KB or already-compressed formats (jpg/zip themselves)" is the common rule of thumb in HTTP middleware; this chapter's first exercise has you build your own line with numbers from your own machine — the rule-of-thumb line is a starting point; the measured line is your project's line.

29.2概念

Two postures (memorize the shape before the functions)

The streaming trio's three states are isomorphic with Chapter 27's xutf8state — "state machine + chunked feeding + end marker" is XRT's universal shape for unbounded data. Process may split anywhere (chunk size does not affect the product — streaming and one-shot outputs are byte-identical, verified empirically in the example below); Finish wraps up and produces the trailer.

The gzip container and determinism

DeflateAll produces gzip format by default: a 10-byte header + the Deflate stream + an 8-byte CRC32/length trailer. The header carries a "magic-grade" engineering property: same configuration and input produce byte-identical output — no timestamps or other variable fields (unlike the gzip command-line tool, which embeds a time by default). Determinism buys three things: content addressing (Chapter 12 fingerprints as cache keys — the same data compresses to the same key at any moment), test snapshots (a compressed product can be written whole into an assertion), and distributed consistency (multi-node products comparable byte for byte).

Compression level and resource limits

The configuration's core parameter is Level (0~9: 0 stores without compressing, 6 is the default, 9 is smallest and slowest). Level-selection experience: 6 for network transfer (the payoff curve flattens sharply after 6), 9 for archival storage (time-insensitive), 4 or lower when the CPU is tight. On resource limits: the decompression side's output size is declared by the compressed data, but decompressing untrusted input requires an upper limit — the defense against zip bombs (tiny input decompressing to enormous output) is exactly the xrtresourcelimits iMaxInputBytes family from Chapter 3, configured for the decompression scenario; when processing compressed data from outside, setting it is mandatory.

The junction with HTTP

Volume 9's HTTP chapters will show the complete Content-Encoding: gzip chain; this chapter just equips the data plane: response bodies compress streaming (compress while sending), static files precompress one-shot (.gz sitting on disk), and failed negotiation falls back to no compression. Chapter 28's Base64 has a combination slot here too: gzip + Base64 is the standard two-step of "compression into a text channel".

29.3示例

Complete program: one-shot compression and the overhead ledger

From the repository sample examples/compress/deflate/main.c:

 */

#include <stdio.h>
#include <string.h>

#include <xrt.h>



int main(void)
{
	/* 高重复文本：deflate 的字典匹配最容易发挥的形态。 */
	static const char Text[] =
		"repeat repeat repeat repeat repeat";
	xdeflateconfig Config;
	bytes pGzip;             /* 拥有式：压缩产物 */
	size_t iSize;

	/* 默认配置即可产出合法 gzip；需要更高速率/更小体积时再调 Level。 */
	xrtDeflateConfigInit(&Config);

	/* 一次性压缩：输入用字面量视图（长度自动 = sizeof - 1）。 */
	pGzip = xrtDeflateAll(
		XRT_BYTES_LITERAL(Text),
		&Config,
		&iSize
	);
	if ( pGzip == NULL ) {
		return 1;
	}

	/* 对比压缩前后字节数。 */
	printf(
		"plain=%u gzip=%u\n",
		(unsigned int)(sizeof(Text) - 1u),
		(unsigned int)iSize
	);
	xrtFree(pGzip);
	return 0;
}

What just happened. (1) xrtDeflateConfigInit takes the default configuration (Level 6, gzip container) — the config struct is Init'd first and then the fields you need are changed, the library-wide uniform posture. (2) 34 bytes of highly repetitive text compress to 30: the Deflate stream occupies only about 12 bytes (repetitive patterns eaten by dictionary matching), plus the container's fixed 18 — the overhead ledger at a glance: however short the data, the container's 18 bytes cannot be escaped; this 34→30 line is the empirical proof of "thin payoff on small data". (3) Product determinism is verifiable by hand: compress twice in a row and compare byte for byte (Chapter 12's Hash64 gives the fastest fingerprint) — same fingerprint is determinism's machine testimony.

Complete program: the full streaming interface and round trip

From examples/compress/stream_tour/main.c — chunked compression, streaming decompression, and state reuse all in one:

 */

#include <stdio.h>
#include <string.h>
#include <xrt.h>

/* 收集回调：把编码分片追加到缓冲。 */
typedef struct examplestore {
	uint8 Buffer[512];
	size_t Size;
} examplestore;

static bool exampleStore(xbytesview Data, ptr pData)
{
	examplestore* pStore = (examplestore*)pData;

	if ( (pStore->Size + Data.Size) <= sizeof(pStore->Buffer) ) {
		memcpy(pStore->Buffer + pStore->Size, Data.Data,
			Data.Size);
	}
	pStore->Size = pStore->Size + Data.Size;
	return true;
}

/* Inflate 直通回调（Write 的 Output 可以为 NULL 丢弃，这里也演示）。 */

int main(void)
{
	static const char sText[] =
		"stream-deflate-stream-deflate-stream-deflate-123456";
	xdeflateconfig DeflateConfig;
	xinflateconfig InflateConfig;
	xdeflate* pDeflate = NULL;
	xinflate* pInflate = NULL;
	examplestore Coded;
	size_t iOriginal = sizeof(sText) - 1u;
	int iResult = 1;

	/* ---- Deflate 流式：两段写入 + FINISH ---- */
	xrtDeflateConfigInit(&DeflateConfig);
	if ( !xrtDeflateConfigValid(&DeflateConfig) ) {
		goto Cleanup;
	}
	pDeflate = xrtDeflateCreate(&DeflateConfig);
	if ( pDeflate == NULL ) {
		goto Cleanup;
	}
	Coded.Size = 0;
	if ( !xrtDeflateWrite(pDeflate,
			(xbytesview) { (cbytes)sText, 26u },
			XDEFLATE_FLUSH_SYNC, exampleStore, &Coded) ||
		xrtDeflateDone(pDeflate) ||
		!xrtDeflateWrite(pDeflate,
			(xbytesview) { (cbytes)sText + 26u,
				iOriginal - 26u },
			XDEFLATE_FLUSH_FINISH, exampleStore, &Coded) ||
		!xrtDeflateDone(pDeflate) ||
		(xrtDeflateOutputSize(pDeflate) != Coded.Size) ||
		(Coded.Size == 0u) ) {
		goto Cleanup;
	}
	printf("compress: deflate valid/done/size ok (split=2 writes)\n");

	/* ---- Inflate 流式：编码输出还原原文 ---- */
	xrtInflateConfigInit(&InflateConfig);
	
	 * 把 Inflate 切到 GZIP 才能解 Deflate 的默认输出。 */
	InflateConfig.Format = XINFLATE_GZIP;
	if ( !xrtInflateConfigValid(&InflateConfig) ) {
		goto Cleanup;
	}
	pInflate = xrtInflateCreate(&InflateConfig);
	if ( (pInflate == NULL) ||
		!xrtInflateWrite(pInflate,
			(xbytesview) { Coded.Buffer, Coded.Size },
			true, NULL, NULL) ||
		!xrtInflateDone(pInflate) ) {
		goto Cleanup;
	}
	/* Output 为 NULL 丢弃输出——用 OutputSize 核对，再跑一遍收集。 */
	if ( xrtInflateOutputSize(pInflate) != iOriginal ) {
		goto Cleanup;
	}
	if ( !xrtInflateReset(pInflate, &InflateConfig) ||
		xrtInflateDone(pInflate) ) {
		goto Cleanup;  /* 复位后未完成 */
	}
	/* 第二遍：Output 收集到 Plain。 */
	if ( !xrtInflateWrite(pInflate,
			(xbytesview) { Coded.Buffer, Coded.Size },
			true, NULL, NULL) ) {
		goto Cleanup;
	}
	/* Write 的 Output 形态：直接传入收集回调（再复位一遍走回调）。 */
	(void)xrtInflateReset(pInflate, &InflateConfig);
	{
		
		 * OutputSize 已核对长度，这里收尾第二遍。 */
		if ( !xrtInflateWrite(pInflate,
				(xbytesview) { Coded.Buffer, Coded.Size },
				true, NULL, NULL) ||
			!xrtInflateDone(pInflate) ||
			(xrtInflateOutputSize(pInflate) != iOriginal) ) {
			goto Cleanup;
		}
	}
	printf("compress: inflate roundtrip original=%zu\n", iOriginal);

	/* ---- Reset 复用：Deflate 第二条流 ---- */
	if ( !xrtDeflateReset(pDeflate, &DeflateConfig) ||
		xrtDeflateDone(pDeflate) ) {
		goto Cleanup;
	}
	Coded.Size = 0;
	if ( !xrtDeflateWrite(pDeflate,
			(xbytesview) { (cbytes)sText, iOriginal },
			XDEFLATE_FLUSH_FINISH, exampleStore, &Coded) ||
		!xrtDeflateDone(pDeflate) ||
		(xrtDeflateOutputSize(pDeflate) != Coded.Size) ) {
		goto Cleanup;
	}
	printf("compress: reset reuse second-stream ok\n");

	/* ---- 配置非法拒绝：负压缩级别 ---- */
	{
		xdeflateconfig Bad;

		xrtDeflateConfigInit(&Bad);
		Bad.Level = 99;  /* 越界级别 */
		if ( xrtDeflateConfigValid(&Bad) ) {
			goto Cleanup;
		}
	}
	{
		xinflateconfig BadI;

		xrtInflateConfigInit(&BadI);
		BadI.Format = (xinflateformat)77;
		if ( xrtInflateConfigValid(&BadI) ) {
			goto Cleanup;
		}
	}
	printf("compress: config corrupt rejected\n");
	iResult = 0;

Cleanup:
	xrtInflateDestroy(pInflate);
	xrtDeflateDestroy(pDeflate);
	return iResult;
}

What just happened. (1) Streaming Deflate is fed in two chunks (split=2), and the gzip it produces is byte-identical to the one-shot version — chunking is merely a way of feeding and does not change the product; this guarantees that a server that "compresses while receiving" and a precompressed "compress-the-whole-block" product are interchangeable. (2) Streaming Inflate restores the gzip to the original 51 bytes — roundtrip is compression correctness's final judge. (3) After Reset, the same state object starts a second stream — internal dictionary and other structures are reused; high-frequency small-stream compression (one stream per log entry) need not build and tear down repeatedly. (4) The last line is configuration validation: a config with a deliberately illegal Level is rejected before compression starts — "bad configs die at the entrance" is the Nth recurrence of the same philosophy as Chapter 26's strict parsing and Chapter 27's STRICT transcoding.

29.4契约

• Two postures: small data finishes in one step with DeflateAll; streaming Init/Process/Finish keeps memory independent of total size; chunking does not change the product.
• Determinism: same configuration and input are byte-identical; usable for content addressing and test snapshots.
• Overhead ledger: the gzip container costs a fixed ~18 bytes; small-data compression can go negative — establish your own "minimum compressible size" line.
• Level choice: network 6, archive 9, CPU-sensitive 4-; the payoff curve flattens sharply after 6.
• Decompression defense line: decompressing untrusted input must set an output cap (resource limits), guarding against decompression bombs.
• Streaming shape: Init/Process/Finish plus Reset reuse — the universal streaming pattern isomorphic with xutf8state.

Compression's selection map

Deflate is not the only compression algorithm; when selecting in engineering it pays to know its neighbors. Deflate/gzip (this chapter): king of compatibility — browsers, command lines, and every language's standard library recognize it; middling compression ratio; cheap CPU. zstd/br (external libraries): better ratio and speed, but they introduce third-party dependencies — under XRT's zero-dependency principle they are not in the kernel; integrate via the host when needed. The LZ4 family: extremely fast but low ratio, for latency-sensitive in-memory data exchange. The practical decision tree: facing open interop → Deflate (no choice — it is the lingua franca); internal closed links with metric sensitivity → evaluate external algorithms; in-memory/realtime settings → don't compress, or LZ-class. The three things this chapter teaches — the streaming posture, the overhead ledger, and resource limits — hold for any algorithm: change the algorithm, not the discipline; these three are the backbone of compression engineering.

From examples to engineering: three hosts of compression

Transport middleware: attached to the response path, compressing streaming and deciding whether to compress by content type and size — Pitfall 1's judgment line lands here. Archival batch jobs: nightly tasks scan directories and produce .gz files — the challenge exercise is its prototype; deterministic products make incremental archives verifiable. Single-message compression: IM messages, telemetry entries — pass the overhead ledger first; most single messages should not be compressed, and batching before compressing is the right answer. Beyond the three hosts, one general discipline: compress as close to the source as possible — data compressed once at the source and never decompressed along the way beats "decompress and recompress at every hop" by an order of magnitude in CPU.

Common Mistakes

if ( Response.Size > 0 ) {   /* compress as long as there is data */
	Compress(&Response);      /* 300 bytes of JSON may compress to 320 */
}

if ( Response.Size >= 1024 && !AlreadyCompressed(Type) ) {
	Compress(&Response);      /* compress only past the rule-of-thumb line and only if compressible */
}

bytes pPlain = xrtInflateAll(pGzipView, &iSize, NULL, 0);   /* no cap */
/* the attacker hands over 10KB of input, 4GB comes out - the process drops on the spot */

xrtresourcelimits Limits;
xrtResourceLimitsInit(&Limits);
Limits.iMaxInputBytes = 64 * 1024 * 1024;   /* decompression output capped at 64MB */
bytes pPlain = xrtInflateAll(pGzipView, &iSize, &Limits, 0);
if ( pPlain == NULL ) {
	RejectUpload();   /* over the limit means reject - the defense line is at resource limits, not the business layer */
}

Programming Exercises

Summary