Chapter 15 The Dynamic Buffer: xbuffer

Chapter Orientation

xbuffer and the previous chapter's xarray are fellow students of the same school: xarray holds "elements", xbuffer holds bytes. It is the standard building block of network packet receipt, protocol encoding, and file I/O — data arrives from every direction (variable length, variable timing), Append adds to the tail, Write writes at a fixed point (auto-zeroing past the end), Take takes the assembled content away with zero copies. This chapter explains these three actions thoroughly and distinguishes the easily confused pair "logical length" and "physical capacity".

15.1引入

Picture a TLS record decoder: data arrives segmented over TCP, one record possibly split across three packets; the header declares "record body length 512 bytes", and you want to know "how much is missing" when the header arrives, and take the whole record at once when the body completes. With a fixed buffer you must guess the maximum record length; with bare malloc splicing, every growth is an allocation and a copy; with xbuffer, Append accumulates segments, Size compares against the header's length, Take takes the complete record — the buffer zeroes and keeps assembling the next. This "accumulate — complete — take" loop is the universal shape of protocol processing, and xbuffer was designed for it.

One easily underestimated dividend: buffer zero-and-reuse decouples allocation counts from "connections × peak messages", making them the same order as "concurrent connections" — one buffer per connection, one Take per message, zero new allocations throughout. In Chapter 6's statistics, the pooled-to-backing ratio shows this curve's shape as soon as you run it.

Fixed-point Write solves another protocol pain: fixed layouts — it makes the natural style of "write the known fields first, backfill the fields that depend on earlier bytes" a two-phase encoding. "Byte 5 is the flag, bytes 10 to 14 the length field" — the body length is unknown when the header arrives; write the flag first, backfill the length later, and the hole in between is auto-zeroed. With a contiguous array you would compute offsets, grow, and zero the gap yourself; Write does it in one step.

15.2概念

The three core actions (memorize them first)

Size and Data are public fields — logical length and data pointer. Reads go straight to fields, edits through functions, per last chapter's conventions. Contrast with xarray's Count: the array's public field is the element count, the buffer's the byte count — "how many records" versus "how many bytes", a difference that decides their consumers: arrays consume structured records, buffers consume raw byte streams.

Logical length and physical capacity

Reserve(4096) arrives in one step when the magnitude is known — in a receive loop this single line drops allocations from "one per packet" to "one or two for the whole run". Resize explicitly changes the logical length (growth zero-fills, truncation cuts). This concept pair exists in every container but is clearest on the byte buffer.

Segmented storage and the coalescing moment

xbuffer's internal storage may be segmented — consecutive Appends do not necessarily live in one block of memory. The external contract is always "you get a contiguous block when taking": Take coalesces before handoff. The design trade-off is clear: the accumulation phase chases allocation efficiency (new segments hang on directly, no old data moved), the take phase alone pays for contiguity (one coalesce). If every Append maintained global contiguity, every arriving segment would move everything — a disaster in receive scenarios. Conversely, if after every Append you read the whole content, segmentation loses its point — that means you actually need "one buffer per message" rather than "one buffer accumulating many messages".

The relation to Chapter 67's xnetbuf is settled here: xnetbuf is the network layer's receive buffer (references, returns, pooling — Volume 7 goes deep), xbuffer a general byte accumulator. The common protocol pipeline is xnetbuf receives segments → copy or reference into xbuffer to complete → Take out a contiguous block for the parser. Different duties, cooperative, not interchangeable.

Take's ownership semantics

Take returns an owning contiguous block (bytes pointer + length out-param); the buffer's logical state zeroes and stays reusable; the returned block is freed with xrtFree. Note what "coalesce" means: with segmented internal storage, Take coalesces once before handing out the contiguous block; this guarantees you always receive contiguous memory — decoders never handle segmentation. This and Chapter 67's network-stream "receive buffer direct handoff" are the same design idea at different layers.

15.3示例

Complete program: append, sparse write, and handoff

From the repository example examples/containers/buffer/main.c:

 */

#include <stdio.h>

#include <xrt.h>



int main(void)
{
	xbuffer tBuffer;
	bytes pResult;
	size_t iSize;

	if ( !xrtBufferInit(&tBuffer) ) {
		return 1;
	}

	




	 */
	if (
		!xrtBufferAppend(&tBuffer, XRT_BYTES_LITERAL("abc")) ||
		!xrtBufferWrite(&tBuffer, 5, XRT_BYTES_LITERAL("z"))
	) {
		xrtBufferUnit(&tBuffer);
		return 2;
	}

	/* 逐字节十六进制转储：61 62 63 00 00 7a。 */
	for ( size_t i = 0; i < tBuffer.Size; i++ ) {
		printf("%02x%s", tBuffer.Data[i],
			i + 1u == tBuffer.Size ? "\n" : " ");
	}

	


	 */
	pResult = xrtBufferTake(&tBuffer, &iSize, NULL);
	if ( (pResult == NULL) || (iSize != 6) ) {
		xrtBufferUnit(&tBuffer);
		return 3;
	}
	xrtFree(pResult);
	xrtBufferUnit(&tBuffer);
	return 0;
}

What just happened. (1) XRT_BYTES_LITERAL("abc") builds a byte view (trailing zero auto-removed — Chapter 3's convention's counterpart in the byte world); after Append, Size = 3. (2) Write(5, "z") writes at offset 5 — past the then-end of 3, offsets 3 and 4 auto-zeroed: in the dump 61 62 63 00 00 7a the two 00s are the holes and 7a is 'z'. Fixed-layout protocol headers' "write the flag first, backfill the length" is exactly this posture. (3) Take takes all 6 bytes — ownership of a contiguous block returned, the iSize out-param carries the length; the buffer zeroes afterward and Unit is safe (no double free). (4) The taken block is freed with xrtFree — the owning convention fully consistent with Chapter 5.

Complete program: capacity and editing in full (the buffer's checkup sheet)

From examples/containers/buffer_tour/main.c, covering Reserve/Resize/Trim, the edit family, and three take styles:

 */

#include <stdio.h>
#include <string.h>
#include <xrt.h>

#define BV(x) ((xbytesview) { (cbytes)(x), sizeof(x) - 1u })

int main(void)
{
	xbuffer* pBuffer = NULL;
	xbuffer* pTaken = NULL;
	xbuffer* pFrom = NULL;
	bytes pData = NULL;
	bytes pSlot = NULL;
	bytes pWrite = NULL;
	int iResult = 1;

	/* ---- 容量族：Reserve / Resize / Trim ---- */
	pBuffer = xrtBufferCreate();
	if ( (pBuffer == NULL) ||
		!xrtBufferReserve(pBuffer, 16u) ||
		!xrtBufferResize(pBuffer, 4u) ||
		(xrtBufferView(pBuffer).Size != 4u) ||
		!xrtBufferTrim(pBuffer) ||
		(xrtBufferView(pBuffer).Size != 4u) ) {
		goto Cleanup;
	}
	/* Resize 扩张填零 + 收缩截断。 */
	if ( !xrtBufferResize(pBuffer, 2u) ||
		(xrtBufferView(pBuffer).Size != 2u) ) {
		goto Cleanup;
	}
	printf("buffer: reserve/resize/trim size=%zu\n",
		xrtBufferView(pBuffer).Size);

	/* ---- 编辑族 ---- */
	/* Assign：整体替换内容。 */
	if ( !xrtBufferAssign(pBuffer, BV("abcdef")) ||
		(xrtBufferView(pBuffer).Size != 6u) ||
		(memcmp(xrtBufferView(pBuffer).Data, "abcdef",
			6u) != 0) ) {
		goto Cleanup;
	}
	/* Insert：中段插入 "XY" → abXYcdef。 */
	if ( !xrtBufferInsert(pBuffer, 2u, BV("XY")) ||
		(xrtBufferView(pBuffer).Size != 8u) ||
		(memcmp(xrtBufferView(pBuffer).Data, "abXYcdef",
			8u) != 0) ) {
		goto Cleanup;
	}
	/* Remove：删 "XY" → abcdef。 */
	if ( !xrtBufferRemove(pBuffer, 2u, 2u) ||
		(xrtBufferView(pBuffer).Size != 6u) ||
		(memcmp(xrtBufferView(pBuffer).Data, "abcdef",
			6u) != 0) ) {
		goto Cleanup;
	}
	/* Add：尾部扩展 2 字节直写 "gh"。 */
	pWrite = xrtBufferAdd(pBuffer, 2u);
	if ( (pWrite == NULL) ||
		(xrtBufferView(pBuffer).Size != 8u) ) {
		goto Cleanup;
	}
	memcpy(pWrite, "gh", 2u);
	/* AppendByte：单字节追加。 */
	if ( !xrtBufferAppendByte(pBuffer, '!') ||
		(xrtBufferView(pBuffer).Size != 9u) ||
		(memcmp(xrtBufferView(pBuffer).Data, "abcdefgh!",
			9u) != 0) ) {
		goto Cleanup;
	}
	
	 * 返回首地址由调用方写入 → 写 "ZZZ" → abZZZcdefgh!。 */
	{
		bytes pSpace = xrtBufferInsertSpace(pBuffer, 2u, 3u);

		if ( (pSpace == NULL) ||
			(xrtBufferView(pBuffer).Size != 12u) ) {
			goto Cleanup;
		}
		memcpy(pSpace, "ZZZ", 3u);
		if ( memcmp(xrtBufferView(pBuffer).Data, "abZZZcdefgh!",
				12u) != 0 ) {
			goto Cleanup;
		}
	}
	printf("buffer: edit add+insert+remove+append assign=9\n");

	/* ---- Clear：清空有效内容，容量保留。 ---- */
	xrtBufferClear(pBuffer);
	if ( xrtBufferView(pBuffer).Size != 0u ) {
		goto Cleanup;
	}

	/* ---- 接管族 ---- */
	/* SetTake：接管 xrtMalloc 内存替换缓冲内容。 */
	pData = (bytes)xrtMalloc(6);
	if ( pData == NULL ) {
		goto Cleanup;
	}
	memcpy(pData, "takeit", 6u);
	pSlot = pData;
	if ( !xrtBufferSetTake(pBuffer, &pSlot, 6u, 6u) ||
		(pSlot != NULL) ||  /* 来源槽被清空 */
		(xrtBufferView(pBuffer).Size != 6u) ||
		(memcmp(xrtBufferView(pBuffer).Data, "takeit",
			6u) != 0) ) {
		goto Cleanup;
	}
	pData = NULL;
	/* CreateTake：创建新缓冲并接管。 */
	pData = (bytes)xrtMalloc(3);
	if ( pData == NULL ) {
		goto Cleanup;
	}
	memcpy(pData, "own", 3u);
	pSlot = pData;
	pTaken = xrtBufferCreateTake(&pSlot, 3u, 3u);
	if ( (pTaken == NULL) ||
		(pSlot != NULL) ||
		(xrtBufferView(pTaken).Size != 3u) ||
		(memcmp(xrtBufferView(pTaken).Data, "own",
			3u) != 0) ) {
		goto Cleanup;
	}
	pData = NULL;
	/* From：复制创建（来源不受影响）。 */
	pFrom = xrtBufferFrom(BV("cp"));
	if ( (pFrom == NULL) ||
		(xrtBufferView(pFrom).Size != 2u) ||
		(memcmp(xrtBufferView(pFrom).Data, "cp", 2u) != 0) ) {
		goto Cleanup;
	}
	printf("buffer: take set-take=6 create-take=3 from=2\n");
	iResult = 0;

Cleanup:
	xrtFree(pData);
	xrtBufferDestroy(pFrom);
	xrtBufferDestroy(pTaken);
	xrtBufferDestroy(pBuffer);
	return iResult;
}

What just happened. (1) Line one verifies the capacity trio: Reserve pre-grows, Resize sets length explicitly, Trim tightens with Size semantics unchanged — note the detail "Size unchanged after Trim": the capacity cut removes only idle space, written content stays as is. (2) Line two verifies the edit family: add, insert, remove, append, and assign (Assign wholesale-replaces content) execute in order with correct values. (3) Line three compares three take postures: Take on an existing buffer, construct-then-take on a fresh buffer, and conversion from a view — three paths converging on a contiguous owning block. Ninety percent of network code uses the first: one reused buffer + one Take per message.

Putting the three actions into a real rhythm

Alone, Append/Write/Take are three functions; inside protocol processing's rhythm they are three beats of one loop. Take "each message = 4-byte length header + body": on any arriving segment, first Append into the buffer; use Size against "the header's declared length already assembled" to judge a message's completeness; when complete, Take it for the parser and the buffer zeroes for the next. In the three-beat loop the buffer is the sole state carrier — no message list, no per-message allocation, no double copying. In Chapter 67 you will see this loop running at a million-per-second cadence inside network-engine callbacks, shaped exactly like the toy version here.

15.4契约

• Public fields: Size (logical length) and Data (data pointer) read directly; edits via functions.
• Write zero-fill: holes past the end auto-zero; the logical length jumps to the write's end.
• Take handoff: returns an owning contiguous block (freed with xrtFree); the buffer zeroes and is reusable; Unit never double-frees.
• No side effects on failure: a failed edit leaves the buffer untouched; cleanup with Unit is always safe.
• Capacity discipline: known magnitude, Reserve first; before long residency, Trim; Trim cuts only idle space, never written content.

Common Mistakes

Both pitfalls orbit one thing: Size and Data are "snapshots at the moment of reading", and any edit (Append/Write/Take) can stale the snapshot. Understand this and the two pitfalls are one pitfall in two guises.

size_t iWant = tBuffer.Size;          /* saved the old length */
bytes pBlock = xrtBufferTake(&tBuffer, &iSize, NULL);
process(pBlock, iWant);               /* iWant is the pre-Take length? correct only by coincidence */

bytes pBlock = xrtBufferTake(&tBuffer, &iSize, NULL);
if ( pBlock == NULL ) {
	return false;
}
process(pBlock, iSize);              /* length from Take's out-param — always this take's true length */

bytes pView = tBuffer.Data;            /* noted the pointer */
xrtBufferAppend(&tBuffer, pMore, 4096);/* growth relocates */
consume(pView, tBuffer.Size);          /* pView dangles */

xrtBufferAppend(&tBuffer, pMore, 4096);
consume(tBuffer.Data, tBuffer.Size);  /* take at use; pointer and length same source, same time */

Programming Exercises

Summary