Chapter 34 JSONL and XLONL: Line-Delimited Data

Chapter Orientation

Chapters 32 and 33 solved how "one document" moves in and out of the value tree; engineering has an equally common second shape — "a sequence of records": server logs with one event per line, bulk imports with one record per line, inter-process exchange with one message per line. JSONL (JSON Lines) turns each line into one complete JSON value; XLONL turns each line into one complete XLON value — many lines together form an xvalue Array, while each line stays individually parseable, locatable, and skippable. This is not a new parser: per-record encoding rules, strictness, and error domains are reused from Chapters 32/33 verbatim; what is added is exactly three things — line framing semantics, cumulative budgets across records, and the translation of per-record error positions into global line numbers and record indices. When to use "one big document" versus "a sequence of records" — this chapter gives the decision rules.

34.1引入

Three scenarios tell the same story: the natural unit of the data is the line, not the document. Server logs: each request appends one event line — replay and statistics parse line by line, bad lines are skipped with their position recorded, and line 4096's stray comma must never discard the whole file. Bulk import: ten million records, one per line — Chapter 32's DOM turns the whole file into one tree (memory multiplied several times over) and SAX streams but has no notion of "record N"; a line-delimited format makes "parse failed at record N" a first-class citizen. Inter-process exchange: internal pipes deliver messages line by line — JSON lines interoperate externally, XLON lines (Chapter 33's bytes/time/set/intmap) round-trip full types internally. All three hosts share one API surface: read into one Array, write as a sequence of lines, errors carry line numbers — framing is the format's business; the semantics remain the value tree's semantics.

34.2概念

Line framing: what counts as one record

The rule worth memorizing is the last one: one line must be exactly one complete value. This is where the "record" semantics come from — the line is both the framing boundary and the validation boundary; {"id":1} {"id":2} squeezed onto one line is an error (one root value too many), and {"id":\n1} spread across lines is an error (not even one complete root value). An escaped \n inside a string does not participate in framing — "{\"a\":1}\n" is one valid string record.

Two-level budgets: per-record inheritance, whole-stream accounting

The two scopes that trip people most: the whole-output budget counts the LF (with MaxOutputBytes=1, the first record "1" commits, and its LF triggers LIMIT); the per-record budget excludes separators (a 1 MiB-max line is 1 MiB to MaxInputBytes). Defaults follow the library-wide convention: 64 MiB overall, 1000000 records and syntax values — tighten before exposing to untrusted input.

Global error location: three coordinates and one index

A per-record error (Chapters 32/33) carries in-record line/column; the line-delimited format translates it into global coordinates — xrtJsonlErrorLocation yields zero-based byte offset, one-based physical line, one-based byte column, and zero-based record index. A blank-line error in strict mode points at the index of the "next pending record"; skipped blank lines still count toward the physical line number — line=2 record=1 means "line 2 is blank and it should have been record 1". The cause chain keeps the specific per-record reason and Kind; the wrapper only adds a shell at RECORD/OUTPUT/IO.

Writing: two forms — callback and file

Serialization writes each Array element as one compact line: the callback form (xrtJsonlWrite) hands chunks to a sink, borrowed bytes valid only during the callback — for streaming straight into a network writer; the file form (xrtJsonlWriteFile) completes the whole serialization in memory first and then atomically replaces the target file — on serialization failure the original file remains untouched. Shared semantics of both forms: compact + LF (PRETTY is rejected), the root must be XVALUE_ARRAY, and configurations are snapshotted by value (mutating the config inside a callback does not affect the in-flight call).

JSONL, XLONL, or neither

The decision variable is isomorphic to Chapter 33's: receiver under your control → XLONL for full types; not under control → JSONL for interop; if the data is really "one configuration", don't split it into lines — and if it is really "a stream of events", don't cram it into one big array.

34.3示例

Complete program: JSONL round trip and error location

From the repository sample examples/data/jsonl/main.c — parse, validate, read back, callback writing, atomic file round trip, and blank-line location in one pass:

 */
#include <stdio.h>
#include <xrt.h>

static bool discard(xbytesview Data, ptr pUserData)
{
	(void)Data;
	(void)pUserData;
	return true;
}

int main(void)
{
	xjsonlreadconfig Read;
	xjsonlwriteconfig Write;
	xjsonllocation Location;
	xvalue* pArray = NULL;
	xvalue* pRead = NULL;
	str Text = NULL;
	size_t Size = 0;
	cstr Path = ".xrt-jsonl-example.jsonl";
	bool Created = false;
	int Result = 1;

	xrtJsonlReadConfigInit(&Read);
	xrtJsonlWriteConfigInit(&Write);
	pArray = xrtJsonlParse(XRT_STR_LITERAL("{\"id\":1}\n\n[2,3]\r\nnull\n"));
	if ( pArray == NULL ) goto done;
	Text = xrtJsonlStringify(pArray, &Size);
	if ( Text == NULL ) goto done;
	if ( !xrtJsonlValid((xstrview){ Text, Size }) ) goto done;
	pRead = xrtJsonlRead((xstrview){ Text, Size }, &Read);
	if ( pRead == NULL ) goto done;
	printf("JSONL: %zu records, %zu bytes\n", xrtValueCount(pRead), Size);
	xrtValueRelease(pRead);
	pRead = NULL;
	if ( !xrtJsonlWrite(pArray, &Write, discard, NULL) ) goto done;
	if ( !xrtJsonlStringifyFile(Path, pArray) ) goto done;
	Created = true;
	pRead = xrtJsonlParseFile(Path);
	if ( pRead == NULL ) goto done;
	xrtValueRelease(pRead);
	pRead = NULL;
	if ( !xrtJsonlWriteFile(Path, pArray, &Write) ) goto done;
	pRead = xrtJsonlReadFile(Path, &Read);
	if ( pRead == NULL ) goto done;
	xrtValueRelease(pRead);
	pRead = NULL;
	Read.Flags |= XJSONL_READ_REJECT_EMPTY_LINES;
	pRead = xrtJsonlRead(XRT_STR_LITERAL("{}\n\n"), &Read);
	if ( pRead != NULL ) goto done;
	if ( !xrtJsonlErrorLocation(xrtGetError(), &Location) ) goto done;
	printf("empty line: line=%zu record=%zu\n", Location.Line, Location.RecordIndex);
	xrtClearError();
	Result = 0;
done:
	xrtValueRelease(pRead);
	xrtValueRelease(pArray);
	xrtFree(Text);
	if ( Created && !xrtFileDelete(Path) ) Result = 1;
	return Result;
}

What just happened. (1) Framing in evidence: the input mixes blank lines with CRLF/LF — all 3 records (object, array, null) arrive intact, and 20 bytes is the actual compact output size (LF included per record). (2) Blank-line policy: with skipping on by default, Valid/Read pass; turning on XJSONL_READ_REJECT_EMPTY_LINES makes "{}\n\n" fail immediately at the blank line before the second record. (3) Global location: the error gives line=2 record=1 — the blank line is on line 2 and it cuts the position after record 1; the caller learns "which line to fix", not a byte range. (4) File loop: StringifyFile atomically replaces → ParseFile reads back → WriteFile (with config) replaces again → ReadFile reads again — any failed step leaves the file in its last complete state.

Complete program: XLONL line-by-line reading and writing

From examples/data/xlonl/main.c — the same framing and budget machinery, with per-record rules swapped to XLON:

 */
#include <stdio.h>
#include <xrt.h>

static bool discard(xbytesview Data, ptr pUserData)
{
	(void)Data;
	(void)pUserData;
	return true;
}

int main(void)
{
	xxlonlreadconfig Read;
	xxlonlwriteconfig Write;
	xxlonllocation Location;
	xvalue* pArray = NULL;
	xvalue* pRead = NULL;
	str Text = NULL;
	size_t Size = 0;
	cstr Path = ".xrt-xlonl-example.xlonl";
	bool Created = false;
	int Result = 1;

	xrtXlonlReadConfigInit(&Read);
	xrtXlonlWriteConfigInit(&Write);
	pArray = xrtXlonlParse(XRT_STR_LITERAL("{\"id\":1}\n\n[2,3]\r\nnull\n"));
	if ( pArray == NULL ) goto done;
	Text = xrtXlonlStringify(pArray, &Size);
	if ( Text == NULL ) goto done;
	if ( !xrtXlonlValid((xstrview){ Text, Size }) ) goto done;
	pRead = xrtXlonlRead((xstrview){ Text, Size }, &Read);
	if ( pRead == NULL ) goto done;
	printf("XLONL: %zu records, %zu bytes\n", xrtValueCount(pRead), Size);
	xrtValueRelease(pRead);
	pRead = NULL;
	if ( !xrtXlonlWrite(pArray, &Write, discard, NULL) ) goto done;
	if ( !xrtXlonlStringifyFile(Path, pArray) ) goto done;
	Created = true;
	pRead = xrtXlonlParseFile(Path);
	if ( pRead == NULL ) goto done;
	xrtValueRelease(pRead);
	pRead = NULL;
	if ( !xrtXlonlWriteFile(Path, pArray, &Write) ) goto done;
	pRead = xrtXlonlReadFile(Path, &Read);
	if ( pRead == NULL ) goto done;
	xrtValueRelease(pRead);
	pRead = NULL;
	Read.Flags |= XXLONL_READ_REJECT_EMPTY_LINES;
	pRead = xrtXlonlRead(XRT_STR_LITERAL("{}\n\n"), &Read);
	if ( pRead != NULL ) goto done;
	if ( !xrtXlonlErrorLocation(xrtGetError(), &Location) ) goto done;
	printf("empty line: line=%zu record=%zu\n", Location.Line, Location.RecordIndex);
	xrtClearError();
	Result = 0;
done:
	xrtValueRelease(pRead);
	xrtValueRelease(pArray);
	xrtFree(Text);
	if ( Created && !xrtFileDelete(Path) ) Result = 1;
	return Result;
}

What just happened. The flow is line-for-line isomorphic to the jsonl sample — not laziness but a design promise made visible: swap the format and the framing, budget structure, and error location all stay; the API shapes correspond one to one. The difference is per-record semantics: each line here is a complete XLON value, and Chapter 33's four extended types go straight into the line stream —

/* Full-type records in an XLONL line stream: after parsing, each lands in place in the value tree */
xvalue* pLog = xrtXlonlParse(XRT_STR_LITERAL(
    "{\"key\":bytes(\"AAEC/w==\"),\"at\":time(\"2026-07-31T00:00:00Z\")}\n"
    "set[80, 443]\n"));
/* pLog[0] is an object (Bytes/Time fields), pLog[1] is a set container — line-by-line replay with zero conversion */

When the built-in bytes tag needs its own total cap, MaxTotalDecodedBytes intercepts before buffer allocation (default 64 MiB) — stopping the "a little per line, a mountain in aggregate" decode bomb.

34.4契约

• One value per line: each line is exactly one complete root value; escaped newlines inside strings do not frame; the last record may lack a newline.
• Failure atomicity: reads never return a partial Array (failures free all partial results); file writes serialize fully, then replace atomically — failures never touch the original.
• Budget scopes: whole input/output include separators and blank lines; per-record does not; output includes each LF — all three scopes are promises, not implementation details.
• Borrowing and ownership: input text and config are borrowed until the call returns; the result Array is released with xrtValueRelease, strings with xrtFree; writing never consumes the caller's references (iteration over a backing snapshot).
• Threading: no implicit locks; each call owns an independent workspace; the caller synchronizes shared mutable input.
• Committed bytes are not rolled back: record bytes already delivered before a synchronous-callback failure cannot be recalled — see Pitfall 2.

From examples to engineering: three hosts of line-delimited data

Server logs (most common): the writer Stringify-writes one line per event, appending or streaming via Write into a sink; the reader Parse-reads into an Array for offline statistics, sending bad lines to an audit log via ErrorLocation and carrying on. Bulk import: Read yields all valid records at once (any failure fails the whole batch, reported by line), or the text is chunked and Parse-processed per chunk for "skip bad chunks, keep importing" — both semantics stand on "the line is the boundary". Inter-process line streams: isomorphic to Chapter 33's rules — JSONL outside, XLONL inside; line streams naturally fit text pipelines (grep, redirection, line-oriented gateways), an operational property document formats cannot offer.

A contrast: the same batch of events, two ways on disk

A batch of request logs can land as one big JSON array (Chapter 32) or as JSONL. Array form: appending means read-modify-rewrite (or the fragile trick of patching a trailing ]); one bad spot makes the whole file unparseable; streaming consumers must wait for the document to close. JSONL form: appending is a plain AppendFile; a corrupted line damages only itself; tail -f consumes as data arrives. The point of the contrast: append-only writing and tolerance of partial corruption are the essential advantages of record streams — as long as the data is an append-only event sequence, the line-delimited format is the right shape; conversely, configurations that need whole-file reading with strongly related fields remain document territory.

Common Mistakes

/* A value the format cannot express is mixed into the tree; hoping SKIP skips the whole record */
xrtJsonlStringify(pArray, &Size);   /* whole call fails: an inexpressible root cannot be skipped */

/* Filter before writing: remove inexpressible records, or degrade them to placeholder objects by convention */
/* External channel: all-JSON types; internal channel: just use XLONL — extended types are natively expressible */
str Text = xrtXlonlStringify(pArray, &Size);

/* Retry in place; the committed half batch is written again */
while ( !xrtJsonlWrite(pArray, &Write, sink, Ctx) )
    ;   /* committed bytes cannot be recalled — retry = duplicated records */

 * (atomic replacement: all-or-nothing by construction) */
if ( !xrtJsonlWriteFile(Path, pArray, &Write) ) { /* original untouched; fix the data and start over */ }

Programming Exercises

Summary