The JSON integration
the engine, pointed at bytes
You have a few gigabytes of NDJSON and one question: total revenue where amount > 150.
Here is the entire program, a case class (Scala's record type) and a query. No codec, no
derives clause (Scala's compile-time codec derivation), no registration; the case
class is the schema:
you write
And here is how that measures against the field, including jsoniter-scala (the fastest JVM JSON
library we know of) driven by its fastest idiom, a hand-written, zero-object JsonReader loop:
measured: JMH, -prof gc
| throughput | allocation | |
|---|---|---|
| ▲fuse (generated scanner) | 391 ops/s | 1.0 B/op |
| jsoniter: hand-written reader, no object | 241 ops/s | 1.7 B/op |
| jsoniter: full codec (parses all 20 fields) | 136 ops/s | 6 464 003 B/op |
| Jackson: tree model (JsonNode) | 94 ops/s | 31 047 492 B/op |
| jawn: AST (JValue) | 45 ops/s | 45 485 209 B/op |
fuse beats jsoniter-scala's fastest idiom 1.6×, and laps the AST parsers 4–9×, at ~zero allocation
Nothing on this page is a JSON library, and none of it is on FArray's roadmap. It exists to answer one question: is the fusion engine about collections, or about pipelines? The engine was pointed, unchanged, at raw bytes. This page measures how far the machinery reaches, and serves as the worked example of how a format plugs in.
Why that number is possible
A JSON object is a product whose fields are columns, sourced from byte ranges instead of
tuple slots. That makes every trick from the optimizer page apply
verbatim, at the byte level: the macro (the code generator that runs inside the compiler, behind
the terminal call) computes the live set (the fields your filters, maps and
aggregates touch) and emits a per-record scanner where every live field gets a typed byte
reader and every other field is skipped without being decoded. A 20-field record queried on one
field costs one number-parse and nineteen byte-skips. No Event object is ever built.
The queries on this page run against a 10 000-record, ~3.2 MB NDJSON buffer of that 20-field
Event: realistic field mix, every query touching 1–2 fields, the projection case where pushdown pays.
Throughput is ops/s over the whole buffer (÷10 000 ≈ per record); every rival runs that library's
best idiom for the same query.
What Json.ndjson[Event](src).stream tells the compiler
The one-liner is doing three jobs at once, and naming them is the key to the whole page. The
expression has type Fuse[Event, Ndjson], and it delivers three different kinds of information to
three different consumers:
1. A runtime value: the bytes. Json.ndjson[Event](src) constructs new NdjsonSource[Event](src, 0, src.length):
a tiny object holding your byte array and implementing the chunk/record contract (for in-memory
bytes, the trivial version: one chunk, records framed by newline scans). Nothing is parsed and no
schema is stored; this object is data, and you can see it again, verbatim, spliced into the top of
the emitted loop in every golden lowering on this page.
2. A compile-time type: Event, the schema. NdjsonSource[T] never stores T; there is no
ClassTag, no field for it, nothing at runtime. Event exists purely in the type, riding along
until a terminal macro runs, and then it becomes the schema: the decoder reflects Event's
case-class fields at compile time. The field names become the interned key bytes the scanner
compares against (JsonScanner.internKey("amount") in the lowering is Event's field name, made
bytes); the field types choose the readers (Double picks readDoubleAt). Change a field name
in the case class and the generated scanner changes with it. The case class is the schema in the
most literal sense available on the JVM.
3. A compile-time type: Ndjson, the router. .stream wraps the source as
Fuse[Event, Ndjson], and Ndjson is a phantom: no values of it exist. Its only job is implicit
routing: the given in Ndjson's companion puts JsonLowering's terminal methods on every
Fuse[_, Ndjson], which is why .sum and .find exist on this value with no imports, why .run
doesn't exist on it at all, and why calling any terminal lands in the hook that names JsonDecode.
The shape type answers "which format module handles this pipeline", at compile time, by scope.
One more piece makes it click: ndjson and stream are inline, so when the terminal macro
receives the pipeline, the receiver tree still contains the new NdjsonSource[Event](...)
construction. The engine looks at that tree's type and switches drivers on it:
// FuseMacro, the actual detection:
if srcTerm.tpe.widen.dealias <:< TypeRepr.of[ByteRecordSource]
then Some('{ $selfBase.asInstanceOf[ByteRecordSource] }) // → the chunk/record driver + decoder
else None // → the array/chunk drivers
So "how can JSON work with fusion" has a short answer: fusion never needed a collection. Stages are markers that never run; the engine needs exactly two things from any source — how to iterate it (the driver, chosen by the source's type in the tree) and how to read one element's columns (the decoder, chosen by the shape type). A byte range plus a schema type answers both, at compile time. The four pieces behind this and the contract at each seam are dissected on Integrating with fusion; this page is the proof they work.
Three entry points, one per place your bytes live; the file and stream variants run in constant memory (one block + the largest record), with the source closed on every exit path:
you write
The demos
Each isolates one optimizer behavior over bytes, with the generated scanner and, where a fair rival exists, that library's best idiom for the same query.
Projection pushdown: read 1 field of 20
The pipeline reads only amount. Every other field is a dead column with no slot: the scanner hands
it to skipValue, which advances past its bytes without decoding or allocating anything. No Event
is built; the fold runs into a var:
you write
the macro emits: one per-record scanner
what we beat: jsoniter-scala (hand-written reader, no object), jawn, and Jackson, all the same query
Lazy decode: a String built only for survivors
category gets a slot, but a string slot is just (start, len) into the buffer. amount decodes
because the filter needs it; category becomes a real String only inside the survivor branch.
Rejected rows never allocate one: the compute-for-survivors sink, operating on bytes:
you write
the macro emits: one per-record scanner
what we beat: jsoniter-scala, narrow codec (only the read fields; the rest skipped)
Discard-DCE: count never decodes the projection
count ignores the element, so map(_.category) is a dead value: the slice is captured, the
String never built, and zero String allocations occur across the run. Dead-column elimination
reaches all the way to the terminal:
you write
the macro emits: one per-record scanner
measured: JMH, -prof gc
| throughput | allocation | |
|---|---|---|
| ▲fuse (no String decoded at all) | 345 ops/s | 1.2 B/op |
| jsoniter: narrow codec | 232 ops/s | 720 002 B/op |
fuse 1.48× faster, and ~zero allocation vs jsoniter's per-record object
Predicate-fail early-out: abandon a record mid-scan
The filter field is decoded during the scan, the predicate runs as soon as it's known, and a
failing record is abandoned there; the scanner never looks for the later projected
field. Here key comes first, payload (a long string) last, and 90% of records fail: each
rejected record costs "scan to key, compare, stop":
you write
the macro emits: one per-record scanner
measured: JMH, -prof gc
| throughput | allocation | |
|---|---|---|
| ▲fuse (rejected records stop at key) | 1207 ops/s | 0.4 B/op |
| jsoniter: narrow codec {key, payload} | 478 ops/s | 1 432 001 B/op |
fuse 2.5× faster, near-zero allocation vs 1.4 MB/op
Multi-aggregate: three answers, one scan, into a case class
aggTo runs sum, count and max in a single fused scan: 3 of 20 fields read, each exactly once, one
unboxed accumulator each, landing in your own case class:
you write
the macro emits: one per-record scanner
The pipeline surface over JSON
.stream hands you the same pipeline type as everywhere else, narrowed to what the record path
supports: filter/map/withFilter stages; the aggregation, search, group-reduce and plan terminal
families (sum, foldLeft, count, find, exists, toList, agg/aggTo, groupSum,
topNBy, …). Records are flat Int/Long/Double/String case classes over one-line NDJSON.
Everything outside that (nested objects, zip, flatMap, collect) is a compile error at the
call site, and an unsupported terminal is not a member. No silent fallback exists.
Scanners are the verbatim post-typer expansion, regenerated to current codegen and checked into
tests/snapshots/. Numbers are JMH (-prof gc) from benchmarks/JsonProjectionBenchmark.