Architecture

This page explains the engineering decisions behind linaldb at a level useful for deciding whether to adopt it: not a line-by-line internals walkthrough. If you want that level of depth, the engine source is at gorigami/linaldb.

The shape of the engine

linaldb is built in Rust as a layered pipeline: your DSL text goes through a lexer (Logos) and a recursive-descent + Pratt parser into a typed AST, which dispatches directly to engine operations: zero string round-trips once parsing is done. Query planning (SELECT, JOIN, …) goes through a logical-plan → physical-plan pipeline with its own optimizer (index selection, predicate pushdown, zone-map partition pruning) before execution.

CLI / REPL / Server / Scripts
            │
   Lexer → Parser → typed AST
            │
   Logical Plan → Physical Plan  (index selection, predicate pushdown)
            │
   Engine (multi-database TensorDb)
            │
   Storage (tensor store, dataset store, hash/vector indexes)
            │
   Persistence (Parquet for datasets, JSON for tensors)

The same execution path serves the CLI, the REPL, and the HTTP server: there’s no separate “server mode” logic that can drift from what the CLI does, and a new DSL feature is automatically available everywhere with no extra wiring.

Why it’s fast

Memory: a three-tier allocation strategy

Not every tensor deserves the same allocation path:

Tensor size	Strategy
≤16 elements	Stack allocation (no heap allocation for the intermediate buffer)
17–255 elements	Direct heap allocation (skips pooling overhead for a size that doesn’t benefit from it)
≥256 elements	Tensor pooling (reuses allocations from a pool of common sizes, 128 up to 8192)

Ephemeral, query-scoped allocations go through an arena (bumpalo) with batch cleanup on context drop and a per-context memory limit (100MB default): a runaway query fails with a clear resource error instead of exhausting host memory.

Execution: SIMD and Rayon, dispatched by size

Operation → element count:
├─ ≥1024 elements → SIMD (NEON on ARM, SSE/AVX on x86_64), if the tensor is contiguous
└─ otherwise       → scalar fallback

Rayon-based parallelism is embedded directly inside the kernel functions, not a separate dispatch tier: it fires once a contiguous tensor operation crosses ~50,000 elements (add/sub/multiply/scale/matmul tiling), and once a dataset batch operation crosses ~10,000 rows. Real, measured effect: 2.5x speedup on 100k-element vectors from Rayon parallelization alone. Every fallible linear-algebra operator (INVERSE, SOLVE, CHOLESKY, EIGENVALUES, …) still errors loudly on a singular or non-symmetric input rather than trading correctness for speed.

Zero-copy where it costs nothing

reshape, transpose, and slice are pure metadata operations: O(1), no data copied, backed by a shared Arc<Vec<f32>> and stride manipulation. This is also the foundation for the DSL’s zero-copy semantics (BIND, LET name = <bare identifier>, dataset() + add_column): aliasing a tensor or attaching it to a dataset column never duplicates the underlying data.

Query execution specifics

• Joins always hash the smaller materialized side (by row count), not a fixed side by join type; it hashes the join keys straight off Value, not a formatted string.
• Vector search (CREATE VECTOR INDEX) clusters automatically (IVF-style k-means, cosine metric) once a column crosses ~64 rows, with no syntax change. An approximate top-k query only probes the nearest few clusters, while an exact `WHERE COSINE_SIM(…)

  threshold` predicate uses a provable per-cluster bound to skip clusters that can’t match, so it never drops a qualifying row.

• Partition pruning kicks in automatically for range predicates once a dataset spans more than one internal 1024-row partition, tracking per-partition column min/max. No index or special syntax is required.
• Dataset batching: rows are processed in 1024-row chunks, with parallel execution once a batch crosses ~10,000 rows, for better cache locality.

Measured impact

Optimization	Impact
Rayon parallelization	2.5x on large tensors
Zero-copy views	Zero allocation for reshape/transpose/slice
Tensor pooling	3–18% improvement
Zero-overhead metadata paths	~10% improvement
SIMD kernels	Platform-dependent, on top of the above

Two dataset models, both real

linaldb maintains two dataset implementations side by side, not one superseding the other:

• Row-oriented (dataset_legacy): the execution substrate JOIN/SELECT/the physical query plan actually run on.
• Zero-copy reference graph (dataset): points at existing TensorIds rather than owning data; materialize_tensor_dataset() converts one into the other when a relational operation needs it.

This is why the tensor-first dataset() constructor (see Data & Resources) can attach a tensor as a column in O(1), while DATASET ... COLUMNS (...) gives you the full relational engine (joins, aggregates, window functions) directly.

Correctness-first design

A theme worth naming explicitly, since it shapes how the engine behaves under pressure: every fallible operation errors loudly instead of degrading silently. A singular matrix passed to INVERSE is an error, not a matrix of NaN. A non-symmetric matrix passed to EIGENVALUES is an error, not a wrong answer computed anyway. This is a deliberate trade: a bit more friction when your input is actually invalid, in exchange for never quietly shipping a wrong number downstream.

Persistence and provenance

Datasets persist as Parquet (native FixedSizeList encoding for Vector/Matrix columns when the data allows it) plus JSON sidecars for schema/stats/lineage; tensors persist as JSON. Every tensor operation and dataset transformation is recorded into one append-only, content-hash-addressed provenance log per database: this is what makes EXPLAIN LINEAGE survive a process restart instead of being an in-session-only debugging aid.

Multi-tenancy and the server

linal serve runs the same engine behind Axum, with database isolation via an X-Linal-Database request header: each request restores the previous active database afterward, so concurrent requests targeting different databases don’t interfere with each other. See Persistence & Server for the full HTTP surface.