Lineage & Linear Algebra

Two things this notebook proves, both against real MNIST-derived data: EXPLAIN LINEAGE produces a real, disk-persisted ancestry chain that survives a genuine process restart (not just in-session memory), and it produces the same shape of tree whether the root is a dataset or a tensor: one unified provenance model.

A real pipeline, lineage inspected at every step

IMPORT DATASET FROM "mnist_features.csv" AS mnist_features
DATASET bright_digits FROM mnist_features FILTER mean_intensity > 0.05 SELECT digit, mean_intensity, ink_fraction
DATASET digit_stats FROM bright_digits GROUP BY digit SELECT digit, AVG(mean_intensity) AS avg_intensity, AVG(ink_fraction) AS avg_ink, COUNT(*) AS n
ALTER DATASET digit_stats ADD COLUMN ink_per_intensity = avg_ink / avg_intensity
SAVE DATASET digit_stats

Four real steps building a four-level ancestry chain:

ADD COMPUTED COLUMN (digit_stats)
  DATASET FROM (GROUP BY) (digit_stats)
    DATASET FROM (bright_digits)
      IMPORT csv (mnist_features)

Surviving a real restart

Drop the Db entirely, start a genuinely fresh instance, LOAD DATASET digit_stats, and the identical four-level chain reconstructs purely from provenance.jsonl on disk, not from anything still resident in memory:

LOAD DATASET digit_stats
EXPLAIN LINEAGE digit_stats
EXPLAIN LINEAGE digit_stats AS JSON

The same tree shape for tensors

A TRANSPOSE/MATMUL chain building a Gram matrix, feeding EIGEN, real provenance, not just dataset lineage:

MATRIX digit_images = [[1.0, 0.5, 0.2], [0.8, 0.9, 0.1], [0.3, 0.2, 0.7]]
LET images_t = TRANSPOSE digit_images
LET gram = MATMUL images_t digit_images
LET vals, vecs = EIGEN gram
EXPLAIN LINEAGE vals

EIGEN (vals)
  MATMUL (gram)
    TRANSPOSE (images_t)
      ROOT (digit_images)
    ROOT (digit_images)

Same EXPLAIN LINEAGE command, same underlying provenance store, whether the root is a DATASET FROM chain or a chain of tensor ops feeding a decomposition: the unification this feature was built around.

A real bug, found by this exact chain

Building this notebook found a real one: a zero-copy TRANSPOSE shares its input’s underlying buffer (by design; that’s what makes it zero-copy), but EXPLAIN LINEAGE’s content-hash ancestry used to hash that raw buffer instead of the tensor’s logical, shape-aware data, so a transposed matrix and its untransposed source could hash identically, misattributing ancestry. Fixed in engine v0.1.81 and PyPI linaldb 0.1.4; this exact TRANSPOSE/MATMUL chain above is the one that found it.

Real dimensionality reduction, no sklearn

PCA ... COMPONENTS 2 on real MNIST pixel data separates visibly by digit class in two components, computed entirely inside the DSL:

LET projected = PCA digit_images COMPONENTS 2