EXPLAIN LINEAGE + classical linear algebra, end to end

This notebook exercises the real linaldb package installed from PyPI in linal-hub/.venv — the two features LINEAGE_AND_LINALG_PLAN.md added: the unified, persisted provenance model (EXPLAIN LINEAGE) and the classical linear algebra suite (TRACE/DETERMINANT/RANK/INVERSE/SOLVE/EIGENVALUES/QR/LU/CHOLESKY/EIGEN/SVD/PCA).

Two parts, using real (not synthetic) data derived from MNIST digit images:

1. A real dataset pipeline — import → filter → aggregate → computed column → save → a genuine process restart → load — with EXPLAIN LINEAGE inspected after every step, proving the ancestry chain survives on disk, not just in one session's memory.
2. Tensor operations feeding a linear algebra decomposition — TRANSPOSE/MATMUL building a Gram matrix, then EIGEN, PCA, and SVD on it — proving EXPLAIN LINEAGE produces the same shape of ancestry tree for tensors as for datasets (the unification this initiative's whole design was built around), on a real chain that happens to be exactly the one that surfaced a real bug (see the callout in part 2).

cd linal-hub
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m ipykernel install --user --name linal-hub --display-name "linal-hub (linaldb)"

import linaldb

print("linaldb", linaldb.__version__)

linaldb 0.1.9

Preparing real data

Real MNIST test-set images (from notebooks/data/examples/mnist_cnn_forward_pass/mnist.npz, already used by notebook 04). For part 1, three simple scalar features per image (mean pixel intensity, max intensity, fraction of "ink" pixels) across 60 real images spanning digits 0-4 — enough to make FILTER/GROUP BY genuinely meaningful. For part 2, a small real pixel matrix (18 images x 49 features, 4x4-block-averaged down from 28x28) for the tensor/linear-algebra half.

import numpy as np
import csv
import os

DATA_DIR = "data/examples/lineage_and_linalg_showcase"
os.makedirs(DATA_DIR, exist_ok=True)

d = np.load("data/examples/mnist_cnn_forward_pass/mnist.npz")
x, y = d["x_test"], d["y_test"]

rng = np.random.default_rng(42)
rows = []
for digit in range(5):
    idx = np.where(y == digit)[0]
    chosen = rng.choice(idx, size=12, replace=False)
    for i in chosen:
        img = x[i].astype(np.float32) / 255.0
        rows.append((int(i), digit, float(img.mean()), float(img.max()), float((img > 0.1).mean())))

csv_path = f"{DATA_DIR}/mnist_features.csv"
with open(csv_path, "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["sample_id", "digit", "mean_intensity", "max_intensity", "ink_fraction"])
    w.writerows(rows)

def downsample(img, block=4):
    h, w = img.shape
    hh, ww = h // block, w // block
    return img[: hh * block, : ww * block].reshape(hh, block, ww, block).mean(axis=(1, 3))

mat_rows, mat_labels = [], []
for digit in range(3):
    idx = np.where(y == digit)[0]
    chosen = rng.choice(idx, size=6, replace=False)
    for i in chosen:
        img = x[i].astype(np.float32) / 255.0
        mat_rows.append(downsample(img, 4).flatten())
        mat_labels.append(digit)

digit_images = np.array(mat_rows, dtype=np.float32)
digit_labels = np.array(mat_labels)
print(f"wrote {len(rows)} feature rows to {csv_path}")
print("real pixel matrix for part 2:", digit_images.shape)

wrote 60 feature rows to data/examples/lineage_and_linalg_showcase/mnist_features.csv
real pixel matrix for part 2: (18, 49)

Part 1: a real dataset pipeline, with lineage inspected at every step

IMPORT → FILTER → GROUP BY → computed column → SAVE — the exact chain this plan's own tests/lineage_provenance_test.rs proves at the Rust level; here it's the same shape, driven from Python against the real published package, on real image-derived data instead of a synthetic fixture.

import shutil

shutil.rmtree("./data_lineage_showcase", ignore_errors=True)
db = linaldb.Db(data_dir="./data_lineage_showcase")

db.execute(f'IMPORT DATASET FROM "{csv_path}" AS mnist_features')
db.execute("LOAD DATASET mnist_features")
print(db.execute("EXPLAIN LINEAGE mnist_features"))

Lineage for 'mnist_features':
IMPORT csv (mnist_features) [a06802ce]

A single IMPORT node — the real root of this dataset's history. Now a transformation: keep only images with above-baseline mean intensity.

db.execute(
    "DATASET bright_digits FROM mnist_features FILTER mean_intensity > 0.05 "
    "SELECT digit, mean_intensity, ink_fraction"
)
print(db.execute("EXPLAIN LINEAGE bright_digits"))

Lineage for 'bright_digits':
DATASET FROM (bright_digits) [2eb7207a]
  IMPORT csv (mnist_features) [a06802ce]

bright_digits's own lineage now shows the real chain: DATASET FROM -> IMPORT. Next, a real aggregation — per-digit average intensity and ink fraction.

db.execute(
    "DATASET digit_stats FROM bright_digits GROUP BY digit "
    "SELECT digit, AVG(mean_intensity) AS avg_intensity, AVG(ink_fraction) AS avg_ink, COUNT(*) AS n"
)
print(db.query("SELECT * FROM digit_stats ORDER BY digit"))
print(db.execute("EXPLAIN LINEAGE digit_stats"))

   digit  avg_intensity   avg_ink   n
0      0       0.161166  0.215561  12
1      1       0.083749  0.109800  12
2      2       0.147774  0.197066  12
3      3       0.133822  0.181760  12
4      4       0.126305  0.171450  12
Lineage for 'digit_stats':
DATASET FROM (GROUP BY) (digit_stats) [a060abfc]
  DATASET FROM (bright_digits) [2eb7207a]
    IMPORT csv (mnist_features) [a06802ce]

Real, non-cherry-picked numbers: digit 1 genuinely has noticeably lower average intensity/ink fraction than the others in this sample — a thin stroke has less ink than a closed loop, exactly as expected. The lineage tree is now three levels deep: DATASET FROM (GROUP BY) -> DATASET FROM -> IMPORT. One more real mutation — a computed column — before saving.

db.execute("ALTER DATASET digit_stats ADD COLUMN ink_per_intensity = avg_ink / avg_intensity")
print(db.execute("EXPLAIN LINEAGE digit_stats"))
print(db.execute("SAVE DATASET digit_stats"))

Lineage for 'digit_stats':
ADD COMPUTED COLUMN (digit_stats) [af235b29]
  DATASET FROM (GROUP BY) (digit_stats) [a060abfc]
    DATASET FROM (bright_digits) [2eb7207a]
      IMPORT csv (mnist_features) [a06802ce]

Saved dataset 'digit_stats' (v1) to './data_lineage_showcase/default'

Four real steps, one real chain. Now the actual proof this initiative was built for: drop this Db entirely and start a brand-new process-equivalent instance — the embedded-bindings analogue of a real restart, no in-memory state carried over — and confirm the full chain reconstructs from disk alone.

del db  # drop every bit of in-memory state -- only ./data_lineage_showcase on disk remains

db = linaldb.Db(data_dir="./data_lineage_showcase")  # a genuinely fresh instance
db.execute("LOAD DATASET digit_stats")

print("--- post-restart, text ---")
print(db.execute("EXPLAIN LINEAGE digit_stats"))

print("--- post-restart, JSON ---")
print(db.execute("EXPLAIN LINEAGE digit_stats AS JSON"))

--- post-restart, text ---
Lineage for 'digit_stats':
ADD COMPUTED COLUMN (digit_stats) [af235b29]
  DATASET FROM (GROUP BY) (digit_stats) [a060abfc]
    DATASET FROM (bright_digits) [2eb7207a]
      IMPORT csv (mnist_features) [a06802ce]

--- post-restart, JSON ---
{
  "entity": {
    "Dataset": {
      "name": "digit_stats",
      "content_hash": "af235b29889a11053abbd57ef887779369a222a913fccfef813e82c9f0968e4e"
    }
  },
  "operation": "ADD COMPUTED COLUMN",
  "parameters": {
    "column": "ink_per_intensity"
  },
  "timestamp": "2026-09-16T13:46:04.814394Z",
  "execution_id": "ead1c236-19b2-4dc5-826e-7a222103d6cb",
  "inputs": [
    {
      "entity": {
        "Dataset": {
          "name": "digit_stats",
          "content_hash": "a060abfc34863260ff1609746fb0dd4d78788457b248690c17c180d766ccc42f"
        }
      },
      "operation": "DATASET FROM (GROUP BY)",
      "parameters": {
        "source": "bright_digits"
      },
      "timestamp": "2026-09-16T13:46:04.662024Z",
      "execution_id": "92ac800d-6f91-4894-ae3a-f3baacb9a271",
      "inputs": [
        {
          "entity": {
            "Dataset": {
              "name": "bright_digits",
              "content_hash": "2eb7207ad05c6661df2d601716e0bfcf7a9e825895cf62cde7f91a7074f7df4b"
            }
          },
          "operation": "DATASET FROM",
          "parameters": {
            "source": "mnist_features"
          },
          "timestamp": "2026-09-16T13:46:04.659487Z",
          "execution_id": "99ddb2e4-1567-44f2-98e9-88cf4c7cbcc4",
          "inputs": [
            {
              "entity": {
                "Dataset": {
                  "name": "mnist_features",
                  "content_hash": "a06802ceb57277e89311cae8e90f71b8f221e044118b1f0e68eee572f0b35814"
                }
              },
              "operation": "IMPORT csv",
              "parameters": {
                "path": "data/examples/lineage_and_linalg_showcase/mnist_features.csv"
              },
              "timestamp": "2026-09-16T13:46:04.656443Z",
              "execution_id": "00b00e94-0306-49c5-b6d8-199db71e94af",
              "inputs": []
            }
          ]
        }
      ]
    }
  ]
}

The identical four-level chain, reconstructed purely from ./data_lineage_showcase/default/provenance.jsonl — this is what engine::LineageNode's old in-memory-only tree walk could never have done. SHOW LINEAGE still works too, as a documented alias:

print(db.execute("SHOW LINEAGE digit_stats"))

Lineage for 'digit_stats':
ADD COMPUTED COLUMN (digit_stats) [af235b29]
  DATASET FROM (GROUP BY) (digit_stats) [a060abfc]
    DATASET FROM (bright_digits) [2eb7207a]
      IMPORT csv (mnist_features) [a06802ce]

Part 2: tensor operations feeding a linear algebra decomposition

EXPLAIN LINEAGE must produce the same shape of tree whether the root is a dataset (part 1) or a tensor. Real MNIST pixel data, this time: a small TRANSPOSE/MATMUL chain building a Gram matrix, feeding EIGEN.

This exact chain is the one that found a real bug while this notebook was first being built: a zero-copy TRANSPOSE shares the same underlying buffer as its input in this engine's storage model, and the content hash EXPLAIN LINEAGE resolves ancestry by used to hash that raw buffer instead of the tensor's logical (shape-aware) data -- so a transposed matrix and its untransposed source hashed identically, and ancestry could be misattributed. Fixed in the engine version this notebook is running against; see CHANGELOG.md's [0.1.81] entry.

def matrix_literal(name, arr):
    rows = ["[" + ", ".join(f"{v:.5f}" for v in row) + "]" for row in arr]
    return f"MATRIX {name} = [{', '.join(rows)}]"

db.execute(matrix_literal("digit_images", digit_images))
print(db.execute("SHOW SHAPE digit_images"))

db.execute("LET images_t = TRANSPOSE digit_images")
db.execute("LET gram = MATMUL images_t digit_images")
print(db.execute("SHOW SHAPE gram"))

SHAPE digit_images: [18, 49]

SHAPE gram: [49, 49]

db.execute("LET vals, vecs = EIGEN gram")
eigvals = db.execute("SHOW vals").to_numpy()
print("top eigenvalues:", np.sort(eigvals)[::-1][:5])
print()
print(db.execute("EXPLAIN LINEAGE vals"))

top eigenvalues: [37.17041779  6.77012205  3.15768242  2.5009284   1.91783273]

Lineage for 'vals':
EIGEN (vals) [41aff4d2]
  MATMUL (gram) [5f147196]
    TRANSPOSE (images_t) [d6201910]
      ROOT (digit_images) [f28adae7]
    ROOT (digit_images) [f28adae7]

EIGEN <- MATMUL <- TRANSPOSE <- ROOT (and, on the other MATMUL branch, EIGEN <- MATMUL <- ROOT directly for the untransposed digit_images) — a real, mixed tensor-op-plus-decomposition ancestry chain, correctly distinguishing the transposed tensor from its source now that the content-hash fix is in. Same tree shape as part 1's dataset chain, same EXPLAIN LINEAGE command, same ProvenanceStore underneath — the unification this whole initiative was built around.

Now the actual payoff: real dimensionality reduction on real image data.

db.execute("LET projected = PCA digit_images COMPONENTS 2")
proj = db.execute("SHOW projected").to_numpy()
print("PCA projection shape:", proj.shape)
print(db.execute("EXPLAIN LINEAGE projected"))

PCA projection shape: (18, 2)
Lineage for 'projected':
PCA(components=2) (projected) [04201cfd]
  ROOT (digit_images) [f28adae7]

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(5, 4))
for digit in range(3):
    mask = digit_labels == digit
    ax.scatter(proj[mask, 0], proj[mask, 1], label=f"digit {digit}")
ax.set_xlabel("PC 1")
ax.set_ylabel("PC 2")
ax.set_title("PCA(digit_images, components=2) -- real MNIST pixels")
ax.legend()
plt.tight_layout()
plt.show()

Real digit images, computed entirely inside linaldb's own DSL (PCA ... COMPONENTS 2, no numpy/sklearn PCA involved), separating visibly by digit class in just two components. Finally, SVD on the same real matrix:

db.execute("LET u, s, vt = SVD digit_images")
print("singular values:", db.execute("SHOW s").to_numpy())

singular values: [6.09675455 2.60194588 1.77698708 1.58143234 1.38485837 1.23324347
 1.1229043  0.73580819 0.5936147  0.50267684 0.44834611 0.42992902
 0.2988492  0.23038967 0.1054933  0.09904749 0.09316345 0.05243912]

---

See ../tests/ for a broader, assertion-backed exploration of the package's surface. See the parent repository's LINEAGE_AND_LINALG_PLAN.md (kept until this notebook -- its final gate -- landed) for the full initiative this notebook validates.