Single-Cell PCA

A live methodological question in single-cell biology: does a linear method like PCA distort the underlying biology? Answered with the real, published 10x Genomics PBMC 3k dataset (2,638 real peripheral blood cells, 8 real published cell types, via Scanpy’s own canonical tutorial object): reduce with linaldb’s own PCA, then check whether the reduced representation alone recovers the real published cell-type labels.

Real dimensionality reduction, verified against numpy

USE DATASET FROM "pbmc3k_expr.npy" AS expr
LET pca_proj = PCA expr_array COMPONENTS 10

Every one of the 10 components matched an independent numpy PCA on the same real matrix to better than 0.999 correlation (up to the sign ambiguity every SVD-based PCA has).

Nearest-centroid classification: 93.4% accuracy, cross-validated two ways

Per-cell-type centroids via AVG_VEC + GROUP BY, then every real cell classified by nearest centroid:

DATASET centroids FROM cells GROUP BY cell_type
    SELECT cell_type AS centroid_type, AVG_VEC(pc) AS centroid, COUNT(*) AS n

WITH classified AS (
    SELECT cell_id, cell_type AS true_type, centroid_type AS predicted_type,
           COSINE_SIM(pc, centroid) AS similarity,
           ROW_NUMBER() OVER (PARTITION BY cell_id ORDER BY similarity DESC) AS rn
    FROM cells JOIN centroids ON COSINE_SIM(pc, centroid) > -2.0
) SELECT cell_id, true_type, predicted_type, similarity FROM classified WHERE rn = 1

2,464 / 2,638 cells correctly classified (93.4%). Independently reconfirmed by recomputing the identical classification from scratch in plain numpy: zero disagreements between the engine’s own answer and the independent recomputation.

A severe bug, found here: silently wrong, not an error

Most bugs this project has found error loudly or produce an obviously-off result. This one didn’t. Two datasets sharing a bare column name (tag) were joined and explicitly qualified in the SELECT list: right_t.tag AS b should show right_t’s values, but used to silently show left_t’s instead:

DATASET left_t COLUMNS (id: Int, tag: String)
INSERT INTO left_t VALUES (1, "LEFT-A")
DATASET right_t COLUMNS (id: Int, tag: String)
INSERT INTO right_t VALUES (1, "RIGHT-X")
SELECT left_t.id AS id, left_t.tag AS a, right_t.tag AS b
FROM left_t JOIN right_t ON left_t.id = right_t.id

b now correctly resolves to "RIGHT-X". Root cause: dsl_expr_to_logical_expr dropped the table qualifier instead of consulting the join schema’s own collision-renaming: silently wrong data, not an error, which is why this class of bug is worth taking more seriously than one that fails loudly. Fixed (engine v0.1.82, PyPI linaldb 0.1.5), and this notebook, re-run against that real published fix, confirms it directly.

Try it yourself

• View the full notebook
• Try it in the Playground