LLM Weight Compression

A live question in LLM research: are transformer weight matrices effectively low-rank? (This is the working hypothesis behind LoRA-style fine-tuning and post-hoc SVD compression.) Answered with real numbers: SVD on all 12 real attention-output-projection matrices from a real pretrained GPT-2 (124M parameters, via HuggingFace transformers), not synthetic data.

Cross-checked against numpy before trusting it

The classical identity ‖A‖²_F = trace(AᵀA), computed via linaldb’s own TRANSPOSE, MATMUL, and TRACE:

MATRIX m = [[4, 0], [3, -5]]
LET mt = TRANSPOSE m
LET mtm = MATMUL mt m
LET fro_sq = TRACE mtm         -- 50.0
LET u, s, vt = SVD m
SHOW s                          -- [6.3246, 3.1623] -- 6.3246^2 + 3.1623^2 = 50.0, matches

On the real 768×768 GPT-2 layer-6 matrix: linaldb’s singular values matched numpy.linalg.svd’s to 0.00e+00 max difference across all 768.

How low-rank, really?

Effective rank at 90%/95% cumulative energy, all 12 layers, all computed by linaldb’s own SVD:

	Layer 0	Layer 6	Layer 11
Effective rank (90% energy)	166	302	336
Effective rank (95% energy)	202	378	416
Full rank	768	768	768

Every layer needs only 20-50% of its full rank to capture 90% of its energy; later layers are more concentrated in their top singular directions.

Does it matter to the real model?

Reconstruct every layer at a shared rank $k$ from linaldb’s own $U$, $s$, $V^T$, patch the real model, measure real perplexity on a real passage (the opening of Pride and Prejudice):

Rank	Storage ratio	Perplexity	vs. baseline (27.11)
8	2.1%	223.72	+196.6 (>8x worse)
256	66.7%	29.37	+2.26
384	100.1%	27.72	+0.61
768 (full)	200.1%	27.11	+0.00

Confirmed independently via linaldb’s own COSINE_SIM on the model’s real next-token probability distribution (50,257-dim, GPT-2’s real vocabulary size): rank 8 drops to 0.860 similarity with the baseline distribution; the rank-384 recovery point reaches 0.965. Both signals agree: degradation concentrates at the low end of the rank range, exactly where the effective-rank numbers predicted.

A real doc/engine mismatch, found here

DSL_REFERENCE.md’s own COSINE_SIM(...) FROM dual example doesn’t actually work: dual isn’t a registered pseudo-table in the real engine, just an unregistered name in that example. Worked around with a real one-row probe dataset instead (DATASET probe COLUMNS (x: Int)). Flagged as a real doc/engine gap worth a follow-up fix.

Try it yourself

• View the full notebook
• Try it in the Playground