Skip to content
linaldblinaldblinaldb

Persistence & Server

Persistence & ingestion

Section titled “Persistence & ingestion”
  • USE DATASET FROM "path" [AS name] [FIELDS (name1, name2, ...)]: load external data (CSV, HDF5, NetCDF, NumPy, Zarr, external Parquet) into the session as ephemeral tensors and a dataset view. Format is auto-detected from the file extension. Without FIELDS, a source bundling fields of different shapes keeps whichever share the first-encountered shape and warns about the rest; with FIELDS, only the named fields are read, and a name that doesn’t exist (or a set of fields that can’t share one row count) is a hard error.
  • IMPORT DATASET FROM "path" [AS name] [FIELDS (...)]: load and normalize external data into a persistent dataset package. Same FIELDS behavior as above.
  • IMPORT CSV FROM "path" AS name: (legacy) auto-infer schema and load CSV.
  • EXPORT [CSV] name TO "path": save to CSV. A Vector/Matrix column is written as a JSON string per cell (CSV has no native nested-data representation). Use SAVE DATASET instead for a native binary (Parquet FixedSizeList) encoding.
  • SAVE DATASET name [TO "path"]: persist to Parquet, including metadata/lineage.
  • LOAD DATASET name [FROM "path"]: restore a persisted dataset.
  • SAVE TENSOR name [TO "path"] / LOAD TENSOR name [FROM "path"]: persist/restore a tensor to/from JSON (lineage preserved).
  • SAVE PIPELINE name [TO "path"] / LOAD PIPELINE name [FROM "path"]: see Pipelines & Vector Search.
  • LIST DATASETS [FROM "path"] (alias LIST DATASET PACKAGES), LIST TENSORS [FROM "path"], LIST DATASET VERSIONS <name>: inventory what’s available.

Scientific data ingestion

Section titled “Scientific data ingestion”

Direct ingestion for multi-dimensional scientific data:

  • HDF5: datasets from groups, flattened into columns.
  • NetCDF (.nc): real CF-convention semantics on top of HDF5-format NetCDF4 files: applies each variable’s scale_factor/add_offset packing and _FillValue/ missing_value masking (mapped to NaN), and surfaces units/standard_name/ long_name as inspectable field metadata. Every variable in one ingested table must still share the same flat element count (FIELDS picks a subset when a file bundles a data variable alongside differently-shaped coordinate variables). A raw (time, lat, lon)-shaped grid variable can’t materialize into a 2D table at all (Cannot materialize tensor with rank > 2); a single-location/region time series (each real variable the same length) is the natural shape for this connector.
  • NumPy: .npy (single vector/matrix) and .npz (named collections).
  • Zarr: V3 stores, recursive group traversal.
  • Parquet (external): generic ingestion of an arbitrary external .parquet file via USE/IMPORT DATASET FROM, distinct from the engine’s own internal Parquet dataset-package format SAVE/LOAD DATASET uses.
-- Only the "labels" field, even though the file also has a differently
-- shaped "embeddings" field.
USE DATASET FROM "vectors.h5" AS d FIELDS (labels)

Instance & session management

Section titled “Instance & session management”
CREATE DATABASE research
USE research
DROP DATABASE obsolete_db
SHOW DATABASES          -- also: SHOW ALL DATABASES

RESET SESSION clears all in-memory registers (tensors and datasets) for the current session.

Diagnostics

Section titled “Diagnostics”

Resource display

  • SHOW <name>: contents of any resource (tensor, legacy dataset, tensor-first dataset). Automatically materializes lazy tensors first.
  • SHOW ALL / SHOW ALL TENSORS: list all in-memory tensors with shapes and data.
  • SHOW ALL DATASETS: list all legacy datasets with row/column counts.
  • SHOW SCHEMA <dataset>: column names and types.
  • SHOW SHAPE <name>: just a tensor’s shape.
  • SHOW INDEXES [<dataset>]: list indexes, optionally filtered to one dataset.
  • SHOW "<string>": print a string literal, useful for annotating script output.

Dataset metadata & versioning

  • SHOW DATASET METADATA <name>: version, hash, origin, author, tags, timestamps.
  • SHOW DATASET VERSIONS <name> (alias LIST DATASET VERSIONS <name>): full schema evolution history.

Query planning

  • EXPLAIN [PLAN] <target>: the logical and physical execution plan. <target> is a SELECT, a DATASET <name> scan (or DATASET ... FROM view), or a SEARCH. EXPLAIN <name> is shorthand for EXPLAIN DATASET <name>.

Lineage & provenance

  • EXPLAIN LINEAGE <name>: the real, persisted derivation ancestry for a tensor or dataset: every IMPORT, DATASET ... FROM, computed column, tensor op, and SAVE, in order. This is genuinely different from EXPLAIN <target> (a query plan): this shows how the data actually got here. Survives a restart (read from a content-hash addressed provenance log, not just in-session state). Works even on a dataset you just LOADed fresh. EXPLAIN LINEAGE <name> AS JSON gives the same ancestry as JSON.
  • AUDIT DATASET <name>: a referential-integrity check: do this dataset’s column references still resolve? (Unrelated to derivation history, despite the similar name.) Only works on tensor-first datasets (built via dataset(), see Data & Resources). Errors Tensor dataset '<name>' not found against an ordinary DATASET COLUMNS (...) dataset, even one that works fine with SHOW/SELECT. Most datasets in this reference (and most real usage) use that legacy form, so this is easy to hit by surprise.
  • DELIVER <dataset> [TO '<path>']: check whether a dataset is deliverable over /delivery (below); points you at SAVE DATASET if it isn’t ready yet. “Doesn’t exist” here means not loaded into the current session: a dataset saved in an earlier linal run invocation needs an explicit LOAD DATASET <name> first, even though it’s already persisted; DELIVER doesn’t check disk on its own.

Server & job management

Section titled “Server & job management”

Run linal serve --port 8080 for remote execution and production workloads.

Background jobs

Endpoint Method Description
/jobs POST Submit a DSL command for background execution: Content-Type: text/plain, the raw command as the body (same contract /execute uses below), not a JSON body. Returns job_id.
/jobs GET List all jobs and their statuses.
/jobs/:id GET Poll a job: Pending, Running, Completed, or Failed.
/jobs/:id/result GET Retrieve the structured result of a completed job.
/jobs/:id DELETE Cancel a Pending job (running/finished jobs can’t be cancelled).

Scheduler: recurring DSL commands on a fixed interval. Unlike /execute//jobs, /schedule POST takes a real JSON body (it’s registering a task definition, not executing a command directly):

Endpoint Method Description
/schedule POST Register a scheduled command (name, command, interval_secs, optional target_db).
/schedule GET List all active scheduled tasks.
/schedule/:id DELETE Remove a scheduled task.

Other endpoints

Endpoint Method Description
/health GET Server health check.
/execute POST Execute a DSL command synchronously: Content-Type: text/plain, one raw statement per request (no trailing semicolon), ?format=json for a JSON response (default is a plain-text “toon” format).
/databases GET List database instances.
/databases/:name POST / DELETE Create / drop a database instance.
/delivery/... GET Read-only Parquet dataset export.

Multi-tenant isolation is via the X-Linal-Database: <db_name> request header: each request restores the previous active database afterward, so concurrent requests with different headers don’t interfere. A database must already exist before you target it with this header. Creating one (CREATE DATABASE <name>) has to run without the header (or with it pointed at an existing database), since the header resolves its target before the statement runs.

The server handles SIGINT/SIGTERM for graceful shutdown, and exposes an OpenAPI/ Swagger UI at /swagger-ui.