Data & Resources

Data types

Relational types

• Int: 64-bit signed integer.
• Float (alias FLOAT32): 32-bit float. Standard precision for tensor values; Vector/Matrix/Tensor elements are always this precision.
• Double (alias FLOAT64): 64-bit float. Use this for large-magnitude scalars (GPS/Unix timestamps, etc.) that exceed Float’s ~7 significant digits: a plain Float column silently rounds these. Arithmetic mixing a Double with Float/Int always promotes to Double. Not available for Vector/Matrix/Tensor elements, which stay Float-only.
• String: UTF-8 text.
• Bool: true/false.
• Complex: a scalar complex number (real + imaginary part). Construct with COMPLEX(re, im); decompose with REAL/IMAG/ABS/PHASE/CONJ (see Querying). Arithmetic mixing Complex with any numeric type always promotes to Complex: it’s the widest scalar type. Has equality (=/!=/IN all work) but no ordering: MIN/MAX/ORDER BY on a Complex column error loudly rather than guess; SUM/AVG work normally, since those are mathematically well-defined for complex numbers. Scalar only, not available for Vector/Matrix/Tensor elements.
• Null: a missing value. Mark a column nullable with a ? suffix: score: Float?.

Tensor types

• Vector(N): a 1D tensor with N elements.
• Matrix(R, C): a 2D tensor with R rows, C columns.
• Tensor(d1, d2, ...): an N-dimensional tensor.

Defining tensors

Quick shorthand:

VECTOR v = [1.0, 2.0, 3.0]
MATRIX m = [[1, 2], [3, 4]]

Explicit form for higher dimensions:

DEFINE t AS TENSOR [2, 2, 2] VALUES [1, 2, 3, 4, 5, 6, 7, 8]

-- STRICT propagates its shape-strictness through any op it's involved in,
-- preventing accidental shape relaxation
DEFINE w AS STRICT TENSOR [3] VALUES [1, 0, 0]

Defining datasets

DATASET diagnostics COLUMNS (
    id: Int,
    region: String,
    score: Float?,           -- nullable column
    features: Vector(128)    -- embedded tensor
)

A second form materializes a dataset from a query against an existing one, like SELECT ... FROM <source> ... but persisted under a new name instead of returned inline:

DATASET seniors FROM employees FILTER age >= 60

DATASET top_scores FROM diagnostics
    FILTER region = "west"
    SELECT region, AVG(score)
    GROUP BY region
    HAVING AVG(score) > 0.5
    ORDER BY region
    LIMIT 10

DATASET <name> FROM <source> [FILTER|WHERE <expr>] [SELECT <cols>] [GROUP BY <cols>] [HAVING <expr>] [ORDER BY <cols>] [LIMIT <n>] [OFFSET <n>]: every clause after FROM <source> is optional and behaves like its SELECT equivalent.

Tensor-first datasets

For a zero-copy, tensor-centric alternative to DATASET COLUMNS (...), construct an empty named dataset and attach existing tensors as columns:

LET ds = dataset("my_dataset")   -- registers an empty dataset, O(1)

VECTOR v_temp = [36.6, 37.1, 36.9]
LET raw = dataset("raw")
raw.add_column(temp, v_temp)     -- <dataset_var>.add_column(<name>, <tensor_var>)

add_column is a metadata-only operation: no data is copied.

Zero-copy aliases and derived resources

• BIND alias TO resource: create a semantic alias to a tensor or dataset.
• ATTACH tensor TO ds.col: link an independent tensor into a dataset column.
• DERIVE target FROM expr: create a new resource with full automatic lineage tracking. expr must be a real computed expression; DERIVE b FROM a (a bare identifier) is a clear error pointing at BIND/LET instead, since aliasing creates no new lineage node.
• LET name = <bare identifier> is also a zero-copy alias, equivalent to BIND name TO <bare identifier>; works for a plain tensor name or a dataset()-constructed tensor-first dataset variable. LAZY LET name = <bare identifier> is a clear error (nothing to defer in a plain alias).

Schema evolution

• ALTER DATASET ds ADD COLUMN col: type [DEFAULT val]
• ALTER DATASET ds ADD COLUMN col = expression [LAZY]
• MATERIALIZE ds: physicalize all LAZY columns in a dataset.
• SET DATASET ds [METADATA] key = "value": attach a string metadata key (the METADATA keyword is optional). Retrieve it with SHOW DATASET METADATA <name> (see Persistence & Server).