◈ LessDB

LessDB

LessDB

A best-of-breed analytical database in Rust — SQL-first, extremely resource-efficient, scaling vertically (many cores, optional GPU) and horizontally (stateless compute over shared object storage), on macOS and Linux.

It combines:

Quickstart

cargo build --release
alias less="$PWD/target/release/less"

less init
less create "CREATE TABLE events (id Int64, kind String, amount Float64, ts DateTime) \
             ENGINE = MergeTree ORDER BY (kind, id) UNIQUE (kind)"
less insert events --csv events.csv
less sql "SELECT kind, count(*), sum(amount) FROM events GROUP BY kind ORDER BY kind"
less                # rich interactive TUI (also: less sql)
less bench --rows 5000000

The bare less command opens a full-screen TUI: table sidebar, scrollable results with column types, query editor with history, and psql-style meta commands (\t tables, \d describe, \p parts, \o optimize, \h help, \q quit). Piped stdout falls back to a plain line-based shell.

SharedMergeTree tables (scale-out storage) work with the same SQL — parts go to the shared object store instead of local disk:

CREATE TABLE events_shared (id Int64, kind String, amount Float64)
ENGINE = SharedMergeTree ORDER BY (kind, id) UNIQUE (kind);

Cloud-native: compute and storage separated

Point SharedMergeTree at object storage and compute nodes become stateless — durable state (data parts and table manifests) lives in the shared store, local disk is only ephemeral buffers/caches:

cargo build --release --features cloud    # S3/GCS/Azure backends
less init --shared s3://my-bucket/lessdb  # config.json remembers it
less create "CREATE TABLE t (...) ENGINE = SharedMergeTree ORDER BY (...)"
# any other node with the same --shared URL sees the table immediately:
less tables --dir /another/compute/node

Backends: s3://bucket/prefix, gcs://bucket/prefix, az://account/container/prefix, file:///path, memory:// (credentials from standard AWS/GOOGLE/AZURE env vars). Reads never touch local disk for shared tables; killed nodes are replaced, not recovered.

Multiple writers on the same shared store coordinate through a CAS metadata layer: part publication is a conditional create of meta.json, and merges claim their input parts first — two nodes can flush and OPTIMIZE concurrently without losing or double-merging rows.

A block cache makes repeated shared reads local: immutable part objects fetched from object storage are served from a bounded LRU (block_cache_bytes, default 256 MiB) and, with block_cache_dir set, a persistent disk tier that survives restarts. less cache prints live hits/misses/bytes.

SQL surface

Everything DataFusion supports: joins (hash/sort-merge), CTEs, window functions, subqueries, EXPLAIN, aggregations. LessDB adds ClickHouse-style CREATE TABLE with ENGINE, ORDER BY / PRIMARY KEY, UNIQUE, and COMPRESSION clauses. Query planning prunes data parts using typed min/max statistics and bloom filters before any I/O.

Integrations

# HTTP API (JSON or Arrow IPC)
less server &  curl -s localhost:7080/v1/sql -d '{"sql":"SELECT 1"}' -H 'content-type: application/json'

# MCP server for AI agents (Claude Desktop, etc.)
less mcp --dir mydb                     # default agent-memory tenant "default"
less mcp --dir mydb --tenant agent-1    # per-agent context/memory/vector namespace
import lessdb
db = lessdb.open("mydb")          # in-process, DuckDB style
db.create_table("CREATE TABLE t (x Int64) ENGINE=MergeTree ORDER BY (x)")
db.insert_json("t", [{"x": 1}, {"x": 2}])
rows = db.sql("SELECT sum(x) FROM t")
const { open } = require("@lessdb/node");
const db = open("mydb");
const ipc = db.queryArrow("SELECT count(*) FROM events"); // Arrow IPC bytes

In-memory contexts & graphs (for AI agents)

LessDB includes a RAM-resident tier that replaces Obsidian-style vaults and standalone graph databases: a property graph of titled, tagged notes with typed links, BFS traversal, shortest paths and ranked search, plus RAM tables with primary-key point lookups and full SQL. Everything persists under <data_dir>/memory/ and is exposed to agents through the same MCP server (context_ and memory_ tools) and the CLI.

The MCP server additionally namespaces each agent's memory per tenant: less mcp --tenant <name> keeps that agent's contexts, memory tables and vector spaces under <data_dir>/tenants/<name>/, so multiple agents can share one server without colliding (less_* SQL tools stay on the shared engine):

less context put proj/lessdb "LessDB" "Analytical DB in Rust" --tag db --kind project
less context put task/123 "GPU kernels" "wgpu filtered-sum + dot" --tag gpu --kind task
less context link task/123 proj/lessdb depends_on
less context find gpu && less context neighbors task/123 && less context path task/123 proj/lessdb

# openCypher subset over the same graph (MATCH/WHERE/RETURN/ORDER BY/SKIP/LIMIT/DISTINCT,
# count/collect/sum/avg/min/max, variable-length paths, CREATE/DELETE/SET)
less cypher "MATCH (t)-[:depends_on*1..2]->(p) RETURN DISTINCT t.title, p.title"
less cypher "MATCH (t:task) WHERE t.title CONTAINS 'gpu' RETURN t.key, t.title ORDER BY t.key SKIP 0 LIMIT 10"

less memory create people --field id:Int64 --field name:Utf8 --pk id
less memory insert people '[{"id":1,"name":"ada"},{"id":2,"name":"grace"}]'
less memory sql "SELECT count(*) FROM people"

Security: the control plane (agents + humans)

One identity model, one role model, one audit trail — both front doors.

# Agent credentials for the MCP door (plaintext shown once; only the
# SHA-256 hash is stored under <data_dir>/auth/tokens.json):
less token create claude --role read --tenant default      # read|write|admin
less token list

# Fail-closed MCP door: tokens required at initialize, every tool call
# role-checked (admin > write > read) before it executes:
less mcp --require-auth

# Every call — caller, tool, role, outcome, SQL, duration — lands in an
# append-only NDJSON trail under <data_dir>/audit/:
less audit --since 24h --caller claude --outcome denied

Tool→permission map: less_query/explain/schema/stats/tables, vector_ reads, context_ reads, memory_sql/get = read; less_optimize, context_put/link/unlink/delete, memory_insert/compact, vector_put = write; vector_create/drop, memory_create = admin. Unknown tools fail safe to read-only. See docs/AGENT-GOVERNANCE.md.

Security: LDAP / Active Directory

Remote interfaces (HTTP server) authenticate against LDAP/AD with role-based authorization — fail-closed group→role mapping:

less init --auth '{
  "ldap": {
    "url": "ldaps://ad.corp.example.com:636",
    "base_dn": "DC=corp,DC=example,DC=com",
    "bind_dn": "CN=lessdb-svc,OU=Services,DC=corp,DC=example,DC=com",
    "bind_password": "…",
    "role_mapping": { "DB-Admins": "admin", "DB-Users": "read", "DB-Writers": "write" }
  }
}'
less server   # now requires HTTP Basic auth; admins can POST /v1/admin/optimize

# HTTPS with rustls — no proxy needed:
less server --tls-cert cert.pem --tls-key key.pem
# (or "tls_cert"/"tls_key" in config.json)

Standard AD flow: service-account bind → locate user DN → rebind as the user (the directory verifies the password) → map groups to roles. Usernames are filter-escaped (no LDAP injection), and auth failures are counted. A dev file authenticator ({"file": {"users": {"alice": {"password": "…", "role": "admin"}}}}) works without a directory.

Observability: Prometheus

GET /metrics exposes Prometheus text metrics (also less metrics): queries + duration histogram, rows, parts written/merged/scanned/pruned (pruning effectiveness on a dashboard), HTTP requests, auth failures, uptime, memory, build info.

scrape_configs:
  - job_name: lessdb
    static_configs: [{ targets: ["db-host:7080"] }]

Native vector search (LanceDB-style)

Embedded vector search with a registry of named vector spaces (each with its own dimension, metric space — l2 / cosine / dot — and index), exact flat search and IVF-PQ approximate nearest neighbors (k-means inverted lists + product quantization, ADC lookup tables, exact re-ranking), an embedding-function registry, and snapshot persistence under <data_dir>/vectors/.

less vector create docs 3 --metric cosine          # flat exact
less vector create big 256 --index ivf_pq --nlist 64 --m 8
less vector add docs --vectors '[[1,0,0],[0,1,0],[0.9,0.1,0]]'                     --payloads '[{"title":"rust"},{"title":"python"},{"title":"rust-like"}]'
less vector search docs '[1,0,0]' --k 2             # ranked hits + payloads
less vector embed "rust database"                   # built-in trigram embedder

SQL-first, too — the vector_search table function works like LanceDB's, including joins over the hits:

SELECT * FROM vector_search('docs', [1.0, 0.0, 0.0], 2);
SELECT id, payload FROM vector_search('docs', [1.0, 0.0, 0.0], 10)
 WHERE payload LIKE '%rust%';

Agents get vector_create / vector_put / vector_search / vector_list / vector_embed / vector_drop through the same MCP server, and the Python & Node SDKs get it through SQL. Production embedding models plug into the embedder registry (VectorRegistry::register_embedder).

Writing data

less insert events --csv events.csv        # header names map to columns
less insert events --jsonl events.jsonl    # NDJSON or JSON array
less insert events --parquet events.parquet
less insert events --arrow events.arrow    # Arrow IPC stream
less sql "INSERT INTO events VALUES (1, 'click', 1.5, '2025-01-01')"
less sql "INSERT INTO events SELECT * FROM other_table"
less sql "COPY (SELECT * FROM events) TO 'export.parquet'"   # parquet | csv | arrow

Embedded sessions: session.register_file("staging", "file.parquet") then INSERT INTO t SELECT * FROM staging. Inserts are WAL-protected: every accepted row is durable before the insert returns, and a crash mid-write recovers exactly-once on reopen (replay skips rows already sealed into parts). Types: integers, floats, bool, string, date, datetime, Decimal(p,s), UUID, Array(T), Map(K,V).

GPU acceleration

cargo build --release --features gpu and less gpu benchmarks the wgpu kernels (Metal on macOS, Vulkan on Linux) against CPU — filtered sums/dot products, the shapes behind WHERE+SUM and join costing. CPU fallback is always available.

Repository layout

crates/    engine, storage, query, catalog, server, mcp, gpu,
           graph (contexts), memory (RAM tables), vector (k-NN/IVF-PQ), cli
sdks/      python (PyO3), node (napi-rs)
skills/    less-query, less-admin, less-context agent skills
docs/      architecture, roadmap, design decisions

See docs/ARCHITECTURE.md for the full design, docs/ROADMAP.md for what's next, and docs/DESIGN-DECISIONS.md for the reasoning behind each choice.

CI

Every push and PR runs the full pipeline on a self-hosted GitHub Actions runner on the local build machine (homelab-lessdb-01, labeled linux-x64): cargo fmt --check, clippy -D warnings, the full test suite, a release build with cloud+gpu features, and a benchmark smoke run whose report is uploaded as a CI artifact. Toolchain pinned via rust-toolchain.toml (Rust 1.96).

License

Apache-2.0