What Is NoKV?

NoKV is a metadata control plane for object-backed agent artifacts: run outputs, log files, checkpoints, reports, and citable evidence in one filesystem-shaped namespace. For the longer interface argument, see Agents Want Filesystems.

It is not a trace database. Keep JSONL, SQLite, or Postgres as the source of truth for runtime events; use NoKV as the agent-facing namespace over the artifacts and evidence those systems produce.

NoKV keeps namespace metadata in its own path-native engine (Holt) and stores file bodies as immutable blocks in S3-compatible object storage such as RustFS, MinIO, Ceph RGW, or AWS S3.

FUSE / SDK / CLI
  -> NoKV metadata service     (self-contained; no separate metadata DB to run)
  -> Holt inode/dentry metadata
  -> S3-compatible object store for file bodies

NoKV owns namespace truth, metadata transactions, snapshots, watches, and object-reference GC. The object store owns byte durability and replication. The metadata engine is built in, so local deployments operate a filesystem rather than a filesystem plus a separate Redis, MySQL, or TiKV cluster.

Why NoKV

Agent workflows are artifact-heavy; their workspaces aren't. Every run leaves behind configs, metrics, logs, checkpoints — and that state scatters across folders, JSON files, object-store keys, and database rows. Agents pay a navigation tax in tokens every time they go looking. NoKV gives that state one address, with the metadata guarantees the workload actually needs:

• Checkpoints publish atomically. Readers see the complete new checkpoint

or the previous one — never a half-written file, even across a crash.

• Snapshots are time travel. Pin a frozen view of any subtree and keep

reading it while jobs write; GC never deletes what a snapshot still needs.

• Changes are events, not polls. Every create, rename, and publish lands as

a typed, replayable event with a cursor.

• Artifacts carry body references and digests, with cleanup of failed

staged uploads.

• Bodies are immutable, versioned blocks. Replacement publishes a new

generation, so node-local caches never invalidate object bytes after publish.

The primary write model is write-once publish, matching how datasets, checkpoints, and artifacts are commonly written.

🤖 The Agent Interface

ls · stat · catalog · find · aggregate · read · grep

Seven verbs, one progressive-disclosure surface: an agent discovers what exists, learns what is queryable, and pays to read only what it needs. Predicates, sort, and projection are pushed into the engine, so a "top-5 runs by val_loss" report costs two calls — one catalog, one find. grep sweeps a subtree and returns line-numbered matches with citable evidence URIs (nokv-native://path@generation:N#L3).

The verbs live in nokv-agent: the tool definitions are LLM-ready JSON schemas, and executeagenttool routes calls over the same AgentNamespace trait whether the namespace is embedded (in-process) or remote (metadata RPC via nokv-client). The crate is transport-free — it depends only on nokv-meta, nokv-object, and nokv-types. See the contributor handbook.

Today the agent verbs ship in the Rust SDK; filesystem operations ship in the nokv CLI and FUSE mount. An MCP server is in development — follow #354.

📊 Measured Evidence

Agent interface. We gave the same agent (gpt-5.4-mini) the same 875-run experiment corpus through two surfaces — raw SQL over SQLite, and the NoKV namespace — across five tasks, 10 repeats per arm and task (100 fully stateless runs), judged against deterministic gold facts neither arm can see:

| Set mean (per 5-task pass) | Raw SQLite | NoKV namespace | | --- | --- | --- | | Tasks solved correctly | 4.40 / 5 | 4.50 / 5 | | Prompt tokens (incl. cached) | 151,572 | 82,827 (−45%) | | Cost (USD, list rates) | $0.0708 | $0.0433 (−39%) |

In this 10-repeat sample, the token gap widens to ~2.4× on the compound-exploration subset, and SQL won the single-shot analytics task — per-task results, wins and losses both, are in the report. Harness, tasks, judge, and the raw telemetry of all 100 runs are committed, so every published number is recomputable: see bench/agent-interface/.

Storage engine. Local engineering baselines, not official MLPerf results. Single-node service numbers are release builds through the NoKV server and Holt metadata path. FUSE comparison numbers depend on kernel/FUSE, object backend, cache settings, and workload shape.

| Workload | Result | | --- | --- | | Metadata create (mdtest, 65k records) | ~127K ops/s (single-writer, batched service path) | | Same, one directory of 65k entries | Same order of throughput; path-native ART does not degrade on large directories | | Checkpoint publish (1 MiB blocks, concurrency 16) | ~1.1 GiB/s in the service/object benchmark | | Dataset read (16 KiB samples, concurrency 16) | ~3,000 samples/s in the service/object benchmark | | Resident metadata | ~1.5 KiB / file in the measured shape | | Atomic checkpoint | Object bytes land first; metadata publishes a new generation atomically |

Same-machine FUSE-vs-FUSE smoke against one RustFS endpoint currently shows NoKV behind JuiceFS on the end-to-end mounted path. That gap is expected to come from FUSE/RPC fixed costs and data-plane cache/writeback maturity, not from the Holt metadata engine alone.

NoKV vs JuiceFS

NoKV follows the same high-level separation used by systems like JuiceFS and 3FS: metadata is separate from file body storage. The difference is that NoKV ships its metadata engine as part of the filesystem and optimizes for agent-workspace and artifact publish/read patterns first.

| | JuiceFS | NoKV | | --- | --- | --- | | Metadata engine | External DB such as Redis, MySQL, or TiKV | Built-in, path-native Holt engine | | Atomic checkpoint publish | POSIX rename/write semantics over the metadata engine | First-class publish-by-generation primitive | | Block model | Slice/block model supporting broad POSIX behavior | Immutable object blocks plus new-generation manifests | | Workspace-native primitives | Layered on top of the filesystem | Snapshots, typed watch, body descriptors, and GC floors are core metadata concepts | | Agent query surface | None | ls/stat/catalog/find/aggregate/read/grep with push-down and line-numbered evidence | | POSIX completeness | Mature production filesystem | P0 subset implemented; still hardening compatibility gates | | Maturity | Production, large deployments | Young Rust implementation; single-node local mode is usable, replication is roadmap |

NoKV is an object-backed filesystem with a sharded Holt metadata plane: multiple metadata shards (one Holt engine each) behind long-running metadata servers, routed through an etcd control plane, with cross-shard grafts presenting a single FUSE namespace across shards. Metadata HA today is single-writer-per-shard with checkpoint-image + shared-log, epoch-fenced failover — not yet consensus replication — so it is not yet a JuiceFS/3FS class production-HA distributed filesystem.

🏗️ Architecture

crates/
  nokv-types     storage-neutral namespace model types
  nokv-protocol  framed metadata RPC DTOs and binary codec
  nokv-meta      schema, MetadataCommand, Holt store, service core
  nokv-control   shard ownership, epochs, and failover coordination
  nokv-object    S3-compatible object body storage helpers
  nokv-agent     transport-free agent tool surface (the seven read verbs)
  nokv-client    Rust SDK over metadata service and object backend
  nokv-fuse      low-level FUSE frontend
  nokv-server    long-running metad process and framed RPC service
  nokv           CLI binary
bench/             system workload benchmark harness docs/              product, architecture, layout, RustFS, and benchmark docs

For artifact and checkpoint publish, object bytes are uploaded first, then the metadata commit publishes the dentry, inode projection, and body manifest atomically. A crash between the two leaves orphan objects for GC, never a corrupt namespace. See Architecture.

🚦 Quick Start

Build and test:

cargo test --workspace
cargo build --release -p nokv --bin nokv

Start a local RustFS-compatible S3 endpoint and create the default bucket:

mkdir -p /tmp/rustfs-data
RUSTFSACCESSKEY=rustfsadmin \
RUSTFSSECRETKEY=rustfsadmin \
rustfs server --address 127.0.0.1:9000 /tmp/rustfs-data &
AWSACCESSKEY_ID=rustfsadmin \ AWSSECRETACCESS_KEY=rustfsadmin \ aws --endpoint-url http://127.0.0.1:9000 \   s3api create-bucket --bucket nokv

By default NoKV expects bucket nokv at http://127.0.0.1:9000 with development credentials rustfsadmin / rustfsadmin. See docs/rustfs.md for other deployment modes.

Start the metadata server, then initialize the namespace. Every other command talks to the server on 127.0.0.1:7777, so keep it running:

cargo run --release -p nokv --bin nokv -- serve &
cargo run --release -p nokv --bin nokv -- init

Publish and read an artifact:

cargo run --release -p nokv --bin nokv -- \
  put-artifact /runs/1/checkpoint.bin ./checkpoint.bin
cargo run --release -p nokv --bin nokv -- \   cat /runs/1/checkpoint.bin > restored.bin

Mount with FUSE:

mkdir -p /tmp/nokv-mount
cargo run --release -p nokv --bin nokv -- \   mount /tmp/nokv-mount

On macOS this requires macFUSE. NoKV passes the noappledouble mount option to avoid Finder/resource-fork AppleDouble sidecars; user xattr roundtrip is covered by the FUSE smoke test.

🧩 Crates

| Crate | Role | | --- | --- | | nokv-types | Storage-neutral namespace model | | nokv-protocol | Framed metadata RPC DTOs and binary codec | | nokv-object | S3-compatible object body storage | | nokv-meta | Schema, MetadataCommand, Holt store, service core | | nokv-control | Shard ownership, epochs, and failover coordination | | nokv-agent | Transport-free agent tool surface (the seven verbs) | | nokv-client | Rust SDK over the metadata service | | nokv-fuse | Low-level FUSE frontend | | nokv-server | Long-running metad process and framed RPC | | nokv | nokv CLI binary |

✅ Current Status

Implemented today:

• low-level FUSE frontend for lookup, getattr, readdir, readdirplus, create,

mkdir, symlink/readlink, rename, unlink, rmdir, read, write, flush, release, fsync, setattr/truncate, hardlink, xattr, advisory locks, special files, statfs, lseek, fallocate, and copyfilerange;

• Holt-backed local metadata service with inode/dentry canonical metadata,

dentry projection, command predicates, command dedupe, and history records;

• chunked object data path where file bodies are split into immutable object

blocks and published by metadata manifest;

• S3-compatible object backend, with RustFS as the local development default;
• Rust SDK and nokv CLI for namespace operations, artifact publish,

metadata server access, and object range reads;

• the seven-verb agent query surface (ls/stat/catalog/find/

aggregate/read/grep) in the dedicated nokv-agent crate, re-exported through the Rust SDK, with LLM-ready tool definitions;

• long-running nokv-server with health, readiness, stats, manual GC, and

framed binary metadata RPC;

• nokv-control shard ownership store (in-memory plus optional etcd-backed

session leases behind the etcd feature) and a server shard-owner guard that installs and renews lease epochs into the metadata commit fence;

• multi-shard distributed metadata: subtree/path-prefix sharding (one Holt engine

per shard), high-bit shard-tagged global inodes, etcd control-plane routing with client re-resolve on owner handoff, and cross-shard grafts that present a single FUSE namespace across shards;

• logical metadata log segment codec/archive/replay foundation, plus controlled

server sync shared-log ACK mode that publishes LogRef before successful RPC ACKs, including grouped independent-batch log segments;

• controlled metadata failover smoke that restores a checkpoint, replays the

shared log, starts the bumped-epoch owner, and accepts a new metadata write;

• local multi-process metadata HA + multi-shard fleet smoke scripts that exercise

etcd ownership, RustFS-backed checkpoint/log archive, owner death, epoch failover, post-failover replay, and a SIGSTOP/SIGCONT stale-owner fence mode;

• a Python SDK (PyO3) and fsspec filesystem with reads, writes, namespace ops,

snapshots, atomic checkpoint publish/resolve, and a torch DataLoader + DCP backend;

• read-only snapshot mounts, snapshot-version reads, typed watch replay, and

FUSE cache invalidation from watch events;

• pending-object GC and metadata history GC tied to snapshot retention.

Not implemented yet:

• consensus-replicated metadata (Raft/Paxos) — HA today is single-writer-per-shard

with checkpoint + shared-log failover, not replicated;

• intra-subtree sharding (a single hot subtree is capped at one shard), learner

read scaling, and chaos-tested failover timing;

• an MCP server for the agent verbs — in development, tracked in

#354;

• Kubernetes CSI packages;
• full POSIX hardening such as ACL enforcement, broad external compatibility

gate coverage, and mature multi-client cache coherence.

Benchmarks

The root bench/ package contains all benchmark entry points. System workload runs use nokv-bench:

cargo run --release -p nokv-bench --bin nokv-bench -- \
  --profile smoke \
  --workload all

Key workloads:

• mdtest-easy and mdtest-hard metadata smoke workloads;
• metadata-negative-lookup, artifact-index-lookup, and

metadata-concurrent-read Holt metadata read-path workloads;

• metadata-durability-batch batch metadata create workload with comparable

local-only and sync-shared-log ACK phases;

• checkpoint-publish object-backed checkpoint publish/read;
• training-read dataset-shaped object reads;
• mlperf-dlio generated MLPerf Storage/DLIO-style training and checkpoint

shape;

• metadata HA smoke through scripts/run-metadata-ha-smoke.sh for owner leases,

epoch fencing, checkpoint restore, shared-log replay, failover RTO timing, and stale-owner write rejection.

All workloads are single-node service runs; see docs/benchmarks.md for the full workload list, profiles, and gates.

The agent-interface benchmark — harness, tasks, judge, report, and the raw telemetry behind the numbers above — lives under bench/agent-interface/ and runs through the same package:

cargo run --release -p nokv-bench --bin yanex-agent-bench -- list-tasks

For the fast AI-training product gate, run:

scripts/run-ai-training-smoke.sh

The default gate covers Holt metadata read concurrency, checkpoint publish, and DLIO-style object reads/writes. Most benchmark workloads are still single-node service runs. Training-cluster claims need separate runs that report replication, cache, object-store, and durability settings.

Run scripts/run-ai-training-smoke.sh fuse-smoke when the local machine has a working FUSE installation and you want the mounted POSIX smoke in the same workflow.

For the local metadata HA gate, run:

scripts/run-metadata-ha-smoke.sh

It requires RustFS, AWS CLI, curl, and either a local etcd binary or NOKVHAETCD_ENDPOINTS pointing at an external etcd cluster. Set NOKVHAMETRICS_JSON=/tmp/nokv-ha.json to keep the emitted HASMOKEMETRICS JSON for CI or benchmark reports. Set NOKVHASTALEOWNERCHAOS=1 to run the local stale-owner fence mode; that mode uses NOKVHAOWNERBBIND for the replacement owner.

📚 Documentation

• Architecture
• Product Design
• AI-Native Metadata HA And Fast Path
• Metadata Schema
• Object Layout
• Checkpointing
• RustFS Backend
• Benchmarks

📄 License

Apache-2.0. See LICENSE.