On-disk substrate¶

Goals of the layout¶

Portable. tar it, mail it, untar it on another machine, run anchor serve --data-dir ./received. Same canvases, same documents.
Inspectable. Plain .json and .md. cat works. jq works. git log of the data folder tells a coherent story.
Separate lifecycles. Deleting a canvas does not delete its source documents. Removing a source document can leave unresolved references on canvases that cited it.
Producer-extensible. Every extension owns a top-level folder. Adding a new extension doesn't change paths owned by existing ones.

The full tree¶

The data/ root below is a project's data directory: the hidden .anchor_data/ folder inside the project folder (<project>/.anchor_data/, or ~/.anchor/envs/<env>/projects/<name>/.anchor_data/ for a managed project). Its internal layout is the same wherever the project lives.

data/  ( = the project's .anchor_data/ )
|
+-- canvases/                       (canvas core)
|   +-- <slug>/
|       +-- meta.json               slug, title, created_at, schema_version
|       +-- events.jsonl            append-only event log, one event per line
|       +-- state.json              cached fold of events at the current version
|
+-- bronze/                         (anchor_pdfs)
|   +-- <slug>/
|       +-- <sha256>.pdf            immutable original bytes owned by this document
|       +-- original.json           slug, filename, sha256, retained legacy ownership
|   +-- <original-filename>.pdf     legacy originals only, never overwritten by new ingest
|
+-- silver/                         (anchor_pdfs)
|   +-- <slug>/
|       +-- index.json              document record: title, filename, source, page_count, outline
|       +-- pages.meta.json         per-page metadata (dimensions, word counts, ...)
|       +-- ingest-report.json      pipeline outcome: status, stage, region_count, ...
|       +-- pages/
|           +-- <n>.md              polished markdown for page n
|           +-- <n>.raw.md          raw Docling markdown for page n
|           +-- <n>.png             rasterised page image
|           +-- <n>.candidates.json region candidates proposed for page n
|           +-- <n>.items.json      uncapped silver source items with stable IDs and cells
|
+-- gold/                           (anchor_pdfs)
|   +-- <slug>/
|       +-- .complete.json          completeness marker: {"complete": true, ...meta}
|       +-- embeddings.json         BGE embeddings for vector search
|       +-- pages/
|           +-- <n>.regions.json    structured regions for page n (id, kind, source_ref, ...)
|           +-- <n>/
|               +-- <region-id>.png cropped screenshot of the region bbox
|
+-- knowledge/                      (anchor_pdfs, derived producer records)
|   +-- <slug>/
|       +-- pages/
|           +-- <n>.regions.json    chart series, normalized specs, calculated artifacts
|
+-- ingest_status/                  (anchor_pdfs, written during active ingestion)
|   +-- <slug>.json                 live activity record: stage, progress, started_at, ...
|
+-- staging/                        (anchor_pdfs, harness-driven ingest sessions)
|   +-- ingest/
|       +-- <session-id>/
|           +-- session.json        session state: slug, steps, status
|           +-- journal.jsonl       per-step append-only log
|           +-- silver/pages/       staged silver pages (pre-publish)
|           +-- gold/pages/         staged gold regions (pre-publish)
|
+-- fmus/                           (anchor_fmus)
|   +-- bronze/
|   |   +-- <slug>.fmu              raw FMU upload
|   +-- models/
|   |   +-- <slug>.json             parsed modelDescription summary (FmuModel)
|   +-- simulations/
|       +-- <sim-id>/
|           +-- run.json            SimulationRun metadata: fmu_slug, params, timestamps
|           +-- series.json         TimeSeries data: signals, time vector
|
+-- cad/                            (anchor_cad)
    +-- bronze/
    |   +-- <slug>.<ext>            raw input model file
    +-- artefacts/
        +-- <slug>/
            +-- document.json       OIP-style document metadata
            +-- model.json          CadModel summary: parameters, parts, geometry

Per-canvas folder, line by line¶

data/canvases/<slug>/
+-- meta.json
|     {
|       "slug": "alfa-laval-eval",
|       "title": "Alfa Laval LKH evaluation",
|       "created_at": 1746000000.0,
|       "schema_version": "0.2"
|     }
|
+-- events.jsonl
|     {"id": "uuid", "ts": ..., "version": 1, "type": "NodeAdded", ...}
|     {"id": "uuid", "ts": ..., "version": 2, "type": "EdgeAdded", ...}
|     ...
|
+-- state.json
      {
        "slug": "alfa-laval-eval",
        "version": 142,
        "nodes": { "n1": {...}, "n2": {...}, ... },
        "edges": { "e1": {...}, ... }
      }

events.jsonl is the truth. state.json is a performance optimisation: without it a cold boot replays every event from the log; with it the server reads the snapshot and replays only events that are newer. Compaction (rewriting the snapshot at checkpoints, optionally truncating event history) is on the roadmap but not implemented; today the log grows monotonically.

Per-document artifact layout (anchor_pdfs)¶

Bronze¶

New originals belong to a document slug. The uploaded filename is display metadata, never the address of a new source. Ingest writes immutable bytes to bronze/<slug>/<sha256>.pdf and records {slug, sha256} in silver/<slug>/index.json under document.source. Extraction and rendering read that exact stored file. Original retrieval verifies the indexed hash; missing or inconsistent authoritative data never falls back to a filename.

original.json records the latest accepted stash, its display filename, and any legacy_sources claims preserved before replacing an old index. It lets a harness fetch an uploaded original before silver exists. Once an index exists, its source binding wins over this pending receipt.

Two slugs may upload the same basename with equal or different bytes: each keeps its own original. Repeating the same bytes reuses that document's hash file. Existing gold-skip and open-session-resume behavior is unchanged unless force is requested. Replacing one slug retains earlier hash files; it does not update another slug's original.

Flat filename originals remain read-only legacy data. Fallback requires one unambiguous owner, or recorded document.source_sha256 fingerprints matching the file for every competing owner. Those fingerprints must come from independent source evidence, not be invented from the currently surviving file. Multiple unproven claims raise an explicit ownership error. Missing originals return unavailable; malformed metadata and fingerprint mismatches raise an error. No automatic migration or repair runs.

Legacy checks conservatively include current filename claims and retained historical claims. Re-ingesting or renaming one owner cannot erase a known conflict. A new document with the same display filename can therefore make an un-fingerprinted legacy original unavailable until its ownership is resolved. Document-owned originals remain available independently.

Existing upload-name and slug validators apply. Unicode filenames are preserved as metadata; path components are rejected. Case variants of existing slugs, trailing-dot aliases, reserved filesystem names, and unsafe source paths are rejected. Identity conflicts fail before activity, silver, gold, embeddings, or harness placeholder/intent publication.

This binds originals, not complete ingestion generations. Forced replacement of one slug can still leave stale pages, gold, embeddings, or polished text. Generation publication and cleanup require separate lifecycle work.

data/bronze/
    alfa-laval-lkh/
        original.json
        <sha256>.pdf
    grundfos-tp/
        original.json
        <sha256>.pdf

Silver¶

Silver stores the Docling extraction for each document under a per-slug directory.

data/silver/alfa-laval-lkh/
    index.json           document record: title, filename, source, page_count, outline
    pages.meta.json      per-page metadata (dimensions, word counts, rotation flags)
    ingest-report.json   outcome written at the end of ingest: status, stage, region_count
    pages/
        1.md             polished markdown for page 1
        1.raw.md         raw Docling output for page 1 (pre-polish)
        1.png            rasterised page image (used by the region extractor)
        1.candidates.json region candidates proposed for page 1
        1.items.json     full source items, text, cells, ordering and extractor metadata
        2.md
        ...

New Docling extractions interleave text, tables, and pictures in the document tree's reading order. Candidate IDs (p<page>-i<index>) and pages.meta.json item IDs follow that sequence; raw Markdown uses the same order. Candidates also retain the Docling traversal ordinal as reading_order, which can have gaps between items. Existing stored IDs are not renumbered. Items without traversal metadata keep their collection order, and Markdown without that metadata retains its geometric fallback.

Candidate text remains a bounded grouping preview. The private full-item artifact uses the same p<page>-i<index> IDs and preserves uncapped source text, precise table cells and extractor metadata. It is written by keyed, text-profile and harness ingest, and selected by the same generation pointer.

Original gold chunks¶

New original chunks store semantic metadata, stable silver membership and display geometry. They do not persist duplicate content, cells or table_topology. Region, gold-map, inspect and content reads reconstruct those fields from the selected generation's full silver items, including the exact rows and columns named by a table_slice. Pointed extraction and value provenance consume the same reconstructed cells.

Source embeddings use title + description + tags + entities when a meaningful description exists. Undescribed coverage chunks fall back to their uncapped reconstructed source text and cells. Producer-derived records retain their stored-content recall. The agent workflow is search, inspect the selected chunk, then answer from its silver evidence; descriptions are search metadata.

Legacy documents without full-item artifacts retain stored source content and cells. Reads do not migrate or rewrite their gold. Producer-derived payloads remain authoritative even when full silver exists.

Knowledge¶

New derive_region records live in knowledge/<slug>/pages/<n>.regions.json. A selected replacement uses knowledge/<slug>/generations/<id>/pages/<n>.regions.json. The same document generation pointer selects silver, gold and knowledge. A replacement starts with empty knowledge, and pinned operations can finish in the retained old generation without writing to the current one.

Derived records retain their producer payload and provenance. Existing derived gold records with a truthy derived_from are read, updated and removed in place. Compatibility region and gold-map reads combine both collections; source coverage and completeness checks exclude derived records. Crops and the combined embedding cache retain their existing gold paths.

Gold¶

Gold stores structured regions, crops, and embeddings under a per-slug directory. A .complete.json marker is written atomically at the end of a successful gold pass. A partial or crashed pass leaves the directory without the marker so the store never presents incomplete gold.

data/gold/alfa-laval-lkh/
    .complete.json       {"complete": true, "mode": "keyed", "model": "gpt-5.4", ...}
    embeddings.json      {"embed_model": "hotchpotch/bekko-embedding-v1-a25m", "dim": 384, "vectors": [...]}
    pages/
        1.regions.json   {"page": 1, "regions": [{id, kind, source_ref, content}, ...]}
        2.regions.json
        ...
        1/
            r0007.png    cropped screenshot of region r0007 on page 1
            r0012.png
            ...
        2/
            ...

A region record inside <n>.regions.json looks like:

{
  "id": "alfa-laval-lkh:r0007",
  "kind": "spec_block",
  "title": "Pump performance - model LKH-5",
  "source_ref": {
    "kind": "pdf-page-bbox",
    "page": 2,
    "bbox": [72.0, 144.0, 540.0, 312.0]
  },
  "content": {
    "text": "...",
    "rows": [...]
  },
  "crops": {
    "png": "2/r0007.png"
  }
}

This is the unit of provenance. Every value on the canvas points back to a region. A spec-table row is a data field on a node; that field carries the region id; the region carries the page and bbox; the bbox is rendered as a highlight in the PDF viewer. There is no detached metadata table: the source ref is the data.

Ingest status¶

A small live-activity record written as the pipeline advances. It lands in its own ingest_status/ directory so it never collides with corpus artifacts and is trivial to list.

data/ingest_status/
    alfa-laval-lkh.json   {"slug": "...", "stage": "gold", "progress": 0.8, ...}
    grundfos-tp.json

The record is created when ingest starts and updated after each stage. It survives a process restart, so an agent polling across process boundaries gets accurate progress without re-running the pipeline.

Staging (harness-driven ingest sessions)¶

In-flight harness-driven sessions live under staging/ingest/ and are invisible to list_documents until the session is finalized and published to silver/gold.

data/staging/ingest/<session-id>/
    session.json          {slug, steps, current_step, status, created_at}
    journal.jsonl         per-step log (one JSON object per line)
    silver/pages/         staged silver pages, same layout as silver/<slug>/pages/
    gold/pages/           staged gold regions, same layout as gold/<slug>/pages/

FMU layout (anchor_fmus)¶

FMUs use a separate subdirectory tree under fmus/ inside the data dir.

data/fmus/
    bronze/
        <slug>.fmu          raw FMU file
    models/
        <slug>.json         parsed modelDescription summary (FmuModel)
    simulations/
        <sim-id>/
            run.json        SimulationRun: fmu_slug, params, started_at, finished_at
            series.json     TimeSeries: signal names, time vector, values

CAD layout (anchor_cad)¶

data/cad/
    bronze/
        <slug>.<ext>        raw input model file (preserves original extension)
    artefacts/
        <slug>/
            document.json   OIP-style document metadata
            model.json      CadModel: parameters, parts, geometry summary

Why JSONL events instead of a database¶

Events are tiny, mostly less than 1 KB. A canvas with 10 000 events is 10 MB.
One file per workspace makes copying, moving, and diffing trivial.
An append-only file is the simplest possible crash-safe write. No transactions, no migrations, no DBA required.
A future Postgres or SQLite store is a port (the WorkspaceStore protocol). The day we need indexed queries across workspaces is the day we add a second implementation. The file format stays.

Multi-tenant note¶

The current layout assumes one user, one machine. The multi-tenant roadmap (separate memory entry) adds a tenants/<tenant>/ prefix and moves canvases under tenants/<tenant>/canvases/. Documents stay shared by default with opt-in scoping. Nothing in the current paths needs to change to accommodate that: they become subpaths.