Diff: architecture/learning-datagraph-architecture.es
From 1c02ec1 to 1c02ec1
+0 / −0 lines
| Before | After |
|---|---|
| --- | --- |
| schema: foundry-doc-v1 | schema: foundry-doc-v1 |
| title: "Learning Datagraph — SLM trajectory loop and apprenticeship queue" | title: "Learning Datagraph — SLM trajectory loop and apprenticeship queue" |
| slug: learning-datagraph-architecture | slug: learning-datagraph-architecture |
| language: en | language: en |
| category: architecture | category: architecture |
| type: topic | type: topic |
| content_type: topic | content_type: topic |
| status: active | status: active |
| bcsc_class: public-disclosure-safe | bcsc_class: public-disclosure-safe |
| last_edited: 2026-05-25 | last_edited: 2026-05-25 |
| editor: pointsav-engineering | editor: pointsav-engineering |
| cites: [] | cites: [] |
| paired_with: learning-datagraph-architecture.es.md | paired_with: learning-datagraph-architecture.es.md |
| --- | --- |
| The platform builds a [[compounding-substrate|compounding substrate]]: every operator interaction with an AI session becomes a structured training tuple, routed through a single auditable boundary ([[doorman-protocol|Doorman]]), captured to an [[worm-ledger-architecture|append-only ledger]], and folded back into the local SLM via periodic fine-tuning. The result is a development environment that learns from how it gets used — code completions improve toward the patterns this operator writes, draft suggestions align closer to the editorial voice this house produces, entity extractions tighten as the graph thickens. | The platform builds a [[compounding-substrate|compounding substrate]]: every operator interaction with an AI session becomes a structured training tuple, routed through a single auditable boundary ([[doorman-protocol|Doorman]]), captured to an [[worm-ledger-architecture|append-only ledger]], and folded back into the local SLM via periodic fine-tuning. The result is a development environment that learns from how it gets used — code completions improve toward the patterns this operator writes, draft suggestions align closer to the editorial voice this house produces, entity extractions tighten as the graph thickens. |
| ## Key Takeaways | ## Key Takeaways |
| - The substrate accumulates training signal through four distinct legs: trajectory capture at session end, an apprenticeship queue that fires on every commit, editorial DPO pairs from the reverse-funnel editorial pipeline, and negative-trajectory distillation from operator corrections. Each leg captures a different dimension of operator intent. | - The substrate accumulates training signal through four distinct legs: trajectory capture at session end, an apprenticeship queue that fires on every commit, editorial DPO pairs from the reverse-funnel editorial pipeline, and negative-trajectory distillation from operator corrections. Each leg captures a different dimension of operator intent. |
| - All training signal passes through the same auditable boundary — [[doorman-protocol|Doorman]] — and lands in the append-only audit ledger. Nothing bypasses the ledger; nothing leaves the local environment. The learning loop is air-gapped and self-contained. | - All training signal passes through the same auditable boundary — [[doorman-protocol|Doorman]] — and lands in the append-only audit ledger. Nothing bypasses the ledger; nothing leaves the local environment. The learning loop is air-gapped and self-contained. |
| - The corpus accumulates with every session. As of mid-2026 the apprenticeship corpus held 502 tuples and the editorial DPO corpus held 34 pairs. These numbers grow without manual curation — the model floor rises as the operator uses the environment. | - The corpus accumulates with every session. As of mid-2026 the apprenticeship corpus held 502 tuples and the editorial DPO corpus held 34 pairs. These numbers grow without manual curation — the model floor rises as the operator uses the environment. |
| - The one leg not yet wired is the structured-entity loop: a `POST /v1/draft/generate` endpoint in [[service-content]] that would ground generation in graph entities. The supporting infrastructure (queue, ledger, hooks, audit routing) is already in place; what remains is a multi-week Rust engineering effort. | - The one leg not yet wired is the structured-entity loop: a `POST /v1/draft/generate` endpoint in [[service-content]] that would ground generation in graph entities. The supporting infrastructure (queue, ledger, hooks, audit routing) is already in place; what remains is a multi-week Rust engineering effort. |
| The substrate has four legs. | The substrate has four legs. |
| **Trajectory capture.** A session-end hook fires at session close, writing a structured JSONL entry to the audit ledger: branch state, uncommitted-file count, head SHA, and a promotion-pending flag. A nightly harvest copies the day's session transcripts into the same ledger, tagged by operator and archive. | **Trajectory capture.** A session-end hook fires at session close, writing a structured JSONL entry to the audit ledger: branch state, uncommitted-file count, head SHA, and a promotion-pending flag. A nightly harvest copies the day's session transcripts into the same ledger, tagged by operator and archive. |
| **Apprenticeship queue.** A post-commit hook emits a brief for every workspace commit. A 15-minute queue drainer calls the local SLM (OLMo-2 7B Q4) against each brief, captures the model's attempt, and writes the `(brief, attempt, actual_diff)` tuple to the [[apprenticeship-substrate|apprenticeship corpus]]. 502 tuples had accumulated as of 2026-05-18. | **Apprenticeship queue.** A post-commit hook emits a brief for every workspace commit. A 15-minute queue drainer calls the local SLM (OLMo-2 7B Q4) against each brief, captures the model's attempt, and writes the `(brief, attempt, actual_diff)` tuple to the [[apprenticeship-substrate|apprenticeship corpus]]. 502 tuples had accumulated as of 2026-05-18. |
| **Editorial DPO pairs.** Every draft that passes through the reverse-funnel editorial pattern — raw to refined to creative-edited — emits two DPO (direct preference optimisation) pairs to the prose-edit corpus. The pair captures the editorial improvement deltas. 34 pairs had accumulated to that date. | **Editorial DPO pairs.** Every draft that passes through the reverse-funnel editorial pattern — raw to refined to creative-edited — emits two DPO (direct preference optimisation) pairs to the prose-edit corpus. The pair captures the editorial improvement deltas. 34 pairs had accumulated to that date. |
| **Negative-trajectory distillation.** An inbox-scanner script reads operator corrections from archived messages and emits negative-trajectory signals to the feedback corpus. This fourth leg captures what the model should not do. | **Negative-trajectory distillation.** An inbox-scanner script reads operator corrections from archived messages and emits negative-trajectory signals to the feedback corpus. This fourth leg captures what the model should not do. |
| What remains to wire — multi-week Rust engineering effort: the structured-entity loop. [[service-content]] (LadybugDB-backed graph) needs a `POST /v1/draft/generate` endpoint that queries the graph for relevant entities, assembles a 2K-token grounded prompt, calls the [[doorman-protocol|Doorman]], and writes the response as a graph-grounded corpus tuple. A LoRA scheduler then wakes Tier B [[yoyo-compute-substrate|GPU compute]] for nightly [[elastic-compute-lora-training-pipeline|adapter training]]. The supporting infrastructure — queue, ledger, hooks, audit-routing — is already in place. | What remains to wire — multi-week Rust engineering effort: the structured-entity loop. [[service-content]] (LadybugDB-backed graph) needs a `POST /v1/draft/generate` endpoint that queries the graph for relevant entities, assembles a 2K-token grounded prompt, calls the [[doorman-protocol|Doorman]], and writes the response as a graph-grounded corpus tuple. A LoRA scheduler then wakes Tier B [[yoyo-compute-substrate|GPU compute]] for nightly [[elastic-compute-lora-training-pipeline|adapter training]]. The supporting infrastructure — queue, ledger, hooks, audit-routing — is already in place. |
| The substrate compounds in two directions: structurally (citation density and supersedence chains thicken with each draft) and generatively (each adapter raises the floor of "raw" so each refinement cycle starts closer to publish-ready). | The substrate compounds in two directions: structurally (citation density and supersedence chains thicken with each draft) and generatively (each adapter raises the floor of "raw" so each refinement cycle starts closer to publish-ready). |
| ## See also | ## See also |
| - [[compounding-substrate]] — the substrate discipline this architecture instantiates | - [[compounding-substrate]] — the substrate discipline this architecture instantiates |
| - [[service-slm]] — the local SLM service that executes model inference in the loop | - [[service-slm]] — the local SLM service that executes model inference in the loop |
| - [[totebox-session]] — the session model that trajectory capture instruments at session end | - [[totebox-session]] — the session model that trajectory capture instruments at session end |
| - [[mailbox-atomicity]] — the atomic prepend discipline that protects the audit ledger from concurrent write races | - [[mailbox-atomicity]] — the atomic prepend discipline that protects the audit ledger from concurrent write races |