Learning Datagraph — SLM trajectory loop and apprenticeship queue

@@ -2,7 +2,7 @@
schema: foundry-doc-v1
title: "Learning Datagraph — SLM trajectory loop and apprenticeship queue"
slug: learning-datagraph-architecture
short_description: "Four-leg training loop turning operator interactions into training tuples — trajectory capture, apprenticeship queue, editorial DPO pairs, and correction distillation."
short_description: "Training loop turning operator interactions into training signal — trajectory capture, apprenticeship queue, and a real GLiNER→OLMo distillation pipeline (not the editorial DPO leg earlier versions of this article described)."
language: en
category: ai
type: topic
@@ -10,7 +10,7 @@ content_type: topic
index_group: entity-extraction-and-the-training-loop
status: active
bcsc_class: public-disclosure-safe
last_edited: 2026-05-25
last_edited: 2026-08-17
editor: pointsav-engineering
cites: []
paired_with: learning-datagraph-architecture.es.md
@@ -21,32 +21,28 @@ The platform builds a [[compounding-substrate|compounding substrate]]: every ope

## Key Takeaways

- The substrate accumulates training signal through four distinct legs: trajectory capture at session end, an apprenticeship queue that fires on every commit, editorial DPO pairs from the reverse-funnel editorial pipeline, and negative-trajectory distillation from operator corrections. Each leg captures a different dimension of operator intent.
- All training signal passes through the same auditable boundary — [[doorman-protocol|Doorman]] — and lands in the append-only audit ledger. Nothing bypasses the ledger; nothing leaves the local environment. The learning loop is air-gapped and self-contained.
- The corpus accumulates with every session. As of mid-2026 the apprenticeship corpus held 502 tuples and the editorial DPO corpus held 34 pairs. These numbers grow without manual curation — the model floor rises as the operator uses the environment.
- **Correction (2026-08-02):** this leg understates real progress. The `POST /v1/draft/generate` endpoint is already built and deployed — `service-content/src/http.rs:176-280` implements it in full (LadybugDB query, prompt packaging, Doorman proxy call), and a live probe returns HTTP 422 (validation error), not 404, confirming the route is live. Per `service-content/CLAUDE.md`'s own status table, the actual remaining gap is a Tier C provider-authentication configuration step, not a multi-week Rust engineering effort building the endpoint from scratch. **Flagged, not resolved** — needs updating to reflect the endpoint's real (built, not-yet-authenticated) status.
- The substrate accumulates training signal through several distinct mechanisms: trajectory capture at session end, an apprenticeship queue that fires on every commit, and a real GLiNER→OLMo distillation pipeline — not the "editorial reverse-funnel DPO" leg earlier versions of this article described, which has no support in the codebase.
- All training signal passes through the same auditable boundary — [[doorman-protocol|Doorman]] — and lands in the append-only audit ledger. The graph-tenant-scoping behavior described below is a real example of that boundary being enforced, not just described.
- Specific corpus-size figures (tuple counts, pair counts) are not repeated here — the only figures verified against real code are dated 2026-04-29 and already stale (14 apprenticeship files, an empty feedback corpus); a fresher-sounding but unverified number would be worse than none.
- `POST /v1/draft/generate` is real, built, and live — `service-content/src/http.rs:352-479` (not the previously-cited `176-280`), confirmed by direct read, not a live probe's status code alone. Its own doc comment calls it the "Tier C Drafting Pipeline" and proxies to **Claude Haiku 4.5** (`anthropic:claude-haiku-4-5-20251001`) — not Claude 3.5 Sonnet as an earlier correction claimed. No `service-content/CLAUDE.md` exists anywhere in the monorepo; that citation and the model-version claim do not correspond to anything in the codebase.

- The one leg not yet wired is the structured-entity loop: a `POST /v1/draft/generate` endpoint in [[service-content]] that would ground generation in graph entities. The supporting infrastructure (queue, ledger, hooks, audit routing) is already in place; what remains is a multi-week Rust engineering effort.
## Training signal mechanisms

## Four legs of training signal
**Trajectory capture.** A session-end hook fires at session close. The real `capture-trajectory.sh` posts a free-text session summary wrapped in an apprenticeship-brief JSON schema (`brief_id`/`senior_role`/`task_type`/`body`) — not a fixed field set of branch-state/uncommitted-file-count/head-SHA as earlier text claimed.

The substrate has four legs.
**Apprenticeship queue.** A post-commit hook fires a shadow brief via `POST /v1/shadow` for commits across 8 clusters — the specific "15-minute drainer" interval in earlier text has no support in the code or in `service-slm/docs/trainer-scoping.md`, the closest real documentation of this mechanism. The local model in this loop is **OLMo 3** (`OLMo-3-7B-Q4_K_M.gguf`, `slm-doorman/src/tier/local.rs`; also `olmo-3-1125-7b-q4` in `slm-core/src/apprenticeship.rs`) — not "OLMo-2 7B Q4"; no reference to an OLMo-2 model exists anywhere in the codebase.

**Trajectory capture.** A session-end hook fires at session close, writing a structured JSONL entry to the audit ledger: branch state, uncommitted-file count, head SHA, and a promotion-pending flag (Correction, 2026-08-02: the real `capture-trajectory.sh` posts a free-text session summary wrapped in an apprenticeship-brief JSON — `brief_id`/`senior_role`/`task_type`/`body` — not this specific field set; flagged, not resolved). A nightly harvest copies the day's session transcripts into the same ledger, tagged by operator and archive.
**The real DPO mechanism is GLiNER→OLMo distillation, not editorial reverse-funnel pairs.** `service-content/src/lib.rs` (`write_gliner_olmo_dpo_pair`, around lines 279–282 and 659–690) shows the actual DPO pipeline: GLiNER (Tier 0 extraction, `:9085`) proposes entities, OLMo 7B (Tier A) is asked to reproduce or improve on the extraction, and the delta becomes a DPO pair — an entity-extraction-quality signal, not an editorial-voice signal from a "raw → refined → creative-edited" pipeline. No such reverse-funnel-to-DPO-pair mechanism exists in the code; earlier text describing one was not grounded in the codebase.

**Apprenticeship queue.** A post-commit hook emits a brief for every workspace commit. A 15-minute queue drainer calls the local SLM (OLMo-2 7B Q4) against each brief, captures the model's attempt, and writes the `(brief, attempt, actual_diff)` tuple to the [[apprenticeship-substrate|apprenticeship corpus]]. 502 tuples had accumulated as of 2026-05-18.
**Negative-trajectory distillation.** An inbox-scanner script reads operator corrections from archived messages and emits negative-trajectory signals to the feedback corpus. Carried forward as previously described; unlike the other three mechanisms above, this one has not been independently checked against the current codebase.

**Editorial DPO pairs.** Every draft that passes through the reverse-funnel editorial pattern — raw to refined to creative-edited — emits two DPO (direct preference optimisation) pairs to the prose-edit corpus. The pair captures the editorial improvement deltas. 34 pairs had accumulated to that date.
## The structured-entity loop, and the tenant-scoping behavior it actually has

**Negative-trajectory distillation.** An inbox-scanner script reads operator corrections from archived messages and emits negative-trajectory signals to the feedback corpus. This fourth leg captures what the model should not do.
`POST /v1/draft/generate` genuinely grounds generation in graph entities — it queries `state.graph.query_context` and an induced edge subgraph before calling out, matching the "grounds generation in graph entities" description. What it does **not** do: call a LoRA scheduler or wake Tier B GPU compute. Earlier text claimed "a LoRA scheduler then wakes Tier B GPU compute for nightly adapter training" downstream of this endpoint — no such wiring exists in `draft_generate`; it is a synchronous Doorman `/v1/audit/proxy` call to a cloud model (Claude Haiku 4.5, see above), full stop.

## Structured-entity loop — the remaining leg
**Not previously documented at all, despite being load-bearing:** every graph query this endpoint makes is subject to the same tenant-isolation enforcement described in [[doorman-protocol]] — `graph_query`/`graph_mutate` scope strictly to the caller's own module, no cross-tenant merge, since the fix confirmed live 2026-07-28. A structured-entity draft for one tenant cannot ground itself in another tenant's graph entities. This matters for exactly the DOCTRINE claim #48 reason [[doorman-protocol]] covers in more depth — customer-owned graph IP never aggregates across tenants without opt-in — and applies here even though nothing in this article's earlier text ever mentioned it.

**Correction (2026-08-02):** see the Key Takeaways correction above — this endpoint is already built and live, not "remains to wire." The description of what it does is otherwise accurate.

What remains to wire — multi-week Rust engineering effort: the structured-entity loop. [[service-content]] (LadybugDB-backed graph) needs a `POST /v1/draft/generate` endpoint that queries the graph for relevant entities, assembles a 2K-token grounded prompt, calls the [[doorman-protocol|Doorman]], and writes the response as a graph-grounded corpus tuple. A LoRA scheduler then wakes Tier B [[yoyo-compute-substrate|GPU compute]] for nightly [[elastic-compute-lora-training-pipeline|adapter training]]. The supporting infrastructure — queue, ledger, hooks, audit-routing — is already in place.

The substrate compounds in two directions: structurally (citation density and supersedence chains thicken with each draft) and generatively (each adapter raises the floor of "raw" so each refinement cycle starts closer to publish-ready).
The substrate compounds in two directions in principle. Structurally, citation density and supersedence chains thicken with each draft. Generatively, each adapter raises the floor of "raw" so refinement starts closer to publish-ready — but that generative half depends on the LoRA training pipeline described in [[elastic-compute-lora-training-pipeline]], which this article does not itself verify.

## See also