Knowledge flow: training loop and ontological DataGraph
← All revisions
fix(ai): Track-B rewrite flow-quality-architecture — entity resolution is 3 stages not 4 (no clustering stage, alias table explicitly deferred as a separate migration), eval gate compares against a static pass-rate threshold not an incumbent-adapter gold set, confidence field is unconditionally overwritten not reconciled despite being the trust-signal field, temporal/bitemporal versioning corrected from claimed-current to its actual planned status (matches this article's own Target-state section, was an internal contradiction), dropped unsourced latency figures (EN+ES)
@@ -11,7 +11,7 @@ status: active audience: vendor-public bcsc_class: planned language_protocol: PROSE-TOPIC last_edited: 2026-06-28 last_edited: 2026-08-17 editor: editorial short_description: "Quality framework for the Totebox knowledge flow, asking whether LoRA adapters measurably improve the model and whether the DataGraph is an accurate ontology." paired_with: flow-quality-architecture.es.md @@ -23,8 +23,8 @@ The Totebox knowledge flow turns prose into two durable assets: an [[ontological ``` prose ─▶ service-extraction ─▶ CORPUS_*.json ─▶ service-content ──▶ GLiNER Tier 0 (~150ms) ─▶ entity spans └──▶ OLMo Tier A (30–137s) ─▶ extraction fallback (Tier 0 unreachable) + async DPO queue ─▶ service-content ──▶ GLiNER Tier 0 (fast — no sourced figure; only a qualitative "~150x faster than OLMo" code comment) ─▶ entity spans └──▶ OLMo Tier A (fallback, no sourced latency figure) ─▶ extraction fallback (Tier 0 unreachable) + async training-pass job └──▶ GPU Tier B (enrichment) ─▶ role/location vectors └──▶ LadybugDB graph ◀── Doorman queries the graph for context before each inference ◀── @@ -32,15 +32,23 @@ training corpus (git-commit shadow tuples + Tier-A-vs-B enrichment pairs) ─▶ TRL SFT / DPO ─▶ PEFT adapter ``` The Tier-0-success path enqueues a `TierAJob` for an asynchronous Tier A *training* pass, not a "DPO queue" as earlier text labeled it — DPO pair generation happens later, downstream in `service-slm`, not at this queue. Two quality questions decide whether the flow is worth its cost: is the **training loop** producing adapters that measurably improve the model, and is the **DataGraph** an accurate, well-resolved ontology rather than a pile of fragments? ## How the loop is intended to close A healthy training loop is a closed circuit: corpus → SFT → on-policy DPO → an **eval gate** that compares the new adapter to the incumbent on a frozen, version-hashed gold set → promotion only on a measured improvement → the promoted adapter **served** on the inference path → its behaviour captured back into the corpus. The gold standard for each stage is well established: adapters attach to all linear projections of the base model and the attachment is asserted post-build; learning rates sit an order of magnitude above full fine-tuning; preference training runs only above a clean, diverse pair floor; and no adapter is promoted that a base-versus-adapter probe cannot distinguish from the base model. A healthy training loop is a closed circuit: corpus → SFT → on-policy DPO → an eval gate → promotion only on a measured improvement → the promoted adapter **served** on the inference path → its behaviour captured back into the corpus. Several stages are real and confirmed: adapters attach to all linear projections of the base model and the attachment is asserted post-build (a fail-closed check on the exact target-module list, in both the SFT and DPO training scripts); learning rates sit an order of magnitude above full fine-tuning (2e-4 vs. a stated 2e-5 full-FT default); preference training runs only above a clean, diverse pair floor (a real `CLEAN_PAIR_FLOOR = 3000` constant, plus per-pair quality gates); and no adapter is promoted that a base-versus-adapter probe cannot distinguish from the base model (a real deploy-gate script implementing exactly this check). **One claim corrected, not just softened**: the eval gate does not compare the new adapter "to the incumbent on a frozen, version-hashed gold set." No incumbent-comparison or version-hashed gold set exists anywhere in `service-slm`. The real gate (`score-gate.sh`) scores an adapter's own completions against a static pass-rate threshold — diff-parse / git-apply / envelope-format correctness — over a randomly-shuffled 10% holdout, freshly split each run, not a comparison against a prior adapter's performance. ## How the ontology is intended to be coherent A coherent DataGraph resolves entities through four stages — blocking, similarity, clustering, and canonicalisation — backed by an alias table so that surface variants ("MCorp", "Woodfine Capital Projects") collapse to a single canonical identity. Facts carry provenance (source document, extractor tier, confidence, timestamp); conflicting facts are reconciled rather than blindly overwritten; relationships are typed edges from a closed, directional ontology; and history is retained so any fact can be read "as of" a point in time. A coherent DataGraph is intended to resolve entities through blocking, similarity, and canonicalisation stages, so that surface variants ("MCorp", "Woodfine Capital Projects") collapse to a single canonical identity. **What's actually built today is narrower than that, and narrower than earlier versions of this article claimed**: real entity resolution (`service-content/src/er.rs`) implements three stages, not four — blocking → similarity → decision bands (auto-merge / review / new) — with no clustering stage. The alias table that would back canonicalization is explicitly not yet implemented; the module's own comment describes it as "an additive migration applied separately." Entity resolution today is pure and side-effect-free, not yet backed by the alias mechanism this section describes. Facts carry partial provenance: a real `confidence` field and a real `source_doc` field exist on every graph entity, with `source_doc` first-write-wins — but there is no `extractor_tier` field, so provenance does not yet capture which tier (GLiNER, OLMo, or GPU) produced a given fact. Conflict handling is mixed, not uniformly "reconciled rather than blindly overwritten": vector fields use a new-wins-if-present merge and `source_doc` is first-write-wins, but `confidence` is unconditionally overwritten on every write — the one field most directly meant to signal trust is the one field that isn't reconciled at all. Relationships genuinely are typed, directional edges from a closed ontology (a real relation-type vocabulary loaded from a CSV file) — this part is accurate as described. **"History is retained so any fact can be read 'as of' a point in time" is not built** — no temporal or bitemporal versioning code exists anywhere in `service-content/src` today, despite being part of the file layout this article's own "Target state" section below describes as a future target. Stating it as already true here, one section above where the same capability is correctly described as planned, was an internal contradiction — corrected by moving the claim to where it belongs, in Target state. ## Target state (planned)