Brief queue substrate

@@ -16,7 +16,7 @@ cites:
paired_with: brief-queue-substrate.es.md
---

The **Brief Queue Substrate** is a durable, file-backed queue that sits between the editorial pipeline and the apprenticeship corpus capture system inside the PointSav platform. Its primary function is to accept brief-execution records — JSON Lines events describing an editorial operation — and hold them reliably until the drain worker can write them to the long-term corpus. This decoupling is what makes Yo-Yo idle-shutdown viable: when the GPU instance shuts down mid-session to avoid idle billing, no in-flight corpus event is lost. The queue itself persists on disk; the drain worker picks up where it left off when the next session begins. The substrate also absorbs spot preemption events from cloud providers, simplifies the capture-edit pipeline by removing inline write dependencies, and positions the platform for the eventual PointSav-LLM continued-pretraining path. It is a deliberate structural choice that keeps the apprenticeship corpus growing continuously regardless of compute-tier availability.
The **Brief Queue Substrate** is a durable, file-backed queue that sits between the editorial pipeline and the [[apprenticeship-substrate|apprenticeship corpus]] capture system inside the PointSav platform. Its primary function is to accept brief-execution records — JSON Lines events describing an editorial operation — and hold them reliably until the drain worker can write them to the long-term corpus. This decoupling is what makes [[yoyo-compute-substrate|Yo-Yo idle-shutdown]] viable: when the GPU instance shuts down mid-session to avoid idle billing, no in-flight corpus event is lost. The queue itself persists on disk; the drain worker picks up where it left off when the next session begins. The substrate also absorbs spot preemption events from cloud providers, simplifies the capture-edit pipeline by removing inline write dependencies, and positions the platform for the eventual PointSav-LLM continued-pretraining path. It is a deliberate structural choice that keeps the apprenticeship corpus growing continuously regardless of compute-tier availability.

## What the queue is

@@ -33,15 +33,15 @@ Each file in the queue is a single JSONL event serialized to disk. File names ar

The Brief Queue Substrate delivers five operational properties that the platform depends on:

**Yo-Yo idle-shutdown tolerance.** The Yo-Yo compute tier (OLMo 3.1 32B Think on GPU burst) shuts down after a configurable idle period to avoid unnecessary billing. Without the queue, any brief-execution event generated during a Yo-Yo session would need to be written to the corpus synchronously before shutdown — coupling the SLM's inference loop to disk I/O and adding latency to every inference call. With the queue, the inference loop writes to `queue/` (a fast local write) and the drain worker handles corpus persistence independently. Shutdown can proceed as soon as the inference completes; no corpus writes are in progress.
**Yo-Yo idle-shutdown tolerance.** The [[yoyo-compute-substrate|Yo-Yo compute tier]] (OLMo 3.1 32B Think on GPU burst) shuts down after a configurable idle period to avoid unnecessary billing. Without the queue, any brief-execution event generated during a Yo-Yo session would need to be written to the corpus synchronously before shutdown — coupling the SLM's inference loop to disk I/O and adding latency to every inference call. With the queue, the inference loop writes to `queue/` (a fast local write) and the drain worker handles corpus persistence independently. Shutdown can proceed as soon as the inference completes; no corpus writes are in progress.

**Spot preemption tolerance.** Cloud spot and preemptible instances can be reclaimed with minimal notice. Events sitting in `queue/` or `queue-in-flight/` survive preemption because the queue directory is on persistent storage, not instance-local ephemeral storage. When a replacement instance starts, the drain worker resumes from the last committed position.

**Capture-edit pipeline simplification.** The previous pattern required the capture pipeline to write directly to corpus JSONL files during the editorial session, which created ordering dependencies between the Doorman's audit-routing logic and the corpus file's append position. The queue removes that dependency: the Doorman writes an event record to `queue/` and returns immediately; the drain worker serializes corpus appends in arrival order without blocking the inference path.

**Customer Tier 1 alignment.** The queue architecture matches the pattern that Ring 1 ingest services (service-fs, [[service-people]], [[service-email]], service-input) already use for durable event capture at the tenant boundary. A customer extending the platform with an additional MCP server can use the same four-directory queue pattern for their ingest events. Consistency across tiers reduces the surface area a contributor needs to understand.
**Customer Tier 1 alignment.** The queue architecture matches the pattern that Ring 1 ingest services ([[service-people]], [[service-email]], [[service-extraction|service-input]]) already use for durable event capture at the [[totebox-archive|tenant]] boundary. A customer extending the platform with an additional MCP server can use the same four-directory queue pattern for their ingest events. Consistency across tiers reduces the surface area a contributor needs to understand.

**Future PointSav-LLM readiness.** The apprenticeship corpus — the accumulation of draft-created, draft-refined, and creative-edited JSONL events — is the intended training signal for a future PointSav-LLM continued-pretraining run [1]. The Brief Queue Substrate ensures that corpus growth is uninterrupted by compute-tier transitions, idle shutdowns, or preemption events. Every editorial session contributes to the corpus regardless of which compute tier handled the inference.
**Future PointSav-LLM readiness.** The [[apprenticeship-substrate|apprenticeship corpus]] — the accumulation of draft-created, draft-refined, and creative-edited JSONL events — is the intended training signal for a future PointSav-LLM continued-pretraining run [1]. The Brief Queue Substrate ensures that corpus growth is uninterrupted by compute-tier transitions, idle shutdowns, or preemption events. Every [[yo-yo-lora-training-pipeline|editorial session]] contributes to the corpus regardless of which compute tier handled the inference.

## Lease mechanics

@@ -59,7 +59,7 @@ The lease pattern does not require a network round-trip, a lock daemon, or a mes

## File-JSONL compared to message brokers

The Brief Queue Substrate stores events as JSONL files on a local filesystem rather than routing them through a hosted message broker. This is an intentional architectural choice grounded in the platform's sovereignty posture and operational constraints, not a limitation to be remedied later.
The Brief Queue Substrate stores events as JSONL files on a local filesystem rather than routing them through a hosted message broker. This is an intentional architectural choice grounded in the platform's [[compounding-substrate|sovereignty posture]] and operational constraints, not a limitation to be remedied later.

Message brokers such as NATS JetStream or Redis Streams offer strong durability guarantees and rich consumer-group semantics. They also introduce a persistent network dependency: the producer cannot write if the broker is unreachable, and the consumer cannot read if the broker has lost its state. For a single-tenant workspace where the queue producer, queue consumer, and corpus storage are all on the same machine or the same cloud-persistent volume, the network round-trip adds latency and a failure mode without adding a capability the platform needs at this scale.