Yo-Yo daily enrichment cycle

@@ -1,161 +1,63 @@
---
schema: foundry-doc-v1
title: "Yo-yo daily enrichment cycle"
title: "Yo-Yo daily enrichment cycle"
slug: yoyo-daily-enrichment-cycle
short_description: "Daily GPU batch window that enriches the DataGraph and accumulates training data — fixed schedule, hard cost cap, and guaranteed VM termination."
short_description: "The nightly two-phase GPU batch window that rebuilds the DataGraph and, once fully enabled, trains adapter weights for the local language model — currently running in DataGraph-only mode."
category: services
index_group: ring-3-ai-gateway
type: topic
content_type: topic
status: stable
bcsc_class: no-disclosure-implication
last_edited: 2026-07-18
bcsc_class: current-fact
last_edited: 2026-08-22
editor: pointsav-engineering
paired_with: yoyo-daily-enrichment-cycle.es.md
---

The Yo-Yo daily enrichment cycle is the automated batch window that runs a GPU-accelerated
inference VM once per day to enrich the [[ontological-datagraph|DataGraph]] and accumulate training data for the
[[pointsav-llm|local language model]]. The cycle runs at a fixed time, enforces a hard cost cap, and
terminates the VM whether the work finishes early or reaches the cap.

## Purpose

The workspace VM runs a 7-billion-parameter language model (OLMo 2 7B) on CPU for
interactive use. This model performs adequately for short prompts but extracts entities
from documents with lower accuracy than a larger GPU-resident model. The daily cycle
addresses this gap by starting a separate GPU VM — the Yo-Yo batch node — that loads
a 32-billion-parameter model and processes a queue of documents that accumulated during
the day.

The products of each cycle are:
- Additional named entities added to the DataGraph (graph store)
- Direct Preference Optimisation (DPO) training pairs written to the enrichment corpus

Each DPO pair records what the 32B model extracted as the preferred output and what the
7B model extracted as the baseline, enabling the 7B model to be fine-tuned toward the
larger model's extraction quality over successive training runs.

## The eight phases

**Major correction (2026-07-18):** this section's phase structure, script name, and the
45-minute hard cap it depends on do not match the live orchestrator. The real script is
`nightly-run.sh` (not `yoyo-daily-cycle.sh` — the same naming drift already flagged on
[[spot-vm-lifecycle-kill-switch]]), and it runs a **two-phase** cycle, not eight: Phase 1
(DataGraph rebuild) and Phase 2 (Training), each with an independently configurable
budget defaulting to `DATAGRAPH_SECONDS=7200` and `TRAINING_SECONDS=7200` — **2 hours
each, roughly 4 hours total**, not the 45-minute cap this article's Budget and cost
section is built around. This matches the already-verified 2-phase structure described
in [[elastic-compute-lora-training-pipeline]] (task-verified earlier this pass) — that
article's Phase 1/Phase 2 framing is the accurate one; this article's eight-phase,
45-minute framing appears to describe an earlier or different design. Zone
(`us-central1-a`, confirmed via `SLM_YOYO_GCP_ZONE` default in the real script) and VM
type/kill-switch path are independently accurate and not in dispute. **The cost figures
below ($0.53/cycle, ~$16/month) are downstream of the wrong 45-minute assumption** — at
the real ~4-hour cycle length and the same $0.71/hour rate, the actual cost would be
roughly 6x higher. **Flagged, not silently rewritten** — the eight named phases below
may map loosely onto sub-steps within the real two-phase script, but that mapping has not
been verified; needs project-totebox confirmation before this section and the cost table
are corrected.

The cycle is a single Bash script (`yoyo-daily-cycle.sh`) that executes eight sequential
phases. The script writes a timestamped log file for each run.

**Phase 1 — VM start.** If the batch VM is not already running, a `gcloud instances start`
command is issued. The VM boots from a persistent disk that retains the model weights and
the inference server configuration from the previous cycle.

**Phase 2 — Inference server health.** The script polls the llama-server health endpoint
(`/health`) at ten-second intervals until it returns `{"status":"ok"}`. Startup consistently
takes approximately 170 seconds from power-on to first healthy response. If the server
does not respond within ten minutes, the cycle aborts and stops the VM.

**Phase 3 — Tier B circuit.** The [[soft-slm-tiered-gateway|local inference gateway]] maintains a circuit breaker for
the Yo-Yo node. The script waits up to two minutes for the circuit to close, confirming
the gateway has registered the VM as reachable. If the circuit does not close, the cycle
continues with a Tier A fallback warning logged.

**Phase 4 — Enrichment drain.** For 40 percent of the cycle budget (18 minutes at the
45-minute cap), the script waits while the gateway processes the pending enrichment queue.
During this window, the [[service-content|content service]] sends document chunks to the Yo-Yo node for
entity extraction and writes DPO pairs to the enrichment corpus. Progress is logged every
60 seconds with entity counts, enrichment pair counts, GPU utilisation, and VRAM usage.

**Phase 5 — Corpus threshold check.** After enrichment, `corpus-threshold.py` runs to
count accumulated training-ready data. If counts exceed the configured threshold, the
script writes dated training marker files to `data/training-pending/`. These markers
are the input to Phase 6.

**Phase 6 — [[yo-yo-lora-training-pipeline|LoRA training]] trigger.** Three gates must all pass for training to run:
training markers must be present, the ML libraries must be installed in the training
virtual environment on the batch VM, and an operator-authored approval tag must exist
for the current date. If all three pass, the script stops the inference server to free
approximately 16 gigabytes of VRAM, then invokes `run-dpo-training.py` over SSH with a
45-percent budget (20 minutes at the 45-minute cap). The `--resume` flag accumulates
daily checkpoints so each run extends the previous day's training rather than starting
from scratch.

**Phase 7 — GCS sync.** If the `SLM_YOYO_WEIGHTS_GCS_BUCKET` environment variable is
set and training markers are present, the enrichment corpus is synchronised to the
configured Cloud Storage bucket. This step is currently disabled pending a future session
that configures the bucket.

**Phase 8 — Hard stop.** The inference server is stopped via SSH, the VM is stopped via
`gcloud instances stop`, and the script waits up to three minutes for the VM to reach
`TERMINATED` status. A summary line records total elapsed time, entity delta, DPO pair
delta, and VM final status.

## Budget and cost

The daily cycle operates under a 45-minute hard cap. The VM is stopped unconditionally
at the end of Phase 8 regardless of whether phases completed normally.

| Item | Value |
|---|---|
| VM type | g2-standard-4 with NVIDIA L4 24 GB |
| Zone | us-central1-a |
| Running cost | approximately $0.71 per hour |
| Cycle cost at 45-minute cap | approximately $0.53 per cycle |
| TERMINATED cost | $0.00 |
| Monthly cost (daily cycles) | approximately $16 per month |

A [[spot-vm-lifecycle-kill-switch|kill switch]] file (`/srv/foundry/data/yoyo-disabled`) suppresses all VM lifecycle
operations immediately. Creating the file prevents Phase 1 from issuing a start command.
Removing the file resumes normal operation on the next scheduled cycle.

An idle monitor timer checks every five minutes whether the VM has been running idle for
more than 30 minutes. If the daily cycle fails to stop the VM, the idle monitor will stop
it as a safety backstop, preventing uncapped cost accumulation.

## DPO pair format

Each enrichment DPO pair is a JSON file written to the feedback directory. The format
is compatible with the TRL DPOTrainer:

```json
{
  "prompt":      "<document chunk text>",
  "chosen":      "[{\"classification\":\"Person\",\"entity_name\":\"...\"}]",
  "rejected":    "[{\"classification\":\"Person\",\"entity_name\":\"...\"}]",
  "source_type": "datagraph-enrichment",
  "worm_id":     "<document identifier>",
  "timestamp":   "<ISO 8601>"
}
```

`chosen` is the 32B model's extraction. `rejected` is the 7B model's extraction. A pair
is only written when both models found at least one entity and the results differ after
normalisation. Pairs where the 7B model found nothing are discarded — they contain no
genuine preference signal.

## Verified test results (2026-06-09)

Three 10-minute test cycles confirmed the pipeline operates correctly end-to-end.

| Cycle | Duration | Entity delta | DPO pairs added | VM final status |
|---|---|---|---|---|
| 1 | 10 min 43 s | +7 | +6 | TERMINATED |
| 2 | 9 min 12 s | +8 | +4 | TERMINATED |
| 3 | 10 min 38 s | +22 | +8 | TERMINATED |

GPU diagnostics in cycle 3: 99% utilisation, 16,151 of 23,034 MB VRAM in use, 73°C.
The Yo-Yo daily enrichment cycle is the nightly batch window on the [[yoyo-compute-substrate|
burst GPU node]] that rebuilds the [[ontological-datagraph|DataGraph]] and, once fully
enabled, trains updated adapter weights for the workspace language model. The cycle runs on
a fixed schedule and always releases the GPU at the end, whether or not both phases complete.

## Two phases, not eight

The cycle is one script running two sequential phases, each with its own
configurable time budget defaulting to two hours — roughly four hours total, not a
forty-five-minute window. The two phases cannot overlap: they need exclusive access to the
same GPU, and the script stops the inference server before the training phase begins.

**Phase 1 — DataGraph rebuild.** The batch VM boots, waits for its inference server to
become healthy, then processes the day's accumulated documents through the Doorman, writing
extracted entities directly to the DataGraph. Full detail: [[service-slm-graph-store-
migration]].

**Phase 2 — Adapter training.** A threshold check counts accumulated training tuples across
two corpus buckets. Once a bucket crosses its clean-pair floor, a training-pending marker
is written and, if configured, the relevant corpus syncs to cloud storage. On the batch VM, a
training script polls for that marker and runs a parameter-efficient fine-tune (QLoRA)
against the base model when one appears.

## Current status: training is not yet active

As of this writing, the training half of the cycle runs in marker-only mode: the threshold
check writes and dispatches the marker, but the training script itself is not yet enabled on
the batch VM's running image — a pending image rebuild is the next step before it goes live.
Every night's cycle today does real DataGraph enrichment; no adapter has yet been produced by
this pipeline running end to end on its own schedule.

## Cost and the hard stop

The VM is stopped unconditionally at the end of the cycle regardless of how far the phases
got, and a kill-switch file can suppress the whole cycle immediately if set. An idle monitor
provides a backstop: if the cycle ever fails to stop the VM itself, the monitor stops it after
a sustained idle period, bounding the worst case. At the real multi-hour cycle length — not
the forty-five-minute figure a stale version of this article assumed — the per-cycle cost is
several times higher than that shorter window would suggest; an exact current figure isn't
republished here since it would need to be re-measured against the real Phase 1/Phase 2
budgets and current cloud pricing, not carried forward from a since-corrected assumption.

## See also

- [[service-slm-graph-store-migration]] — the DataGraph rebuild that is Phase 1 of this cycle
- [[elastic-compute-lora-training-pipeline]] — the fuller two-phase pipeline description, including the training phase's real QLoRA configuration
- [[service-slm]] — the service that orchestrates the pipeline