SLM Rust stack architecture

@@ -10,115 +10,67 @@ index_group: the-doorman-boundary
short_description: "The full Rust dependency graph and binary architecture for service-slm, the Doorman service that mediates every inference call in the PointSav platform."
status: active
bcsc_class: public-disclosure-safe
last_edited: 2026-05-01
last_edited: 2026-08-17
editor: pointsav-engineering
cites: []
paired_with: slm-stack-architecture.es.md

---

**Correction (2026-08-02):** most of the specific stack described below does not match the real `service-slm` codebase. **Inference**: the real runtime is **llama-server (llama.cpp)** at Tier A and **vLLM** at Tier B — both external, non-Rust binaries called over HTTP (`service-slm/ARCHITECTURE.md:225-233`); `mistral.rs` and `candle` are not deployed anywhere (candle is listed only as a hypothetical future path). **Storage**: the real crate uses `rusqlite`, not `sqlx` (zero `sqlx` hits in `Cargo.lock`). **Document processing / orchestration / observability**: `oxidize-pdf`, `docx-rust`, `calamine`, `pulldown-cmark`, `apalis`, `object_store`, `opentelemetry-rust`, `sigstore-rs`, and `mupdf-rs` — none appear anywhere in the workspace's 286-crate `Cargo.lock`. **Binary/crate layout**: the "Flat binary architecture" section below invents `slm-ledger`, `slm-compute`, `slm-memory-kv`, `slm-memory-adapters`, `slm-inference-local`, `slm-inference-remote`, `slm-api`, `slm-cli` as a single-binary design; the real workspace has five crates (`slm-core`, `slm-doorman`, `slm-doorman-server`, `adapter-hub`, `slm-mcp-server`) with two actual `[[bin]]` targets, not one. What's confirmed accurate: the axum/tower/tokio/reqwest/tracing base, the `cargo-deny`/`deny.toml` license-policy mechanism, and LadybugDB as the knowledge-graph store. **Flagged, not resolved** — this needs a rewrite of the "canonical stack" and "flat binary architecture" sections against the real dependency graph, not a wording fix.
**What changed in this rewrite.** A 2026-08-02 correction note on this article confirmed most of the specific stack described in earlier versions did not match the real `service-slm` codebase, and asked for a rewrite of the "canonical stack" and "flat binary architecture" sections against the real dependency graph rather than a wording fix. This is that rewrite — re-verified against source again, not just carrying the 2026-08-02 note forward, and it found further problems the note hadn't caught: an "L1/L2/L3" framework unrelated to the platform's real one, a third external service with no support in the codebase, and a self-contradiction about which inference engine is actually deployed.

[[service-slm|`service-slm`]] is built as a single, statically-linked Rust binary. Every direct dependency in the stack is either pure Rust or Rust bindings to a permissively licensed native library. No copyleft licenses appear anywhere in the dependency graph, which means [[pointsav-overview|PointSav]] holds an unrestricted right to fork, modify, and redistribute the entire codebase. This property is called the "We Own It" criterion.
[[service-slm|`service-slm`]] ships as a Rust cargo workspace of five real crates (`slm-core`, `slm-doorman`, `slm-doorman-server`, `adapter-hub`, `slm-mcp-server`) with two real `[[bin]]` targets (`slm-doorman-server`, `slm-mcp-server`) — not the single statically-linked binary earlier text claimed; no static-link (musl or similar) target configuration exists in the toolchain config. The workspace's own `ARCHITECTURE.md` calls this a "flat architecture" per binary, not one binary for the entire system.

The choice of Rust is not a language preference. It is an engineering constraint imposed by the intended deployment target — [[totebox-os|ToteboxOS]] appliance hardware, where a CPython interpreter plus a large ML framework does not fit in the available memory envelope, and where cold-start predictability and the absence of a garbage collector are operational requirements rather than optional improvements.

## Why L2 Rust, not L3 Rust
## The real "We Own It" framework — not the Rust-dependency taxonomy earlier text invented

Three distinct levels of "Rust-ness" are commonly conflated:

| Level | Meaning | Achievable for service-slm? |
|---|---|---|
| L1 — Source Rust | All code written by PointSav is Rust | Yes |
| L2 — Direct-deps Rust | Every crate directly depended on is a Rust crate (may internally FFI to C/C++) | Yes |
| L3 — Transitive Rust | Every line in the entire dependency tree, including GPU kernels, is Rust | No — and not the right goal |

L3 is unachievable for GPU inference and graph databases in 2026 because CUDA kernels and columnar storage engines have a twenty-year C++ inheritance. L2 is achievable and sufficient. The "We Own It" test is a license question, not a language question: `MIT + Apache-2.0 = we own it`.
Earlier versions of this article described a three-level "L1/L2/L3 Rust-ness" table grading how much of the dependency tree is Rust versus FFI, and called that the "We Own It" test. **That table does not correspond to any real framework in this codebase.** The actual "We Own It" concept, documented in `substrate/llm-substrate-decision.md` and `service-slm/docs/yoyo-training-substrate-and-service-content-integration.md`, grades **LLM openness**, not dependency-graph composition: L1 is open weights, L2 adds a permissive license, L3 requires the entire lineage — weights, training data, and code — to be openly licensed. OLMo 3 is cited in the real documentation as satisfying L3 under *this* framework, which is a claim about the model, not about how much of `service-slm`'s own Cargo dependency tree is written in Rust. The license-hygiene property earlier text was reaching for (no copyleft anywhere in the dependency graph, so PointSav holds an unrestricted right to fork, modify, and redistribute) is real and enforced by `cargo-deny` — it just isn't what "We Own It" means in this codebase's own vocabulary, and conflating the two invents a framework that doesn't exist.

## The canonical stack

### Inference layer

The inference runtime for service-slm is **mistral.rs**, a statically-linked Rust binary that ships with FlashAttention V2/V3, PagedAttention, prefix caching, and LoRA hot-swap per token. It exposes an OpenAI-compatible HTTP endpoint, which is the wire protocol the Doorman uses for Tier A (local) and Tier B (GPU burst) calls.

The foundation ML framework underneath mistral.rs is **candle** (Apache-2.0/MIT dual license, Hugging Face). If mistral.rs ever diverges from platform requirements, candle provides a clean rebuild path without re-architecting the stack.
The inference runtime is **not** a Rust binary. Tier A runs **llama-server (llama.cpp)** and Tier B runs **vLLM** — both external, non-Rust processes called over HTTP (`service-slm/ARCHITECTURE.md`). `mistral.rs` and `candle` are not deployed anywhere in the current stack; `candle` appears only as a hypothetical future path in documentation, not as production infrastructure. Earlier text's claim that vLLM was "the Phase 1 trial inference engine, replaced by mistral.rs in Phase 2" is self-contradictory with the rest of this article's own correction history — vLLM is the actual, current Tier B runtime, with no replacement planned or underway.

The OLMo 3 model family is the production base model selection. OLMo 3 carries an Apache 2.0 code license and an Open Data Commons license for training data, making it the only major open-weight family whose entire lineage — weights, training data, and code — is permissively licensed end-to-end. This is the requirement for the [[apprenticeship-substrate]] training path, where PointSav exercises the right to run continued pretraining on customer-accumulated signal.

### HTTP and async runtime

The [[doorman-protocol|Doorman]]'s inbound HTTP surface is served by **axum** (MIT), with **tower** middleware for retries, timeouts, and backpressure, running on the **tokio** async runtime (MIT). Outbound HTTP calls — to Cloud Run GPU instances ([[yoyo-compute-substrate|Yo-Yo]]), to the Tier C external API allowlist — use **hyper** and **reqwest**.
The [[doorman-protocol|Doorman]]'s inbound HTTP surface is served by **axum** (MIT), with **tower** middleware for retries, timeouts, and backpressure, running on the **tokio** async runtime (MIT). Outbound HTTP calls use **reqwest**. These dependencies are confirmed in the real `Cargo.toml`.

### Storage and state

The audit ledger uses **sqlx** with an SQLite backend for local, append-only structured storage. The long-term knowledge graph (held by [[service-content]]) is held by LadybugDB, addressed through Rust bindings (MIT). Cloud object storage for model weights and LoRA adapter artefacts is abstracted through **object_store** (Apache-2.0).

### Document processing

The document ingest path uses **oxidize-pdf** for PDF parsing (pure Rust, zero C dependencies, 99.3% success rate across real-world PDFs at 3,000–4,000 pages per second), **docx-rust** for `.docx` files, **calamine** for `.xlsx` spreadsheets, and **pulldown-cmark** for Markdown.

**mupdf-rs is explicitly excluded** from the dependency graph. It carries an AGPL-3.0 license, which would taint the binary if linked. The `cargo-deny` CI policy enforces this exclusion automatically on every commit.

### Orchestration

Internal job orchestration uses **apalis** (MIT) — a job-processing library with step-based workflow composition and tower middleware compatibility. apalis fits the service-slm work shape: sanitise, send, await, receive, rehydrate. It introduces no Python runtime dependency, which is the meaningful distinction from Python-native workflow engines used in the [[service-content|`service-content`]] derivative pipeline.
The audit ledger uses **rusqlite** with an SQLite backend, not `sqlx` as earlier text claimed — `Cargo.lock` has zero `sqlx` matches across all 286 packages in the workspace. The long-term knowledge graph (held by [[service-content]]) is LadybugDB. No `object_store` dependency was found for model-weight/adapter cloud storage; earlier text's claim there is unconfirmed.

### Observability and supply-chain security
### Document processing, orchestration, and observability — not real dependencies

Structured logging and distributed tracing are provided by the **tracing** crate family (MIT), with **opentelemetry-rust** for OpenTelemetry export to the SOC 3 audit pipeline. Container image signing and OCI artefact attestation use **sigstore-rs** (Apache-2.0).
Earlier text named a substantial document-processing and orchestration stack — `oxidize-pdf`, `docx-rust`, `calamine`, `pulldown-cmark` for document ingest; `apalis` for job orchestration; `opentelemetry-rust` for tracing export; `sigstore-rs` for artifact signing; `mupdf-rs` as an explicitly-excluded AGPL dependency. **None of these appear anywhere in the workspace's `Cargo.lock`.** This entire section of earlier text was invented, not merely imprecise — there is no real evidence any of this document-processing/orchestration/observability stack exists in `service-slm` today.

**cargo-deny** runs in CI on every commit and enforces license policy across the full transitive dependency tree, blocking any new dependency that introduces AGPL, GPL, LGPL, BSL, or custom community licenses. This turns manual license discipline into an automated build constraint.
### License hygiene — confirmed real, with minor corrections

## Flat binary architecture

The cargo workspace produces one binary: `slm-cli`. Logical modules communicate via Rust function calls, not RPC. External calls — to Cloud Run, to the Mooncake KV cache sidecar, to Tier C API endpoints, to LadybugDB — are the only network boundaries.

```
service-slm/
├── crates/
│ ├── slm-core/ shared types, moduleId discipline
│ ├── slm-doorman/ sanitise / send / receive / rehydrate
│ ├── slm-ledger/ append-only audit trail (SQLite + CSV)
│ ├── slm-compute/ Cloud Run driver, container management
│ ├── slm-memory-kv/ LMCache + Mooncake wire protocol client
│ ├── slm-memory-adapters/ LoRA adapter registry and loader
│ ├── slm-inference-local/ mistral.rs-backed local inference
│ ├── slm-inference-remote/ GPU burst driver
│ ├── slm-api/ axum inbound endpoints
│ └── slm-cli/ binary entry point
└── xtask/ build helpers, release automation
```

This is the shape a ToteboxOS appliance component requires: one process, one log stream, one set of metrics, one binary to sign with Sigstore, one configuration file.
`cargo-deny` genuinely runs in CI with a real `deny.toml` policy file, confirmed by direct read. The allowed-license list is largely as earlier text described (`MIT`, `Apache-2.0`, `BSD-2-Clause`, `BSD-3-Clause`, `ISC`, `MPL-2.0` file-level, `Zlib`), with two corrections: the real file also allows `Apache-2.0 WITH LLVM-exception` and `CC0-1.0`, both omitted from earlier text; and the Unicode license entries are `Unicode-DFS-2016`/`Unicode-3.0` in the real file, not the bare `Unicode-DFS` earlier text used.

## ToteboxOS integration

The binary architecture is motivated in part by ToteboxOS deployment constraints. A CPython stack plus a GPU inference framework does not fit in the memory envelope available on constrained appliance hardware. A Rust binary with a quantised inference runtime operating in CPU mode does.

The relevant constraints per ToteboxOS Laptop-A hardware profile (~550 MB available headroom after core services):
**The binding constraint on Laptop-A hosts is the 4 GB RAM envelope** — not the "~550 MB available headroom" figure earlier text cited, which has no source anywhere in the codebase; the real `ARCHITECTURE.md` states the 4 GB figure directly.

- Static binary, no interpreter warmup — seconds, not minutes to first inference
- Static binary per `[[bin]]` target, no interpreter warmup — seconds, not minutes to first inference
- No garbage collector, no interpreter heap
- True parallelism across cores without a global interpreter lock
- Cross-compilation via `cargo build --target aarch64-unknown-linux-gnu` for ARM ToteboxOS targets

## Three external non-Rust services
## Two external non-Rust services — not three

Three services in the Yo-Yo compute substrate sit outside the Rust binary, all behind stable network protocols:
Earlier versions of this article listed three external non-Rust services in the Yo-Yo compute substrate, including "SkyPilot" for multi-cloud GPU orchestration. **SkyPilot has zero references anywhere in the monorepo** and is dropped here rather than carried forward unverified. The two real ones:

**LMCache + Mooncake Store** (Python control plane + C++ Mooncake Transfer Engine): the KV cache tier that persists prefill state across GPU node teardowns. service-slm holds a Rust client that speaks to Mooncake over HTTP and TCP. No FFI coupling. Both are Apache-2.0 licensed.

**vLLM** (Python): the Phase 1 trial inference engine. Replaced by mistral.rs in Phase 2. Apache-2.0.

**SkyPilot** (Python): multi-cloud GPU orchestration. Used when Cloud Run GPU alone is insufficient. Apache-2.0.

All three are behind stable network protocols. service-slm depends on the wire protocol, not the implementation. Swapping any of them requires changing one client module.

## License hygiene

The mandatory rule for the entire dependency graph: every entry is one of `MIT`, `Apache-2.0`, `BSD-2-Clause`, `BSD-3-Clause`, `ISC`, `Unicode-DFS`, `MPL-2.0` (file-level), or `Zlib`. Anything else fails the CI build.
**vLLM** (Python): the real, current Tier B inference engine (see Inference layer, above) — not a superseded trial, as earlier text claimed. Apache-2.0.

`cargo-deny` with a `deny.toml` policy file enforces this. The `deny.toml` is committed to the repository and reviewed at every dependency addition.
Both are behind stable network protocols; service-slm depends on the wire protocol, not the implementation. Swapping either requires changing one client module.

## See also