docs(cache): end-to-end numbers, operator runbook, and a vmrun gotcha

Measured through the real pipeline, not the unit tests: a Rust job goes from
186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which
is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore.

The test that decides whether this ships is row 4: a deliberate type error in
fury-core, built warm. The cache restored 645 MB of dependencies and the build
still went red with E0308, exit 1, publishing nothing. A cache that can turn a
red build green is the worst failure a CI system has because it is silent --
re-run that row whenever the prune rules change.

AGENTS.md #17 records something found while validating: `vmrun list` is
per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone
is very much alive, which makes the "no VMs running" check before a
deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and
was already the recommended diagnosis.
This commit is contained in:
2026-07-22 22:15:32 +02:00
parent 15b1676375
commit 9f78742111
4 changed files with 117 additions and 4 deletions
+47
View File
@@ -608,6 +608,53 @@ ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb
See [§4.6](#46-sweep-orphans) (`ci vm cleanup`).
### Cargo dependency cache
Rust jobs on `LinuxRust2404` reuse compiled **dependency** artifacts between
runs. Measured on furyrs: a warm job pays ~8s to skip ~120s of dependency
compilation on a 134s job. Workspace crates are never cached, so a restored
entry cannot carry the code under test.
Store: `/var/lib/ci/cache/cargo`, one immutable file per key
(`v1-<sha256>.tar.zst`, ~645 MB each). A key is
`sha256(Cargo.lock) + rustc version + host triple`.
```bash
ls -lh /var/lib/ci/cache/cargo/ # what is stored
du -sh /var/lib/ci/cache/cargo/ # total
```
What the job log tells you:
| Line | Meaning |
|---|---|
| `[cache] hit for v1-… (645 MB) — restoring` | Entry reused; dependencies will not recompile |
| `[cache] miss for v1-… — building cold` | First run for this `Cargo.lock`+toolchain; the job publishes at the end |
| `[cache] prune: ok — 6 workspace member(s) stripped, …` | Prune passed its assertion; publishing |
| `[cache] prune refused — publishing nothing` | **The assertion tripped.** An artifact still referenced workspace source. Nothing was stored; the build result itself is unaffected. Investigate before trusting the feature |
| `[cache] restore skipped: …` / `publish skipped: …` | Transport or disk problem. The build ran cold and is still valid |
Turning it off, cheapest first:
1. One workflow: `cargo-cache: 'off'` in the composite action inputs.
2. This host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`.
3. One job: `ci job --no-cargo-cache`.
4. Everything, without deleting a file: bump `CACHE_FORMAT` in
`src/ci_orchestrator/cargo_cache.py`. Every key changes, so every existing
entry becomes unreachable. Use this when the prune rules change and older
entries can no longer be trusted.
Deleting entries by hand is always safe — they are immutable and independent,
and each one costs exactly one cold build to regenerate:
```bash
sudo -u ci-runner rm /var/lib/ci/cache/cargo/*.tar.zst
```
`ci retention run` prunes entries older than the tighter of `--retention-days`
and `[cargo_cache].retention_days`, so the low-disk aggressive window reaches
them too.
### Publishing `.gitea/` — committed is not deployed
act_runner never executes this working copy. For every job it re-clones the