From 9f7874211123b9e21a361688b1b9d52f05721fa0 Mon Sep 17 00:00:00 2001 From: Simone Date: Wed, 22 Jul 2026 22:15:32 +0200 Subject: [PATCH] docs(cache): end-to-end numbers, operator runbook, and a vmrun gotcha Measured through the real pipeline, not the unit tests: a Rust job goes from 186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore. The test that decides whether this ships is row 4: a deliberate type error in fury-core, built warm. The cache restored 645 MB of dependencies and the build still went red with E0308, exit 1, publishing nothing. A cache that can turn a red build green is the worst failure a CI system has because it is silent -- re-run that row whenever the prune rules change. AGENTS.md #17 records something found while validating: `vmrun list` is per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone is very much alive, which makes the "no VMs running" check before a deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and was already the recommended diagnosis. --- AGENTS.md | 2 + TODO.md | 24 ++++++++-- docs/RUNBOOK-PhaseC.md | 47 +++++++++++++++++++ plans/cargo-dependency-cache-measurements.md | 48 ++++++++++++++++++++ 4 files changed, 117 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 41c5e7b..767163c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -283,3 +283,5 @@ Configurazione in `PSScriptAnalyzerSettings.psd1`. Regole principali: 15. **Step 7 in timeout: distinguere tre cause diverse che danno lo stesso messaggio** — `getGuestIPAddress` risponde `The VMware Tools are not running` sia quando il guest non ha IP, sia quando ha l'IP ma i Tools non rispondono. Ordine di verifica, dal più veloce: (a) **collisione** — `ps | grep vmware-vmx`, processo più vecchio del deploy corrente → vedi #14; (b) **DHCP host incastrato** — `ip -4 addr show vmnet8` deve dare `192.168.79.1/24` e `ps -ef | grep vmnet-dhcpd` deve mostrarlo vivo; se no, `sudo systemctl restart vmware-networks` (o `vmware-host.service`); (c) **guest senza lease o Tools non pronti** — cercare il MAC della VMX in `/etc/vmware/vmnet8/dhcpd/dhcpd.leases` e in `ip neigh show dev vmnet8`: se un lease c'è, il guest è vivo e mancano solo i Tools, quindi sondare TCP/22 direttamente invece di aspettare `getGuestIPAddress`. 16. **Delta di snapshot orfani dopo `--force`** — `deploy-*` cancella `.vmsd` (il database snapshot) per evitare l'errore di resize sulla catena, ma **non** cancella i file delta `-00000N.vmdk`. Restano su disco con un `parentCID` che non corrisponde più al base ricreato: innocui (fuori dal percorso di boot, la VMX punta al base con `parentCID=ffffffff`) ma confondono la diagnosi e vanno rimossi a mano. Verifica della catena: `head -c 2000 .vmdk | strings | grep -E '^CID|parentCID'`. + +17. **`vmrun list` è per-utente e mente se lo lanci con l'utente sbagliato** — le VM della CI le avvia `ci-runner`; `vmrun -T ws list` eseguito come `simone` (o come root) risponde `Total running VMs: 0` **anche mentre un clone sta girando**, perché vede solo l'istanza VMware dell'utente corrente. Occorso il 2026-07-22 durante la validazione della cache cargo: `vmrun list` diceva 0 e `ps` mostrava un `vmware-vmx` vivo da 19 minuti. Conseguenza pratica: il check "0 VM accese" prima di un `deploy --force` (vedi #14) **non è affidabile** se non lo si esegue come `ci-runner`. Usare `sudo -u ci-runner vmrun -T ws list`, oppure — meglio, perché indipendente dall'utente — `ps -eo pid,lstart,etime,args | grep '[v]mware-vmx'`, che è anche la diagnosi già raccomandata in #14 e #15. diff --git a/TODO.md b/TODO.md index bdac67d..2d68587 100644 --- a/TODO.md +++ b/TODO.md @@ -35,7 +35,7 @@ _Last updated: 2026-05-11 (post Sprint 15: §7.5 DONE — Rinomina Setup-* → I | [x] | Gitea Workflow Integration | 5 | 0 | tutti completati — example.yml usa composite action | | [x] | §1 Sicurezza & hardening | 3 | 0 | 1.1, 1.4, 1.6 done; 1.2/1.3/1.5/1.7/1.8 deferred (home lab) | | [x] | §2 Affidabilità & resilienza | 7 | 0 | tutto done (2.1-2.7) | -| [ ] | §3 Performance & throughput | 4 | 1 | 3.1/3.2/3.3/3.6 done; 3.4/3.5 deferred; **3.8 cache cargo pianificata** | +| [x] | §3 Performance & throughput | 5 | 0 | 3.1/3.2/3.3/3.6/3.8 done; 3.4/3.5 deferred — **§3 CHIUSO** | | [x] | §4 Osservabilità & manutenzione | 3 | 0 | 4.1/4.3/4.4 done (4.2 Prometheus, 4.5 Dashboard rimossi) | | [x] | §5 Qualità codice & test | 5 | 0 | 5.1/5.2/5.3/5.4/5.5 done | | [x] | §6 Scalabilità & estensioni | 5 | 0 | 6.1-6.2 done, 6.3 deferred (home lab), 6.4-6.5 done, 6.6 done | @@ -938,8 +938,19 @@ cioè nelle invocazioni manuali/burn-in. `Start-Job`/`pwsh`/`$IsWindows`. Lo shim resta solo per il runner Windows, dove `CI_VENV_PYTHON` è sempre settato. Il bug non è più raggiungibile sull'host Linux. -### 3.8 [P2] [ ] Cache delle dipendenze cargo per i job Rust — PIANIFICATO (2026-07-22) -File: [plans/cargo-dependency-cache.md](plans/cargo-dependency-cache.md). +### 3.8 [P2] [x] Cache delle dipendenze cargo per i job Rust — COMPLETATO 2026-07-22 +File: [src/ci_orchestrator/cargo_cache.py](src/ci_orchestrator/cargo_cache.py), +[src/ci_orchestrator/assets/cargo-cache-prune.sh](src/ci_orchestrator/assets/cargo-cache-prune.sh), +piano in [plans/cargo-dependency-cache.md](plans/cargo-dependency-cache.md), +numeri in [plans/cargo-dependency-cache-measurements.md](plans/cargo-dependency-cache-measurements.md). + +**Risultato misurato end-to-end**: job Rust da **186 s → 84 s** (fase build +150 s → 47 s). Con sorgente modificato — lo scenario CI vero — 75 s. Entry da +645 MB, restore 8 s. `--no-cargo-cache` e i tag `v*` restano a freddo. + +**La prova che conta**: errore di compilazione deliberato in `fury-core`, +run a caldo → `error[E0308]`, job exit 1. La cache non può nascondere una +regressione. Da ripetere ogni volta che cambiano le regole di potatura. Misurato in furyrs#12: dei ~133 s di un job Rust, ~120 s sono la compilazione delle ~490 dipendenze esterne, che `Cargo.lock` committato tiene **bit-identiche @@ -975,7 +986,12 @@ concorrenti. cambio di `Cargo.lock`, centinaia di MB l'una) senza estendere `retention run` ripeterebbe l'errore di §2.9 su scala maggiore. -**Prossimo passo**: Task 0 del piano — misurare prima di costruire, come in #12. +**Nota emersa implementando**: l'asserzione prevista dal piano era tautologica +— ri-cercava gli stessi glob che aveva appena cancellato, quindi provava solo +che `rm` fosse andato a buon fine, non che i pattern coprissero la realtà. +Sostituita con un oracolo indipendente: i depfile `.d` che cargo scrive dicono +da quali sorgenti ogni artefatto è stato compilato. Dettaglio nel documento +delle misure. ### 8.1 [P3] [ ] Mirror GitHub Actions su Gitea locale (rimuovi dipendenza github.com) File: [.gitea/workflows/lint.yml](.gitea/workflows/lint.yml), Gitea (org `actions`). diff --git a/docs/RUNBOOK-PhaseC.md b/docs/RUNBOOK-PhaseC.md index e6f21eb..b841a33 100644 --- a/docs/RUNBOOK-PhaseC.md +++ b/docs/RUNBOOK-PhaseC.md @@ -608,6 +608,53 @@ ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb See [§4.6](#46-sweep-orphans) (`ci vm cleanup`). +### Cargo dependency cache + +Rust jobs on `LinuxRust2404` reuse compiled **dependency** artifacts between +runs. Measured on furyrs: a warm job pays ~8s to skip ~120s of dependency +compilation on a 134s job. Workspace crates are never cached, so a restored +entry cannot carry the code under test. + +Store: `/var/lib/ci/cache/cargo`, one immutable file per key +(`v1-.tar.zst`, ~645 MB each). A key is +`sha256(Cargo.lock) + rustc version + host triple`. + +```bash +ls -lh /var/lib/ci/cache/cargo/ # what is stored +du -sh /var/lib/ci/cache/cargo/ # total +``` + +What the job log tells you: + +| Line | Meaning | +|---|---| +| `[cache] hit for v1-… (645 MB) — restoring` | Entry reused; dependencies will not recompile | +| `[cache] miss for v1-… — building cold` | First run for this `Cargo.lock`+toolchain; the job publishes at the end | +| `[cache] prune: ok — 6 workspace member(s) stripped, …` | Prune passed its assertion; publishing | +| `[cache] prune refused — publishing nothing` | **The assertion tripped.** An artifact still referenced workspace source. Nothing was stored; the build result itself is unaffected. Investigate before trusting the feature | +| `[cache] restore skipped: …` / `publish skipped: …` | Transport or disk problem. The build ran cold and is still valid | + +Turning it off, cheapest first: + +1. One workflow: `cargo-cache: 'off'` in the composite action inputs. +2. This host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`. +3. One job: `ci job --no-cargo-cache`. +4. Everything, without deleting a file: bump `CACHE_FORMAT` in + `src/ci_orchestrator/cargo_cache.py`. Every key changes, so every existing + entry becomes unreachable. Use this when the prune rules change and older + entries can no longer be trusted. + +Deleting entries by hand is always safe — they are immutable and independent, +and each one costs exactly one cold build to regenerate: + +```bash +sudo -u ci-runner rm /var/lib/ci/cache/cargo/*.tar.zst +``` + +`ci retention run` prunes entries older than the tighter of `--retention-days` +and `[cargo_cache].retention_days`, so the low-disk aggressive window reaches +them too. + ### Publishing `.gitea/` — committed is not deployed act_runner never executes this working copy. For every job it re-clones the diff --git a/plans/cargo-dependency-cache-measurements.md b/plans/cargo-dependency-cache-measurements.md index 836c3c4..857f4dd 100644 --- a/plans/cargo-dependency-cache-measurements.md +++ b/plans/cargo-dependency-cache-measurements.md @@ -56,6 +56,54 @@ The publish path costs 15.7s but runs only on a miss, i.e. once per Transfer is not the bottleneck anyone feared: the virtual NIC moves the entry at 140–180 MiB/s, so 645 MiB crosses in under 4 seconds each way. +## End-to-end, through `ci job` (2026-07-22) + +Full pipeline against `LinuxRust2404`, furyrs `b8ea567`, `--skip-artifact`. +"Build" is the build phase alone; "total" includes clone, boot (~35s) and +teardown. + +| # | Scenario | Build | Total | Cache line in the log | +|---|---|---:|---:|---| +| 1 | Cold, empty store | 150s | 186s | `miss` → `prune: ok` → `published … (645 MB)` | +| 2 | Warm, same commit | **47s** | **84s** | `hit … — restoring` | +| 3 | Warm, source modified | **38s** | **75s** | `hit … — restoring` | +| 4 | Warm, deliberate compile error | — | — | `hit`, then **build fails, exit 1** | +| 5 | `--no-cargo-cache` | 133s | 170s | none — cache never touched | + +**Saving: 103s on the build phase (150 → 47), 102s on the total (186 → 84).** +Row 3 is the real CI scenario and is the fastest of the warm runs, because the +touched crate is small. + +Row 2's log names every workspace crate being rebuilt — + +``` +Compiling fury-core, fury-gui, fury-ipc, fury-backend, furyd, fury-cli +``` + +— and no dependency. That is the design working: dependencies restored, +workspace recompiled from source, every time. + +### Row 4 is the one that decides whether this ships + +A deliberate type error was pushed to a scratch branch and built warm: + +``` +[cache] hit for v1-1311… (645 MB) — restoring +error[E0308]: mismatched types + | --- ^^^^^^^^^^^^^^^^^^^ expected `u32`, found `&str` +error: could not compile `fury-core` (lib) due to 1 previous error +JOB_EXIT=1 +``` + +The cache restored 645 MB of dependencies and the regression still went red. A +cache that can turn a red build green is the worst failure a CI system has, +because it is silent; this is the test that rules it out, and it is worth +re-running whenever the prune rules change. + +The failed job also published nothing — the store still held exactly one entry +and `.incoming/` was empty — because publishing sits after the build-failure +check. A broken tree cannot become an entry. + ## Notes that changed the implementation **`zstd` is already in the template.** `/usr/bin/zstd` is present on