diff --git a/AGENTS.md b/AGENTS.md index 41c5e7b..767163c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -283,3 +283,5 @@ Configurazione in `PSScriptAnalyzerSettings.psd1`. Regole principali: 15. **Step 7 in timeout: distinguere tre cause diverse che danno lo stesso messaggio** — `getGuestIPAddress` risponde `The VMware Tools are not running` sia quando il guest non ha IP, sia quando ha l'IP ma i Tools non rispondono. Ordine di verifica, dal più veloce: (a) **collisione** — `ps | grep vmware-vmx`, processo più vecchio del deploy corrente → vedi #14; (b) **DHCP host incastrato** — `ip -4 addr show vmnet8` deve dare `192.168.79.1/24` e `ps -ef | grep vmnet-dhcpd` deve mostrarlo vivo; se no, `sudo systemctl restart vmware-networks` (o `vmware-host.service`); (c) **guest senza lease o Tools non pronti** — cercare il MAC della VMX in `/etc/vmware/vmnet8/dhcpd/dhcpd.leases` e in `ip neigh show dev vmnet8`: se un lease c'è, il guest è vivo e mancano solo i Tools, quindi sondare TCP/22 direttamente invece di aspettare `getGuestIPAddress`. 16. **Delta di snapshot orfani dopo `--force`** — `deploy-*` cancella `.vmsd` (il database snapshot) per evitare l'errore di resize sulla catena, ma **non** cancella i file delta `-00000N.vmdk`. Restano su disco con un `parentCID` che non corrisponde più al base ricreato: innocui (fuori dal percorso di boot, la VMX punta al base con `parentCID=ffffffff`) ma confondono la diagnosi e vanno rimossi a mano. Verifica della catena: `head -c 2000 .vmdk | strings | grep -E '^CID|parentCID'`. + +17. **`vmrun list` è per-utente e mente se lo lanci con l'utente sbagliato** — le VM della CI le avvia `ci-runner`; `vmrun -T ws list` eseguito come `simone` (o come root) risponde `Total running VMs: 0` **anche mentre un clone sta girando**, perché vede solo l'istanza VMware dell'utente corrente. Occorso il 2026-07-22 durante la validazione della cache cargo: `vmrun list` diceva 0 e `ps` mostrava un `vmware-vmx` vivo da 19 minuti. Conseguenza pratica: il check "0 VM accese" prima di un `deploy --force` (vedi #14) **non è affidabile** se non lo si esegue come `ci-runner`. Usare `sudo -u ci-runner vmrun -T ws list`, oppure — meglio, perché indipendente dall'utente — `ps -eo pid,lstart,etime,args | grep '[v]mware-vmx'`, che è anche la diagnosi già raccomandata in #14 e #15. diff --git a/TODO.md b/TODO.md index bdac67d..2d68587 100644 --- a/TODO.md +++ b/TODO.md @@ -35,7 +35,7 @@ _Last updated: 2026-05-11 (post Sprint 15: §7.5 DONE — Rinomina Setup-* → I | [x] | Gitea Workflow Integration | 5 | 0 | tutti completati — example.yml usa composite action | | [x] | §1 Sicurezza & hardening | 3 | 0 | 1.1, 1.4, 1.6 done; 1.2/1.3/1.5/1.7/1.8 deferred (home lab) | | [x] | §2 Affidabilità & resilienza | 7 | 0 | tutto done (2.1-2.7) | -| [ ] | §3 Performance & throughput | 4 | 1 | 3.1/3.2/3.3/3.6 done; 3.4/3.5 deferred; **3.8 cache cargo pianificata** | +| [x] | §3 Performance & throughput | 5 | 0 | 3.1/3.2/3.3/3.6/3.8 done; 3.4/3.5 deferred — **§3 CHIUSO** | | [x] | §4 Osservabilità & manutenzione | 3 | 0 | 4.1/4.3/4.4 done (4.2 Prometheus, 4.5 Dashboard rimossi) | | [x] | §5 Qualità codice & test | 5 | 0 | 5.1/5.2/5.3/5.4/5.5 done | | [x] | §6 Scalabilità & estensioni | 5 | 0 | 6.1-6.2 done, 6.3 deferred (home lab), 6.4-6.5 done, 6.6 done | @@ -938,8 +938,19 @@ cioè nelle invocazioni manuali/burn-in. `Start-Job`/`pwsh`/`$IsWindows`. Lo shim resta solo per il runner Windows, dove `CI_VENV_PYTHON` è sempre settato. Il bug non è più raggiungibile sull'host Linux. -### 3.8 [P2] [ ] Cache delle dipendenze cargo per i job Rust — PIANIFICATO (2026-07-22) -File: [plans/cargo-dependency-cache.md](plans/cargo-dependency-cache.md). +### 3.8 [P2] [x] Cache delle dipendenze cargo per i job Rust — COMPLETATO 2026-07-22 +File: [src/ci_orchestrator/cargo_cache.py](src/ci_orchestrator/cargo_cache.py), +[src/ci_orchestrator/assets/cargo-cache-prune.sh](src/ci_orchestrator/assets/cargo-cache-prune.sh), +piano in [plans/cargo-dependency-cache.md](plans/cargo-dependency-cache.md), +numeri in [plans/cargo-dependency-cache-measurements.md](plans/cargo-dependency-cache-measurements.md). + +**Risultato misurato end-to-end**: job Rust da **186 s → 84 s** (fase build +150 s → 47 s). Con sorgente modificato — lo scenario CI vero — 75 s. Entry da +645 MB, restore 8 s. `--no-cargo-cache` e i tag `v*` restano a freddo. + +**La prova che conta**: errore di compilazione deliberato in `fury-core`, +run a caldo → `error[E0308]`, job exit 1. La cache non può nascondere una +regressione. Da ripetere ogni volta che cambiano le regole di potatura. Misurato in furyrs#12: dei ~133 s di un job Rust, ~120 s sono la compilazione delle ~490 dipendenze esterne, che `Cargo.lock` committato tiene **bit-identiche @@ -975,7 +986,12 @@ concorrenti. cambio di `Cargo.lock`, centinaia di MB l'una) senza estendere `retention run` ripeterebbe l'errore di §2.9 su scala maggiore. -**Prossimo passo**: Task 0 del piano — misurare prima di costruire, come in #12. +**Nota emersa implementando**: l'asserzione prevista dal piano era tautologica +— ri-cercava gli stessi glob che aveva appena cancellato, quindi provava solo +che `rm` fosse andato a buon fine, non che i pattern coprissero la realtà. +Sostituita con un oracolo indipendente: i depfile `.d` che cargo scrive dicono +da quali sorgenti ogni artefatto è stato compilato. Dettaglio nel documento +delle misure. ### 8.1 [P3] [ ] Mirror GitHub Actions su Gitea locale (rimuovi dipendenza github.com) File: [.gitea/workflows/lint.yml](.gitea/workflows/lint.yml), Gitea (org `actions`). diff --git a/docs/RUNBOOK-PhaseC.md b/docs/RUNBOOK-PhaseC.md index e6f21eb..b841a33 100644 --- a/docs/RUNBOOK-PhaseC.md +++ b/docs/RUNBOOK-PhaseC.md @@ -608,6 +608,53 @@ ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb See [§4.6](#46-sweep-orphans) (`ci vm cleanup`). +### Cargo dependency cache + +Rust jobs on `LinuxRust2404` reuse compiled **dependency** artifacts between +runs. Measured on furyrs: a warm job pays ~8s to skip ~120s of dependency +compilation on a 134s job. Workspace crates are never cached, so a restored +entry cannot carry the code under test. + +Store: `/var/lib/ci/cache/cargo`, one immutable file per key +(`v1-.tar.zst`, ~645 MB each). A key is +`sha256(Cargo.lock) + rustc version + host triple`. + +```bash +ls -lh /var/lib/ci/cache/cargo/ # what is stored +du -sh /var/lib/ci/cache/cargo/ # total +``` + +What the job log tells you: + +| Line | Meaning | +|---|---| +| `[cache] hit for v1-… (645 MB) — restoring` | Entry reused; dependencies will not recompile | +| `[cache] miss for v1-… — building cold` | First run for this `Cargo.lock`+toolchain; the job publishes at the end | +| `[cache] prune: ok — 6 workspace member(s) stripped, …` | Prune passed its assertion; publishing | +| `[cache] prune refused — publishing nothing` | **The assertion tripped.** An artifact still referenced workspace source. Nothing was stored; the build result itself is unaffected. Investigate before trusting the feature | +| `[cache] restore skipped: …` / `publish skipped: …` | Transport or disk problem. The build ran cold and is still valid | + +Turning it off, cheapest first: + +1. One workflow: `cargo-cache: 'off'` in the composite action inputs. +2. This host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`. +3. One job: `ci job --no-cargo-cache`. +4. Everything, without deleting a file: bump `CACHE_FORMAT` in + `src/ci_orchestrator/cargo_cache.py`. Every key changes, so every existing + entry becomes unreachable. Use this when the prune rules change and older + entries can no longer be trusted. + +Deleting entries by hand is always safe — they are immutable and independent, +and each one costs exactly one cold build to regenerate: + +```bash +sudo -u ci-runner rm /var/lib/ci/cache/cargo/*.tar.zst +``` + +`ci retention run` prunes entries older than the tighter of `--retention-days` +and `[cargo_cache].retention_days`, so the low-disk aggressive window reaches +them too. + ### Publishing `.gitea/` — committed is not deployed act_runner never executes this working copy. For every job it re-clones the diff --git a/plans/cargo-dependency-cache-measurements.md b/plans/cargo-dependency-cache-measurements.md index 836c3c4..857f4dd 100644 --- a/plans/cargo-dependency-cache-measurements.md +++ b/plans/cargo-dependency-cache-measurements.md @@ -56,6 +56,54 @@ The publish path costs 15.7s but runs only on a miss, i.e. once per Transfer is not the bottleneck anyone feared: the virtual NIC moves the entry at 140–180 MiB/s, so 645 MiB crosses in under 4 seconds each way. +## End-to-end, through `ci job` (2026-07-22) + +Full pipeline against `LinuxRust2404`, furyrs `b8ea567`, `--skip-artifact`. +"Build" is the build phase alone; "total" includes clone, boot (~35s) and +teardown. + +| # | Scenario | Build | Total | Cache line in the log | +|---|---|---:|---:|---| +| 1 | Cold, empty store | 150s | 186s | `miss` → `prune: ok` → `published … (645 MB)` | +| 2 | Warm, same commit | **47s** | **84s** | `hit … — restoring` | +| 3 | Warm, source modified | **38s** | **75s** | `hit … — restoring` | +| 4 | Warm, deliberate compile error | — | — | `hit`, then **build fails, exit 1** | +| 5 | `--no-cargo-cache` | 133s | 170s | none — cache never touched | + +**Saving: 103s on the build phase (150 → 47), 102s on the total (186 → 84).** +Row 3 is the real CI scenario and is the fastest of the warm runs, because the +touched crate is small. + +Row 2's log names every workspace crate being rebuilt — + +``` +Compiling fury-core, fury-gui, fury-ipc, fury-backend, furyd, fury-cli +``` + +— and no dependency. That is the design working: dependencies restored, +workspace recompiled from source, every time. + +### Row 4 is the one that decides whether this ships + +A deliberate type error was pushed to a scratch branch and built warm: + +``` +[cache] hit for v1-1311… (645 MB) — restoring +error[E0308]: mismatched types + | --- ^^^^^^^^^^^^^^^^^^^ expected `u32`, found `&str` +error: could not compile `fury-core` (lib) due to 1 previous error +JOB_EXIT=1 +``` + +The cache restored 645 MB of dependencies and the regression still went red. A +cache that can turn a red build green is the worst failure a CI system has, +because it is silent; this is the test that rules it out, and it is worth +re-running whenever the prune rules change. + +The failed job also published nothing — the store still held exactly one entry +and `.incoming/` was empty — because publishing sits after the build-failure +check. A broken tree cannot become an entry. + ## Notes that changed the implementation **`zstd` is already in the template.** `/usr/bin/zstd` is present on