From bd49c4c455bc3e78a4adee687f4bbaaca874f07a Mon Sep 17 00:00:00 2001 From: Simone Date: Wed, 22 Jul 2026 20:32:38 +0200 Subject: [PATCH] docs(cache): plan the cargo dependency cache, and why not cargo-chef Thread entry furyrs#12 measured where a 133s Rust job actually goes: ~120s is compiling the ~490 external dependencies, which a committed Cargo.lock keeps bit-identical between runs. The workspace crates cost 7s. #13 settled the policy -- reuse dependency artifacts, never workspace ones, and keep v* tags cold. The plan departs from #12's cargo-chef shape for a concrete reason: the orchestrator receives build-command as an opaque string (furyrs runs six cargo invocations across three feature sets) and `cargo chef cook` must be handed the same flags as the real build. Rather than guess them, invert the order -- build normally, then prune the workspace members' artifacts out of target/ and publish the remainder. No knowledge of the command, one entry holding every feature graph, no new tool in the template. That makes the safety property checkable rather than argued: after the prune, an assertion looks for any surviving workspace fingerprint and the job refuses to publish if it finds one. A missing entry costs 120s; a poisoned one costs a wrong verdict, which is the worst failure a build system has because it is silent. Store is one immutable file per key with an atomic rename, so there is no shared mutable directory and no advisory locking -- the objection furyrs raised about VMware Shared Folders under concurrency. Retention of the store is in the same plan: adding an unbounded directory without it would repeat TODO 2.9 at a larger scale. --- TODO.md | 41 +- plans/cargo-dependency-cache.md | 1375 +++++++++++++++++++++++++++++++ 2 files changed, 1415 insertions(+), 1 deletion(-) create mode 100644 plans/cargo-dependency-cache.md diff --git a/TODO.md b/TODO.md index 2a81318..bdac67d 100644 --- a/TODO.md +++ b/TODO.md @@ -35,7 +35,7 @@ _Last updated: 2026-05-11 (post Sprint 15: §7.5 DONE — Rinomina Setup-* → I | [x] | Gitea Workflow Integration | 5 | 0 | tutti completati — example.yml usa composite action | | [x] | §1 Sicurezza & hardening | 3 | 0 | 1.1, 1.4, 1.6 done; 1.2/1.3/1.5/1.7/1.8 deferred (home lab) | | [x] | §2 Affidabilità & resilienza | 7 | 0 | tutto done (2.1-2.7) | -| [x] | §3 Performance & throughput | 4 | 0 | 3.1/3.2/3.3/3.6 done; 3.4/3.5 deferred — **§3 CHIUSO** | +| [ ] | §3 Performance & throughput | 4 | 1 | 3.1/3.2/3.3/3.6 done; 3.4/3.5 deferred; **3.8 cache cargo pianificata** | | [x] | §4 Osservabilità & manutenzione | 3 | 0 | 4.1/4.3/4.4 done (4.2 Prometheus, 4.5 Dashboard rimossi) | | [x] | §5 Qualità codice & test | 5 | 0 | 5.1/5.2/5.3/5.4/5.5 done | | [x] | §6 Scalabilità & estensioni | 5 | 0 | 6.1-6.2 done, 6.3 deferred (home lab), 6.4-6.5 done, 6.6 done | @@ -938,6 +938,45 @@ cioè nelle invocazioni manuali/burn-in. `Start-Job`/`pwsh`/`$IsWindows`. Lo shim resta solo per il runner Windows, dove `CI_VENV_PYTHON` è sempre settato. Il bug non è più raggiungibile sull'host Linux. +### 3.8 [P2] [ ] Cache delle dipendenze cargo per i job Rust — PIANIFICATO (2026-07-22) +File: [plans/cargo-dependency-cache.md](plans/cargo-dependency-cache.md). + +Misurato in furyrs#12: dei ~133 s di un job Rust, ~120 s sono la compilazione +delle ~490 dipendenze esterne, che `Cargo.lock` committato tiene **bit-identiche +fra un run e l'altro**. I crate del workspace costano 7 s. Il registry — che +avevamo indicato per primo — vale 9,3 s (6,8%) e costerebbe +800 MB su template +e backup: **scartato esplicitamente**, non è un rinvio. + +Decisione presa con furyrs in #13: riusare solo gli artefatti delle dipendenze, +**mai** quelli del workspace; i tag `v*` restano a freddo. + +Il piano si discosta dalla forma `cargo-chef` proposta in #12 per un motivo +concreto: l'orchestratore riceve `build-command` come stringa opaca (furyrs ne +lancia sei, con tre set di feature diversi) e `cargo chef cook` va invocato con +gli stessi flag del build vero. Invece di indovinarli, si fa il contrario — +build normale, poi **potatura** degli artefatti dei membri del workspace da +`target/`, e si pubblica il resto. Nessuna conoscenza del comando, una sola +entry che contiene tutti i grafi di feature, nessun tool nuovo nel template. + +La proprietà di sicurezza che furyrs ha chiesto diventa così **verificabile**: +dopo la potatura, un'asserzione controlla che non sopravviva nessun fingerprint +di un membro del workspace. Se ne trova uno, il job **non pubblica**. Fail +closed — una entry mancante costa 120 s, una avvelenata costa un verdetto +sbagliato, che è il guasto peggiore che un sistema di build possa avere. + +Store: un file immutabile per chiave sotto `/var/lib/ci/cache/cargo`, chiave +`sha256(Cargo.lock) + rustc -V + host triple`, pubblicazione con `rename(2)` +atomico. Niente directory mutabile condivisa, niente lock — che è il punto +sollevato da furyrs sul locking avvisorio delle Shared Folders con 3 job +concorrenti. + +**Nota di collegamento con §2.9**: il piano include la retention dello store +*nello stesso ciclo*. Aggiungere una directory illimitata (una entry per +cambio di `Cargo.lock`, centinaia di MB l'una) senza estendere `retention run` +ripeterebbe l'errore di §2.9 su scala maggiore. + +**Prossimo passo**: Task 0 del piano — misurare prima di costruire, come in #12. + ### 8.1 [P3] [ ] Mirror GitHub Actions su Gitea locale (rimuovi dipendenza github.com) File: [.gitea/workflows/lint.yml](.gitea/workflows/lint.yml), Gitea (org `actions`). diff --git a/plans/cargo-dependency-cache.md b/plans/cargo-dependency-cache.md new file mode 100644 index 0000000..6cd29dd --- /dev/null +++ b/plans/cargo-dependency-cache.md @@ -0,0 +1,1375 @@ +# Cargo Dependency Cache — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Cut ~120s off every Rust CI job on `LinuxRust2404` by reusing the compiled **dependency** artifacts between runs, while making it structurally impossible for a workspace crate's artifacts to survive into the cache. + +**Architecture:** After a successful build, prune every workspace-member artifact out of the guest's `target/`, assert none survived, and publish what remains to a host-side content-addressed store keyed by `sha256(Cargo.lock) + rustc version + host triple`. The next job with the same key restores that tarball into its clone before building. Entries are immutable — one file per key, published by atomic `rename(2)` from a single writer — so there is no shared mutable directory and no locking. + +**Tech Stack:** Python 3.12 (`click`, `paramiko` via the existing `SshTransport`), POSIX shell in the guest, `tar` + `zstd`/`gzip`, pytest. + +## Global Constraints + +- `click.echo` everywhere — no bare `print()`. +- No `subprocess.run(..., shell=True)`. +- No global variables — state lives in `Config` and objects passed explicitly. +- All public functions and exported helpers have type hints. `mypy --strict src` must pass. +- `ruff check src tests/python` must pass. +- Coverage gate: **≥90%** on `ci_orchestrator`. +- Application code imports `backends.load_backend`, never `WorkstationVmrunBackend`. +- Do not touch the Windows/WinRM branch in any commit that touches the Linux/SSH branch (`AGENTS.md` §"Errori frequenti" #4). +- Cache store lives under `config.paths.root` (`/var/lib/ci` on the Linux host), owned by `ci-runner`. + +--- + +## Design decisions, and where they depart from thread entry #12 + +Entry #12 proposed the `cargo-chef` principle: pre-build a dependency-only +`target/` from a skeleton workspace, key it on `digest(Cargo.lock) + toolchain ++ feature set`. Two things forced a different mechanism. + +**1. The build command is opaque to us.** The orchestrator receives +`build-command` as a single string and runs it. furyrs's pipeline is five +distinct cargo invocations with three different feature sets. `cargo chef cook` +must be handed the same flags as the real build to produce the right dependency +graph — we do not have them, and parsing an arbitrary shell string to recover +them is a guess that fails silently (a wrong cook produces a cache that misses +every time, costing time instead of saving it). + +**2. Pruning after the fact needs no such knowledge.** Whatever cargo +invocations ran, every artifact they produced is in `target/`. Delete the +workspace members' artifacts and what remains is exactly the dependency set — +for all feature variants at once, in one entry. This is the mechanism +`Swatinem/rust-cache` uses in the wild. + +Consequences, all favourable: + +| | #12's proposal | This plan | +|---|---|---| +| Needs to know the cargo flags | yes | no | +| Entries per job | one per feature set | one, holding every variant cargo actually built | +| `feature set` in the cache key | required | **dropped** — cargo already encodes the feature set in each artifact's own `-` suffix, so variants coexist without colliding | +| New tool in the template | `cargo-chef` | none | + +The safety property furyrs asked for is unchanged and is now *checkable*: the +cache must contain no workspace-crate artifact. The prune step is followed by an +**assertion** that no fingerprint directory for any workspace member survives. +If one does, the job **refuses to publish** and says so. Fail closed: a missing +cache entry costs 120 seconds, a poisoned one costs a wrong verdict. + +**Scope gate.** The cache engages only when `Cargo.lock` exists in the build +workdir. Non-Rust jobs are untouched, with no flag to set. + +**Cold on release tags.** `refs/tags/v*` builds run with the cache disabled, per +furyrs's constraint in #13: those verify a release artifact, not a change, and +the clean room is the point. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `src/ci_orchestrator/cargo_cache.py` | **Create.** Pure host-side logic: key derivation from parts, store paths, lookup, atomic publish, age-based prune. No SSH, no click, no I/O beyond the store directory. | +| `src/ci_orchestrator/assets/cargo-cache-prune.sh` | **Create.** Guest-side prune + fail-closed assertion. Shipped per job over SCP, *not* baked into the template, so changing it needs no template rebuild. It lives **inside the package** and is read through `importlib.resources`, following `winimage/assets/`: the production venv is installed non-editable, so anything resolved relative to the repo root is absent at runtime. | +| `src/ci_orchestrator/assets/__init__.py` | **Create.** Empty — makes the asset directory an importable package so `importlib.resources` can find it. | +| `tests/python/test_cargo_cache.py` | **Create.** Unit tests for the store module. | +| `src/ci_orchestrator/config.py` | **Modify.** `[cargo_cache]` section → `CargoCacheConfig`. | +| `src/ci_orchestrator/commands/build.py` | **Modify.** `_linux_build` gains restore-before / save-after. Windows branch untouched. | +| `src/ci_orchestrator/commands/job.py` | **Modify.** `--cargo-cache/--no-cargo-cache`, plumbed to `_linux_build`. | +| `src/ci_orchestrator/commands/retention.py` | **Modify.** Purge cache entries older than the retention window. | +| `.gitea/actions/local-ci-build/action.yml` | **Modify.** `cargo-cache` input (`auto`/`on`/`off`), `auto` = off for `refs/tags/v*`. | +| `docs/RUNBOOK-PhaseC.md` | **Modify.** Operator section: where the store lives, how to read a miss, how to purge. | + +--- + +### Task 0: Measure before building any of it + +No production code. Entry #12 set the standard — the registry cache was +abandoned because measurement showed it bought 9.3s for +800 MB. Do the same +here before committing to the transfer cost. + +**Files:** +- Create: `plans/cargo-dependency-cache-measurements.md` + +- [ ] **Step 1: Clone the Rust template and run furyrs's pipeline once, cold** + +```bash +sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator vm new \ + --template /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx \ + --snapshot BaseClean-LinuxRust --job-id cachemeasure +# note the clone VMX path it prints, then: +sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator wait-ready \ + --vmx --guest-os linux +``` + +Inside the guest, clone furyrs and run its `ci.yml` build command to completion. + +- [ ] **Step 2: Record the five numbers that decide the design** + +```bash +ssh -i /var/lib/ci/keys/ci_linux ci_build@ ' + du -sb /opt/ci/build/target # 1. target/ raw + time tar -C /opt/ci/build -c target | wc -c # 2. tar cost + size + time tar -C /opt/ci/build -c target | zstd -3 | wc -c # 3. zstd -3 cost + size + command -v zstd || echo "ZSTD ABSENT" # 4. is zstd even present? + nproc; free -m' +``` + +Then, host side, time the transfer of a file that size over SCP to the guest. + +- [ ] **Step 3: Write the numbers down and state the go/no-go** + +Record in `plans/cargo-dependency-cache-measurements.md`: raw size, pruned size, +compressed size, compress time, transfer time, extract time. The feature is +worth building if `restore + extract` is comfortably under the ~120s it saves. +If `zstd` is absent, note it — Task 3 falls back to `gzip` and Task 3 Step 7 +adds `zstd` to the toolchain script for the next rebuild. + +- [ ] **Step 4: Destroy the clone** + +```bash +sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator vm remove --vmx +``` + +- [ ] **Step 5: Commit** + +```bash +git add plans/cargo-dependency-cache-measurements.md +git commit -m "docs(cache): measure the cargo target/ cache before building it" +``` + +--- + +### Task 1: The store — key derivation and atomic publish + +**Files:** +- Create: `src/ci_orchestrator/cargo_cache.py` +- Test: `tests/python/test_cargo_cache.py` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces, all used by Task 3 and Task 6: + - `CACHE_FORMAT: int` — bump to invalidate every entry at once. + - `cache_key(*, lock_digest: str, rustc_version: str, host_triple: str) -> str` + - `class CargoCacheStore` with + `__init__(self, root: Path, *, suffix: str = ".tar.zst") -> None`, + `path_for(self, key: str) -> Path`, + `lookup(self, key: str) -> Path | None`, + `publish(self, key: str, source: Path) -> Path`, + `prune_older_than(self, cutoff: datetime, *, what_if: bool = False) -> list[Path]` + +- [ ] **Step 1: Write the failing tests** + +```python +"""Tests for ci_orchestrator.cargo_cache — key derivation and the entry store.""" + +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest + +from ci_orchestrator.cargo_cache import CACHE_FORMAT, CargoCacheStore, cache_key + + +# ── key derivation ─────────────────────────────────────────────────────────── + + +def test_key_is_stable_for_identical_inputs() -> None: + a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64") + b = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64") + assert a == b + + +def test_key_changes_with_lock_digest() -> None: + a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64") + b = cache_key(lock_digest="def", rustc_version="rustc 1.97.1", host_triple="x86_64") + assert a != b + + +def test_key_changes_with_toolchain() -> None: + a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64") + b = cache_key(lock_digest="abc", rustc_version="rustc 1.98.0", host_triple="x86_64") + assert a != b + + +def test_key_is_filesystem_safe() -> None: + key = cache_key( + lock_digest="a/b+c=", rustc_version="rustc 1.97.1 (x /y)", host_triple="x86_64" + ) + assert key.isalnum() or all(c.isalnum() or c == "-" for c in key) + assert "/" not in key + + +def test_key_carries_the_format_version() -> None: + key = cache_key(lock_digest="abc", rustc_version="r", host_triple="t") + assert key.startswith(f"v{CACHE_FORMAT}-") + + +# ── store ──────────────────────────────────────────────────────────────────── + + +def test_lookup_returns_none_when_absent(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + assert store.lookup("v1-deadbeef") is None + + +def test_publish_then_lookup_finds_the_entry(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "incoming.tar.zst" + src.write_bytes(b"payload") + published = store.publish("v1-deadbeef", src) + assert store.lookup("v1-deadbeef") == published + assert published.read_bytes() == b"payload" + + +def test_publish_consumes_the_source_file(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "incoming.tar.zst" + src.write_bytes(b"payload") + store.publish("v1-deadbeef", src) + assert not src.exists(), "publish must move, not copy — a copy doubles peak disk" + + +def test_publish_overwrites_an_existing_key_atomically(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + first = tmp_path / "a.tar.zst" + first.write_bytes(b"first") + store.publish("v1-key", first) + second = tmp_path / "b.tar.zst" + second.write_bytes(b"second") + published = store.publish("v1-key", second) + assert published.read_bytes() == b"second" + + +def test_publish_leaves_no_temp_files_behind(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "incoming.tar.zst" + src.write_bytes(b"payload") + store.publish("v1-key", src) + leftovers = list((tmp_path / ".incoming").glob("*")) if (tmp_path / ".incoming").is_dir() else [] + assert leftovers == [] + + +def test_publish_missing_source_raises(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + with pytest.raises(FileNotFoundError): + store.publish("v1-key", tmp_path / "nope.tar.zst") + + +def test_lookup_rejects_a_key_with_a_path_separator(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + with pytest.raises(ValueError, match="unsafe cache key"): + store.lookup("../../etc/passwd") + + +# ── prune ──────────────────────────────────────────────────────────────────── + + +def _age(path: Path, days: int) -> None: + old = (datetime.now(tz=UTC) - timedelta(days=days)).timestamp() + import os + + os.utime(path, (old, old)) + + +def test_prune_removes_entries_older_than_the_cutoff(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "old.tar.zst" + src.write_bytes(b"x") + entry = store.publish("v1-old", src) + _age(entry, days=40) + + removed = store.prune_older_than(datetime.now(tz=UTC) - timedelta(days=30)) + assert removed == [entry] + assert not entry.exists() + + +def test_prune_keeps_fresh_entries(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "new.tar.zst" + src.write_bytes(b"x") + entry = store.publish("v1-new", src) + + removed = store.prune_older_than(datetime.now(tz=UTC) - timedelta(days=30)) + assert removed == [] + assert entry.exists() + + +def test_prune_what_if_reports_without_deleting(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path) + src = tmp_path / "old.tar.zst" + src.write_bytes(b"x") + entry = store.publish("v1-old", src) + _age(entry, days=40) + + removed = store.prune_older_than( + datetime.now(tz=UTC) - timedelta(days=30), what_if=True + ) + assert removed == [entry] + assert entry.exists() + + +def test_prune_on_a_missing_store_is_a_noop(tmp_path: Path) -> None: + store = CargoCacheStore(tmp_path / "never-created") + assert store.prune_older_than(datetime.now(tz=UTC)) == [] +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/python/test_cargo_cache.py -q` +Expected: collection error — `ModuleNotFoundError: No module named 'ci_orchestrator.cargo_cache'` + +- [ ] **Step 3: Write the implementation** + +```python +"""Content-addressed store for compiled Rust dependency artifacts. + +A cache entry is one immutable file per key. Publishing is a ``rename(2)`` +within the store directory, so a reader either sees the previous entry or the +new one and never a half-written file — no advisory locking, which is exactly +what a VMware shared folder cannot give us with several jobs in flight. + +Nothing here knows about SSH or guests: it takes a finished tarball and files +it, or hands back the path of an existing one. That keeps the interesting +logic testable without a VM. +""" + +from __future__ import annotations + +import hashlib +import os +import re +from datetime import UTC, datetime +from pathlib import Path + +# Bump to invalidate every entry at once — e.g. when the prune script changes +# what it strips, making older entries untrustworthy. +CACHE_FORMAT = 1 + +_SAFE_KEY = re.compile(r"^v\d+-[0-9a-f]{32,64}$") +_INCOMING = ".incoming" + + +def cache_key(*, lock_digest: str, rustc_version: str, host_triple: str) -> str: + """Derive the entry key from the inputs that make artifacts incompatible. + + ``lock_digest`` is the sha256 of ``Cargo.lock``: it changes exactly when + the dependency set changes, which is the property that makes this cache + sound. The feature set is deliberately *not* part of the key — cargo + encodes it in each artifact's own hash suffix, so several feature graphs + coexist inside one entry instead of fragmenting it. + """ + digest = hashlib.sha256( + "\0".join((lock_digest, rustc_version, host_triple)).encode("utf-8") + ).hexdigest() + return f"v{CACHE_FORMAT}-{digest}" + + +class CargoCacheStore: + """A directory of immutable cache entries, one file per key.""" + + def __init__(self, root: Path, *, suffix: str = ".tar.zst") -> None: + self.root = Path(root) + self.suffix = suffix + + # ── paths ──────────────────────────────────────────────────────────── + + def _checked(self, key: str) -> str: + if not _SAFE_KEY.match(key): + raise ValueError(f"unsafe cache key: {key!r}") + return key + + def path_for(self, key: str) -> Path: + return self.root / f"{self._checked(key)}{self.suffix}" + + # ── read ───────────────────────────────────────────────────────────── + + def lookup(self, key: str) -> Path | None: + """Return the entry path for ``key``, or ``None`` on a miss.""" + candidate = self.path_for(key) + return candidate if candidate.is_file() else None + + # ── write ──────────────────────────────────────────────────────────── + + def publish(self, key: str, source: Path) -> Path: + """Move ``source`` into the store under ``key``. Returns the entry path. + + The move goes through a staging directory on the same filesystem so the + final step is a rename: concurrent publishers of the same key both + produce a valid entry and the last one wins, with no window in which a + reader can observe a partial file. + """ + source = Path(source) + if not source.is_file(): + raise FileNotFoundError(f"nothing to publish: {source}") + target = self.path_for(key) + staging = self.root / _INCOMING + staging.mkdir(parents=True, exist_ok=True) + tmp = staging / f"{key}.{os.getpid()}{self.suffix}" + try: + os.replace(source, tmp) + except OSError: + # Different filesystem — fall back to a copy into staging, which + # still keeps the rename below atomic. + import shutil + + shutil.copy2(source, tmp) + source.unlink(missing_ok=True) + os.replace(tmp, target) + return target + + # ── maintenance ────────────────────────────────────────────────────── + + def prune_older_than( + self, cutoff: datetime, *, what_if: bool = False + ) -> list[Path]: + """Delete entries last modified before ``cutoff``. Returns what went.""" + if not self.root.is_dir(): + return [] + removed: list[Path] = [] + for entry in sorted(self.root.glob(f"*{self.suffix}")): + if not entry.is_file(): + continue + mtime = datetime.fromtimestamp(entry.stat().st_mtime, tz=UTC) + if mtime >= cutoff: + continue + removed.append(entry) + if not what_if: + entry.unlink(missing_ok=True) + return removed + + +__all__ = ["CACHE_FORMAT", "CargoCacheStore", "cache_key"] +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/python/test_cargo_cache.py -q` +Expected: all pass. + +- [ ] **Step 5: Lint and type-check** + +Run: `.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src` +Expected: `All checks passed!` and `Success: no issues found`. + +- [ ] **Step 6: Commit** + +```bash +git add src/ci_orchestrator/cargo_cache.py tests/python/test_cargo_cache.py +git commit -m "feat(cache): content-addressed store for cargo dependency artifacts" +``` + +--- + +### Task 2: The guest-side prune, and the assertion that makes it safe + +**Files:** +- Create: `src/ci_orchestrator/assets/cargo-cache-prune.sh` +- Create: `src/ci_orchestrator/assets/__init__.py` (empty) + +**Interfaces:** +- Consumes: nothing. +- Produces: a script invoked by Task 3 as + `bash cargo-cache-prune.sh `, exit 0 on a clean + prune, exit 2 when a workspace artifact survived (caller must not publish), + exit 1 on a usage/environment error. Task 3 locates it with + `importlib.resources.as_file(resources.files("ci_orchestrator.assets").joinpath("cargo-cache-prune.sh"))`. + +- [ ] **Step 0: Create the asset package** + +```bash +mkdir -p src/ci_orchestrator/assets +touch src/ci_orchestrator/assets/__init__.py +``` + +`[tool.hatch.build.targets.wheel]` already declares `packages = +["src/ci_orchestrator"]`, so a sub-package with a data file is included in the +wheel without further configuration — the same route +`winimage/assets/autounattend.win2025.xml` takes. + +- [ ] **Step 1: Write the script** + +```bash +#!/usr/bin/env bash +# Strip every workspace-crate artifact out of target/, then prove none survived. +# +# The cache is only sound while it contains dependency artifacts alone. A +# workspace crate's artifact carries the code under test, and reusing one can +# turn a red build green -- the worst failure a CI system has, because it is +# silent. So this script ends by asserting its own work and exits non-zero if +# the assertion fails; the caller must treat that as "do not publish". +set -euo pipefail + +TARGET_DIR="${1:?usage: cargo-cache-prune.sh }" +MANIFEST="${2:?usage: cargo-cache-prune.sh }" + +[ -d "$TARGET_DIR" ] || { echo "prune: no such target dir: $TARGET_DIR" >&2; exit 1; } +command -v cargo >/dev/null || { echo "prune: cargo not on PATH" >&2; exit 1; } + +# --no-deps restricts the listing to workspace members, which is precisely the +# set whose artifacts must never be cached. +members="$(cargo metadata --no-deps --format-version 1 --manifest-path "$MANIFEST" \ + | python3 -c 'import json,sys; print("\n".join(p["name"] for p in json.load(sys.stdin)["packages"]))')" + +if [ -z "$members" ]; then + echo "prune: cargo metadata listed no workspace members" >&2 + exit 1 +fi + +# Incremental state is large, machine-specific and never reusable across +# clones; examples and docs are not dependencies of anything we build. +rm -rf "$TARGET_DIR"/*/incremental "$TARGET_DIR"/*/examples "$TARGET_DIR"/doc 2>/dev/null || true + +prune_one() { + name="$1" + snake="${name//-/_}" + # Fingerprints and build-script output live in per-crate directories. + find "$TARGET_DIR" -maxdepth 3 \ + \( -path "*/.fingerprint/$name-*" \ + -o -path "*/.fingerprint/$snake-*" \ + -o -path "*/build/$name-*" \ + -o -path "*/build/$snake-*" \) \ + -prune -exec rm -rf {} + 2>/dev/null || true + # Compiled outputs: rlib/rmeta/d in deps/, plus binaries at the profile root. + find "$TARGET_DIR" -maxdepth 3 \ + \( -name "$snake-*" -o -name "lib$snake-*" -o -name "lib$snake.*" \ + -o -name "$name" -o -name "$name-*" \) \ + -exec rm -rf {} + 2>/dev/null || true +} + +while IFS= read -r name; do + [ -n "$name" ] || continue + prune_one "$name" +done </dev/null || true)" + [ -z "$hit" ] || leaked="$leaked $name" +done <&2 + exit 2 +fi + +echo "prune: ok -- $(printf '%s\n' "$members" | wc -l) workspace member(s) stripped" +``` + +- [ ] **Step 2: Make it executable and shellcheck it** + +```bash +chmod +x src/ci_orchestrator/assets/cargo-cache-prune.sh +shellcheck src/ci_orchestrator/assets/cargo-cache-prune.sh || true # advisory; SC2044 on find is expected +``` + +- [ ] **Step 3: Prove the assertion actually fires** + +This is the one guard the whole feature's safety rests on, so exercise it +against a real cargo workspace rather than trusting the code reading. In a +LinuxRust2404 clone (reuse the Task 0 procedure): + +```bash +# after a real build of the furyrs workspace: +bash cargo-cache-prune.sh /opt/ci/build/target /opt/ci/build/Cargo.toml +echo "exit=$?" # expect 0, and a "prune: ok" line + +# now re-create one workspace fingerprint by hand and re-run +mkdir -p /opt/ci/build/target/debug/.fingerprint/fury-core-deadbeef +bash cargo-cache-prune.sh /opt/ci/build/target /opt/ci/build/Cargo.toml +echo "exit=$?" # expect 2, with "REFUSING to publish" +``` + +Both outcomes must be observed before this task is considered done. If the +second run exits 0, the prune glob is deleting the directory it is supposed to +detect — widen the assertion, do not narrow the test. + +- [ ] **Step 4: Verify the dependency artifacts survived** + +```bash +ls /opt/ci/build/target/*/deps/ | head # expect many libserde-*, libtokio-* etc. +du -sh /opt/ci/build/target # record; compare against Task 0's raw size +``` + +Expected: dependency `.rlib`/`.rmeta` files are still present. A prune that +emptied `deps/` is a bug — record the pruned size in +`plans/cargo-dependency-cache-measurements.md`. + +- [ ] **Step 5: Commit** + +```bash +git add src/ci_orchestrator/assets/ plans/cargo-dependency-cache-measurements.md +git commit -m "feat(cache): guest-side prune of workspace artifacts, with a fail-closed assert" +``` + +--- + +### Task 3: Restore before the build, publish after it + +**Files:** +- Modify: `src/ci_orchestrator/commands/build.py` (`_linux_build` only — leave `_windows_build` alone) +- Test: `tests/python/test_commands_build.py` + +**Interfaces:** +- Consumes: `CargoCacheStore`, `cache_key`, `CACHE_FORMAT` from Task 1; + `ci_orchestrator/assets/cargo-cache-prune.sh` from Task 2. +- Produces: `_linux_build(..., cargo_cache_root: Path | None = None)` — `None` + disables the feature entirely, which is the default so no existing caller + changes behaviour. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/python/test_commands_build.py`. `FakeSsh` already exists in +that file; if its recorded commands are not exposed as `.commands`, add that +list first in the same style as the existing fake. + +```python +# ── cargo dependency cache ─────────────────────────────────────────────────── + + +def test_linux_build_without_cache_root_runs_no_cache_commands(monkeypatch, tmp_path): + fake = _install_fake_ssh(monkeypatch) + _linux_build(**_linux_kwargs(clone_url="https://example/r.git")) + assert not any("cargo-cache" in c for c in fake.commands) + + +def test_linux_build_skips_cache_when_no_cargo_lock(monkeypatch, tmp_path): + fake = _install_fake_ssh(monkeypatch) + fake.result_for("test -f", returncode=1) # no Cargo.lock in the workdir + _linux_build(**_linux_kwargs(clone_url="https://example/r.git", + cargo_cache_root=tmp_path)) + assert not any("cargo-cache-prune" in c for c in fake.commands) + + +def test_linux_build_restores_an_existing_entry(monkeypatch, tmp_path): + from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key + + fake = _install_fake_ssh(monkeypatch) + fake.result_for("sha256sum", stdout="deadbeef Cargo.lock\n") + fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n") + fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n") + + store = CargoCacheStore(tmp_path) + key = cache_key(lock_digest="deadbeef", + rustc_version="rustc 1.97.1 (abc 2026-01-01)", + host_triple="x86_64-unknown-linux-gnu") + seed = tmp_path / "seed.tar.zst" + seed.write_bytes(b"entry") + store.publish(key, seed) + + _linux_build(**_linux_kwargs(clone_url="https://example/r.git", + cargo_cache_root=tmp_path)) + assert any("tar" in c and "target" in c for c in fake.commands) + assert fake.copied, "a cache hit must upload the entry to the guest" + + +def test_linux_build_publishes_on_a_miss(monkeypatch, tmp_path): + from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key + + fake = _install_fake_ssh(monkeypatch) + fake.result_for("sha256sum", stdout="cafe Cargo.lock\n") + fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n") + fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n") + fake.fetch_writes = b"fresh-entry" + + _linux_build(**_linux_kwargs(clone_url="https://example/r.git", + cargo_cache_root=tmp_path)) + + key = cache_key(lock_digest="cafe", + rustc_version="rustc 1.97.1 (abc 2026-01-01)", + host_triple="x86_64-unknown-linux-gnu") + assert CargoCacheStore(tmp_path).lookup(key) is not None + + +def test_linux_build_does_not_publish_when_the_prune_assertion_fails(monkeypatch, tmp_path): + from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key + + fake = _install_fake_ssh(monkeypatch) + fake.result_for("sha256sum", stdout="cafe Cargo.lock\n") + fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n") + fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n") + fake.result_for("cargo-cache-prune", returncode=2, + stderr="prune: REFUSING to publish -- workspace artifacts survived: fury-core") + + _linux_build(**_linux_kwargs(clone_url="https://example/r.git", + cargo_cache_root=tmp_path)) + + key = cache_key(lock_digest="cafe", + rustc_version="rustc 1.97.1 (abc 2026-01-01)", + host_triple="x86_64-unknown-linux-gnu") + assert CargoCacheStore(tmp_path).lookup(key) is None, ( + "a failed prune assertion must never reach the store" + ) + + +def test_cache_failure_does_not_fail_the_build(monkeypatch, tmp_path): + fake = _install_fake_ssh(monkeypatch) + fake.result_for("sha256sum", returncode=1, stderr="boom") + # Must not raise: the cache is an optimisation, never a failure mode. + _linux_build(**_linux_kwargs(clone_url="https://example/r.git", + cargo_cache_root=tmp_path)) +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `.venv/bin/python -m pytest tests/python/test_commands_build.py -q -k cargo_cache` +Expected: FAIL — `_linux_build() got an unexpected keyword argument 'cargo_cache_root'` + +- [ ] **Step 3: Add the cache helpers to `build.py`** + +Insert above `_linux_build`: + +```python +_PRUNE_ASSET = "cargo-cache-prune.sh" +_GUEST_PRUNE = "/tmp/cargo-cache-prune.sh" +_GUEST_TARBALL = "/tmp/cargo-cache-entry.tar" + + +def _cargo_cache_probe(t: SshTransport, workdir: str) -> tuple[str, str] | None: + """Return ``(key, compressor)`` for this workspace, or ``None`` to skip. + + ``None`` means "not a Rust build, or we could not read what we need" — the + cache is an optimisation and every uncertainty here resolves to skipping it. + """ + if t.run(f"test -f {_sh_quote(workdir + '/Cargo.lock')}", check=False).returncode != 0: + return None + lock = t.run(f"sha256sum {_sh_quote(workdir + '/Cargo.lock')}", check=False) + version = t.run("rustc -V", check=False) + triple = t.run("rustc -vV | sed -n 's/^host: //p'", check=False) + if not (lock.ok and version.ok and triple.ok): + click.echo("[cache] could not read Cargo.lock digest or rustc version; skipping") + return None + compressor = "zstd" if t.run("command -v zstd", check=False).ok else "gzip" + key = cache_key( + lock_digest=lock.stdout.split()[0], + rustc_version=version.stdout.strip(), + host_triple=triple.stdout.strip(), + ) + return key, compressor + + +def _cargo_cache_restore( + t: SshTransport, store: CargoCacheStore, key: str, workdir: str, compressor: str +) -> bool: + """Upload and unpack an existing entry. Returns True on a hit.""" + entry = store.lookup(key) + if entry is None: + click.echo(f"[cache] miss for {key} — building cold") + return False + size_mb = entry.stat().st_size / 1_048_576 + click.echo(f"[cache] hit for {key} ({size_mb:.0f} MB) — restoring") + guest_entry = f"{_GUEST_TARBALL}.{'zst' if compressor == 'zstd' else 'gz'}" + t.copy(str(entry), guest_entry) + flag = "--zstd" if compressor == "zstd" else "-z" + t.run( + f"tar {flag} -xf {_sh_quote(guest_entry)} -C {_sh_quote(workdir)} && " + f"rm -f {_sh_quote(guest_entry)}" + ) + return True + + +def _cargo_cache_publish( + t: SshTransport, store: CargoCacheStore, key: str, workdir: str, compressor: str +) -> None: + """Prune workspace artifacts, then file the remainder under ``key``. + + A non-zero prune means the fail-closed assertion tripped: say so loudly and + publish nothing. A poisoned entry would be far more expensive than a miss. + """ + # as_file() because the production venv is non-editable: the script is + # package data, not a path relative to a source checkout. + asset = resources.files("ci_orchestrator.assets").joinpath(_PRUNE_ASSET) + with resources.as_file(asset) as prune_script: + t.copy(str(prune_script), _GUEST_PRUNE) + pruned = t.run( + f"bash {_GUEST_PRUNE} {_sh_quote(workdir + '/target')} " + f"{_sh_quote(workdir + '/Cargo.toml')}", + check=False, + ) + click.echo(f"[cache] {pruned.stdout.strip() or pruned.stderr.strip()}") + if pruned.returncode != 0: + click.echo("[cache] prune refused — publishing nothing", err=True) + return + + ext = "zst" if compressor == "zstd" else "gz" + flag = "--zstd" if compressor == "zstd" else "-z" + guest_tar = f"{_GUEST_TARBALL}.{ext}" + packed = t.run( + f"tar {flag} -cf {_sh_quote(guest_tar)} -C {_sh_quote(workdir)} target", + check=False, + ) + if not packed.ok: + click.echo("[cache] packing failed — publishing nothing", err=True) + return + + with tempfile.TemporaryDirectory() as td: + local = Path(td) / f"entry.tar.{ext}" + t.fetch(guest_tar, local) + published = store.publish(key, local) + click.echo( + f"[cache] published {key} " + f"({published.stat().st_size / 1_048_576:.0f} MB)" + ) +``` + +Add to the imports at the top of `build.py`: + +```python +from importlib import resources + +from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key +``` + +- [ ] **Step 4: Call them from `_linux_build`** + +Add the parameter to the signature (keyword-only, defaulted, so no existing +caller changes): + +```python + use_xvfb: bool = False, + cargo_cache_root: Path | None = None, +) -> None: +``` + +Immediately **after** the source-mtime normalisation `t.run(find ... touch ...)` +and **before** `env_prefix = ...`, insert: + +```python + # Cargo dependency cache. Everything here is best-effort: a cache + # problem must never turn a good build red, so failures degrade to a + # cold build with a message. + cache_key_value: str | None = None + compressor = "gzip" + store: CargoCacheStore | None = None + if cargo_cache_root is not None: + try: + store = CargoCacheStore(cargo_cache_root) + probed = _cargo_cache_probe(t, workdir) + if probed is not None: + cache_key_value, compressor = probed + _cargo_cache_restore(t, store, cache_key_value, workdir, compressor) + except (TransportError, OSError, ValueError) as exc: + click.echo(f"[cache] restore skipped: {exc}", err=True) + cache_key_value = None +``` + +Immediately **after** the build-failure check (`if result.returncode != 0: raise ...`) +and **before** the `if skip_artifact:` block, insert: + +```python + # Only publish on a miss: in the steady state this uploads nothing. + if store is not None and cache_key_value and store.lookup(cache_key_value) is None: + try: + _cargo_cache_publish(t, store, cache_key_value, workdir, compressor) + except (TransportError, OSError, ValueError) as exc: + click.echo(f"[cache] publish skipped: {exc}", err=True) +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `.venv/bin/python -m pytest tests/python/test_commands_build.py -q` +Expected: all pass, including the pre-existing tests (the Windows branch and +every non-Rust Linux path must be untouched). + +- [ ] **Step 6: Lint, type-check, full suite with the coverage gate** + +Run: +```bash +.venv/bin/python -m ruff check src tests/python +.venv/bin/python -m mypy --strict src +.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90 +``` +Expected: clean, clean, and coverage ≥90%. + +- [ ] **Step 7: If Task 0 found `zstd` absent, add it to the toolchain script** + +Only if Step 4 of Task 0 recorded `ZSTD ABSENT`. In +`template/Install-CIToolchain-LinuxRust2404.sh`, add `zstd` to the apt package +list, next to the existing build tooling. It takes effect at the next template +rebuild; until then the `gzip` fallback above carries the feature. + +- [ ] **Step 8: Commit** + +```bash +git add src/ci_orchestrator/commands/build.py tests/python/test_commands_build.py \ + template/Install-CIToolchain-LinuxRust2404.sh +git commit -m "feat(cache): restore and publish the cargo dependency cache around Linux builds" +``` + +--- + +### Task 4: CLI flag and config section + +**Files:** +- Modify: `src/ci_orchestrator/config.py` +- Modify: `src/ci_orchestrator/commands/job.py` +- Modify: `config.example.toml` +- Test: `tests/python/test_config.py`, `tests/python/test_commands_job.py` + +**Interfaces:** +- Consumes: `_linux_build(..., cargo_cache_root=...)` from Task 3. +- Produces: `Config.cargo_cache: CargoCacheConfig` with fields + `enabled: bool = True`, `root: Path` (default `/cache/cargo`), + `retention_days: int = 30`; and `ci job --cargo-cache/--no-cargo-cache`. + +- [ ] **Step 1: Write the failing config test** + +Append to `tests/python/test_config.py`: + +```python +def test_cargo_cache_defaults_under_paths_root(tmp_path: Path) -> None: + cfg_file = tmp_path / "config.toml" + cfg_file.write_text( + '[paths]\nroot = "/var/lib/ci"\ntemplates = "/var/lib/ci/templates"\n' + 'build_vms = "/var/lib/ci/build-vms"\nartifacts = "/var/lib/ci/artifacts"\n' + 'keys = "/var/lib/ci/keys"\n', + encoding="utf-8", + ) + cfg = load_config(config_path=cfg_file, env={}) + assert cfg.cargo_cache.enabled is True + assert cfg.cargo_cache.root == Path("/var/lib/ci/cache/cargo") + + +def test_cargo_cache_section_overrides_the_defaults(tmp_path: Path) -> None: + cfg_file = tmp_path / "config.toml" + cfg_file.write_text( + '[paths]\nroot = "/var/lib/ci"\ntemplates = "/var/lib/ci/templates"\n' + 'build_vms = "/var/lib/ci/build-vms"\nartifacts = "/var/lib/ci/artifacts"\n' + 'keys = "/var/lib/ci/keys"\n' + '\n[cargo_cache]\nenabled = false\nroot = "/mnt/fast/cargo"\n' + "retention_days = 7\n", + encoding="utf-8", + ) + cfg = load_config(config_path=cfg_file, env={}) + assert cfg.cargo_cache.enabled is False + assert cfg.cargo_cache.root == Path("/mnt/fast/cargo") + assert cfg.cargo_cache.retention_days == 7 +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `.venv/bin/python -m pytest tests/python/test_config.py -q -k cargo_cache` +Expected: FAIL — `AttributeError: 'Config' object has no attribute 'cargo_cache'` + +- [ ] **Step 3: Add the dataclass and loader wiring** + +In `config.py`, beside the other section dataclasses: + +```python +@dataclass(frozen=True) +class CargoCacheConfig: + """Where compiled Rust dependency artifacts are filed between jobs.""" + + enabled: bool = True + root: Path = Path("/var/lib/ci/cache/cargo") + retention_days: int = 30 +``` + +Add the field to `Config` (after `ip_pool`, so no positional call breaks): + +```python + cargo_cache: CargoCacheConfig = field(default_factory=CargoCacheConfig) +``` + +In `load_config`, after the `ip_pool` block: + +```python + cc_raw = data.get("cargo_cache", {}) if isinstance(data, dict) else {} + cargo_cache = CargoCacheConfig( + enabled=bool(cc_raw.get("enabled", True)), + root=Path(cc_raw.get("root", paths.root / "cache" / "cargo")), + retention_days=int(cc_raw.get("retention_days", 30)), + ) +``` + +and pass `cargo_cache=cargo_cache` into the `Config(...)` construction. + +- [ ] **Step 4: Add the CLI flag to `job.py`** + +Add the option beside `--use-shared-cache`: + +```python +@click.option( + "--cargo-cache/--no-cargo-cache", + "cargo_cache", + default=True, + show_default=True, + help="Reuse compiled Rust dependency artifacts between jobs (Linux only; " + "engages only when the workspace has a Cargo.lock).", +) +``` + +Add `cargo_cache: bool,` to the `job(...)` signature, and in the Linux branch +of Phase 4 pass it down: + +```python + use_xvfb=use_xvfb, + cargo_cache_root=( + config.cargo_cache.root + if cargo_cache and config.cargo_cache.enabled + else None + ), +``` + +- [ ] **Step 5: Write the failing job test, then run the suite** + +Append to `tests/python/test_commands_job.py`, in the style of the existing +`_linux_build` call-capture tests: + +```python +def test_job_passes_cache_root_to_linux_build(monkeypatch, tmp_path): + captured = _capture_linux_build(monkeypatch) + _run_job(monkeypatch, tmp_path, extra_args=[]) + assert captured["cargo_cache_root"] is not None + + +def test_job_no_cargo_cache_disables_it(monkeypatch, tmp_path): + captured = _capture_linux_build(monkeypatch) + _run_job(monkeypatch, tmp_path, extra_args=["--no-cargo-cache"]) + assert captured["cargo_cache_root"] is None +``` + +Run: `.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90` +Expected: all pass, coverage ≥90%. + +- [ ] **Step 6: Document the section in `config.example.toml`** + +```toml +# Compiled Rust dependency artifacts, reused between jobs. Engages only for +# Linux guests whose build workdir contains a Cargo.lock; workspace crates are +# stripped before an entry is stored, so a cached entry can never carry the +# code under test. +[cargo_cache] +enabled = true +root = "/var/lib/ci/cache/cargo" +retention_days = 30 +``` + +- [ ] **Step 7: Lint, type-check, commit** + +```bash +.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src +git add src/ci_orchestrator/config.py src/ci_orchestrator/commands/job.py \ + config.example.toml tests/python/test_config.py tests/python/test_commands_job.py +git commit -m "feat(cache): --cargo-cache flag and [cargo_cache] config section" +``` + +--- + +### Task 5: Action input, and cold builds on release tags + +**Files:** +- Modify: `.gitea/actions/local-ci-build/action.yml` + +**Interfaces:** +- Consumes: `ci job --cargo-cache/--no-cargo-cache` from Task 4. +- Produces: action input `cargo-cache` with values `auto` (default), `on`, `off`. + +- [ ] **Step 1: Add the input** + +In the `inputs:` block, after `use-shared-cache`: + +```yaml + cargo-cache: + description: > + Reuse compiled Rust dependency artifacts between jobs: 'auto' (default) + caches on branches and pull requests but builds release tags (refs/tags/v*) + cold, 'on' always caches, 'off' never does. Only ever engages for Linux + guests whose workspace has a Cargo.lock. Workspace crates are stripped + from an entry before it is stored, so a restored cache cannot carry the + code under test -- but a release build verifies an artifact rather than a + change, so 'auto' gives it the clean room. + required: false + default: 'auto' +``` + +- [ ] **Step 2: Pass it into the env block** + +Beside `INPUT_USE_SHARED_CACHE`: + +```yaml + INPUT_CARGO_CACHE: ${{ inputs.cargo-cache }} + GITHUB_REF_FOR_CACHE: ${{ github.ref }} +``` + +- [ ] **Step 3: Resolve it next to the other input resolution** + +In the same PowerShell step that resolves `$resolvedSnapshotName`: + +```powershell + # Release tags build cold: they verify a release artifact, not a + # change, so the clean room is worth the two minutes (furyrs#13). + $cargoCacheMode = if ($env:INPUT_CARGO_CACHE) { $env:INPUT_CARGO_CACHE } else { 'auto' } + switch ($cargoCacheMode) { + 'on' { $useCargoCache = $true } + 'off' { $useCargoCache = $false } + 'auto' { + $useCargoCache = -not ($env:GITHUB_REF_FOR_CACHE -like 'refs/tags/v*') + } + default { + throw "cargo-cache must be one of 'auto', 'on', 'off' — got '$cargoCacheMode'." + } + } + Write-Host "[job] Cargo cache: $(if ($useCargoCache) { 'on' } else { 'off (cold)' })" +``` + +- [ ] **Step 4: Add the flag to the argument list** + +Where the other `ci job` switches are appended: + +```powershell + if (-not $useCargoCache) { $jobArgs += '--no-cargo-cache' } +``` + +- [ ] **Step 5: Verify the action parses and the branches behave** + +```bash +.venv/bin/python -c "import yaml,sys; yaml.safe_load(open('.gitea/actions/local-ci-build/action.yml')); print('yaml ok')" +pwsh -NoProfile -Command " + \$env:INPUT_CARGO_CACHE='auto'; \$env:GITHUB_REF_FOR_CACHE='refs/tags/v1.2.3' + # paste the switch block; expect: off (cold) +" +``` + +Expected: `yaml ok`, and the `auto` + tag combination selects the cold path. +If `pwsh` is not installed on this host (Phase C removed it), verify on the +Windows runner or by reading the branch — but record which you did. + +- [ ] **Step 6: Commit, then publish — the action only runs from the server** + +```bash +git add .gitea/actions/local-ci-build/action.yml +git commit -m "feat(action): cargo-cache input, cold on release tags" +git push origin main +sudo -u ci-runner /opt/ci/venv/bin/python tools/ci-action-published.py +``` + +Expected: `OK — .gitea/ matches origin/main`. Committing is not deploying for +anything under `.gitea/`; this check is the proof. + +--- + +### Task 6: Retention — the store must not grow without bound + +**Files:** +- Modify: `src/ci_orchestrator/commands/retention.py` +- Test: `tests/python/test_commands_retention.py` + +**Interfaces:** +- Consumes: `CargoCacheStore.prune_older_than` from Task 1, + `Config.cargo_cache.retention_days` from Task 4. + +TODO §2.9 records that `retention run` already warns about low disk while +freeing 0.0 GB, because it only governs artifacts, logs and IP leases. Adding a +new unbounded directory without adding it to retention repeats that mistake at +a larger scale — an entry is hundreds of megabytes and one is minted per +`Cargo.lock` change. + +- [ ] **Step 1: Write the failing test** + +```python +def test_retention_purges_stale_cargo_cache_entries(tmp_path, monkeypatch): + import os + from datetime import UTC, datetime, timedelta + + from ci_orchestrator.cargo_cache import CargoCacheStore + + store_root = tmp_path / "cache" / "cargo" + store_root.mkdir(parents=True) + old = store_root / ("v1-" + "a" * 40 + ".tar.zst") + old.write_bytes(b"x") + stamp = (datetime.now(tz=UTC) - timedelta(days=90)).timestamp() + os.utime(old, (stamp, stamp)) + + _run_retention(monkeypatch, tmp_path, retention_days=30) + assert not old.exists() + + +def test_retention_what_if_keeps_cargo_cache_entries(tmp_path, monkeypatch): + import os + from datetime import UTC, datetime, timedelta + + store_root = tmp_path / "cache" / "cargo" + store_root.mkdir(parents=True) + old = store_root / ("v1-" + "a" * 40 + ".tar.zst") + old.write_bytes(b"x") + stamp = (datetime.now(tz=UTC) - timedelta(days=90)).timestamp() + os.utime(old, (stamp, stamp)) + + _run_retention(monkeypatch, tmp_path, retention_days=30, what_if=True) + assert old.exists() +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `.venv/bin/python -m pytest tests/python/test_commands_retention.py -q -k cargo` +Expected: FAIL — the entry still exists. + +- [ ] **Step 3: Implement** + +In `retention_run`, after the existing artifact and log purges: + +```python + if config.cargo_cache.enabled: + cache_cutoff = datetime.now(tz=UTC) - timedelta( + days=min(effective_days, config.cargo_cache.retention_days) + ) + removed = CargoCacheStore(config.cargo_cache.root).prune_older_than( + cache_cutoff, what_if=what_if + ) + freed_mb = sum(p.stat().st_size for p in removed if p.exists()) / 1_048_576 + verb = "WhatIf: would purge" if what_if else "Purged" + click.echo( + f"[RetentionPolicy] {verb} {len(removed)} cargo cache entr" + f"{'y' if len(removed) == 1 else 'ies'}" + + (f" ({freed_mb:.0f} MB)" if what_if else "") + ) +``` + +Add the import: `from ci_orchestrator.cargo_cache import CargoCacheStore`. + +- [ ] **Step 4: Run tests, lint, type-check** + +```bash +.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90 +.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src +``` + +- [ ] **Step 5: Commit** + +```bash +git add src/ci_orchestrator/commands/retention.py tests/python/test_commands_retention.py +git commit -m "feat(cache): purge stale cargo cache entries in retention run" +``` + +--- + +### Task 7: End-to-end validation on the real workspace, and report the numbers + +**Files:** +- Modify: `plans/cargo-dependency-cache-measurements.md` +- Modify: `docs/RUNBOOK-PhaseC.md` +- Modify: `TODO.md` + +- [ ] **Step 1: Cold run — populates the store** + +The build command is the `build-command:` block of +`/home/simone/_Workspace/furyrs/.gitea/workflows/ci.yml` (line 60 onward) — +copy it verbatim into a file rather than retyping it, because it carries the +`/etc/profile.d/rust.sh` sourcing and the `--locked` flags that make the run +representative: + +```bash +sed -n '/^ build-command: |/,/^ [a-z-]*:/p' \ + /home/simone/_Workspace/furyrs/.gitea/workflows/ci.yml \ + | sed '1d;$d;s/^ //' > /tmp/furyrs-build.sh + +sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator job \ + --job-id cache-e2e-cold \ + --repo-url https://gitea.emulab.it/Simone/furyrs.git --branch main \ + --template-path /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx \ + --snapshot-name BaseClean-LinuxRust --guest-os linux \ + --build-command "$(cat /tmp/furyrs-build.sh)" --skip-artifact +``` + +At the time of writing that block runs, in order: `cargo fmt --all -- --check`, +`cargo test --workspace --locked`, `cargo test -p fury-core --features +ddr5-scaffold --locked`, `cargo clippy -p fury-cli --all-targets --features +direct-backend --locked -- -D warnings`, `cargo clippy --workspace +--all-targets --locked -- -D warnings`, `cargo clippy -p fury-gui --features +gui --bin fury-gui --locked -- -D warnings`, and `cargo audit`. Three distinct +feature sets, which is exactly the case the single-entry design has to handle. + +Expected in the log: `[cache] miss for v1-… — building cold`, then +`[cache] prune: ok — N workspace member(s) stripped`, then +`[cache] published v1-… (NNN MB)`. Record the total job time. + +- [ ] **Step 2: Warm run — same commit** + +Re-run the identical command with `--job-id cache-e2e-warm`. + +Expected: `[cache] hit for v1-… (NNN MB) — restoring`, no `published` line +(publish happens on a miss only), and a materially shorter build phase. Record +the total. + +- [ ] **Step 3: The run that actually matters — warm, with modified source** + +Push a trivial source change to a scratch branch of furyrs and run against it. +This is the real CI scenario; #12 measured 6.8s for the build step here. + +Expected: cache hit, workspace crates recompiled, dependencies not. + +- [ ] **Step 4: Prove the cache cannot hide a regression** + +Introduce a deliberate compile error in a workspace crate on a scratch branch +and run warm. + +Expected: **the build fails.** If it passes, stop and do not ship: the prune is +leaking workspace artifacts and the assertion in Task 2 is not catching it. +This single test is what the whole safety argument reduces to. + +- [ ] **Step 5: Confirm the release-tag path is cold** + +Run once with `--no-cargo-cache` and confirm the log shows a cold build and no +`[cache] hit` line. + +- [ ] **Step 6: Write the numbers into the measurements file** + +Cold total, warm-identical total, warm-modified total, entry size on disk, +restore time, publish time. State the net saving against the 133s baseline from +#12/#13. + +- [ ] **Step 7: Document for the operator** + +Add to `docs/RUNBOOK-PhaseC.md` §6 (Maintenance): where the store lives +(`/var/lib/ci/cache/cargo`), how to read `[cache] hit`/`miss`/`REFUSING`, how to +invalidate everything (bump `CACHE_FORMAT`), and how to purge by hand +(`rm /var/lib/ci/cache/cargo/*.tar.zst` — entries are immutable, deleting one is +always safe and costs one cold build). + +- [ ] **Step 8: Close §7 in TODO.md and the thread** + +Mark the cargo-cache entry done in `TODO.md` with the measured numbers, then +append a `CI → furyrs` entry to +`/home/simone/_Workspace/furyrs/docs/ci-rust-integration.md` closing #7: what +was built, the deviation from #12's cargo-chef shape and why, the measured +saving, that release tags are cold as they asked, and that `cargo-cache: 'off'` +is available as a per-workflow override. + +- [ ] **Step 9: Commit and publish** + +```bash +git add plans/cargo-dependency-cache-measurements.md docs/RUNBOOK-PhaseC.md TODO.md +git commit -m "docs(cache): end-to-end numbers for the cargo dependency cache" +git push origin main +sudo -u ci-runner /opt/ci/venv/bin/python tools/ci-action-published.py +``` + +--- + +## Rollback + +Every piece is opt-out at a different level, cheapest first: + +1. Per workflow: `cargo-cache: 'off'` in the composite action inputs. +2. Per host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`. +3. Everything at once: bump `CACHE_FORMAT` in `cargo_cache.py` — every existing + entry becomes unreachable and every job goes cold, with no file to delete. +4. Disk: `rm /var/lib/ci/cache/cargo/*.tar.zst`. Entries are immutable and + independent; deleting any subset costs exactly one cold build each. + +No step needs a template rebuild, so none of them has to be coordinated with +furyrs's pin-bump window (thread #4).