Documented with `sudo -u ci-runner`, copied from the neighbouring tools that call the Gitea API. Wrong here: the guard fetches, fetching writes .git/FETCH_HEAD, and that file belongs to the committing user -- so as ci-runner it died with "impossibile aprire '.git/FETCH_HEAD': Permesso negato" before comparing anything. A guard that cannot run is worse than no guard, since the failure looks like infrastructure noise rather than an answer. The question it asks is "does what I have match what is published", so it belongs to whoever holds the working tree.
1378 lines
53 KiB
Markdown
1378 lines
53 KiB
Markdown
# Cargo Dependency Cache — Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Cut ~120s off every Rust CI job on `LinuxRust2404` by reusing the compiled **dependency** artifacts between runs, while making it structurally impossible for a workspace crate's artifacts to survive into the cache.
|
|
|
|
**Architecture:** After a successful build, prune every workspace-member artifact out of the guest's `target/`, assert none survived, and publish what remains to a host-side content-addressed store keyed by `sha256(Cargo.lock) + rustc version + host triple`. The next job with the same key restores that tarball into its clone before building. Entries are immutable — one file per key, published by atomic `rename(2)` from a single writer — so there is no shared mutable directory and no locking.
|
|
|
|
**Tech Stack:** Python 3.12 (`click`, `paramiko` via the existing `SshTransport`), POSIX shell in the guest, `tar` + `zstd`/`gzip`, pytest.
|
|
|
|
## Global Constraints
|
|
|
|
- `click.echo` everywhere — no bare `print()`.
|
|
- No `subprocess.run(..., shell=True)`.
|
|
- No global variables — state lives in `Config` and objects passed explicitly.
|
|
- All public functions and exported helpers have type hints. `mypy --strict src` must pass.
|
|
- `ruff check src tests/python` must pass.
|
|
- Coverage gate: **≥90%** on `ci_orchestrator`.
|
|
- Application code imports `backends.load_backend`, never `WorkstationVmrunBackend`.
|
|
- Do not touch the Windows/WinRM branch in any commit that touches the Linux/SSH branch (`AGENTS.md` §"Errori frequenti" #4).
|
|
- Cache store lives under `config.paths.root` (`/var/lib/ci` on the Linux host), owned by `ci-runner`.
|
|
|
|
---
|
|
|
|
## Design decisions, and where they depart from thread entry #12
|
|
|
|
Entry #12 proposed the `cargo-chef` principle: pre-build a dependency-only
|
|
`target/` from a skeleton workspace, key it on `digest(Cargo.lock) + toolchain
|
|
+ feature set`. Two things forced a different mechanism.
|
|
|
|
**1. The build command is opaque to us.** The orchestrator receives
|
|
`build-command` as a single string and runs it. furyrs's pipeline is five
|
|
compiling cargo invocations across four feature configurations. `cargo chef cook`
|
|
must be handed the same flags as the real build to produce the right dependency
|
|
graph — we do not have them, and parsing an arbitrary shell string to recover
|
|
them is a guess that fails silently (a wrong cook produces a cache that misses
|
|
every time, costing time instead of saving it).
|
|
|
|
**2. Pruning after the fact needs no such knowledge.** Whatever cargo
|
|
invocations ran, every artifact they produced is in `target/`. Delete the
|
|
workspace members' artifacts and what remains is exactly the dependency set —
|
|
for all feature variants at once, in one entry. This is the mechanism
|
|
`Swatinem/rust-cache` uses in the wild.
|
|
|
|
Consequences, all favourable:
|
|
|
|
| | #12's proposal | This plan |
|
|
|---|---|---|
|
|
| Needs to know the cargo flags | yes | no |
|
|
| Entries per job | one per feature set | one, holding every variant cargo actually built |
|
|
| `feature set` in the cache key | required | **dropped** — cargo already encodes the feature set in each artifact's own `-<hash>` suffix, so variants coexist without colliding |
|
|
| New tool in the template | `cargo-chef` | none |
|
|
|
|
The safety property furyrs asked for is unchanged and is now *checkable*: the
|
|
cache must contain no workspace-crate artifact. The prune step is followed by an
|
|
**assertion** that no fingerprint directory for any workspace member survives.
|
|
If one does, the job **refuses to publish** and says so. Fail closed: a missing
|
|
cache entry costs 120 seconds, a poisoned one costs a wrong verdict.
|
|
|
|
**Scope gate.** The cache engages only when `Cargo.lock` exists in the build
|
|
workdir. Non-Rust jobs are untouched, with no flag to set.
|
|
|
|
**Cold on release tags.** `refs/tags/v*` builds run with the cache disabled, per
|
|
furyrs's constraint in #13: those verify a release artifact, not a change, and
|
|
the clean room is the point.
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
| File | Responsibility |
|
|
|---|---|
|
|
| `src/ci_orchestrator/cargo_cache.py` | **Create.** Pure host-side logic: key derivation from parts, store paths, lookup, atomic publish, age-based prune. No SSH, no click, no I/O beyond the store directory. |
|
|
| `src/ci_orchestrator/assets/cargo-cache-prune.sh` | **Create.** Guest-side prune + fail-closed assertion. Shipped per job over SCP, *not* baked into the template, so changing it needs no template rebuild. It lives **inside the package** and is read through `importlib.resources`, following `winimage/assets/`: the production venv is installed non-editable, so anything resolved relative to the repo root is absent at runtime. |
|
|
| `src/ci_orchestrator/assets/__init__.py` | **Create.** Empty — makes the asset directory an importable package so `importlib.resources` can find it. |
|
|
| `tests/python/test_cargo_cache.py` | **Create.** Unit tests for the store module. |
|
|
| `src/ci_orchestrator/config.py` | **Modify.** `[cargo_cache]` section → `CargoCacheConfig`. |
|
|
| `src/ci_orchestrator/commands/build.py` | **Modify.** `_linux_build` gains restore-before / save-after. Windows branch untouched. |
|
|
| `src/ci_orchestrator/commands/job.py` | **Modify.** `--cargo-cache/--no-cargo-cache`, plumbed to `_linux_build`. |
|
|
| `src/ci_orchestrator/commands/retention.py` | **Modify.** Purge cache entries older than the retention window. |
|
|
| `.gitea/actions/local-ci-build/action.yml` | **Modify.** `cargo-cache` input (`auto`/`on`/`off`), `auto` = off for `refs/tags/v*`. |
|
|
| `docs/RUNBOOK-PhaseC.md` | **Modify.** Operator section: where the store lives, how to read a miss, how to purge. |
|
|
|
|
---
|
|
|
|
### Task 0: Measure before building any of it
|
|
|
|
No production code. Entry #12 set the standard — the registry cache was
|
|
abandoned because measurement showed it bought 9.3s for +800 MB. Do the same
|
|
here before committing to the transfer cost.
|
|
|
|
**Files:**
|
|
- Create: `plans/cargo-dependency-cache-measurements.md`
|
|
|
|
- [ ] **Step 1: Clone the Rust template and run furyrs's pipeline once, cold**
|
|
|
|
```bash
|
|
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator vm new \
|
|
--template /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx \
|
|
--snapshot BaseClean-LinuxRust --job-id cachemeasure
|
|
# note the clone VMX path it prints, then:
|
|
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator wait-ready \
|
|
--vmx <clone.vmx> --guest-os linux
|
|
```
|
|
|
|
Inside the guest, clone furyrs and run its `ci.yml` build command to completion.
|
|
|
|
- [ ] **Step 2: Record the five numbers that decide the design**
|
|
|
|
```bash
|
|
ssh -i /var/lib/ci/keys/ci_linux ci_build@<ip> '
|
|
du -sb /opt/ci/build/target # 1. target/ raw
|
|
time tar -C /opt/ci/build -c target | wc -c # 2. tar cost + size
|
|
time tar -C /opt/ci/build -c target | zstd -3 | wc -c # 3. zstd -3 cost + size
|
|
command -v zstd || echo "ZSTD ABSENT" # 4. is zstd even present?
|
|
nproc; free -m'
|
|
```
|
|
|
|
Then, host side, time the transfer of a file that size over SCP to the guest.
|
|
|
|
- [ ] **Step 3: Write the numbers down and state the go/no-go**
|
|
|
|
Record in `plans/cargo-dependency-cache-measurements.md`: raw size, pruned size,
|
|
compressed size, compress time, transfer time, extract time. The feature is
|
|
worth building if `restore + extract` is comfortably under the ~120s it saves.
|
|
If `zstd` is absent, note it — Task 3 falls back to `gzip` and Task 3 Step 7
|
|
adds `zstd` to the toolchain script for the next rebuild.
|
|
|
|
- [ ] **Step 4: Destroy the clone**
|
|
|
|
```bash
|
|
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator vm remove --vmx <clone.vmx>
|
|
```
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add plans/cargo-dependency-cache-measurements.md
|
|
git commit -m "docs(cache): measure the cargo target/ cache before building it"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 1: The store — key derivation and atomic publish
|
|
|
|
**Files:**
|
|
- Create: `src/ci_orchestrator/cargo_cache.py`
|
|
- Test: `tests/python/test_cargo_cache.py`
|
|
|
|
**Interfaces:**
|
|
- Consumes: nothing from earlier tasks.
|
|
- Produces, all used by Task 3 and Task 6:
|
|
- `CACHE_FORMAT: int` — bump to invalidate every entry at once.
|
|
- `cache_key(*, lock_digest: str, rustc_version: str, host_triple: str) -> str`
|
|
- `class CargoCacheStore` with
|
|
`__init__(self, root: Path, *, suffix: str = ".tar.zst") -> None`,
|
|
`path_for(self, key: str) -> Path`,
|
|
`lookup(self, key: str) -> Path | None`,
|
|
`publish(self, key: str, source: Path) -> Path`,
|
|
`prune_older_than(self, cutoff: datetime, *, what_if: bool = False) -> list[Path]`
|
|
|
|
- [ ] **Step 1: Write the failing tests**
|
|
|
|
```python
|
|
"""Tests for ci_orchestrator.cargo_cache — key derivation and the entry store."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from datetime import UTC, datetime, timedelta
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from ci_orchestrator.cargo_cache import CACHE_FORMAT, CargoCacheStore, cache_key
|
|
|
|
|
|
# ── key derivation ───────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_key_is_stable_for_identical_inputs() -> None:
|
|
a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64")
|
|
b = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64")
|
|
assert a == b
|
|
|
|
|
|
def test_key_changes_with_lock_digest() -> None:
|
|
a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64")
|
|
b = cache_key(lock_digest="def", rustc_version="rustc 1.97.1", host_triple="x86_64")
|
|
assert a != b
|
|
|
|
|
|
def test_key_changes_with_toolchain() -> None:
|
|
a = cache_key(lock_digest="abc", rustc_version="rustc 1.97.1", host_triple="x86_64")
|
|
b = cache_key(lock_digest="abc", rustc_version="rustc 1.98.0", host_triple="x86_64")
|
|
assert a != b
|
|
|
|
|
|
def test_key_is_filesystem_safe() -> None:
|
|
key = cache_key(
|
|
lock_digest="a/b+c=", rustc_version="rustc 1.97.1 (x /y)", host_triple="x86_64"
|
|
)
|
|
assert key.isalnum() or all(c.isalnum() or c == "-" for c in key)
|
|
assert "/" not in key
|
|
|
|
|
|
def test_key_carries_the_format_version() -> None:
|
|
key = cache_key(lock_digest="abc", rustc_version="r", host_triple="t")
|
|
assert key.startswith(f"v{CACHE_FORMAT}-")
|
|
|
|
|
|
# ── store ────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_lookup_returns_none_when_absent(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
assert store.lookup("v1-deadbeef") is None
|
|
|
|
|
|
def test_publish_then_lookup_finds_the_entry(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "incoming.tar.zst"
|
|
src.write_bytes(b"payload")
|
|
published = store.publish("v1-deadbeef", src)
|
|
assert store.lookup("v1-deadbeef") == published
|
|
assert published.read_bytes() == b"payload"
|
|
|
|
|
|
def test_publish_consumes_the_source_file(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "incoming.tar.zst"
|
|
src.write_bytes(b"payload")
|
|
store.publish("v1-deadbeef", src)
|
|
assert not src.exists(), "publish must move, not copy — a copy doubles peak disk"
|
|
|
|
|
|
def test_publish_overwrites_an_existing_key_atomically(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
first = tmp_path / "a.tar.zst"
|
|
first.write_bytes(b"first")
|
|
store.publish("v1-key", first)
|
|
second = tmp_path / "b.tar.zst"
|
|
second.write_bytes(b"second")
|
|
published = store.publish("v1-key", second)
|
|
assert published.read_bytes() == b"second"
|
|
|
|
|
|
def test_publish_leaves_no_temp_files_behind(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "incoming.tar.zst"
|
|
src.write_bytes(b"payload")
|
|
store.publish("v1-key", src)
|
|
leftovers = list((tmp_path / ".incoming").glob("*")) if (tmp_path / ".incoming").is_dir() else []
|
|
assert leftovers == []
|
|
|
|
|
|
def test_publish_missing_source_raises(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
with pytest.raises(FileNotFoundError):
|
|
store.publish("v1-key", tmp_path / "nope.tar.zst")
|
|
|
|
|
|
def test_lookup_rejects_a_key_with_a_path_separator(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
with pytest.raises(ValueError, match="unsafe cache key"):
|
|
store.lookup("../../etc/passwd")
|
|
|
|
|
|
# ── prune ────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def _age(path: Path, days: int) -> None:
|
|
old = (datetime.now(tz=UTC) - timedelta(days=days)).timestamp()
|
|
import os
|
|
|
|
os.utime(path, (old, old))
|
|
|
|
|
|
def test_prune_removes_entries_older_than_the_cutoff(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "old.tar.zst"
|
|
src.write_bytes(b"x")
|
|
entry = store.publish("v1-old", src)
|
|
_age(entry, days=40)
|
|
|
|
removed = store.prune_older_than(datetime.now(tz=UTC) - timedelta(days=30))
|
|
assert removed == [entry]
|
|
assert not entry.exists()
|
|
|
|
|
|
def test_prune_keeps_fresh_entries(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "new.tar.zst"
|
|
src.write_bytes(b"x")
|
|
entry = store.publish("v1-new", src)
|
|
|
|
removed = store.prune_older_than(datetime.now(tz=UTC) - timedelta(days=30))
|
|
assert removed == []
|
|
assert entry.exists()
|
|
|
|
|
|
def test_prune_what_if_reports_without_deleting(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path)
|
|
src = tmp_path / "old.tar.zst"
|
|
src.write_bytes(b"x")
|
|
entry = store.publish("v1-old", src)
|
|
_age(entry, days=40)
|
|
|
|
removed = store.prune_older_than(
|
|
datetime.now(tz=UTC) - timedelta(days=30), what_if=True
|
|
)
|
|
assert removed == [entry]
|
|
assert entry.exists()
|
|
|
|
|
|
def test_prune_on_a_missing_store_is_a_noop(tmp_path: Path) -> None:
|
|
store = CargoCacheStore(tmp_path / "never-created")
|
|
assert store.prune_older_than(datetime.now(tz=UTC)) == []
|
|
```
|
|
|
|
- [ ] **Step 2: Run the tests to verify they fail**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_cargo_cache.py -q`
|
|
Expected: collection error — `ModuleNotFoundError: No module named 'ci_orchestrator.cargo_cache'`
|
|
|
|
- [ ] **Step 3: Write the implementation**
|
|
|
|
```python
|
|
"""Content-addressed store for compiled Rust dependency artifacts.
|
|
|
|
A cache entry is one immutable file per key. Publishing is a ``rename(2)``
|
|
within the store directory, so a reader either sees the previous entry or the
|
|
new one and never a half-written file — no advisory locking, which is exactly
|
|
what a VMware shared folder cannot give us with several jobs in flight.
|
|
|
|
Nothing here knows about SSH or guests: it takes a finished tarball and files
|
|
it, or hands back the path of an existing one. That keeps the interesting
|
|
logic testable without a VM.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import os
|
|
import re
|
|
from datetime import UTC, datetime
|
|
from pathlib import Path
|
|
|
|
# Bump to invalidate every entry at once — e.g. when the prune script changes
|
|
# what it strips, making older entries untrustworthy.
|
|
CACHE_FORMAT = 1
|
|
|
|
_SAFE_KEY = re.compile(r"^v\d+-[0-9a-f]{32,64}$")
|
|
_INCOMING = ".incoming"
|
|
|
|
|
|
def cache_key(*, lock_digest: str, rustc_version: str, host_triple: str) -> str:
|
|
"""Derive the entry key from the inputs that make artifacts incompatible.
|
|
|
|
``lock_digest`` is the sha256 of ``Cargo.lock``: it changes exactly when
|
|
the dependency set changes, which is the property that makes this cache
|
|
sound. The feature set is deliberately *not* part of the key — cargo
|
|
encodes it in each artifact's own hash suffix, so several feature graphs
|
|
coexist inside one entry instead of fragmenting it.
|
|
"""
|
|
digest = hashlib.sha256(
|
|
"\0".join((lock_digest, rustc_version, host_triple)).encode("utf-8")
|
|
).hexdigest()
|
|
return f"v{CACHE_FORMAT}-{digest}"
|
|
|
|
|
|
class CargoCacheStore:
|
|
"""A directory of immutable cache entries, one file per key."""
|
|
|
|
def __init__(self, root: Path, *, suffix: str = ".tar.zst") -> None:
|
|
self.root = Path(root)
|
|
self.suffix = suffix
|
|
|
|
# ── paths ────────────────────────────────────────────────────────────
|
|
|
|
def _checked(self, key: str) -> str:
|
|
if not _SAFE_KEY.match(key):
|
|
raise ValueError(f"unsafe cache key: {key!r}")
|
|
return key
|
|
|
|
def path_for(self, key: str) -> Path:
|
|
return self.root / f"{self._checked(key)}{self.suffix}"
|
|
|
|
# ── read ─────────────────────────────────────────────────────────────
|
|
|
|
def lookup(self, key: str) -> Path | None:
|
|
"""Return the entry path for ``key``, or ``None`` on a miss."""
|
|
candidate = self.path_for(key)
|
|
return candidate if candidate.is_file() else None
|
|
|
|
# ── write ────────────────────────────────────────────────────────────
|
|
|
|
def publish(self, key: str, source: Path) -> Path:
|
|
"""Move ``source`` into the store under ``key``. Returns the entry path.
|
|
|
|
The move goes through a staging directory on the same filesystem so the
|
|
final step is a rename: concurrent publishers of the same key both
|
|
produce a valid entry and the last one wins, with no window in which a
|
|
reader can observe a partial file.
|
|
"""
|
|
source = Path(source)
|
|
if not source.is_file():
|
|
raise FileNotFoundError(f"nothing to publish: {source}")
|
|
target = self.path_for(key)
|
|
staging = self.root / _INCOMING
|
|
staging.mkdir(parents=True, exist_ok=True)
|
|
tmp = staging / f"{key}.{os.getpid()}{self.suffix}"
|
|
try:
|
|
os.replace(source, tmp)
|
|
except OSError:
|
|
# Different filesystem — fall back to a copy into staging, which
|
|
# still keeps the rename below atomic.
|
|
import shutil
|
|
|
|
shutil.copy2(source, tmp)
|
|
source.unlink(missing_ok=True)
|
|
os.replace(tmp, target)
|
|
return target
|
|
|
|
# ── maintenance ──────────────────────────────────────────────────────
|
|
|
|
def prune_older_than(
|
|
self, cutoff: datetime, *, what_if: bool = False
|
|
) -> list[Path]:
|
|
"""Delete entries last modified before ``cutoff``. Returns what went."""
|
|
if not self.root.is_dir():
|
|
return []
|
|
removed: list[Path] = []
|
|
for entry in sorted(self.root.glob(f"*{self.suffix}")):
|
|
if not entry.is_file():
|
|
continue
|
|
mtime = datetime.fromtimestamp(entry.stat().st_mtime, tz=UTC)
|
|
if mtime >= cutoff:
|
|
continue
|
|
removed.append(entry)
|
|
if not what_if:
|
|
entry.unlink(missing_ok=True)
|
|
return removed
|
|
|
|
|
|
__all__ = ["CACHE_FORMAT", "CargoCacheStore", "cache_key"]
|
|
```
|
|
|
|
- [ ] **Step 4: Run the tests to verify they pass**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_cargo_cache.py -q`
|
|
Expected: all pass.
|
|
|
|
- [ ] **Step 5: Lint and type-check**
|
|
|
|
Run: `.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src`
|
|
Expected: `All checks passed!` and `Success: no issues found`.
|
|
|
|
- [ ] **Step 6: Commit**
|
|
|
|
```bash
|
|
git add src/ci_orchestrator/cargo_cache.py tests/python/test_cargo_cache.py
|
|
git commit -m "feat(cache): content-addressed store for cargo dependency artifacts"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: The guest-side prune, and the assertion that makes it safe
|
|
|
|
**Files:**
|
|
- Create: `src/ci_orchestrator/assets/cargo-cache-prune.sh`
|
|
- Create: `src/ci_orchestrator/assets/__init__.py` (empty)
|
|
|
|
**Interfaces:**
|
|
- Consumes: nothing.
|
|
- Produces: a script invoked by Task 3 as
|
|
`bash cargo-cache-prune.sh <target-dir> <manifest-path>`, exit 0 on a clean
|
|
prune, exit 2 when a workspace artifact survived (caller must not publish),
|
|
exit 1 on a usage/environment error. Task 3 locates it with
|
|
`importlib.resources.as_file(resources.files("ci_orchestrator.assets").joinpath("cargo-cache-prune.sh"))`.
|
|
|
|
- [ ] **Step 0: Create the asset package**
|
|
|
|
```bash
|
|
mkdir -p src/ci_orchestrator/assets
|
|
touch src/ci_orchestrator/assets/__init__.py
|
|
```
|
|
|
|
`[tool.hatch.build.targets.wheel]` already declares `packages =
|
|
["src/ci_orchestrator"]`, so a sub-package with a data file is included in the
|
|
wheel without further configuration — the same route
|
|
`winimage/assets/autounattend.win2025.xml` takes.
|
|
|
|
- [ ] **Step 1: Write the script**
|
|
|
|
```bash
|
|
#!/usr/bin/env bash
|
|
# Strip every workspace-crate artifact out of target/, then prove none survived.
|
|
#
|
|
# The cache is only sound while it contains dependency artifacts alone. A
|
|
# workspace crate's artifact carries the code under test, and reusing one can
|
|
# turn a red build green -- the worst failure a CI system has, because it is
|
|
# silent. So this script ends by asserting its own work and exits non-zero if
|
|
# the assertion fails; the caller must treat that as "do not publish".
|
|
set -euo pipefail
|
|
|
|
TARGET_DIR="${1:?usage: cargo-cache-prune.sh <target-dir> <manifest-path>}"
|
|
MANIFEST="${2:?usage: cargo-cache-prune.sh <target-dir> <manifest-path>}"
|
|
|
|
[ -d "$TARGET_DIR" ] || { echo "prune: no such target dir: $TARGET_DIR" >&2; exit 1; }
|
|
command -v cargo >/dev/null || { echo "prune: cargo not on PATH" >&2; exit 1; }
|
|
|
|
# --no-deps restricts the listing to workspace members, which is precisely the
|
|
# set whose artifacts must never be cached.
|
|
members="$(cargo metadata --no-deps --format-version 1 --manifest-path "$MANIFEST" \
|
|
| python3 -c 'import json,sys; print("\n".join(p["name"] for p in json.load(sys.stdin)["packages"]))')"
|
|
|
|
if [ -z "$members" ]; then
|
|
echo "prune: cargo metadata listed no workspace members" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Incremental state is large, machine-specific and never reusable across
|
|
# clones; examples and docs are not dependencies of anything we build.
|
|
rm -rf "$TARGET_DIR"/*/incremental "$TARGET_DIR"/*/examples "$TARGET_DIR"/doc 2>/dev/null || true
|
|
|
|
prune_one() {
|
|
name="$1"
|
|
snake="${name//-/_}"
|
|
# Fingerprints and build-script output live in per-crate directories.
|
|
find "$TARGET_DIR" -maxdepth 3 \
|
|
\( -path "*/.fingerprint/$name-*" \
|
|
-o -path "*/.fingerprint/$snake-*" \
|
|
-o -path "*/build/$name-*" \
|
|
-o -path "*/build/$snake-*" \) \
|
|
-prune -exec rm -rf {} + 2>/dev/null || true
|
|
# Compiled outputs: rlib/rmeta/d in deps/, plus binaries at the profile root.
|
|
find "$TARGET_DIR" -maxdepth 3 \
|
|
\( -name "$snake-*" -o -name "lib$snake-*" -o -name "lib$snake.*" \
|
|
-o -name "$name" -o -name "$name-*" \) \
|
|
-exec rm -rf {} + 2>/dev/null || true
|
|
}
|
|
|
|
while IFS= read -r name; do
|
|
[ -n "$name" ] || continue
|
|
prune_one "$name"
|
|
done <<EOF
|
|
$members
|
|
EOF
|
|
|
|
# The assertion. A surviving fingerprint means cargo would consider that
|
|
# workspace crate already built and skip recompiling it on restore.
|
|
leaked=""
|
|
while IFS= read -r name; do
|
|
[ -n "$name" ] || continue
|
|
snake="${name//-/_}"
|
|
hit="$(find "$TARGET_DIR" -maxdepth 3 \
|
|
\( -path "*/.fingerprint/$name-*" -o -path "*/.fingerprint/$snake-*" \) \
|
|
-print -quit 2>/dev/null || true)"
|
|
[ -z "$hit" ] || leaked="$leaked $name"
|
|
done <<EOF
|
|
$members
|
|
EOF
|
|
|
|
if [ -n "$leaked" ]; then
|
|
echo "prune: REFUSING to publish -- workspace artifacts survived:$leaked" >&2
|
|
exit 2
|
|
fi
|
|
|
|
echo "prune: ok -- $(printf '%s\n' "$members" | wc -l) workspace member(s) stripped"
|
|
```
|
|
|
|
- [ ] **Step 2: Make it executable and shellcheck it**
|
|
|
|
```bash
|
|
chmod +x src/ci_orchestrator/assets/cargo-cache-prune.sh
|
|
shellcheck src/ci_orchestrator/assets/cargo-cache-prune.sh || true # advisory; SC2044 on find is expected
|
|
```
|
|
|
|
- [ ] **Step 3: Prove the assertion actually fires**
|
|
|
|
This is the one guard the whole feature's safety rests on, so exercise it
|
|
against a real cargo workspace rather than trusting the code reading. In a
|
|
LinuxRust2404 clone (reuse the Task 0 procedure):
|
|
|
|
```bash
|
|
# after a real build of the furyrs workspace:
|
|
bash cargo-cache-prune.sh /opt/ci/build/target /opt/ci/build/Cargo.toml
|
|
echo "exit=$?" # expect 0, and a "prune: ok" line
|
|
|
|
# now re-create one workspace fingerprint by hand and re-run
|
|
mkdir -p /opt/ci/build/target/debug/.fingerprint/fury-core-deadbeef
|
|
bash cargo-cache-prune.sh /opt/ci/build/target /opt/ci/build/Cargo.toml
|
|
echo "exit=$?" # expect 2, with "REFUSING to publish"
|
|
```
|
|
|
|
Both outcomes must be observed before this task is considered done. If the
|
|
second run exits 0, the prune glob is deleting the directory it is supposed to
|
|
detect — widen the assertion, do not narrow the test.
|
|
|
|
- [ ] **Step 4: Verify the dependency artifacts survived**
|
|
|
|
```bash
|
|
ls /opt/ci/build/target/*/deps/ | head # expect many libserde-*, libtokio-* etc.
|
|
du -sh /opt/ci/build/target # record; compare against Task 0's raw size
|
|
```
|
|
|
|
Expected: dependency `.rlib`/`.rmeta` files are still present. A prune that
|
|
emptied `deps/` is a bug — record the pruned size in
|
|
`plans/cargo-dependency-cache-measurements.md`.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/ci_orchestrator/assets/ plans/cargo-dependency-cache-measurements.md
|
|
git commit -m "feat(cache): guest-side prune of workspace artifacts, with a fail-closed assert"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Restore before the build, publish after it
|
|
|
|
**Files:**
|
|
- Modify: `src/ci_orchestrator/commands/build.py` (`_linux_build` only — leave `_windows_build` alone)
|
|
- Test: `tests/python/test_commands_build.py`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `CargoCacheStore`, `cache_key`, `CACHE_FORMAT` from Task 1;
|
|
`ci_orchestrator/assets/cargo-cache-prune.sh` from Task 2.
|
|
- Produces: `_linux_build(..., cargo_cache_root: Path | None = None)` — `None`
|
|
disables the feature entirely, which is the default so no existing caller
|
|
changes behaviour.
|
|
|
|
- [ ] **Step 1: Write the failing tests**
|
|
|
|
Append to `tests/python/test_commands_build.py`. `FakeSsh` already exists in
|
|
that file; if its recorded commands are not exposed as `.commands`, add that
|
|
list first in the same style as the existing fake.
|
|
|
|
```python
|
|
# ── cargo dependency cache ───────────────────────────────────────────────────
|
|
|
|
|
|
def test_linux_build_without_cache_root_runs_no_cache_commands(monkeypatch, tmp_path):
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git"))
|
|
assert not any("cargo-cache" in c for c in fake.commands)
|
|
|
|
|
|
def test_linux_build_skips_cache_when_no_cargo_lock(monkeypatch, tmp_path):
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
fake.result_for("test -f", returncode=1) # no Cargo.lock in the workdir
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git",
|
|
cargo_cache_root=tmp_path))
|
|
assert not any("cargo-cache-prune" in c for c in fake.commands)
|
|
|
|
|
|
def test_linux_build_restores_an_existing_entry(monkeypatch, tmp_path):
|
|
from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key
|
|
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
fake.result_for("sha256sum", stdout="deadbeef Cargo.lock\n")
|
|
fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n")
|
|
fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n")
|
|
|
|
store = CargoCacheStore(tmp_path)
|
|
key = cache_key(lock_digest="deadbeef",
|
|
rustc_version="rustc 1.97.1 (abc 2026-01-01)",
|
|
host_triple="x86_64-unknown-linux-gnu")
|
|
seed = tmp_path / "seed.tar.zst"
|
|
seed.write_bytes(b"entry")
|
|
store.publish(key, seed)
|
|
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git",
|
|
cargo_cache_root=tmp_path))
|
|
assert any("tar" in c and "target" in c for c in fake.commands)
|
|
assert fake.copied, "a cache hit must upload the entry to the guest"
|
|
|
|
|
|
def test_linux_build_publishes_on_a_miss(monkeypatch, tmp_path):
|
|
from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key
|
|
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
fake.result_for("sha256sum", stdout="cafe Cargo.lock\n")
|
|
fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n")
|
|
fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n")
|
|
fake.fetch_writes = b"fresh-entry"
|
|
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git",
|
|
cargo_cache_root=tmp_path))
|
|
|
|
key = cache_key(lock_digest="cafe",
|
|
rustc_version="rustc 1.97.1 (abc 2026-01-01)",
|
|
host_triple="x86_64-unknown-linux-gnu")
|
|
assert CargoCacheStore(tmp_path).lookup(key) is not None
|
|
|
|
|
|
def test_linux_build_does_not_publish_when_the_prune_assertion_fails(monkeypatch, tmp_path):
|
|
from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key
|
|
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
fake.result_for("sha256sum", stdout="cafe Cargo.lock\n")
|
|
fake.result_for("rustc -V", stdout="rustc 1.97.1 (abc 2026-01-01)\n")
|
|
fake.result_for("rustc -vV", stdout="host: x86_64-unknown-linux-gnu\n")
|
|
fake.result_for("cargo-cache-prune", returncode=2,
|
|
stderr="prune: REFUSING to publish -- workspace artifacts survived: fury-core")
|
|
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git",
|
|
cargo_cache_root=tmp_path))
|
|
|
|
key = cache_key(lock_digest="cafe",
|
|
rustc_version="rustc 1.97.1 (abc 2026-01-01)",
|
|
host_triple="x86_64-unknown-linux-gnu")
|
|
assert CargoCacheStore(tmp_path).lookup(key) is None, (
|
|
"a failed prune assertion must never reach the store"
|
|
)
|
|
|
|
|
|
def test_cache_failure_does_not_fail_the_build(monkeypatch, tmp_path):
|
|
fake = _install_fake_ssh(monkeypatch)
|
|
fake.result_for("sha256sum", returncode=1, stderr="boom")
|
|
# Must not raise: the cache is an optimisation, never a failure mode.
|
|
_linux_build(**_linux_kwargs(clone_url="https://example/r.git",
|
|
cargo_cache_root=tmp_path))
|
|
```
|
|
|
|
- [ ] **Step 2: Run the tests to verify they fail**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_commands_build.py -q -k cargo_cache`
|
|
Expected: FAIL — `_linux_build() got an unexpected keyword argument 'cargo_cache_root'`
|
|
|
|
- [ ] **Step 3: Add the cache helpers to `build.py`**
|
|
|
|
Insert above `_linux_build`:
|
|
|
|
```python
|
|
_PRUNE_ASSET = "cargo-cache-prune.sh"
|
|
_GUEST_PRUNE = "/tmp/cargo-cache-prune.sh"
|
|
_GUEST_TARBALL = "/tmp/cargo-cache-entry.tar"
|
|
|
|
|
|
def _cargo_cache_probe(t: SshTransport, workdir: str) -> tuple[str, str] | None:
|
|
"""Return ``(key, compressor)`` for this workspace, or ``None`` to skip.
|
|
|
|
``None`` means "not a Rust build, or we could not read what we need" — the
|
|
cache is an optimisation and every uncertainty here resolves to skipping it.
|
|
"""
|
|
if t.run(f"test -f {_sh_quote(workdir + '/Cargo.lock')}", check=False).returncode != 0:
|
|
return None
|
|
lock = t.run(f"sha256sum {_sh_quote(workdir + '/Cargo.lock')}", check=False)
|
|
version = t.run("rustc -V", check=False)
|
|
triple = t.run("rustc -vV | sed -n 's/^host: //p'", check=False)
|
|
if not (lock.ok and version.ok and triple.ok):
|
|
click.echo("[cache] could not read Cargo.lock digest or rustc version; skipping")
|
|
return None
|
|
compressor = "zstd" if t.run("command -v zstd", check=False).ok else "gzip"
|
|
key = cache_key(
|
|
lock_digest=lock.stdout.split()[0],
|
|
rustc_version=version.stdout.strip(),
|
|
host_triple=triple.stdout.strip(),
|
|
)
|
|
return key, compressor
|
|
|
|
|
|
def _cargo_cache_restore(
|
|
t: SshTransport, store: CargoCacheStore, key: str, workdir: str, compressor: str
|
|
) -> bool:
|
|
"""Upload and unpack an existing entry. Returns True on a hit."""
|
|
entry = store.lookup(key)
|
|
if entry is None:
|
|
click.echo(f"[cache] miss for {key} — building cold")
|
|
return False
|
|
size_mb = entry.stat().st_size / 1_048_576
|
|
click.echo(f"[cache] hit for {key} ({size_mb:.0f} MB) — restoring")
|
|
guest_entry = f"{_GUEST_TARBALL}.{'zst' if compressor == 'zstd' else 'gz'}"
|
|
t.copy(str(entry), guest_entry)
|
|
flag = "--zstd" if compressor == "zstd" else "-z"
|
|
t.run(
|
|
f"tar {flag} -xf {_sh_quote(guest_entry)} -C {_sh_quote(workdir)} && "
|
|
f"rm -f {_sh_quote(guest_entry)}"
|
|
)
|
|
return True
|
|
|
|
|
|
def _cargo_cache_publish(
|
|
t: SshTransport, store: CargoCacheStore, key: str, workdir: str, compressor: str
|
|
) -> None:
|
|
"""Prune workspace artifacts, then file the remainder under ``key``.
|
|
|
|
A non-zero prune means the fail-closed assertion tripped: say so loudly and
|
|
publish nothing. A poisoned entry would be far more expensive than a miss.
|
|
"""
|
|
# as_file() because the production venv is non-editable: the script is
|
|
# package data, not a path relative to a source checkout.
|
|
asset = resources.files("ci_orchestrator.assets").joinpath(_PRUNE_ASSET)
|
|
with resources.as_file(asset) as prune_script:
|
|
t.copy(str(prune_script), _GUEST_PRUNE)
|
|
pruned = t.run(
|
|
f"bash {_GUEST_PRUNE} {_sh_quote(workdir + '/target')} "
|
|
f"{_sh_quote(workdir + '/Cargo.toml')}",
|
|
check=False,
|
|
)
|
|
click.echo(f"[cache] {pruned.stdout.strip() or pruned.stderr.strip()}")
|
|
if pruned.returncode != 0:
|
|
click.echo("[cache] prune refused — publishing nothing", err=True)
|
|
return
|
|
|
|
ext = "zst" if compressor == "zstd" else "gz"
|
|
flag = "--zstd" if compressor == "zstd" else "-z"
|
|
guest_tar = f"{_GUEST_TARBALL}.{ext}"
|
|
packed = t.run(
|
|
f"tar {flag} -cf {_sh_quote(guest_tar)} -C {_sh_quote(workdir)} target",
|
|
check=False,
|
|
)
|
|
if not packed.ok:
|
|
click.echo("[cache] packing failed — publishing nothing", err=True)
|
|
return
|
|
|
|
with tempfile.TemporaryDirectory() as td:
|
|
local = Path(td) / f"entry.tar.{ext}"
|
|
t.fetch(guest_tar, local)
|
|
published = store.publish(key, local)
|
|
click.echo(
|
|
f"[cache] published {key} "
|
|
f"({published.stat().st_size / 1_048_576:.0f} MB)"
|
|
)
|
|
```
|
|
|
|
Add to the imports at the top of `build.py`:
|
|
|
|
```python
|
|
from importlib import resources
|
|
|
|
from ci_orchestrator.cargo_cache import CargoCacheStore, cache_key
|
|
```
|
|
|
|
- [ ] **Step 4: Call them from `_linux_build`**
|
|
|
|
Add the parameter to the signature (keyword-only, defaulted, so no existing
|
|
caller changes):
|
|
|
|
```python
|
|
use_xvfb: bool = False,
|
|
cargo_cache_root: Path | None = None,
|
|
) -> None:
|
|
```
|
|
|
|
Immediately **after** the source-mtime normalisation `t.run(find ... touch ...)`
|
|
and **before** `env_prefix = ...`, insert:
|
|
|
|
```python
|
|
# Cargo dependency cache. Everything here is best-effort: a cache
|
|
# problem must never turn a good build red, so failures degrade to a
|
|
# cold build with a message.
|
|
cache_key_value: str | None = None
|
|
compressor = "gzip"
|
|
store: CargoCacheStore | None = None
|
|
if cargo_cache_root is not None:
|
|
try:
|
|
store = CargoCacheStore(cargo_cache_root)
|
|
probed = _cargo_cache_probe(t, workdir)
|
|
if probed is not None:
|
|
cache_key_value, compressor = probed
|
|
_cargo_cache_restore(t, store, cache_key_value, workdir, compressor)
|
|
except (TransportError, OSError, ValueError) as exc:
|
|
click.echo(f"[cache] restore skipped: {exc}", err=True)
|
|
cache_key_value = None
|
|
```
|
|
|
|
Immediately **after** the build-failure check (`if result.returncode != 0: raise ...`)
|
|
and **before** the `if skip_artifact:` block, insert:
|
|
|
|
```python
|
|
# Only publish on a miss: in the steady state this uploads nothing.
|
|
if store is not None and cache_key_value and store.lookup(cache_key_value) is None:
|
|
try:
|
|
_cargo_cache_publish(t, store, cache_key_value, workdir, compressor)
|
|
except (TransportError, OSError, ValueError) as exc:
|
|
click.echo(f"[cache] publish skipped: {exc}", err=True)
|
|
```
|
|
|
|
- [ ] **Step 5: Run the tests to verify they pass**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_commands_build.py -q`
|
|
Expected: all pass, including the pre-existing tests (the Windows branch and
|
|
every non-Rust Linux path must be untouched).
|
|
|
|
- [ ] **Step 6: Lint, type-check, full suite with the coverage gate**
|
|
|
|
Run:
|
|
```bash
|
|
.venv/bin/python -m ruff check src tests/python
|
|
.venv/bin/python -m mypy --strict src
|
|
.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90
|
|
```
|
|
Expected: clean, clean, and coverage ≥90%.
|
|
|
|
- [ ] **Step 7: If Task 0 found `zstd` absent, add it to the toolchain script**
|
|
|
|
Only if Step 4 of Task 0 recorded `ZSTD ABSENT`. In
|
|
`template/Install-CIToolchain-LinuxRust2404.sh`, add `zstd` to the apt package
|
|
list, next to the existing build tooling. It takes effect at the next template
|
|
rebuild; until then the `gzip` fallback above carries the feature.
|
|
|
|
- [ ] **Step 8: Commit**
|
|
|
|
```bash
|
|
git add src/ci_orchestrator/commands/build.py tests/python/test_commands_build.py \
|
|
template/Install-CIToolchain-LinuxRust2404.sh
|
|
git commit -m "feat(cache): restore and publish the cargo dependency cache around Linux builds"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: CLI flag and config section
|
|
|
|
**Files:**
|
|
- Modify: `src/ci_orchestrator/config.py`
|
|
- Modify: `src/ci_orchestrator/commands/job.py`
|
|
- Modify: `config.example.toml`
|
|
- Test: `tests/python/test_config.py`, `tests/python/test_commands_job.py`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `_linux_build(..., cargo_cache_root=...)` from Task 3.
|
|
- Produces: `Config.cargo_cache: CargoCacheConfig` with fields
|
|
`enabled: bool = True`, `root: Path` (default `<paths.root>/cache/cargo`),
|
|
`retention_days: int = 30`; and `ci job --cargo-cache/--no-cargo-cache`.
|
|
|
|
- [ ] **Step 1: Write the failing config test**
|
|
|
|
Append to `tests/python/test_config.py`:
|
|
|
|
```python
|
|
def test_cargo_cache_defaults_under_paths_root(tmp_path: Path) -> None:
|
|
cfg_file = tmp_path / "config.toml"
|
|
cfg_file.write_text(
|
|
'[paths]\nroot = "/var/lib/ci"\ntemplates = "/var/lib/ci/templates"\n'
|
|
'build_vms = "/var/lib/ci/build-vms"\nartifacts = "/var/lib/ci/artifacts"\n'
|
|
'keys = "/var/lib/ci/keys"\n',
|
|
encoding="utf-8",
|
|
)
|
|
cfg = load_config(config_path=cfg_file, env={})
|
|
assert cfg.cargo_cache.enabled is True
|
|
assert cfg.cargo_cache.root == Path("/var/lib/ci/cache/cargo")
|
|
|
|
|
|
def test_cargo_cache_section_overrides_the_defaults(tmp_path: Path) -> None:
|
|
cfg_file = tmp_path / "config.toml"
|
|
cfg_file.write_text(
|
|
'[paths]\nroot = "/var/lib/ci"\ntemplates = "/var/lib/ci/templates"\n'
|
|
'build_vms = "/var/lib/ci/build-vms"\nartifacts = "/var/lib/ci/artifacts"\n'
|
|
'keys = "/var/lib/ci/keys"\n'
|
|
'\n[cargo_cache]\nenabled = false\nroot = "/mnt/fast/cargo"\n'
|
|
"retention_days = 7\n",
|
|
encoding="utf-8",
|
|
)
|
|
cfg = load_config(config_path=cfg_file, env={})
|
|
assert cfg.cargo_cache.enabled is False
|
|
assert cfg.cargo_cache.root == Path("/mnt/fast/cargo")
|
|
assert cfg.cargo_cache.retention_days == 7
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to verify it fails**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_config.py -q -k cargo_cache`
|
|
Expected: FAIL — `AttributeError: 'Config' object has no attribute 'cargo_cache'`
|
|
|
|
- [ ] **Step 3: Add the dataclass and loader wiring**
|
|
|
|
In `config.py`, beside the other section dataclasses:
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class CargoCacheConfig:
|
|
"""Where compiled Rust dependency artifacts are filed between jobs."""
|
|
|
|
enabled: bool = True
|
|
root: Path = Path("/var/lib/ci/cache/cargo")
|
|
retention_days: int = 30
|
|
```
|
|
|
|
Add the field to `Config` (after `ip_pool`, so no positional call breaks):
|
|
|
|
```python
|
|
cargo_cache: CargoCacheConfig = field(default_factory=CargoCacheConfig)
|
|
```
|
|
|
|
In `load_config`, after the `ip_pool` block:
|
|
|
|
```python
|
|
cc_raw = data.get("cargo_cache", {}) if isinstance(data, dict) else {}
|
|
cargo_cache = CargoCacheConfig(
|
|
enabled=bool(cc_raw.get("enabled", True)),
|
|
root=Path(cc_raw.get("root", paths.root / "cache" / "cargo")),
|
|
retention_days=int(cc_raw.get("retention_days", 30)),
|
|
)
|
|
```
|
|
|
|
and pass `cargo_cache=cargo_cache` into the `Config(...)` construction.
|
|
|
|
- [ ] **Step 4: Add the CLI flag to `job.py`**
|
|
|
|
Add the option beside `--use-shared-cache`:
|
|
|
|
```python
|
|
@click.option(
|
|
"--cargo-cache/--no-cargo-cache",
|
|
"cargo_cache",
|
|
default=True,
|
|
show_default=True,
|
|
help="Reuse compiled Rust dependency artifacts between jobs (Linux only; "
|
|
"engages only when the workspace has a Cargo.lock).",
|
|
)
|
|
```
|
|
|
|
Add `cargo_cache: bool,` to the `job(...)` signature, and in the Linux branch
|
|
of Phase 4 pass it down:
|
|
|
|
```python
|
|
use_xvfb=use_xvfb,
|
|
cargo_cache_root=(
|
|
config.cargo_cache.root
|
|
if cargo_cache and config.cargo_cache.enabled
|
|
else None
|
|
),
|
|
```
|
|
|
|
- [ ] **Step 5: Write the failing job test, then run the suite**
|
|
|
|
Append to `tests/python/test_commands_job.py`, in the style of the existing
|
|
`_linux_build` call-capture tests:
|
|
|
|
```python
|
|
def test_job_passes_cache_root_to_linux_build(monkeypatch, tmp_path):
|
|
captured = _capture_linux_build(monkeypatch)
|
|
_run_job(monkeypatch, tmp_path, extra_args=[])
|
|
assert captured["cargo_cache_root"] is not None
|
|
|
|
|
|
def test_job_no_cargo_cache_disables_it(monkeypatch, tmp_path):
|
|
captured = _capture_linux_build(monkeypatch)
|
|
_run_job(monkeypatch, tmp_path, extra_args=["--no-cargo-cache"])
|
|
assert captured["cargo_cache_root"] is None
|
|
```
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90`
|
|
Expected: all pass, coverage ≥90%.
|
|
|
|
- [ ] **Step 6: Document the section in `config.example.toml`**
|
|
|
|
```toml
|
|
# Compiled Rust dependency artifacts, reused between jobs. Engages only for
|
|
# Linux guests whose build workdir contains a Cargo.lock; workspace crates are
|
|
# stripped before an entry is stored, so a cached entry can never carry the
|
|
# code under test.
|
|
[cargo_cache]
|
|
enabled = true
|
|
root = "/var/lib/ci/cache/cargo"
|
|
retention_days = 30
|
|
```
|
|
|
|
- [ ] **Step 7: Lint, type-check, commit**
|
|
|
|
```bash
|
|
.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src
|
|
git add src/ci_orchestrator/config.py src/ci_orchestrator/commands/job.py \
|
|
config.example.toml tests/python/test_config.py tests/python/test_commands_job.py
|
|
git commit -m "feat(cache): --cargo-cache flag and [cargo_cache] config section"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Action input, and cold builds on release tags
|
|
|
|
**Files:**
|
|
- Modify: `.gitea/actions/local-ci-build/action.yml`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `ci job --cargo-cache/--no-cargo-cache` from Task 4.
|
|
- Produces: action input `cargo-cache` with values `auto` (default), `on`, `off`.
|
|
|
|
- [ ] **Step 1: Add the input**
|
|
|
|
In the `inputs:` block, after `use-shared-cache`:
|
|
|
|
```yaml
|
|
cargo-cache:
|
|
description: >
|
|
Reuse compiled Rust dependency artifacts between jobs: 'auto' (default)
|
|
caches on branches and pull requests but builds release tags (refs/tags/v*)
|
|
cold, 'on' always caches, 'off' never does. Only ever engages for Linux
|
|
guests whose workspace has a Cargo.lock. Workspace crates are stripped
|
|
from an entry before it is stored, so a restored cache cannot carry the
|
|
code under test -- but a release build verifies an artifact rather than a
|
|
change, so 'auto' gives it the clean room.
|
|
required: false
|
|
default: 'auto'
|
|
```
|
|
|
|
- [ ] **Step 2: Pass it into the env block**
|
|
|
|
Beside `INPUT_USE_SHARED_CACHE`:
|
|
|
|
```yaml
|
|
INPUT_CARGO_CACHE: ${{ inputs.cargo-cache }}
|
|
GITHUB_REF_FOR_CACHE: ${{ github.ref }}
|
|
```
|
|
|
|
- [ ] **Step 3: Resolve it next to the other input resolution**
|
|
|
|
In the same PowerShell step that resolves `$resolvedSnapshotName`:
|
|
|
|
```powershell
|
|
# Release tags build cold: they verify a release artifact, not a
|
|
# change, so the clean room is worth the two minutes (furyrs#13).
|
|
$cargoCacheMode = if ($env:INPUT_CARGO_CACHE) { $env:INPUT_CARGO_CACHE } else { 'auto' }
|
|
switch ($cargoCacheMode) {
|
|
'on' { $useCargoCache = $true }
|
|
'off' { $useCargoCache = $false }
|
|
'auto' {
|
|
$useCargoCache = -not ($env:GITHUB_REF_FOR_CACHE -like 'refs/tags/v*')
|
|
}
|
|
default {
|
|
throw "cargo-cache must be one of 'auto', 'on', 'off' — got '$cargoCacheMode'."
|
|
}
|
|
}
|
|
Write-Host "[job] Cargo cache: $(if ($useCargoCache) { 'on' } else { 'off (cold)' })"
|
|
```
|
|
|
|
- [ ] **Step 4: Add the flag to the argument list**
|
|
|
|
Where the other `ci job` switches are appended:
|
|
|
|
```powershell
|
|
if (-not $useCargoCache) { $jobArgs += '--no-cargo-cache' }
|
|
```
|
|
|
|
- [ ] **Step 5: Verify the action parses and the branches behave**
|
|
|
|
```bash
|
|
.venv/bin/python -c "import yaml,sys; yaml.safe_load(open('.gitea/actions/local-ci-build/action.yml')); print('yaml ok')"
|
|
pwsh -NoProfile -Command "
|
|
\$env:INPUT_CARGO_CACHE='auto'; \$env:GITHUB_REF_FOR_CACHE='refs/tags/v1.2.3'
|
|
# paste the switch block; expect: off (cold)
|
|
"
|
|
```
|
|
|
|
Expected: `yaml ok`, and the `auto` + tag combination selects the cold path.
|
|
If `pwsh` is not installed on this host (Phase C removed it), verify on the
|
|
Windows runner or by reading the branch — but record which you did.
|
|
|
|
- [ ] **Step 6: Commit, then publish — the action only runs from the server**
|
|
|
|
```bash
|
|
git add .gitea/actions/local-ci-build/action.yml
|
|
git commit -m "feat(action): cargo-cache input, cold on release tags"
|
|
git push origin main
|
|
/opt/ci/venv/bin/python tools/ci-action-published.py
|
|
```
|
|
|
|
Expected: `OK — .gitea/ matches origin/main`. Committing is not deploying for
|
|
anything under `.gitea/`; this check is the proof.
|
|
|
|
---
|
|
|
|
### Task 6: Retention — the store must not grow without bound
|
|
|
|
**Files:**
|
|
- Modify: `src/ci_orchestrator/commands/retention.py`
|
|
- Test: `tests/python/test_commands_retention.py`
|
|
|
|
**Interfaces:**
|
|
- Consumes: `CargoCacheStore.prune_older_than` from Task 1,
|
|
`Config.cargo_cache.retention_days` from Task 4.
|
|
|
|
TODO §2.9 records that `retention run` already warns about low disk while
|
|
freeing 0.0 GB, because it only governs artifacts, logs and IP leases. Adding a
|
|
new unbounded directory without adding it to retention repeats that mistake at
|
|
a larger scale — an entry is hundreds of megabytes and one is minted per
|
|
`Cargo.lock` change.
|
|
|
|
- [ ] **Step 1: Write the failing test**
|
|
|
|
```python
|
|
def test_retention_purges_stale_cargo_cache_entries(tmp_path, monkeypatch):
|
|
import os
|
|
from datetime import UTC, datetime, timedelta
|
|
|
|
from ci_orchestrator.cargo_cache import CargoCacheStore
|
|
|
|
store_root = tmp_path / "cache" / "cargo"
|
|
store_root.mkdir(parents=True)
|
|
old = store_root / ("v1-" + "a" * 40 + ".tar.zst")
|
|
old.write_bytes(b"x")
|
|
stamp = (datetime.now(tz=UTC) - timedelta(days=90)).timestamp()
|
|
os.utime(old, (stamp, stamp))
|
|
|
|
_run_retention(monkeypatch, tmp_path, retention_days=30)
|
|
assert not old.exists()
|
|
|
|
|
|
def test_retention_what_if_keeps_cargo_cache_entries(tmp_path, monkeypatch):
|
|
import os
|
|
from datetime import UTC, datetime, timedelta
|
|
|
|
store_root = tmp_path / "cache" / "cargo"
|
|
store_root.mkdir(parents=True)
|
|
old = store_root / ("v1-" + "a" * 40 + ".tar.zst")
|
|
old.write_bytes(b"x")
|
|
stamp = (datetime.now(tz=UTC) - timedelta(days=90)).timestamp()
|
|
os.utime(old, (stamp, stamp))
|
|
|
|
_run_retention(monkeypatch, tmp_path, retention_days=30, what_if=True)
|
|
assert old.exists()
|
|
```
|
|
|
|
- [ ] **Step 2: Run it to verify it fails**
|
|
|
|
Run: `.venv/bin/python -m pytest tests/python/test_commands_retention.py -q -k cargo`
|
|
Expected: FAIL — the entry still exists.
|
|
|
|
- [ ] **Step 3: Implement**
|
|
|
|
In `retention_run`, after the existing artifact and log purges:
|
|
|
|
```python
|
|
if config.cargo_cache.enabled:
|
|
cache_cutoff = datetime.now(tz=UTC) - timedelta(
|
|
days=min(effective_days, config.cargo_cache.retention_days)
|
|
)
|
|
removed = CargoCacheStore(config.cargo_cache.root).prune_older_than(
|
|
cache_cutoff, what_if=what_if
|
|
)
|
|
freed_mb = sum(p.stat().st_size for p in removed if p.exists()) / 1_048_576
|
|
verb = "WhatIf: would purge" if what_if else "Purged"
|
|
click.echo(
|
|
f"[RetentionPolicy] {verb} {len(removed)} cargo cache entr"
|
|
f"{'y' if len(removed) == 1 else 'ies'}"
|
|
+ (f" ({freed_mb:.0f} MB)" if what_if else "")
|
|
)
|
|
```
|
|
|
|
Add the import: `from ci_orchestrator.cargo_cache import CargoCacheStore`.
|
|
|
|
- [ ] **Step 4: Run tests, lint, type-check**
|
|
|
|
```bash
|
|
.venv/bin/python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90
|
|
.venv/bin/python -m ruff check src tests/python && .venv/bin/python -m mypy --strict src
|
|
```
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add src/ci_orchestrator/commands/retention.py tests/python/test_commands_retention.py
|
|
git commit -m "feat(cache): purge stale cargo cache entries in retention run"
|
|
```
|
|
|
|
---
|
|
|
|
### Task 7: End-to-end validation on the real workspace, and report the numbers
|
|
|
|
**Files:**
|
|
- Modify: `plans/cargo-dependency-cache-measurements.md`
|
|
- Modify: `docs/RUNBOOK-PhaseC.md`
|
|
- Modify: `TODO.md`
|
|
|
|
- [ ] **Step 1: Cold run — populates the store**
|
|
|
|
The build command is the `build-command:` block of
|
|
`/home/simone/_Workspace/furyrs/.gitea/workflows/ci.yml` (line 60 onward) —
|
|
copy it verbatim into a file rather than retyping it, because it carries the
|
|
`/etc/profile.d/rust.sh` sourcing and the `--locked` flags that make the run
|
|
representative:
|
|
|
|
```bash
|
|
sed -n '/^ build-command: |/,/^ [a-z-]*:/p' \
|
|
/home/simone/_Workspace/furyrs/.gitea/workflows/ci.yml \
|
|
| sed '1d;$d;s/^ //' > /tmp/furyrs-build.sh
|
|
|
|
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator job \
|
|
--job-id cache-e2e-cold \
|
|
--repo-url https://gitea.emulab.it/Simone/furyrs.git --branch main \
|
|
--template-path /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx \
|
|
--snapshot-name BaseClean-LinuxRust --guest-os linux \
|
|
--build-command "$(cat /tmp/furyrs-build.sh)" --skip-artifact
|
|
```
|
|
|
|
At the time of writing that block runs, in order: `cargo fmt --all -- --check`,
|
|
`cargo test --workspace --locked`, `cargo test -p fury-core --features
|
|
ddr5-scaffold --locked`, `cargo clippy -p fury-cli --all-targets --features
|
|
direct-backend --locked -- -D warnings`, `cargo clippy --workspace
|
|
--all-targets --locked -- -D warnings`, `cargo clippy -p fury-gui --features
|
|
gui --bin fury-gui --locked -- -D warnings`, and `cargo audit`. Five of those
|
|
compile, across four feature configurations — default, `ddr5-scaffold`,
|
|
`direct-backend`, `gui` — which is exactly the case the single-entry design has
|
|
to handle.
|
|
|
|
Expected in the log: `[cache] miss for v1-… — building cold`, then
|
|
`[cache] prune: ok — N workspace member(s) stripped`, then
|
|
`[cache] published v1-… (NNN MB)`. Record the total job time.
|
|
|
|
- [ ] **Step 2: Warm run — same commit**
|
|
|
|
Re-run the identical command with `--job-id cache-e2e-warm`.
|
|
|
|
Expected: `[cache] hit for v1-… (NNN MB) — restoring`, no `published` line
|
|
(publish happens on a miss only), and a materially shorter build phase. Record
|
|
the total.
|
|
|
|
- [ ] **Step 3: The run that actually matters — warm, with modified source**
|
|
|
|
Push a trivial source change to a scratch branch of furyrs and run against it.
|
|
This is the real CI scenario; #12 measured 6.8s for the build step here.
|
|
|
|
Expected: cache hit, workspace crates recompiled, dependencies not.
|
|
|
|
- [ ] **Step 4: Prove the cache cannot hide a regression**
|
|
|
|
Introduce a deliberate compile error in a workspace crate on a scratch branch
|
|
and run warm.
|
|
|
|
Expected: **the build fails.** If it passes, stop and do not ship: the prune is
|
|
leaking workspace artifacts and the assertion in Task 2 is not catching it.
|
|
This single test is what the whole safety argument reduces to.
|
|
|
|
- [ ] **Step 5: Confirm the release-tag path is cold**
|
|
|
|
Run once with `--no-cargo-cache` and confirm the log shows a cold build and no
|
|
`[cache] hit` line.
|
|
|
|
- [ ] **Step 6: Write the numbers into the measurements file**
|
|
|
|
Cold total, warm-identical total, warm-modified total, entry size on disk,
|
|
restore time, publish time. State the net saving against the 133s baseline from
|
|
#12/#13.
|
|
|
|
- [ ] **Step 7: Document for the operator**
|
|
|
|
Add to `docs/RUNBOOK-PhaseC.md` §6 (Maintenance): where the store lives
|
|
(`/var/lib/ci/cache/cargo`), how to read `[cache] hit`/`miss`/`REFUSING`, how to
|
|
invalidate everything (bump `CACHE_FORMAT`), and how to purge by hand
|
|
(`rm /var/lib/ci/cache/cargo/*.tar.zst` — entries are immutable, deleting one is
|
|
always safe and costs one cold build).
|
|
|
|
- [ ] **Step 8: Close §7 in TODO.md and the thread**
|
|
|
|
Mark the cargo-cache entry done in `TODO.md` with the measured numbers, then
|
|
append a `CI → furyrs` entry to
|
|
`/home/simone/_Workspace/furyrs/docs/ci-rust-integration.md` closing #7: what
|
|
was built, the deviation from #12's cargo-chef shape and why, the measured
|
|
saving, that release tags are cold as they asked, and that `cargo-cache: 'off'`
|
|
is available as a per-workflow override.
|
|
|
|
- [ ] **Step 9: Commit and publish**
|
|
|
|
```bash
|
|
git add plans/cargo-dependency-cache-measurements.md docs/RUNBOOK-PhaseC.md TODO.md
|
|
git commit -m "docs(cache): end-to-end numbers for the cargo dependency cache"
|
|
git push origin main
|
|
/opt/ci/venv/bin/python tools/ci-action-published.py
|
|
```
|
|
|
|
---
|
|
|
|
## Rollback
|
|
|
|
Every piece is opt-out at a different level, cheapest first:
|
|
|
|
1. Per workflow: `cargo-cache: 'off'` in the composite action inputs.
|
|
2. Per host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`.
|
|
3. Everything at once: bump `CACHE_FORMAT` in `cargo_cache.py` — every existing
|
|
entry becomes unreachable and every job goes cold, with no file to delete.
|
|
4. Disk: `rm /var/lib/ci/cache/cargo/*.tar.zst`. Entries are immutable and
|
|
independent; deleting any subset costs exactly one cold build each.
|
|
|
|
No step needs a template rebuild, so none of them has to be coordinated with
|
|
furyrs's pin-bump window (thread #4).
|