Files
local-ci-cd-system/plans/cargo-dependency-cache-measurements.md
T
Simone 9f78742111 docs(cache): end-to-end numbers, operator runbook, and a vmrun gotcha
Measured through the real pipeline, not the unit tests: a Rust job goes from
186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which
is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore.

The test that decides whether this ships is row 4: a deliberate type error in
fury-core, built warm. The cache restored 645 MB of dependencies and the build
still went red with E0308, exit 1, publishing nothing. A cache that can turn a
red build green is the worst failure a CI system has because it is silent --
re-run that row whenever the prune rules change.

AGENTS.md #17 records something found while validating: `vmrun list` is
per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone
is very much alive, which makes the "no VMs running" check before a
deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and
was already the recommended diagnosis.
2026-07-22 22:15:32 +02:00

6.1 KiB
Raw Blame History

Cargo dependency cache — measurements

Measured 2026-07-22 on a LinuxRust2404 linked clone (4 vCPU, 3.9 GB RAM, rustc 1.97.1, zstd present at /usr/bin/zstd), running furyrs b8ea567 with the build-command block from its .gitea/workflows/ci.yml verbatim.

Entry #12 of the furyrs thread set the rule: measure before building. There it killed the registry cache — 9.3s of saving for +800 MB on the template. Here the numbers point the other way, decisively.

Baseline

Cold pipeline, empty target/ 134.7s

Matches the 133s furyrs reported for job 1021 and the 126.6s measured in #12, so the clone is representative.

Sizes

Bytes Human
target/ raw 3,206,670,175 3.0 GiB
tar of raw target/ 3,215,503,360 3.0 GiB
tar + zstd -3 -T0, raw 810,108,028 773 MiB
target/ after prune 2,719,971,105 2.5 GiB
entry as stored (tar --zstd, pruned) 676,585,779 645 MiB

Pruning strips 487 MB (15%) of raw target/, which is the workspace crates' own artifacts — 42 fingerprint directories across the six members (fury-core 11, fury-gui 12, fury-ipc 6, fury-backend 5, fury-cli 5, furyd 3). 1,928 dependency artifacts and 903 dependency fingerprints survive.

Costs

Step Seconds Paid by
prune + assert 0.21 publish
pack (tar --zstd) 10.88 publish
fetch guest→host (140 MiB/s) 4.62 publish
publish total 15.7 only on a cache miss
upload host→guest (179 MiB/s) 3.60 restore
extract 4.52 restore
restore total 8.1 every warm job

Verdict: build it

A warm job pays 8.1s to skip roughly 120s of dependency compilation — the ~490 external crates that a committed Cargo.lock keeps bit-identical between runs. Net saving ≈ 112s on a 134.7s job, about 83%.

The publish path costs 15.7s but runs only on a miss, i.e. once per Cargo.lock change, so it is amortised across every run in between.

Transfer is not the bottleneck anyone feared: the virtual NIC moves the entry at 140–180 MiB/s, so 645 MiB crosses in under 4 seconds each way.

End-to-end, through ci job (2026-07-22)

Full pipeline against LinuxRust2404, furyrs b8ea567, --skip-artifact. "Build" is the build phase alone; "total" includes clone, boot (~35s) and teardown.

# Scenario Build Total Cache line in the log
1 Cold, empty store 150s 186s miss → prune: ok → published … (645 MB)
2 Warm, same commit 47s 84s hit … — restoring
3 Warm, source modified 38s 75s hit … — restoring
4 Warm, deliberate compile error — — hit, then build fails, exit 1
5 --no-cargo-cache 133s 170s none — cache never touched

Saving: 103s on the build phase (150 → 47), 102s on the total (186 → 84). Row 3 is the real CI scenario and is the fastest of the warm runs, because the touched crate is small.

Row 2's log names every workspace crate being rebuilt —

Compiling fury-core, fury-gui, fury-ipc, fury-backend, furyd, fury-cli

— and no dependency. That is the design working: dependencies restored, workspace recompiled from source, every time.

Row 4 is the one that decides whether this ships

A deliberate type error was pushed to a scratch branch and built warm:

[cache] hit for v1-1311… (645 MB) — restoring
error[E0308]: mismatched types
   |            ---   ^^^^^^^^^^^^^^^^^^^ expected `u32`, found `&str`
error: could not compile `fury-core` (lib) due to 1 previous error
JOB_EXIT=1

The cache restored 645 MB of dependencies and the regression still went red. A cache that can turn a red build green is the worst failure a CI system has, because it is silent; this is the test that rules it out, and it is worth re-running whenever the prune rules change.

The failed job also published nothing — the store still held exactly one entry and .incoming/ was empty — because publishing sits after the build-failure check. A broken tree cannot become an entry.

Notes that changed the implementation

zstd is already in the template. /usr/bin/zstd is present on LinuxRust2404, so the gzip fallback is dead code in practice and no template rebuild is needed. Keep the fallback anyway — it costs one branch and keeps the feature working on a plain LinuxBuild2404.

The name-based assertion was worthless and was replaced. The plan's original check re-scanned for */.fingerprint/<member>-* after deleting exactly those paths. Asked to catch a planted fury-core-deadbeefcafe fingerprint, it exited 0 — the prune had deleted the evidence before the assertion looked. An assertion built from the same globs as the deletion can only prove that rm ran; it is blind to the failure that matters, a glob that never matched the artifact at all.

The replacement asks a different oracle. Cargo records in each .d depfile the source files an artifact was built from: a dependency points into /opt/rust/cargo/registry/..., anything built from workspace source points back into the workspace. After a correct prune, zero surviving depfiles reference workspace source outside target/. Verified negatively as well — an artifact named totallyunrelated-abc123.d (which no prune glob matches) declaring a dependency on fury-core/src/lib.rs makes the script exit 2 and refuse to publish.

Check Clean tree Planted fingerprint Planted foreign artifact
name-based (original) 0 0 — missed it 0 — missed it
depfile-based (shipped) 0 0 (prune removed it, correctly) 2 — refused

Feature variants coexist in one entry, as predicted. deps/ holds libclap-943062c20e623534.rmeta and libclap-abe395d6778d44ce.rmeta side by side — the same crate compiled under two different feature resolutions. This is why the feature set is not part of the cache key: cargo already separates the variants by hash, so one entry serves all of furyrs's four feature configurations instead of fragmenting into four.