Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1a96330de251dfdb51582ebda3ac6b6973022caf
local-ci-cd-system/docs/RUNBOOK-PhaseC.md
T
Simone 9f78742111 docs(cache): end-to-end numbers, operator runbook, and a vmrun gotcha 
Measured through the real pipeline, not the unit tests: a Rust job goes from
186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which
is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore.

The test that decides whether this ships is row 4: a deliberate type error in
fury-core, built warm. The cache restored 645 MB of dependencies and the build
still went red with E0308, exit 1, publishing nothing. A cache that can turn a
red build green is the worst failure a CI system has because it is silent --
re-run that row whenever the prune rules change.

AGENTS.md #17 records something found while validating: `vmrun list` is
per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone
is very much alive, which makes the "no VMs running" check before a
deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and
was already the recommended diagnosis.
2026-07-22 22:15:32 +02:00

766 lines
35 KiB
Markdown
Raw Blame History

# Runbook — Manual CLI Reference (Linux host, pwsh-free)
A complete, hands-on reference for every `ci_orchestrator` command you can run
by hand on the Linux host. Unless noted otherwise, commands run as the
**`ci-runner`** service user against the live configuration at
`/var/lib/ci/config.toml`.
Every command can be written two ways:
- **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form** — `ci <cmd>`, after installing the alias (see [§0](#0-setup-do-once)).
The examples use the short form. The paths and template names below are the
real ones on this host, so most blocks are copy-paste-ready as written.
---
## 0. Setup (do once)
### 0.1 Refresh the production venv
Mandatory after every code change or merge — the venv is installed
non-editable, so changes only take effect once reinstalled:
```bash
cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
```
### 0.2 Install the `ci` alias
```bash
cd /opt/ci/local-ci-cd-system
./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent)
source ~/.bashrc # or open a new shell
ci validate host # smoke-test the alias
```
Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs
sudo); `--uninstall` removes the block. The script defines two helpers:
| Helper | Runs as | Use for |
| ------- | ------------------ | ----------------------------------------- |
| `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs |
| `ci-me` | the current user | read-only checks where sudo is just noise |
### 0.3 The live environment
| Item | Value |
| ------------------ | -------------------------------------------------------------------------------------- |
| Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` |
| Configuration | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) |
| Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` |
---
## 1. Command index
| Command | Purpose | Section |
| ----------------------------- | ---------------------------------------------------- | --------------------------------------------------- |
| `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) |
| `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) |
| `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) |
| `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) |
| `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) |
| `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) |
| `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) |
| `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) |
| `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) |
| `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) |
| `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) |
| `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) |
| `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) |
| `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) |
| `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) |
| `ci template deploy-rust` | Build the Rust Linux template VM (LinuxRust2404) | [§5](#5-credentials--template-provisioning) |
| `ci template check-win-iso` | Report the newest WS2025 ISO from massgrave + cache status | [§5](#5-credentials--template-provisioning) |
| `ci template patch-win-iso` | Inject autounattend + no-prompt boot into a Windows ISO | [§5](#5-credentials--template-provisioning) |
| `ci template deploy-win` | Build the Windows Server 2025 template VM (download→patch→deploy) | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-rust` | Provision the Rust Linux template over SSH | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) |
| `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) |
| `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) |
| `tools/ci-action-published.py` | Is the `.gitea/` CI runs the one in this checkout? | [§6](#6-maintenance) |
| `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) |
| `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) |
| `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) |
---
## 2. Health, monitoring & reporting
These are safe — none of them boots a VM.
### `validate host`
Run this first, every day. It checks vmrun, template snapshots, the keyring,
`/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0`
with every line marked `[OK ]`; any `[FAIL]` line names the broken check.
```bash
ci validate host ; echo "exit=$?"
```
### Find a running guest's IP
`vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must
be **powered on** with Tools running. Run it as `ci-runner` — the VMs are
started by that user, so its `vmrun` inventory is the one that can see them.
```bash
# Which VMs are powered on right now
sudo -u ci-runner vmrun list
# IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
```
> **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print
> `Error: The VMware Tools are not running...` on stdout when Tools are down. A
> line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest`
Connects to a booted guest, runs a trivial command, and reports the explicit
cause on failure (connect vs. auth vs. command). Capture the IP into a variable
so the probe is genuinely copy-paste-ready:
```bash
# Linux guest over SSH
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux
# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows
```
### `monitor disk`
```bash
ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
```
### `monitor runner`
Restarts the runner up to N times per window. Optional: `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
```bash
ci monitor runner --service-name act-runner --max-restarts 3
```
### `report job`
```bash
ci report job --last 10 # 10 most recent, as a table
ci report job --failed # failures only
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON output
```
---
## 3. Full pipeline (`job`)
This is the normal way to run a build: a single command clones the template,
waits for the guest, runs the build, collects artifacts, and destroys the
clone. It is exactly what the runner invokes on each push, and you can run it
by hand too.
The examples use `Simone/burnin-dummy`, whose `build.sh` simulates a real build
(sleeps ~30 s, then writes `dist/build-info.txt`) — so the run exercises the
full clone → build → artifact-collect → destroy path end to end.
```bash
ci job \
--job-id manual-$(date +%s) \
--template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
--snapshot-name BaseClean-Linux \
--repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
--branch main \
--guest-os linux \
--build-command "bash build.sh" \
--guest-artifact-source dist \
--use-git-clone
```
The VM boots headless by default. To watch it in the Workstation GUI while it
runs, add `--gui` (needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)):
```bash
ci job \
--job-id manual-$(date +%s) \
--template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
--snapshot-name BaseClean-Linux \
--repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
--branch main \
--guest-os linux \
--build-command "bash build.sh" \
--guest-artifact-source dist \
--use-git-clone \
--gui
```
`--job-id`, `--repo-url`, `--branch`, and `--template-path` are required. The
default `--snapshot-name` is `BaseClean`, so Linux jobs must pass
`--snapshot-name BaseClean-Linux` explicitly. Other useful flags:
- **Source**: `--commit`, `--submodules` / `--no-submodules`,
`--use-git-clone` / `--host-clone`, `--use-shared-cache`
- **Build**: `--configuration Release`, `--guest-artifact-source dist`,
`--skip-artifact`, `--xvfb` (Linux GUI builds),
`--extra-env-json '{"KEY":"VALUE"}'`
- **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`,
`--snapshot-name`, `--ready-timeout`, `--gui` / `--nogui` (default `--nogui`,
headless; `--gui` opens the Workstation window and needs an X display —
see [§4.7](#47-open-a-vm-in-the-workstation-gui))
- **Auth**: `--gitea-credential-target` (private repos)
> For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing))
> over hand-rolling a `job` invocation.
---
## 4. Manual VM lifecycle
These are the stages that `job` runs internally. Drive them individually when
debugging a single phase. **Run the blocks below in the same shell session** —
each one reuses the `$VMX` and `$IP` variables captured by an earlier step, so
there are no fragile path globs to expand.
### 4.1 Clone a VM
`ci vm new` prints the clone's VMX path (and nothing else) on stdout, so capture
it straight into a variable. A timestamped job-id keeps repeated runs from
colliding.
```bash
VMX=$(ci vm new \
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
--snapshot BaseClean-Linux \
--clone-base-dir /var/lib/ci/build-vms \
--job-id "dbg-$(date +%s)" \
--guest-os linux \
--start --gui)
echo "clone VMX: $VMX"
```
`--gui` opens the clone in the Workstation window so you can watch it while you
debug (it needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)).
Drop `--gui` for a headless boot.
### 4.2 Wait until reachable, then capture the IP
`wait-ready` blocks until SSH is up. The clean guest IP comes from
`vmrun getGuestIPAddress`; capture it for the next steps.
```bash
ci wait-ready \
--vmx "$VMX" \
--guest-os linux \
--ssh-user ci_build \
--ssh-key-path /var/lib/ci/keys/ci_linux \
--timeout 180
IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX")
echo "guest IP: $IP"
```
For WinRM guests, add `--credential-target BuildVMGuest` to `wait-ready`.
### 4.3 Run the build
Unlike `job`, `ci build run` does **not** inject the Gitea PAT, so an in-guest
`--clone-url` only works for a public repo. For a private repo (like
`burnin-dummy`), upload a local source directory with `--host-source-dir`
instead — here the checked-out copy of `burnin-dummy` that ships in this repo.
The build stages its `--guest-artifact-source` (`dist`) into `/opt/ci/output`,
which is where step 4.4 collects from.
Windows guests use `--credential-target BuildVMGuest` in place of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; add `--xvfb` for Linux GUI builds or
`--use-shared-cache` for the shared cache.
```bash
ci build run \
--ip-address "$IP" \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \
--build-command "bash build.sh" \
--guest-artifact-source dist
```
### 4.4 Collect artifacts
`build run` stages the artifact under `/opt/ci/output`, so collect from there
(not from the `dist` source directory).
```bash
ci artifacts collect \
--ip-address "$IP" \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--guest-artifact-path /opt/ci/output \
--host-artifact-dir /var/lib/ci/artifacts/dbg-manual \
--include-logs --job-id dbg-manual
```
### 4.5 Destroy the clone
```bash
ci vm remove --vmx "$VMX"
ci vm remove --vmx "$VMX" --force # skip the soft stop
```
### 4.6 Sweep orphans
Also a maintenance task — see [§6](#6-maintenance).
```bash
ci vm cleanup --what-if # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
```
### 4.7 Open a VM in the Workstation GUI
Launches the **interactive** VMware Workstation GUI (not the headless vmrun
path) so you can watch or poke a clone by hand. On a normal desktop session it
just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`:
Reuse the `$VMX` captured in [§4.1](#41-clone-a-vm), or set it to a clone's VMX
path (`VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1)` picks the first
existing clone):
```bash
ci vmware # bare GUI, no VM
ci vmware --vmx "$VMX" # open the VM
ci vmware --vmx "$VMX" --power-on # power on (-x)
ci vmware --vmx "$VMX" --fullscreen # power on + fullscreen (-X)
```
Other flags: `--new-window` (`-n`), `--display :0` (force a display when
`$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the
command prints its PID and a log path under `/tmp/vmware-*.log`.
**How the display works.** VMware connects through the X **host ACL**, not the
cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is
unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant —
normally already present (`xhost` lists `SI:localuser:ci-runner`).
**Making the grant persistent.** An xhost grant is per-session and is wiped on
logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart
entry in the desktop user's home that re-applies it at every graphical login.
You can also (re)run that step standalone:
```bash
cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall
```
This applies the grant immediately (when a display is reachable) and writes
`~simone/.config/autostart/ci-runner-xhost.desktop` for future logins.
**One-off, without the script.** Add the grant from the logged-in desktop user
(not `ci-runner`):
```bash
xhost +SI:localuser:ci-runner
```
`vmware` pre-flights the display with `xdpyinfo` and prints this exact command
on failure rather than dying silently.
---
## 5. Credentials & template provisioning
### `creds set`
Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`).
The password never appears on the command line — verify it afterwards with the
`validate host` keyring check.
```bash
ci creds set --target BuildVMGuest --user ci_build # hidden prompt
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
```
> The leading space before `printf` keeps the secret out of shell history.
### Template management
Run `ci template <sub> --help` for the (many) per-template options.
```bash
ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-rust # build the Rust Linux template VM (LinuxRust2404)
ci template deploy-win # build the Windows Server 2025 template VM (see below)
ci template prepare-linux # provision an existing Linux template over SSH
ci template prepare-rust # provision the Rust Linux template over SSH
ci template prepare-win # provision a Windows template over WinRM (Linux-host driven)
ci template backup # timestamped backup of the template directories
```
### `deploy-rust` / `prepare-rust` — the Rust Linux template (LinuxRust2404)
A separate template for Rust workspaces (e.g. Tauri apps): same Ubuntu 24.04
cloud image and guest layout as `LinuxBuild2404`, but `prepare-rust` runs
`template/Install-CIToolchain-LinuxRust2404.sh`, which additionally bakes:
- the Tauri v2 webview/GTK development libraries
(`libwebkit2gtk-4.1-dev`, `libgtk-3-dev`, `libayatana-appindicator3-dev`,
`librsvg2-dev`);
- a system-wide Rust toolchain via rustup under `/opt/rust`
(`RUSTUP_HOME=/opt/rust/rustup`, `CARGO_HOME=/opt/rust/cargo`) with the
`rustfmt` and `clippy` components. The tree is `chown`ed to `ci_build`
**after** the installs, not only before — otherwise every write the build
user makes (toolchain sync, dependency download into
`CARGO_HOME/registry`, `cargo install`) fails with `Permission denied`.
Exposure is threefold because the orchestrator runs build commands in a
**non-login** shell (`exec_command` + `sh -c`), which never sources
`profile.d`: `/etc/profile.d/rust.sh` for interactive shells,
`/etc/environment` (read by `pam_env` at SSH session open) for the two
home variables, and `/usr/local/bin` symlinks for the rustup proxies;
- the `cargo-audit` binary (skip with `--no-cargo-audit`). The audit
**advisory DB is never baked** — `cargo audit` fetches it at runtime.
```bash
ci template deploy-rust # same flags as deploy-linux; 60 GB disk default
ci template prepare-rust # default VMX: <CI_TEMPLATES>/LinuxRust2404/...
# snapshot: BaseClean-LinuxRust
ci template prepare-rust --rust-channel 1.97.1 # pin an exact toolchain
# Both phases in one run, unattended. deploy and prepare stay separate
# commands — --then-prepare only chains them, reusing the VM deploy left
# running (no second power-on, no second IP detection).
#
# Run from the repo root: the guest toolchain script is looked up under
# <cwd>/template/. With --then-prepare this is checked BEFORE the deploy
# starts, so a wrong directory costs two seconds instead of failing the
# hand-off after a ~45-minute deploy. Pass --toolchain-script to run from
# anywhere.
cd /opt/ci/local-ci-cd-system
ci template deploy-rust --then-prepare --rust-channel 1.97.1
ci template deploy-linux --then-prepare
# from any directory
ci template deploy-rust --then-prepare --rust-channel 1.97.1 \
--toolchain-script /opt/ci/local-ci-cd-system/template/Install-CIToolchain-LinuxRust2404.sh
```
`--snapshot-name` on `deploy-linux`/`deploy-rust` names the snapshot that
`prepare-*` will create (`BaseClean-Linux` / `BaseClean-LinuxRust`) — the
deploy commands take no snapshot of their own, unlike `deploy-win`. With
`--then-prepare` the value is handed to `prepare-*`, so overriding it on the
deploy side changes what actually gets created.
Both `prepare-linux` and `prepare-rust` default `--vmx` to their template VMX
under `<CI_TEMPLATES>` (`LinuxBuild2404` / `LinuxRust2404`) when neither
`--vmx` nor `--ip` is passed — so the plain `ci template prepare-rust` just
works after `deploy-rust`. Pass `--ip` to provision a running guest without
VM lifecycle (no auto power-on/shutdown/snapshot).
`--rust-channel` defaults to `stable` — a moving target: when a new stable
ships, `rustup show` in a job re-downloads the toolchain at runtime. Pin an
exact version for a fully deterministic snapshot (the consumer repo decides
this via its `rust-toolchain.toml`). Refresh the snapshot when the pinned
channel or the apt security baseline moves.
Two ways to route jobs to it:
1. **Per job (no host change).** Pass `template-path` + `snapshot-name` to the
`local-ci-build` composite action. This cannot leave a job unschedulable,
which is why consumer repos default to it.
2. **By label** — `runs-on: rust-build`. This needs a **second act_runner
instance**, not just an extra label on the existing one: act_runner reads a
single `runner.envs` block per process and never tells the workflow which
label picked the job (there is no `RUNNER_LABELS` variable — v1.0.4 exposes
only `RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION`). A label can select a
template only by selecting a runner whose envs name that template.
```bash
# 1. mint a registration token from the stored GiteaPAT (0600 file, never
# printed — pass it by path so it stays out of `ps` and shell history)
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
--out /var/lib/ci/.rust-runner-regtoken
# 2. provision + register + enable the unit
sudo ./scripts/install-rust-runner.sh \
--token-file /var/lib/ci/.rust-runner-regtoken
# 3. the token is single-use — destroy it
sudo shred -u /var/lib/ci/.rust-runner-regtoken
```
Provisions `/var/lib/ci/runner-rust/` from `deploy/runner-rust-config.yaml`
(label `rust-build:host`, `GITEA_CI_LINUX_TEMPLATE_PATH` →
`LinuxRust2404`, `GITEA_CI_LINUX_SNAPSHOT_NAME` → `BaseClean-LinuxRust`),
registers it, and installs `act-runner-rust.service`. Its `capacity` adds
to the main runner's — keep the sum inside the host RAM budget (currently
2 + 1 = 3 concurrent build VMs).
**Verify the label is announced before any workflow depends on it**: a
`runs-on` value that no runner advertises leaves the job unschedulable with
no error anywhere — silently zero runs.
```bash
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py \
--label rust-build # exit 1 unless an ONLINE runner announces it
```
Note that `runner.file` in the config is an **absolute** path. act_runner
resolves a relative `file:` against the process CWD, so registering from
anywhere else drops the registration state (which contains the runner
token) into that directory and the daemon then fails with
`registration file not found`.
Both runner services share **one binary**, `/opt/ci/act_runner/act_runner`.
Replacing it upgrades `act-runner.service` and `act-runner-rust.service`
together — a staged rollout needs a second binary alongside it and the
`ExecStart` of one unit pointed at it. Current version is v1.0.4; the
upgrade to v2.x was evaluated and deferred (TODO §6.5).
Note that `guest-os: 'Auto'` cannot pick a template by itself, for the same
reason: the label is invisible to the action, and the VMX `guestOS` field is
only readable *after* the clone. With `Auto` the action requires
`template-path` **and** `snapshot-name` and fails fast otherwise, instead of
silently falling back to the Windows template as it did before.
### `deploy-win` — build the Windows Server 2025 template
Takes a Windows Server 2025 install ISO — downloaded from `--iso-url` (cached
under `/var/lib/ci/iso/` by file name) or a local `--win-iso` — patches it
(injects `autounattend.xml` + `post-install.ps1` and swaps the EFI boot image to
the no-prompt variant), runs the unattended install, swaps the NIC e1000e →
vmxnet3 and takes the cold `PostInstall` snapshot. VMware Tools are mounted as a
second CD.
```bash
# Download + deploy (e.g. a massgrave direct link), headless
ci template deploy-win --iso-url 'https://example/win2025.iso'
ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 <HASH> # verify
ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso # local ISO, watch install
```
**Finding / preparing an ISO from massgrave.** The massgrave "updated" link is
behind a Cloudflare-Turnstile file host, so the download itself can't be
scripted — but you can resolve the newest build and pre-patch a manual download:
```bash
ci template check-win-iso # newest WS2025 build + URL + cache status
# → download that URL in a browser into /var/lib/ci/iso/<filename>, then:
ci template patch-win-iso --in /var/lib/ci/iso/<filename> # → <filename>.patched.iso
ci template deploy-win --win-iso /var/lib/ci/iso/<filename>.patched.iso
```
(`deploy-win` patches the ISO itself too, so `patch-win-iso` is optional — use it
to produce/inspect the bootable no-prompt ISO separately.)
Notes:
- Provide **exactly one** of `--iso-url` / `--win-iso`. A downloaded ISO is
cached by its URL file name and reused on re-runs.
- The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO
from massgrave. (UUP dump was dropped: on Linux it only yields the RTM
`install.wim`, so Windows Update had to install the whole cumulative anyway.)
- Needs `p7zip-full`, `xorriso` (installed by `setup-host-linux.sh`) and ~16 GiB
scratch for the ISO repack. The patched ISO is cached per VM dir and only
rebuilt when the base ISO or answer files change.
- After it finishes: `ci template prepare-win` then
`ci validate guest --winrm --guest-os windows`.
---
## 6. Maintenance
### `retention run`
Purges old artifacts and logs. Always dry-run first. Retention switches to the
aggressive window automatically when free space drops below `--min-free-gb`.
```bash
ci retention run --what-if # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
```
### Orphan sweep
See [§4.6](#46-sweep-orphans) (`ci vm cleanup`).
### Cargo dependency cache
Rust jobs on `LinuxRust2404` reuse compiled **dependency** artifacts between
runs. Measured on furyrs: a warm job pays ~8s to skip ~120s of dependency
compilation on a 134s job. Workspace crates are never cached, so a restored
entry cannot carry the code under test.
Store: `/var/lib/ci/cache/cargo`, one immutable file per key
(`v1-<sha256>.tar.zst`, ~645 MB each). A key is
`sha256(Cargo.lock) + rustc version + host triple`.
```bash
ls -lh /var/lib/ci/cache/cargo/ # what is stored
du -sh /var/lib/ci/cache/cargo/ # total
```
What the job log tells you:
| Line | Meaning |
|---|---|
| `[cache] hit for v1-… (645 MB) — restoring` | Entry reused; dependencies will not recompile |
| `[cache] miss for v1-… — building cold` | First run for this `Cargo.lock`+toolchain; the job publishes at the end |
| `[cache] prune: ok — 6 workspace member(s) stripped, …` | Prune passed its assertion; publishing |
| `[cache] prune refused — publishing nothing` | **The assertion tripped.** An artifact still referenced workspace source. Nothing was stored; the build result itself is unaffected. Investigate before trusting the feature |
| `[cache] restore skipped: …` / `publish skipped: …` | Transport or disk problem. The build ran cold and is still valid |
Turning it off, cheapest first:
1. One workflow: `cargo-cache: 'off'` in the composite action inputs.
2. This host: `enabled = false` under `[cargo_cache]` in `/var/lib/ci/config.toml`.
3. One job: `ci job --no-cargo-cache`.
4. Everything, without deleting a file: bump `CACHE_FORMAT` in
`src/ci_orchestrator/cargo_cache.py`. Every key changes, so every existing
entry becomes unreachable. Use this when the prune rules change and older
entries can no longer be trusted.
Deleting entries by hand is always safe — they are immutable and independent,
and each one costs exactly one cold build to regenerate:
```bash
sudo -u ci-runner rm /var/lib/ci/cache/cargo/*.tar.zst
```
`ci retention run` prunes entries older than the tighter of `--retention-days`
and `[cargo_cache].retention_days`, so the low-disk aggressive window reaches
them too.
### Publishing `.gitea/` — committed is not deployed
act_runner never executes this working copy. For every job it re-clones the
repository **from the Gitea server** at the ref the workflow names
(`...local-ci-build@main`) and runs that copy. Anything under `.gitea/` is
therefore inert until it is pushed, and the gap is invisible from here: `git
log` shows the fix, CI runs the old file.
This has already cost a consumer a day of debugging — 59 commits sat unpushed
while a downstream repo chased a snapshot-name bug that was fixed locally.
Check before announcing any action change as done:
```bash
/opt/ci/venv/bin/python tools/ci-action-published.py
# OK — .gitea/ matches origin/main; CI runs what you see here.
# FAIL — lists the drifted files, exit 1
```
Run it as the user who commits, **not** under `sudo -u ci-runner`: it fetches,
which needs write access to `.git`, and `.git/FETCH_HEAD` here belongs to the
committing user.
It catches both ways of being unpublished, committed-but-unpushed and
edited-but-uncommitted, and fetches first so it compares against the server
rather than a stale remote-tracking ref (`--no-fetch` to skip the network).
---
## 7. Phase C testing
Burn-in, benchmark, and smoke checks — all native Python, no pwsh.
### `smoke run`
One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts:
exit `0`, an artifact directory under `/var/lib/ci/artifacts/<job-id>/`, and a
`job/success` event in `invoke-ci.jsonl`.
```bash
ci smoke run --guest-os linux # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end
```
### `bench measure`
Times the clone / start / IP / transport / destroy phases and appends a row to
`benchmark.jsonl` (legacy field names preserved for trend continuity).
```bash
ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4 # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
```
### `bench run`
Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts
that all jobs exit `0`, no orphan clone directory is left behind, and no stale
`vm-start.lock` remains. The JSON report lands under
`/var/lib/ci/artifacts/bench/<timestamp>.json`.
```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
```
---
## 8. Cutover checklist & uninstalling pwsh
Repeat for at least one week with pwsh **still installed** as an A/B safety net:
- [ ] The normal `git push` CI pipeline stays green.
- [ ] `ci validate host` is green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks).
- [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`.
- [ ] `ci bench measure` timings are comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
Only after a fully green week:
```bash
pwsh --version
sudo apt-get remove powershell
ci validate host # must still exit 0 without pwsh
```
To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the
original `.ps1` scripts remain in `scripts/archived/`.
---
## 9. Troubleshooting
| Symptom | Cause | Fix |
| -------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------- |
| `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` |
| `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` |
| `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP |
| `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` |
| IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; `vmware-modules-rebuild.service` rebuilds) |
| VMs won't start after a reboot — no `/dev/vmnet*`, `vmware-vmx` dies `status=237/KEYRING` | Host stack didn't come up at boot: `vmmon` unloaded + vmnet DHCP/NAT down (the flaky sysv `vmware.service` failed to generate — `systemd-sysv-generator: Cannot find unit vmware.service`) | `sudo systemctl start vmware-host.service`, then confirm it's enabled. Provisioned by `scripts/install-vmware-host-services.sh` — see [deploy/systemd](../deploy/systemd/README.md#servizi-vmware-host-prerequisito-avvio-vm) |
| `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |
Reference in New Issue View Git Blame Copy Permalink