local-ci-cd-system/docs/RUNBOOK-PhaseC.md

Runbook — Manual CLI Reference (Linux host, pwsh-free)

A complete, hands-on reference for every ci_orchestrator command you can run by hand on the Linux host. Unless noted otherwise, commands run as the ci-runner service user against the live configuration at /var/lib/ci/config.toml.

Every command can be written two ways:

• Full form — sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>
• Short form — ci <cmd>, after installing the alias (see §0).

The examples use the short form. The paths and template names below are the real ones on this host, so most blocks are copy-paste-ready as written.

---

0. Setup (do once)

0.1 Refresh the production venv

Mandatory after every code change or merge — the venv is installed non-editable, so changes only take effect once reinstalled:

cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'

0.2 Install the ci alias

cd /opt/ci/local-ci-cd-system
./scripts/install-ci-alias.sh        # writes to ~/.bashrc and ~/.zshrc (idempotent)
source ~/.bashrc                     # or open a new shell
ci validate host                     # smoke-test the alias

Options: --system installs /etc/profile.d/ci-alias.sh for all users (needs sudo); --uninstall removes the block. The script defines two helpers:

Helper	Runs as	Use for
ci	ci-runner (sudo)	everything — writes keyring, starts VMs
ci-me	the current user	read-only checks where sudo is just noise

0.3 The live environment

Item	Value
Service user	ci-runner · production venv /opt/ci/venv/bin/python
Configuration	/var/lib/ci/config.toml
Linux template	/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx (snapshot BaseClean-Linux)
Windows templates	/var/lib/ci/templates/WinBuild2025 · WinBuild2022 (snapshot BaseClean)
Clones / artifacts	/var/lib/ci/build-vms · /var/lib/ci/artifacts
Linux SSH access	key /var/lib/ci/keys/ci_linux, user ci_build, credential target BuildVMGuest

---

1. Command index

Command	Purpose	Section
ci validate host	Check host prerequisites (no VM)	§2
ci validate guest --host IP	Probe a running guest's transport	§2
ci monitor disk	Free-space alert	§2
ci monitor runner	Watch and auto-restart act_runner	§2
ci report job	Job summaries from the JSONL logs	§2
ci job ...	Full pipeline: clone → build → collect → destroy	§3
ci vm new ...	Linked-clone a template	§4
ci wait-ready --vmx ...	Poll until SSH/WinRM is ready	§4
ci build run ...	Run a build inside a running guest	§4
ci artifacts collect ...	Copy artifacts out of a guest	§4
ci vm remove --vmx ...	Stop and delete one clone	§4
ci vm cleanup	Sweep orphaned clones and stale locks	§4 · §6
ci vmware --vmx ...	Open a VMX in the VMware Workstation GUI	§4
ci creds set --user ...	Store guest credentials in the keyring	§5
ci template deploy-linux	Build the Linux template VM	§5
ci template deploy-rust	Build the Rust Linux template VM (LinuxRust2404)	§5
ci template check-win-iso	Report the newest WS2025 ISO from massgrave + cache status	§5
ci template patch-win-iso	Inject autounattend + no-prompt boot into a Windows ISO	§5
ci template deploy-win	Build the Windows Server 2025 template VM (download→patch→deploy)	§5
ci template prepare-linux	Provision the Linux template over SSH	§5
ci template prepare-rust	Provision the Rust Linux template over SSH	§5
ci template prepare-win	Provision a Windows template over WinRM	§5
ci template backup	Timestamped copy of the template directories	§5
ci retention run	Purge old artifacts and logs	§6
tools/ci-action-published.py	Is the .gitea/ CI runs the one in this checkout?	§6
ci bench measure ...	Phase-timing benchmark	§7
ci bench run ...	Concurrency burn-in	§7
ci smoke run ...	One end-to-end job with assertions	§7

---

2. Health, monitoring & reporting

These are safe — none of them boots a VM.

validate host

Run this first, every day. It checks vmrun, template snapshots, the keyring, /var/lib/ci permissions, and the act-runner unit. A clean run exits 0 with every line marked [OK ]; any [FAIL] line names the broken check.

ci validate host ; echo "exit=$?"

Find a running guest's IP

vmrun getGuestIPAddress reads the address from VMware Tools, so the VM must be powered on with Tools running. Run it as ci-runner — the VMs are started by that user, so its vmrun inventory is the one that can see them.

# Which VMs are powered on right now
sudo -u ci-runner vmrun list

# IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx

# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx

# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx

Gotcha (AGENTS.md #1). getGuestIPAddress can exit 0 and still print Error: The VMware Tools are not running... on stdout when Tools are down. A line beginning with Error: means no IP yet — wait for boot, don't parse it.

validate guest

Connects to a booted guest, runs a trivial command, and reports the explicit cause on failure (connect vs. auth vs. command). Capture the IP into a variable so the probe is genuinely copy-paste-ready:

# Linux guest over SSH
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux

# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

monitor disk

ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json                      # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"

monitor runner

Restarts the runner up to N times per window. Optional: --webhook-url, --gitea-url, --gitea-credential-target.

ci monitor runner --service-name act-runner --max-restarts 3

report job

ci report job --last 10                          # 10 most recent, as a table
ci report job --failed                           # failures only
ci report job --job-id smoke-20260607-2030       # phase breakdown for one job
ci report job --last 20 --json                   # JSON output

---

3. Full pipeline (job)

This is the normal way to run a build: a single command clones the template, waits for the guest, runs the build, collects artifacts, and destroys the clone. It is exactly what the runner invokes on each push, and you can run it by hand too.

The examples use Simone/burnin-dummy, whose build.sh simulates a real build (sleeps ~30 s, then writes dist/build-info.txt) — so the run exercises the full clone → build → artifact-collect → destroy path end to end.

ci job \
  --job-id manual-$(date +%s) \
  --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot-name BaseClean-Linux \
  --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
  --branch main \
  --guest-os linux \
  --build-command "bash build.sh" \
  --guest-artifact-source dist \
  --use-git-clone

The VM boots headless by default. To watch it in the Workstation GUI while it runs, add --gui (needs an X display — see §4.7):

ci job \
  --job-id manual-$(date +%s) \
  --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot-name BaseClean-Linux \
  --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
  --branch main \
  --guest-os linux \
  --build-command "bash build.sh" \
  --guest-artifact-source dist \
  --use-git-clone \
  --gui

--job-id, --repo-url, --branch, and --template-path are required. The default --snapshot-name is BaseClean, so Linux jobs must pass --snapshot-name BaseClean-Linux explicitly. Other useful flags:

• Source: --commit, --submodules / --no-submodules, --use-git-clone / --host-clone, --use-shared-cache
• Build: --configuration Release, --guest-artifact-source dist, --skip-artifact, --xvfb (Linux GUI builds), --extra-env-json '{"KEY":"VALUE"}'
• VM: --guest-cpu N, --guest-memory-mb N, --template-path, --snapshot-name, --ready-timeout, --gui / --nogui (default --nogui, headless; --gui opens the Workstation window and needs an X display — see §4.7)
• Auth: --gitea-credential-target (private repos)

For a zero-config end-to-end check, prefer smoke run (§7) over hand-rolling a job invocation.

---

4. Manual VM lifecycle

These are the stages that job runs internally. Drive them individually when debugging a single phase. Run the blocks below in the same shell session — each one reuses the $VMX and $IP variables captured by an earlier step, so there are no fragile path globs to expand.

4.1 Clone a VM

ci vm new prints the clone's VMX path (and nothing else) on stdout, so capture it straight into a variable. A timestamped job-id keeps repeated runs from colliding.

VMX=$(ci vm new \
  --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot BaseClean-Linux \
  --clone-base-dir /var/lib/ci/build-vms \
  --job-id "dbg-$(date +%s)" \
  --guest-os linux \
  --start --gui)
echo "clone VMX: $VMX"

--gui opens the clone in the Workstation window so you can watch it while you debug (it needs an X display — see §4.7). Drop --gui for a headless boot.

4.2 Wait until reachable, then capture the IP

wait-ready blocks until SSH is up. The clean guest IP comes from vmrun getGuestIPAddress; capture it for the next steps.

ci wait-ready \
  --vmx "$VMX" \
  --guest-os linux \
  --ssh-user ci_build \
  --ssh-key-path /var/lib/ci/keys/ci_linux \
  --timeout 180

IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX")
echo "guest IP: $IP"

For WinRM guests, add --credential-target BuildVMGuest to wait-ready.

4.3 Run the build

Unlike job, ci build run does not inject the Gitea PAT, so an in-guest --clone-url only works for a public repo. For a private repo (like burnin-dummy), upload a local source directory with --host-source-dir instead — here the checked-out copy of burnin-dummy that ships in this repo. The build stages its --guest-artifact-source (dist) into /opt/ci/output, which is where step 4.4 collects from.

Windows guests use --credential-target BuildVMGuest in place of the SSH flags. Repeatable --extra-env KEY=VALUE; add --xvfb for Linux GUI builds or --use-shared-cache for the shared cache.

ci build run \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \
  --build-command "bash build.sh" \
  --guest-artifact-source dist

4.4 Collect artifacts

build run stages the artifact under /opt/ci/output, so collect from there (not from the dist source directory).

ci artifacts collect \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --guest-artifact-path /opt/ci/output \
  --host-artifact-dir /var/lib/ci/artifacts/dbg-manual \
  --include-logs --job-id dbg-manual

4.5 Destroy the clone

ci vm remove --vmx "$VMX"
ci vm remove --vmx "$VMX" --force        # skip the soft stop

4.6 Sweep orphans

Also a maintenance task — see §6.

ci vm cleanup --what-if                                              # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock

4.7 Open a VM in the Workstation GUI

Launches the interactive VMware Workstation GUI (not the headless vmrun path) so you can watch or poke a clone by hand. On a normal desktop session it just works — ci vmware behaves exactly like sudo -u ci-runner vmware:

Reuse the $VMX captured in §4.1, or set it to a clone's VMX path (VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1) picks the first existing clone):

ci vmware                                # bare GUI, no VM
ci vmware --vmx "$VMX"                    # open the VM
ci vmware --vmx "$VMX" --power-on         # power on (-x)
ci vmware --vmx "$VMX" --fullscreen       # power on + fullscreen (-X)

Other flags: --new-window (-n), --display :0 (force a display when $DISPLAY isn't forwarded), --vmware-path. The GUI is spawned detached; the command prints its PID and a log path under /tmp/vmware-*.log.

How the display works. VMware connects through the X host ACL, not the cookie. The inherited XAUTHORITY (the desktop user's ~/.Xauthority) is unreadable by ci-runner, so vmware drops it and relies on an xhost grant — normally already present (xhost lists SI:localuser:ci-runner).

Making the grant persistent. An xhost grant is per-session and is wiped on logout or X restart. setup-host-linux.sh (Step 8) installs an XDG autostart entry in the desktop user's home that re-applies it at every graphical login. You can also (re)run that step standalone:

cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall

This applies the grant immediately (when a display is reachable) and writes ~simone/.config/autostart/ci-runner-xhost.desktop for future logins.

One-off, without the script. Add the grant from the logged-in desktop user (not ci-runner):

xhost +SI:localuser:ci-runner

vmware pre-flights the display with xdpyinfo and prints this exact command on failure rather than dying silently.

---

5. Credentials & template provisioning

creds set

Stores the guest login in the keyring (replaces Set-CIGuestCredential.ps1). The password never appears on the command line — verify it afterwards with the validate host keyring check.

ci creds set --target BuildVMGuest --user ci_build           # hidden prompt
 printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin

The leading space before printf keeps the secret out of shell history.

Template management

Run ci template <sub> --help for the (many) per-template options.

ci template deploy-linux         # build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-rust          # build the Rust Linux template VM (LinuxRust2404)
ci template deploy-win           # build the Windows Server 2025 template VM (see below)
ci template prepare-linux        # provision an existing Linux template over SSH
ci template prepare-rust         # provision the Rust Linux template over SSH
ci template prepare-win          # provision a Windows template over WinRM (Linux-host driven)
ci template backup               # timestamped backup of the template directories

deploy-rust / prepare-rust — the Rust Linux template (LinuxRust2404)

A separate template for Rust workspaces (e.g. Tauri apps): same Ubuntu 24.04 cloud image and guest layout as LinuxBuild2404, but prepare-rust runs template/Install-CIToolchain-LinuxRust2404.sh, which additionally bakes:

• the Tauri v2 webview/GTK development libraries (libwebkit2gtk-4.1-dev, libgtk-3-dev, libayatana-appindicator3-dev, librsvg2-dev);
• a system-wide Rust toolchain via rustup under /opt/rust (RUSTUP_HOME=/opt/rust/rustup, CARGO_HOME=/opt/rust/cargo) with the rustfmt and clippy components. The tree is chowned to ci_build after the installs, not only before — otherwise every write the build user makes (toolchain sync, dependency download into CARGO_HOME/registry, cargo install) fails with Permission denied. Exposure is threefold because the orchestrator runs build commands in a non-login shell (exec_command + sh -c), which never sources profile.d: /etc/profile.d/rust.sh for interactive shells, /etc/environment (read by pam_env at SSH session open) for the two home variables, and /usr/local/bin symlinks for the rustup proxies;
• the cargo-audit binary (skip with --no-cargo-audit). The audit advisory DB is never baked — cargo audit fetches it at runtime.

ci template deploy-rust                      # same flags as deploy-linux; 60 GB disk default
ci template prepare-rust                     # default VMX: <CI_TEMPLATES>/LinuxRust2404/...
                                             # snapshot: BaseClean-LinuxRust
ci template prepare-rust --rust-channel 1.97.1   # pin an exact toolchain

# Both phases in one run, unattended. deploy and prepare stay separate
# commands — --then-prepare only chains them, reusing the VM deploy left
# running (no second power-on, no second IP detection).
#
# Run from the repo root: the guest toolchain script is looked up under
# <cwd>/template/. With --then-prepare this is checked BEFORE the deploy
# starts, so a wrong directory costs two seconds instead of failing the
# hand-off after a ~45-minute deploy. Pass --toolchain-script to run from
# anywhere.
cd /opt/ci/local-ci-cd-system
ci template deploy-rust --then-prepare --rust-channel 1.97.1
ci template deploy-linux --then-prepare

# from any directory
ci template deploy-rust --then-prepare --rust-channel 1.97.1 \
    --toolchain-script /opt/ci/local-ci-cd-system/template/Install-CIToolchain-LinuxRust2404.sh

--snapshot-name on deploy-linux/deploy-rust names the snapshot that prepare-* will create (BaseClean-Linux / BaseClean-LinuxRust) — the deploy commands take no snapshot of their own, unlike deploy-win. With --then-prepare the value is handed to prepare-*, so overriding it on the deploy side changes what actually gets created.

Both prepare-linux and prepare-rust default --vmx to their template VMX under <CI_TEMPLATES> (LinuxBuild2404 / LinuxRust2404) when neither --vmx nor --ip is passed — so the plain ci template prepare-rust just works after deploy-rust. Pass --ip to provision a running guest without VM lifecycle (no auto power-on/shutdown/snapshot).

--rust-channel defaults to stable — a moving target: when a new stable ships, rustup show in a job re-downloads the toolchain at runtime. Pin an exact version for a fully deterministic snapshot (the consumer repo decides this via its rust-toolchain.toml). Refresh the snapshot when the pinned channel or the apt security baseline moves.

Two ways to route jobs to it:

1. Per job (no host change). Pass template-path + snapshot-name to the local-ci-build composite action. This cannot leave a job unschedulable, which is why consumer repos default to it.

2. By label — runs-on: rust-build. This needs a second act_runner instance, not just an extra label on the existing one: act_runner reads a single runner.envs block per process and never tells the workflow which label picked the job (there is no RUNNER_LABELS variable — v1.0.4 exposes only RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION). A label can select a template only by selecting a runner whose envs name that template.

   # 1. mint a registration token from the stored GiteaPAT (0600 file, never
   #    printed — pass it by path so it stays out of `ps` and shell history)
   sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
       --out /var/lib/ci/.rust-runner-regtoken
   
   # 2. provision + register + enable the unit
   sudo ./scripts/install-rust-runner.sh \
       --token-file /var/lib/ci/.rust-runner-regtoken
   
   # 3. the token is single-use — destroy it
   sudo shred -u /var/lib/ci/.rust-runner-regtoken
   

   Provisions /var/lib/ci/runner-rust/ from deploy/runner-rust-config.yaml (label rust-build:host, GITEA_CI_LINUX_TEMPLATE_PATH → LinuxRust2404, GITEA_CI_LINUX_SNAPSHOT_NAME → BaseClean-LinuxRust), registers it, and installs act-runner-rust.service. Its capacity adds to the main runner's — keep the sum inside the host RAM budget (currently 2 + 1 = 3 concurrent build VMs).

   Verify the label is announced before any workflow depends on it: a runs-on value that no runner advertises leaves the job unschedulable with no error anywhere — silently zero runs.

   sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py \
       --label rust-build     # exit 1 unless an ONLINE runner announces it
   

   Note that runner.file in the config is an absolute path. act_runner resolves a relative file: against the process CWD, so registering from anywhere else drops the registration state (which contains the runner token) into that directory and the daemon then fails with registration file not found.

   Both runner services share one binary, /opt/ci/act_runner/act_runner. Replacing it upgrades act-runner.service and act-runner-rust.service together — a staged rollout needs a second binary alongside it and the ExecStart of one unit pointed at it. Current version is v1.0.4; the upgrade to v2.x was evaluated and deferred (TODO §6.5).

Note that guest-os: 'Auto' cannot pick a template by itself, for the same reason: the label is invisible to the action, and the VMX guestOS field is only readable after the clone. With Auto the action requires template-path and snapshot-name and fails fast otherwise, instead of silently falling back to the Windows template as it did before.

deploy-win — build the Windows Server 2025 template

Takes a Windows Server 2025 install ISO — downloaded from --iso-url (cached under /var/lib/ci/iso/ by file name) or a local --win-iso — patches it (injects autounattend.xml + post-install.ps1 and swaps the EFI boot image to the no-prompt variant), runs the unattended install, swaps the NIC e1000e → vmxnet3 and takes the cold PostInstall snapshot. VMware Tools are mounted as a second CD.

# Download + deploy (e.g. a massgrave direct link), headless
ci template deploy-win --iso-url 'https://example/win2025.iso'
ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 <HASH>   # verify
ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso   # local ISO, watch install

Finding / preparing an ISO from massgrave. The massgrave "updated" link is behind a Cloudflare-Turnstile file host, so the download itself can't be scripted — but you can resolve the newest build and pre-patch a manual download:

ci template check-win-iso                       # newest WS2025 build + URL + cache status
# → download that URL in a browser into /var/lib/ci/iso/<filename>, then:
ci template patch-win-iso --in /var/lib/ci/iso/<filename>   # → <filename>.patched.iso
ci template deploy-win --win-iso /var/lib/ci/iso/<filename>.patched.iso

(deploy-win patches the ISO itself too, so patch-win-iso is optional — use it to produce/inspect the bootable no-prompt ISO separately.)

Notes:

• Provide exactly one of --iso-url / --win-iso. A downloaded ISO is cached by its URL file name and reused on re-runs.
• The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO from massgrave. (UUP dump was dropped: on Linux it only yields the RTM install.wim, so Windows Update had to install the whole cumulative anyway.)
• Needs p7zip-full, xorriso (installed by setup-host-linux.sh) and ~16 GiB scratch for the ISO repack. The patched ISO is cached per VM dir and only rebuilt when the base ISO or answer files change.
• After it finishes: ci template prepare-win then ci validate guest --winrm --guest-os windows.

---

6. Maintenance

retention run

Purges old artifacts and logs. Always dry-run first. Retention switches to the aggressive window automatically when free space drops below --min-free-gb.

ci retention run --what-if                                                       # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50

Orphan sweep

See §4.6 (ci vm cleanup).

Publishing .gitea/ — committed is not deployed

act_runner never executes this working copy. For every job it re-clones the repository from the Gitea server at the ref the workflow names (...local-ci-build@main) and runs that copy. Anything under .gitea/ is therefore inert until it is pushed, and the gap is invisible from here: git log shows the fix, CI runs the old file.

This has already cost a consumer a day of debugging — 59 commits sat unpushed while a downstream repo chased a snapshot-name bug that was fixed locally. Check before announcing any action change as done:

/opt/ci/venv/bin/python tools/ci-action-published.py
# OK   — .gitea/ matches origin/main; CI runs what you see here.
# FAIL — lists the drifted files, exit 1

Run it as the user who commits, not under sudo -u ci-runner: it fetches, which needs write access to .git, and .git/FETCH_HEAD here belongs to the committing user.

It catches both ways of being unpublished, committed-but-unpushed and edited-but-uncommitted, and fetches first so it compares against the server rather than a stale remote-tracking ref (--no-fetch to skip the network).

---

7. Phase C testing

Burn-in, benchmark, and smoke checks — all native Python, no pwsh.

smoke run

One end-to-end job plus assertions (replaces Test-Smoke.ps1). It asserts: exit 0, an artifact directory under /var/lib/ci/artifacts/<job-id>/, and a job/success event in invoke-ci.jsonl.

ci smoke run --guest-os linux                       # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip       # real Linux build, end-to-end

bench measure

Times the clone / start / IP / transport / destroy phases and appends a row to benchmark.jsonl (legacy field names preserved for trend continuity).

ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4    # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl

bench run

Concurrency burn-in (replaces Test-CapacityBurnIn.ps1). Each round asserts that all jobs exit 0, no orphan clone directory is left behind, and no stale vm-start.lock remains. The JSON report lands under /var/lib/ci/artifacts/bench/<timestamp>.json.

ci bench run --guest-os linux   --concurrency 2 --rounds 2    # quick confidence
ci bench run --guest-os linux   --concurrency 4 --rounds 10   # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10   # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"

---

8. Cutover checklist & uninstalling pwsh

Repeat for at least one week with pwsh still installed as an A/B safety net:

• The normal git push CI pipeline stays green.
• ci validate host is green daily.
• ci bench run --guest-os linux --concurrency 4 --rounds 10 is green (no orphans or locks).
• ci smoke run is green on both --guest-os linux and --guest-os windows.
• ci bench measure timings are comparable to historical baselines.
• No procedure forced a fallback to a .ps1 on the Linux host.

Only after a fully green week:

pwsh --version
sudo apt-get remove powershell
ci validate host                # must still exit 0 without pwsh

To roll back, sudo ./setup-host-linux.sh --with-pwsh reinstalls pwsh; the original .ps1 scripts remain in scripts/archived/.

---

9. Troubleshooting

Symptom	Cause	Fix
No such command 'bench'	Production venv not refreshed	§0.1 — pip install .
validate host keyring [FAIL]	Credential missing or wrong user	§5 — creds set
validate guest connect failure	VM not booted, wrong IP, or firewall	Confirm the VM is running and re-check its IP
bench run round FAIL: orphan	Clone was not destroyed	ci vm cleanup; check vmrun and /var/lib/ci/build-vms
WinRM jobs drop at concurrency 4	Known instability (TODO §3.6)	Use --concurrency 3
IP-acquire timeout (Linux)	vmnet8 DHCP wedged	sudo systemctl restart vmware-networks
vmrun "operation was canceled"	vmmon/vmnet unbuilt after a kernel bump	sudo vmware-modconfig --console --install-all (or reboot; vmware-modules-rebuild.service rebuilds)
VMs won't start after a reboot — no /dev/vmnet*, vmware-vmx dies status=237/KEYRING	Host stack didn't come up at boot: vmmon unloaded + vmnet DHCP/NAT down (the flaky sysv vmware.service failed to generate — systemd-sysv-generator: Cannot find unit vmware.service)	sudo systemctl start vmware-host.service, then confirm it's enabled. Provisioned by scripts/install-vmware-host-services.sh — see deploy/systemd
ci: command not found	Alias not loaded	source ~/.bashrc, or re-run §0.2