local-ci-cd-system/docs/RUNBOOK-PhaseC.md

Runbook — Manual CLI Reference (Linux host, pwsh-free)

A complete, hands-on reference for every ci_orchestrator command you can run by hand on the Linux host. Unless noted otherwise, commands run as the ci-runner service user against the live configuration at /var/lib/ci/config.toml.

Every command can be written two ways:

• Full form — sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>
• Short form — ci <cmd>, after installing the alias (see §0).

The examples use the short form. The paths and template names below are the real ones on this host, so most blocks are copy-paste-ready as written.

---

0. Setup (do once)

0.1 Refresh the production venv

Mandatory after every code change or merge — the venv is installed non-editable, so changes only take effect once reinstalled:

cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'

0.2 Install the ci alias

cd /opt/ci/local-ci-cd-system
./scripts/install-ci-alias.sh        # writes to ~/.bashrc and ~/.zshrc (idempotent)
source ~/.bashrc                     # or open a new shell
ci validate host                     # smoke-test the alias

Options: --system installs /etc/profile.d/ci-alias.sh for all users (needs sudo); --uninstall removes the block. The script defines two helpers:

Helper	Runs as	Use for
ci	ci-runner (sudo)	everything — writes keyring, starts VMs
ci-me	the current user	read-only checks where sudo is just noise

0.3 The live environment

Item	Value
Service user	ci-runner · production venv /opt/ci/venv/bin/python
Configuration	/var/lib/ci/config.toml
Linux template	/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx (snapshot BaseClean-Linux)
Windows templates	/var/lib/ci/templates/WinBuild2025 · WinBuild2022 (snapshot BaseClean)
Clones / artifacts	/var/lib/ci/build-vms · /var/lib/ci/artifacts
Linux SSH access	key /var/lib/ci/keys/ci_linux, user ci_build, credential target BuildVMGuest

---

1. Command index

Command	Purpose	Section
ci validate host	Check host prerequisites (no VM)	§2
ci validate guest --host IP	Probe a running guest's transport	§2
ci monitor disk	Free-space alert	§2
ci monitor runner	Watch and auto-restart act_runner	§2
ci report job	Job summaries from the JSONL logs	§2
ci job ...	Full pipeline: clone → build → collect → destroy	§3
ci vm new ...	Linked-clone a template	§4
ci wait-ready --vmx ...	Poll until SSH/WinRM is ready	§4
ci build run ...	Run a build inside a running guest	§4
ci artifacts collect ...	Copy artifacts out of a guest	§4
ci vm remove --vmx ...	Stop and delete one clone	§4
ci vm cleanup	Sweep orphaned clones and stale locks	§4 · §6
ci vmware --vmx ...	Open a VMX in the VMware Workstation GUI	§4
ci creds set --user ...	Store guest credentials in the keyring	§5
ci template deploy-linux	Build the Linux template VM	§5
ci template check-win-iso	Report the newest WS2025 ISO from massgrave + cache status	§5
ci template patch-win-iso	Inject autounattend + no-prompt boot into a Windows ISO	§5
ci template deploy-win	Build the Windows Server 2025 template VM (download→patch→deploy)	§5
ci template prepare-linux	Provision the Linux template over SSH	§5
ci template prepare-win	Provision a Windows template over WinRM	§5
ci template backup	Timestamped copy of the template directories	§5
ci retention run	Purge old artifacts and logs	§6
ci bench measure ...	Phase-timing benchmark	§7
ci bench run ...	Concurrency burn-in	§7
ci smoke run ...	One end-to-end job with assertions	§7

---

2. Health, monitoring & reporting

These are safe — none of them boots a VM.

validate host

Run this first, every day. It checks vmrun, template snapshots, the keyring, /var/lib/ci permissions, and the act-runner unit. A clean run exits 0 with every line marked [OK ]; any [FAIL] line names the broken check.

ci validate host ; echo "exit=$?"

Find a running guest's IP

vmrun getGuestIPAddress reads the address from VMware Tools, so the VM must be powered on with Tools running. Run it as ci-runner — the VMs are started by that user, so its vmrun inventory is the one that can see them.

# Which VMs are powered on right now
sudo -u ci-runner vmrun list

# IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx

# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx

# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx

Gotcha (AGENTS.md #1). getGuestIPAddress can exit 0 and still print Error: The VMware Tools are not running... on stdout when Tools are down. A line beginning with Error: means no IP yet — wait for boot, don't parse it.

validate guest

Connects to a booted guest, runs a trivial command, and reports the explicit cause on failure (connect vs. auth vs. command). Capture the IP into a variable so the probe is genuinely copy-paste-ready:

# Linux guest over SSH
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux

# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

monitor disk

ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json                      # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"

monitor runner

Restarts the runner up to N times per window. Optional: --webhook-url, --gitea-url, --gitea-credential-target.

ci monitor runner --service-name act-runner --max-restarts 3

report job

ci report job --last 10                          # 10 most recent, as a table
ci report job --failed                           # failures only
ci report job --job-id smoke-20260607-2030       # phase breakdown for one job
ci report job --last 20 --json                   # JSON output

---

3. Full pipeline (job)

This is the normal way to run a build: a single command clones the template, waits for the guest, runs the build, collects artifacts, and destroys the clone. It is exactly what the runner invokes on each push, and you can run it by hand too.

The examples use Simone/burnin-dummy, whose build.sh simulates a real build (sleeps ~30 s, then writes dist/build-info.txt) — so the run exercises the full clone → build → artifact-collect → destroy path end to end.

ci job \
  --job-id manual-$(date +%s) \
  --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot-name BaseClean-Linux \
  --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
  --branch main \
  --guest-os linux \
  --build-command "bash build.sh" \
  --guest-artifact-source dist \
  --use-git-clone

The VM boots headless by default. To watch it in the Workstation GUI while it runs, add --gui (needs an X display — see §4.7):

ci job \
  --job-id manual-$(date +%s) \
  --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot-name BaseClean-Linux \
  --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
  --branch main \
  --guest-os linux \
  --build-command "bash build.sh" \
  --guest-artifact-source dist \
  --use-git-clone \
  --gui

--job-id, --repo-url, --branch, and --template-path are required. The default --snapshot-name is BaseClean, so Linux jobs must pass --snapshot-name BaseClean-Linux explicitly. Other useful flags:

• Source: --commit, --submodules / --no-submodules, --use-git-clone / --host-clone, --use-shared-cache
• Build: --configuration Release, --guest-artifact-source dist, --skip-artifact, --xvfb (Linux GUI builds), --extra-env-json '{"KEY":"VALUE"}'
• VM: --guest-cpu N, --guest-memory-mb N, --template-path, --snapshot-name, --ready-timeout, --gui / --nogui (default --nogui, headless; --gui opens the Workstation window and needs an X display — see §4.7)
• Auth: --gitea-credential-target (private repos)

For a zero-config end-to-end check, prefer smoke run (§7) over hand-rolling a job invocation.

---

4. Manual VM lifecycle

These are the stages that job runs internally. Drive them individually when debugging a single phase. Run the blocks below in the same shell session — each one reuses the $VMX and $IP variables captured by an earlier step, so there are no fragile path globs to expand.

4.1 Clone a VM

ci vm new prints the clone's VMX path (and nothing else) on stdout, so capture it straight into a variable. A timestamped job-id keeps repeated runs from colliding.

VMX=$(ci vm new \
  --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot BaseClean-Linux \
  --clone-base-dir /var/lib/ci/build-vms \
  --job-id "dbg-$(date +%s)" \
  --guest-os linux \
  --start --gui)
echo "clone VMX: $VMX"

--gui opens the clone in the Workstation window so you can watch it while you debug (it needs an X display — see §4.7). Drop --gui for a headless boot.

4.2 Wait until reachable, then capture the IP

wait-ready blocks until SSH is up. The clean guest IP comes from vmrun getGuestIPAddress; capture it for the next steps.

ci wait-ready \
  --vmx "$VMX" \
  --guest-os linux \
  --ssh-user ci_build \
  --ssh-key-path /var/lib/ci/keys/ci_linux \
  --timeout 180

IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX")
echo "guest IP: $IP"

For WinRM guests, add --credential-target BuildVMGuest to wait-ready.

4.3 Run the build

Unlike job, ci build run does not inject the Gitea PAT, so an in-guest --clone-url only works for a public repo. For a private repo (like burnin-dummy), upload a local source directory with --host-source-dir instead — here the checked-out copy of burnin-dummy that ships in this repo. The build stages its --guest-artifact-source (dist) into /opt/ci/output, which is where step 4.4 collects from.

Windows guests use --credential-target BuildVMGuest in place of the SSH flags. Repeatable --extra-env KEY=VALUE; add --xvfb for Linux GUI builds or --use-shared-cache for the shared cache.

ci build run \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \
  --build-command "bash build.sh" \
  --guest-artifact-source dist

4.4 Collect artifacts

build run stages the artifact under /opt/ci/output, so collect from there (not from the dist source directory).

ci artifacts collect \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --guest-artifact-path /opt/ci/output \
  --host-artifact-dir /var/lib/ci/artifacts/dbg-manual \
  --include-logs --job-id dbg-manual

4.5 Destroy the clone

ci vm remove --vmx "$VMX"
ci vm remove --vmx "$VMX" --force        # skip the soft stop

4.6 Sweep orphans

Also a maintenance task — see §6.

ci vm cleanup --what-if                                              # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock

4.7 Open a VM in the Workstation GUI

Launches the interactive VMware Workstation GUI (not the headless vmrun path) so you can watch or poke a clone by hand. On a normal desktop session it just works — ci vmware behaves exactly like sudo -u ci-runner vmware:

Reuse the $VMX captured in §4.1, or set it to a clone's VMX path (VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1) picks the first existing clone):

ci vmware                                # bare GUI, no VM
ci vmware --vmx "$VMX"                    # open the VM
ci vmware --vmx "$VMX" --power-on         # power on (-x)
ci vmware --vmx "$VMX" --fullscreen       # power on + fullscreen (-X)

Other flags: --new-window (-n), --display :0 (force a display when $DISPLAY isn't forwarded), --vmware-path. The GUI is spawned detached; the command prints its PID and a log path under /tmp/vmware-*.log.

How the display works. VMware connects through the X host ACL, not the cookie. The inherited XAUTHORITY (the desktop user's ~/.Xauthority) is unreadable by ci-runner, so vmware drops it and relies on an xhost grant — normally already present (xhost lists SI:localuser:ci-runner).

Making the grant persistent. An xhost grant is per-session and is wiped on logout or X restart. setup-host-linux.sh (Step 8) installs an XDG autostart entry in the desktop user's home that re-applies it at every graphical login. You can also (re)run that step standalone:

cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall

This applies the grant immediately (when a display is reachable) and writes ~simone/.config/autostart/ci-runner-xhost.desktop for future logins.

One-off, without the script. Add the grant from the logged-in desktop user (not ci-runner):

xhost +SI:localuser:ci-runner

vmware pre-flights the display with xdpyinfo and prints this exact command on failure rather than dying silently.

---

5. Credentials & template provisioning

creds set

Stores the guest login in the keyring (replaces Set-CIGuestCredential.ps1). The password never appears on the command line — verify it afterwards with the validate host keyring check.

ci creds set --target BuildVMGuest --user ci_build           # hidden prompt
 printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin

The leading space before printf keeps the secret out of shell history.

Template management

Run ci template <sub> --help for the (many) per-template options.

ci template deploy-linux         # build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-win           # build the Windows Server 2025 template VM (see below)
ci template prepare-linux        # provision an existing Linux template over SSH
ci template prepare-win          # provision a Windows template over WinRM (Linux-host driven)
ci template backup               # timestamped backup of the template directories

deploy-win — build the Windows Server 2025 template

Takes a Windows Server 2025 install ISO — downloaded from --iso-url (cached under /var/lib/ci/iso/ by file name) or a local --win-iso — patches it (injects autounattend.xml + post-install.ps1 and swaps the EFI boot image to the no-prompt variant), runs the unattended install, swaps the NIC e1000e → vmxnet3 and takes the cold PostInstall snapshot. VMware Tools are mounted as a second CD.

# Download + deploy (e.g. a massgrave direct link), headless
ci template deploy-win --iso-url 'https://example/win2025.iso'
ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 <HASH>   # verify
ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso   # local ISO, watch install

Finding / preparing an ISO from massgrave. The massgrave "updated" link is behind a Cloudflare-Turnstile file host, so the download itself can't be scripted — but you can resolve the newest build and pre-patch a manual download:

ci template check-win-iso                       # newest WS2025 build + URL + cache status
# → download that URL in a browser into /var/lib/ci/iso/<filename>, then:
ci template patch-win-iso --in /var/lib/ci/iso/<filename>   # → <filename>.patched.iso
ci template deploy-win --win-iso /var/lib/ci/iso/<filename>.patched.iso

(deploy-win patches the ISO itself too, so patch-win-iso is optional — use it to produce/inspect the bootable no-prompt ISO separately.)

Notes:

• Provide exactly one of --iso-url / --win-iso. A downloaded ISO is cached by its URL file name and reused on re-runs.
• The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO from massgrave. (UUP dump was dropped: on Linux it only yields the RTM install.wim, so Windows Update had to install the whole cumulative anyway.)
• Needs p7zip-full, xorriso (installed by setup-host-linux.sh) and ~16 GiB scratch for the ISO repack. The patched ISO is cached per VM dir and only rebuilt when the base ISO or answer files change.
• After it finishes: ci template prepare-win then ci validate guest --winrm --guest-os windows.

---

6. Maintenance

retention run

Purges old artifacts and logs. Always dry-run first. Retention switches to the aggressive window automatically when free space drops below --min-free-gb.

ci retention run --what-if                                                       # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50

Orphan sweep

See §4.6 (ci vm cleanup).

---

7. Phase C testing

Burn-in, benchmark, and smoke checks — all native Python, no pwsh.

smoke run

One end-to-end job plus assertions (replaces Test-Smoke.ps1). It asserts: exit 0, an artifact directory under /var/lib/ci/artifacts/<job-id>/, and a job/success event in invoke-ci.jsonl.

ci smoke run --guest-os linux                       # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip       # real Linux build, end-to-end

bench measure

Times the clone / start / IP / transport / destroy phases and appends a row to benchmark.jsonl (legacy field names preserved for trend continuity).

ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4    # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl

bench run

Concurrency burn-in (replaces Test-CapacityBurnIn.ps1). Each round asserts that all jobs exit 0, no orphan clone directory is left behind, and no stale vm-start.lock remains. The JSON report lands under /var/lib/ci/artifacts/bench/<timestamp>.json.

ci bench run --guest-os linux   --concurrency 2 --rounds 2    # quick confidence
ci bench run --guest-os linux   --concurrency 4 --rounds 10   # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10   # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"

---

8. Cutover checklist & uninstalling pwsh

Repeat for at least one week with pwsh still installed as an A/B safety net:

• The normal git push CI pipeline stays green.
• ci validate host is green daily.
• ci bench run --guest-os linux --concurrency 4 --rounds 10 is green (no orphans or locks).
• ci smoke run is green on both --guest-os linux and --guest-os windows.
• ci bench measure timings are comparable to historical baselines.
• No procedure forced a fallback to a .ps1 on the Linux host.

Only after a fully green week:

pwsh --version
sudo apt-get remove powershell
ci validate host                # must still exit 0 without pwsh

To roll back, sudo ./setup-host-linux.sh --with-pwsh reinstalls pwsh; the original .ps1 scripts remain in scripts/archived/.

---

9. Troubleshooting

Symptom	Cause	Fix
No such command 'bench'	Production venv not refreshed	§0.1 — pip install .
validate host keyring [FAIL]	Credential missing or wrong user	§5 — creds set
validate guest connect failure	VM not booted, wrong IP, or firewall	Confirm the VM is running and re-check its IP
bench run round FAIL: orphan	Clone was not destroyed	ci vm cleanup; check vmrun and /var/lib/ci/build-vms
WinRM jobs drop at concurrency 4	Known instability (TODO §3.6)	Use --concurrency 3
IP-acquire timeout (Linux)	vmnet8 DHCP wedged	sudo systemctl restart vmware-networks
vmrun "operation was canceled"	vmmon/vmnet unbuilt after a kernel bump	sudo vmware-modconfig --console --install-all (or reboot; the systemd guard rebuilds)
ci: command not found	Alias not loaded	source ~/.bashrc, or re-run §0.2