local-ci-cd-system/docs/RUNBOOK-PhaseC.md

Runbook — Manual CLI Reference (Linux host, pwsh-free)

A complete, hands-on reference for every ci_orchestrator command you can run by hand on the Linux host. Unless noted otherwise, commands run as the ci-runner service user against the live configuration at /var/lib/ci/config.toml.

Every command can be written two ways:

• Full form — sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>
• Short form — ci <cmd>, after installing the alias (see §0).

The examples use the short form. The paths and template names below are the real ones on this host, so most blocks are copy-paste-ready as written.

---

0. Setup (do once)

0.1 Refresh the production venv

Mandatory after every code change or merge — the venv is installed non-editable, so changes only take effect once reinstalled:

cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'

0.2 Install the ci alias

cd /opt/ci/local-ci-cd-system
./scripts/install-ci-alias.sh        # writes to ~/.bashrc and ~/.zshrc (idempotent)
source ~/.bashrc                     # or open a new shell
ci validate host                     # smoke-test the alias

Options: --system installs /etc/profile.d/ci-alias.sh for all users (needs sudo); --uninstall removes the block. The script defines two helpers:

Helper	Runs as	Use for
ci	ci-runner (sudo)	everything — writes keyring, starts VMs
ci-me	the current user	read-only checks where sudo is just noise

0.3 The live environment

Item	Value
Service user	ci-runner · production venv /opt/ci/venv/bin/python
Configuration	/var/lib/ci/config.toml
Linux template	/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx (snapshot BaseClean-Linux)
Windows templates	/var/lib/ci/templates/WinBuild2025 · WinBuild2022 (snapshot BaseClean)
Clones / artifacts	/var/lib/ci/build-vms · /var/lib/ci/artifacts
Linux SSH access	key /var/lib/ci/keys/ci_linux, user ci_build, credential target BuildVMGuest

---

1. Command index

Command	Purpose	Section
ci validate host	Check host prerequisites (no VM)	§2
ci validate guest --host IP	Probe a running guest's transport	§2
ci monitor disk	Free-space alert	§2
ci monitor runner	Watch and auto-restart act_runner	§2
ci report job	Job summaries from the JSONL logs	§2
ci job ...	Full pipeline: clone → build → collect → destroy	§3
ci vm new ...	Linked-clone a template	§4
ci wait-ready --vmx ...	Poll until SSH/WinRM is ready	§4
ci build run ...	Run a build inside a running guest	§4
ci artifacts collect ...	Copy artifacts out of a guest	§4
ci vm remove --vmx ...	Stop and delete one clone	§4
ci vm cleanup	Sweep orphaned clones and stale locks	§4 · §6
ci vmware --vmx ...	Open a VMX in the VMware Workstation GUI	§4
ci creds set --user ...	Store guest credentials in the keyring	§5
ci template deploy-linux	Build the Linux template VM	§5
ci template prepare-linux	Provision the Linux template over SSH	§5
ci template prepare-win	Provision a Windows template over WinRM	§5
ci template backup	Timestamped copy of the template directories	§5
ci retention run	Purge old artifacts and logs	§6
ci bench measure ...	Phase-timing benchmark	§7
ci bench run ...	Concurrency burn-in	§7
ci smoke run ...	One end-to-end job with assertions	§7

---

2. Health, monitoring & reporting

These are safe — none of them boots a VM.

validate host

Run this first, every day. It checks vmrun, template snapshots, the keyring, /var/lib/ci permissions, and the act-runner unit. A clean run exits 0 with every line marked [OK ]; any [FAIL] line names the broken check.

ci validate host ; echo "exit=$?"

Find a running guest's IP

vmrun getGuestIPAddress reads the address from VMware Tools, so the VM must be powered on with Tools running. Run it as ci-runner — the VMs are started by that user, so its vmrun inventory is the one that can see them.

# Which VMs are powered on right now
sudo -u ci-runner vmrun list

# IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx

# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx

# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx

Gotcha (AGENTS.md #1). getGuestIPAddress can exit 0 and still print Error: The VMware Tools are not running... on stdout when Tools are down. A line beginning with Error: means no IP yet — wait for boot, don't parse it.

validate guest

Connects to a booted guest, runs a trivial command, and reports the explicit cause on failure (connect vs. auth vs. command). Capture the IP into a variable so the probe is genuinely copy-paste-ready:

# Linux guest over SSH
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux

# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows

monitor disk

ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json                      # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"

monitor runner

Restarts the runner up to N times per window. Optional: --webhook-url, --gitea-url, --gitea-credential-target.

ci monitor runner --service-name act-runner --max-restarts 3

report job

ci report job --last 10                          # 10 most recent, as a table
ci report job --failed                           # failures only
ci report job --job-id smoke-20260607-2030       # phase breakdown for one job
ci report job --last 20 --json                   # JSON output

---

3. Full pipeline (job)

This is the normal way to run a build: a single command clones the template, waits for the guest, runs the build, collects artifacts, and destroys the clone. It is exactly what the runner invokes on each push, and you can run it by hand too.

ci job \
  --job-id manual-$(date +%s) \
  --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot-name BaseClean-Linux \
  --repo-url https://gitea.emulab.it/Simone/local-ci-cd-system.git \
  --branch main \
  --guest-os linux \
  --build-command "echo build-ok" \
  --use-git-clone

--job-id, --repo-url, --branch, and --template-path are required. The default --snapshot-name is BaseClean, so Linux jobs must pass --snapshot-name BaseClean-Linux explicitly. Other useful flags:

• Source: --commit, --submodules / --no-submodules, --use-git-clone / --host-clone, --use-shared-cache
• Build: --configuration Release, --guest-artifact-source dist, --skip-artifact, --xvfb (Linux GUI builds), --extra-env-json '{"KEY":"VALUE"}'
• VM: --guest-cpu N, --guest-memory-mb N, --template-path, --snapshot-name, --ready-timeout, --gui / --nogui (default --nogui, headless; --gui opens the Workstation window and needs an X display — see §4.7)
• Auth: --gitea-credential-target (private repos)

For a zero-config end-to-end check, prefer smoke run (§7) over hand-rolling a job invocation.

---

4. Manual VM lifecycle

These are the stages that job runs internally. Drive them individually when debugging a single phase.

4.1 Clone a VM

ci vm new \
  --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot BaseClean-Linux \
  --clone-base-dir /var/lib/ci/build-vms \
  --job-id dbg-1 \
  --guest-os linux \
  --start                          # prints the clone's VMX path on stdout

4.2 Wait until reachable

Prints the guest IP once ready. Use --ip-address to skip polling, or --credential-target for WinRM guests.

ci wait-ready \
  --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
  --guest-os linux \
  --ssh-user ci_build \
  --ssh-key-path /var/lib/ci/keys/ci_linux \
  --timeout 180

4.3 Run the build

The guest must already be up — capture its IP first (see §2). Windows guests use --credential-target BuildVMGuest in place of the SSH flags. Repeatable --extra-env KEY=VALUE; add --xvfb for Linux GUI builds or --use-shared-cache for the shared cache.

ci build run \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --clone-url https://gitea.emulab.it/Simone/local-ci-cd-system.git --clone-branch main \
  --build-command "echo build-ok" \
  --guest-artifact-source dist

4.4 Collect artifacts

ci artifacts collect \
  --ip-address "$IP" \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --guest-artifact-path dist \
  --host-artifact-dir /var/lib/ci/artifacts/dbg-1 \
  --include-logs --job-id dbg-1

4.5 Destroy the clone

ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --force   # skip the soft stop

4.6 Sweep orphans

Also a maintenance task — see §6.

ci vm cleanup --what-if                                              # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock

4.7 Open a VM in the Workstation GUI

Launches the interactive VMware Workstation GUI (not the headless vmrun path) so you can watch or poke a clone by hand. On a normal desktop session it just works — ci vmware behaves exactly like sudo -u ci-runner vmware:

ci vmware                                                            # bare GUI, no VM
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --power-on    # power on (-x)
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --fullscreen  # power on + fullscreen (-X)

Other flags: --new-window (-n), --display :0 (force a display when $DISPLAY isn't forwarded), --vmware-path. The GUI is spawned detached; the command prints its PID and a log path under /tmp/vmware-*.log.

How the display works. VMware connects through the X host ACL, not the cookie. The inherited XAUTHORITY (the desktop user's ~/.Xauthority) is unreadable by ci-runner, so vmware drops it and relies on an xhost grant — normally already present (xhost lists SI:localuser:ci-runner).

Making the grant persistent. An xhost grant is per-session and is wiped on logout or X restart. setup-host-linux.sh (Step 8) installs an XDG autostart entry in the desktop user's home that re-applies it at every graphical login. You can also (re)run that step standalone:

cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall

This applies the grant immediately (when a display is reachable) and writes ~simone/.config/autostart/ci-runner-xhost.desktop for future logins.

One-off, without the script. Add the grant from the logged-in desktop user (not ci-runner):

xhost +SI:localuser:ci-runner

vmware pre-flights the display with xdpyinfo and prints this exact command on failure rather than dying silently.

---

5. Credentials & template provisioning

creds set

Stores the guest login in the keyring (replaces Set-CIGuestCredential.ps1). The password never appears on the command line — verify it afterwards with the validate host keyring check.

ci creds set --target BuildVMGuest --user ci_build           # hidden prompt
 printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin

The leading space before printf keeps the secret out of shell history.

Template management

Run ci template <sub> --help for the (many) per-template options.

ci template deploy-linux         # build the Linux template VM from scratch (Ubuntu 24.04)
ci template prepare-linux        # provision an existing Linux template over SSH
ci template prepare-win          # provision a Windows template over WinRM (Linux-host driven)
ci template backup               # timestamped backup of the template directories

---

6. Maintenance

retention run

Purges old artifacts and logs. Always dry-run first. Retention switches to the aggressive window automatically when free space drops below --min-free-gb.

ci retention run --what-if                                                       # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50

Orphan sweep

See §4.6 (ci vm cleanup).

---

7. Phase C testing

Burn-in, benchmark, and smoke checks — all native Python, no pwsh.

smoke run

One end-to-end job plus assertions (replaces Test-Smoke.ps1). It asserts: exit 0, an artifact directory under /var/lib/ci/artifacts/<job-id>/, and a job/success event in invoke-ci.jsonl.

ci smoke run --guest-os linux                       # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip       # real Linux build, end-to-end

bench measure

Times the clone / start / IP / transport / destroy phases and appends a row to benchmark.jsonl (legacy field names preserved for trend continuity).

ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4    # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl

bench run

Concurrency burn-in (replaces Test-CapacityBurnIn.ps1). Each round asserts that all jobs exit 0, no orphan clone directory is left behind, and no stale vm-start.lock remains. The JSON report lands under /var/lib/ci/artifacts/bench/<timestamp>.json.

ci bench run --guest-os linux   --concurrency 2 --rounds 2    # quick confidence
ci bench run --guest-os linux   --concurrency 4 --rounds 10   # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10   # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"

---

8. Cutover checklist & uninstalling pwsh

Repeat for at least one week with pwsh still installed as an A/B safety net:

• The normal git push CI pipeline stays green.
• ci validate host is green daily.
• ci bench run --guest-os linux --concurrency 4 --rounds 10 is green (no orphans or locks).
• ci smoke run is green on both --guest-os linux and --guest-os windows.
• ci bench measure timings are comparable to historical baselines.
• No procedure forced a fallback to a .ps1 on the Linux host.

Only after a fully green week:

pwsh --version
sudo apt-get remove powershell
ci validate host                # must still exit 0 without pwsh

To roll back, sudo ./setup-host-linux.sh --with-pwsh reinstalls pwsh; the original .ps1 scripts remain in scripts/archived/.

---

9. Troubleshooting

Symptom	Cause	Fix
No such command 'bench'	Production venv not refreshed	§0.1 — pip install .
validate host keyring [FAIL]	Credential missing or wrong user	§5 — creds set
validate guest connect failure	VM not booted, wrong IP, or firewall	Confirm the VM is running and re-check its IP
bench run round FAIL: orphan	Clone was not destroyed	ci vm cleanup; check vmrun and /var/lib/ci/build-vms
WinRM jobs drop at concurrency 4	Known instability (TODO §3.6)	Use --concurrency 3
IP-acquire timeout (Linux)	vmnet8 DHCP wedged	sudo systemctl restart vmware-networks
vmrun "operation was canceled"	vmmon/vmnet unbuilt after a kernel bump	sudo vmware-modconfig --console --install-all (or reboot; the systemd guard rebuilds)
ci: command not found	Alias not loaded	source ~/.bashrc, or re-run §0.2