local-ci-cd-system/docs/RUNBOOK-PhaseC.md

RUNBOOK — manual CLI reference (Linux host, pwsh-free)

Simplified but complete reference for every ci_orchestrator command you can run by hand on the Linux host. All commands run as the ci-runner service user against the live config (/var/lib/ci/config.toml).

Two notations are used:

• Full form: sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>
• Short form (alias): ci <cmd> — after installing the helper (see §0).

Examples below use the short form. Drop ci for the literal command.

---

0. Setup (do once)

0.1 Refresh the production venv (mandatory after a code change/merge)

cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .        # non-editable, see CLAUDE.md
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'

0.2 Install the ci alias

./scripts/install-ci-alias.sh        # into ~/.bashrc + ~/.zshrc (idempotent)
#   --system     install /etc/profile.d/ci-alias.sh for all users (needs sudo)
#   --uninstall  remove it
source ~/.bashrc                      # or open a new shell
ci validate host                      # smoke test the alias

The script defines two helpers:

Helper	Runs as	Use for
ci	ci-runner (sudo)	everything (writes keyring, starts VMs)
ci-me	current user	read-only checks where sudo is noise

0.3 Live environment

Thing	Value
Service user	ci-runner · Prod venv /opt/ci/venv/bin/python
Config	/var/lib/ci/config.toml
Linux template	/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx (snap BaseClean-Linux)
Windows templates	/var/lib/ci/templates/WinBuild2025 · WinBuild2022 (snap BaseClean)
Clones / Artifacts	/var/lib/ci/build-vms · /var/lib/ci/artifacts
Linux SSH key	/var/lib/ci/keys/ci_linux · cred target BuildVMGuest

---

1. Quick reference — every command

Command	Does	Section
ci validate host	host prerequisites check (no VM)	§2
ci validate guest --host IP	probe a running guest's transport	§2
ci monitor disk	free-space alert	§2
ci monitor runner	watch + auto-restart act_runner	§2
ci report job	job summaries from JSONL logs	§2
ci job ...	full pipeline clone→build→collect→destroy	§3
ci vm new ...	linked-clone a template	§4
ci wait-ready --vmx ...	poll until SSH/WinRM ready	§4
ci build run ...	run a build in a running guest	§4
ci artifacts collect ...	copy artifacts out of a guest	§4
ci vm remove --vmx ...	stop + delete one clone	§4
ci vm cleanup	sweep orphaned clones + stale locks	§4 / §6
ci vmgui --vmx ...	open a VMX in the VMware Workstation GUI	§4
ci creds set --user ...	store guest credentials in keyring	§5
ci template deploy-linux	build the Linux template VM	§5
ci template prepare-linux	provision Linux template over SSH	§5
ci template prepare-win	provision Windows template over WinRM	§5
ci template backup	timestamped copy of template dirs	§5
ci retention run	purge old artifacts/logs	§6
ci bench measure ...	phase-timing benchmark	§7
ci bench run ...	concurrency burn-in	§7
ci smoke run ...	one E2E job + assertions	§7

---

2. Health, monitoring & reporting (safe, no VM boot)

validate host — run this first, every day

ci validate host ; echo "exit=$?"

Checks vmrun, template snapshots, keyring, /var/lib/ci permissions, act-runner unit. Exit 0 + all [OK ]; a [FAIL] line names the broken check.

validate guest — diagnose a running guest's transport

ci validate guest --host 192.168.79.200 --ssh   --guest-os linux     # SSH
ci validate guest --host 192.168.79.201 --winrm --guest-os windows   # WinRM

Connects + runs a trivial command, printing the explicit cause on failure (connect vs auth vs command). The guest must already be booted.

monitor disk — free-space alert

ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json            # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"

monitor runner — keep act_runner alive

ci monitor runner --service-name act-runner --max-restarts 3

Restarts the runner up to N times per window; optional --webhook-url, --gitea-url, --gitea-credential-target.

report job — read job history

ci report job --last 10                 # 10 most recent, table
ci report job --failed                  # only failures
ci report job --job-id smoke-20260607-2030   # phase breakdown for one job
ci report job --last 20 --json          # JSON out

---

3. Full pipeline (job) — the normal way to run a build

One command does everything: clone → wait-ready → build → collect → destroy. This is what the runner invokes per push; you can run it by hand too.

# Linux build, in-guest git clone, no artifact packaging
ci job \
  --job-id manual-$(date +%s) \
  --repo-url https://gitea.emulab.it/Simone/myrepo.git \
  --branch main \
  --guest-os linux \
  --build-command "make all" \
  --use-git-clone

Key flags: --commit, --configuration Release, --guest-artifact-source dist, --submodules/--no-submodules, --use-git-clone/--host-clone, --use-shared-cache, --skip-artifact, --xvfb (Linux GUI builds), --guest-cpu N, --guest-memory-mb N, --template-path, --snapshot-name, --ready-timeout, --extra-env-json '{"K":"V"}', --gitea-credential-target (private repos). Required: --job-id --repo-url --branch.

For a zero-config end-to-end check, prefer smoke run (§7) over hand-rolling job.

---

4. Manual VM lifecycle (the steps job runs internally)

Use these to drive a build stage-by-stage, e.g. when debugging a single phase.

4.1 Clone a VM

ci vm new \
  --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot BaseClean-Linux \
  --clone-base-dir /var/lib/ci/build-vms \
  --job-id dbg-1 \
  --guest-os linux \
  --start                       # prints the clone VMX path on stdout

4.2 Wait until reachable

ci wait-ready \
  --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
  --guest-os linux \
  --ssh-user ci_build \
  --ssh-key-path /var/lib/ci/keys/ci_linux \
  --timeout 180

Prints the guest IP when ready. --ip-address skips polling; --credential-target for WinRM guests.

4.3 Run the build (guest already up)

ci build run \
  --ip-address 192.168.79.200 \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
  --build-command "make all" \
  --guest-artifact-source dist

Windows guests use --credential-target BuildVMGuest instead of the SSH flags. Repeatable --extra-env KEY=VALUE; --xvfb Linux GUI; --use-shared-cache.

4.4 Collect artifacts

ci artifacts collect \
  --ip-address 192.168.79.200 \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --guest-artifact-path dist \
  --host-artifact-dir /var/lib/ci/artifacts/dbg-1 \
  --include-logs --job-id dbg-1

4.5 Destroy the clone

ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force        # skip soft stop

4.6 Sweep orphans (also a maintenance task, §6)

ci vm cleanup --what-if                  # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock

4.7 Open a VM in the Workstation GUI (interactive debugging)

Launches the interactive VMware Workstation GUI (not the headless vmrun path) so you can watch/poke a clone by hand.

# authorise the service user on your desktop X session first (run as YOUR user):
xhost +SI:localuser:ci-runner

# then open the VMX as ci-runner, forcing the display:
ci vmgui --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --display :0
ci vmgui --vmx <path> --display :0 --power-on      # also power it on (-x)
ci vmgui --vmx <path> --display :0 --fullscreen    # power on + fullscreen (-X)

# bare GUI, no VM loaded (just open Workstation):
ci vmgui --display :0

The GUI needs an X display; sudo -u ci-runner strips $DISPLAY, so pass --display :0 (or export DISPLAY). Other flags: --new-window (-n), --vmware-path to override the binary. The command spawns the GUI detached and returns immediately (prints the child PID).

---

5. Credentials & template provisioning

creds set — store the guest login (replaces Set-CIGuestCredential.ps1)

ci creds set --target BuildVMGuest --user ci_build           # hidden prompt
 printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin

Password is never on the command line. Verify via validate host keyring check. (Leading space before printf keeps the secret out of shell history.)

Template management

# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux

# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux

# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win

# Timestamped backup of the template directories
ci template backup

Run ci template <sub> --help for the (many) per-template options.

---

6. Maintenance

retention run — purge old artifacts & logs

ci retention run --what-if                          # dry run first
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50

Switches to aggressive retention automatically when free space drops below --min-free-gb.

Orphan sweep — see §4.6 (vm cleanup).

---

7. Phase C testing (burn-in / benchmark / smoke)

smoke run — one E2E job + assertions (replaces Test-Smoke.ps1)

ci smoke run --guest-os linux                       # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip       # real Linux build E2E

Asserts: exit 0 + artifact dir under /var/lib/ci/artifacts/<job-id>/ + job/success event in invoke-ci.jsonl.

bench measure — phase timings (replaces Measure-CIBenchmark.ps1)

ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4    # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl

Times clone/start/IP/transport/destroy; appends to benchmark.jsonl (legacy field names preserved for trend continuity).

bench run — concurrency burn-in (replaces Test-CapacityBurnIn.ps1)

ci bench run --guest-os linux --concurrency 2 --rounds 2     # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10    # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10  # WinRM unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"

Per round asserts: all jobs exit 0, no orphan clone dir, no stale vm-start.lock. JSON report under /var/lib/ci/artifacts/bench/<ts>.json.

---

8. Cutover checklist & uninstalling pwsh

Repeat for ≥1 week with pwsh still installed (A/B safety net):

• Normal git push CI pipeline green.
• ci validate host green daily.
• ci bench run --guest-os linux --concurrency 4 --rounds 10 green (no orphans/locks).
• ci smoke run green on --guest-os linux and --guest-os windows.
• ci bench measure timings comparable to historical baselines.
• No procedure forced a fallback to a .ps1 on the Linux host.

Then — only after a fully green week:

pwsh --version
sudo apt-get remove powershell
ci validate host                # must still exit 0 without pwsh

Rollback: sudo ./setup-host-linux.sh --with-pwsh reinstalls pwsh; the original .ps1 scripts remain in scripts/.

---

9. Troubleshooting quick map

Symptom	Cause	Fix
No such command 'bench'	prod venv not refreshed	§0.1 pip install .
validate host keyring [FAIL]	credential missing/wrong user	§5 creds set
validate guest connect failure	VM not booted / wrong IP / firewall	confirm VM running + IP
bench run round FAIL: orphan	clone not destroyed	ci vm cleanup; check vmrun + /var/lib/ci/build-vms
WinRM jobs drop at concurrency 4	known instability (TODO §3.6)	use --concurrency 3
IP acquire timeout (Linux)	vmnet8 DHCP wedged	sudo systemctl restart vmware-networks
vmrun "operation was canceled"	vmmon/vmnet unbuilt after kernel bump	sudo vmware-modconfig --console --install-all (or reboot; systemd guard rebuilds)
ci: command not found	alias not loaded	source ~/.bashrc or re-run §0.2