# Runbook — Manual CLI Reference (Linux host, pwsh-free) A complete, hands-on reference for every `ci_orchestrator` command you can run by hand on the Linux host. Unless noted otherwise, commands run as the **`ci-runner`** service user against the live configuration at `/var/lib/ci/config.toml`. Every command can be written two ways: - **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ` - **Short form** — `ci `, after installing the alias (see [§0](#0-setup-do-once)). The examples use the short form. The paths and template names below are the real ones on this host, so most blocks are copy-paste-ready as written. --- ## 0. Setup (do once) ### 0.1 Refresh the production venv Mandatory after every code change or merge — the venv is installed non-editable, so changes only take effect once reinstalled: ```bash cd /opt/ci/local-ci-cd-system sudo /opt/ci/venv/bin/python -m pip install . sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds' ``` ### 0.2 Install the `ci` alias ```bash cd /opt/ci/local-ci-cd-system ./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent) source ~/.bashrc # or open a new shell ci validate host # smoke-test the alias ``` Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs sudo); `--uninstall` removes the block. The script defines two helpers: | Helper | Runs as | Use for | | ------- | ------------------ | ----------------------------------------- | | `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs | | `ci-me` | the current user | read-only checks where sudo is just noise | ### 0.3 The live environment | Item | Value | | ------------------ | -------------------------------------------------------------------------------------- | | Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` | | Configuration | `/var/lib/ci/config.toml` | | Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) | | Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) | | Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | | Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` | --- ## 1. Command index | Command | Purpose | Section | | ----------------------------- | ---------------------------------------------------- | --------------------------------------------------- | | `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) | | `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) | | `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) | | `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) | | `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) | | `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) | | `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) | | `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) | | `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) | | `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) | | `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) | | `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) | | `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) | | `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) | | `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) | | `ci template check-win-iso` | Report the newest WS2025 ISO from massgrave + cache status | [§5](#5-credentials--template-provisioning) | | `ci template patch-win-iso` | Inject autounattend + no-prompt boot into a Windows ISO | [§5](#5-credentials--template-provisioning) | | `ci template deploy-win` | Build the Windows Server 2025 template VM (download→patch→deploy) | [§5](#5-credentials--template-provisioning) | | `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) | | `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) | | `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) | | `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) | | `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) | | `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) | | `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) | --- ## 2. Health, monitoring & reporting These are safe — none of them boots a VM. ### `validate host` Run this first, every day. It checks vmrun, template snapshots, the keyring, `/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0` with every line marked `[OK ]`; any `[FAIL]` line names the broken check. ```bash ci validate host ; echo "exit=$?" ``` ### Find a running guest's IP `vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must be **powered on** with Tools running. Run it as `ci-runner` — the VMs are started by that user, so its `vmrun` inventory is the one that can see them. ```bash # Which VMs are powered on right now sudo -u ci-runner vmrun list # IP of the Linux template sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx # IP of a Windows template sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx # IP of a build clone (glob the job directory) sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx ``` > **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print > `Error: The VMware Tools are not running...` on stdout when Tools are down. A > line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it. ### `validate guest` Connects to a booted guest, runs a trivial command, and reports the explicit cause on failure (connect vs. auth vs. command). Capture the IP into a variable so the probe is genuinely copy-paste-ready: ```bash # Linux guest over SSH IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx) ci validate guest --host "$IP" --ssh --guest-os linux # Windows guest over WinRM IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx) ci validate guest --host "$IP" --winrm --guest-os windows IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx) ci validate guest --host "$IP" --winrm --guest-os windows ``` ### `monitor disk` ```bash ci monitor disk --min-free-gb 50 ci monitor disk --min-free-gb 50 --json # machine-readable one-liner ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK" ``` ### `monitor runner` Restarts the runner up to N times per window. Optional: `--webhook-url`, `--gitea-url`, `--gitea-credential-target`. ```bash ci monitor runner --service-name act-runner --max-restarts 3 ``` ### `report job` ```bash ci report job --last 10 # 10 most recent, as a table ci report job --failed # failures only ci report job --job-id smoke-20260607-2030 # phase breakdown for one job ci report job --last 20 --json # JSON output ``` --- ## 3. Full pipeline (`job`) This is the normal way to run a build: a single command clones the template, waits for the guest, runs the build, collects artifacts, and destroys the clone. It is exactly what the runner invokes on each push, and you can run it by hand too. The examples use `Simone/burnin-dummy`, whose `build.sh` simulates a real build (sleeps ~30 s, then writes `dist/build-info.txt`) — so the run exercises the full clone → build → artifact-collect → destroy path end to end. ```bash ci job \ --job-id manual-$(date +%s) \ --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot-name BaseClean-Linux \ --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \ --branch main \ --guest-os linux \ --build-command "bash build.sh" \ --guest-artifact-source dist \ --use-git-clone ``` The VM boots headless by default. To watch it in the Workstation GUI while it runs, add `--gui` (needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)): ```bash ci job \ --job-id manual-$(date +%s) \ --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot-name BaseClean-Linux \ --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \ --branch main \ --guest-os linux \ --build-command "bash build.sh" \ --guest-artifact-source dist \ --use-git-clone \ --gui ``` `--job-id`, `--repo-url`, `--branch`, and `--template-path` are required. The default `--snapshot-name` is `BaseClean`, so Linux jobs must pass `--snapshot-name BaseClean-Linux` explicitly. Other useful flags: - **Source**: `--commit`, `--submodules` / `--no-submodules`, `--use-git-clone` / `--host-clone`, `--use-shared-cache` - **Build**: `--configuration Release`, `--guest-artifact-source dist`, `--skip-artifact`, `--xvfb` (Linux GUI builds), `--extra-env-json '{"KEY":"VALUE"}'` - **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`, `--snapshot-name`, `--ready-timeout`, `--gui` / `--nogui` (default `--nogui`, headless; `--gui` opens the Workstation window and needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)) - **Auth**: `--gitea-credential-target` (private repos) > For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing)) > over hand-rolling a `job` invocation. --- ## 4. Manual VM lifecycle These are the stages that `job` runs internally. Drive them individually when debugging a single phase. **Run the blocks below in the same shell session** — each one reuses the `$VMX` and `$IP` variables captured by an earlier step, so there are no fragile path globs to expand. ### 4.1 Clone a VM `ci vm new` prints the clone's VMX path (and nothing else) on stdout, so capture it straight into a variable. A timestamped job-id keeps repeated runs from colliding. ```bash VMX=$(ci vm new \ --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot BaseClean-Linux \ --clone-base-dir /var/lib/ci/build-vms \ --job-id "dbg-$(date +%s)" \ --guest-os linux \ --start --gui) echo "clone VMX: $VMX" ``` `--gui` opens the clone in the Workstation window so you can watch it while you debug (it needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)). Drop `--gui` for a headless boot. ### 4.2 Wait until reachable, then capture the IP `wait-ready` blocks until SSH is up. The clean guest IP comes from `vmrun getGuestIPAddress`; capture it for the next steps. ```bash ci wait-ready \ --vmx "$VMX" \ --guest-os linux \ --ssh-user ci_build \ --ssh-key-path /var/lib/ci/keys/ci_linux \ --timeout 180 IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX") echo "guest IP: $IP" ``` For WinRM guests, add `--credential-target BuildVMGuest` to `wait-ready`. ### 4.3 Run the build Unlike `job`, `ci build run` does **not** inject the Gitea PAT, so an in-guest `--clone-url` only works for a public repo. For a private repo (like `burnin-dummy`), upload a local source directory with `--host-source-dir` instead — here the checked-out copy of `burnin-dummy` that ships in this repo. The build stages its `--guest-artifact-source` (`dist`) into `/opt/ci/output`, which is where step 4.4 collects from. Windows guests use `--credential-target BuildVMGuest` in place of the SSH flags. Repeatable `--extra-env KEY=VALUE`; add `--xvfb` for Linux GUI builds or `--use-shared-cache` for the shared cache. ```bash ci build run \ --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \ --build-command "bash build.sh" \ --guest-artifact-source dist ``` ### 4.4 Collect artifacts `build run` stages the artifact under `/opt/ci/output`, so collect from there (not from the `dist` source directory). ```bash ci artifacts collect \ --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --guest-artifact-path /opt/ci/output \ --host-artifact-dir /var/lib/ci/artifacts/dbg-manual \ --include-logs --job-id dbg-manual ``` ### 4.5 Destroy the clone ```bash ci vm remove --vmx "$VMX" ci vm remove --vmx "$VMX" --force # skip the soft stop ``` ### 4.6 Sweep orphans Also a maintenance task — see [§6](#6-maintenance). ```bash ci vm cleanup --what-if # list only ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock ``` ### 4.7 Open a VM in the Workstation GUI Launches the **interactive** VMware Workstation GUI (not the headless vmrun path) so you can watch or poke a clone by hand. On a normal desktop session it just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`: Reuse the `$VMX` captured in [§4.1](#41-clone-a-vm), or set it to a clone's VMX path (`VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1)` picks the first existing clone): ```bash ci vmware # bare GUI, no VM ci vmware --vmx "$VMX" # open the VM ci vmware --vmx "$VMX" --power-on # power on (-x) ci vmware --vmx "$VMX" --fullscreen # power on + fullscreen (-X) ``` Other flags: `--new-window` (`-n`), `--display :0` (force a display when `$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the command prints its PID and a log path under `/tmp/vmware-*.log`. **How the display works.** VMware connects through the X **host ACL**, not the cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant — normally already present (`xhost` lists `SI:localuser:ci-runner`). **Making the grant persistent.** An xhost grant is per-session and is wiped on logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in the desktop user's home that re-applies it at every graphical login. You can also (re)run that step standalone: ```bash cd /opt/ci/local-ci-cd-system # --desktop-user defaults to $SUDO_USER sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone # To remove the grant and the autostart entry: sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall ``` This applies the grant immediately (when a display is reachable) and writes `~simone/.config/autostart/ci-runner-xhost.desktop` for future logins. **One-off, without the script.** Add the grant from the logged-in desktop user (not `ci-runner`): ```bash xhost +SI:localuser:ci-runner ``` `vmware` pre-flights the display with `xdpyinfo` and prints this exact command on failure rather than dying silently. --- ## 5. Credentials & template provisioning ### `creds set` Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`). The password never appears on the command line — verify it afterwards with the `validate host` keyring check. ```bash ci creds set --target BuildVMGuest --user ci_build # hidden prompt printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin ``` > The leading space before `printf` keeps the secret out of shell history. ### Template management Run `ci template --help` for the (many) per-template options. ```bash ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04) ci template deploy-win # build the Windows Server 2025 template VM (see below) ci template prepare-linux # provision an existing Linux template over SSH ci template prepare-win # provision a Windows template over WinRM (Linux-host driven) ci template backup # timestamped backup of the template directories ``` ### `deploy-win` — build the Windows Server 2025 template Takes a Windows Server 2025 install ISO — downloaded from `--iso-url` (cached under `/var/lib/ci/iso/` by file name) or a local `--win-iso` — patches it (injects `autounattend.xml` + `post-install.ps1` and swaps the EFI boot image to the no-prompt variant), runs the unattended install, swaps the NIC e1000e → vmxnet3 and takes the cold `PostInstall` snapshot. VMware Tools are mounted as a second CD. ```bash # Download + deploy (e.g. a massgrave direct link), headless ci template deploy-win --iso-url 'https://example/win2025.iso' ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 # verify ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso # local ISO, watch install ``` **Finding / preparing an ISO from massgrave.** The massgrave "updated" link is behind a Cloudflare-Turnstile file host, so the download itself can't be scripted — but you can resolve the newest build and pre-patch a manual download: ```bash ci template check-win-iso # newest WS2025 build + URL + cache status # → download that URL in a browser into /var/lib/ci/iso/, then: ci template patch-win-iso --in /var/lib/ci/iso/ # → .patched.iso ci template deploy-win --win-iso /var/lib/ci/iso/.patched.iso ``` (`deploy-win` patches the ISO itself too, so `patch-win-iso` is optional — use it to produce/inspect the bootable no-prompt ISO separately.) Notes: - Provide **exactly one** of `--iso-url` / `--win-iso`. A downloaded ISO is cached by its URL file name and reused on re-runs. - The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO from massgrave. (UUP dump was dropped: on Linux it only yields the RTM `install.wim`, so Windows Update had to install the whole cumulative anyway.) - Needs `p7zip-full`, `xorriso` (installed by `setup-host-linux.sh`) and ~16 GiB scratch for the ISO repack. The patched ISO is cached per VM dir and only rebuilt when the base ISO or answer files change. - After it finishes: `ci template prepare-win` then `ci validate guest --winrm --guest-os windows`. --- ## 6. Maintenance ### `retention run` Purges old artifacts and logs. Always dry-run first. Retention switches to the aggressive window automatically when free space drops below `--min-free-gb`. ```bash ci retention run --what-if # dry run ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50 ``` ### Orphan sweep See [§4.6](#46-sweep-orphans) (`ci vm cleanup`). --- ## 7. Phase C testing Burn-in, benchmark, and smoke checks — all native Python, no pwsh. ### `smoke run` One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts: exit `0`, an artifact directory under `/var/lib/ci/artifacts//`, and a `job/success` event in `invoke-ci.jsonl`. ```bash ci smoke run --guest-os linux # no-op marker job (fastest) ci smoke run --guest-os windows ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end ``` ### `bench measure` Times the clone / start / IP / transport / destroy phases and appends a row to `benchmark.jsonl` (legacy field names preserved for trend continuity). ```bash ci bench measure --guest-os linux --iterations 1 ci bench measure --guest-os linux --iterations 4 # match historical baselines sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl ``` ### `bench run` Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts that all jobs exit `0`, no orphan clone directory is left behind, and no stale `vm-start.lock` remains. The JSON report lands under `/var/lib/ci/artifacts/bench/.json`. ```bash ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6) sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)" ``` --- ## 8. Cutover checklist & uninstalling pwsh Repeat for at least one week with pwsh **still installed** as an A/B safety net: - [ ] The normal `git push` CI pipeline stays green. - [ ] `ci validate host` is green daily. - [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks). - [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`. - [ ] `ci bench measure` timings are comparable to historical baselines. - [ ] No procedure forced a fallback to a `.ps1` on the Linux host. Only after a fully green week: ```bash pwsh --version sudo apt-get remove powershell ci validate host # must still exit 0 without pwsh ``` To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original `.ps1` scripts remain in `scripts/archived/`. --- ## 9. Troubleshooting | Symptom | Cause | Fix | | -------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------- | | `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` | | `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` | | `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP | | `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` | | WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` | | IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | | `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; the systemd guard rebuilds) | | `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |