Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(runbook): rewrite Phase C runbook for clarity

Browse Source 
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
This commit is contained in:
Simone
2026-06-08 21:31:25 +02:00
parent c274d334d3
commit 027e2ff4b6
1 changed files with 259 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/RUNBOOK-PhaseC.md
+259 -190
View File
@@ -1,155 +1,182 @@
# RUNBOOK — manual CLI reference (Linux host, pwsh-free)
# Runbook — Manual CLI Reference (Linux host, pwsh-free)
Simplified but **complete** reference for every `ci_orchestrator` command you can
run by hand on the Linux host. All commands run as the **`ci-runner`** service
user against the live config (`/var/lib/ci/config.toml`).
A complete, hands-on reference for every `ci_orchestrator` command you can run
by hand on the Linux host. Unless noted otherwise, commands run as the
**`ci-runner`** service user against the live configuration at
`/var/lib/ci/config.toml`.
Two notations are used:
Every command can be written two ways:
- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form (alias):** `ci <cmd>` — after installing the helper (see §0).
- **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form** — `ci <cmd>`, after installing the alias (see [§0](#0-setup-do-once)).
Examples below use the **short form**. Drop `ci` for the literal command.
The examples use the short form. The paths and template names below are the
real ones on this host, so most blocks are copy-paste-ready as written.
---
## 0. Setup (do once)
### 0.1 Refresh the production venv (mandatory after a code change/merge)
### 0.1 Refresh the production venv
Mandatory after every code change or merge — the venv is installed
non-editable, so changes only take effect once reinstalled:
```bash
cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md
sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
```
### 0.2 Install the `ci` alias
```bash
./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent)
# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo)
# --uninstall remove it
source ~/.bashrc # or open a new shell
ci validate host # smoke test the alias
cd /opt/ci/local-ci-cd-system
./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent)
source ~/.bashrc # or open a new shell
ci validate host # smoke-test the alias
```
The script defines two helpers:
Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs
sudo); `--uninstall` removes the block. The script defines two helpers:
| Helper | Runs as | Use for |
| ------- | ------------------ | --------------------------------------- |
| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) |
| `ci-me` | current user | read-only checks where sudo is noise |
| Helper | Runs as | Use for |
| ------- | ------------------ | ----------------------------------------- |
| `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs |
| `ci-me` | the current user | read-only checks where sudo is just noise |
### 0.3 Live environment
### 0.3 The live environment
| Thing | Value |
| Item | Value |
| ------------------ | ---------------------------------------------------------------------------------- |
| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` |
| Config | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) |
| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` |
| Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` |
| Configuration | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) |
| Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` |
---
## 1. Quick reference — every command
## 1. Command index
| Command | Does | Section |
| Command | Purpose | Section |
| ----------------------------- | --------------------------------------------- | ------- |
| `ci validate host` | host prerequisites check (no VM) | §2 |
| `ci validate guest --host IP` | probe a running guest's transport | §2 |
| `ci monitor disk` | free-space alert | §2 |
| `ci monitor runner` | watch + auto-restart act_runner | §2 |
| `ci report job` | job summaries from JSONL logs | §2 |
| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 |
| `ci vm new ...` | linked-clone a template | §4 |
| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 |
| `ci build run ...` | run a build in a running guest | §4 |
| `ci artifacts collect ...` | copy artifacts out of a guest | §4 |
| `ci vm remove --vmx ...` | stop + delete one clone | §4 |
| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 |
| `ci vmware --vmx ...` | open a VMX in the VMware Workstation GUI | §4 |
| `ci creds set --user ...` | store guest credentials in keyring | §5 |
| `ci template deploy-linux` | build the Linux template VM | §5 |
| `ci template prepare-linux` | provision Linux template over SSH | §5 |
| `ci template prepare-win` | provision Windows template over WinRM | §5 |
| `ci template backup` | timestamped copy of template dirs | §5 |
| `ci retention run` | purge old artifacts/logs | §6 |
| `ci bench measure ...` | phase-timing benchmark | §7 |
| `ci bench run ...` | concurrency burn-in | §7 |
| `ci smoke run ...` | one E2E job + assertions | §7 |
| `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) |
| `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) |
| `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) |
| `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) |
| `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) |
| `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) |
| `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) |
| `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) |
| `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) |
| `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) |
| `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) |
| `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) |
| `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) |
| `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) |
| `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) |
| `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) |
| `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) |
| `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) |
| `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) |
| `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) |
---
## 2. Health, monitoring & reporting (safe, no VM boot)
## 2. Health, monitoring & reporting
These are safe — none of them boots a VM.
### `validate host`
Run this first, every day. It checks vmrun, template snapshots, the keyring,
`/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0`
with every line marked `[OK ]`; any `[FAIL]` line names the broken check.
### `validate host` — run this first, every day
```bash
ci validate host ; echo "exit=$?"
```
Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner`
unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check.
### Find a running guest's IP (needed for `validate guest`)
`vmrun getGuestIPAddress` reads the IP from VMware Tools — the VM must be
**powered on** and Tools running. Run it as `ci-runner` (the VMs are started by
that user, so its `vmrun` inventory is the one that sees them):
### Find a running guest's IP
`vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must
be **powered on** with Tools running. Run it as `ci-runner` — the VMs are
started by that user, so its `vmrun` inventory is the one that can see them.
```bash
# Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# A build clone (glob the job dir)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
# List which VMs are powered on first
# Which VMs are powered on right now
sudo -u ci-runner vmrun list
```
> **Gotcha (AGENTS.md #1):** `getGuestIPAddress` can exit `0` and print
> `Error: The VMware Tools are not running...` on stdout when Tools are down.
> A line starting with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest` — diagnose a running guest's transport
# IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
```
> **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print
> `Error: The VMware Tools are not running...` on stdout when Tools are down. A
> line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest`
Connects to a booted guest, runs a trivial command, and reports the explicit
cause on failure (connect vs. auth vs. command). Capture the IP into a variable
so the probe is genuinely copy-paste-ready:
```bash
ci validate guest --host 192.168.79.128 --ssh --guest-os linux # SSH (Linux)
ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM (Windows)
# Linux guest over SSH
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux
# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows
```
Connects + runs a trivial command, printing the explicit cause on failure
(connect vs auth vs command). The guest must already be booted — get its IP
with `getGuestIPAddress` above.
### `monitor disk`
### `monitor disk` — free-space alert
```bash
ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
```
### `monitor runner` — keep act_runner alive
### `monitor runner`
Restarts the runner up to N times per window. Optional: `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
```bash
ci monitor runner --service-name act-runner --max-restarts 3
```
Restarts the runner up to N times per window; optional `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
### `report job` — read job history
### `report job`
```bash
ci report job --last 10 # 10 most recent, table
ci report job --failed # only failures
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON out
ci report job --last 10 # 10 most recent, as a table
ci report job --failed # failures only
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON output
```
---
## 3. Full pipeline (`job`) — the normal way to run a build
## 3. Full pipeline (`job`)
One command does everything: clone → wait-ready → build → collect → destroy.
This is what the runner invokes per push; you can run it by hand too.
This is the normal way to run a build: a single command clones the template,
waits for the guest, runs the build, collects artifacts, and destroys the
clone. It is exactly what the runner invokes on each push, and you can run it
by hand too.
```bash
# Linux build, in-guest git clone, no artifact packaging
ci job \
--job-id manual-$(date +%s) \
--repo-url https://gitea.emulab.it/Simone/myrepo.git \
@@ -159,21 +186,29 @@ ci job \
--use-git-clone
```
Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`,
`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`,
`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`,
`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`,
`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`.
`--job-id`, `--repo-url`, and `--branch` are required. Other useful flags:
> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`.
- **Source**: `--commit`, `--submodules` / `--no-submodules`,
`--use-git-clone` / `--host-clone`, `--use-shared-cache`
- **Build**: `--configuration Release`, `--guest-artifact-source dist`,
`--skip-artifact`, `--xvfb` (Linux GUI builds),
`--extra-env-json '{"KEY":"VALUE"}'`
- **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`,
`--snapshot-name`, `--ready-timeout`
- **Auth**: `--gitea-credential-target` (private repos)
> For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing))
> over hand-rolling a `job` invocation.
---
## 4. Manual VM lifecycle (the steps `job` runs internally)
## 4. Manual VM lifecycle
Use these to drive a build stage-by-stage, e.g. when debugging a single phase.
These are the stages that `job` runs internally. Drive them individually when
debugging a single phase.
### 4.1 Clone a VM
```bash
ci vm new \
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
@@ -181,10 +216,14 @@ ci vm new \
--clone-base-dir /var/lib/ci/build-vms \
--job-id dbg-1 \
--guest-os linux \
--start # prints the clone VMX path on stdout
--start # prints the clone's VMX path on stdout
```
### 4.2 Wait until reachable
Prints the guest IP once ready. Use `--ip-address` to skip polling, or
`--credential-target` for WinRM guests.
```bash
ci wait-ready \
--vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
@@ -193,26 +232,29 @@ ci wait-ready \
--ssh-key-path /var/lib/ci/keys/ci_linux \
--timeout 180
```
Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target`
for WinRM guests.
### 4.3 Run the build (guest already up)
### 4.3 Run the build
The guest must already be up — capture its IP first (see
[§2](#find-a-running-guests-ip)). Windows guests use `--credential-target
BuildVMGuest` in place of the SSH flags. Repeatable `--extra-env KEY=VALUE`;
add `--xvfb` for Linux GUI builds or `--use-shared-cache` for the shared cache.
```bash
ci build run \
--ip-address 192.168.79.200 \
--ip-address "$IP" \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
--build-command "make all" \
--guest-artifact-source dist
```
Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`.
### 4.4 Collect artifacts
```bash
ci artifacts collect \
--ip-address 192.168.79.200 \
--ip-address "$IP" \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--guest-artifact-path dist \
@@ -221,166 +263,193 @@ ci artifacts collect \
```
### 4.5 Destroy the clone
```bash
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force # skip soft stop
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --force # skip the soft stop
```
### 4.6 Sweep orphans (also a maintenance task, §6)
### 4.6 Sweep orphans
Also a maintenance task — see [§6](#6-maintenance).
```bash
ci vm cleanup --what-if # list only
ci vm cleanup --what-if # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
```
### 4.7 Open a VM in the Workstation GUI (interactive debugging)
Launches the **interactive** VMware Workstation GUI (not the headless vmrun path)
so you can watch/poke a clone by hand.
### 4.7 Open a VM in the Workstation GUI
Launches the **interactive** VMware Workstation GUI (not the headless vmrun
path) so you can watch or poke a clone by hand. On a normal desktop session it
just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`:
On a normal desktop session it just works — `sudo` already forwards `DISPLAY`,
and `vmware` behaves exactly like `sudo -u ci-runner vmware`:
```bash
ci vmware # bare GUI, no VM
ci vmware # bare GUI, no VM
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vmware --vmx <path> --power-on # power it on (-x)
ci vmware --vmx <path> --fullscreen # power on + fullscreen (-X)
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --power-on # power on (-x)
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --fullscreen # power on + fullscreen (-X)
```
Other flags: `--new-window` (`-n`), `--display :0` (force a display if `$DISPLAY`
isn't forwarded), `--vmware-path`. Spawns detached; prints PID + log path.
**Why it works**: VMware connects via the X **host ACL**, not the cookie. The
inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by
ci-runner, so `vmware` drops it and relies on the xhost grant — which is normally
already present (`xhost` lists `SI:localuser:ci-runner`).
Other flags: `--new-window` (`-n`), `--display :0` (force a display when
`$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the
command prints its PID and a log path under `/tmp/vmware-*.log`.
**How the display works.** VMware connects through the X **host ACL**, not the
cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is
unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant —
normally already present (`xhost` lists `SI:localuser:ci-runner`).
**Making the grant persistent.** An xhost grant is per-session and is wiped on
logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart
entry in the desktop user's home that re-applies it at every graphical login.
You can also (re)run that step standalone:
The xhost grant is **per-session**: it is wiped on logout / X restart. To make
it persistent, `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in
the desktop user's home that re-applies the grant at every graphical login. You
can (re)run it standalone:
```bash
# As root (or via sudo); --desktop-user defaults to $SUDO_USER
cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# Remove the grant + autostart entry:
# To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall
```
This applies the grant immediately (if a display is reachable) and writes
This applies the grant immediately (when a display is reachable) and writes
`~simone/.config/autostart/ci-runner-xhost.desktop` for future logins.
**One-off, without the script**: add the grant from the logged-in desktop user
(not ci-runner):
**One-off, without the script.** Add the grant from the logged-in desktop user
(not `ci-runner`):
```bash
xhost +SI:localuser:ci-runner
```
`vmware` pre-flights the display (`xdpyinfo`) and prints this exact command on
failure instead of dying silently; the GUI's output is captured in `/tmp/vmware-*.log`.
`vmware` pre-flights the display with `xdpyinfo` and prints this exact command
on failure rather than dying silently.
---
## 5. Credentials & template provisioning
### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1)
### `creds set`
Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`).
The password never appears on the command line — verify it afterwards with the
`validate host` keyring check.
```bash
ci creds set --target BuildVMGuest --user ci_build # hidden prompt
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
```
Password is never on the command line. Verify via `validate host` keyring check.
(Leading space before `printf` keeps the secret out of shell history.)
> The leading space before `printf` keeps the secret out of shell history.
### Template management
```bash
# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux
# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux
# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win
# Timestamped backup of the template directories
ci template backup
```
Run `ci template <sub> --help` for the (many) per-template options.
```bash
ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04)
ci template prepare-linux # provision an existing Linux template over SSH
ci template prepare-win # provision a Windows template over WinRM (Linux-host driven)
ci template backup # timestamped backup of the template directories
```
---
## 6. Maintenance
### `retention run` — purge old artifacts & logs
### `retention run`
Purges old artifacts and logs. Always dry-run first. Retention switches to the
aggressive window automatically when free space drops below `--min-free-gb`.
```bash
ci retention run --what-if # dry run first
ci retention run --what-if # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
```
Switches to aggressive retention automatically when free space drops below
`--min-free-gb`.
### Orphan sweep — see §4.6 (`vm cleanup`).
### Orphan sweep
See [§4.6](#46-sweep-orphans) (`ci vm cleanup`).
---
## 7. Phase C testing (burn-in / benchmark / smoke)
## 7. Phase C testing
Burn-in, benchmark, and smoke checks — all native Python, no pwsh.
### `smoke run`
One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts:
exit `0`, an artifact directory under `/var/lib/ci/artifacts/<job-id>/`, and a
`job/success` event in `invoke-ci.jsonl`.
### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1)
```bash
ci smoke run --guest-os linux # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E
ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end
```
Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts/<job-id>/` +
`job/success` event in `invoke-ci.jsonl`.
### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1)
### `bench measure`
Times the clone / start / IP / transport / destroy phases and appends a row to
`benchmark.jsonl` (legacy field names preserved for trend continuity).
```bash
ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4 # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
```
Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl`
(legacy field names preserved for trend continuity).
### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1)
### `bench run`
Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts
that all jobs exit `0`, no orphan clone directory is left behind, and no stale
`vm-start.lock` remains. The JSON report lands under
`/var/lib/ci/artifacts/bench/<timestamp>.json`.
```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6)
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
```
Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`.
JSON report under `/var/lib/ci/artifacts/bench/<ts>.json`.
---
## 8. Cutover checklist & uninstalling pwsh
Repeat for ≥1 week with pwsh **still installed** (A/B safety net):
Repeat for at least one week with pwsh **still installed** as an A/B safety net:
- [ ] Normal `git push` CI pipeline green.
- [ ] `ci validate host` green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks).
- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`.
- [ ] `ci bench measure` timings comparable to historical baselines.
- [ ] The normal `git push` CI pipeline stays green.
- [ ] `ci validate host` is green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks).
- [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`.
- [ ] `ci bench measure` timings are comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
Then — only after a fully green week:
Only after a fully green week:
```bash
pwsh --version
sudo apt-get remove powershell
ci validate host # must still exit 0 without pwsh
```
Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original
`.ps1` scripts remain in `scripts/`.
To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the
original `.ps1` scripts remain in `scripts/archived/`.
---
## 9. Troubleshooting quick map
## 9. Troubleshooting
| Symptom | Cause | Fix |
| -------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------- |
| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` |
| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` |
| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP |
| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` |
| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) |
| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 |
| Symptom | Cause | Fix |
| ---------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------- |
| `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` |
| `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` |
| `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP |
| `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` |
| IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; the systemd guard rebuilds) |
| `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |