From 027e2ff4b6fd46c084ac96058bd52b5559e300c0 Mon Sep 17 00:00:00 2001 From: Simone Date: Mon, 8 Jun 2026 21:31:25 +0200 Subject: [PATCH] docs(runbook): rewrite Phase C runbook for clarity Replace the terse, abbreviation-heavy style with clean prose and consistent formatting. Make command blocks copy-paste-ready on this host: capture the guest IP into a shell variable for `validate guest` / `build run`, expand globbed VMX paths, and group the `job` flags into readable lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note at scripts/archived/. No commands or flags changed. --- docs/RUNBOOK-PhaseC.md | 449 ++++++++++++++++++++++++----------------- 1 file changed, 259 insertions(+), 190 deletions(-) diff --git a/docs/RUNBOOK-PhaseC.md b/docs/RUNBOOK-PhaseC.md index 23bc20f..c1785cf 100644 --- a/docs/RUNBOOK-PhaseC.md +++ b/docs/RUNBOOK-PhaseC.md @@ -1,155 +1,182 @@ -# RUNBOOK — manual CLI reference (Linux host, pwsh-free) +# Runbook — Manual CLI Reference (Linux host, pwsh-free) -Simplified but **complete** reference for every `ci_orchestrator` command you can -run by hand on the Linux host. All commands run as the **`ci-runner`** service -user against the live config (`/var/lib/ci/config.toml`). +A complete, hands-on reference for every `ci_orchestrator` command you can run +by hand on the Linux host. Unless noted otherwise, commands run as the +**`ci-runner`** service user against the live configuration at +`/var/lib/ci/config.toml`. -Two notations are used: +Every command can be written two ways: -- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ` -- **Short form (alias):** `ci ` — after installing the helper (see §0). +- **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ` +- **Short form** — `ci `, after installing the alias (see [§0](#0-setup-do-once)). -Examples below use the **short form**. Drop `ci` for the literal command. +The examples use the short form. The paths and template names below are the +real ones on this host, so most blocks are copy-paste-ready as written. --- ## 0. Setup (do once) -### 0.1 Refresh the production venv (mandatory after a code change/merge) +### 0.1 Refresh the production venv + +Mandatory after every code change or merge — the venv is installed +non-editable, so changes only take effect once reinstalled: ```bash cd /opt/ci/local-ci-cd-system -sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md +sudo /opt/ci/venv/bin/python -m pip install . sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds' ``` ### 0.2 Install the `ci` alias ```bash -./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent) -# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo) -# --uninstall remove it -source ~/.bashrc # or open a new shell -ci validate host # smoke test the alias +cd /opt/ci/local-ci-cd-system +./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent) +source ~/.bashrc # or open a new shell +ci validate host # smoke-test the alias ``` -The script defines two helpers: +Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs +sudo); `--uninstall` removes the block. The script defines two helpers: -| Helper | Runs as | Use for | -| ------- | ------------------ | --------------------------------------- | -| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) | -| `ci-me` | current user | read-only checks where sudo is noise | +| Helper | Runs as | Use for | +| ------- | ------------------ | ----------------------------------------- | +| `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs | +| `ci-me` | the current user | read-only checks where sudo is just noise | -### 0.3 Live environment +### 0.3 The live environment -| Thing | Value | +| Item | Value | | ------------------ | ---------------------------------------------------------------------------------- | -| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` | -| Config | `/var/lib/ci/config.toml` | -| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) | -| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) | -| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | -| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` | +| Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` | +| Configuration | `/var/lib/ci/config.toml` | +| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) | +| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) | +| Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | +| Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` | --- -## 1. Quick reference — every command +## 1. Command index -| Command | Does | Section | +| Command | Purpose | Section | | ----------------------------- | --------------------------------------------- | ------- | -| `ci validate host` | host prerequisites check (no VM) | §2 | -| `ci validate guest --host IP` | probe a running guest's transport | §2 | -| `ci monitor disk` | free-space alert | §2 | -| `ci monitor runner` | watch + auto-restart act_runner | §2 | -| `ci report job` | job summaries from JSONL logs | §2 | -| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 | -| `ci vm new ...` | linked-clone a template | §4 | -| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 | -| `ci build run ...` | run a build in a running guest | §4 | -| `ci artifacts collect ...` | copy artifacts out of a guest | §4 | -| `ci vm remove --vmx ...` | stop + delete one clone | §4 | -| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 | -| `ci vmware --vmx ...` | open a VMX in the VMware Workstation GUI | §4 | -| `ci creds set --user ...` | store guest credentials in keyring | §5 | -| `ci template deploy-linux` | build the Linux template VM | §5 | -| `ci template prepare-linux` | provision Linux template over SSH | §5 | -| `ci template prepare-win` | provision Windows template over WinRM | §5 | -| `ci template backup` | timestamped copy of template dirs | §5 | -| `ci retention run` | purge old artifacts/logs | §6 | -| `ci bench measure ...` | phase-timing benchmark | §7 | -| `ci bench run ...` | concurrency burn-in | §7 | -| `ci smoke run ...` | one E2E job + assertions | §7 | +| `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) | +| `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) | +| `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) | +| `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) | +| `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) | +| `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) | +| `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) | +| `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) | +| `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) | +| `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) | +| `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) | +| `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) | +| `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) | +| `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) | +| `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) | +| `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) | +| `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) | +| `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) | +| `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) | +| `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) | +| `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) | +| `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) | --- -## 2. Health, monitoring & reporting (safe, no VM boot) +## 2. Health, monitoring & reporting + +These are safe — none of them boots a VM. + +### `validate host` + +Run this first, every day. It checks vmrun, template snapshots, the keyring, +`/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0` +with every line marked `[OK ]`; any `[FAIL]` line names the broken check. -### `validate host` — run this first, every day ```bash ci validate host ; echo "exit=$?" ``` -Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner` -unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check. -### Find a running guest's IP (needed for `validate guest`) -`vmrun getGuestIPAddress` reads the IP from VMware Tools — the VM must be -**powered on** and Tools running. Run it as `ci-runner` (the VMs are started by -that user, so its `vmrun` inventory is the one that sees them): +### Find a running guest's IP + +`vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must +be **powered on** with Tools running. Run it as `ci-runner` — the VMs are +started by that user, so its `vmrun` inventory is the one that can see them. + ```bash -# Linux template -sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx -# Windows template -sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx -# A build clone (glob the job dir) -sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx - -# List which VMs are powered on first +# Which VMs are powered on right now sudo -u ci-runner vmrun list -``` -> **Gotcha (AGENTS.md #1):** `getGuestIPAddress` can exit `0` and print -> `Error: The VMware Tools are not running...` on stdout when Tools are down. -> A line starting with `Error:` means *no IP yet* — wait for boot, don't parse it. -### `validate guest` — diagnose a running guest's transport +# IP of the Linux template +sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx + +# IP of a Windows template +sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx + +# IP of a build clone (glob the job directory) +sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx +``` + +> **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print +> `Error: The VMware Tools are not running...` on stdout when Tools are down. A +> line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it. + +### `validate guest` + +Connects to a booted guest, runs a trivial command, and reports the explicit +cause on failure (connect vs. auth vs. command). Capture the IP into a variable +so the probe is genuinely copy-paste-ready: + ```bash -ci validate guest --host 192.168.79.128 --ssh --guest-os linux # SSH (Linux) -ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM (Windows) +# Linux guest over SSH +IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx) +ci validate guest --host "$IP" --ssh --guest-os linux + +# Windows guest over WinRM +IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx) +ci validate guest --host "$IP" --winrm --guest-os windows ``` -Connects + runs a trivial command, printing the explicit cause on failure -(connect vs auth vs command). The guest must already be booted — get its IP -with `getGuestIPAddress` above. + +### `monitor disk` -### `monitor disk` — free-space alert ```bash ci monitor disk --min-free-gb 50 -ci monitor disk --min-free-gb 50 --json # machine-readable one-liner +ci monitor disk --min-free-gb 50 --json # machine-readable one-liner ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK" ``` -### `monitor runner` — keep act_runner alive +### `monitor runner` + +Restarts the runner up to N times per window. Optional: `--webhook-url`, +`--gitea-url`, `--gitea-credential-target`. + ```bash ci monitor runner --service-name act-runner --max-restarts 3 ``` -Restarts the runner up to N times per window; optional `--webhook-url`, -`--gitea-url`, `--gitea-credential-target`. -### `report job` — read job history +### `report job` + ```bash -ci report job --last 10 # 10 most recent, table -ci report job --failed # only failures -ci report job --job-id smoke-20260607-2030 # phase breakdown for one job -ci report job --last 20 --json # JSON out +ci report job --last 10 # 10 most recent, as a table +ci report job --failed # failures only +ci report job --job-id smoke-20260607-2030 # phase breakdown for one job +ci report job --last 20 --json # JSON output ``` --- -## 3. Full pipeline (`job`) — the normal way to run a build +## 3. Full pipeline (`job`) -One command does everything: clone → wait-ready → build → collect → destroy. -This is what the runner invokes per push; you can run it by hand too. +This is the normal way to run a build: a single command clones the template, +waits for the guest, runs the build, collects artifacts, and destroys the +clone. It is exactly what the runner invokes on each push, and you can run it +by hand too. ```bash -# Linux build, in-guest git clone, no artifact packaging ci job \ --job-id manual-$(date +%s) \ --repo-url https://gitea.emulab.it/Simone/myrepo.git \ @@ -159,21 +186,29 @@ ci job \ --use-git-clone ``` -Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`, -`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`, -`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`, -`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`, -`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`. +`--job-id`, `--repo-url`, and `--branch` are required. Other useful flags: -> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`. +- **Source**: `--commit`, `--submodules` / `--no-submodules`, + `--use-git-clone` / `--host-clone`, `--use-shared-cache` +- **Build**: `--configuration Release`, `--guest-artifact-source dist`, + `--skip-artifact`, `--xvfb` (Linux GUI builds), + `--extra-env-json '{"KEY":"VALUE"}'` +- **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`, + `--snapshot-name`, `--ready-timeout` +- **Auth**: `--gitea-credential-target` (private repos) + +> For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing)) +> over hand-rolling a `job` invocation. --- -## 4. Manual VM lifecycle (the steps `job` runs internally) +## 4. Manual VM lifecycle -Use these to drive a build stage-by-stage, e.g. when debugging a single phase. +These are the stages that `job` runs internally. Drive them individually when +debugging a single phase. ### 4.1 Clone a VM + ```bash ci vm new \ --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ @@ -181,10 +216,14 @@ ci vm new \ --clone-base-dir /var/lib/ci/build-vms \ --job-id dbg-1 \ --guest-os linux \ - --start # prints the clone VMX path on stdout + --start # prints the clone's VMX path on stdout ``` ### 4.2 Wait until reachable + +Prints the guest IP once ready. Use `--ip-address` to skip polling, or +`--credential-target` for WinRM guests. + ```bash ci wait-ready \ --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \ @@ -193,26 +232,29 @@ ci wait-ready \ --ssh-key-path /var/lib/ci/keys/ci_linux \ --timeout 180 ``` -Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target` -for WinRM guests. -### 4.3 Run the build (guest already up) +### 4.3 Run the build + +The guest must already be up — capture its IP first (see +[§2](#find-a-running-guests-ip)). Windows guests use `--credential-target +BuildVMGuest` in place of the SSH flags. Repeatable `--extra-env KEY=VALUE`; +add `--xvfb` for Linux GUI builds or `--use-shared-cache` for the shared cache. + ```bash ci build run \ - --ip-address 192.168.79.200 \ + --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \ --build-command "make all" \ --guest-artifact-source dist ``` -Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags. -Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`. ### 4.4 Collect artifacts + ```bash ci artifacts collect \ - --ip-address 192.168.79.200 \ + --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --guest-artifact-path dist \ @@ -221,166 +263,193 @@ ci artifacts collect \ ``` ### 4.5 Destroy the clone + ```bash ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx -ci vm remove --vmx --force # skip soft stop +ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --force # skip the soft stop ``` -### 4.6 Sweep orphans (also a maintenance task, §6) +### 4.6 Sweep orphans + +Also a maintenance task — see [§6](#6-maintenance). + ```bash -ci vm cleanup --what-if # list only +ci vm cleanup --what-if # list only ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock ``` -### 4.7 Open a VM in the Workstation GUI (interactive debugging) -Launches the **interactive** VMware Workstation GUI (not the headless vmrun path) -so you can watch/poke a clone by hand. +### 4.7 Open a VM in the Workstation GUI + +Launches the **interactive** VMware Workstation GUI (not the headless vmrun +path) so you can watch or poke a clone by hand. On a normal desktop session it +just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`: -On a normal desktop session it just works — `sudo` already forwards `DISPLAY`, -and `vmware` behaves exactly like `sudo -u ci-runner vmware`: ```bash -ci vmware # bare GUI, no VM +ci vmware # bare GUI, no VM ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx -ci vmware --vmx --power-on # power it on (-x) -ci vmware --vmx --fullscreen # power on + fullscreen (-X) +ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --power-on # power on (-x) +ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --fullscreen # power on + fullscreen (-X) ``` -Other flags: `--new-window` (`-n`), `--display :0` (force a display if `$DISPLAY` -isn't forwarded), `--vmware-path`. Spawns detached; prints PID + log path. -**Why it works**: VMware connects via the X **host ACL**, not the cookie. The -inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by -ci-runner, so `vmware` drops it and relies on the xhost grant — which is normally -already present (`xhost` lists `SI:localuser:ci-runner`). +Other flags: `--new-window` (`-n`), `--display :0` (force a display when +`$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the +command prints its PID and a log path under `/tmp/vmware-*.log`. + +**How the display works.** VMware connects through the X **host ACL**, not the +cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is +unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant — +normally already present (`xhost` lists `SI:localuser:ci-runner`). + +**Making the grant persistent.** An xhost grant is per-session and is wiped on +logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart +entry in the desktop user's home that re-applies it at every graphical login. +You can also (re)run that step standalone: -The xhost grant is **per-session**: it is wiped on logout / X restart. To make -it persistent, `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in -the desktop user's home that re-applies the grant at every graphical login. You -can (re)run it standalone: ```bash -# As root (or via sudo); --desktop-user defaults to $SUDO_USER +cd /opt/ci/local-ci-cd-system +# --desktop-user defaults to $SUDO_USER sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone -# Remove the grant + autostart entry: +# To remove the grant and the autostart entry: sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall ``` -This applies the grant immediately (if a display is reachable) and writes + +This applies the grant immediately (when a display is reachable) and writes `~simone/.config/autostart/ci-runner-xhost.desktop` for future logins. -**One-off, without the script**: add the grant from the logged-in desktop user -(not ci-runner): +**One-off, without the script.** Add the grant from the logged-in desktop user +(not `ci-runner`): + ```bash xhost +SI:localuser:ci-runner ``` -`vmware` pre-flights the display (`xdpyinfo`) and prints this exact command on -failure instead of dying silently; the GUI's output is captured in `/tmp/vmware-*.log`. + +`vmware` pre-flights the display with `xdpyinfo` and prints this exact command +on failure rather than dying silently. --- ## 5. Credentials & template provisioning -### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1) +### `creds set` + +Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`). +The password never appears on the command line — verify it afterwards with the +`validate host` keyring check. + ```bash ci creds set --target BuildVMGuest --user ci_build # hidden prompt printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin ``` -Password is never on the command line. Verify via `validate host` keyring check. -(Leading space before `printf` keeps the secret out of shell history.) + +> The leading space before `printf` keeps the secret out of shell history. ### Template management -```bash -# Build the Linux template VM from scratch (Ubuntu 24.04) -ci template deploy-linux -# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep) -ci template prepare-linux - -# Provision a Windows template over WinRM (Linux-host driven) -ci template prepare-win - -# Timestamped backup of the template directories -ci template backup -``` Run `ci template --help` for the (many) per-template options. +```bash +ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04) +ci template prepare-linux # provision an existing Linux template over SSH +ci template prepare-win # provision a Windows template over WinRM (Linux-host driven) +ci template backup # timestamped backup of the template directories +``` + --- ## 6. Maintenance -### `retention run` — purge old artifacts & logs +### `retention run` + +Purges old artifacts and logs. Always dry-run first. Retention switches to the +aggressive window automatically when free space drops below `--min-free-gb`. + ```bash -ci retention run --what-if # dry run first +ci retention run --what-if # dry run ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50 ``` -Switches to aggressive retention automatically when free space drops below -`--min-free-gb`. -### Orphan sweep — see §4.6 (`vm cleanup`). +### Orphan sweep + +See [§4.6](#46-sweep-orphans) (`ci vm cleanup`). --- -## 7. Phase C testing (burn-in / benchmark / smoke) +## 7. Phase C testing + +Burn-in, benchmark, and smoke checks — all native Python, no pwsh. + +### `smoke run` + +One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts: +exit `0`, an artifact directory under `/var/lib/ci/artifacts//`, and a +`job/success` event in `invoke-ci.jsonl`. -### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1) ```bash ci smoke run --guest-os linux # no-op marker job (fastest) ci smoke run --guest-os windows -ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E +ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end ``` -Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts//` + -`job/success` event in `invoke-ci.jsonl`. -### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1) +### `bench measure` + +Times the clone / start / IP / transport / destroy phases and appends a row to +`benchmark.jsonl` (legacy field names preserved for trend continuity). + ```bash ci bench measure --guest-os linux --iterations 1 ci bench measure --guest-os linux --iterations 4 # match historical baselines sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl ``` -Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl` -(legacy field names preserved for trend continuity). -### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1) +### `bench run` + +Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts +that all jobs exit `0`, no orphan clone directory is left behind, and no stale +`vm-start.lock` remains. The JSON report lands under +`/var/lib/ci/artifacts/bench/.json`. + ```bash -ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence -ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate -ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6) +ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence +ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate +ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6) sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)" ``` -Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`. -JSON report under `/var/lib/ci/artifacts/bench/.json`. --- ## 8. Cutover checklist & uninstalling pwsh -Repeat for ≥1 week with pwsh **still installed** (A/B safety net): +Repeat for at least one week with pwsh **still installed** as an A/B safety net: -- [ ] Normal `git push` CI pipeline green. -- [ ] `ci validate host` green daily. -- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks). -- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`. -- [ ] `ci bench measure` timings comparable to historical baselines. +- [ ] The normal `git push` CI pipeline stays green. +- [ ] `ci validate host` is green daily. +- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks). +- [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`. +- [ ] `ci bench measure` timings are comparable to historical baselines. - [ ] No procedure forced a fallback to a `.ps1` on the Linux host. -Then — only after a fully green week: +Only after a fully green week: ```bash pwsh --version sudo apt-get remove powershell ci validate host # must still exit 0 without pwsh ``` -Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original -`.ps1` scripts remain in `scripts/`. + +To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the +original `.ps1` scripts remain in `scripts/archived/`. --- -## 9. Troubleshooting quick map +## 9. Troubleshooting -| Symptom | Cause | Fix | -| -------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------- | -| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` | -| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` | -| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP | -| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` | -| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` | -| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | -| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) | -| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 | +| Symptom | Cause | Fix | +| ---------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------- | +| `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` | +| `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` | +| `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP | +| `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` | +| WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` | +| IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | +| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; the systemd guard rebuilds) | +| `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |