From f13b24956e217d04a546b02aa88b6ec7b85aae9a Mon Sep 17 00:00:00 2001 From: Simone Date: Mon, 8 Jun 2026 00:11:36 +0200 Subject: [PATCH] docs: add complete manual CLI runbook (all commands + ci alias + vm open) Rewrite docs/RUNBOOK-PhaseC.md as a full manual-command reference: every ci_orchestrator command with a working example as the ci-runner user, the `ci` alias setup, quick-reference table, manual VM lifecycle (incl. vm open GUI launch with the xhost/--display procedure), templating, maintenance, Phase C testing, cutover checklist and a troubleshooting map. Co-Authored-By: Claude Opus 4.8 --- docs/RUNBOOK-PhaseC.md | 343 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 343 insertions(+) create mode 100644 docs/RUNBOOK-PhaseC.md diff --git a/docs/RUNBOOK-PhaseC.md b/docs/RUNBOOK-PhaseC.md new file mode 100644 index 0000000..4e0c10c --- /dev/null +++ b/docs/RUNBOOK-PhaseC.md @@ -0,0 +1,343 @@ +# RUNBOOK — manual CLI reference (Linux host, pwsh-free) + +Simplified but **complete** reference for every `ci_orchestrator` command you can +run by hand on the Linux host. All commands run as the **`ci-runner`** service +user against the live config (`/var/lib/ci/config.toml`). + +Two notations are used: + +- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ` +- **Short form (alias):** `ci ` — after installing the helper (see §0). + +Examples below use the **short form**. Drop `ci` for the literal command. + +--- + +## 0. Setup (do once) + +### 0.1 Refresh the production venv (mandatory after a code change/merge) + +```bash +cd /opt/ci/local-ci-cd-system +sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md +sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds' +``` + +### 0.2 Install the `ci` alias + +```bash +./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent) +# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo) +# --uninstall remove it +source ~/.bashrc # or open a new shell +ci validate host # smoke test the alias +``` + +The script defines two helpers: + +| Helper | Runs as | Use for | +|---------|---------------|---------| +| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) | +| `ci-me` | current user | read-only checks where sudo is noise | + +### 0.3 Live environment + +| Thing | Value | +|-------|-------| +| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` | +| Config | `/var/lib/ci/config.toml` | +| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) | +| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) | +| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | +| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` | + +--- + +## 1. Quick reference — every command + +| Command | Does | Section | +|---------|------|---------| +| `ci validate host` | host prerequisites check (no VM) | §2 | +| `ci validate guest --host IP` | probe a running guest's transport | §2 | +| `ci monitor disk` | free-space alert | §2 | +| `ci monitor runner` | watch + auto-restart act_runner | §2 | +| `ci report job` | job summaries from JSONL logs | §2 | +| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 | +| `ci vm new ...` | linked-clone a template | §4 | +| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 | +| `ci build run ...` | run a build in a running guest | §4 | +| `ci artifacts collect ...` | copy artifacts out of a guest | §4 | +| `ci vm remove --vmx ...` | stop + delete one clone | §4 | +| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 | +| `ci vm open --vmx ...` | open a VMX in the VMware Workstation GUI | §4 | +| `ci creds set --user ...` | store guest credentials in keyring | §5 | +| `ci template deploy-linux` | build the Linux template VM | §5 | +| `ci template prepare-linux` | provision Linux template over SSH | §5 | +| `ci template prepare-win` | provision Windows template over WinRM | §5 | +| `ci template backup` | timestamped copy of template dirs | §5 | +| `ci retention run` | purge old artifacts/logs | §6 | +| `ci bench measure ...` | phase-timing benchmark | §7 | +| `ci bench run ...` | concurrency burn-in | §7 | +| `ci smoke run ...` | one E2E job + assertions | §7 | + +--- + +## 2. Health, monitoring & reporting (safe, no VM boot) + +### `validate host` — run this first, every day +```bash +ci validate host ; echo "exit=$?" +``` +Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner` +unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check. + +### `validate guest` — diagnose a running guest's transport +```bash +ci validate guest --host 192.168.79.200 --ssh --guest-os linux # SSH +ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM +``` +Connects + runs a trivial command, printing the explicit cause on failure +(connect vs auth vs command). The guest must already be booted. + +### `monitor disk` — free-space alert +```bash +ci monitor disk --min-free-gb 50 +ci monitor disk --min-free-gb 50 --json # machine-readable one-liner +ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK" +``` + +### `monitor runner` — keep act_runner alive +```bash +ci monitor runner --service-name act-runner --max-restarts 3 +``` +Restarts the runner up to N times per window; optional `--webhook-url`, +`--gitea-url`, `--gitea-credential-target`. + +### `report job` — read job history +```bash +ci report job --last 10 # 10 most recent, table +ci report job --failed # only failures +ci report job --job-id smoke-20260607-2030 # phase breakdown for one job +ci report job --last 20 --json # JSON out +``` + +--- + +## 3. Full pipeline (`job`) — the normal way to run a build + +One command does everything: clone → wait-ready → build → collect → destroy. +This is what the runner invokes per push; you can run it by hand too. + +```bash +# Linux build, in-guest git clone, no artifact packaging +ci job \ + --job-id manual-$(date +%s) \ + --repo-url https://gitea.emulab.it/Simone/myrepo.git \ + --branch main \ + --guest-os linux \ + --build-command "make all" \ + --use-git-clone +``` + +Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`, +`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`, +`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`, +`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`, +`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`. + +> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`. + +--- + +## 4. Manual VM lifecycle (the steps `job` runs internally) + +Use these to drive a build stage-by-stage, e.g. when debugging a single phase. + +### 4.1 Clone a VM +```bash +ci vm new \ + --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ + --snapshot BaseClean-Linux \ + --clone-base-dir /var/lib/ci/build-vms \ + --job-id dbg-1 \ + --guest-os linux \ + --start # prints the clone VMX path on stdout +``` + +### 4.2 Wait until reachable +```bash +ci wait-ready \ + --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \ + --guest-os linux \ + --ssh-user ci_build \ + --ssh-key-path /var/lib/ci/keys/ci_linux \ + --timeout 180 +``` +Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target` +for WinRM guests. + +### 4.3 Run the build (guest already up) +```bash +ci build run \ + --ip-address 192.168.79.200 \ + --guest-os linux \ + --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ + --clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \ + --build-command "make all" \ + --guest-artifact-source dist +``` +Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags. +Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`. + +### 4.4 Collect artifacts +```bash +ci artifacts collect \ + --ip-address 192.168.79.200 \ + --guest-os linux \ + --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ + --guest-artifact-path dist \ + --host-artifact-dir /var/lib/ci/artifacts/dbg-1 \ + --include-logs --job-id dbg-1 +``` + +### 4.5 Destroy the clone +```bash +ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx +ci vm remove --vmx --force # skip soft stop +``` + +### 4.6 Sweep orphans (also a maintenance task, §6) +```bash +ci vm cleanup --what-if # list only +ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock +``` + +### 4.7 Open a VM in the Workstation GUI (interactive debugging) +Launches the **interactive** VMware Workstation GUI (not the headless vmrun path) +so you can watch/poke a clone by hand. + +```bash +# authorise the service user on your desktop X session first (run as YOUR user): +xhost +SI:localuser:ci-runner + +# then open the VMX as ci-runner, forcing the display: +ci vm open --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --display :0 +ci vm open --vmx --display :0 --power-on # also power it on (-x) +ci vm open --vmx --display :0 --fullscreen # power on + fullscreen (-X) +``` +The GUI needs an X display; `sudo -u ci-runner` strips `$DISPLAY`, so pass +`--display :0` (or export `DISPLAY`). Other flags: `--new-window` (`-n`), +`--vmware-path` to override the binary. The command spawns the GUI detached and +returns immediately (prints the child PID). + +--- + +## 5. Credentials & template provisioning + +### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1) +```bash +ci creds set --target BuildVMGuest --user ci_build # hidden prompt + printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin +``` +Password is never on the command line. Verify via `validate host` keyring check. +(Leading space before `printf` keeps the secret out of shell history.) + +### Template management +```bash +# Build the Linux template VM from scratch (Ubuntu 24.04) +ci template deploy-linux + +# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep) +ci template prepare-linux + +# Provision a Windows template over WinRM (Linux-host driven) +ci template prepare-win + +# Timestamped backup of the template directories +ci template backup +``` +Run `ci template --help` for the (many) per-template options. + +--- + +## 6. Maintenance + +### `retention run` — purge old artifacts & logs +```bash +ci retention run --what-if # dry run first +ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50 +``` +Switches to aggressive retention automatically when free space drops below +`--min-free-gb`. + +### Orphan sweep — see §4.6 (`vm cleanup`). + +--- + +## 7. Phase C testing (burn-in / benchmark / smoke) + +### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1) +```bash +ci smoke run --guest-os linux # no-op marker job (fastest) +ci smoke run --guest-os windows +ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E +``` +Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts//` + +`job/success` event in `invoke-ci.jsonl`. + +### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1) +```bash +ci bench measure --guest-os linux --iterations 1 +ci bench measure --guest-os linux --iterations 4 # match historical baselines +sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl +``` +Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl` +(legacy field names preserved for trend continuity). + +### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1) +```bash +ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence +ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate +ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6) +sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)" +``` +Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`. +JSON report under `/var/lib/ci/artifacts/bench/.json`. + +--- + +## 8. Cutover checklist & uninstalling pwsh + +Repeat for ≥1 week with pwsh **still installed** (A/B safety net): + +- [ ] Normal `git push` CI pipeline green. +- [ ] `ci validate host` green daily. +- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks). +- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`. +- [ ] `ci bench measure` timings comparable to historical baselines. +- [ ] No procedure forced a fallback to a `.ps1` on the Linux host. + +Then — only after a fully green week: + +```bash +pwsh --version +sudo apt-get remove powershell +ci validate host # must still exit 0 without pwsh +``` +Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original +`.ps1` scripts remain in `scripts/`. + +--- + +## 9. Troubleshooting quick map + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` | +| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` | +| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP | +| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` | +| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` | +| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | +| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) | +| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 |