Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(runbook): rewrite Phase C runbook for clarity

Browse Source 
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
This commit is contained in:
Simone
2026-06-08 21:31:25 +02:00
parent c274d334d3
commit 027e2ff4b6
1 changed files with 259 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/RUNBOOK-PhaseC.md
+259 -190
View File
@@ -1,155 +1,182 @@
# RUNBOOK — manual CLI reference (Linux host, pwsh-free) # Runbook — Manual CLI Reference (Linux host, pwsh-free)
Simplified but **complete** reference for every `ci_orchestrator` command you can A complete, hands-on reference for every `ci_orchestrator` command you can run
run by hand on the Linux host. All commands run as the **`ci-runner`** service by hand on the Linux host. Unless noted otherwise, commands run as the
user against the live config (`/var/lib/ci/config.toml`). **`ci-runner`** service user against the live configuration at
`/var/lib/ci/config.toml`.
Two notations are used: Every command can be written two ways:
- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>` - **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form (alias):** `ci <cmd>` — after installing the helper (see §0). - **Short form** — `ci <cmd>`, after installing the alias (see [§0](#0-setup-do-once)).
Examples below use the **short form**. Drop `ci` for the literal command. The examples use the short form. The paths and template names below are the
real ones on this host, so most blocks are copy-paste-ready as written.
--- ---
## 0. Setup (do once) ## 0. Setup (do once)
### 0.1 Refresh the production venv (mandatory after a code change/merge) ### 0.1 Refresh the production venv
Mandatory after every code change or merge — the venv is installed
non-editable, so changes only take effect once reinstalled:
```bash ```bash
cd /opt/ci/local-ci-cd-system cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md sudo /opt/ci/venv/bin/python -m pip install .
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds' sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
``` ```
### 0.2 Install the `ci` alias ### 0.2 Install the `ci` alias
```bash ```bash
./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent) cd /opt/ci/local-ci-cd-system
# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo) ./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent)
# --uninstall remove it source ~/.bashrc # or open a new shell
source ~/.bashrc # or open a new shell ci validate host # smoke-test the alias
ci validate host # smoke test the alias
``` ```
The script defines two helpers: Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs
sudo); `--uninstall` removes the block. The script defines two helpers:
| Helper | Runs as | Use for | | Helper | Runs as | Use for |
| ------- | ------------------ | --------------------------------------- | | ------- | ------------------ | ----------------------------------------- |
| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) | | `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs |
| `ci-me` | current user | read-only checks where sudo is noise | | `ci-me` | the current user | read-only checks where sudo is just noise |
### 0.3 Live environment ### 0.3 The live environment
| Thing | Value | | Item | Value |
| ------------------ | ---------------------------------------------------------------------------------- | | ------------------ | ---------------------------------------------------------------------------------- |
| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` | | Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` |
| Config | `/var/lib/ci/config.toml` | | Configuration | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) | | Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) | | Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) |
| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | | Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` | | Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` |
--- ---
## 1. Quick reference — every command ## 1. Command index
| Command | Does | Section | | Command | Purpose | Section |
| ----------------------------- | --------------------------------------------- | ------- | | ----------------------------- | --------------------------------------------- | ------- |
| `ci validate host` | host prerequisites check (no VM) | §2 | | `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) |
| `ci validate guest --host IP` | probe a running guest's transport | §2 | | `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) |
| `ci monitor disk` | free-space alert | §2 | | `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) |
| `ci monitor runner` | watch + auto-restart act_runner | §2 | | `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) |
| `ci report job` | job summaries from JSONL logs | §2 | | `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) |
| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 | | `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) |
| `ci vm new ...` | linked-clone a template | §4 | | `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) |
| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 | | `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) |
| `ci build run ...` | run a build in a running guest | §4 | | `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) |
| `ci artifacts collect ...` | copy artifacts out of a guest | §4 | | `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) |
| `ci vm remove --vmx ...` | stop + delete one clone | §4 | | `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) |
| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 | | `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) |
| `ci vmware --vmx ...` | open a VMX in the VMware Workstation GUI | §4 | | `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) |
| `ci creds set --user ...` | store guest credentials in keyring | §5 | | `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) |
| `ci template deploy-linux` | build the Linux template VM | §5 | | `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-linux` | provision Linux template over SSH | §5 | | `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) |
| `ci template prepare-win` | provision Windows template over WinRM | §5 | | `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) |
| `ci template backup` | timestamped copy of template dirs | §5 | | `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) |
| `ci retention run` | purge old artifacts/logs | §6 | | `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) |
| `ci bench measure ...` | phase-timing benchmark | §7 | | `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) |
| `ci bench run ...` | concurrency burn-in | §7 | | `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) |
| `ci smoke run ...` | one E2E job + assertions | §7 | | `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) |
--- ---
## 2. Health, monitoring & reporting (safe, no VM boot) ## 2. Health, monitoring & reporting
These are safe — none of them boots a VM.
### `validate host`
Run this first, every day. It checks vmrun, template snapshots, the keyring,
`/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0`
with every line marked `[OK ]`; any `[FAIL]` line names the broken check.
### `validate host` — run this first, every day
```bash ```bash
ci validate host ; echo "exit=$?" ci validate host ; echo "exit=$?"
``` ```
Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner`
unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check.
### Find a running guest's IP (needed for `validate guest`) ### Find a running guest's IP
`vmrun getGuestIPAddress` reads the IP from VMware Tools — the VM must be
**powered on** and Tools running. Run it as `ci-runner` (the VMs are started by `vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must
that user, so its `vmrun` inventory is the one that sees them): be **powered on** with Tools running. Run it as `ci-runner` — the VMs are
started by that user, so its `vmrun` inventory is the one that can see them.
```bash ```bash
# Linux template # Which VMs are powered on right now
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# A build clone (glob the job dir)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
# List which VMs are powered on first
sudo -u ci-runner vmrun list sudo -u ci-runner vmrun list
```
> **Gotcha (AGENTS.md #1):** `getGuestIPAddress` can exit `0` and print
> `Error: The VMware Tools are not running...` on stdout when Tools are down.
> A line starting with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest` — diagnose a running guest's transport # IP of the Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# IP of a Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# IP of a build clone (glob the job directory)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
```
> **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print
> `Error: The VMware Tools are not running...` on stdout when Tools are down. A
> line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest`
Connects to a booted guest, runs a trivial command, and reports the explicit
cause on failure (connect vs. auth vs. command). Capture the IP into a variable
so the probe is genuinely copy-paste-ready:
```bash ```bash
ci validate guest --host 192.168.79.128 --ssh --guest-os linux # SSH (Linux) # Linux guest over SSH
ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM (Windows) IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
ci validate guest --host "$IP" --ssh --guest-os linux
# Windows guest over WinRM
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
ci validate guest --host "$IP" --winrm --guest-os windows
``` ```
Connects + runs a trivial command, printing the explicit cause on failure
(connect vs auth vs command). The guest must already be booted — get its IP ### `monitor disk`
with `getGuestIPAddress` above.
### `monitor disk` — free-space alert
```bash ```bash
ci monitor disk --min-free-gb 50 ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK" ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
``` ```
### `monitor runner` — keep act_runner alive ### `monitor runner`
Restarts the runner up to N times per window. Optional: `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
```bash ```bash
ci monitor runner --service-name act-runner --max-restarts 3 ci monitor runner --service-name act-runner --max-restarts 3
``` ```
Restarts the runner up to N times per window; optional `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
### `report job` — read job history ### `report job`
```bash ```bash
ci report job --last 10 # 10 most recent, table ci report job --last 10 # 10 most recent, as a table
ci report job --failed # only failures ci report job --failed # failures only
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON out ci report job --last 20 --json # JSON output
``` ```
--- ---
## 3. Full pipeline (`job`) — the normal way to run a build ## 3. Full pipeline (`job`)
One command does everything: clone → wait-ready → build → collect → destroy. This is the normal way to run a build: a single command clones the template,
This is what the runner invokes per push; you can run it by hand too. waits for the guest, runs the build, collects artifacts, and destroys the
clone. It is exactly what the runner invokes on each push, and you can run it
by hand too.
```bash ```bash
# Linux build, in-guest git clone, no artifact packaging
ci job \ ci job \
--job-id manual-$(date +%s) \ --job-id manual-$(date +%s) \
--repo-url https://gitea.emulab.it/Simone/myrepo.git \ --repo-url https://gitea.emulab.it/Simone/myrepo.git \
@@ -159,21 +186,29 @@ ci job \
--use-git-clone --use-git-clone
``` ```
Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`, `--job-id`, `--repo-url`, and `--branch` are required. Other useful flags:
`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`,
`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`,
`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`,
`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`.
> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`. - **Source**: `--commit`, `--submodules` / `--no-submodules`,
`--use-git-clone` / `--host-clone`, `--use-shared-cache`
- **Build**: `--configuration Release`, `--guest-artifact-source dist`,
`--skip-artifact`, `--xvfb` (Linux GUI builds),
`--extra-env-json '{"KEY":"VALUE"}'`
- **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`,
`--snapshot-name`, `--ready-timeout`
- **Auth**: `--gitea-credential-target` (private repos)
> For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing))
> over hand-rolling a `job` invocation.
--- ---
## 4. Manual VM lifecycle (the steps `job` runs internally) ## 4. Manual VM lifecycle
Use these to drive a build stage-by-stage, e.g. when debugging a single phase. These are the stages that `job` runs internally. Drive them individually when
debugging a single phase.
### 4.1 Clone a VM ### 4.1 Clone a VM
```bash ```bash
ci vm new \ ci vm new \
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
@@ -181,10 +216,14 @@ ci vm new \
--clone-base-dir /var/lib/ci/build-vms \ --clone-base-dir /var/lib/ci/build-vms \
--job-id dbg-1 \ --job-id dbg-1 \
--guest-os linux \ --guest-os linux \
--start # prints the clone VMX path on stdout --start # prints the clone's VMX path on stdout
``` ```
### 4.2 Wait until reachable ### 4.2 Wait until reachable
Prints the guest IP once ready. Use `--ip-address` to skip polling, or
`--credential-target` for WinRM guests.
```bash ```bash
ci wait-ready \ ci wait-ready \
--vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \ --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
@@ -193,26 +232,29 @@ ci wait-ready \
--ssh-key-path /var/lib/ci/keys/ci_linux \ --ssh-key-path /var/lib/ci/keys/ci_linux \
--timeout 180 --timeout 180
``` ```
Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target`
for WinRM guests.
### 4.3 Run the build (guest already up) ### 4.3 Run the build
The guest must already be up — capture its IP first (see
[§2](#find-a-running-guests-ip)). Windows guests use `--credential-target
BuildVMGuest` in place of the SSH flags. Repeatable `--extra-env KEY=VALUE`;
add `--xvfb` for Linux GUI builds or `--use-shared-cache` for the shared cache.
```bash ```bash
ci build run \ ci build run \
--ip-address 192.168.79.200 \ --ip-address "$IP" \
--guest-os linux \ --guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \ --clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
--build-command "make all" \ --build-command "make all" \
--guest-artifact-source dist --guest-artifact-source dist
``` ```
Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`.
### 4.4 Collect artifacts ### 4.4 Collect artifacts
```bash ```bash
ci artifacts collect \ ci artifacts collect \
--ip-address 192.168.79.200 \ --ip-address "$IP" \
--guest-os linux \ --guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--guest-artifact-path dist \ --guest-artifact-path dist \
@@ -221,166 +263,193 @@ ci artifacts collect \
``` ```
### 4.5 Destroy the clone ### 4.5 Destroy the clone
```bash ```bash
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force # skip soft stop ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --force # skip the soft stop
``` ```
### 4.6 Sweep orphans (also a maintenance task, §6) ### 4.6 Sweep orphans
Also a maintenance task — see [§6](#6-maintenance).
```bash ```bash
ci vm cleanup --what-if # list only ci vm cleanup --what-if # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
``` ```
### 4.7 Open a VM in the Workstation GUI (interactive debugging) ### 4.7 Open a VM in the Workstation GUI
Launches the **interactive** VMware Workstation GUI (not the headless vmrun path)
so you can watch/poke a clone by hand. Launches the **interactive** VMware Workstation GUI (not the headless vmrun
path) so you can watch or poke a clone by hand. On a normal desktop session it
just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`:
On a normal desktop session it just works — `sudo` already forwards `DISPLAY`,
and `vmware` behaves exactly like `sudo -u ci-runner vmware`:
```bash ```bash
ci vmware # bare GUI, no VM ci vmware # bare GUI, no VM
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vmware --vmx <path> --power-on # power it on (-x) ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --power-on # power on (-x)
ci vmware --vmx <path> --fullscreen # power on + fullscreen (-X) ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --fullscreen # power on + fullscreen (-X)
``` ```
Other flags: `--new-window` (`-n`), `--display :0` (force a display if `$DISPLAY`
isn't forwarded), `--vmware-path`. Spawns detached; prints PID + log path.
**Why it works**: VMware connects via the X **host ACL**, not the cookie. The Other flags: `--new-window` (`-n`), `--display :0` (force a display when
inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by `$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the
ci-runner, so `vmware` drops it and relies on the xhost grant — which is normally command prints its PID and a log path under `/tmp/vmware-*.log`.
already present (`xhost` lists `SI:localuser:ci-runner`).
**How the display works.** VMware connects through the X **host ACL**, not the
cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is
unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant —
normally already present (`xhost` lists `SI:localuser:ci-runner`).
**Making the grant persistent.** An xhost grant is per-session and is wiped on
logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart
entry in the desktop user's home that re-applies it at every graphical login.
You can also (re)run that step standalone:
The xhost grant is **per-session**: it is wiped on logout / X restart. To make
it persistent, `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in
the desktop user's home that re-applies the grant at every graphical login. You
can (re)run it standalone:
```bash ```bash
# As root (or via sudo); --desktop-user defaults to $SUDO_USER cd /opt/ci/local-ci-cd-system
# --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# Remove the grant + autostart entry: # To remove the grant and the autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall
``` ```
This applies the grant immediately (if a display is reachable) and writes
This applies the grant immediately (when a display is reachable) and writes
`~simone/.config/autostart/ci-runner-xhost.desktop` for future logins. `~simone/.config/autostart/ci-runner-xhost.desktop` for future logins.
**One-off, without the script**: add the grant from the logged-in desktop user **One-off, without the script.** Add the grant from the logged-in desktop user
(not ci-runner): (not `ci-runner`):
```bash ```bash
xhost +SI:localuser:ci-runner xhost +SI:localuser:ci-runner
``` ```
`vmware` pre-flights the display (`xdpyinfo`) and prints this exact command on
failure instead of dying silently; the GUI's output is captured in `/tmp/vmware-*.log`. `vmware` pre-flights the display with `xdpyinfo` and prints this exact command
on failure rather than dying silently.
--- ---
## 5. Credentials & template provisioning ## 5. Credentials & template provisioning
### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1) ### `creds set`
Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`).
The password never appears on the command line — verify it afterwards with the
`validate host` keyring check.
```bash ```bash
ci creds set --target BuildVMGuest --user ci_build # hidden prompt ci creds set --target BuildVMGuest --user ci_build # hidden prompt
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
``` ```
Password is never on the command line. Verify via `validate host` keyring check.
(Leading space before `printf` keeps the secret out of shell history.) > The leading space before `printf` keeps the secret out of shell history.
### Template management ### Template management
```bash
# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux
# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux
# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win
# Timestamped backup of the template directories
ci template backup
```
Run `ci template <sub> --help` for the (many) per-template options. Run `ci template <sub> --help` for the (many) per-template options.
```bash
ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04)
ci template prepare-linux # provision an existing Linux template over SSH
ci template prepare-win # provision a Windows template over WinRM (Linux-host driven)
ci template backup # timestamped backup of the template directories
```
--- ---
## 6. Maintenance ## 6. Maintenance
### `retention run` — purge old artifacts & logs ### `retention run`
Purges old artifacts and logs. Always dry-run first. Retention switches to the
aggressive window automatically when free space drops below `--min-free-gb`.
```bash ```bash
ci retention run --what-if # dry run first ci retention run --what-if # dry run
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50 ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
``` ```
Switches to aggressive retention automatically when free space drops below
`--min-free-gb`.
### Orphan sweep — see §4.6 (`vm cleanup`). ### Orphan sweep
See [§4.6](#46-sweep-orphans) (`ci vm cleanup`).
--- ---
## 7. Phase C testing (burn-in / benchmark / smoke) ## 7. Phase C testing
Burn-in, benchmark, and smoke checks — all native Python, no pwsh.
### `smoke run`
One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts:
exit `0`, an artifact directory under `/var/lib/ci/artifacts/<job-id>/`, and a
`job/success` event in `invoke-ci.jsonl`.
### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1)
```bash ```bash
ci smoke run --guest-os linux # no-op marker job (fastest) ci smoke run --guest-os linux # no-op marker job (fastest)
ci smoke run --guest-os windows ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end
``` ```
Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts/<job-id>/` +
`job/success` event in `invoke-ci.jsonl`.
### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1) ### `bench measure`
Times the clone / start / IP / transport / destroy phases and appends a row to
`benchmark.jsonl` (legacy field names preserved for trend continuity).
```bash ```bash
ci bench measure --guest-os linux --iterations 1 ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4 # match historical baselines ci bench measure --guest-os linux --iterations 4 # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
``` ```
Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl`
(legacy field names preserved for trend continuity).
### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1) ### `bench run`
Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts
that all jobs exit `0`, no orphan clone directory is left behind, and no stale
`vm-start.lock` remains. The JSON report lands under
`/var/lib/ci/artifacts/bench/<timestamp>.json`.
```bash ```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6) ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)" sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
``` ```
Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`.
JSON report under `/var/lib/ci/artifacts/bench/<ts>.json`.
--- ---
## 8. Cutover checklist & uninstalling pwsh ## 8. Cutover checklist & uninstalling pwsh
Repeat for ≥1 week with pwsh **still installed** (A/B safety net): Repeat for at least one week with pwsh **still installed** as an A/B safety net:
- [ ] Normal `git push` CI pipeline green. - [ ] The normal `git push` CI pipeline stays green.
- [ ] `ci validate host` green daily. - [ ] `ci validate host` is green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks). - [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks).
- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`. - [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`.
- [ ] `ci bench measure` timings comparable to historical baselines. - [ ] `ci bench measure` timings are comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host. - [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
Then — only after a fully green week: Only after a fully green week:
```bash ```bash
pwsh --version pwsh --version
sudo apt-get remove powershell sudo apt-get remove powershell
ci validate host # must still exit 0 without pwsh ci validate host # must still exit 0 without pwsh
``` ```
Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original
`.ps1` scripts remain in `scripts/`. To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the
original `.ps1` scripts remain in `scripts/archived/`.
--- ---
## 9. Troubleshooting quick map ## 9. Troubleshooting
| Symptom | Cause | Fix | | Symptom | Cause | Fix |
| -------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------- | | ---------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------- |
| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` | | `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` |
| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` | | `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` |
| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP | | `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP |
| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` | | `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` | | WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` |
| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | | IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) | | `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; the systemd guard rebuilds) |
| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 | | `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |