Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
local-ci-cd-system/docs/RUNBOOK-PhaseC.md
T
Simone 005662e359 fix(vmgui): pre-flight X display, stop failing silently 
`vmgui` sent the child's stderr to /dev/null, so when the X display was
unreachable (the common case: sudo -u ci-runner loses the desktop user's
XAUTHORITY and xhost wasn't granted) the GUI died with "cannot open display"
and the command still printed success — "nothing opens".

Now it:
- pre-flights the display with xdpyinfo and aborts with the exact
  `xhost +SI:localuser:ci-runner` hint instead of spawning a doomed process;
- drops an inherited XAUTHORITY that this user can't read (falls back to
  host-based xhost auth);
- logs the GUI's stdio to /tmp/vmgui-*.log and grace-polls 1.5s, surfacing the
  log if the GUI exits immediately;
- errors (non-zero) when no DISPLAY is resolvable.

Runbook §4.7 updated with the "if nothing opens" guidance. 13 tests; green ≥90%.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 00:25:37 +02:00

349 lines
13 KiB
Markdown
Raw Permalink Blame History

# RUNBOOK — manual CLI reference (Linux host, pwsh-free)
Simplified but **complete** reference for every `ci_orchestrator` command you can
run by hand on the Linux host. All commands run as the **`ci-runner`** service
user against the live config (`/var/lib/ci/config.toml`).
Two notations are used:
- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form (alias):** `ci <cmd>` — after installing the helper (see §0).
Examples below use the **short form**. Drop `ci` for the literal command.
---
## 0. Setup (do once)
### 0.1 Refresh the production venv (mandatory after a code change/merge)
```bash
cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
```
### 0.2 Install the `ci` alias
```bash
./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent)
# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo)
# --uninstall remove it
source ~/.bashrc # or open a new shell
ci validate host # smoke test the alias
```
The script defines two helpers:
| Helper | Runs as | Use for |
|---------|---------------|---------|
| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) |
| `ci-me` | current user | read-only checks where sudo is noise |
### 0.3 Live environment
| Thing | Value |
|-------|-------|
| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` |
| Config | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) |
| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` |
---
## 1. Quick reference — every command
| Command | Does | Section |
|---------|------|---------|
| `ci validate host` | host prerequisites check (no VM) | §2 |
| `ci validate guest --host IP` | probe a running guest's transport | §2 |
| `ci monitor disk` | free-space alert | §2 |
| `ci monitor runner` | watch + auto-restart act_runner | §2 |
| `ci report job` | job summaries from JSONL logs | §2 |
| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 |
| `ci vm new ...` | linked-clone a template | §4 |
| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 |
| `ci build run ...` | run a build in a running guest | §4 |
| `ci artifacts collect ...` | copy artifacts out of a guest | §4 |
| `ci vm remove --vmx ...` | stop + delete one clone | §4 |
| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 |
| `ci vmgui --vmx ...` | open a VMX in the VMware Workstation GUI | §4 |
| `ci creds set --user ...` | store guest credentials in keyring | §5 |
| `ci template deploy-linux` | build the Linux template VM | §5 |
| `ci template prepare-linux` | provision Linux template over SSH | §5 |
| `ci template prepare-win` | provision Windows template over WinRM | §5 |
| `ci template backup` | timestamped copy of template dirs | §5 |
| `ci retention run` | purge old artifacts/logs | §6 |
| `ci bench measure ...` | phase-timing benchmark | §7 |
| `ci bench run ...` | concurrency burn-in | §7 |
| `ci smoke run ...` | one E2E job + assertions | §7 |
---
## 2. Health, monitoring & reporting (safe, no VM boot)
### `validate host` — run this first, every day
```bash
ci validate host ; echo "exit=$?"
```
Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner`
unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check.
### `validate guest` — diagnose a running guest's transport
```bash
ci validate guest --host 192.168.79.200 --ssh --guest-os linux # SSH
ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM
```
Connects + runs a trivial command, printing the explicit cause on failure
(connect vs auth vs command). The guest must already be booted.
### `monitor disk` — free-space alert
```bash
ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
```
### `monitor runner` — keep act_runner alive
```bash
ci monitor runner --service-name act-runner --max-restarts 3
```
Restarts the runner up to N times per window; optional `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
### `report job` — read job history
```bash
ci report job --last 10 # 10 most recent, table
ci report job --failed # only failures
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON out
```
---
## 3. Full pipeline (`job`) — the normal way to run a build
One command does everything: clone → wait-ready → build → collect → destroy.
This is what the runner invokes per push; you can run it by hand too.
```bash
# Linux build, in-guest git clone, no artifact packaging
ci job \
--job-id manual-$(date +%s) \
--repo-url https://gitea.emulab.it/Simone/myrepo.git \
--branch main \
--guest-os linux \
--build-command "make all" \
--use-git-clone
```
Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`,
`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`,
`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`,
`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`,
`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`.
> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`.
---
## 4. Manual VM lifecycle (the steps `job` runs internally)
Use these to drive a build stage-by-stage, e.g. when debugging a single phase.
### 4.1 Clone a VM
```bash
ci vm new \
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
--snapshot BaseClean-Linux \
--clone-base-dir /var/lib/ci/build-vms \
--job-id dbg-1 \
--guest-os linux \
--start # prints the clone VMX path on stdout
```
### 4.2 Wait until reachable
```bash
ci wait-ready \
--vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
--guest-os linux \
--ssh-user ci_build \
--ssh-key-path /var/lib/ci/keys/ci_linux \
--timeout 180
```
Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target`
for WinRM guests.
### 4.3 Run the build (guest already up)
```bash
ci build run \
--ip-address 192.168.79.200 \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
--build-command "make all" \
--guest-artifact-source dist
```
Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`.
### 4.4 Collect artifacts
```bash
ci artifacts collect \
--ip-address 192.168.79.200 \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--guest-artifact-path dist \
--host-artifact-dir /var/lib/ci/artifacts/dbg-1 \
--include-logs --job-id dbg-1
```
### 4.5 Destroy the clone
```bash
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force # skip soft stop
```
### 4.6 Sweep orphans (also a maintenance task, §6)
```bash
ci vm cleanup --what-if # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
```
### 4.7 Open a VM in the Workstation GUI (interactive debugging)
Launches the **interactive** VMware Workstation GUI (not the headless vmrun path)
so you can watch/poke a clone by hand.
**Required once per desktop session** — authorise `ci-runner` on your X server
(run as YOUR logged-in user, not ci-runner):
```bash
xhost +SI:localuser:ci-runner
```
Then open the GUI as ci-runner, forcing the display:
```bash
ci vmgui --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --display :0
ci vmgui --vmx <path> --display :0 --power-on # also power it on (-x)
ci vmgui --vmx <path> --display :0 --fullscreen # power on + fullscreen (-X)
ci vmgui --display :0 # bare GUI, no VM loaded
```
**If nothing opens**: you skipped the `xhost` step. `vmgui` now pre-flights the
display (`xdpyinfo`) and fails with the exact `xhost` command instead of dying
silently; it also drops an unreadable inherited `XAUTHORITY` and logs the GUI's
output to `/tmp/vmgui-*.log`. The GUI needs an X display — `sudo -u ci-runner`
loses your cookie, so `xhost` + `--display :0` is the reliable combo. Other
flags: `--new-window` (`-n`), `--vmware-path`. Spawns detached, prints PID + log.
---
## 5. Credentials & template provisioning
### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1)
```bash
ci creds set --target BuildVMGuest --user ci_build # hidden prompt
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
```
Password is never on the command line. Verify via `validate host` keyring check.
(Leading space before `printf` keeps the secret out of shell history.)
### Template management
```bash
# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux
# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux
# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win
# Timestamped backup of the template directories
ci template backup
```
Run `ci template <sub> --help` for the (many) per-template options.
---
## 6. Maintenance
### `retention run` — purge old artifacts & logs
```bash
ci retention run --what-if # dry run first
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
```
Switches to aggressive retention automatically when free space drops below
`--min-free-gb`.
### Orphan sweep — see §4.6 (`vm cleanup`).
---
## 7. Phase C testing (burn-in / benchmark / smoke)
### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1)
```bash
ci smoke run --guest-os linux # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E
```
Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts/<job-id>/` +
`job/success` event in `invoke-ci.jsonl`.
### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1)
```bash
ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4 # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
```
Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl`
(legacy field names preserved for trend continuity).
### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1)
```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
```
Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`.
JSON report under `/var/lib/ci/artifacts/bench/<ts>.json`.
---
## 8. Cutover checklist & uninstalling pwsh
Repeat for ≥1 week with pwsh **still installed** (A/B safety net):
- [ ] Normal `git push` CI pipeline green.
- [ ] `ci validate host` green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks).
- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`.
- [ ] `ci bench measure` timings comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
Then — only after a fully green week:
```bash
pwsh --version
sudo apt-get remove powershell
ci validate host # must still exit 0 without pwsh
```
Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original
`.ps1` scripts remain in `scripts/`.
---
## 9. Troubleshooting quick map
| Symptom | Cause | Fix |
|---------|-------|-----|
| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` |
| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` |
| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP |
| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` |
| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) |
| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 |
Reference in New Issue View Git Blame Copy Permalink