Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7f615333b6bc3f793f7aedfb186fddec67dae4cc
local-ci-cd-system/docs/RUNBOOK-PhaseC.md
T
Simone 7f615333b6 docs(runbook): add guest IP discovery to §2 
Document `vmrun getGuestIPAddress` (run as ci-runner) for templates and
clones as the prerequisite for `validate guest`, with the AGENTS.md #1
"Error: on stdout" gotcha. Update the validate-guest Linux example to the
live template IP.
2026-06-08 21:26:07 +02:00

387 lines
17 KiB
Markdown
Raw Blame History

# RUNBOOK — manual CLI reference (Linux host, pwsh-free)
Simplified but **complete** reference for every `ci_orchestrator` command you can
run by hand on the Linux host. All commands run as the **`ci-runner`** service
user against the live config (`/var/lib/ci/config.toml`).
Two notations are used:
- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form (alias):** `ci <cmd>` — after installing the helper (see §0).
Examples below use the **short form**. Drop `ci` for the literal command.
---
## 0. Setup (do once)
### 0.1 Refresh the production venv (mandatory after a code change/merge)
```bash
cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install . # non-editable, see CLAUDE.md
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
```
### 0.2 Install the `ci` alias
```bash
./scripts/install-ci-alias.sh # into ~/.bashrc + ~/.zshrc (idempotent)
# --system install /etc/profile.d/ci-alias.sh for all users (needs sudo)
# --uninstall remove it
source ~/.bashrc # or open a new shell
ci validate host # smoke test the alias
```
The script defines two helpers:
| Helper | Runs as | Use for |
| ------- | ------------------ | --------------------------------------- |
| `ci` | `ci-runner` (sudo) | everything (writes keyring, starts VMs) |
| `ci-me` | current user | read-only checks where sudo is noise |
### 0.3 Live environment
| Thing | Value |
| ------------------ | ---------------------------------------------------------------------------------- |
| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` |
| Config | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) |
| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` |
---
## 1. Quick reference — every command
| Command | Does | Section |
| ----------------------------- | --------------------------------------------- | ------- |
| `ci validate host` | host prerequisites check (no VM) | §2 |
| `ci validate guest --host IP` | probe a running guest's transport | §2 |
| `ci monitor disk` | free-space alert | §2 |
| `ci monitor runner` | watch + auto-restart act_runner | §2 |
| `ci report job` | job summaries from JSONL logs | §2 |
| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 |
| `ci vm new ...` | linked-clone a template | §4 |
| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 |
| `ci build run ...` | run a build in a running guest | §4 |
| `ci artifacts collect ...` | copy artifacts out of a guest | §4 |
| `ci vm remove --vmx ...` | stop + delete one clone | §4 |
| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 |
| `ci vmware --vmx ...` | open a VMX in the VMware Workstation GUI | §4 |
| `ci creds set --user ...` | store guest credentials in keyring | §5 |
| `ci template deploy-linux` | build the Linux template VM | §5 |
| `ci template prepare-linux` | provision Linux template over SSH | §5 |
| `ci template prepare-win` | provision Windows template over WinRM | §5 |
| `ci template backup` | timestamped copy of template dirs | §5 |
| `ci retention run` | purge old artifacts/logs | §6 |
| `ci bench measure ...` | phase-timing benchmark | §7 |
| `ci bench run ...` | concurrency burn-in | §7 |
| `ci smoke run ...` | one E2E job + assertions | §7 |
---
## 2. Health, monitoring & reporting (safe, no VM boot)
### `validate host` — run this first, every day
```bash
ci validate host ; echo "exit=$?"
```
Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner`
unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check.
### Find a running guest's IP (needed for `validate guest`)
`vmrun getGuestIPAddress` reads the IP from VMware Tools — the VM must be
**powered on** and Tools running. Run it as `ci-runner` (the VMs are started by
that user, so its `vmrun` inventory is the one that sees them):
```bash
# Linux template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
# Windows template
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
# A build clone (glob the job dir)
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
# List which VMs are powered on first
sudo -u ci-runner vmrun list
```
> **Gotcha (AGENTS.md #1):** `getGuestIPAddress` can exit `0` and print
> `Error: The VMware Tools are not running...` on stdout when Tools are down.
> A line starting with `Error:` means *no IP yet* — wait for boot, don't parse it.
### `validate guest` — diagnose a running guest's transport
```bash
ci validate guest --host 192.168.79.128 --ssh --guest-os linux # SSH (Linux)
ci validate guest --host 192.168.79.201 --winrm --guest-os windows # WinRM (Windows)
```
Connects + runs a trivial command, printing the explicit cause on failure
(connect vs auth vs command). The guest must already be booted — get its IP
with `getGuestIPAddress` above.
### `monitor disk` — free-space alert
```bash
ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
```
### `monitor runner` — keep act_runner alive
```bash
ci monitor runner --service-name act-runner --max-restarts 3
```
Restarts the runner up to N times per window; optional `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.
### `report job` — read job history
```bash
ci report job --last 10 # 10 most recent, table
ci report job --failed # only failures
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
ci report job --last 20 --json # JSON out
```
---
## 3. Full pipeline (`job`) — the normal way to run a build
One command does everything: clone → wait-ready → build → collect → destroy.
This is what the runner invokes per push; you can run it by hand too.
```bash
# Linux build, in-guest git clone, no artifact packaging
ci job \
--job-id manual-$(date +%s) \
--repo-url https://gitea.emulab.it/Simone/myrepo.git \
--branch main \
--guest-os linux \
--build-command "make all" \
--use-git-clone
```
Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`,
`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`,
`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`,
`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`,
`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`.
> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`.
---
## 4. Manual VM lifecycle (the steps `job` runs internally)
Use these to drive a build stage-by-stage, e.g. when debugging a single phase.
### 4.1 Clone a VM
```bash
ci vm new \
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
--snapshot BaseClean-Linux \
--clone-base-dir /var/lib/ci/build-vms \
--job-id dbg-1 \
--guest-os linux \
--start # prints the clone VMX path on stdout
```
### 4.2 Wait until reachable
```bash
ci wait-ready \
--vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
--guest-os linux \
--ssh-user ci_build \
--ssh-key-path /var/lib/ci/keys/ci_linux \
--timeout 180
```
Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target`
for WinRM guests.
### 4.3 Run the build (guest already up)
```bash
ci build run \
--ip-address 192.168.79.200 \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
--build-command "make all" \
--guest-artifact-source dist
```
Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`.
### 4.4 Collect artifacts
```bash
ci artifacts collect \
--ip-address 192.168.79.200 \
--guest-os linux \
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
--guest-artifact-path dist \
--host-artifact-dir /var/lib/ci/artifacts/dbg-1 \
--include-logs --job-id dbg-1
```
### 4.5 Destroy the clone
```bash
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force # skip soft stop
```
### 4.6 Sweep orphans (also a maintenance task, §6)
```bash
ci vm cleanup --what-if # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
```
### 4.7 Open a VM in the Workstation GUI (interactive debugging)
Launches the **interactive** VMware Workstation GUI (not the headless vmrun path)
so you can watch/poke a clone by hand.
On a normal desktop session it just works — `sudo` already forwards `DISPLAY`,
and `vmware` behaves exactly like `sudo -u ci-runner vmware`:
```bash
ci vmware # bare GUI, no VM
ci vmware --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vmware --vmx <path> --power-on # power it on (-x)
ci vmware --vmx <path> --fullscreen # power on + fullscreen (-X)
```
Other flags: `--new-window` (`-n`), `--display :0` (force a display if `$DISPLAY`
isn't forwarded), `--vmware-path`. Spawns detached; prints PID + log path.
**Why it works**: VMware connects via the X **host ACL**, not the cookie. The
inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by
ci-runner, so `vmware` drops it and relies on the xhost grant — which is normally
already present (`xhost` lists `SI:localuser:ci-runner`).
The xhost grant is **per-session**: it is wiped on logout / X restart. To make
it persistent, `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in
the desktop user's home that re-applies the grant at every graphical login. You
can (re)run it standalone:
```bash
# As root (or via sudo); --desktop-user defaults to $SUDO_USER
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
# Remove the grant + autostart entry:
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall
```
This applies the grant immediately (if a display is reachable) and writes
`~simone/.config/autostart/ci-runner-xhost.desktop` for future logins.
**One-off, without the script**: add the grant from the logged-in desktop user
(not ci-runner):
```bash
xhost +SI:localuser:ci-runner
```
`vmware` pre-flights the display (`xdpyinfo`) and prints this exact command on
failure instead of dying silently; the GUI's output is captured in `/tmp/vmware-*.log`.
---
## 5. Credentials & template provisioning
### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1)
```bash
ci creds set --target BuildVMGuest --user ci_build # hidden prompt
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
```
Password is never on the command line. Verify via `validate host` keyring check.
(Leading space before `printf` keeps the secret out of shell history.)
### Template management
```bash
# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux
# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux
# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win
# Timestamped backup of the template directories
ci template backup
```
Run `ci template <sub> --help` for the (many) per-template options.
---
## 6. Maintenance
### `retention run` — purge old artifacts & logs
```bash
ci retention run --what-if # dry run first
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
```
Switches to aggressive retention automatically when free space drops below
`--min-free-gb`.
### Orphan sweep — see §4.6 (`vm cleanup`).
---
## 7. Phase C testing (burn-in / benchmark / smoke)
### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1)
```bash
ci smoke run --guest-os linux # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip # real Linux build E2E
```
Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts/<job-id>/` +
`job/success` event in `invoke-ci.jsonl`.
### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1)
```bash
ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4 # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
```
Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl`
(legacy field names preserved for trend continuity).
### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1)
```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
```
Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`.
JSON report under `/var/lib/ci/artifacts/bench/<ts>.json`.
---
## 8. Cutover checklist & uninstalling pwsh
Repeat for ≥1 week with pwsh **still installed** (A/B safety net):
- [ ] Normal `git push` CI pipeline green.
- [ ] `ci validate host` green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks).
- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`.
- [ ] `ci bench measure` timings comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
Then — only after a fully green week:
```bash
pwsh --version
sudo apt-get remove powershell
ci validate host # must still exit 0 without pwsh
```
Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original
`.ps1` scripts remain in `scripts/`.
---
## 9. Troubleshooting quick map
| Symptom | Cause | Fix |
| -------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------- |
| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` |
| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` |
| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP |
| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` |
| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) |
| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 |
Reference in New Issue View Git Blame Copy Permalink