local-ci-cd-system/docs/RUNBOOK-PhaseC.md

# RUNBOOK — manual CLI reference (Linux host, pwsh-free)

Simplified but **complete** reference for every `ci_orchestrator` command you can
run by hand on the Linux host. All commands run as the **`ci-runner`** service
user against the live config (`/var/lib/ci/config.toml`).

Two notations are used:

- **Full form:** `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
- **Short form (alias):** `ci <cmd>` — after installing the helper (see §0).

Examples below use the **short form**. Drop `ci` for the literal command.

---

## 0. Setup (do once)

### 0.1 Refresh the production venv (mandatory after a code change/merge)

```bash
cd /opt/ci/local-ci-cd-system
sudo /opt/ci/venv/bin/python -m pip install .        # non-editable, see CLAUDE.md
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
```

### 0.2 Install the `ci` alias

```bash
./scripts/install-ci-alias.sh        # into ~/.bashrc + ~/.zshrc (idempotent)
#   --system     install /etc/profile.d/ci-alias.sh for all users (needs sudo)
#   --uninstall  remove it
source ~/.bashrc                      # or open a new shell
ci validate host                      # smoke test the alias
```

The script defines two helpers:

| Helper  | Runs as       | Use for |
|---------|---------------|---------|
| `ci`    | `ci-runner` (sudo) | everything (writes keyring, starts VMs) |
| `ci-me` | current user  | read-only checks where sudo is noise |

### 0.3 Live environment

| Thing | Value |
|-------|-------|
| Service user | `ci-runner` · Prod venv `/opt/ci/venv/bin/python` |
| Config | `/var/lib/ci/config.toml` |
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snap `BaseClean-Linux`) |
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snap `BaseClean`) |
| Clones / Artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
| Linux SSH key | `/var/lib/ci/keys/ci_linux` · cred target `BuildVMGuest` |

---

## 1. Quick reference — every command

| Command | Does | Section |
|---------|------|---------|
| `ci validate host` | host prerequisites check (no VM) | §2 |
| `ci validate guest --host IP` | probe a running guest's transport | §2 |
| `ci monitor disk` | free-space alert | §2 |
| `ci monitor runner` | watch + auto-restart act_runner | §2 |
| `ci report job` | job summaries from JSONL logs | §2 |
| `ci job ...` | **full pipeline** clone→build→collect→destroy | §3 |
| `ci vm new ...` | linked-clone a template | §4 |
| `ci wait-ready --vmx ...` | poll until SSH/WinRM ready | §4 |
| `ci build run ...` | run a build in a running guest | §4 |
| `ci artifacts collect ...` | copy artifacts out of a guest | §4 |
| `ci vm remove --vmx ...` | stop + delete one clone | §4 |
| `ci vm cleanup` | sweep orphaned clones + stale locks | §4 / §6 |
| `ci vm open --vmx ...` | open a VMX in the VMware Workstation GUI | §4 |
| `ci creds set --user ...` | store guest credentials in keyring | §5 |
| `ci template deploy-linux` | build the Linux template VM | §5 |
| `ci template prepare-linux` | provision Linux template over SSH | §5 |
| `ci template prepare-win` | provision Windows template over WinRM | §5 |
| `ci template backup` | timestamped copy of template dirs | §5 |
| `ci retention run` | purge old artifacts/logs | §6 |
| `ci bench measure ...` | phase-timing benchmark | §7 |
| `ci bench run ...` | concurrency burn-in | §7 |
| `ci smoke run ...` | one E2E job + assertions | §7 |

---

## 2. Health, monitoring & reporting (safe, no VM boot)

### `validate host` — run this first, every day
```bash
ci validate host ; echo "exit=$?"
```
Checks vmrun, template snapshots, keyring, `/var/lib/ci` permissions, `act-runner`
unit. Exit `0` + all `[OK ]`; a `[FAIL]` line names the broken check.

### `validate guest` — diagnose a running guest's transport
```bash
ci validate guest --host 192.168.79.200 --ssh   --guest-os linux     # SSH
ci validate guest --host 192.168.79.201 --winrm --guest-os windows   # WinRM
```
Connects + runs a trivial command, printing the explicit cause on failure
(connect vs auth vs command). The guest must already be booted.

### `monitor disk` — free-space alert
```bash
ci monitor disk --min-free-gb 50
ci monitor disk --min-free-gb 50 --json            # machine-readable one-liner
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
```

### `monitor runner` — keep act_runner alive
```bash
ci monitor runner --service-name act-runner --max-restarts 3
```
Restarts the runner up to N times per window; optional `--webhook-url`,
`--gitea-url`, `--gitea-credential-target`.

### `report job` — read job history
```bash
ci report job --last 10                 # 10 most recent, table
ci report job --failed                  # only failures
ci report job --job-id smoke-20260607-2030   # phase breakdown for one job
ci report job --last 20 --json          # JSON out
```

---

## 3. Full pipeline (`job`) — the normal way to run a build

One command does everything: clone → wait-ready → build → collect → destroy.
This is what the runner invokes per push; you can run it by hand too.

```bash
# Linux build, in-guest git clone, no artifact packaging
ci job \
  --job-id manual-$(date +%s) \
  --repo-url https://gitea.emulab.it/Simone/myrepo.git \
  --branch main \
  --guest-os linux \
  --build-command "make all" \
  --use-git-clone
```

Key flags: `--commit`, `--configuration Release`, `--guest-artifact-source dist`,
`--submodules/--no-submodules`, `--use-git-clone/--host-clone`, `--use-shared-cache`,
`--skip-artifact`, `--xvfb` (Linux GUI builds), `--guest-cpu N`, `--guest-memory-mb N`,
`--template-path`, `--snapshot-name`, `--ready-timeout`, `--extra-env-json '{"K":"V"}'`,
`--gitea-credential-target` (private repos). Required: `--job-id --repo-url --branch`.

> For a zero-config end-to-end check, prefer `smoke run` (§7) over hand-rolling `job`.

---

## 4. Manual VM lifecycle (the steps `job` runs internally)

Use these to drive a build stage-by-stage, e.g. when debugging a single phase.

### 4.1 Clone a VM
```bash
ci vm new \
  --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
  --snapshot BaseClean-Linux \
  --clone-base-dir /var/lib/ci/build-vms \
  --job-id dbg-1 \
  --guest-os linux \
  --start                       # prints the clone VMX path on stdout
```

### 4.2 Wait until reachable
```bash
ci wait-ready \
  --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx \
  --guest-os linux \
  --ssh-user ci_build \
  --ssh-key-path /var/lib/ci/keys/ci_linux \
  --timeout 180
```
Prints the guest IP when ready. `--ip-address` skips polling; `--credential-target`
for WinRM guests.

### 4.3 Run the build (guest already up)
```bash
ci build run \
  --ip-address 192.168.79.200 \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --clone-url https://gitea.emulab.it/Simone/myrepo.git --clone-branch main \
  --build-command "make all" \
  --guest-artifact-source dist
```
Windows guests use `--credential-target BuildVMGuest` instead of the SSH flags.
Repeatable `--extra-env KEY=VALUE`; `--xvfb` Linux GUI; `--use-shared-cache`.

### 4.4 Collect artifacts
```bash
ci artifacts collect \
  --ip-address 192.168.79.200 \
  --guest-os linux \
  --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
  --guest-artifact-path dist \
  --host-artifact-dir /var/lib/ci/artifacts/dbg-1 \
  --include-logs --job-id dbg-1
```

### 4.5 Destroy the clone
```bash
ci vm remove --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx
ci vm remove --vmx <path> --force        # skip soft stop
```

### 4.6 Sweep orphans (also a maintenance task, §6)
```bash
ci vm cleanup --what-if                  # list only
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
```

### 4.7 Open a VM in the Workstation GUI (interactive debugging)
Launches the **interactive** VMware Workstation GUI (not the headless vmrun path)
so you can watch/poke a clone by hand.

```bash
# authorise the service user on your desktop X session first (run as YOUR user):
xhost +SI:localuser:ci-runner

# then open the VMX as ci-runner, forcing the display:
ci vm open --vmx /var/lib/ci/build-vms/Clone_dbg-1_*/Clone_dbg-1_*.vmx --display :0
ci vm open --vmx <path> --display :0 --power-on      # also power it on (-x)
ci vm open --vmx <path> --display :0 --fullscreen    # power on + fullscreen (-X)
```
The GUI needs an X display; `sudo -u ci-runner` strips `$DISPLAY`, so pass
`--display :0` (or export `DISPLAY`). Other flags: `--new-window` (`-n`),
`--vmware-path` to override the binary. The command spawns the GUI detached and
returns immediately (prints the child PID).

---

## 5. Credentials & template provisioning

### `creds set` — store the guest login (replaces Set-CIGuestCredential.ps1)
```bash
ci creds set --target BuildVMGuest --user ci_build           # hidden prompt
 printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
```
Password is never on the command line. Verify via `validate host` keyring check.
(Leading space before `printf` keeps the secret out of shell history.)

### Template management
```bash
# Build the Linux template VM from scratch (Ubuntu 24.04)
ci template deploy-linux

# Provision an existing Linux template over SSH (installs toolchain, sets snapshot prep)
ci template prepare-linux

# Provision a Windows template over WinRM (Linux-host driven)
ci template prepare-win

# Timestamped backup of the template directories
ci template backup
```
Run `ci template <sub> --help` for the (many) per-template options.

---

## 6. Maintenance

### `retention run` — purge old artifacts & logs
```bash
ci retention run --what-if                          # dry run first
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
```
Switches to aggressive retention automatically when free space drops below
`--min-free-gb`.

### Orphan sweep — see §4.6 (`vm cleanup`).

---

## 7. Phase C testing (burn-in / benchmark / smoke)

### `smoke run` — one E2E job + assertions (replaces Test-Smoke.ps1)
```bash
ci smoke run --guest-os linux                       # no-op marker job (fastest)
ci smoke run --guest-os windows
ci smoke run --guest-os linux --preset ns7zip       # real Linux build E2E
```
Asserts: exit 0 + artifact dir under `/var/lib/ci/artifacts/<job-id>/` +
`job/success` event in `invoke-ci.jsonl`.

### `bench measure` — phase timings (replaces Measure-CIBenchmark.ps1)
```bash
ci bench measure --guest-os linux --iterations 1
ci bench measure --guest-os linux --iterations 4    # match historical baselines
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
```
Times clone/start/IP/transport/destroy; appends to `benchmark.jsonl`
(legacy field names preserved for trend continuity).

### `bench run` — concurrency burn-in (replaces Test-CapacityBurnIn.ps1)
```bash
ci bench run --guest-os linux --concurrency 2 --rounds 2     # quick confidence
ci bench run --guest-os linux --concurrency 4 --rounds 10    # standard gate
ci bench run --guest-os windows --concurrency 3 --rounds 10  # WinRM unstable at 4x (TODO §3.6)
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
```
Per round asserts: all jobs exit 0, no orphan clone dir, no stale `vm-start.lock`.
JSON report under `/var/lib/ci/artifacts/bench/<ts>.json`.

---

## 8. Cutover checklist & uninstalling pwsh

Repeat for ≥1 week with pwsh **still installed** (A/B safety net):

- [ ] Normal `git push` CI pipeline green.
- [ ] `ci validate host` green daily.
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` green (no orphans/locks).
- [ ] `ci smoke run` green on `--guest-os linux` **and** `--guest-os windows`.
- [ ] `ci bench measure` timings comparable to historical baselines.
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.

Then — only after a fully green week:

```bash
pwsh --version
sudo apt-get remove powershell
ci validate host                # must still exit 0 without pwsh
```
Rollback: `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original
`.ps1` scripts remain in `scripts/`.

---

## 9. Troubleshooting quick map

| Symptom | Cause | Fix |
|---------|-------|-----|
| `No such command 'bench'` | prod venv not refreshed | §0.1 `pip install .` |
| `validate host` keyring `[FAIL]` | credential missing/wrong user | §5 `creds set` |
| `validate guest` connect failure | VM not booted / wrong IP / firewall | confirm VM running + IP |
| `bench run` round FAIL: orphan | clone not destroyed | `ci vm cleanup`; check vmrun + `/var/lib/ci/build-vms` |
| WinRM jobs drop at concurrency 4 | known instability (TODO §3.6) | use `--concurrency 3` |
| IP acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; systemd guard rebuilds) |
| `ci: command not found` | alias not loaded | `source ~/.bashrc` or re-run §0.2 |