# Runbook — Manual CLI Reference (Linux host, pwsh-free) A complete, hands-on reference for every `ci_orchestrator` command you can run by hand on the Linux host. Unless noted otherwise, commands run as the **`ci-runner`** service user against the live configuration at `/var/lib/ci/config.toml`. Every command can be written two ways: - **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ` - **Short form** — `ci `, after installing the alias (see [§0](#0-setup-do-once)). The examples use the short form. The paths and template names below are the real ones on this host, so most blocks are copy-paste-ready as written. --- ## 0. Setup (do once) ### 0.1 Refresh the production venv Mandatory after every code change or merge — the venv is installed non-editable, so changes only take effect once reinstalled: ```bash cd /opt/ci/local-ci-cd-system sudo /opt/ci/venv/bin/python -m pip install . sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds' ``` ### 0.2 Install the `ci` alias ```bash cd /opt/ci/local-ci-cd-system ./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent) source ~/.bashrc # or open a new shell ci validate host # smoke-test the alias ``` Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs sudo); `--uninstall` removes the block. The script defines two helpers: | Helper | Runs as | Use for | | ------- | ------------------ | ----------------------------------------- | | `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs | | `ci-me` | the current user | read-only checks where sudo is just noise | ### 0.3 The live environment | Item | Value | | ------------------ | -------------------------------------------------------------------------------------- | | Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` | | Configuration | `/var/lib/ci/config.toml` | | Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) | | Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) | | Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` | | Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` | --- ## 1. Command index | Command | Purpose | Section | | ----------------------------- | ---------------------------------------------------- | --------------------------------------------------- | | `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) | | `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) | | `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) | | `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) | | `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) | | `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) | | `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) | | `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) | | `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) | | `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) | | `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) | | `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) | | `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) | | `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) | | `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) | | `ci template deploy-rust` | Build the Rust Linux template VM (LinuxRust2404) | [§5](#5-credentials--template-provisioning) | | `ci template check-win-iso` | Report the newest WS2025 ISO from massgrave + cache status | [§5](#5-credentials--template-provisioning) | | `ci template patch-win-iso` | Inject autounattend + no-prompt boot into a Windows ISO | [§5](#5-credentials--template-provisioning) | | `ci template deploy-win` | Build the Windows Server 2025 template VM (download→patch→deploy) | [§5](#5-credentials--template-provisioning) | | `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) | | `ci template prepare-rust` | Provision the Rust Linux template over SSH | [§5](#5-credentials--template-provisioning) | | `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) | | `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) | | `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) | | `tools/ci-action-published.py` | Is the `.gitea/` CI runs the one in this checkout? | [§6](#6-maintenance) | | `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) | | `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) | | `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) | --- ## 2. Health, monitoring & reporting These are safe — none of them boots a VM. ### `validate host` Run this first, every day. It checks vmrun, template snapshots, the keyring, `/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0` with every line marked `[OK ]`; any `[FAIL]` line names the broken check. ```bash ci validate host ; echo "exit=$?" ``` ### Find a running guest's IP `vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must be **powered on** with Tools running. Run it as `ci-runner` — the VMs are started by that user, so its `vmrun` inventory is the one that can see them. ```bash # Which VMs are powered on right now sudo -u ci-runner vmrun list # IP of the Linux template sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx # IP of a Windows template sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx # IP of a build clone (glob the job directory) sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx ``` > **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print > `Error: The VMware Tools are not running...` on stdout when Tools are down. A > line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it. ### `validate guest` Connects to a booted guest, runs a trivial command, and reports the explicit cause on failure (connect vs. auth vs. command). Capture the IP into a variable so the probe is genuinely copy-paste-ready: ```bash # Linux guest over SSH IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx) ci validate guest --host "$IP" --ssh --guest-os linux # Windows guest over WinRM IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx) ci validate guest --host "$IP" --winrm --guest-os windows IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx) ci validate guest --host "$IP" --winrm --guest-os windows ``` ### `monitor disk` ```bash ci monitor disk --min-free-gb 50 ci monitor disk --min-free-gb 50 --json # machine-readable one-liner ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK" ``` ### `monitor runner` Restarts the runner up to N times per window. Optional: `--webhook-url`, `--gitea-url`, `--gitea-credential-target`. ```bash ci monitor runner --service-name act-runner --max-restarts 3 ``` ### `report job` ```bash ci report job --last 10 # 10 most recent, as a table ci report job --failed # failures only ci report job --job-id smoke-20260607-2030 # phase breakdown for one job ci report job --last 20 --json # JSON output ``` --- ## 3. Full pipeline (`job`) This is the normal way to run a build: a single command clones the template, waits for the guest, runs the build, collects artifacts, and destroys the clone. It is exactly what the runner invokes on each push, and you can run it by hand too. The examples use `Simone/burnin-dummy`, whose `build.sh` simulates a real build (sleeps ~30 s, then writes `dist/build-info.txt`) — so the run exercises the full clone → build → artifact-collect → destroy path end to end. ```bash ci job \ --job-id manual-$(date +%s) \ --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot-name BaseClean-Linux \ --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \ --branch main \ --guest-os linux \ --build-command "bash build.sh" \ --guest-artifact-source dist \ --use-git-clone ``` The VM boots headless by default. To watch it in the Workstation GUI while it runs, add `--gui` (needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)): ```bash ci job \ --job-id manual-$(date +%s) \ --template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot-name BaseClean-Linux \ --repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \ --branch main \ --guest-os linux \ --build-command "bash build.sh" \ --guest-artifact-source dist \ --use-git-clone \ --gui ``` `--job-id`, `--repo-url`, `--branch`, and `--template-path` are required. The default `--snapshot-name` is `BaseClean`, so Linux jobs must pass `--snapshot-name BaseClean-Linux` explicitly. Other useful flags: - **Source**: `--commit`, `--submodules` / `--no-submodules`, `--use-git-clone` / `--host-clone`, `--use-shared-cache` - **Build**: `--configuration Release`, `--guest-artifact-source dist`, `--skip-artifact`, `--xvfb` (Linux GUI builds), `--extra-env-json '{"KEY":"VALUE"}'` - **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`, `--snapshot-name`, `--ready-timeout`, `--gui` / `--nogui` (default `--nogui`, headless; `--gui` opens the Workstation window and needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)) - **Auth**: `--gitea-credential-target` (private repos) > For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing)) > over hand-rolling a `job` invocation. --- ## 4. Manual VM lifecycle These are the stages that `job` runs internally. Drive them individually when debugging a single phase. **Run the blocks below in the same shell session** — each one reuses the `$VMX` and `$IP` variables captured by an earlier step, so there are no fragile path globs to expand. ### 4.1 Clone a VM `ci vm new` prints the clone's VMX path (and nothing else) on stdout, so capture it straight into a variable. A timestamped job-id keeps repeated runs from colliding. ```bash VMX=$(ci vm new \ --template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \ --snapshot BaseClean-Linux \ --clone-base-dir /var/lib/ci/build-vms \ --job-id "dbg-$(date +%s)" \ --guest-os linux \ --start --gui) echo "clone VMX: $VMX" ``` `--gui` opens the clone in the Workstation window so you can watch it while you debug (it needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)). Drop `--gui` for a headless boot. ### 4.2 Wait until reachable, then capture the IP `wait-ready` blocks until SSH is up. The clean guest IP comes from `vmrun getGuestIPAddress`; capture it for the next steps. ```bash ci wait-ready \ --vmx "$VMX" \ --guest-os linux \ --ssh-user ci_build \ --ssh-key-path /var/lib/ci/keys/ci_linux \ --timeout 180 IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX") echo "guest IP: $IP" ``` For WinRM guests, add `--credential-target BuildVMGuest` to `wait-ready`. ### 4.3 Run the build Unlike `job`, `ci build run` does **not** inject the Gitea PAT, so an in-guest `--clone-url` only works for a public repo. For a private repo (like `burnin-dummy`), upload a local source directory with `--host-source-dir` instead — here the checked-out copy of `burnin-dummy` that ships in this repo. The build stages its `--guest-artifact-source` (`dist`) into `/opt/ci/output`, which is where step 4.4 collects from. Windows guests use `--credential-target BuildVMGuest` in place of the SSH flags. Repeatable `--extra-env KEY=VALUE`; add `--xvfb` for Linux GUI builds or `--use-shared-cache` for the shared cache. ```bash ci build run \ --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \ --build-command "bash build.sh" \ --guest-artifact-source dist ``` ### 4.4 Collect artifacts `build run` stages the artifact under `/opt/ci/output`, so collect from there (not from the `dist` source directory). ```bash ci artifacts collect \ --ip-address "$IP" \ --guest-os linux \ --ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \ --guest-artifact-path /opt/ci/output \ --host-artifact-dir /var/lib/ci/artifacts/dbg-manual \ --include-logs --job-id dbg-manual ``` ### 4.5 Destroy the clone ```bash ci vm remove --vmx "$VMX" ci vm remove --vmx "$VMX" --force # skip the soft stop ``` ### 4.6 Sweep orphans Also a maintenance task — see [§6](#6-maintenance). ```bash ci vm cleanup --what-if # list only ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock ``` ### 4.7 Open a VM in the Workstation GUI Launches the **interactive** VMware Workstation GUI (not the headless vmrun path) so you can watch or poke a clone by hand. On a normal desktop session it just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`: Reuse the `$VMX` captured in [§4.1](#41-clone-a-vm), or set it to a clone's VMX path (`VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1)` picks the first existing clone): ```bash ci vmware # bare GUI, no VM ci vmware --vmx "$VMX" # open the VM ci vmware --vmx "$VMX" --power-on # power on (-x) ci vmware --vmx "$VMX" --fullscreen # power on + fullscreen (-X) ``` Other flags: `--new-window` (`-n`), `--display :0` (force a display when `$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the command prints its PID and a log path under `/tmp/vmware-*.log`. **How the display works.** VMware connects through the X **host ACL**, not the cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant — normally already present (`xhost` lists `SI:localuser:ci-runner`). **Making the grant persistent.** An xhost grant is per-session and is wiped on logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart entry in the desktop user's home that re-applies it at every graphical login. You can also (re)run that step standalone: ```bash cd /opt/ci/local-ci-cd-system # --desktop-user defaults to $SUDO_USER sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone # To remove the grant and the autostart entry: sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall ``` This applies the grant immediately (when a display is reachable) and writes `~simone/.config/autostart/ci-runner-xhost.desktop` for future logins. **One-off, without the script.** Add the grant from the logged-in desktop user (not `ci-runner`): ```bash xhost +SI:localuser:ci-runner ``` `vmware` pre-flights the display with `xdpyinfo` and prints this exact command on failure rather than dying silently. --- ## 5. Credentials & template provisioning ### `creds set` Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`). The password never appears on the command line — verify it afterwards with the `validate host` keyring check. ```bash ci creds set --target BuildVMGuest --user ci_build # hidden prompt printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin ``` > The leading space before `printf` keeps the secret out of shell history. ### Template management Run `ci template --help` for the (many) per-template options. ```bash ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04) ci template deploy-rust # build the Rust Linux template VM (LinuxRust2404) ci template deploy-win # build the Windows Server 2025 template VM (see below) ci template prepare-linux # provision an existing Linux template over SSH ci template prepare-rust # provision the Rust Linux template over SSH ci template prepare-win # provision a Windows template over WinRM (Linux-host driven) ci template backup # timestamped backup of the template directories ``` ### `deploy-rust` / `prepare-rust` — the Rust Linux template (LinuxRust2404) A separate template for Rust workspaces (e.g. Tauri apps): same Ubuntu 24.04 cloud image and guest layout as `LinuxBuild2404`, but `prepare-rust` runs `template/Install-CIToolchain-LinuxRust2404.sh`, which additionally bakes: - the Tauri v2 webview/GTK development libraries (`libwebkit2gtk-4.1-dev`, `libgtk-3-dev`, `libayatana-appindicator3-dev`, `librsvg2-dev`); - a system-wide Rust toolchain via rustup under `/opt/rust` (`RUSTUP_HOME=/opt/rust/rustup`, `CARGO_HOME=/opt/rust/cargo`) with the `rustfmt` and `clippy` components. The tree is `chown`ed to `ci_build` **after** the installs, not only before — otherwise every write the build user makes (toolchain sync, dependency download into `CARGO_HOME/registry`, `cargo install`) fails with `Permission denied`. Exposure is threefold because the orchestrator runs build commands in a **non-login** shell (`exec_command` + `sh -c`), which never sources `profile.d`: `/etc/profile.d/rust.sh` for interactive shells, `/etc/environment` (read by `pam_env` at SSH session open) for the two home variables, and `/usr/local/bin` symlinks for the rustup proxies; - the `cargo-audit` binary (skip with `--no-cargo-audit`). The audit **advisory DB is never baked** — `cargo audit` fetches it at runtime. ```bash ci template deploy-rust # same flags as deploy-linux; 60 GB disk default ci template prepare-rust # default VMX: /LinuxRust2404/... # snapshot: BaseClean-LinuxRust ci template prepare-rust --rust-channel 1.97.1 # pin an exact toolchain # Both phases in one run, unattended. deploy and prepare stay separate # commands — --then-prepare only chains them, reusing the VM deploy left # running (no second power-on, no second IP detection). # # Run from the repo root: the guest toolchain script is looked up under # /template/. With --then-prepare this is checked BEFORE the deploy # starts, so a wrong directory costs two seconds instead of failing the # hand-off after a ~45-minute deploy. Pass --toolchain-script to run from # anywhere. cd /opt/ci/local-ci-cd-system ci template deploy-rust --then-prepare --rust-channel 1.97.1 ci template deploy-linux --then-prepare # from any directory ci template deploy-rust --then-prepare --rust-channel 1.97.1 \ --toolchain-script /opt/ci/local-ci-cd-system/template/Install-CIToolchain-LinuxRust2404.sh ``` `--snapshot-name` on `deploy-linux`/`deploy-rust` names the snapshot that `prepare-*` will create (`BaseClean-Linux` / `BaseClean-LinuxRust`) — the deploy commands take no snapshot of their own, unlike `deploy-win`. With `--then-prepare` the value is handed to `prepare-*`, so overriding it on the deploy side changes what actually gets created. Both `prepare-linux` and `prepare-rust` default `--vmx` to their template VMX under `` (`LinuxBuild2404` / `LinuxRust2404`) when neither `--vmx` nor `--ip` is passed — so the plain `ci template prepare-rust` just works after `deploy-rust`. Pass `--ip` to provision a running guest without VM lifecycle (no auto power-on/shutdown/snapshot). `--rust-channel` defaults to `stable` — a moving target: when a new stable ships, `rustup show` in a job re-downloads the toolchain at runtime. Pin an exact version for a fully deterministic snapshot (the consumer repo decides this via its `rust-toolchain.toml`). Refresh the snapshot when the pinned channel or the apt security baseline moves. Two ways to route jobs to it: 1. **Per job (no host change).** Pass `template-path` + `snapshot-name` to the `local-ci-build` composite action. This cannot leave a job unschedulable, which is why consumer repos default to it. 2. **By label** — `runs-on: rust-build`. This needs a **second act_runner instance**, not just an extra label on the existing one: act_runner reads a single `runner.envs` block per process and never tells the workflow which label picked the job (there is no `RUNNER_LABELS` variable — v1.0.4 exposes only `RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION`). A label can select a template only by selecting a runner whose envs name that template. ```bash # 1. mint a registration token from the stored GiteaPAT (0600 file, never # printed — pass it by path so it stays out of `ps` and shell history) sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \ --out /var/lib/ci/.rust-runner-regtoken # 2. provision + register + enable the unit sudo ./scripts/install-rust-runner.sh \ --token-file /var/lib/ci/.rust-runner-regtoken # 3. the token is single-use — destroy it sudo shred -u /var/lib/ci/.rust-runner-regtoken ``` Provisions `/var/lib/ci/runner-rust/` from `deploy/runner-rust-config.yaml` (label `rust-build:host`, `GITEA_CI_LINUX_TEMPLATE_PATH` → `LinuxRust2404`, `GITEA_CI_LINUX_SNAPSHOT_NAME` → `BaseClean-LinuxRust`), registers it, and installs `act-runner-rust.service`. Its `capacity` adds to the main runner's — keep the sum inside the host RAM budget (currently 2 + 1 = 3 concurrent build VMs). **Verify the label is announced before any workflow depends on it**: a `runs-on` value that no runner advertises leaves the job unschedulable with no error anywhere — silently zero runs. ```bash sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py \ --label rust-build # exit 1 unless an ONLINE runner announces it ``` Note that `runner.file` in the config is an **absolute** path. act_runner resolves a relative `file:` against the process CWD, so registering from anywhere else drops the registration state (which contains the runner token) into that directory and the daemon then fails with `registration file not found`. Both runner services share **one binary**, `/opt/ci/act_runner/act_runner`. Replacing it upgrades `act-runner.service` and `act-runner-rust.service` together — a staged rollout needs a second binary alongside it and the `ExecStart` of one unit pointed at it. Current version is v1.0.4; the upgrade to v2.x was evaluated and deferred (TODO §6.5). Note that `guest-os: 'Auto'` cannot pick a template by itself, for the same reason: the label is invisible to the action, and the VMX `guestOS` field is only readable *after* the clone. With `Auto` the action requires `template-path` **and** `snapshot-name` and fails fast otherwise, instead of silently falling back to the Windows template as it did before. ### `deploy-win` — build the Windows Server 2025 template Takes a Windows Server 2025 install ISO — downloaded from `--iso-url` (cached under `/var/lib/ci/iso/` by file name) or a local `--win-iso` — patches it (injects `autounattend.xml` + `post-install.ps1` and swaps the EFI boot image to the no-prompt variant), runs the unattended install, swaps the NIC e1000e → vmxnet3 and takes the cold `PostInstall` snapshot. VMware Tools are mounted as a second CD. ```bash # Download + deploy (e.g. a massgrave direct link), headless ci template deploy-win --iso-url 'https://example/win2025.iso' ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 # verify ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso # local ISO, watch install ``` **Finding / preparing an ISO from massgrave.** The massgrave "updated" link is behind a Cloudflare-Turnstile file host, so the download itself can't be scripted — but you can resolve the newest build and pre-patch a manual download: ```bash ci template check-win-iso # newest WS2025 build + URL + cache status # → download that URL in a browser into /var/lib/ci/iso/, then: ci template patch-win-iso --in /var/lib/ci/iso/ # → .patched.iso ci template deploy-win --win-iso /var/lib/ci/iso/.patched.iso ``` (`deploy-win` patches the ISO itself too, so `patch-win-iso` is optional — use it to produce/inspect the bootable no-prompt ISO separately.) Notes: - Provide **exactly one** of `--iso-url` / `--win-iso`. A downloaded ISO is cached by its URL file name and reused on re-runs. - The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO from massgrave. (UUP dump was dropped: on Linux it only yields the RTM `install.wim`, so Windows Update had to install the whole cumulative anyway.) - Needs `p7zip-full`, `xorriso` (installed by `setup-host-linux.sh`) and ~16 GiB scratch for the ISO repack. The patched ISO is cached per VM dir and only rebuilt when the base ISO or answer files change. - After it finishes: `ci template prepare-win` then `ci validate guest --winrm --guest-os windows`. --- ## 6. Maintenance ### `retention run` Purges old artifacts and logs. Always dry-run first. Retention switches to the aggressive window automatically when free space drops below `--min-free-gb`. ```bash ci retention run --what-if # dry run ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50 ``` ### Orphan sweep See [§4.6](#46-sweep-orphans) (`ci vm cleanup`). ### Publishing `.gitea/` — committed is not deployed act_runner never executes this working copy. For every job it re-clones the repository **from the Gitea server** at the ref the workflow names (`...local-ci-build@main`) and runs that copy. Anything under `.gitea/` is therefore inert until it is pushed, and the gap is invisible from here: `git log` shows the fix, CI runs the old file. This has already cost a consumer a day of debugging — 59 commits sat unpushed while a downstream repo chased a snapshot-name bug that was fixed locally. Check before announcing any action change as done: ```bash /opt/ci/venv/bin/python tools/ci-action-published.py # OK — .gitea/ matches origin/main; CI runs what you see here. # FAIL — lists the drifted files, exit 1 ``` Run it as the user who commits, **not** under `sudo -u ci-runner`: it fetches, which needs write access to `.git`, and `.git/FETCH_HEAD` here belongs to the committing user. It catches both ways of being unpublished, committed-but-unpushed and edited-but-uncommitted, and fetches first so it compares against the server rather than a stale remote-tracking ref (`--no-fetch` to skip the network). --- ## 7. Phase C testing Burn-in, benchmark, and smoke checks — all native Python, no pwsh. ### `smoke run` One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts: exit `0`, an artifact directory under `/var/lib/ci/artifacts//`, and a `job/success` event in `invoke-ci.jsonl`. ```bash ci smoke run --guest-os linux # no-op marker job (fastest) ci smoke run --guest-os windows ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end ``` ### `bench measure` Times the clone / start / IP / transport / destroy phases and appends a row to `benchmark.jsonl` (legacy field names preserved for trend continuity). ```bash ci bench measure --guest-os linux --iterations 1 ci bench measure --guest-os linux --iterations 4 # match historical baselines sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl ``` ### `bench run` Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts that all jobs exit `0`, no orphan clone directory is left behind, and no stale `vm-start.lock` remains. The JSON report lands under `/var/lib/ci/artifacts/bench/.json`. ```bash ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6) sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)" ``` --- ## 8. Cutover checklist & uninstalling pwsh Repeat for at least one week with pwsh **still installed** as an A/B safety net: - [ ] The normal `git push` CI pipeline stays green. - [ ] `ci validate host` is green daily. - [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks). - [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`. - [ ] `ci bench measure` timings are comparable to historical baselines. - [ ] No procedure forced a fallback to a `.ps1` on the Linux host. Only after a fully green week: ```bash pwsh --version sudo apt-get remove powershell ci validate host # must still exit 0 without pwsh ``` To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the original `.ps1` scripts remain in `scripts/archived/`. --- ## 9. Troubleshooting | Symptom | Cause | Fix | | -------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------- | | `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` | | `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` | | `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP | | `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` | | WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` | | IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` | | `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; `vmware-modules-rebuild.service` rebuilds) | | VMs won't start after a reboot — no `/dev/vmnet*`, `vmware-vmx` dies `status=237/KEYRING` | Host stack didn't come up at boot: `vmmon` unloaded + vmnet DHCP/NAT down (the flaky sysv `vmware.service` failed to generate — `systemd-sysv-generator: Cannot find unit vmware.service`) | `sudo systemctl start vmware-host.service`, then confirm it's enabled. Provisioned by `scripts/install-vmware-host-services.sh` — see [deploy/systemd](../deploy/systemd/README.md#servizi-vmware-host-prerequisito-avvio-vm) | | `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |