Simone
14a67ead79
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force against a powered-on VM, the three distinct causes behind the identical "VMware Tools are not running" message, and the orphan snapshot deltas --force leaves behind after deleting the .vmsd. - docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check first, with a Quick Reference entry. - docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login shell constraints on /opt/rust, --then-prepare, and how to route a label to a template (a second runner instance, with the verification step). - TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by hand; the automated policy is still blind to what actually weighs. - TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked against the binary), so the two-runner design is not a stopgap.
692 lines
32 KiB
Markdown
692 lines
32 KiB
Markdown
# Runbook — Manual CLI Reference (Linux host, pwsh-free)
|
|
|
|
A complete, hands-on reference for every `ci_orchestrator` command you can run
|
|
by hand on the Linux host. Unless noted otherwise, commands run as the
|
|
**`ci-runner`** service user against the live configuration at
|
|
`/var/lib/ci/config.toml`.
|
|
|
|
Every command can be written two ways:
|
|
|
|
- **Full form** — `sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator <cmd>`
|
|
- **Short form** — `ci <cmd>`, after installing the alias (see [§0](#0-setup-do-once)).
|
|
|
|
The examples use the short form. The paths and template names below are the
|
|
real ones on this host, so most blocks are copy-paste-ready as written.
|
|
|
|
---
|
|
|
|
## 0. Setup (do once)
|
|
|
|
### 0.1 Refresh the production venv
|
|
|
|
Mandatory after every code change or merge — the venv is installed
|
|
non-editable, so changes only take effect once reinstalled:
|
|
|
|
```bash
|
|
cd /opt/ci/local-ci-cd-system
|
|
sudo /opt/ci/venv/bin/python -m pip install .
|
|
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help | grep -E 'bench|smoke|validate|creds'
|
|
```
|
|
|
|
### 0.2 Install the `ci` alias
|
|
|
|
```bash
|
|
cd /opt/ci/local-ci-cd-system
|
|
./scripts/install-ci-alias.sh # writes to ~/.bashrc and ~/.zshrc (idempotent)
|
|
source ~/.bashrc # or open a new shell
|
|
ci validate host # smoke-test the alias
|
|
```
|
|
|
|
Options: `--system` installs `/etc/profile.d/ci-alias.sh` for all users (needs
|
|
sudo); `--uninstall` removes the block. The script defines two helpers:
|
|
|
|
| Helper | Runs as | Use for |
|
|
| ------- | ------------------ | ----------------------------------------- |
|
|
| `ci` | `ci-runner` (sudo) | everything — writes keyring, starts VMs |
|
|
| `ci-me` | the current user | read-only checks where sudo is just noise |
|
|
|
|
### 0.3 The live environment
|
|
|
|
| Item | Value |
|
|
| ------------------ | -------------------------------------------------------------------------------------- |
|
|
| Service user | `ci-runner` · production venv `/opt/ci/venv/bin/python` |
|
|
| Configuration | `/var/lib/ci/config.toml` |
|
|
| Linux template | `/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx` (snapshot `BaseClean-Linux`) |
|
|
| Windows templates | `/var/lib/ci/templates/WinBuild2025` · `WinBuild2022` (snapshot `BaseClean`) |
|
|
| Clones / artifacts | `/var/lib/ci/build-vms` · `/var/lib/ci/artifacts` |
|
|
| Linux SSH access | key `/var/lib/ci/keys/ci_linux`, user `ci_build`, credential target `BuildVMGuest` |
|
|
|
|
---
|
|
|
|
## 1. Command index
|
|
|
|
| Command | Purpose | Section |
|
|
| ----------------------------- | ---------------------------------------------------- | --------------------------------------------------- |
|
|
| `ci validate host` | Check host prerequisites (no VM) | [§2](#2-health-monitoring--reporting) |
|
|
| `ci validate guest --host IP` | Probe a running guest's transport | [§2](#2-health-monitoring--reporting) |
|
|
| `ci monitor disk` | Free-space alert | [§2](#2-health-monitoring--reporting) |
|
|
| `ci monitor runner` | Watch and auto-restart act_runner | [§2](#2-health-monitoring--reporting) |
|
|
| `ci report job` | Job summaries from the JSONL logs | [§2](#2-health-monitoring--reporting) |
|
|
| `ci job ...` | **Full pipeline**: clone → build → collect → destroy | [§3](#3-full-pipeline-job) |
|
|
| `ci vm new ...` | Linked-clone a template | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci wait-ready --vmx ...` | Poll until SSH/WinRM is ready | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci build run ...` | Run a build inside a running guest | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci artifacts collect ...` | Copy artifacts out of a guest | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci vm remove --vmx ...` | Stop and delete one clone | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci vm cleanup` | Sweep orphaned clones and stale locks | [§4](#4-manual-vm-lifecycle) · [§6](#6-maintenance) |
|
|
| `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) |
|
|
| `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template deploy-rust` | Build the Rust Linux template VM (LinuxRust2404) | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template check-win-iso` | Report the newest WS2025 ISO from massgrave + cache status | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template patch-win-iso` | Inject autounattend + no-prompt boot into a Windows ISO | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template deploy-win` | Build the Windows Server 2025 template VM (download→patch→deploy) | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template prepare-rust` | Provision the Rust Linux template over SSH | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) |
|
|
| `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) |
|
|
| `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) |
|
|
| `ci bench measure ...` | Phase-timing benchmark | [§7](#7-phase-c-testing) |
|
|
| `ci bench run ...` | Concurrency burn-in | [§7](#7-phase-c-testing) |
|
|
| `ci smoke run ...` | One end-to-end job with assertions | [§7](#7-phase-c-testing) |
|
|
|
|
---
|
|
|
|
## 2. Health, monitoring & reporting
|
|
|
|
These are safe — none of them boots a VM.
|
|
|
|
### `validate host`
|
|
|
|
Run this first, every day. It checks vmrun, template snapshots, the keyring,
|
|
`/var/lib/ci` permissions, and the `act-runner` unit. A clean run exits `0`
|
|
with every line marked `[OK ]`; any `[FAIL]` line names the broken check.
|
|
|
|
```bash
|
|
ci validate host ; echo "exit=$?"
|
|
```
|
|
|
|
### Find a running guest's IP
|
|
|
|
`vmrun getGuestIPAddress` reads the address from VMware Tools, so the VM must
|
|
be **powered on** with Tools running. Run it as `ci-runner` — the VMs are
|
|
started by that user, so its `vmrun` inventory is the one that can see them.
|
|
|
|
```bash
|
|
# Which VMs are powered on right now
|
|
sudo -u ci-runner vmrun list
|
|
|
|
# IP of the Linux template
|
|
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx
|
|
|
|
# IP of a Windows template
|
|
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx
|
|
|
|
# IP of a build clone (glob the job directory)
|
|
sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/build-vms/Clone_*/Clone_*.vmx
|
|
```
|
|
|
|
> **Gotcha (AGENTS.md #1).** `getGuestIPAddress` can exit `0` and still print
|
|
> `Error: The VMware Tools are not running...` on stdout when Tools are down. A
|
|
> line beginning with `Error:` means *no IP yet* — wait for boot, don't parse it.
|
|
|
|
### `validate guest`
|
|
|
|
Connects to a booted guest, runs a trivial command, and reports the explicit
|
|
cause on failure (connect vs. auth vs. command). Capture the IP into a variable
|
|
so the probe is genuinely copy-paste-ready:
|
|
|
|
```bash
|
|
# Linux guest over SSH
|
|
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx)
|
|
ci validate guest --host "$IP" --ssh --guest-os linux
|
|
|
|
# Windows guest over WinRM
|
|
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2022/WinBuild2022.vmx)
|
|
ci validate guest --host "$IP" --winrm --guest-os windows
|
|
|
|
IP=$(sudo -u ci-runner vmrun getGuestIPAddress /var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx)
|
|
ci validate guest --host "$IP" --winrm --guest-os windows
|
|
```
|
|
|
|
### `monitor disk`
|
|
|
|
```bash
|
|
ci monitor disk --min-free-gb 50
|
|
ci monitor disk --min-free-gb 50 --json # machine-readable one-liner
|
|
ci monitor disk --min-free-gb 50 --webhook-url "$DISCORD_WEBHOOK"
|
|
```
|
|
|
|
### `monitor runner`
|
|
|
|
Restarts the runner up to N times per window. Optional: `--webhook-url`,
|
|
`--gitea-url`, `--gitea-credential-target`.
|
|
|
|
```bash
|
|
ci monitor runner --service-name act-runner --max-restarts 3
|
|
```
|
|
|
|
### `report job`
|
|
|
|
```bash
|
|
ci report job --last 10 # 10 most recent, as a table
|
|
ci report job --failed # failures only
|
|
ci report job --job-id smoke-20260607-2030 # phase breakdown for one job
|
|
ci report job --last 20 --json # JSON output
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Full pipeline (`job`)
|
|
|
|
This is the normal way to run a build: a single command clones the template,
|
|
waits for the guest, runs the build, collects artifacts, and destroys the
|
|
clone. It is exactly what the runner invokes on each push, and you can run it
|
|
by hand too.
|
|
|
|
The examples use `Simone/burnin-dummy`, whose `build.sh` simulates a real build
|
|
(sleeps ~30 s, then writes `dist/build-info.txt`) — so the run exercises the
|
|
full clone → build → artifact-collect → destroy path end to end.
|
|
|
|
```bash
|
|
ci job \
|
|
--job-id manual-$(date +%s) \
|
|
--template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
|
|
--snapshot-name BaseClean-Linux \
|
|
--repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
|
|
--branch main \
|
|
--guest-os linux \
|
|
--build-command "bash build.sh" \
|
|
--guest-artifact-source dist \
|
|
--use-git-clone
|
|
```
|
|
|
|
The VM boots headless by default. To watch it in the Workstation GUI while it
|
|
runs, add `--gui` (needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)):
|
|
|
|
```bash
|
|
ci job \
|
|
--job-id manual-$(date +%s) \
|
|
--template-path /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
|
|
--snapshot-name BaseClean-Linux \
|
|
--repo-url https://gitea.emulab.it/Simone/burnin-dummy.git \
|
|
--branch main \
|
|
--guest-os linux \
|
|
--build-command "bash build.sh" \
|
|
--guest-artifact-source dist \
|
|
--use-git-clone \
|
|
--gui
|
|
```
|
|
|
|
`--job-id`, `--repo-url`, `--branch`, and `--template-path` are required. The
|
|
default `--snapshot-name` is `BaseClean`, so Linux jobs must pass
|
|
`--snapshot-name BaseClean-Linux` explicitly. Other useful flags:
|
|
|
|
- **Source**: `--commit`, `--submodules` / `--no-submodules`,
|
|
`--use-git-clone` / `--host-clone`, `--use-shared-cache`
|
|
- **Build**: `--configuration Release`, `--guest-artifact-source dist`,
|
|
`--skip-artifact`, `--xvfb` (Linux GUI builds),
|
|
`--extra-env-json '{"KEY":"VALUE"}'`
|
|
- **VM**: `--guest-cpu N`, `--guest-memory-mb N`, `--template-path`,
|
|
`--snapshot-name`, `--ready-timeout`, `--gui` / `--nogui` (default `--nogui`,
|
|
headless; `--gui` opens the Workstation window and needs an X display —
|
|
see [§4.7](#47-open-a-vm-in-the-workstation-gui))
|
|
- **Auth**: `--gitea-credential-target` (private repos)
|
|
|
|
> For a zero-config end-to-end check, prefer `smoke run` ([§7](#7-phase-c-testing))
|
|
> over hand-rolling a `job` invocation.
|
|
|
|
---
|
|
|
|
## 4. Manual VM lifecycle
|
|
|
|
These are the stages that `job` runs internally. Drive them individually when
|
|
debugging a single phase. **Run the blocks below in the same shell session** —
|
|
each one reuses the `$VMX` and `$IP` variables captured by an earlier step, so
|
|
there are no fragile path globs to expand.
|
|
|
|
### 4.1 Clone a VM
|
|
|
|
`ci vm new` prints the clone's VMX path (and nothing else) on stdout, so capture
|
|
it straight into a variable. A timestamped job-id keeps repeated runs from
|
|
colliding.
|
|
|
|
```bash
|
|
VMX=$(ci vm new \
|
|
--template /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx \
|
|
--snapshot BaseClean-Linux \
|
|
--clone-base-dir /var/lib/ci/build-vms \
|
|
--job-id "dbg-$(date +%s)" \
|
|
--guest-os linux \
|
|
--start --gui)
|
|
echo "clone VMX: $VMX"
|
|
```
|
|
|
|
`--gui` opens the clone in the Workstation window so you can watch it while you
|
|
debug (it needs an X display — see [§4.7](#47-open-a-vm-in-the-workstation-gui)).
|
|
Drop `--gui` for a headless boot.
|
|
|
|
### 4.2 Wait until reachable, then capture the IP
|
|
|
|
`wait-ready` blocks until SSH is up. The clean guest IP comes from
|
|
`vmrun getGuestIPAddress`; capture it for the next steps.
|
|
|
|
```bash
|
|
ci wait-ready \
|
|
--vmx "$VMX" \
|
|
--guest-os linux \
|
|
--ssh-user ci_build \
|
|
--ssh-key-path /var/lib/ci/keys/ci_linux \
|
|
--timeout 180
|
|
|
|
IP=$(sudo -u ci-runner vmrun getGuestIPAddress "$VMX")
|
|
echo "guest IP: $IP"
|
|
```
|
|
|
|
For WinRM guests, add `--credential-target BuildVMGuest` to `wait-ready`.
|
|
|
|
### 4.3 Run the build
|
|
|
|
Unlike `job`, `ci build run` does **not** inject the Gitea PAT, so an in-guest
|
|
`--clone-url` only works for a public repo. For a private repo (like
|
|
`burnin-dummy`), upload a local source directory with `--host-source-dir`
|
|
instead — here the checked-out copy of `burnin-dummy` that ships in this repo.
|
|
The build stages its `--guest-artifact-source` (`dist`) into `/opt/ci/output`,
|
|
which is where step 4.4 collects from.
|
|
|
|
Windows guests use `--credential-target BuildVMGuest` in place of the SSH flags.
|
|
Repeatable `--extra-env KEY=VALUE`; add `--xvfb` for Linux GUI builds or
|
|
`--use-shared-cache` for the shared cache.
|
|
|
|
```bash
|
|
ci build run \
|
|
--ip-address "$IP" \
|
|
--guest-os linux \
|
|
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
|
|
--host-source-dir /opt/ci/local-ci-cd-system/gitea/burnin-dummy \
|
|
--build-command "bash build.sh" \
|
|
--guest-artifact-source dist
|
|
```
|
|
|
|
### 4.4 Collect artifacts
|
|
|
|
`build run` stages the artifact under `/opt/ci/output`, so collect from there
|
|
(not from the `dist` source directory).
|
|
|
|
```bash
|
|
ci artifacts collect \
|
|
--ip-address "$IP" \
|
|
--guest-os linux \
|
|
--ssh-user ci_build --ssh-key-path /var/lib/ci/keys/ci_linux \
|
|
--guest-artifact-path /opt/ci/output \
|
|
--host-artifact-dir /var/lib/ci/artifacts/dbg-manual \
|
|
--include-logs --job-id dbg-manual
|
|
```
|
|
|
|
### 4.5 Destroy the clone
|
|
|
|
```bash
|
|
ci vm remove --vmx "$VMX"
|
|
ci vm remove --vmx "$VMX" --force # skip the soft stop
|
|
```
|
|
|
|
### 4.6 Sweep orphans
|
|
|
|
Also a maintenance task — see [§6](#6-maintenance).
|
|
|
|
```bash
|
|
ci vm cleanup --what-if # list only
|
|
ci vm cleanup --max-age-hours 4 --lock-file /var/lib/ci/vm-start.lock
|
|
```
|
|
|
|
### 4.7 Open a VM in the Workstation GUI
|
|
|
|
Launches the **interactive** VMware Workstation GUI (not the headless vmrun
|
|
path) so you can watch or poke a clone by hand. On a normal desktop session it
|
|
just works — `ci vmware` behaves exactly like `sudo -u ci-runner vmware`:
|
|
|
|
Reuse the `$VMX` captured in [§4.1](#41-clone-a-vm), or set it to a clone's VMX
|
|
path (`VMX=$(ls /var/lib/ci/build-vms/Clone_*/*.vmx | head -1)` picks the first
|
|
existing clone):
|
|
|
|
```bash
|
|
ci vmware # bare GUI, no VM
|
|
ci vmware --vmx "$VMX" # open the VM
|
|
ci vmware --vmx "$VMX" --power-on # power on (-x)
|
|
ci vmware --vmx "$VMX" --fullscreen # power on + fullscreen (-X)
|
|
```
|
|
|
|
Other flags: `--new-window` (`-n`), `--display :0` (force a display when
|
|
`$DISPLAY` isn't forwarded), `--vmware-path`. The GUI is spawned detached; the
|
|
command prints its PID and a log path under `/tmp/vmware-*.log`.
|
|
|
|
**How the display works.** VMware connects through the X **host ACL**, not the
|
|
cookie. The inherited `XAUTHORITY` (the desktop user's `~/.Xauthority`) is
|
|
unreadable by `ci-runner`, so `vmware` drops it and relies on an xhost grant —
|
|
normally already present (`xhost` lists `SI:localuser:ci-runner`).
|
|
|
|
**Making the grant persistent.** An xhost grant is per-session and is wiped on
|
|
logout or X restart. `setup-host-linux.sh` (Step 8) installs an XDG autostart
|
|
entry in the desktop user's home that re-applies it at every graphical login.
|
|
You can also (re)run that step standalone:
|
|
|
|
```bash
|
|
cd /opt/ci/local-ci-cd-system
|
|
# --desktop-user defaults to $SUDO_USER
|
|
sudo CI_RUN_USER=ci-runner bash scripts/install-vmware-gui-access.sh --desktop-user simone
|
|
# To remove the grant and the autostart entry:
|
|
sudo bash scripts/install-vmware-gui-access.sh --desktop-user simone --uninstall
|
|
```
|
|
|
|
This applies the grant immediately (when a display is reachable) and writes
|
|
`~simone/.config/autostart/ci-runner-xhost.desktop` for future logins.
|
|
|
|
**One-off, without the script.** Add the grant from the logged-in desktop user
|
|
(not `ci-runner`):
|
|
|
|
```bash
|
|
xhost +SI:localuser:ci-runner
|
|
```
|
|
|
|
`vmware` pre-flights the display with `xdpyinfo` and prints this exact command
|
|
on failure rather than dying silently.
|
|
|
|
---
|
|
|
|
## 5. Credentials & template provisioning
|
|
|
|
### `creds set`
|
|
|
|
Stores the guest login in the keyring (replaces `Set-CIGuestCredential.ps1`).
|
|
The password never appears on the command line — verify it afterwards with the
|
|
`validate host` keyring check.
|
|
|
|
```bash
|
|
ci creds set --target BuildVMGuest --user ci_build # hidden prompt
|
|
printf '%s' 'SuperSecret123' | ci creds set --target BuildVMGuest --user ci_build --password-stdin
|
|
```
|
|
|
|
> The leading space before `printf` keeps the secret out of shell history.
|
|
|
|
### Template management
|
|
|
|
Run `ci template <sub> --help` for the (many) per-template options.
|
|
|
|
```bash
|
|
ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04)
|
|
ci template deploy-rust # build the Rust Linux template VM (LinuxRust2404)
|
|
ci template deploy-win # build the Windows Server 2025 template VM (see below)
|
|
ci template prepare-linux # provision an existing Linux template over SSH
|
|
ci template prepare-rust # provision the Rust Linux template over SSH
|
|
ci template prepare-win # provision a Windows template over WinRM (Linux-host driven)
|
|
ci template backup # timestamped backup of the template directories
|
|
```
|
|
|
|
### `deploy-rust` / `prepare-rust` — the Rust Linux template (LinuxRust2404)
|
|
|
|
A separate template for Rust workspaces (e.g. Tauri apps): same Ubuntu 24.04
|
|
cloud image and guest layout as `LinuxBuild2404`, but `prepare-rust` runs
|
|
`template/Install-CIToolchain-LinuxRust2404.sh`, which additionally bakes:
|
|
|
|
- the Tauri v2 webview/GTK development libraries
|
|
(`libwebkit2gtk-4.1-dev`, `libgtk-3-dev`, `libayatana-appindicator3-dev`,
|
|
`librsvg2-dev`);
|
|
- a system-wide Rust toolchain via rustup under `/opt/rust`
|
|
(`RUSTUP_HOME=/opt/rust/rustup`, `CARGO_HOME=/opt/rust/cargo`) with the
|
|
`rustfmt` and `clippy` components. The tree is `chown`ed to `ci_build`
|
|
**after** the installs, not only before — otherwise every write the build
|
|
user makes (toolchain sync, dependency download into
|
|
`CARGO_HOME/registry`, `cargo install`) fails with `Permission denied`.
|
|
Exposure is threefold because the orchestrator runs build commands in a
|
|
**non-login** shell (`exec_command` + `sh -c`), which never sources
|
|
`profile.d`: `/etc/profile.d/rust.sh` for interactive shells,
|
|
`/etc/environment` (read by `pam_env` at SSH session open) for the two
|
|
home variables, and `/usr/local/bin` symlinks for the rustup proxies;
|
|
- the `cargo-audit` binary (skip with `--no-cargo-audit`). The audit
|
|
**advisory DB is never baked** — `cargo audit` fetches it at runtime.
|
|
|
|
```bash
|
|
ci template deploy-rust # same flags as deploy-linux; 60 GB disk default
|
|
ci template prepare-rust # default VMX: <CI_TEMPLATES>/LinuxRust2404/...
|
|
# snapshot: BaseClean-LinuxRust
|
|
ci template prepare-rust --rust-channel 1.97.1 # pin an exact toolchain
|
|
|
|
# Both phases in one run, unattended. deploy and prepare stay separate
|
|
# commands — --then-prepare only chains them, reusing the VM deploy left
|
|
# running (no second power-on, no second IP detection).
|
|
#
|
|
# Run from the repo root: the guest toolchain script is looked up under
|
|
# <cwd>/template/. With --then-prepare this is checked BEFORE the deploy
|
|
# starts, so a wrong directory costs two seconds instead of failing the
|
|
# hand-off after a ~45-minute deploy. Pass --toolchain-script to run from
|
|
# anywhere.
|
|
cd /opt/ci/local-ci-cd-system
|
|
ci template deploy-rust --then-prepare --rust-channel 1.97.1
|
|
ci template deploy-linux --then-prepare
|
|
|
|
# from any directory
|
|
ci template deploy-rust --then-prepare --rust-channel 1.97.1 \
|
|
--toolchain-script /opt/ci/local-ci-cd-system/template/Install-CIToolchain-LinuxRust2404.sh
|
|
```
|
|
|
|
`--snapshot-name` on `deploy-linux`/`deploy-rust` names the snapshot that
|
|
`prepare-*` will create (`BaseClean-Linux` / `BaseClean-LinuxRust`) — the
|
|
deploy commands take no snapshot of their own, unlike `deploy-win`. With
|
|
`--then-prepare` the value is handed to `prepare-*`, so overriding it on the
|
|
deploy side changes what actually gets created.
|
|
|
|
Both `prepare-linux` and `prepare-rust` default `--vmx` to their template VMX
|
|
under `<CI_TEMPLATES>` (`LinuxBuild2404` / `LinuxRust2404`) when neither
|
|
`--vmx` nor `--ip` is passed — so the plain `ci template prepare-rust` just
|
|
works after `deploy-rust`. Pass `--ip` to provision a running guest without
|
|
VM lifecycle (no auto power-on/shutdown/snapshot).
|
|
|
|
`--rust-channel` defaults to `stable` — a moving target: when a new stable
|
|
ships, `rustup show` in a job re-downloads the toolchain at runtime. Pin an
|
|
exact version for a fully deterministic snapshot (the consumer repo decides
|
|
this via its `rust-toolchain.toml`). Refresh the snapshot when the pinned
|
|
channel or the apt security baseline moves.
|
|
|
|
Two ways to route jobs to it:
|
|
|
|
1. **Per job (no host change).** Pass `template-path` + `snapshot-name` to the
|
|
`local-ci-build` composite action. This cannot leave a job unschedulable,
|
|
which is why consumer repos default to it.
|
|
2. **By label** — `runs-on: rust-build`. This needs a **second act_runner
|
|
instance**, not just an extra label on the existing one: act_runner reads a
|
|
single `runner.envs` block per process and never tells the workflow which
|
|
label picked the job (there is no `RUNNER_LABELS` variable — v1.0.4 exposes
|
|
only `RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION`). A label can select a
|
|
template only by selecting a runner whose envs name that template.
|
|
|
|
```bash
|
|
# 1. mint a registration token from the stored GiteaPAT (0600 file, never
|
|
# printed — pass it by path so it stays out of `ps` and shell history)
|
|
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
|
|
--out /var/lib/ci/.rust-runner-regtoken
|
|
|
|
# 2. provision + register + enable the unit
|
|
sudo ./scripts/install-rust-runner.sh \
|
|
--token-file /var/lib/ci/.rust-runner-regtoken
|
|
|
|
# 3. the token is single-use — destroy it
|
|
sudo shred -u /var/lib/ci/.rust-runner-regtoken
|
|
```
|
|
|
|
Provisions `/var/lib/ci/runner-rust/` from `deploy/runner-rust-config.yaml`
|
|
(label `rust-build:host`, `GITEA_CI_LINUX_TEMPLATE_PATH` →
|
|
`LinuxRust2404`, `GITEA_CI_LINUX_SNAPSHOT_NAME` → `BaseClean-LinuxRust`),
|
|
registers it, and installs `act-runner-rust.service`. Its `capacity` adds
|
|
to the main runner's — keep the sum inside the host RAM budget (currently
|
|
2 + 1 = 3 concurrent build VMs).
|
|
|
|
**Verify the label is announced before any workflow depends on it**: a
|
|
`runs-on` value that no runner advertises leaves the job unschedulable with
|
|
no error anywhere — silently zero runs.
|
|
|
|
```bash
|
|
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py \
|
|
--label rust-build # exit 1 unless an ONLINE runner announces it
|
|
```
|
|
|
|
Note that `runner.file` in the config is an **absolute** path. act_runner
|
|
resolves a relative `file:` against the process CWD, so registering from
|
|
anywhere else drops the registration state (which contains the runner
|
|
token) into that directory and the daemon then fails with
|
|
`registration file not found`.
|
|
|
|
Both runner services share **one binary**, `/opt/ci/act_runner/act_runner`.
|
|
Replacing it upgrades `act-runner.service` and `act-runner-rust.service`
|
|
together — a staged rollout needs a second binary alongside it and the
|
|
`ExecStart` of one unit pointed at it. Current version is v1.0.4; the
|
|
upgrade to v2.x was evaluated and deferred (TODO §6.5).
|
|
|
|
Note that `guest-os: 'Auto'` cannot pick a template by itself, for the same
|
|
reason: the label is invisible to the action, and the VMX `guestOS` field is
|
|
only readable *after* the clone. With `Auto` the action requires
|
|
`template-path` **and** `snapshot-name` and fails fast otherwise, instead of
|
|
silently falling back to the Windows template as it did before.
|
|
|
|
### `deploy-win` — build the Windows Server 2025 template
|
|
|
|
Takes a Windows Server 2025 install ISO — downloaded from `--iso-url` (cached
|
|
under `/var/lib/ci/iso/` by file name) or a local `--win-iso` — patches it
|
|
(injects `autounattend.xml` + `post-install.ps1` and swaps the EFI boot image to
|
|
the no-prompt variant), runs the unattended install, swaps the NIC e1000e →
|
|
vmxnet3 and takes the cold `PostInstall` snapshot. VMware Tools are mounted as a
|
|
second CD.
|
|
|
|
```bash
|
|
# Download + deploy (e.g. a massgrave direct link), headless
|
|
ci template deploy-win --iso-url 'https://example/win2025.iso'
|
|
ci template deploy-win --iso-url 'https://example/win2025.iso' --iso-sha256 <HASH> # verify
|
|
ci template deploy-win --gui --win-iso /var/lib/ci/iso/win2025.iso # local ISO, watch install
|
|
```
|
|
|
|
**Finding / preparing an ISO from massgrave.** The massgrave "updated" link is
|
|
behind a Cloudflare-Turnstile file host, so the download itself can't be
|
|
scripted — but you can resolve the newest build and pre-patch a manual download:
|
|
|
|
```bash
|
|
ci template check-win-iso # newest WS2025 build + URL + cache status
|
|
# → download that URL in a browser into /var/lib/ci/iso/<filename>, then:
|
|
ci template patch-win-iso --in /var/lib/ci/iso/<filename> # → <filename>.patched.iso
|
|
ci template deploy-win --win-iso /var/lib/ci/iso/<filename>.patched.iso
|
|
```
|
|
|
|
(`deploy-win` patches the ISO itself too, so `patch-win-iso` is optional — use it
|
|
to produce/inspect the bootable no-prompt ISO separately.)
|
|
|
|
Notes:
|
|
- Provide **exactly one** of `--iso-url` / `--win-iso`. A downloaded ISO is
|
|
cached by its URL file name and reused on re-runs.
|
|
- The supplied ISO is the install media you give it — e.g. an updated WS2025 ISO
|
|
from massgrave. (UUP dump was dropped: on Linux it only yields the RTM
|
|
`install.wim`, so Windows Update had to install the whole cumulative anyway.)
|
|
- Needs `p7zip-full`, `xorriso` (installed by `setup-host-linux.sh`) and ~16 GiB
|
|
scratch for the ISO repack. The patched ISO is cached per VM dir and only
|
|
rebuilt when the base ISO or answer files change.
|
|
- After it finishes: `ci template prepare-win` then
|
|
`ci validate guest --winrm --guest-os windows`.
|
|
|
|
---
|
|
|
|
## 6. Maintenance
|
|
|
|
### `retention run`
|
|
|
|
Purges old artifacts and logs. Always dry-run first. Retention switches to the
|
|
aggressive window automatically when free space drops below `--min-free-gb`.
|
|
|
|
```bash
|
|
ci retention run --what-if # dry run
|
|
ci retention run --retention-days 30 --aggressive-retention-days 7 --min-free-gb 50
|
|
```
|
|
|
|
### Orphan sweep
|
|
|
|
See [§4.6](#46-sweep-orphans) (`ci vm cleanup`).
|
|
|
|
---
|
|
|
|
## 7. Phase C testing
|
|
|
|
Burn-in, benchmark, and smoke checks — all native Python, no pwsh.
|
|
|
|
### `smoke run`
|
|
|
|
One end-to-end job plus assertions (replaces `Test-Smoke.ps1`). It asserts:
|
|
exit `0`, an artifact directory under `/var/lib/ci/artifacts/<job-id>/`, and a
|
|
`job/success` event in `invoke-ci.jsonl`.
|
|
|
|
```bash
|
|
ci smoke run --guest-os linux # no-op marker job (fastest)
|
|
ci smoke run --guest-os windows
|
|
ci smoke run --guest-os linux --preset ns7zip # real Linux build, end-to-end
|
|
```
|
|
|
|
### `bench measure`
|
|
|
|
Times the clone / start / IP / transport / destroy phases and appends a row to
|
|
`benchmark.jsonl` (legacy field names preserved for trend continuity).
|
|
|
|
```bash
|
|
ci bench measure --guest-os linux --iterations 1
|
|
ci bench measure --guest-os linux --iterations 4 # match historical baselines
|
|
sudo -u ci-runner tail -n1 /var/lib/ci/artifacts/benchmark.jsonl
|
|
```
|
|
|
|
### `bench run`
|
|
|
|
Concurrency burn-in (replaces `Test-CapacityBurnIn.ps1`). Each round asserts
|
|
that all jobs exit `0`, no orphan clone directory is left behind, and no stale
|
|
`vm-start.lock` remains. The JSON report lands under
|
|
`/var/lib/ci/artifacts/bench/<timestamp>.json`.
|
|
|
|
```bash
|
|
ci bench run --guest-os linux --concurrency 2 --rounds 2 # quick confidence
|
|
ci bench run --guest-os linux --concurrency 4 --rounds 10 # standard gate
|
|
ci bench run --guest-os windows --concurrency 3 --rounds 10 # WinRM is unstable at 4x (TODO §3.6)
|
|
sudo -u ci-runner cat "$(ls -t /var/lib/ci/artifacts/bench/*.json | head -1)"
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Cutover checklist & uninstalling pwsh
|
|
|
|
Repeat for at least one week with pwsh **still installed** as an A/B safety net:
|
|
|
|
- [ ] The normal `git push` CI pipeline stays green.
|
|
- [ ] `ci validate host` is green daily.
|
|
- [ ] `ci bench run --guest-os linux --concurrency 4 --rounds 10` is green (no orphans or locks).
|
|
- [ ] `ci smoke run` is green on both `--guest-os linux` and `--guest-os windows`.
|
|
- [ ] `ci bench measure` timings are comparable to historical baselines.
|
|
- [ ] No procedure forced a fallback to a `.ps1` on the Linux host.
|
|
|
|
Only after a fully green week:
|
|
|
|
```bash
|
|
pwsh --version
|
|
sudo apt-get remove powershell
|
|
ci validate host # must still exit 0 without pwsh
|
|
```
|
|
|
|
To roll back, `sudo ./setup-host-linux.sh --with-pwsh` reinstalls pwsh; the
|
|
original `.ps1` scripts remain in `scripts/archived/`.
|
|
|
|
---
|
|
|
|
## 9. Troubleshooting
|
|
|
|
| Symptom | Cause | Fix |
|
|
| -------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------- |
|
|
| `No such command 'bench'` | Production venv not refreshed | [§0.1](#01-refresh-the-production-venv) — `pip install .` |
|
|
| `validate host` keyring `[FAIL]` | Credential missing or wrong user | [§5](#5-credentials--template-provisioning) — `creds set` |
|
|
| `validate guest` connect failure | VM not booted, wrong IP, or firewall | Confirm the VM is running and re-check its IP |
|
|
| `bench run` round FAIL: orphan | Clone was not destroyed | `ci vm cleanup`; check vmrun and `/var/lib/ci/build-vms` |
|
|
| WinRM jobs drop at concurrency 4 | Known instability (TODO §3.6) | Use `--concurrency 3` |
|
|
| IP-acquire timeout (Linux) | vmnet8 DHCP wedged | `sudo systemctl restart vmware-networks` |
|
|
| `vmrun` "operation was canceled" | vmmon/vmnet unbuilt after a kernel bump | `sudo vmware-modconfig --console --install-all` (or reboot; `vmware-modules-rebuild.service` rebuilds) |
|
|
| VMs won't start after a reboot — no `/dev/vmnet*`, `vmware-vmx` dies `status=237/KEYRING` | Host stack didn't come up at boot: `vmmon` unloaded + vmnet DHCP/NAT down (the flaky sysv `vmware.service` failed to generate — `systemd-sysv-generator: Cannot find unit vmware.service`) | `sudo systemctl start vmware-host.service`, then confirm it's enabled. Provisioned by `scripts/install-vmware-host-services.sh` — see [deploy/systemd](../deploy/systemd/README.md#servizi-vmware-host-prerequisito-avvio-vm) |
|
|
| `ci: command not found` | Alias not loaded | `source ~/.bashrc`, or re-run [§0.2](#02-install-the-ci-alias) |
|