act_runner never runs this working copy. Every job re-clones the repository
from the Gitea server at the ref the workflow names, so a fix under .gitea/
is inert until it is pushed -- and nothing here says so. `git log` shows the
fix while CI keeps running the old file.
That gap cost a downstream repo a day: 59 commits sat unpushed while they
debugged a snapshot-name bug we had already fixed and announced as done.
ci-action-published.py compares the working tree against the remote ref,
which catches both committed-but-unpushed and edited-but-uncommitted in one
diff. It fetches first, so it compares against the server rather than a
remote-tracking ref that can itself be stale -- the same failure mode one
level down. Exit 1 on drift, 2 when the check could not run.
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force
against a powered-on VM, the three distinct causes behind the identical
"VMware Tools are not running" message, and the orphan snapshot deltas
--force leaves behind after deleting the .vmsd.
- docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check
first, with a Quick Reference entry.
- docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login
shell constraints on /opt/rust, --then-prepare, and how to route a label to a
template (a second runner instance, with the verification step).
- TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it
warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by
hand; the automated policy is still blind to what actually weighs.
- TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the
trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked
against the binary), so the two-runner design is not a stopgap.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
Verified the whole §4 chain end-to-end on the live host and fixed every
break:
- Capture the clone VMX from `vm new` stdout into $VMX and the guest IP
via `vmrun getGuestIPAddress` into $IP, then reuse both downstream.
Removes the fragile `Clone_dbg-1_*` globs that broke when more than one
clone shared a job-id ("unexpected extra argument").
- §4.3: `build run` does not inject the Gitea PAT (only `job` does), so the
private burnin-dummy repo can't be cloned in-guest. Use --host-source-dir
against the local copy instead.
- §4.4: collect from /opt/ci/output (where build run stages the artifact),
not the `dist` source dir — the old path failed with "tar: dist: Cannot
open".
- §4.1 now boots with --gui (manual debugging); §4.7 reuses $VMX.
Confirmed green: vm new → wait-ready → build → collect (build-info.txt +
manifest.json) → vm remove, and the §3 `job` pipeline (SUCCESS in 66s).
Mirror the `job` flag on the manual lifecycle: `vm new --start` was
hardcoded headless; add --gui/--nogui (default --nogui) so a clone can be
booted into the Workstation GUI window for interactive debugging. The
start banner reflects the mode.
Also update the runbook: §4.1 shows the --gui variant, and §3 notes that
the burnin-dummy build.sh is a ~30s fake build writing dist/build-info.txt
so the examples exercise the full pipeline.
Switch the §3 and §4.3 examples to Simone/burnin-dummy (the canonical
burn-in test repo): `bash build.sh` (fake ~30s build) producing a `dist/`
artifact, so the examples exercise a real build + artifact collection end
to end.
The Phase-2 VM start was hardcoded to headless. Expose a --gui/--nogui
toggle (default --nogui, preserving current behaviour); --gui starts the
clone with the Workstation GUI window (needs an X display, see §4.7). The
phase banner now reflects the chosen mode.
Replace the non-existent myrepo.git placeholder (which fails with
"repository not found") with this repo's URL and a harmless
`echo build-ok` build command, so §3 and §4.3 run as pasted.
The `job` command enforces --template-path (job.py:447) and defaults
--snapshot-name to BaseClean, which is wrong for Linux. The §3 example
omitted both and failed with "--template-path is required". Add both
flags (BaseClean-Linux) and note them as required.
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
Document `vmrun getGuestIPAddress` (run as ci-runner) for templates and
clones as the prerequisite for `validate guest`, with the AGENTS.md #1
"Error: on stdout" gotcha. Update the validate-guest Linux example to the
live template IP.
§4.7 now explains the grant is per-session and points to setup Step 8 /
the standalone install-vmware-gui-access.sh, keeping the manual one-off
xhost command as a fallback.
The GUI connects via the X host ACL (xhost), not the inherited XAUTHORITY
cookie (unreadable by ci-runner). On a normal session the grant is already
present and `ci vmgui` works with no --display/xhost dance — sudo forwards
DISPLAY. xhost is only needed if you hit "cannot open X display". Drop the
overstated "--display :0 required" framing from §4.7.
`vmgui` sent the child's stderr to /dev/null, so when the X display was
unreachable (the common case: sudo -u ci-runner loses the desktop user's
XAUTHORITY and xhost wasn't granted) the GUI died with "cannot open display"
and the command still printed success — "nothing opens".
Now it:
- pre-flights the display with xdpyinfo and aborts with the exact
`xhost +SI:localuser:ci-runner` hint instead of spawning a doomed process;
- drops an inherited XAUTHORITY that this user can't read (falls back to
host-based xhost auth);
- logs the GUI's stdio to /tmp/vmgui-*.log and grace-polls 1.5s, surfacing the
log if the GUI exits immediately;
- errors (non-zero) when no DISPLAY is resolvable.
Runbook §4.7 updated with the "if nothing opens" guidance. 13 tests; green ≥90%.
Add top-level `vmgui` to open the VMware Workstation GUI (renamed from the
earlier `vm open`). --vmx is optional: with it the GUI opens that VM, without
it the GUI starts bare. Power-on/fullscreen flags ignored (with a notice) when
no VMX is given. Needs an X display — sudo -u ci-runner strips $DISPLAY, so
--display / xhost; see runbook §4.7. 9 tests; suite green ≥90%.
Rewrite docs/RUNBOOK-PhaseC.md as a full manual-command reference: every
ci_orchestrator command with a working example as the ci-runner user, the `ci`
alias setup, quick-reference table, manual VM lifecycle (incl. vm open GUI
launch with the xhost/--display procedure), templating, maintenance, Phase C
testing, cutover checklist and a troubleshooting map.
Simone