Committed by accident in 3680f07 while working on the massgrave ISO
resolver: a saved ZeroFS search page, 52 KB, referenced by nothing in
src/, tests/, tools/, scripts/, template/ or deploy/. It carries a
Google Analytics tag and a third-party ad-injector script, neither of
which belongs in this repository.
Removed from HEAD rather than rebased out: furyrs cites 7cca938 by SHA
in their coordination thread, and rewriting history would dangle that
reference. The blob stays reachable in history.
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force
against a powered-on VM, the three distinct causes behind the identical
"VMware Tools are not running" message, and the orphan snapshot deltas
--force leaves behind after deleting the .vmsd.
- docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check
first, with a Quick Reference entry.
- docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login
shell constraints on /opt/rust, --then-prepare, and how to route a label to a
template (a second runner instance, with the verification step).
- TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it
warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by
hand; the automated policy is still blind to what actually weighs.
- TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the
trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked
against the binary), so the two-runner design is not a stopgap.
`runs-on: rust-build` needs a second act_runner instance, not another label on
the existing one: act_runner resolves a single runner.envs block per process
and never tells the workflow which label picked the job. A label can select a
template only by selecting a runner whose envs name that template. Registered
as ci-linux-rust (id 3), online, announcing rust-build.
- deploy/runner-rust-config.yaml: label rust-build:host only, envs pointing at
LinuxRust2404 + BaseClean-LinuxRust, its own cache dir (two processes must
not share one actcache database), capacity 1 on top of the main runner's 2.
runner.file is absolute: act_runner resolves a relative `file:` against the
process CWD, so registering from elsewhere drops the registration state --
which holds the runner token -- wherever it was invoked, and the daemon then
fails with "registration file not found".
- scripts/install-rust-runner.sh: idempotent, --dry-run, refuses to touch the
main runner directory, requires the template to exist first. Takes
--token-file rather than --token so the secret stays out of the process
argument list; the dry-run echo redacts it.
- tools/gitea-runner-token.py: mints a registration token from the stored
GiteaPAT and writes it to a 0600 file. Neither secret is ever printed. Sets
an explicit User-Agent because Cloudflare rejects urllib's default with
error 1010.
- tools/gitea-runner-list.py: --label NAME exits non-zero unless an ONLINE
runner announces it. A runs-on value no runner advertises leaves jobs
unschedulable with no error anywhere, which is exactly the failure the
consumer repo warned about.
Backups had grown to 195 GB -- half the disk, which was at 94% -- because
retention run covers only artifacts, logs and IP leases, none of which weigh
anything here. The three generations were byte-identical per template: the
templates rarely change, so the extra copies bought nothing. WinBuildTest alone
cost 26 GB per generation and has been unused since 2026-06-10.
act_runner does not expose the job's runs-on label to the workflow: v1.0.4
provides only RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION, verified against
the binary. The `$env:RUNNER_LABELS -match 'linux'` check was therefore always
false, so guest-os: 'Auto' silently resolved to Windows and selected the
Windows template for Linux jobs. It was masked only because every caller passes
guest-os explicitly or overrides template-path.
'Auto' cannot pick a template on its own -- the VMX guestOS field is only
readable after the clone -- so it now requires template-path and snapshot-name
and errors otherwise, instead of falling back to the wrong template.
Add GITEA_CI_LINUX_SNAPSHOT_NAME so a runner instance dedicated to a Linux
variant can declare its own snapshot instead of every workflow repeating it;
the Linux snapshot was hardcoded to BaseClean-Linux.
BREAKING: a workflow using the composite with neither guest-os nor
template-path now fails instead of silently using the Windows template.
Four defects, each of which could waste a full template rebuild:
1. getGuestIPAddress was called without -wait, so vmrun answered "The VMware
Tools are not running" unless the Tools happened to have registered the IP
at that exact instant. A guest that was booted, on the network and answering
SSH could still burn the entire 10-minute Step 7 budget. This race is the
likely cause of every intermittent "no guest IP" incident so far, including
ones previously attributed to wedged vmnet DHCP. AGENTS.md #10 already
prescribed -wait; the code did not follow it.
2. --force did not check whether the target VM was powered on. It deleted the
live .vmdk.lck as a stale artifact and Step 4 rewrote the base VMDK under
the running guest, killing it; the damage surfaced ten minutes later as an
IP timeout because `vmrun list` still showed the OLD instance.
3. Step 6 accepted "VM appears in vmrun list" as proof of power-on, which a
pre-existing instance satisfies. Check the exit code of `vmrun start`.
4. The cloud-init seed ISO was detached while the VM was still running, and
VMware rewrote the VMX from its in-memory copy at power-off, restoring
ide1:0.present = "TRUE". Every clone would boot with the seed attached
(AGENTS.md #6). Detach it in prepare, after power-off and before the
snapshot.
Also in this change:
- --then-prepare chains deploy into the matching prepare in one run, reusing
the VM deploy leaves running. The two commands stay separately invocable.
The guest toolchain script is resolved and verified BEFORE any VM work, so a
wrong working directory costs two seconds instead of failing the hand-off
after a 45-minute deploy; --toolchain-script allows running from anywhere.
- Corrected the final messages: deploy-rust told the user to run prepare-linux,
and both deploy commands reported a snapshot name (PostInstall-Linux) that no
command ever creates. The prepare hand-off hint passed a bare filename to
--template and omitted --snapshot, so copy-pasting it cloned BaseClean, the
Windows snapshot.
- template backup --exclude, repeatable, for retired templates. An unknown name
or --exclude without --all-templates is an error rather than a silent no-op.
test_deploy_linux_full_happy_path declared the VM already running before Step 6,
an impossible state that masked the new --force guard; its vmrun mock is now
stateful.
The chown ran before rustup and cargo-audit were installed, so everything
those created under RUSTUP_HOME/CARGO_HOME was left root-owned: only the two
empty directories existing at chown time belonged to ci_build. Measured on the
previous snapshot: 18350 root-owned paths, including cargo/registry. Every Rust
build with external dependencies failed with "Permission denied (os error 13)".
Re-assert ownership after the installs, and assert writability rather than mere
existence -- test -x on the binaries could not catch this, so a broken template
passed its own build and failed in consumers' jobs instead.
Also wire the toolchain for non-login shells. The orchestrator runs build
commands via paramiko exec_command + `sh -c`, which never sources
/etc/profile.d, so profile.d alone was not enough: add RUSTUP_HOME/CARGO_HOME
to /etc/environment (read by pam_env at SSH session open) and symlink the
rustup proxies into /usr/local/bin.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
Setting DelegationConsole/DelegationTerminal to the Windows Terminal GUID made
WT the console host for the cmd.exe shell. With Shell=cmd.exe and
AutoRestartShell=1, closing the WT-hosted shell window breaks the cmd<->WT host
handshake: Winlogon's relaunch leaves a bare grey desktop with no shell instead
of a fresh cmd. Classic conhost restarts cleanly and is ~82 MB lighter (no
WindowsTerminal process in the idle baseline).
Comment out the four delegation reg adds (HKCU + Default hive) so the shell
falls back to conhost. Kept commented (not deleted) to document the rationale.
schtasks.exe /Change on a non-existent task writes to stderr and exits 1.
Inside the disk-cleanup function (EAP=Stop, wrapped by the exit-code-sensitive
prepare-win cleanup invoker), that leaked $LASTEXITCODE=1 all the way to the
result marker — DISM runs via Start-Process and does not reset it — so the
step failed with "Disk cleanup failed (exit 1)" even though everything worked.
On a freshly-deployed WS2025 the UpdateOrchestrator "Schedule Scan" task does
not exist, so this fired every run.
Replace the schtasks loop with Get-ScheduledTask | Disable-ScheduledTask,
which no-ops silently on a missing task and sets no process exit code.
Follow-up to fb5de6f; applied to both toolchains.
The disk-cleanup function stopped wuauserv before DISM /StartComponentCleanup
/ResetBase but only re-enforced Start=4 AFTER it. /ResetBase drives heavy CBS
servicing (TiWorker); during that window WaaSMedicSvc woke, flipped wuauserv
Start back to Manual, and a "Schedule Scan" task started it. If the pass was
interrupted there (reboot/crash), the template was left with WU running and
Manual startup — a half-prepared state where Windows Update keeps firing on a
supposedly-locked-down build template.
Move the lockdown ahead of DISM: set wuauserv Start=4, apply the
WaaSMedicSvc registry write-deny ACL (the only reliable lever — the service
is protected and cannot itself be disabled), and disable the WindowsUpdate /
UpdateOrchestrator scan tasks, all BEFORE /ResetBase rather than only after.
The post-DISM re-enforce stays as a belt-and-suspenders check.
Applied identically to the WinBuild2025 and WinBuild2022 toolchains.
An ephemeral headless build VM carries a lot of dead weight. Split the trim
by cost so heavy servicing never runs under the live explorer shell (which
crash-loops during CBS churn at first logon):
post-install.win.ps1 (light, runs under explorer at first logon):
- disable waste services: audio, camera, CDP/sync, push (Wpn), telemetry
(DiagTrack/InventorySvc), search index (WSearch), DPS/WDI, MSDTC, TrkWks,
Themes, MapsBroker, RetailDemo, PcaSvc, WSAIFabricSvc; BITS -> on-demand
- disable per-user service templates (CDPUserSvc/OneSyncSvc/WpnUserService/...)
so no per-session clone spawns
- remove in-box apps: FeedbackHub/GetHelp/Getstarted/OfficeHub (appx +
provisioned), Snip & Sketch, Paint, RDP client (mstsc) — all best-effort
prepare-win (heavy, runs under cmd shell, before the final reboot so the
following DISM cleanup compacts the freed component store):
- Uninstall-WindowsFeature WindowsAdminCenterSetup, XPS-Viewer,
Wireless-Networking, Windows-Defender (-NoRestart)
- Remove-WindowsCapability AzureArcSetup (payload; auto-start already off)
- uninstall Edge + EdgeWebView2 and disable their leftover updater services
- gated by --skip-trim
Server Manager auto-launch was already disabled (task + reg + GPO policy);
its exe is part of the OS and cannot be uninstalled. Defender was already
runtime-off; removing the feature drops the AV stack entirely.
run_streaming used index-based _drain, which read new records from
ps.output / ps.streams.{information,error} but never released the consumed
ones. pypsrp retains every record (each a heavy deserialised PSObject) for
the whole invocation, so a long prepare-win (Windows Update + toolchain,
thousands of lines) grew the host python process RAM without bound.
Replace _drain with _drain_clear, which emits each pending record then
deletes the consumed prefix (`del seq[:n]`). Safe because draining runs on
the same thread as poll_invoke — nothing appends while we mutate. Track an
explicit had_error flag since the error stream is now cleared as we go.
PowerShell 5.1 strips an empty-string argument ('') when invoking a native
exe, so `reg add ... /d '' /f` reached reg.exe as `reg add ... /d /f`: reg
consumed /f as the data value, lost the force flag, and prompted
"overwrite? (Yes/No)" on stdin. FirstLogon runs headless with no stdin, so
post-install hung forever right after the "Explorer UX tweaks applied" log
line. Route both empty-value reg adds (HKCU + Default hive) through
`cmd /c`, which preserves the empty "" so the value is set and /f stays.
Regression from 8095f00 (wallpaper removal feature).
Prepend a guard that runs `autoload -Uz compinit && compinit` when compdef
isn't defined, so the generated zsh completion works without oh-my-zsh or if
sourced before compinit.
Generate a click completion file for bash/zsh and source it from the alias
block. Click's generated script queries completions by invoking the command
— but `ci` is a sudo shell function (env vars don't survive sudo, and `env
VAR=x ci` can't call a function), so rewrite the query to call the venv
python directly (read-only, no sudo) and drop zsh's `$+commands[ci]` guard.
Writes ~/.ci-completion.<shell>; removed cleanly on --uninstall (block strip).
Invoking `python -m ci_orchestrator` now reports prog_name "ci" — matching
the shell alias — so `--help` reads naturally and click shell completion
keys off the _CI_COMPLETE env var.
The base ISO's volume label (e.g. "26100.1_X64_EN-US") contains characters
outside strict ISO 9660 d-characters, so xorriso warned "-volid text does
not comply to ISO 9660 / ECMA 119 rules" on every repack. Map the label to
[A-Z0-9_] (cap 16). Cosmetic only — Windows boots from the El Torito images,
not the volid — but keeps the output clean.
Remove only the wallpaper image (Wallpaper="" + WallpaperStyle=0) and leave
Control Panel\Colors\Background at the WS2025 default, so the desktop shows
the default grey instead of black.
A headless CI build VM needs no wallpaper. post-install now clears the
Wallpaper value + sets a black background for the autologin Administrator
(HKCU) and the Default user hive (so any future profile inherits it).
autounattend FirstLogonCommands — which run post-install.ps1 — are triggered
by explorer starting as the shell. Setting Winlogon\Shell=cmd.exe in the
specialize pass stopped explorer, so post-install never ran on a fresh
deploy.
Remove the shell switch from specialize; explorer runs once for the first
logon (deploy-win does no heavy servicing, so no crash-loop risk) and
post-install.ps1 sets Shell=cmd.exe at its end — subsequent boots and all of
prepare-win run under cmd. Keep the (shell-independent) Server Manager
auto-launch disable in specialize.
Add specialize-pass RunSynchronousCommands to disable the ServerManager
scheduled task (the actual auto-launcher on WS2025) plus the HKLM/policy
DoNotOpen* keys, so Server Manager never pops up — not even at the first
logon, before post-install runs. Redundant with post-install.
Add a Microsoft-Windows-Deployment RunSynchronousCommand in the specialize
pass that sets Winlogon\Shell=cmd.exe before the first logon. A freshly
deployed template then never starts explorer — not even the transient
desktop during OOBE/install — closing the last window where the shell could
crash-loop during servicing. post-install.ps1 and prepare-win still
re-affirm it for redundancy / existing templates.
Up front, read Winlogon\Shell; if it isn't already cmd.exe, set it and do a
clean reboot before running the toolchain — so the shell is genuinely cmd
(not a killed-explorer half-state) for the whole servicing run. Idempotent:
when the shell is already cmd (re-runs), skip the reboot. Replaces the
earlier kill-explorer approach.
Set Winlogon\Shell=cmd.exe and drop the live explorer at the very start of
prepare-win, before the toolchain runs, so every reboot in the Windows
Update 3010 loop already boots without explorer — eliminating the shell
crash-loop during heavy servicing, not just at the final reboot. Still
re-affirmed before the final reboot and baked into post-install.
Setting Winlogon\Shell=cmd.exe only in the install-time post-install meant
existing templates (deployed earlier, or reverted to a pre-cmd snapshot)
kept explorer as the shell — re-running prepare-win never changed it, since
prepare runs the toolchain, not post-install. Set Shell=cmd.exe in the
prepare-win autologin re-affirm (before the final reboot) so any template
adopts it on the next prepare.
A headless CI build VM never needs the interactive desktop (builds run over
WinRM in Session 0), and a running explorer is fragile: heavy servicing
(Windows Update / DISM component cleanup) destabilises the live shell and
explorer crash-loops until the next reboot, spiking CPU and tripping the
pre-snapshot idle check.
Set Winlogon\Shell = cmd.exe in post-install so explorer never auto-starts
— removing that whole failure class at the root. Autologin is unchanged;
explorer remains launchable on demand (run explorer.exe, or via RDP).
Verified on the live WinBuildTest VM: post-reboot no explorer, 0 shell-stop
events, ~2% idle CPU.
Heavy servicing (DISM StartComponentCleanup in the -CleanupOnly step, like
WU finalization) destabilises the running shell session — explorer
crash-loops until the next reboot. Cleanup ran *after* the final reboot, so
the template ended on a crash-looping desktop and the idle check aborted.
Reorder: disk cleanup + autologin re-affirm now run before the final
reboot, making that reboot the last action before the idle check — the
shell comes up against fully-settled components and the snapshot captures a
clean, idle desktop.
The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
Microsoft rotated the unversioned https://dot.net/v1/dotnet-install.ps1, so
the pinned SHA256 was stale and the toolchain aborted ("Possible MITM or
wrong installer version"). Independently re-downloaded the official script
(builds.dotnet.microsoft.com) — its hash matches the guest's observed
value, confirming it is the legitimate current script, not tampering.
Update the pin in both WinBuild2025 and WinBuild2022 toolchains to
6585899AED55FF6AE13DBE1E8C3B878F2D00433520E7EFBE250B75DB948B7DA9 and note
that this pin must be refreshed whenever MS updates the script in place.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-win patched its input into vm_dir/install-noprompt.iso and only its
own sidecar cache could skip the repack — so feeding an already-patched ISO
(e.g. from `template patch-win-iso`, or via --win-iso) triggered a wasteful
second extract+repack that double-injected the answer files.
deploy-win now detects a patched input (autounattend.xml present at the ISO
root via `iso_patch.iso_has_root_file`) and uses it as-is, skipping Step 4.
Raw ISOs are still patched once and cached as before.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
The ISO repack (extract + xorriso, ~minutes) ran on every deploy. Cache it
per VM dir: store an inputs key (base ISO size+mtime + rendered
autounattend.xml + post-install.ps1) in a `.iso.inputs` sidecar next to
install-noprompt.iso. Step 4 skips the repack when the patched ISO and a
matching sidecar already exist, and re-patches automatically when the base
ISO or either answer file changes.
The patched install ISO is mounted as a CD and has setup.exe at its root
(Windows Server Setup). The first-logon CD scan matched it before the
VMware Tools CD and ran `setup.exe /s /v...`, which Windows Setup rejects
("unknown command-line option [/s]"), and copied the multi-GB install
media instead of the Tools payload.
Skip any CD that carries sources\install.wim / install.esd (the OS
installer) and prefer setup64.exe (VMware Tools x64 has it; Windows media
does not) when resolving the Tools setup.
A headless CI build VM has no USB devices — input is the VMware PS/2
keyboard/mouse + SVGA console. Drop the usb/ehci/usb_xhci controllers from
the generated VMX (one explicit `usb.present = "FALSE"`), trimming device
enumeration at boot. Step 9's usb* line stripping becomes a harmless no-op.
xorriso -extract reads the ISO9660 name tree, which on a Windows ISO has
no Rock Ridge — names come back upper-cased/truncated, so the
`efi/microsoft/boot/efisys_noprompt.bin` lookup failed ("missing from
ISO") and a repack from that tree would corrupt Windows' long file names.
Extract with 7z instead (reads the UDF/Joliet tree, preserving the real
lower-case paths), excluding 7z's synthetic `[BOOT]` dump. Verified
end-to-end on the cached 26100.32860 ISO: 7z extract → xorriso repack
yields a 5 GB ISO with autounattend.xml + post-install.ps1 at the root and
efisys_noprompt.bin as the UEFI El Torito image (no boot prompt).
Also: cap the Joliet volume id at 16 chars (silence xorriso warning), add
a 7z preflight check, and add p7zip-full to setup-host-linux.sh.
New `ci template deploy-win` builds the Windows Server 2025 template VM on
the Linux host, mirroring the legacy Deploy-WinBuild2025.ps1 ten-step flow
in Python:
- winimage/uup.py: resolve the newest (or pinned) WS2025 build via the
uupdump API and build a cached, MS-signed ISO via the uup converter.
- winimage/autounattend.py + assets/: render autounattend.xml and the
guest post-install.ps1 (ported from the PS heredoc; VMware Tools now
mounted as a 2nd CD, scanned at first logon).
- winimage/iso_patch.py: xorriso repack injecting the answer files at the
ISO root and swapping the EFI boot image to efisys_noprompt.bin
(no "press any key" prompt — essential for an unattended headless install).
- commands/template.py: the deploy-win command + _winbuild_vmx_lines
(the 82-key VMX), _remove_vmx_lines, _wait_install_flag, _vmrun_soft_stop.
Acquire → patch → VMDK → VMX → install → flag wait → strip media +
NIC e1000e→vmxnet3 → verify IP → cold snapshot.
Deterministic by build (--build), with --win-iso to bypass UUP. Tests
mirror the prepare-win mocking style; total coverage 93%.
Add aria2 + wimtools (wimlib-imagex) + xorriso + genisoimage + cabextract
+ chntpw to setup-host-linux.sh (Step 3) — the UUP dump → ISO conversion
and the no-prompt ISO repack that `ci template deploy-win` needs. Also
create the <CI_ROOT>/iso cache dir in the layout step.
Verified the whole §4 chain end-to-end on the live host and fixed every
break:
- Capture the clone VMX from `vm new` stdout into $VMX and the guest IP
via `vmrun getGuestIPAddress` into $IP, then reuse both downstream.
Removes the fragile `Clone_dbg-1_*` globs that broke when more than one
clone shared a job-id ("unexpected extra argument").
- §4.3: `build run` does not inject the Gitea PAT (only `job` does), so the
private burnin-dummy repo can't be cloned in-guest. Use --host-source-dir
against the local copy instead.
- §4.4: collect from /opt/ci/output (where build run stages the artifact),
not the `dist` source dir — the old path failed with "tar: dist: Cannot
open".
- §4.1 now boots with --gui (manual debugging); §4.7 reuses $VMX.
Confirmed green: vm new → wait-ready → build → collect (build-info.txt +
manifest.json) → vm remove, and the §3 `job` pipeline (SUCCESS in 66s).
Mirror the `job` flag on the manual lifecycle: `vm new --start` was
hardcoded headless; add --gui/--nogui (default --nogui) so a clone can be
booted into the Workstation GUI window for interactive debugging. The
start banner reflects the mode.
Also update the runbook: §4.1 shows the --gui variant, and §3 notes that
the burnin-dummy build.sh is a ~30s fake build writing dist/build-info.txt
so the examples exercise the full pipeline.
Switch the §3 and §4.3 examples to Simone/burnin-dummy (the canonical
burn-in test repo): `bash build.sh` (fake ~30s build) producing a `dist/`
artifact, so the examples exercise a real build + artifact collection end
to end.
The Phase-2 VM start was hardcoded to headless. Expose a --gui/--nogui
toggle (default --nogui, preserving current behaviour); --gui starts the
clone with the Workstation GUI window (needs an X display, see §4.7). The
phase banner now reflects the chosen mode.
Replace the non-existent myrepo.git placeholder (which fails with
"repository not found") with this repo's URL and a harmless
`echo build-ok` build command, so §3 and §4.3 run as pasted.
The `job` command enforces --template-path (job.py:447) and defaults
--snapshot-name to BaseClean, which is wrong for Linux. The §3 example
omitted both and failed with "--template-path is required". Add both
flags (BaseClean-Linux) and note them as required.
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
The shim mapping table now notes the .ps1 live in scripts/archived/
(frozen, lint-exempt, no live caller) and gains the Phase C host tools
(bench/smoke/validate host/creds set). Fix the "stays in PowerShell"
list: drop Test-CapacityBurnIn (archived) and the non-existent
Measure-CIBenchmark; keep retention/backup/template/e2e/Windows-side.
Document `vmrun getGuestIPAddress` (run as ci-runner) for templates and
clones as the prerequisite for `validate guest`, with the AGENTS.md #1
"Error: on stdout" gotcha. Update the validate-guest Linux example to the
live template IP.
Move 16 .ps1 files to scripts/archived/ — they have no live caller:
- 10 pure shims (New-BuildVM, Remove-BuildVM, Cleanup-OrphanedBuildVMs,
Wait-VMReady, Invoke-RemoteBuild, Get-BuildArtifacts, Watch-DiskSpace,
Watch-RunnerHealth, Get-CIJobSummary, Invoke-CIJob) that only forwarded
to `ci_orchestrator`; the live Gitea composite action now calls
`python -m ci_orchestrator job` directly.
- 6 Phase-C host tools reimplemented natively in Python (Start-BurnInTest
[+ -Linux], Test-CapacityBurnIn → bench; Test-Smoke → smoke run;
Validate-HostState → validate host; Set-CIGuestCredential → creds set).
Add scripts/archived/README.md mapping each file to its `ci` replacement,
and exempt scripts/archived/ from the PSScriptAnalyzer lint job.
Kept (still used / not yet ported): Register-CIScheduledTasks,
Backup-CITemplate, Invoke-RetentionPolicy, Set-TemplateSharedFolders,
Test-CIGuestWinRM, the e2e build harnesses, and all template/ guest
provisioning scripts.
§4.7 now explains the grant is per-session and points to setup Step 8 /
the standalone install-vmware-gui-access.sh, keeping the manual one-off
xhost command as a fallback.
`ci vmware` runs as ci-runner and relies on host-based X auth (xhost),
but the grant is per-session and wiped on logout / X restart, so the GUI
launch fails with "cannot open X display" after every desktop relogin.
Add scripts/install-vmware-gui-access.sh: installs an XDG autostart entry
in the desktop user's home that re-applies `xhost +SI:localuser:ci-runner`
at each graphical login, and applies it immediately when a display is
reachable. Desktop user resolves from --desktop-user / $CI_DESKTOP_USER /
$SUDO_USER (never root); service user from $CI_RUN_USER. --uninstall
removes the autostart entry and revokes the live grant.
Wire it into setup-host-linux.sh as Step 8, with --desktop-user (default
$SUDO_USER) and --skip-gui-access for headless hosts.
The GUI connects via the X host ACL (xhost), not the inherited XAUTHORITY
cookie (unreadable by ci-runner). On a normal session the grant is already
present and `ci vmgui` works with no --display/xhost dance — sudo forwards
DISPLAY. xhost is only needed if you hit "cannot open X display". Drop the
overstated "--display :0 required" framing from §4.7.
`vmgui` sent the child's stderr to /dev/null, so when the X display was
unreachable (the common case: sudo -u ci-runner loses the desktop user's
XAUTHORITY and xhost wasn't granted) the GUI died with "cannot open display"
and the command still printed success — "nothing opens".
Now it:
- pre-flights the display with xdpyinfo and aborts with the exact
`xhost +SI:localuser:ci-runner` hint instead of spawning a doomed process;
- drops an inherited XAUTHORITY that this user can't read (falls back to
host-based xhost auth);
- logs the GUI's stdio to /tmp/vmgui-*.log and grace-polls 1.5s, surfacing the
log if the GUI exits immediately;
- errors (non-zero) when no DISPLAY is resolvable.
Runbook §4.7 updated with the "if nothing opens" guidance. 13 tests; green ≥90%.
Add top-level `vmgui` to open the VMware Workstation GUI (renamed from the
earlier `vm open`). --vmx is optional: with it the GUI opens that VM, without
it the GUI starts bare. Power-on/fullscreen flags ignored (with a notice) when
no VMX is given. Needs an X display — sudo -u ci-runner strips $DISPLAY, so
--display / xhost; see runbook §4.7. 9 tests; suite green ≥90%.
Rewrite docs/RUNBOOK-PhaseC.md as a full manual-command reference: every
ci_orchestrator command with a working example as the ci-runner user, the `ci`
alias setup, quick-reference table, manual VM lifecycle (incl. vm open GUI
launch with the xhost/--display procedure), templating, maintenance, Phase C
testing, cutover checklist and a troubleshooting map.
scripts/install-ci-alias.sh installs `ci` (runs the orchestrator as ci-runner
via the prod venv) and `ci-me` (current user). Idempotent, marker-bounded;
supports --system (/etc/profile.d), --uninstall. setup-host-linux.sh gains a
step [7/7] that installs it system-wide, passing CI_PROD_PYTHON/CI_RUN_USER.
Open a VMX in the interactive Workstation GUI (not the headless vmrun path) for
hands-on debugging of a clone. Spawns `vmware` detached (own session, stdio
detached) and returns the child PID.
Flags: --vmx (required), --power-on (-x), --fullscreen (-X), --new-window (-n),
--display (sudo -u ci-runner strips $DISPLAY, so allow forcing it),
--vmware-path. Warns when no X display is set. 7 tests; vm.py 96%.
- CLAUDE.md: Linux host no longer runs pwsh; PS 5.1 constraints scoped to
Windows guests + Windows runner.
- RUNBOOK: add "Phase C — Python equivalents" table; §6-§13 benchmark records
kept as historical provenance.
- TODO §3.7: close $IsWindows shim bug (resolved-by-replacement — bench run
bypasses the shim).
- plans: mark C1-C8 done + docs C9; flag operational cutover/uninstall pending.
Apply analysis §4 option 1: pssa job runs on windows-build with
shell: powershell, gated to workflow_dispatch so push/PR don't queue it when
no Windows runner is registered. The python job stays on linux-build. No job
on a Linux runner uses pwsh anymore.
Remove _Common.psm1, _Transport.psm1 and their Pester test, plus their sole
consumer Measure-CIBenchmark.ps1 — all replaced by `bench measure`. Leaving
the script would dangle an Import-Module on a deleted .psm1.
Add four native command groups so the Linux host no longer needs pwsh for
manual ops. __main__ registers all four at once, so they ship together.
- bench run (C2): N concurrent jobs in-process via concurrent.futures,
per-round orphan-clone + stale-lock assertions. Replaces
Test-CapacityBurnIn.ps1 + Start-BurnInTest*.ps1. No
Start-Job/pwsh/$IsWindows.
- bench measure (C3): clone/start/IP/transport/destroy timings appended to
benchmark.jsonl (legacy field names kept). Replaces
Measure-CIBenchmark.ps1.
- smoke run (C4): one E2E job; asserts exit 0 + artifact dir + job/success
event. ns7zip/nsinnounp presets. Replaces Test-Smoke.ps1
+ Test-*Build-Linux.ps1.
- validate host (C5): vmrun/snapshot/keyring/permissions/systemd checks.
- validate guest(C6): transport probe with explicit failure cause.
- creds set (C6): keyring writer matching credentials.py read scheme;
password via stdin/prompt only. Replaces
Set-CIGuestCredential.ps1.
All groups import backends.load_backend only (Phase D ESXi path stays open).
Suite green, coverage 95.10% (gate 90%); new modules at 100%.
Phase B is closed (checklist Passi 1–9 [x], benchmark matrix §6–§14
recorded). Move the two finished working docs into the dated archive:
plans/PhaseB-user-checklist.md → plans/archived/2026-06-07/
plans/benchmark-windows-host.md → plans/archived/2026-06-07/
plans/ now holds only the forward-looking plans (idea-3 Phase C pwsh
removal, idea-4 Phase D ESXi, ideas-overview). Fixed the live links that
pointed at the moved/relocated files: idea-3 §4, idea-2 §8 (relative path),
and ideas-overview (idea-1/idea-2 now under archived/, Phase A+B marked ✅).
RUNBOOK §15: dual-boot operation — Linux⇄Windows switch procedure, per-OS
CI-stack table (roots/runner/vmrun), current GRUB behaviour (DEFAULT=0,
TIMEOUT=0 hidden → boots straight to Linux), post-boot verification commands
for both OSes, and an optional GRUB tweak (left unapplied — boot-to-Linux is
intentional so the box returns as the Linux CI host after any unattended
reboot).
Checklist: Passo 9 done; fixed the stale global tracking table (Passi 6–9
were still [ ] despite being complete). Phase B is now fully closed (1–9 [x]).
§10: single-job Measure-CIBenchmark on LinuxBuild2404 (DHCP, 10 iter) on the
Linux host — boot total 8.82s avg, near-deterministic even on DHCP (IP-acquire
7.45s, σ≈0.01). Guest-vs-guest table: Linux guest ~63% faster boot total than
the Windows static-IP guest (§8) on the same host.
plans/benchmark-windows-host.md: dual-boot plan to collect the symmetric
Windows-host numbers (§11 Win static, §12 Linux DHCP, §13 concurrent burn-in)
so the host×guest×IP-mode matrix is complete and Linux-vs-Windows is a
column-for-column comparison. Includes Windows paths, Credential Manager
prereqs, and the BaseClean-Linux snapshot caveat.
Concurrent capacity burn-in run on the Linux host as ci-runner with the
static-IP pool: WinBuild2025 40/40 PASS (~78.6s/round) and LinuxBuild2404
40/40 PASS (~70.2s/round) — 80/80 jobs, 20/20 rounds, near-deterministic
(static IP eliminates the DHCP IP-acquire variance of the §6/§7 baselines).
- RUNBOOK §9: concurrent burn-in result + ip-pool reset/recovery note.
- PhaseB-user-checklist Passo 7/8 marked done; fixed two checklist bugs:
run as ci-runner (the simone path needs g+w on the /var/lib/ci root for
ip-pool.tmp, not just the sub-dirs), and the Linux command was missing
-SnapshotName BaseClean-Linux (default BaseClean is Windows-only).
- idea-2-linux-host §8 DoD: Phase B complete (B1–B7 + >2 weeks stability).
- TODO §2.8 [P2]: ip-pool is not reconciled against live clones; a killed
job leaks its lease permanently and exhausts the 4-IP pool. Proposed fix.
Postmortem of the Linux template deploy/prepare 'no IP' investigation:
separates the three distinct causes (apt index stale from --skip-update,
deploy --force snapshot-chain, wedged host vmnet-dhcpd) and documents the
recovery (restart vmware-networks). Changelog entry summarising the
toolchain, xvfb, --force cleanup and cloud-init seal changes.
Add an --xvfb switch to 'build run' and 'job' that wraps the Linux build
command in 'xvfb-run -a sh -c ...', giving GUI toolkits (GTK4/PyGObject)
a headless $DISPLAY. Exported env vars precede xvfb-run so they
propagate into the virtual-display session. No-op (with a notice) for
Windows guests.
The local-ci-build composite action exposes it as the 'xvfb' input
(default false), forwarded as INPUT_XVFB -> --xvfb, mirroring the
use-shared-cache/skip-artifact wiring.
Docs: WORKFLOW-AUTHORING.md parameter table + Linux example note;
LINUX-TEMPLATE-SETUP.md Step 4a (the template ships python3-gi
gir1.2-gtk-4.0 xvfb). Adds build-run unit tests (wrap, plain, Windows
no-op).
deploy-linux --force reused the VM directory but left any prior snapshot
chain in place, so vmware-vdiskmanager refused the Step 4 resize with
'disk is part of a snapshot chain'. Add _clean_force_artifacts to remove
snapshot deltas (.vmdk), .vmsn, .vmsd and stale .lck before resize.
prepare-linux Step 7d seals the image: drop the stale cloud-init netplan
(leaving only the MAC-agnostic 99-ci-dhcp-allnics), regenerate networkd
config, and disable cloud-init so it never re-runs on linked clones.
deploy user-data sets package_update/upgrade: false so cloud-init does
not hold the dpkg lock at first boot and delay open-vm-tools past the
IP-detection poll; the step-7 IP timeout is raised 300s -> 600s as a
safety net for slow first boots.
Adds unit tests for _clean_force_artifacts (snapshot chain, lock dir,
clean-dir no-op) and updates prepare-linux stubs for the seal step.
Step 4a installs python3-gi gir1.2-gtk-4.0 xvfb for headless PyGObject
GUI tests, with import-gi and Xvfb assertions.
Step 1 now always runs apt-get update; --skip-update only skips the
(slow) full upgrade. Previously --skip-update skipped the index refresh
too, so a fresh package (gir1.2-gtk-4.0/xvfb) could not be located.
Linux clones could boot with the RTC interpreted as local time (guest
runs hours ahead of UTC). NTP then steps the clock backward at runtime,
landing after source mtimes were stamped, so make reported "Clock skew
detected" and future-dated files.
- build run: enable NTP and block until NTPSynchronized before stamping
the sources, so the backward correction happens pre-touch. Drop
`hwclock -s`, which read the local-time RTC and pushed the clock further
ahead.
- template prepare-linux: set-local-rtc 0 so clones interpret the RTC as
UTC and boot at the correct time; NTP then only nudges, never steps 2h.
Linux/SSH path only; WinRM branch untouched.
- Introduced a new test to verify behavior when SSH transport connects but is not ready, ensuring it returns False after the deadline.
- Added comprehensive tests for the `deploy-linux` template, covering various scenarios including missing public keys, VM directory existence, SHA256 mismatches, and failures during ISO creation and VM conversion.
- Implemented tests for SSH-related functions, including connection retries and streaming command execution, ensuring proper handling of exit statuses and progress indications.
- Enhanced existing tests to cover edge cases and ensure robustness in the deployment process.
- Implement tests for disk alert JSON output to include a message field when below threshold.
- Add tests for runner status when Gitea is online but no webhook URL is provided.
- Introduce tests for job report details to ensure no error line appears when there are no failure events.
- Add tests to skip empty JSONL files in job report listing.
- Install SIGTERM handler that raises KeyboardInterrupt so act_runner
workflow cancellations trigger the existing VM cleanup path
- Resolve guest_os before the ip_pool block and skip guestinfo IP
injection for Linux (template doesn't read VMX guestinfo to configure
network, causing SSH probe to fail against the wrong IP)
Simone