Add specialize-pass RunSynchronousCommands to disable the ServerManager
scheduled task (the actual auto-launcher on WS2025) plus the HKLM/policy
DoNotOpen* keys, so Server Manager never pops up — not even at the first
logon, before post-install runs. Redundant with post-install.
Add a Microsoft-Windows-Deployment RunSynchronousCommand in the specialize
pass that sets Winlogon\Shell=cmd.exe before the first logon. A freshly
deployed template then never starts explorer — not even the transient
desktop during OOBE/install — closing the last window where the shell could
crash-loop during servicing. post-install.ps1 and prepare-win still
re-affirm it for redundancy / existing templates.
Up front, read Winlogon\Shell; if it isn't already cmd.exe, set it and do a
clean reboot before running the toolchain — so the shell is genuinely cmd
(not a killed-explorer half-state) for the whole servicing run. Idempotent:
when the shell is already cmd (re-runs), skip the reboot. Replaces the
earlier kill-explorer approach.
Set Winlogon\Shell=cmd.exe and drop the live explorer at the very start of
prepare-win, before the toolchain runs, so every reboot in the Windows
Update 3010 loop already boots without explorer — eliminating the shell
crash-loop during heavy servicing, not just at the final reboot. Still
re-affirmed before the final reboot and baked into post-install.
Setting Winlogon\Shell=cmd.exe only in the install-time post-install meant
existing templates (deployed earlier, or reverted to a pre-cmd snapshot)
kept explorer as the shell — re-running prepare-win never changed it, since
prepare runs the toolchain, not post-install. Set Shell=cmd.exe in the
prepare-win autologin re-affirm (before the final reboot) so any template
adopts it on the next prepare.
A headless CI build VM never needs the interactive desktop (builds run over
WinRM in Session 0), and a running explorer is fragile: heavy servicing
(Windows Update / DISM component cleanup) destabilises the live shell and
explorer crash-loops until the next reboot, spiking CPU and tripping the
pre-snapshot idle check.
Set Winlogon\Shell = cmd.exe in post-install so explorer never auto-starts
— removing that whole failure class at the root. Autologin is unchanged;
explorer remains launchable on demand (run explorer.exe, or via RDP).
Verified on the live WinBuildTest VM: post-reboot no explorer, 0 shell-stop
events, ~2% idle CPU.
Heavy servicing (DISM StartComponentCleanup in the -CleanupOnly step, like
WU finalization) destabilises the running shell session — explorer
crash-loops until the next reboot. Cleanup ran *after* the final reboot, so
the template ended on a crash-looping desktop and the idle check aborted.
Reorder: disk cleanup + autologin re-affirm now run before the final
reboot, making that reboot the last action before the idle check — the
shell comes up against fully-settled components and the snapshot captures a
clean, idle desktop.
The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-win patched its input into vm_dir/install-noprompt.iso and only its
own sidecar cache could skip the repack — so feeding an already-patched ISO
(e.g. from `template patch-win-iso`, or via --win-iso) triggered a wasteful
second extract+repack that double-injected the answer files.
deploy-win now detects a patched input (autounattend.xml present at the ISO
root via `iso_patch.iso_has_root_file`) and uses it as-is, skipping Step 4.
Raw ISOs are still patched once and cached as before.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
The ISO repack (extract + xorriso, ~minutes) ran on every deploy. Cache it
per VM dir: store an inputs key (base ISO size+mtime + rendered
autounattend.xml + post-install.ps1) in a `.iso.inputs` sidecar next to
install-noprompt.iso. Step 4 skips the repack when the patched ISO and a
matching sidecar already exist, and re-patches automatically when the base
ISO or either answer file changes.
The patched install ISO is mounted as a CD and has setup.exe at its root
(Windows Server Setup). The first-logon CD scan matched it before the
VMware Tools CD and ran `setup.exe /s /v...`, which Windows Setup rejects
("unknown command-line option [/s]"), and copied the multi-GB install
media instead of the Tools payload.
Skip any CD that carries sources\install.wim / install.esd (the OS
installer) and prefer setup64.exe (VMware Tools x64 has it; Windows media
does not) when resolving the Tools setup.
A headless CI build VM has no USB devices — input is the VMware PS/2
keyboard/mouse + SVGA console. Drop the usb/ehci/usb_xhci controllers from
the generated VMX (one explicit `usb.present = "FALSE"`), trimming device
enumeration at boot. Step 9's usb* line stripping becomes a harmless no-op.
xorriso -extract reads the ISO9660 name tree, which on a Windows ISO has
no Rock Ridge — names come back upper-cased/truncated, so the
`efi/microsoft/boot/efisys_noprompt.bin` lookup failed ("missing from
ISO") and a repack from that tree would corrupt Windows' long file names.
Extract with 7z instead (reads the UDF/Joliet tree, preserving the real
lower-case paths), excluding 7z's synthetic `[BOOT]` dump. Verified
end-to-end on the cached 26100.32860 ISO: 7z extract → xorriso repack
yields a 5 GB ISO with autounattend.xml + post-install.ps1 at the root and
efisys_noprompt.bin as the UEFI El Torito image (no boot prompt).
Also: cap the Joliet volume id at 16 chars (silence xorriso warning), add
a 7z preflight check, and add p7zip-full to setup-host-linux.sh.
New `ci template deploy-win` builds the Windows Server 2025 template VM on
the Linux host, mirroring the legacy Deploy-WinBuild2025.ps1 ten-step flow
in Python:
- winimage/uup.py: resolve the newest (or pinned) WS2025 build via the
uupdump API and build a cached, MS-signed ISO via the uup converter.
- winimage/autounattend.py + assets/: render autounattend.xml and the
guest post-install.ps1 (ported from the PS heredoc; VMware Tools now
mounted as a 2nd CD, scanned at first logon).
- winimage/iso_patch.py: xorriso repack injecting the answer files at the
ISO root and swapping the EFI boot image to efisys_noprompt.bin
(no "press any key" prompt — essential for an unattended headless install).
- commands/template.py: the deploy-win command + _winbuild_vmx_lines
(the 82-key VMX), _remove_vmx_lines, _wait_install_flag, _vmrun_soft_stop.
Acquire → patch → VMDK → VMX → install → flag wait → strip media +
NIC e1000e→vmxnet3 → verify IP → cold snapshot.
Deterministic by build (--build), with --win-iso to bypass UUP. Tests
mirror the prepare-win mocking style; total coverage 93%.
Mirror the `job` flag on the manual lifecycle: `vm new --start` was
hardcoded headless; add --gui/--nogui (default --nogui) so a clone can be
booted into the Workstation GUI window for interactive debugging. The
start banner reflects the mode.
Also update the runbook: §4.1 shows the --gui variant, and §3 notes that
the burnin-dummy build.sh is a ~30s fake build writing dist/build-info.txt
so the examples exercise the full pipeline.
The Phase-2 VM start was hardcoded to headless. Expose a --gui/--nogui
toggle (default --nogui, preserving current behaviour); --gui starts the
clone with the Workstation GUI window (needs an X display, see §4.7). The
phase banner now reflects the chosen mode.
`vmgui` sent the child's stderr to /dev/null, so when the X display was
unreachable (the common case: sudo -u ci-runner loses the desktop user's
XAUTHORITY and xhost wasn't granted) the GUI died with "cannot open display"
and the command still printed success — "nothing opens".
Now it:
- pre-flights the display with xdpyinfo and aborts with the exact
`xhost +SI:localuser:ci-runner` hint instead of spawning a doomed process;
- drops an inherited XAUTHORITY that this user can't read (falls back to
host-based xhost auth);
- logs the GUI's stdio to /tmp/vmgui-*.log and grace-polls 1.5s, surfacing the
log if the GUI exits immediately;
- errors (non-zero) when no DISPLAY is resolvable.
Runbook §4.7 updated with the "if nothing opens" guidance. 13 tests; green ≥90%.
Add top-level `vmgui` to open the VMware Workstation GUI (renamed from the
earlier `vm open`). --vmx is optional: with it the GUI opens that VM, without
it the GUI starts bare. Power-on/fullscreen flags ignored (with a notice) when
no VMX is given. Needs an X display — sudo -u ci-runner strips $DISPLAY, so
--display / xhost; see runbook §4.7. 9 tests; suite green ≥90%.
Open a VMX in the interactive Workstation GUI (not the headless vmrun path) for
hands-on debugging of a clone. Spawns `vmware` detached (own session, stdio
detached) and returns the child PID.
Flags: --vmx (required), --power-on (-x), --fullscreen (-X), --new-window (-n),
--display (sudo -u ci-runner strips $DISPLAY, so allow forcing it),
--vmware-path. Warns when no X display is set. 7 tests; vm.py 96%.
Add four native command groups so the Linux host no longer needs pwsh for
manual ops. __main__ registers all four at once, so they ship together.
- bench run (C2): N concurrent jobs in-process via concurrent.futures,
per-round orphan-clone + stale-lock assertions. Replaces
Test-CapacityBurnIn.ps1 + Start-BurnInTest*.ps1. No
Start-Job/pwsh/$IsWindows.
- bench measure (C3): clone/start/IP/transport/destroy timings appended to
benchmark.jsonl (legacy field names kept). Replaces
Measure-CIBenchmark.ps1.
- smoke run (C4): one E2E job; asserts exit 0 + artifact dir + job/success
event. ns7zip/nsinnounp presets. Replaces Test-Smoke.ps1
+ Test-*Build-Linux.ps1.
- validate host (C5): vmrun/snapshot/keyring/permissions/systemd checks.
- validate guest(C6): transport probe with explicit failure cause.
- creds set (C6): keyring writer matching credentials.py read scheme;
password via stdin/prompt only. Replaces
Set-CIGuestCredential.ps1.
All groups import backends.load_backend only (Phase D ESXi path stays open).
Suite green, coverage 95.10% (gate 90%); new modules at 100%.
Add an --xvfb switch to 'build run' and 'job' that wraps the Linux build
command in 'xvfb-run -a sh -c ...', giving GUI toolkits (GTK4/PyGObject)
a headless $DISPLAY. Exported env vars precede xvfb-run so they
propagate into the virtual-display session. No-op (with a notice) for
Windows guests.
The local-ci-build composite action exposes it as the 'xvfb' input
(default false), forwarded as INPUT_XVFB -> --xvfb, mirroring the
use-shared-cache/skip-artifact wiring.
Docs: WORKFLOW-AUTHORING.md parameter table + Linux example note;
LINUX-TEMPLATE-SETUP.md Step 4a (the template ships python3-gi
gir1.2-gtk-4.0 xvfb). Adds build-run unit tests (wrap, plain, Windows
no-op).
deploy-linux --force reused the VM directory but left any prior snapshot
chain in place, so vmware-vdiskmanager refused the Step 4 resize with
'disk is part of a snapshot chain'. Add _clean_force_artifacts to remove
snapshot deltas (.vmdk), .vmsn, .vmsd and stale .lck before resize.
prepare-linux Step 7d seals the image: drop the stale cloud-init netplan
(leaving only the MAC-agnostic 99-ci-dhcp-allnics), regenerate networkd
config, and disable cloud-init so it never re-runs on linked clones.
deploy user-data sets package_update/upgrade: false so cloud-init does
not hold the dpkg lock at first boot and delay open-vm-tools past the
IP-detection poll; the step-7 IP timeout is raised 300s -> 600s as a
safety net for slow first boots.
Adds unit tests for _clean_force_artifacts (snapshot chain, lock dir,
clean-dir no-op) and updates prepare-linux stubs for the seal step.
Linux clones could boot with the RTC interpreted as local time (guest
runs hours ahead of UTC). NTP then steps the clock backward at runtime,
landing after source mtimes were stamped, so make reported "Clock skew
detected" and future-dated files.
- build run: enable NTP and block until NTPSynchronized before stamping
the sources, so the backward correction happens pre-touch. Drop
`hwclock -s`, which read the local-time RTC and pushed the clock further
ahead.
- template prepare-linux: set-local-rtc 0 so clones interpret the RTC as
UTC and boot at the correct time; NTP then only nudges, never steps 2h.
Linux/SSH path only; WinRM branch untouched.
- Introduced a new test to verify behavior when SSH transport connects but is not ready, ensuring it returns False after the deadline.
- Added comprehensive tests for the `deploy-linux` template, covering various scenarios including missing public keys, VM directory existence, SHA256 mismatches, and failures during ISO creation and VM conversion.
- Implemented tests for SSH-related functions, including connection retries and streaming command execution, ensuring proper handling of exit statuses and progress indications.
- Enhanced existing tests to cover edge cases and ensure robustness in the deployment process.
- Install SIGTERM handler that raises KeyboardInterrupt so act_runner
workflow cancellations trigger the existing VM cleanup path
- Resolve guest_os before the ip_pool block and skip guestinfo IP
injection for Linux (template doesn't read VMX guestinfo to configure
network, causing SSH probe to fail against the wrong IP)
- WinRM MaxProcessesPerShell: apply 100 if below threshold before asserting
- DesktopHeap SharedSection: apply 4096 KB registry value if below threshold before asserting
- Replace cleanmgr (hangs in WinRM Session 0, no window station) with direct
PowerShell cleanup for dumps, WER files and Recycle Bin; DISM already covers
component store and WU cache
- vcpkg git clone/checkout: wrap with EAP=Continue to suppress PS5.1
NativeCommandError on git stderr (gotcha #3 from AGENTS.md)
- template.py: filter CI_EXITCODE:N marker from displayed output
- ci-static-ip.ps1: switch from vmware-rpctool (removed in recent Tools) to
vmtoolsd --cmd; use temp-file redirect to capture output from native cmd
Cleanup now runs after the system reboots post-install (fully settled
state), not right after the WU loop. Sequence: WU loop → static IP →
validation → final reboot → WinRM ready → [cleanup] → autologin →
shutdown → snapshot.
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
pypsrp alters terminal tty settings during WinRM sessions. This causes
readline() at the snapshot prompt to receive \r (not \n) on Enter,
printing ^M repeatedly without ever unblocking.
Fix: save termios attributes at command start (before any WinRM work,
terminal is clean) and restore them with tcsetattr(TCSADRAIN) before
the interactive snapshot prompt. Gracefully skips save/restore when
stdin is not a tty (tests, pipes, CI).
Revert the pre-flight snapshot check approach (commit 97d9dab) — checking
at start was architecturally wrong (snapshot state may change during
provisioning). The prompt stays at the end where it belongs.
pypsrp alters terminal tty state during remote sessions. Any readline()
call after WinRM work receives \r instead of \n on Enter (terminal in
raw mode), causing the prompt to spin printing ^M and never return.
Fix: check for existing snapshot and ask the user at the very beginning
of the command, before any WinRM/pypsrp connections are opened, when the
terminal is still in canonical mode. If the user declines, abort early.
At the end, if the snapshot still exists (race-safe), delete silently
and recreate — no second prompt needed.
input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
execute_ps is fully blocking with no streaming. Add _run_with_heartbeat()
which runs the WinRM call in a daemon thread and prints
" ... still running (Ns elapsed) ..." every 30 s so the operator can
distinguish a hung session from a slow Windows Update pass.
Used only for the toolchain script invocation (the slow step); all other
transport.run() calls keep the direct path.
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
Eliminates VMware-Tools IP-detect polling (observed: 39–177 s variance on
Linux host B7 benchmark). When [ip_pool] is configured the orchestrator:
1. Allocates a slot IP from a JSON-backed pool (fcntl-locked on Linux)
2. Injects guestinfo.ip-assignment / netmask / gateway into the cloned VMX
before vmrun start
3. Skips _wait_for_ip entirely — uses the pre-assigned IP immediately
4. Releases the slot in _destroy_clone (all paths: success/failure/interrupt)
New modules / changes:
src/ci_orchestrator/ip_pool.py IpSlotPool + _file_lock context manager
src/ci_orchestrator/config.py IpPoolConfig dataclass; [ip_pool] TOML
src/ci_orchestrator/commands/job.py _inject_guestinfo_ip helper; pool wiring
tests/python/test_ip_pool.py 12 unit tests (acquire/release/persist)
tests/python/test_config.py ip_pool config parsing tests
tests/python/test_commands_job.py pool path + _inject_guestinfo_ip tests
config.example.toml [ip_pool] section (commented example)
Coverage 90.02%; mypy --strict clean.
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Simone