The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-win patched its input into vm_dir/install-noprompt.iso and only its
own sidecar cache could skip the repack — so feeding an already-patched ISO
(e.g. from `template patch-win-iso`, or via --win-iso) triggered a wasteful
second extract+repack that double-injected the answer files.
deploy-win now detects a patched input (autounattend.xml present at the ISO
root via `iso_patch.iso_has_root_file`) and uses it as-is, skipping Step 4.
Raw ISOs are still patched once and cached as before.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
The ISO repack (extract + xorriso, ~minutes) ran on every deploy. Cache it
per VM dir: store an inputs key (base ISO size+mtime + rendered
autounattend.xml + post-install.ps1) in a `.iso.inputs` sidecar next to
install-noprompt.iso. Step 4 skips the repack when the patched ISO and a
matching sidecar already exist, and re-patches automatically when the base
ISO or either answer file changes.
A headless CI build VM has no USB devices — input is the VMware PS/2
keyboard/mouse + SVGA console. Drop the usb/ehci/usb_xhci controllers from
the generated VMX (one explicit `usb.present = "FALSE"`), trimming device
enumeration at boot. Step 9's usb* line stripping becomes a harmless no-op.
xorriso -extract reads the ISO9660 name tree, which on a Windows ISO has
no Rock Ridge — names come back upper-cased/truncated, so the
`efi/microsoft/boot/efisys_noprompt.bin` lookup failed ("missing from
ISO") and a repack from that tree would corrupt Windows' long file names.
Extract with 7z instead (reads the UDF/Joliet tree, preserving the real
lower-case paths), excluding 7z's synthetic `[BOOT]` dump. Verified
end-to-end on the cached 26100.32860 ISO: 7z extract → xorriso repack
yields a 5 GB ISO with autounattend.xml + post-install.ps1 at the root and
efisys_noprompt.bin as the UEFI El Torito image (no boot prompt).
Also: cap the Joliet volume id at 16 chars (silence xorriso warning), add
a 7z preflight check, and add p7zip-full to setup-host-linux.sh.
New `ci template deploy-win` builds the Windows Server 2025 template VM on
the Linux host, mirroring the legacy Deploy-WinBuild2025.ps1 ten-step flow
in Python:
- winimage/uup.py: resolve the newest (or pinned) WS2025 build via the
uupdump API and build a cached, MS-signed ISO via the uup converter.
- winimage/autounattend.py + assets/: render autounattend.xml and the
guest post-install.ps1 (ported from the PS heredoc; VMware Tools now
mounted as a 2nd CD, scanned at first logon).
- winimage/iso_patch.py: xorriso repack injecting the answer files at the
ISO root and swapping the EFI boot image to efisys_noprompt.bin
(no "press any key" prompt — essential for an unattended headless install).
- commands/template.py: the deploy-win command + _winbuild_vmx_lines
(the 82-key VMX), _remove_vmx_lines, _wait_install_flag, _vmrun_soft_stop.
Acquire → patch → VMDK → VMX → install → flag wait → strip media +
NIC e1000e→vmxnet3 → verify IP → cold snapshot.
Deterministic by build (--build), with --win-iso to bypass UUP. Tests
mirror the prepare-win mocking style; total coverage 93%.
deploy-linux --force reused the VM directory but left any prior snapshot
chain in place, so vmware-vdiskmanager refused the Step 4 resize with
'disk is part of a snapshot chain'. Add _clean_force_artifacts to remove
snapshot deltas (.vmdk), .vmsn, .vmsd and stale .lck before resize.
prepare-linux Step 7d seals the image: drop the stale cloud-init netplan
(leaving only the MAC-agnostic 99-ci-dhcp-allnics), regenerate networkd
config, and disable cloud-init so it never re-runs on linked clones.
deploy user-data sets package_update/upgrade: false so cloud-init does
not hold the dpkg lock at first boot and delay open-vm-tools past the
IP-detection poll; the step-7 IP timeout is raised 300s -> 600s as a
safety net for slow first boots.
Adds unit tests for _clean_force_artifacts (snapshot chain, lock dir,
clean-dir no-op) and updates prepare-linux stubs for the seal step.
Linux clones could boot with the RTC interpreted as local time (guest
runs hours ahead of UTC). NTP then steps the clock backward at runtime,
landing after source mtimes were stamped, so make reported "Clock skew
detected" and future-dated files.
- build run: enable NTP and block until NTPSynchronized before stamping
the sources, so the backward correction happens pre-touch. Drop
`hwclock -s`, which read the local-time RTC and pushed the clock further
ahead.
- template prepare-linux: set-local-rtc 0 so clones interpret the RTC as
UTC and boot at the correct time; NTP then only nudges, never steps 2h.
Linux/SSH path only; WinRM branch untouched.
- Introduced a new test to verify behavior when SSH transport connects but is not ready, ensuring it returns False after the deadline.
- Added comprehensive tests for the `deploy-linux` template, covering various scenarios including missing public keys, VM directory existence, SHA256 mismatches, and failures during ISO creation and VM conversion.
- Implemented tests for SSH-related functions, including connection retries and streaming command execution, ensuring proper handling of exit statuses and progress indications.
- Enhanced existing tests to cover edge cases and ensure robustness in the deployment process.
- WinRM MaxProcessesPerShell: apply 100 if below threshold before asserting
- DesktopHeap SharedSection: apply 4096 KB registry value if below threshold before asserting
- Replace cleanmgr (hangs in WinRM Session 0, no window station) with direct
PowerShell cleanup for dumps, WER files and Recycle Bin; DISM already covers
component store and WU cache
- vcpkg git clone/checkout: wrap with EAP=Continue to suppress PS5.1
NativeCommandError on git stderr (gotcha #3 from AGENTS.md)
- template.py: filter CI_EXITCODE:N marker from displayed output
- ci-static-ip.ps1: switch from vmware-rpctool (removed in recent Tools) to
vmtoolsd --cmd; use temp-file redirect to capture output from native cmd
Cleanup now runs after the system reboots post-install (fully settled
state), not right after the WU loop. Sequence: WU loop → static IP →
validation → final reboot → WinRM ready → [cleanup] → autologin →
shutdown → snapshot.
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
pypsrp alters terminal tty settings during WinRM sessions. This causes
readline() at the snapshot prompt to receive \r (not \n) on Enter,
printing ^M repeatedly without ever unblocking.
Fix: save termios attributes at command start (before any WinRM work,
terminal is clean) and restore them with tcsetattr(TCSADRAIN) before
the interactive snapshot prompt. Gracefully skips save/restore when
stdin is not a tty (tests, pipes, CI).
Revert the pre-flight snapshot check approach (commit 97d9dab) — checking
at start was architecturally wrong (snapshot state may change during
provisioning). The prompt stays at the end where it belongs.
pypsrp alters terminal tty state during remote sessions. Any readline()
call after WinRM work receives \r instead of \n on Enter (terminal in
raw mode), causing the prompt to spin printing ^M and never return.
Fix: check for existing snapshot and ask the user at the very beginning
of the command, before any WinRM/pypsrp connections are opened, when the
terminal is still in canonical mode. If the user declines, abort early.
At the end, if the snapshot still exists (race-safe), delete silently
and recreate — no second prompt needed.
input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
execute_ps is fully blocking with no streaming. Add _run_with_heartbeat()
which runs the WinRM call in a daemon thread and prints
" ... still running (Ns elapsed) ..." every 30 s so the operator can
distinguish a hung session from a slow Windows Update pass.
Used only for the toolchain script invocation (the slow step); all other
transport.run() calls keep the direct path.
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Simone