A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-linux --force reused the VM directory but left any prior snapshot
chain in place, so vmware-vdiskmanager refused the Step 4 resize with
'disk is part of a snapshot chain'. Add _clean_force_artifacts to remove
snapshot deltas (.vmdk), .vmsn, .vmsd and stale .lck before resize.
prepare-linux Step 7d seals the image: drop the stale cloud-init netplan
(leaving only the MAC-agnostic 99-ci-dhcp-allnics), regenerate networkd
config, and disable cloud-init so it never re-runs on linked clones.
deploy user-data sets package_update/upgrade: false so cloud-init does
not hold the dpkg lock at first boot and delay open-vm-tools past the
IP-detection poll; the step-7 IP timeout is raised 300s -> 600s as a
safety net for slow first boots.
Adds unit tests for _clean_force_artifacts (snapshot chain, lock dir,
clean-dir no-op) and updates prepare-linux stubs for the seal step.
- Introduced a new test to verify behavior when SSH transport connects but is not ready, ensuring it returns False after the deadline.
- Added comprehensive tests for the `deploy-linux` template, covering various scenarios including missing public keys, VM directory existence, SHA256 mismatches, and failures during ISO creation and VM conversion.
- Implemented tests for SSH-related functions, including connection retries and streaming command execution, ensuring proper handling of exit statuses and progress indications.
- Enhanced existing tests to cover edge cases and ensure robustness in the deployment process.
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Simone