input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
execute_ps is fully blocking with no streaming. Add _run_with_heartbeat()
which runs the WinRM call in a daemon thread and prints
" ... still running (Ns elapsed) ..." every 30 s so the operator can
distinguish a hung session from a slow Windows Update pass.
Used only for the toolchain script invocation (the slow step); all other
transport.run() calls keep the direct path.
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
Eliminates VMware-Tools IP-detect polling (observed: 39–177 s variance on
Linux host B7 benchmark). When [ip_pool] is configured the orchestrator:
1. Allocates a slot IP from a JSON-backed pool (fcntl-locked on Linux)
2. Injects guestinfo.ip-assignment / netmask / gateway into the cloned VMX
before vmrun start
3. Skips _wait_for_ip entirely — uses the pre-assigned IP immediately
4. Releases the slot in _destroy_clone (all paths: success/failure/interrupt)
New modules / changes:
src/ci_orchestrator/ip_pool.py IpSlotPool + _file_lock context manager
src/ci_orchestrator/config.py IpPoolConfig dataclass; [ip_pool] TOML
src/ci_orchestrator/commands/job.py _inject_guestinfo_ip helper; pool wiring
tests/python/test_ip_pool.py 12 unit tests (acquire/release/persist)
tests/python/test_config.py ip_pool config parsing tests
tests/python/test_commands_job.py pool path + _inject_guestinfo_ip tests
config.example.toml [ip_pool] section (commented example)
Coverage 90.02%; mypy --strict clean.
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
After a memory-heavy parallel build the guest can momentarily fail to
launch wsmprovhost (WSManFault 1018), which broke artifact collect
(now does Compress-Archive + fetch over WinRM). run() retries up to 4x
with exponential backoff on transient WSMan markers, dropping the
cached shell so a fresh one is opened once the guest recovers.
- SshTransport.run_streaming: incremental channel recv, forwards
stdout/stderr live (no more one-block-at-end). _linux_build uses it
with PYTHONUNBUFFERED so the guest python flushes promptly.
- _tail_lines split out for failure messages (no double print).
- Windows live streaming deferred (TODO 8.2) — pypsrp execute_ps
returns only at completion.
- build-ns7zip.yml: use-git-clone:'true' (in-guest), repo-url ->
https://gitea.emulab.it/... (guest-reachable; host SSH alias was
host-only). Requires GiteaPAT in the LocalSystem keyring or a public
nsis-plugin-ns7zip.
On a non-zero build the guest stdout/stderr was written unflushed
then a ClickException raised, so the cause was lost (only
"build command failed (exit 1)" surfaced). _report_build_output now
emits the output with markers, flushes both streams, and returns the
last lines which are appended to the failure message — diagnosable
even if act_runner buffering drops the body. Applied to Linux + Windows.
_host_clone hid all stderr ("exit 128" with no cause). Surface the
last stderr lines with scheme://user:pass@ credentials masked, so
failures (e.g. unresolved SSH alias under LocalSystem) are diagnosable.
The job command previously discarded --use-git-clone and --submodules
(`del`) and always in-guest cloned with submodules off — making both
action inputs dead no-ops (submodule repos silently broken).
- job.py: --use-git-clone/--host-clone and --submodules/--no-submodules
are real booleans defaulting to ON. Default = in-guest clone with
submodules. --host-clone clones on the host into a temp dir
(_host_clone) and transfers a zip; temp dir cleaned in finally.
clone_submodules now honoured (was hardcoded False).
- action.yml: use-git-clone / submodules default 'true'; forward the
explicit positive/negative flag so the Python default is overridden
deterministically.
- self-test.yml: add smoke-{windows,linux}-hostclone jobs covering the
host-side transport (default jobs already cover in-guest).
- tests: default-transport, host-clone, and --no-submodules coverage.
Windows produced an intermediate C:\CI\output\artifacts.zip that
upload-artifact then re-zipped (zip-in-zip), while Linux deposited raw
files. Make both paths consistent: the build stages raw outputs into a
collect dir; collect zips it only as a transport archive, fetches, and
extracts into the host artifact dir; actions/upload-artifact produces
the single final archive.
- _windows_build: param artifact_zip -> output_dir; stage raw files
(in-place when source IS the collect dir, else wipe+copy), no
Compress-Archive; absolute artifact-source resolved.
- _windows_collect: zip guest dir as transport, fetch, extract (mirror
_linux_collect); guest_path is now the collect directory.
- job.py / build_run CLI: pass output_dir; rename --guest-artifact-zip
-> --guest-output-dir.
- Tests updated for the raw-file contract.
git clone --depth 1 --branch X only fetches the branch tip. When the
branch advances between workflow trigger and runner pickup, checking
out the pinned commit failed with "reference is not a tree: <sha>".
Fetch that exact commit (git fetch --depth 1 origin <sha>) before
checkout, on both Linux (SSH) and Windows (WinRM) paths.
Two bugs made the Linux smoke "succeed" with an empty artifact:
- artifact_source was always joined as workdir + '/' + src, so an
absolute source (/opt/ci/output) became /opt/ci/build//opt/ci/output
and was never found — only a non-fatal WARNING was logged.
- `rm -rf output_dir` ran unconditionally; when the build writes
directly into the collect dir (the smoke case: output == source),
it deleted the build output before collection.
Resolve absolute vs workdir-relative; when the source is the output
dir, collect in place without wiping; a missing/empty source is now a
hard failure instead of a silent empty artifact.
Two Windows-guest build bugs:
- The build command was wrapped in `cmd /c`, so PowerShell-authored
commands failed ("'New-Item' is not recognized"). Run it as PS
(seed LASTEXITCODE=0 so PS-only commands exit cleanly).
- artifact_source was always Join-Path'd with the workdir; an absolute
source (C:\CI\output) became C:\CI\build\C:\CI\output. Use the path
as-is when rooted, else Join-Path the workdir.
pypsrp defaults to auth='negotiate', which attempts Kerberos with no
domain and fails under SSPI with SEC_E_UNKNOWN_CREDENTIALS for the
workgroup ci_build account. Default to 'ntlm' (configurable). The
guest username must be host-qualified (WINBUILD-2025\ci_build) and
stored that way in the credential vault.
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
Simone
Simone