Open a VMX in the interactive Workstation GUI (not the headless vmrun path) for
hands-on debugging of a clone. Spawns `vmware` detached (own session, stdio
detached) and returns the child PID.
Flags: --vmx (required), --power-on (-x), --fullscreen (-X), --new-window (-n),
--display (sudo -u ci-runner strips $DISPLAY, so allow forcing it),
--vmware-path. Warns when no X display is set. 7 tests; vm.py 96%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add four native command groups so the Linux host no longer needs pwsh for
manual ops. __main__ registers all four at once, so they ship together.
- bench run (C2): N concurrent jobs in-process via concurrent.futures,
per-round orphan-clone + stale-lock assertions. Replaces
Test-CapacityBurnIn.ps1 + Start-BurnInTest*.ps1. No
Start-Job/pwsh/$IsWindows.
- bench measure (C3): clone/start/IP/transport/destroy timings appended to
benchmark.jsonl (legacy field names kept). Replaces
Measure-CIBenchmark.ps1.
- smoke run (C4): one E2E job; asserts exit 0 + artifact dir + job/success
event. ns7zip/nsinnounp presets. Replaces Test-Smoke.ps1
+ Test-*Build-Linux.ps1.
- validate host (C5): vmrun/snapshot/keyring/permissions/systemd checks.
- validate guest(C6): transport probe with explicit failure cause.
- creds set (C6): keyring writer matching credentials.py read scheme;
password via stdin/prompt only. Replaces
Set-CIGuestCredential.ps1.
All groups import backends.load_backend only (Phase D ESXi path stays open).
Suite green, coverage 95.10% (gate 90%); new modules at 100%.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an --xvfb switch to 'build run' and 'job' that wraps the Linux build
command in 'xvfb-run -a sh -c ...', giving GUI toolkits (GTK4/PyGObject)
a headless $DISPLAY. Exported env vars precede xvfb-run so they
propagate into the virtual-display session. No-op (with a notice) for
Windows guests.
The local-ci-build composite action exposes it as the 'xvfb' input
(default false), forwarded as INPUT_XVFB -> --xvfb, mirroring the
use-shared-cache/skip-artifact wiring.
Docs: WORKFLOW-AUTHORING.md parameter table + Linux example note;
LINUX-TEMPLATE-SETUP.md Step 4a (the template ships python3-gi
gir1.2-gtk-4.0 xvfb). Adds build-run unit tests (wrap, plain, Windows
no-op).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
deploy-linux --force reused the VM directory but left any prior snapshot
chain in place, so vmware-vdiskmanager refused the Step 4 resize with
'disk is part of a snapshot chain'. Add _clean_force_artifacts to remove
snapshot deltas (.vmdk), .vmsn, .vmsd and stale .lck before resize.
prepare-linux Step 7d seals the image: drop the stale cloud-init netplan
(leaving only the MAC-agnostic 99-ci-dhcp-allnics), regenerate networkd
config, and disable cloud-init so it never re-runs on linked clones.
deploy user-data sets package_update/upgrade: false so cloud-init does
not hold the dpkg lock at first boot and delay open-vm-tools past the
IP-detection poll; the step-7 IP timeout is raised 300s -> 600s as a
safety net for slow first boots.
Adds unit tests for _clean_force_artifacts (snapshot chain, lock dir,
clean-dir no-op) and updates prepare-linux stubs for the seal step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Introduced a new test to verify behavior when SSH transport connects but is not ready, ensuring it returns False after the deadline.
- Added comprehensive tests for the `deploy-linux` template, covering various scenarios including missing public keys, VM directory existence, SHA256 mismatches, and failures during ISO creation and VM conversion.
- Implemented tests for SSH-related functions, including connection retries and streaming command execution, ensuring proper handling of exit statuses and progress indications.
- Enhanced existing tests to cover edge cases and ensure robustness in the deployment process.
- Implement tests for disk alert JSON output to include a message field when below threshold.
- Add tests for runner status when Gitea is online but no webhook URL is provided.
- Introduce tests for job report details to ensure no error line appears when there are no failure events.
- Add tests to skip empty JSONL files in job report listing.
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Eliminates VMware-Tools IP-detect polling (observed: 39–177 s variance on
Linux host B7 benchmark). When [ip_pool] is configured the orchestrator:
1. Allocates a slot IP from a JSON-backed pool (fcntl-locked on Linux)
2. Injects guestinfo.ip-assignment / netmask / gateway into the cloned VMX
before vmrun start
3. Skips _wait_for_ip entirely — uses the pre-assigned IP immediately
4. Releases the slot in _destroy_clone (all paths: success/failure/interrupt)
New modules / changes:
src/ci_orchestrator/ip_pool.py IpSlotPool + _file_lock context manager
src/ci_orchestrator/config.py IpPoolConfig dataclass; [ip_pool] TOML
src/ci_orchestrator/commands/job.py _inject_guestinfo_ip helper; pool wiring
tests/python/test_ip_pool.py 12 unit tests (acquire/release/persist)
tests/python/test_config.py ip_pool config parsing tests
tests/python/test_commands_job.py pool path + _inject_guestinfo_ip tests
config.example.toml [ip_pool] section (commented example)
Coverage 90.02%; mypy --strict clean.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- SshTransport.run_streaming: incremental channel recv, forwards
stdout/stderr live (no more one-block-at-end). _linux_build uses it
with PYTHONUNBUFFERED so the guest python flushes promptly.
- _tail_lines split out for failure messages (no double print).
- Windows live streaming deferred (TODO 8.2) — pypsrp execute_ps
returns only at completion.
- build-ns7zip.yml: use-git-clone:'true' (in-guest), repo-url ->
https://gitea.emulab.it/... (guest-reachable; host SSH alias was
host-only). Requires GiteaPAT in the LocalSystem keyring or a public
nsis-plugin-ns7zip.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The job command previously discarded --use-git-clone and --submodules
(`del`) and always in-guest cloned with submodules off — making both
action inputs dead no-ops (submodule repos silently broken).
- job.py: --use-git-clone/--host-clone and --submodules/--no-submodules
are real booleans defaulting to ON. Default = in-guest clone with
submodules. --host-clone clones on the host into a temp dir
(_host_clone) and transfers a zip; temp dir cleaned in finally.
clone_submodules now honoured (was hardcoded False).
- action.yml: use-git-clone / submodules default 'true'; forward the
explicit positive/negative flag so the Python default is overridden
deterministically.
- self-test.yml: add smoke-{windows,linux}-hostclone jobs covering the
host-side transport (default jobs already cover in-guest).
- tests: default-transport, host-clone, and --no-submodules coverage.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Windows produced an intermediate C:\CI\output\artifacts.zip that
upload-artifact then re-zipped (zip-in-zip), while Linux deposited raw
files. Make both paths consistent: the build stages raw outputs into a
collect dir; collect zips it only as a transport archive, fetches, and
extracts into the host artifact dir; actions/upload-artifact produces
the single final archive.
- _windows_build: param artifact_zip -> output_dir; stage raw files
(in-place when source IS the collect dir, else wipe+copy), no
Compress-Archive; absolute artifact-source resolved.
- _windows_collect: zip guest dir as transport, fetch, extract (mirror
_linux_collect); guest_path is now the collect directory.
- job.py / build_run CLI: pass output_dir; rename --guest-artifact-zip
-> --guest-output-dir.
- Tests updated for the raw-file contract.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12 dedicated tests covering: (#9) clone_linked surfaces 'should not be powered on' as typed BackendError; (#10) is_running uses 'vmrun list' only and never falls back to getGuestIPAddress (incl. on vmrun list failure); (#11) two consecutive 'vm new' invocations with identical inputs MUST produce distinct clone names/destinations (timestamp suffix), guarding the orchestrator-side safety net against the machine-id duplicate-IP failure mode; (#12) SshTransport defaults to paramiko AutoAddPolicy with no known_hosts, never shells out to ssh-keygen/ssh.exe/scp.exe, wraps paramiko import/exec/SFTP errors as TransportConnectError. Coverage: 80.10% (gate 80%).
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
Simone
Simone