- Implement tests for disk alert JSON output to include a message field when below threshold.
- Add tests for runner status when Gitea is online but no webhook URL is provided.
- Introduce tests for job report details to ensure no error line appears when there are no failure events.
- Add tests to skip empty JSONL files in job report listing.
- Install SIGTERM handler that raises KeyboardInterrupt so act_runner
workflow cancellations trigger the existing VM cleanup path
- Resolve guest_os before the ip_pool block and skip guestinfo IP
injection for Linux (template doesn't read VMX guestinfo to configure
network, causing SSH probe to fail against the wrong IP)
- Add -StaticIP/-Netmask/-Gateway params; injects guestinfo into cloned
VMX before start, mirroring job.py _inject_guestinfo_ip behaviour
- Add -GuestInfoOnly switch to Get-GuestIPAddress: skips vmrun
getGuestIPAddress fallback to prevent race where transient DHCP address
is returned before ci-static-ip.ps1 finishes reconfiguring the NIC
- Benchmark passes -GuestInfoOnly automatically when -StaticIP is set
- staticIP field added to benchmark.jsonl for DHCP vs static comparison
- WinRM MaxProcessesPerShell: apply 100 if below threshold before asserting
- DesktopHeap SharedSection: apply 4096 KB registry value if below threshold before asserting
- Replace cleanmgr (hangs in WinRM Session 0, no window station) with direct
PowerShell cleanup for dumps, WER files and Recycle Bin; DISM already covers
component store and WU cache
- vcpkg git clone/checkout: wrap with EAP=Continue to suppress PS5.1
NativeCommandError on git stderr (gotcha #3 from AGENTS.md)
- template.py: filter CI_EXITCODE:N marker from displayed output
- ci-static-ip.ps1: switch from vmware-rpctool (removed in recent Tools) to
vmtoolsd --cmd; use temp-file redirect to capture output from native cmd
Cleanup now runs after the system reboots post-install (fully settled
state), not right after the WU loop. Sequence: WU loop → static IP →
validation → final reboot → WinRM ready → [cleanup] → autologin →
shutdown → snapshot.
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
pypsrp alters terminal tty settings during WinRM sessions. This causes
readline() at the snapshot prompt to receive \r (not \n) on Enter,
printing ^M repeatedly without ever unblocking.
Fix: save termios attributes at command start (before any WinRM work,
terminal is clean) and restore them with tcsetattr(TCSADRAIN) before
the interactive snapshot prompt. Gracefully skips save/restore when
stdin is not a tty (tests, pipes, CI).
Revert the pre-flight snapshot check approach (commit 97d9dab) — checking
at start was architecturally wrong (snapshot state may change during
provisioning). The prompt stays at the end where it belongs.
pypsrp alters terminal tty state during remote sessions. Any readline()
call after WinRM work receives \r instead of \n on Enter (terminal in
raw mode), causing the prompt to spin printing ^M and never return.
Fix: check for existing snapshot and ask the user at the very beginning
of the command, before any WinRM/pypsrp connections are opened, when the
terminal is still in canonical mode. If the user declines, abort early.
At the end, if the snapshot still exists (race-safe), delete silently
and recreate — no second prompt needed.
input() with a prompt string on some terminals (zsh + Linux Mint) echoes
^M and the prompt doesn't flush before reading, causing the prompt to
appear stuck or not accept input.
Replace with explicit sys.stdout.write()+flush() + sys.stdin.readline()
which guarantees the prompt is flushed before blocking on stdin, and
readline() + .strip() handles both \n and \r\n line endings correctly.
Tests updated to use CliRunner(input=) which populates sys.stdin inside
the runner context (monkeypatching sys.stdin.readline directly is ignored
by CliRunner which replaces sys.stdin entirely).
click.confirm() doesn't strip \r — "y\r" doesn't match "y" and the prompt
loops or fails. Replace with raw input().strip().rstrip('\r').lower() which
handles both \n and \r\n line endings.
Also add --recreate-snapshot flag to skip the interactive prompt entirely
(useful for scripted re-provisioning).
Update snapshot_exists_skip/recreate tests to monkeypatch builtins.input
instead of using CliRunner input=.
execute_ps is fully blocking with no streaming. Add _run_with_heartbeat()
which runs the WinRM call in a daemon thread and prints
" ... still running (Ns elapsed) ..." every 30 s so the operator can
distinguish a hung session from a slow Windows Update pass.
Used only for the toolchain script invocation (the slow step); all other
transport.run() calls keep the direct path.
- --store-credential: after snapshot, writes build-user credentials into
the system keyring using the two-entry scheme (service:meta + service)
that KeyringCredentialStore.get() reads. Works on Windows (Credential
Manager) and Linux (SecretService / file backend).
- --credential-target: keyring target name, default BuildVMGuest (matches
config.guest_cred_target).
- Rewrote module docstring: full prose documentation of both `template
backup` and `template prepare-win` including all 12 provisioning steps.
- 2 new tests covering --store-credential with custom and default targets.
- mypy --strict clean; coverage 90.03%.
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
Without Import-Module PSWSMan the native WSMan library isn't registered
and New-PSSession hangs/fails silently. Import guarded by $IsWindows so
Windows PS 5.1 is unaffected.
WSMan TrustedHosts is a Windows WinRM client concept — does not exist
on Linux. Guard the configure+restore blocks with $isLinuxHost so the
Prepare scripts run without elevation errors from a Linux host.
-SkipCACheck/-SkipCNCheck in sessionOptions is sufficient on Linux.
Parameter not available in PS 7 on Linux. -SkipCACheck -SkipCNCheck
sufficient for self-signed lab certs; revocation check is irrelevant
for self-signed certs regardless.
Test-NetConnection is Windows-only. Replace with cross-platform TcpClient
async-connect helper (same pattern as Measure-CIBenchmark.ps1) so the
Prepare scripts run on the Linux host under PowerShell.
Windows (Prepare-WinBuild2025/2022): read ci-static-ip.ps1 from
guest-setup/windows/ on the host, push content over WinRM, register the
CI-StaticIp scheduled task (AtStartup/SYSTEM). Adds StaticIpTask and
StaticIpScript post-setup assertions.
Linux (Install-CIToolchain-Linux2404.sh): embed ci-report-ip.sh and
ci-report-ip.service content inline via heredoc tee, replacing the broken
cp-from-relative-path approach (the script runs inside the VM where the
host-side guest-setup/ directory does not exist).
template/guest-setup/linux/ci-report-ip.sh
Updated version: checks guestinfo.ip-assignment first (static path),
falls back to DHCP detection if absent. Applies static IP before
network-online.target so SSH starts on the correct address immediately.
template/guest-setup/linux/ci-report-ip.service
Drops 'network.target' from After= — runs before network is up so the
static IP is configured before SSH/other services start.
template/guest-setup/windows/ci-static-ip.ps1
Reads guestinfo.ip-assignment via vmware-rpctool at system startup (SYSTEM
account Task Scheduler task). Disables DHCP, applies static IP+gateway,
reports back via guestinfo.ci-ip. No-op when ip-assignment is absent.
template/Install-CIToolchain-Linux2404.sh
Step 7b now copies scripts from guest-setup/linux/ instead of embedding
the old DHCP-only inline script.
Eliminates VMware-Tools IP-detect polling (observed: 39–177 s variance on
Linux host B7 benchmark). When [ip_pool] is configured the orchestrator:
1. Allocates a slot IP from a JSON-backed pool (fcntl-locked on Linux)
2. Injects guestinfo.ip-assignment / netmask / gateway into the cloned VMX
before vmrun start
3. Skips _wait_for_ip entirely — uses the pre-assigned IP immediately
4. Releases the slot in _destroy_clone (all paths: success/failure/interrupt)
New modules / changes:
src/ci_orchestrator/ip_pool.py IpSlotPool + _file_lock context manager
src/ci_orchestrator/config.py IpPoolConfig dataclass; [ip_pool] TOML
src/ci_orchestrator/commands/job.py _inject_guestinfo_ip helper; pool wiring
tests/python/test_ip_pool.py 12 unit tests (acquire/release/persist)
tests/python/test_config.py ip_pool config parsing tests
tests/python/test_commands_job.py pool path + _inject_guestinfo_ip tests
config.example.toml [ip_pool] section (commented example)
Coverage 90.02%; mypy --strict clean.
4-iteration run on Linux Mint host, WinBuild2025/BaseClean.
Clone/Start/Ready/Destroy within ±20% of Windows baseline.
IP-acquire avg 99.7 s vs 58.2 s Windows — high variance (39–177 s),
attributed to VMware Tools reporting latency, not orchestrator regression.
Measure-CIBenchmark.ps1:
- Default params now Linux-aware via $IsWindows conditional
(TemplatePath, CloneBaseDir, VmrunPath, OutputDir)
- Add -GuestOS Auto/Windows/Linux param; Auto reads VMX guestOS field
(ubuntu-* → Linux, else Windows)
- Replace Test-NetConnection (Windows-only) with Test-TcpPort helper
using raw TcpClient async connect — works on PS5.1 and PS7/Linux
- Probe SSH/22 for Linux guests, WinRM/5986 for Windows guests
- Rename winrm→ready throughout ($t, JSONL field readySec, column header)
- Add guestOS field to JSONL output
_Common.psm1 / Invoke-VmrunBounded:
- Replace taskkill /F /T with platform-aware kill:
Windows → taskkill; Linux → Process.Kill(entireProcessTree) (.NET 5+)
nsis-plugin-ns7zip is a real C++ project — wrong for lifecycle testing.
burnin-dummy simulates ~30s work and writes to dist/ with no external
deps. Documents one-time prerequisites (group, keyring, Gitea push).
Without an explicit build command, job.py runs dotnet build (which fails
on nsis-plugin-ns7zip in a fresh VM) and expects bin\CIOutput as artifact
source. With a trivial BuildCommand, artifact_source defaults to "dist"
and the lightweight command always succeeds.
Also documents one-time simone group/keyring prerequisites for running
the burn-in as simone with ci-runner group access.
All shims defaulted to F:\CI\python\venv\Scripts\python.exe when
CI_VENV_PYTHON was not set. On Linux host this caused instant failure.
Now uses $IsWindows (PS 6+ automatic var; absent on PS 5.1 = Windows)
to select /opt/ci/venv/bin/python on Linux. Windows paths unchanged.
Affects: Invoke-CIJob, Get-BuildArtifacts, Cleanup-OrphanedBuildVMs,
New-BuildVM, Remove-BuildVM, Wait-VMReady, Get-CIJobSummary,
Invoke-RemoteBuild, Watch-DiskSpace, Watch-RunnerHealth,
Set-CIGuestCredential, Test-CIGuestWinRM.
Passing $psExe as a second -ArgumentList item caused double-wrapping of
the string array, collapsing all args into one string when splatted.
Computing $IsWindows inside the worker avoids the extra param entirely.
powershell.exe does not exist on Linux — child process launch failed
instantly (ExitCode null, Elapsed 00:00). Now selects powershell.exe
on Windows and pwsh on Linux via $IsWindows automatic variable.
Data from Measure-CIBenchmark.ps1 × 4 (2026-05-17, commit 5b46bba):
avg boot 60.6s, IP-acquire dominant (20–85s, σ≈26s). Adds ±20% ranges
for B7 comparison and guidance on sample size given IP-phase variance.
Burn-in (4 jobs × 10 rounds), 7-day stability run, and RUNBOOK.md
Linux section are still open. Checklist stays active until all
acceptance criteria are met.
Moves all A/B closeouts, checklists, idea docs, and implementation plans
to plans/archived/2026-05-23/. Both phases are production-stable.
Active plans remaining in plans/:
- idea-3-powershell-removal.md (Phase C — in progress)
- idea-3-esxi-support.md (Phase D — future)
- ideas-overview.md (roadmap reference)
Relocates the graphify BFS/path query script from scripts/ (a PowerShell
folder) into a new tools/ directory for Python developer utilities.
The script is not part of the CI pipeline — it's a local query tool for
navigating graphify-out/graph.json during codebase investigation.
ns7zip matrix Win+Linux passes after the WinRM quotas + desktop-heap
fixes landed in the template scripts. Monitoring window completed
without regressions.
Parallel MSBuild fan-out (e.g. ns7zip x86/x64/x86-ansi) over pypsrp
Linux→Win fails with MSBuild MSB6003 / Win32 1816 'Not enough quota'
because two default guest limits are too tight for concurrent native
processes via WinRM:
- Plugin Microsoft.PowerShell MaxMemoryPerShellMB = 150 MB
(pypsrp uses the plugin endpoint, not the Shell endpoint that
Deploy was previously bumping)
- Non-interactive desktop heap SharedSection 3rd field = 768 KB
(Session 0 csrss; CreateProcess fails when 3+ cl.exe spawn)
Pre-migration Win→Win went through Invoke-Command on PSSession with
quotas inherited from the local process, so the limits never bit.
Changes:
- Deploy-WinBuild2025.ps1 / Deploy-WinBuild2022.ps1: also set the
plugin-level MaxMemoryPerShellMB=2048, bump MaxProcessesPerShell
25→100, patch SharedSection 3rd field 768→4096 (reboot needed for
csrss to pick up, applied by post-install shutdown).
- Install-CIToolchain-WinBuild2025/2022.ps1: add Assert-Step for
plugin quota, MaxProcessesPerShell≥100, SharedSection≥4096; fail
hard if any are under threshold.
- docs/WINDOWS-TEMPLATE-SETUP.md: new "WinRM quotas e desktop heap"
section explaining the diagnosis + one-shot patch for legacy
templates; updated Step 3 validation row + troubleshooting entry.
- AGENTS.md: error #13 with symptom, root cause, and pointer to docs.
If ci-backup-template.service is killed mid-run (SIGTERM), the Python
finally block that restarts act-runner.service is skipped. A subsequent
backup then finds the runner already inactive, _stop_runner returns
False, and the finally restart is gated off — leaving the runner down
indefinitely (incident: 21 mag 2026, runner stayed off ~80 min).
Add ExecStopPost=-/bin/systemctl start act-runner.service so systemd
guarantees the runner is restarted regardless of how the unit exits
(success, failure, signal). The `-` prefix preserves the unit's own
exit status.
- Add plans/idea-3-powershell-removal.md: Phase C goal is to eliminate
the pwsh dependency from the Linux host by porting the remaining PS
scripts (bench run, validate host, smoke run) to Python sub-commands.
- Update idea-3-esxi-support.md header: Fase C → Fase D.
- Update ideas-overview.md: new four-phase table (A/B/C/D) with
prerequisiti and criteri for Phase C.
- Rename all "Hook futuri Fase C" (ESXi) → "Hook futuri Fase D" in
implementation-plan-A-B.md; add Phase C/D entries to summary and
references section.
- Update PhaseB-user-checklist.md end note and CLAUDE.md to point
Phase C → pwsh removal, Phase D → ESXi.
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Update --cov-fail-under in lint workflow + docs (AGENTS, README, ARCHITECTURE). Note: A5 closeout reported 80.10%% so this run will likely fail until additional tests are added.
Configure system Python path once in runner/config.yaml (runner.envs) instead of probing N candidate locations in the workflow. Workflow now fails fast with a clear error if the env is unset or points to a missing file. Document the new env in AGENTS.md.
Reorder candidates to put C:\Program Files\Python314 first (verified install path on the production host). Drop py.exe -3.11 hint to plain -3 since host has only 3.14.
act_runner SYSTEM context may not have py.exe or python.exe in PATH. Try a list of common install paths (C:\Python3XX, Program Files, LocalAppData) and allow override via CI_PYTHON_LAUNCHER env var.
Native stderr from a broken python.exe shim ('No Python at ...') becomes a terminating exception under \Continue='Stop' BEFORE \0 can be read. Wrap the probe in try/catch + SilentlyContinue.
The venv at F:\CI\python\venv may exist but be broken if its pyvenv.cfg points to a deleted base interpreter (e.g. a stale actions/setup-python tool_cache). Detect this by running 'python --version' and rebuild the venv with py.exe -3.11 if it fails.
act_runner runs as SYSTEM and does not have plain 'python' in PATH; use py.exe launcher (-3.11) to bootstrap the venv when missing. The production venv at F:\CI\python\venv normally already exists, so the bootstrap is just a fallback. Also align coverage threshold to 80% (raised in A5, see AGENTS.md).
Update active docs (PhaseA/B checklists, README) after rename of feature/python-rewrite-phase-a -> feature/python-rewrite-and-linux-migration. Historical A1-A5 closeouts left untouched as they document state at the time of writing.
AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
12 dedicated tests covering: (#9) clone_linked surfaces 'should not be powered on' as typed BackendError; (#10) is_running uses 'vmrun list' only and never falls back to getGuestIPAddress (incl. on vmrun list failure); (#11) two consecutive 'vm new' invocations with identical inputs MUST produce distinct clone names/destinations (timestamp suffix), guarding the orchestrator-side safety net against the machine-id duplicate-IP failure mode; (#12) SshTransport defaults to paramiko AutoAddPolicy with no known_hosts, never shells out to ssh-keygen/ssh.exe/scp.exe, wraps paramiko import/exec/SFTP errors as TransportConnectError. Coverage: 80.10% (gate 80%).
The composite action now invokes & $venvPython -m ci_orchestrator job ... directly instead of splatting a hashtable into Invoke-CIJob.ps1, removing the only mismatch between the new shim's positional args contract and the action's previous PascalCase splatting. runner/config.yaml exports PYTHONIOENCODING=utf-8 so non-ASCII build output survives the act_runner -> Gitea round-trip.
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
A1 items were completed but only A1-closeout.md was updated. Mark all A1 checkboxes done in implementation-plan-A-B.md (master checklist + step-A1 attivita + definizione di fatto), leaving only the real-VM PoC pending (hardware validation deferred).
The runner's act_runner runs as SYSTEM, which only sees the Machine PATH.
Host has Python 3.14 installed at C:\Program Files\Python314\; once that
path is added to the machine PATH (one-time admin op) the simple
Get-Command python lookup works without network downloads.
The act_runner service account doesn't have python in PATH. Delegating
to actions/setup-python@v5 which downloads/caches Python 3.11 reliably.
Also bump timeout-minutes 15 -> 20 to absorb first-run pip install.
Gitea Actions only scans .gitea/workflows/ and .github/workflows/. The
file under gitea/workflows/ (no leading dot) was never picked up. Other
files in gitea/workflows/ remain as templates copied into consumer repos.
- Create a new document `plans/implementation-plan-A-B.md` that consolidates the implementation plans for the Python rewrite (Phase A) and the migration to Linux Mint (Phase B).
- The document includes detailed sections such as executive summary, checklist, prerequisites, step-by-step activities, architectural hooks for future phases, risk matrix, and references.
- Ensure the plan is self-contained, adhering to specified structure and style guidelines, and does not introduce new technologies or modify existing files.
Simone