Measure-CIBenchmark.ps1:
- Default params now Linux-aware via $IsWindows conditional
(TemplatePath, CloneBaseDir, VmrunPath, OutputDir)
- Add -GuestOS Auto/Windows/Linux param; Auto reads VMX guestOS field
(ubuntu-* → Linux, else Windows)
- Replace Test-NetConnection (Windows-only) with Test-TcpPort helper
using raw TcpClient async connect — works on PS5.1 and PS7/Linux
- Probe SSH/22 for Linux guests, WinRM/5986 for Windows guests
- Rename winrm→ready throughout ($t, JSONL field readySec, column header)
- Add guestOS field to JSONL output
_Common.psm1 / Invoke-VmrunBounded:
- Replace taskkill /F /T with platform-aware kill:
Windows → taskkill; Linux → Process.Kill(entireProcessTree) (.NET 5+)
nsis-plugin-ns7zip is a real C++ project — wrong for lifecycle testing.
burnin-dummy simulates ~30s work and writes to dist/ with no external
deps. Documents one-time prerequisites (group, keyring, Gitea push).
Without an explicit build command, job.py runs dotnet build (which fails
on nsis-plugin-ns7zip in a fresh VM) and expects bin\CIOutput as artifact
source. With a trivial BuildCommand, artifact_source defaults to "dist"
and the lightweight command always succeeds.
Also documents one-time simone group/keyring prerequisites for running
the burn-in as simone with ci-runner group access.
All shims defaulted to F:\CI\python\venv\Scripts\python.exe when
CI_VENV_PYTHON was not set. On Linux host this caused instant failure.
Now uses $IsWindows (PS 6+ automatic var; absent on PS 5.1 = Windows)
to select /opt/ci/venv/bin/python on Linux. Windows paths unchanged.
Affects: Invoke-CIJob, Get-BuildArtifacts, Cleanup-OrphanedBuildVMs,
New-BuildVM, Remove-BuildVM, Wait-VMReady, Get-CIJobSummary,
Invoke-RemoteBuild, Watch-DiskSpace, Watch-RunnerHealth,
Set-CIGuestCredential, Test-CIGuestWinRM.
Passing $psExe as a second -ArgumentList item caused double-wrapping of
the string array, collapsing all args into one string when splatted.
Computing $IsWindows inside the worker avoids the extra param entirely.
powershell.exe does not exist on Linux — child process launch failed
instantly (ExitCode null, Elapsed 00:00). Now selects powershell.exe
on Windows and pwsh on Linux via $IsWindows automatic variable.
Data from Measure-CIBenchmark.ps1 × 4 (2026-05-17, commit 5b46bba):
avg boot 60.6s, IP-acquire dominant (20–85s, σ≈26s). Adds ±20% ranges
for B7 comparison and guidance on sample size given IP-phase variance.
Burn-in (4 jobs × 10 rounds), 7-day stability run, and RUNBOOK.md
Linux section are still open. Checklist stays active until all
acceptance criteria are met.
Moves all A/B closeouts, checklists, idea docs, and implementation plans
to plans/archived/2026-05-23/. Both phases are production-stable.
Active plans remaining in plans/:
- idea-3-powershell-removal.md (Phase C — in progress)
- idea-3-esxi-support.md (Phase D — future)
- ideas-overview.md (roadmap reference)
Relocates the graphify BFS/path query script from scripts/ (a PowerShell
folder) into a new tools/ directory for Python developer utilities.
The script is not part of the CI pipeline — it's a local query tool for
navigating graphify-out/graph.json during codebase investigation.
ns7zip matrix Win+Linux passes after the WinRM quotas + desktop-heap
fixes landed in the template scripts. Monitoring window completed
without regressions.
Parallel MSBuild fan-out (e.g. ns7zip x86/x64/x86-ansi) over pypsrp
Linux→Win fails with MSBuild MSB6003 / Win32 1816 'Not enough quota'
because two default guest limits are too tight for concurrent native
processes via WinRM:
- Plugin Microsoft.PowerShell MaxMemoryPerShellMB = 150 MB
(pypsrp uses the plugin endpoint, not the Shell endpoint that
Deploy was previously bumping)
- Non-interactive desktop heap SharedSection 3rd field = 768 KB
(Session 0 csrss; CreateProcess fails when 3+ cl.exe spawn)
Pre-migration Win→Win went through Invoke-Command on PSSession with
quotas inherited from the local process, so the limits never bit.
Changes:
- Deploy-WinBuild2025.ps1 / Deploy-WinBuild2022.ps1: also set the
plugin-level MaxMemoryPerShellMB=2048, bump MaxProcessesPerShell
25→100, patch SharedSection 3rd field 768→4096 (reboot needed for
csrss to pick up, applied by post-install shutdown).
- Install-CIToolchain-WinBuild2025/2022.ps1: add Assert-Step for
plugin quota, MaxProcessesPerShell≥100, SharedSection≥4096; fail
hard if any are under threshold.
- docs/WINDOWS-TEMPLATE-SETUP.md: new "WinRM quotas e desktop heap"
section explaining the diagnosis + one-shot patch for legacy
templates; updated Step 3 validation row + troubleshooting entry.
- AGENTS.md: error #13 with symptom, root cause, and pointer to docs.
If ci-backup-template.service is killed mid-run (SIGTERM), the Python
finally block that restarts act-runner.service is skipped. A subsequent
backup then finds the runner already inactive, _stop_runner returns
False, and the finally restart is gated off — leaving the runner down
indefinitely (incident: 21 mag 2026, runner stayed off ~80 min).
Add ExecStopPost=-/bin/systemctl start act-runner.service so systemd
guarantees the runner is restarted regardless of how the unit exits
(success, failure, signal). The `-` prefix preserves the unit's own
exit status.
- Add plans/idea-3-powershell-removal.md: Phase C goal is to eliminate
the pwsh dependency from the Linux host by porting the remaining PS
scripts (bench run, validate host, smoke run) to Python sub-commands.
- Update idea-3-esxi-support.md header: Fase C → Fase D.
- Update ideas-overview.md: new four-phase table (A/B/C/D) with
prerequisiti and criteri for Phase C.
- Rename all "Hook futuri Fase C" (ESXi) → "Hook futuri Fase D" in
implementation-plan-A-B.md; add Phase C/D entries to summary and
references section.
- Update PhaseB-user-checklist.md end note and CLAUDE.md to point
Phase C → pwsh removal, Phase D → ESXi.
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Update --cov-fail-under in lint workflow + docs (AGENTS, README, ARCHITECTURE). Note: A5 closeout reported 80.10%% so this run will likely fail until additional tests are added.
Configure system Python path once in runner/config.yaml (runner.envs) instead of probing N candidate locations in the workflow. Workflow now fails fast with a clear error if the env is unset or points to a missing file. Document the new env in AGENTS.md.
Reorder candidates to put C:\Program Files\Python314 first (verified install path on the production host). Drop py.exe -3.11 hint to plain -3 since host has only 3.14.
act_runner SYSTEM context may not have py.exe or python.exe in PATH. Try a list of common install paths (C:\Python3XX, Program Files, LocalAppData) and allow override via CI_PYTHON_LAUNCHER env var.
Native stderr from a broken python.exe shim ('No Python at ...') becomes a terminating exception under \Continue='Stop' BEFORE \0 can be read. Wrap the probe in try/catch + SilentlyContinue.
The venv at F:\CI\python\venv may exist but be broken if its pyvenv.cfg points to a deleted base interpreter (e.g. a stale actions/setup-python tool_cache). Detect this by running 'python --version' and rebuild the venv with py.exe -3.11 if it fails.
act_runner runs as SYSTEM and does not have plain 'python' in PATH; use py.exe launcher (-3.11) to bootstrap the venv when missing. The production venv at F:\CI\python\venv normally already exists, so the bootstrap is just a fallback. Also align coverage threshold to 80% (raised in A5, see AGENTS.md).
Update active docs (PhaseA/B checklists, README) after rename of feature/python-rewrite-phase-a -> feature/python-rewrite-and-linux-migration. Historical A1-A5 closeouts left untouched as they document state at the time of writing.
AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
12 dedicated tests covering: (#9) clone_linked surfaces 'should not be powered on' as typed BackendError; (#10) is_running uses 'vmrun list' only and never falls back to getGuestIPAddress (incl. on vmrun list failure); (#11) two consecutive 'vm new' invocations with identical inputs MUST produce distinct clone names/destinations (timestamp suffix), guarding the orchestrator-side safety net against the machine-id duplicate-IP failure mode; (#12) SshTransport defaults to paramiko AutoAddPolicy with no known_hosts, never shells out to ssh-keygen/ssh.exe/scp.exe, wraps paramiko import/exec/SFTP errors as TransportConnectError. Coverage: 80.10% (gate 80%).
The composite action now invokes & $venvPython -m ci_orchestrator job ... directly instead of splatting a hashtable into Invoke-CIJob.ps1, removing the only mismatch between the new shim's positional args contract and the action's previous PascalCase splatting. runner/config.yaml exports PYTHONIOENCODING=utf-8 so non-ASCII build output survives the act_runner -> Gitea round-trip.
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
A1 items were completed but only A1-closeout.md was updated. Mark all A1 checkboxes done in implementation-plan-A-B.md (master checklist + step-A1 attivita + definizione di fatto), leaving only the real-VM PoC pending (hardware validation deferred).
The runner's act_runner runs as SYSTEM, which only sees the Machine PATH.
Host has Python 3.14 installed at C:\Program Files\Python314\; once that
path is added to the machine PATH (one-time admin op) the simple
Get-Command python lookup works without network downloads.
The act_runner service account doesn't have python in PATH. Delegating
to actions/setup-python@v5 which downloads/caches Python 3.11 reliably.
Also bump timeout-minutes 15 -> 20 to absorb first-run pip install.
Gitea Actions only scans .gitea/workflows/ and .github/workflows/. The
file under gitea/workflows/ (no leading dot) was never picked up. Other
files in gitea/workflows/ remain as templates copied into consumer repos.
- Create a new document `plans/implementation-plan-A-B.md` that consolidates the implementation plans for the Python rewrite (Phase A) and the migration to Linux Mint (Phase B).
- The document includes detailed sections such as executive summary, checklist, prerequisites, step-by-step activities, architectural hooks for future phases, risk matrix, and references.
- Ensure the plan is self-contained, adhering to specified structure and style guidelines, and does not introduce new technologies or modify existing files.
Simone