- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force
against a powered-on VM, the three distinct causes behind the identical
"VMware Tools are not running" message, and the orphan snapshot deltas
--force leaves behind after deleting the .vmsd.
- docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check
first, with a Quick Reference entry.
- docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login
shell constraints on /opt/rust, --then-prepare, and how to route a label to a
template (a second runner instance, with the verification step).
- TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it
warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by
hand; the automated policy is still blind to what actually weighs.
- TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the
trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked
against the binary), so the two-runner design is not a stopgap.
The shim mapping table now notes the .ps1 live in scripts/archived/
(frozen, lint-exempt, no live caller) and gains the Phase C host tools
(bench/smoke/validate host/creds set). Fix the "stays in PowerShell"
list: drop Test-CapacityBurnIn (archived) and the non-existent
Measure-CIBenchmark; keep retention/backup/template/e2e/Windows-side.
Parallel MSBuild fan-out (e.g. ns7zip x86/x64/x86-ansi) over pypsrp
Linux→Win fails with MSBuild MSB6003 / Win32 1816 'Not enough quota'
because two default guest limits are too tight for concurrent native
processes via WinRM:
- Plugin Microsoft.PowerShell MaxMemoryPerShellMB = 150 MB
(pypsrp uses the plugin endpoint, not the Shell endpoint that
Deploy was previously bumping)
- Non-interactive desktop heap SharedSection 3rd field = 768 KB
(Session 0 csrss; CreateProcess fails when 3+ cl.exe spawn)
Pre-migration Win→Win went through Invoke-Command on PSSession with
quotas inherited from the local process, so the limits never bit.
Changes:
- Deploy-WinBuild2025.ps1 / Deploy-WinBuild2022.ps1: also set the
plugin-level MaxMemoryPerShellMB=2048, bump MaxProcessesPerShell
25→100, patch SharedSection 3rd field 768→4096 (reboot needed for
csrss to pick up, applied by post-install shutdown).
- Install-CIToolchain-WinBuild2025/2022.ps1: add Assert-Step for
plugin quota, MaxProcessesPerShell≥100, SharedSection≥4096; fail
hard if any are under threshold.
- docs/WINDOWS-TEMPLATE-SETUP.md: new "WinRM quotas e desktop heap"
section explaining the diagnosis + one-shot patch for legacy
templates; updated Step 3 validation row + troubleshooting entry.
- AGENTS.md: error #13 with symptom, root cause, and pointer to docs.
Reconcile docs with the end-to-end pipeline fixes:
- upload/download-artifact @v4 -> @v3 (Gitea GHES) in WORKFLOW-AUTHORING
and workflow-example.yml; add Common Mistakes rows (v4, action ref
form + DEFAULT_ACTIONS_URL, public action repo).
- BEST-PRACTICES / README / HOST-SETUP: guest credential must live in
the LocalSystem vault with a host-qualified username; document
Set-CIGuestCredential.ps1 / Test-CIGuestWinRM.ps1 and auth=ntlm.
- README / AGENTS / HOST-SETUP: production venv install is NON-editable
(LocalSystem); no CI workflow may install into it.
- HOST-SETUP: add DEFAULT_ACTIONS_URL=github + full-URL uses: + public
repo + @main requirements discovered during validation.
- Correct stale repo name local-ci-system -> local-ci-cd-system.
Update --cov-fail-under in lint workflow + docs (AGENTS, README, ARCHITECTURE). Note: A5 closeout reported 80.10%% so this run will likely fail until additional tests are added.
Configure system Python path once in runner/config.yaml (runner.envs) instead of probing N candidate locations in the workflow. Workflow now fails fast with a clear error if the env is unset or points to a missing file. Document the new env in AGENTS.md.
AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
AGENTS.md (new):
- Environment reference for AI agents: host specs, PS 5.1 constraints,
dir structure, VM templates, transport layer, style rules
- Lessons #9 (snapshot cold state) and #10 (vmrun list vs getGuestIPAddress)
ARCHITECTURE.md:
- Add SSH/SCP transport section (Linux VMs)
- Update system diagram to include _Transport.psm1, SSH transport path
- Network layout: add SSH port 22 for Linux VMs
- Timeline: Linux readiness check phases documented
CI-FLOW.md:
- Prerequisites table: add Template VM (Linux), SSH key row
- Step 6 (readiness): document Windows vs Linux phases
- Step 7 (build) and Step 8 (artifacts): document Linux/SSH paths
- ci-report-ip guestinfo mechanism referenced
HOST-SETUP.md:
- Directory tree: add LinuxBuild2404 template dir and keys/
- Post-setup checklist: add step 5 (SSH key generation + Linux template)
- Fix duplicate step numbering (56, 67)
- Reference Paths table: add Linux template VMX, snapshot, SSH key
LINUX-TEMPLATE-SETUP.md:
- Rewritten as implementation record (no longer a plan)
- Fase A user-data: minimal (user+SSH key only, no packages/runcmd)
- All phases marked complete with actual paths and snapshot names
WINDOWS-TEMPLATE-SETUP.md: minor updates for consistency
TODO.md:
- Summary: update last-updated to 2026-05-11, Sprint 9 description
- Linux Build VM row: all 5 items complete
- Prossimi passi: Test Plan + §6.6 Tier-2 as next priorities
Simone
Simone