The chown ran before rustup and cargo-audit were installed, so everything
those created under RUSTUP_HOME/CARGO_HOME was left root-owned: only the two
empty directories existing at chown time belonged to ci_build. Measured on the
previous snapshot: 18350 root-owned paths, including cargo/registry. Every Rust
build with external dependencies failed with "Permission denied (os error 13)".
Re-assert ownership after the installs, and assert writability rather than mere
existence -- test -x on the binaries could not catch this, so a broken template
passed its own build and failed in consumers' jobs instead.
Also wire the toolchain for non-login shells. The orchestrator runs build
commands via paramiko exec_command + `sh -c`, which never sources
/etc/profile.d, so profile.d alone was not enough: add RUSTUP_HOME/CARGO_HOME
to /etc/environment (read by pam_env at SSH session open) and symlink the
rustup proxies into /usr/local/bin.
schtasks.exe /Change on a non-existent task writes to stderr and exits 1.
Inside the disk-cleanup function (EAP=Stop, wrapped by the exit-code-sensitive
prepare-win cleanup invoker), that leaked $LASTEXITCODE=1 all the way to the
result marker — DISM runs via Start-Process and does not reset it — so the
step failed with "Disk cleanup failed (exit 1)" even though everything worked.
On a freshly-deployed WS2025 the UpdateOrchestrator "Schedule Scan" task does
not exist, so this fired every run.
Replace the schtasks loop with Get-ScheduledTask | Disable-ScheduledTask,
which no-ops silently on a missing task and sets no process exit code.
Follow-up to fb5de6f; applied to both toolchains.
The disk-cleanup function stopped wuauserv before DISM /StartComponentCleanup
/ResetBase but only re-enforced Start=4 AFTER it. /ResetBase drives heavy CBS
servicing (TiWorker); during that window WaaSMedicSvc woke, flipped wuauserv
Start back to Manual, and a "Schedule Scan" task started it. If the pass was
interrupted there (reboot/crash), the template was left with WU running and
Manual startup — a half-prepared state where Windows Update keeps firing on a
supposedly-locked-down build template.
Move the lockdown ahead of DISM: set wuauserv Start=4, apply the
WaaSMedicSvc registry write-deny ACL (the only reliable lever — the service
is protected and cannot itself be disabled), and disable the WindowsUpdate /
UpdateOrchestrator scan tasks, all BEFORE /ResetBase rather than only after.
The post-DISM re-enforce stays as a belt-and-suspenders check.
Applied identically to the WinBuild2025 and WinBuild2022 toolchains.
Microsoft rotated the unversioned https://dot.net/v1/dotnet-install.ps1, so
the pinned SHA256 was stale and the toolchain aborted ("Possible MITM or
wrong installer version"). Independently re-downloaded the official script
(builds.dotnet.microsoft.com) — its hash matches the guest's observed
value, confirming it is the legitimate current script, not tampering.
Update the pin in both WinBuild2025 and WinBuild2022 toolchains to
6585899AED55FF6AE13DBE1E8C3B878F2D00433520E7EFBE250B75DB948B7DA9 and note
that this pin must be refreshed whenever MS updates the script in place.
Step 4a installs python3-gi gir1.2-gtk-4.0 xvfb for headless PyGObject
GUI tests, with import-gi and Xvfb assertions.
Step 1 now always runs apt-get update; --skip-update only skips the
(slow) full upgrade. Previously --skip-update skipped the index refresh
too, so a fresh package (gir1.2-gtk-4.0/xvfb) could not be located.
- WinRM MaxProcessesPerShell: apply 100 if below threshold before asserting
- DesktopHeap SharedSection: apply 4096 KB registry value if below threshold before asserting
- Replace cleanmgr (hangs in WinRM Session 0, no window station) with direct
PowerShell cleanup for dumps, WER files and Recycle Bin; DISM already covers
component store and WU cache
- vcpkg git clone/checkout: wrap with EAP=Continue to suppress PS5.1
NativeCommandError on git stderr (gotcha #3 from AGENTS.md)
- template.py: filter CI_EXITCODE:N marker from displayed output
- ci-static-ip.ps1: switch from vmware-rpctool (removed in recent Tools) to
vmtoolsd --cmd; use temp-file redirect to capture output from native cmd
Move disk cleanup out of the Windows Update loop:
- PS scripts: extract Invoke-DiskCleanup function, add -CleanupOnly switch
for standalone post-update cleanup pass.
- Python command: always pass -SkipCleanup:$true during WU loop; after
loop exits 0, run a separate -CleanupOnly invocation (skipped if
--skip-cleanup). Fails fast if cleanup-only returns nonzero.
- Tests: 3 new tests for cleanup-runs-after-loop, skip-cleanup suppresses
it, and cleanup-failure propagates as ClickException.
- ruff --fix: remove unused shutil import, sort imports, fix UP037/UP035.
Without Import-Module PSWSMan the native WSMan library isn't registered
and New-PSSession hangs/fails silently. Import guarded by $IsWindows so
Windows PS 5.1 is unaffected.
WSMan TrustedHosts is a Windows WinRM client concept — does not exist
on Linux. Guard the configure+restore blocks with $isLinuxHost so the
Prepare scripts run without elevation errors from a Linux host.
-SkipCACheck/-SkipCNCheck in sessionOptions is sufficient on Linux.
Parameter not available in PS 7 on Linux. -SkipCACheck -SkipCNCheck
sufficient for self-signed lab certs; revocation check is irrelevant
for self-signed certs regardless.
Test-NetConnection is Windows-only. Replace with cross-platform TcpClient
async-connect helper (same pattern as Measure-CIBenchmark.ps1) so the
Prepare scripts run on the Linux host under PowerShell.
Windows (Prepare-WinBuild2025/2022): read ci-static-ip.ps1 from
guest-setup/windows/ on the host, push content over WinRM, register the
CI-StaticIp scheduled task (AtStartup/SYSTEM). Adds StaticIpTask and
StaticIpScript post-setup assertions.
Linux (Install-CIToolchain-Linux2404.sh): embed ci-report-ip.sh and
ci-report-ip.service content inline via heredoc tee, replacing the broken
cp-from-relative-path approach (the script runs inside the VM where the
host-side guest-setup/ directory does not exist).
template/guest-setup/linux/ci-report-ip.sh
Updated version: checks guestinfo.ip-assignment first (static path),
falls back to DHCP detection if absent. Applies static IP before
network-online.target so SSH starts on the correct address immediately.
template/guest-setup/linux/ci-report-ip.service
Drops 'network.target' from After= — runs before network is up so the
static IP is configured before SSH/other services start.
template/guest-setup/windows/ci-static-ip.ps1
Reads guestinfo.ip-assignment via vmware-rpctool at system startup (SYSTEM
account Task Scheduler task). Disables DHCP, applies static IP+gateway,
reports back via guestinfo.ci-ip. No-op when ip-assignment is absent.
template/Install-CIToolchain-Linux2404.sh
Step 7b now copies scripts from guest-setup/linux/ instead of embedding
the old DHCP-only inline script.
Parallel MSBuild fan-out (e.g. ns7zip x86/x64/x86-ansi) over pypsrp
Linux→Win fails with MSBuild MSB6003 / Win32 1816 'Not enough quota'
because two default guest limits are too tight for concurrent native
processes via WinRM:
- Plugin Microsoft.PowerShell MaxMemoryPerShellMB = 150 MB
(pypsrp uses the plugin endpoint, not the Shell endpoint that
Deploy was previously bumping)
- Non-interactive desktop heap SharedSection 3rd field = 768 KB
(Session 0 csrss; CreateProcess fails when 3+ cl.exe spawn)
Pre-migration Win→Win went through Invoke-Command on PSSession with
quotas inherited from the local process, so the limits never bit.
Changes:
- Deploy-WinBuild2025.ps1 / Deploy-WinBuild2022.ps1: also set the
plugin-level MaxMemoryPerShellMB=2048, bump MaxProcessesPerShell
25→100, patch SharedSection 3rd field 768→4096 (reboot needed for
csrss to pick up, applied by post-install shutdown).
- Install-CIToolchain-WinBuild2025/2022.ps1: add Assert-Step for
plugin quota, MaxProcessesPerShell≥100, SharedSection≥4096; fail
hard if any are under threshold.
- docs/WINDOWS-TEMPLATE-SETUP.md: new "WinRM quotas e desktop heap"
section explaining the diagnosis + one-shot patch for legacy
templates; updated Step 3 validation row + troubleshooting entry.
- AGENTS.md: error #13 with symptom, root cause, and pointer to docs.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
Each Ubuntu clone derived from the same snapshot shared an identical
machine-id, causing systemd-networkd to use the same DHCP client-id
and receive the same NAT IP triggering IP collision failures.
Step 7b truncates /etc/machine-id and removes /var/lib/dbus/machine-id
before the snapshot is taken. On first boot each clone generates a
fresh machine-id from /dev/urandom -> unique DHCP client-id -> unique IP.
Setup-WinBuild2025.ps1:
- New [switch] \ param (after \)
- Step 12 block wrapped with if (\) { skip-message } else { ... }
- Final gate Tier-2 Assert-Step wrapped with if (-not \)
Prepare-WinBuild2025.ps1:
- New [switch] \ param forwarded to Setup via setupArgs
Validate-SetupState.ps1:
- New [switch] \ param
- \ passed into both remote scriptblocks via -ArgumentList
- Tier-2 Chk calls guarded with if (-not \) in both blocks
Setup-WinBuild2025.ps1: default SevenZipVersion '24.07' → '26.01'
TODO.md: Tier-1 table + bullet list updated to current versions
TEST-PLAN: expected version strings and checklist updated
Root cause: Assert-Step 'Final' 'dotnet/python/msbuild' block contains a
registry PATH refresh that overwrites $env:PATH, discarding the temporary
session injection added in §6.6. Next assertion (7z command available) then fails.
Fix: use SetEnvironmentVariable('PATH', ..., 'Machine') to permanently register
C:\Program Files\7-Zip in the system PATH — same pattern used for dotnet and MSBuild.
Follow-up registry refresh ensures $env:PATH is also current.
Two root causes:
1. Git skip check used C:\BuildTools\Git but Git installs to
C:\Program Files\Git (default). Changed $gitDir to correct path.
2. 7-Zip MSI does not register in system PATH. Registry refresh
(GetEnvironmentVariable Machine+User) was picking up empty result.
Fixed: inject sevenZipDir directly into $env:PATH if not present.
PATH refresh was inside the 'else' block (only during fresh install).
If tools already exist from previous run, PATH not refreshed → validation fails.
Move PATH refresh outside if/else for both Git and 7-Zip:
- Already installed: skip download+install, but refresh PATH anyway
- Fresh install: download+install, then refresh PATH
Ensures validations can always resolve git/7z commands, even on retry runs.
7-Zip MSI installer adds to system PATH but PowerShell session doesn't
see it until PATH is refreshed. Refresh via:
$env:PATH = [System.Environment]::GetEnvironmentVariable('PATH', 'Machine') + ';' +
[System.Environment]::GetEnvironmentVariable('PATH', 'User')
Same pattern already used for Git installer. Fixes validation error:
'7z command resolvable' now passes after installation.
GitHub release tag: v2.54.0.windows.1
Filename in release: Git-2.54.0-64-bit.exe (no .windows.1)
Extract base version for filename via regex:
$gitVersionBase = $GitVersion -replace '\.windows\.\d+$', ''
URL now correctly formed as:
.../download/v2.54.0.windows.1/Git-2.54.0-64-bit.exe
Reverted GitVersion back to '2.54.0.windows.1' (needed for tag).
Version 2.47.0.windows.1 no longer available on GitHub (network errors).
Updated to latest stable 2.54.0.windows.1 with SHA256 hash pinning (§1.3).
Hash: 2b96e7854f0520f0f6b709c21041d9801b1be44d5e1a0d9fa621b2fbc40f1983
Source: https://github.com/git-for-windows/git/releases
This fixes download failures in Setup-WinBuild2025 Step 11 (Tier-1 Toolchain).
§6.6 Tier-1 Toolchain (Git for Windows, 7-Zip) now included in final
state validation. Checks:
- 'Git for Windows (git.exe in PATH)'
- '7-Zip (7z.exe in PATH)'
Added to both initial validation pass and re-validation pass (post-remediation).
Implement §6.6 minimal viable product: install Git for Windows and 7-Zip
to the template VM. Critical prerequisites for §3.3 (In-VM Git clone).
Step 11 additions:
- Git for Windows (latest LTS configurable) — silent installer, adds to PATH
- 7-Zip (latest stable configurable) — MSI installer to Program Files
- SHA256 hash pinning for both tools (currently unpinned, warn at runtime)
- Assert-Step validation for git/7z command availability
- Final pre-snapshot gate checks for both tools + cleanup of installers
Parameters added:
- -GitVersion (default: 2.47.0.windows.1)
- -SevenZipVersion (default: 24.07)
Rationale:
- Enables §3.3 in-VM Git clone (git clone directly in VM instead of host-zip-transfer)
- Supports cross-platform builds and multi-tool workflows
- Fallback from §3.2 7-Zip compression finally available in template
- Positions for future expansion of Tier-1 tools (NSIS, CMake, Node.js, WiX, etc.)
Updated documentation:
- Setup-WinBuild2025.ps1 docstring: Step 11 added to list + step ordering rationale
- Removed NOTE about Git not being installed
- Final validation now checks: 9 assertions (was 7)
Activates Windows against kms8.msguides.com before Windows Update runs
so WU sees correct license state. Uses cscript //NoLogo to route slmgr
output to stdout (dialogs hang in WinRM sessions). Best-effort: activation
failure emits a warning but does not abort Setup — CI builds function
within the grace period.
Placed between Step 3 (WinRM/network confirmed) and Step 4 (user creation).
Simone
Simone