The chown ran before rustup and cargo-audit were installed, so everything
those created under RUSTUP_HOME/CARGO_HOME was left root-owned: only the two
empty directories existing at chown time belonged to ci_build. Measured on the
previous snapshot: 18350 root-owned paths, including cargo/registry. Every Rust
build with external dependencies failed with "Permission denied (os error 13)".
Re-assert ownership after the installs, and assert writability rather than mere
existence -- test -x on the binaries could not catch this, so a broken template
passed its own build and failed in consumers' jobs instead.
Also wire the toolchain for non-login shells. The orchestrator runs build
commands via paramiko exec_command + `sh -c`, which never sources
/etc/profile.d, so profile.d alone was not enough: add RUSTUP_HOME/CARGO_HOME
to /etc/environment (read by pam_env at SSH session open) and symlink the
rustup proxies into /usr/local/bin.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
Setting DelegationConsole/DelegationTerminal to the Windows Terminal GUID made
WT the console host for the cmd.exe shell. With Shell=cmd.exe and
AutoRestartShell=1, closing the WT-hosted shell window breaks the cmd<->WT host
handshake: Winlogon's relaunch leaves a bare grey desktop with no shell instead
of a fresh cmd. Classic conhost restarts cleanly and is ~82 MB lighter (no
WindowsTerminal process in the idle baseline).
Comment out the four delegation reg adds (HKCU + Default hive) so the shell
falls back to conhost. Kept commented (not deleted) to document the rationale.
schtasks.exe /Change on a non-existent task writes to stderr and exits 1.
Inside the disk-cleanup function (EAP=Stop, wrapped by the exit-code-sensitive
prepare-win cleanup invoker), that leaked $LASTEXITCODE=1 all the way to the
result marker — DISM runs via Start-Process and does not reset it — so the
step failed with "Disk cleanup failed (exit 1)" even though everything worked.
On a freshly-deployed WS2025 the UpdateOrchestrator "Schedule Scan" task does
not exist, so this fired every run.
Replace the schtasks loop with Get-ScheduledTask | Disable-ScheduledTask,
which no-ops silently on a missing task and sets no process exit code.
Follow-up to fb5de6f; applied to both toolchains.
The disk-cleanup function stopped wuauserv before DISM /StartComponentCleanup
/ResetBase but only re-enforced Start=4 AFTER it. /ResetBase drives heavy CBS
servicing (TiWorker); during that window WaaSMedicSvc woke, flipped wuauserv
Start back to Manual, and a "Schedule Scan" task started it. If the pass was
interrupted there (reboot/crash), the template was left with WU running and
Manual startup — a half-prepared state where Windows Update keeps firing on a
supposedly-locked-down build template.
Move the lockdown ahead of DISM: set wuauserv Start=4, apply the
WaaSMedicSvc registry write-deny ACL (the only reliable lever — the service
is protected and cannot itself be disabled), and disable the WindowsUpdate /
UpdateOrchestrator scan tasks, all BEFORE /ResetBase rather than only after.
The post-DISM re-enforce stays as a belt-and-suspenders check.
Applied identically to the WinBuild2025 and WinBuild2022 toolchains.
An ephemeral headless build VM carries a lot of dead weight. Split the trim
by cost so heavy servicing never runs under the live explorer shell (which
crash-loops during CBS churn at first logon):
post-install.win.ps1 (light, runs under explorer at first logon):
- disable waste services: audio, camera, CDP/sync, push (Wpn), telemetry
(DiagTrack/InventorySvc), search index (WSearch), DPS/WDI, MSDTC, TrkWks,
Themes, MapsBroker, RetailDemo, PcaSvc, WSAIFabricSvc; BITS -> on-demand
- disable per-user service templates (CDPUserSvc/OneSyncSvc/WpnUserService/...)
so no per-session clone spawns
- remove in-box apps: FeedbackHub/GetHelp/Getstarted/OfficeHub (appx +
provisioned), Snip & Sketch, Paint, RDP client (mstsc) — all best-effort
prepare-win (heavy, runs under cmd shell, before the final reboot so the
following DISM cleanup compacts the freed component store):
- Uninstall-WindowsFeature WindowsAdminCenterSetup, XPS-Viewer,
Wireless-Networking, Windows-Defender (-NoRestart)
- Remove-WindowsCapability AzureArcSetup (payload; auto-start already off)
- uninstall Edge + EdgeWebView2 and disable their leftover updater services
- gated by --skip-trim
Server Manager auto-launch was already disabled (task + reg + GPO policy);
its exe is part of the OS and cannot be uninstalled. Defender was already
runtime-off; removing the feature drops the AV stack entirely.
run_streaming used index-based _drain, which read new records from
ps.output / ps.streams.{information,error} but never released the consumed
ones. pypsrp retains every record (each a heavy deserialised PSObject) for
the whole invocation, so a long prepare-win (Windows Update + toolchain,
thousands of lines) grew the host python process RAM without bound.
Replace _drain with _drain_clear, which emits each pending record then
deletes the consumed prefix (`del seq[:n]`). Safe because draining runs on
the same thread as poll_invoke — nothing appends while we mutate. Track an
explicit had_error flag since the error stream is now cleared as we go.
PowerShell 5.1 strips an empty-string argument ('') when invoking a native
exe, so `reg add ... /d '' /f` reached reg.exe as `reg add ... /d /f`: reg
consumed /f as the data value, lost the force flag, and prompted
"overwrite? (Yes/No)" on stdin. FirstLogon runs headless with no stdin, so
post-install hung forever right after the "Explorer UX tweaks applied" log
line. Route both empty-value reg adds (HKCU + Default hive) through
`cmd /c`, which preserves the empty "" so the value is set and /f stays.
Regression from 8095f00 (wallpaper removal feature).
Prepend a guard that runs `autoload -Uz compinit && compinit` when compdef
isn't defined, so the generated zsh completion works without oh-my-zsh or if
sourced before compinit.
Generate a click completion file for bash/zsh and source it from the alias
block. Click's generated script queries completions by invoking the command
— but `ci` is a sudo shell function (env vars don't survive sudo, and `env
VAR=x ci` can't call a function), so rewrite the query to call the venv
python directly (read-only, no sudo) and drop zsh's `$+commands[ci]` guard.
Writes ~/.ci-completion.<shell>; removed cleanly on --uninstall (block strip).
Invoking `python -m ci_orchestrator` now reports prog_name "ci" — matching
the shell alias — so `--help` reads naturally and click shell completion
keys off the _CI_COMPLETE env var.
The base ISO's volume label (e.g. "26100.1_X64_EN-US") contains characters
outside strict ISO 9660 d-characters, so xorriso warned "-volid text does
not comply to ISO 9660 / ECMA 119 rules" on every repack. Map the label to
[A-Z0-9_] (cap 16). Cosmetic only — Windows boots from the El Torito images,
not the volid — but keeps the output clean.
Remove only the wallpaper image (Wallpaper="" + WallpaperStyle=0) and leave
Control Panel\Colors\Background at the WS2025 default, so the desktop shows
the default grey instead of black.
A headless CI build VM needs no wallpaper. post-install now clears the
Wallpaper value + sets a black background for the autologin Administrator
(HKCU) and the Default user hive (so any future profile inherits it).
autounattend FirstLogonCommands — which run post-install.ps1 — are triggered
by explorer starting as the shell. Setting Winlogon\Shell=cmd.exe in the
specialize pass stopped explorer, so post-install never ran on a fresh
deploy.
Remove the shell switch from specialize; explorer runs once for the first
logon (deploy-win does no heavy servicing, so no crash-loop risk) and
post-install.ps1 sets Shell=cmd.exe at its end — subsequent boots and all of
prepare-win run under cmd. Keep the (shell-independent) Server Manager
auto-launch disable in specialize.
Add specialize-pass RunSynchronousCommands to disable the ServerManager
scheduled task (the actual auto-launcher on WS2025) plus the HKLM/policy
DoNotOpen* keys, so Server Manager never pops up — not even at the first
logon, before post-install runs. Redundant with post-install.
Add a Microsoft-Windows-Deployment RunSynchronousCommand in the specialize
pass that sets Winlogon\Shell=cmd.exe before the first logon. A freshly
deployed template then never starts explorer — not even the transient
desktop during OOBE/install — closing the last window where the shell could
crash-loop during servicing. post-install.ps1 and prepare-win still
re-affirm it for redundancy / existing templates.
Up front, read Winlogon\Shell; if it isn't already cmd.exe, set it and do a
clean reboot before running the toolchain — so the shell is genuinely cmd
(not a killed-explorer half-state) for the whole servicing run. Idempotent:
when the shell is already cmd (re-runs), skip the reboot. Replaces the
earlier kill-explorer approach.
Set Winlogon\Shell=cmd.exe and drop the live explorer at the very start of
prepare-win, before the toolchain runs, so every reboot in the Windows
Update 3010 loop already boots without explorer — eliminating the shell
crash-loop during heavy servicing, not just at the final reboot. Still
re-affirmed before the final reboot and baked into post-install.
Setting Winlogon\Shell=cmd.exe only in the install-time post-install meant
existing templates (deployed earlier, or reverted to a pre-cmd snapshot)
kept explorer as the shell — re-running prepare-win never changed it, since
prepare runs the toolchain, not post-install. Set Shell=cmd.exe in the
prepare-win autologin re-affirm (before the final reboot) so any template
adopts it on the next prepare.
A headless CI build VM never needs the interactive desktop (builds run over
WinRM in Session 0), and a running explorer is fragile: heavy servicing
(Windows Update / DISM component cleanup) destabilises the live shell and
explorer crash-loops until the next reboot, spiking CPU and tripping the
pre-snapshot idle check.
Set Winlogon\Shell = cmd.exe in post-install so explorer never auto-starts
— removing that whole failure class at the root. Autologin is unchanged;
explorer remains launchable on demand (run explorer.exe, or via RDP).
Verified on the live WinBuildTest VM: post-reboot no explorer, 0 shell-stop
events, ~2% idle CPU.
Heavy servicing (DISM StartComponentCleanup in the -CleanupOnly step, like
WU finalization) destabilises the running shell session — explorer
crash-loops until the next reboot. Cleanup ran *after* the final reboot, so
the template ended on a crash-looping desktop and the idle check aborted.
Reorder: disk cleanup + autologin re-affirm now run before the final
reboot, making that reboot the last action before the idle check — the
shell comes up against fully-settled components and the snapshot captures a
clean, idle desktop.
The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
Microsoft rotated the unversioned https://dot.net/v1/dotnet-install.ps1, so
the pinned SHA256 was stale and the toolchain aborted ("Possible MITM or
wrong installer version"). Independently re-downloaded the official script
(builds.dotnet.microsoft.com) — its hash matches the guest's observed
value, confirming it is the legitimate current script, not tampering.
Update the pin in both WinBuild2025 and WinBuild2022 toolchains to
6585899AED55FF6AE13DBE1E8C3B878F2D00433520E7EFBE250B75DB948B7DA9 and note
that this pin must be refreshed whenever MS updates the script in place.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-win patched its input into vm_dir/install-noprompt.iso and only its
own sidecar cache could skip the repack — so feeding an already-patched ISO
(e.g. from `template patch-win-iso`, or via --win-iso) triggered a wasteful
second extract+repack that double-injected the answer files.
deploy-win now detects a patched input (autounattend.xml present at the ISO
root via `iso_patch.iso_has_root_file`) and uses it as-is, skipping Step 4.
Raw ISOs are still patched once and cached as before.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
The ISO repack (extract + xorriso, ~minutes) ran on every deploy. Cache it
per VM dir: store an inputs key (base ISO size+mtime + rendered
autounattend.xml + post-install.ps1) in a `.iso.inputs` sidecar next to
install-noprompt.iso. Step 4 skips the repack when the patched ISO and a
matching sidecar already exist, and re-patches automatically when the base
ISO or either answer file changes.
The patched install ISO is mounted as a CD and has setup.exe at its root
(Windows Server Setup). The first-logon CD scan matched it before the
VMware Tools CD and ran `setup.exe /s /v...`, which Windows Setup rejects
("unknown command-line option [/s]"), and copied the multi-GB install
media instead of the Tools payload.
Skip any CD that carries sources\install.wim / install.esd (the OS
installer) and prefer setup64.exe (VMware Tools x64 has it; Windows media
does not) when resolving the Tools setup.
A headless CI build VM has no USB devices — input is the VMware PS/2
keyboard/mouse + SVGA console. Drop the usb/ehci/usb_xhci controllers from
the generated VMX (one explicit `usb.present = "FALSE"`), trimming device
enumeration at boot. Step 9's usb* line stripping becomes a harmless no-op.
xorriso -extract reads the ISO9660 name tree, which on a Windows ISO has
no Rock Ridge — names come back upper-cased/truncated, so the
`efi/microsoft/boot/efisys_noprompt.bin` lookup failed ("missing from
ISO") and a repack from that tree would corrupt Windows' long file names.
Extract with 7z instead (reads the UDF/Joliet tree, preserving the real
lower-case paths), excluding 7z's synthetic `[BOOT]` dump. Verified
end-to-end on the cached 26100.32860 ISO: 7z extract → xorriso repack
yields a 5 GB ISO with autounattend.xml + post-install.ps1 at the root and
efisys_noprompt.bin as the UEFI El Torito image (no boot prompt).
Also: cap the Joliet volume id at 16 chars (silence xorriso warning), add
a 7z preflight check, and add p7zip-full to setup-host-linux.sh.
New `ci template deploy-win` builds the Windows Server 2025 template VM on
the Linux host, mirroring the legacy Deploy-WinBuild2025.ps1 ten-step flow
in Python:
- winimage/uup.py: resolve the newest (or pinned) WS2025 build via the
uupdump API and build a cached, MS-signed ISO via the uup converter.
- winimage/autounattend.py + assets/: render autounattend.xml and the
guest post-install.ps1 (ported from the PS heredoc; VMware Tools now
mounted as a 2nd CD, scanned at first logon).
- winimage/iso_patch.py: xorriso repack injecting the answer files at the
ISO root and swapping the EFI boot image to efisys_noprompt.bin
(no "press any key" prompt — essential for an unattended headless install).
- commands/template.py: the deploy-win command + _winbuild_vmx_lines
(the 82-key VMX), _remove_vmx_lines, _wait_install_flag, _vmrun_soft_stop.
Acquire → patch → VMDK → VMX → install → flag wait → strip media +
NIC e1000e→vmxnet3 → verify IP → cold snapshot.
Deterministic by build (--build), with --win-iso to bypass UUP. Tests
mirror the prepare-win mocking style; total coverage 93%.
Add aria2 + wimtools (wimlib-imagex) + xorriso + genisoimage + cabextract
+ chntpw to setup-host-linux.sh (Step 3) — the UUP dump → ISO conversion
and the no-prompt ISO repack that `ci template deploy-win` needs. Also
create the <CI_ROOT>/iso cache dir in the layout step.
Verified the whole §4 chain end-to-end on the live host and fixed every
break:
- Capture the clone VMX from `vm new` stdout into $VMX and the guest IP
via `vmrun getGuestIPAddress` into $IP, then reuse both downstream.
Removes the fragile `Clone_dbg-1_*` globs that broke when more than one
clone shared a job-id ("unexpected extra argument").
- §4.3: `build run` does not inject the Gitea PAT (only `job` does), so the
private burnin-dummy repo can't be cloned in-guest. Use --host-source-dir
against the local copy instead.
- §4.4: collect from /opt/ci/output (where build run stages the artifact),
not the `dist` source dir — the old path failed with "tar: dist: Cannot
open".
- §4.1 now boots with --gui (manual debugging); §4.7 reuses $VMX.
Confirmed green: vm new → wait-ready → build → collect (build-info.txt +
manifest.json) → vm remove, and the §3 `job` pipeline (SUCCESS in 66s).
Mirror the `job` flag on the manual lifecycle: `vm new --start` was
hardcoded headless; add --gui/--nogui (default --nogui) so a clone can be
booted into the Workstation GUI window for interactive debugging. The
start banner reflects the mode.
Also update the runbook: §4.1 shows the --gui variant, and §3 notes that
the burnin-dummy build.sh is a ~30s fake build writing dist/build-info.txt
so the examples exercise the full pipeline.
Switch the §3 and §4.3 examples to Simone/burnin-dummy (the canonical
burn-in test repo): `bash build.sh` (fake ~30s build) producing a `dist/`
artifact, so the examples exercise a real build + artifact collection end
to end.
The Phase-2 VM start was hardcoded to headless. Expose a --gui/--nogui
toggle (default --nogui, preserving current behaviour); --gui starts the
clone with the Workstation GUI window (needs an X display, see §4.7). The
phase banner now reflects the chosen mode.
Replace the non-existent myrepo.git placeholder (which fails with
"repository not found") with this repo's URL and a harmless
`echo build-ok` build command, so §3 and §4.3 run as pasted.
The `job` command enforces --template-path (job.py:447) and defaults
--snapshot-name to BaseClean, which is wrong for Linux. The §3 example
omitted both and failed with "--template-path is required". Add both
flags (BaseClean-Linux) and note them as required.
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
The shim mapping table now notes the .ps1 live in scripts/archived/
(frozen, lint-exempt, no live caller) and gains the Phase C host tools
(bench/smoke/validate host/creds set). Fix the "stays in PowerShell"
list: drop Test-CapacityBurnIn (archived) and the non-existent
Measure-CIBenchmark; keep retention/backup/template/e2e/Windows-side.
Document `vmrun getGuestIPAddress` (run as ci-runner) for templates and
clones as the prerequisite for `validate guest`, with the AGENTS.md #1
"Error: on stdout" gotcha. Update the validate-guest Linux example to the
live template IP.
Move 16 .ps1 files to scripts/archived/ — they have no live caller:
- 10 pure shims (New-BuildVM, Remove-BuildVM, Cleanup-OrphanedBuildVMs,
Wait-VMReady, Invoke-RemoteBuild, Get-BuildArtifacts, Watch-DiskSpace,
Watch-RunnerHealth, Get-CIJobSummary, Invoke-CIJob) that only forwarded
to `ci_orchestrator`; the live Gitea composite action now calls
`python -m ci_orchestrator job` directly.
- 6 Phase-C host tools reimplemented natively in Python (Start-BurnInTest
[+ -Linux], Test-CapacityBurnIn → bench; Test-Smoke → smoke run;
Validate-HostState → validate host; Set-CIGuestCredential → creds set).
Add scripts/archived/README.md mapping each file to its `ci` replacement,
and exempt scripts/archived/ from the PSScriptAnalyzer lint job.
Kept (still used / not yet ported): Register-CIScheduledTasks,
Backup-CITemplate, Invoke-RetentionPolicy, Set-TemplateSharedFolders,
Test-CIGuestWinRM, the e2e build harnesses, and all template/ guest
provisioning scripts.
Simone