All shims defaulted to F:\CI\python\venv\Scripts\python.exe when
CI_VENV_PYTHON was not set. On Linux host this caused instant failure.
Now uses $IsWindows (PS 6+ automatic var; absent on PS 5.1 = Windows)
to select /opt/ci/venv/bin/python on Linux. Windows paths unchanged.
Affects: Invoke-CIJob, Get-BuildArtifacts, Cleanup-OrphanedBuildVMs,
New-BuildVM, Remove-BuildVM, Wait-VMReady, Get-CIJobSummary,
Invoke-RemoteBuild, Watch-DiskSpace, Watch-RunnerHealth,
Set-CIGuestCredential, Test-CIGuestWinRM.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Passing $psExe as a second -ArgumentList item caused double-wrapping of
the string array, collapsing all args into one string when splatted.
Computing $IsWindows inside the worker avoids the extra param entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
powershell.exe does not exist on Linux — child process launch failed
instantly (ExitCode null, Elapsed 00:00). Now selects powershell.exe
on Windows and pwsh on Linux via $IsWindows automatic variable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The blind -X -> --x shims forwarded PowerShell common parameters
(-ErrorAction, -Verbose, ...) to the Python CLI, e.g.
Measure-CIBenchmark.ps1 -> Remove-BuildVM.ps1 -ErrorAction
SilentlyContinue produced `ci_orchestrator vm remove --error-action`
-> "No such option" -> destroy failed (orphan VM left). Skip
common switches and consume value-taking ones. Applied to all 7
blind-loop shims; param-style shims (New-BuildVM, Invoke-RemoteBuild,
Get-BuildArtifacts) are not affected. PSSA clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove-Securely: suppress PSUseShouldProcessForStateChangingFunctions
(internal best-effort shred; ShouldProcess prompts are undesirable)
+ add [CmdletBinding()].
- Replace the two empty catch blocks (Set-CIGuestCredential.ps1,
Test-CIGuestWinRM.ps1) with Write-Verbose so PSAvoidUsingEmptyCatchBlock
is satisfied while keeping best-effort behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reproduces the orchestrator WinRM readiness probe as SYSTEM (the
act_runner account) and surfaces the real error the silent
is_ready() swallows: TCP reachability, guest hostname from the TLS
cert CN, and an auth x username-form matrix, with a classified
fix recommendation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stores a guest-VM credential into the LocalSystem keyring vault via a
transient SYSTEM scheduled task, using the production venv's keyring
(the backend ci_orchestrator reads). Fixes the per-user vault mismatch
that made act_runner (LocalSystem) fail with "Credential not found in
keyring" for cmdkey/UI-stored credentials.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
Each Linux BuildVM clone gets a new SSH host key for the same
NAT IP, causing 'host key has changed' errors with accept-new mode.
After IP detection, ssh-keygen -R removes the old entry from the
CI-local known_hosts file (F:\CI\State\known_hosts by default)
before Phase 4/5 make any SSH connection. The file is created if
it does not exist yet. Windows jobs are unaffected.
gitea/burnin-dummy/build.sh: bash equivalent of build.ps1 (sleep 30,
writes dist/build-info.txt). Needed because powershell is not present
on LinuxBuild2404 template.
scripts/Start-BurnInTest-Linux.ps1: burn-in wrapper for LinuxBuild2404
template (SnapshotName=BaseClean-Linux, GuestOS=Linux,
BuildCommand='bash build.sh'). Mirrors Start-BurnInTest.ps1.
gitea/burnin-dummy/build.ps1:
Fake build script to commit to the burnin-dummy repo.
Creates dist\build-info.txt (date, host, pid) after ~30s ping delay.
Invoke-RemoteBuild packages dist\ as artifacts.zip.
Start-BurnInTest.ps1:
Default BuildCommand updated from ping-only delay to build.ps1 invocation.
Run with -NoSkipArtifact to exercise the full artifact pipeline.
In PS 5.1, $(if (...) { '-SkipArtifact' }) inside a call produces the
literal string '-SkipArtifact' bound as a positional argument (-> IPList),
not interpreted as a switch name.
Fixed by building a hashtable and setting SkipArtifact = \True when needed.
PROBLEM: Invoke-RemoteBuild.ps1 was creating a placeholder artifact when the build
produced no dist/ directory. In production this hides build defects (build exits 0
but produces nothing = silent failure treated as success).
SOLUTION: Add explicit -SkipArtifact switch to the call chain:
Invoke-RemoteBuild.ps1:
- Add -SkipArtifact switch. When set: skips packaging entirely (no zip created).
- When absent (production default): missing GuestArtifactSource is a hard error
with a clear message pointing to -SkipArtifact as the opt-in for no-output builds.
Invoke-CIJob.ps1:
- Add -SkipArtifact switch. Passes it to Invoke-RemoteBuild.ps1.
- Phase 6 (Get-BuildArtifacts) is entirely skipped when set; logs 'skipped' event.
Test-CapacityBurnIn.ps1:
- Add -SkipArtifact switch. Adds '-SkipArtifact' flag to each child job argList.
Start-BurnInTest.ps1:
- Always passes -SkipArtifact by default (burn-in uses delay commands, no output).
- Add -NoSkipArtifact override switch for runs that do produce real artifacts.
- Remove -NoSkipArtifact escape from wrapper: default burn-in never needs artifacts.
Invoke-RemoteBuild.ps1: when BuildCommand succeeds (exit 0) but GuestArtifactSource
(default: dist) does not exist, instead of silently skipping packaging, now creates a
placeholder 'build-ok.txt' in the artifact output dir and zips it. This ensures
Get-BuildArtifacts.ps1 always finds artifacts.zip regardless of whether the build
produced real output files. Correct for burn-in, smoke, and delay-only build commands.
Start-BurnInTest.ps1: revert default BuildCommand to 'ping 127.0.0.1 -n 31 > nul'
(~30s delay, no artifact output) now that the pipeline handles missing dist gracefully.
The previous 'if not exist dist mkdir dist' workaround is no longer needed.
'exit 0' left C:\CI\build\dist non-existent -> packaging skipped -> Get-BuildArtifacts
could not find artifacts.zip -> all 4 jobs FAIL at artifact collection phase.
New default: 'if not exist dist mkdir dist & echo burnin_ok > dist\marker.txt'
This creates dist\marker.txt relative to C:\CI\build (GuestWorkDir), so the
packaging step finds it, creates artifacts.zip, and Get-BuildArtifacts succeeds.
Also updated docblock: clarify command runs via cmd /c; replace Start-Sleep example
(PS cmdlet, does not work via WinRM cmd) with ping delay equivalent.
scripts/Test-CapacityBurnIn.ps1 (new): N-way parallel burn-in harness using Start-Job
to launch N concurrent Invoke-CIJob.ps1 child processes for R rounds. After each round
asserts: all VMs destroyed, all jobs got distinct IPs, vm-start.lock gone, ip-leases
empty, wall time recorded. Params: -RepoUrl (mandatory), -IPList = @() (optional,
auto-detects via VMware Tools when empty), -GiteaCredentialTarget = 'GiteaPAT',
-Parallelism = 4, -Rounds = 3, -RoundTimeoutMinutes = 60, plus all Invoke-CIJob params.
scripts/Start-BurnInTest.ps1 (new): lab wrapper with all default params hardcoded
(template F:\CI\Templates\WinBuild2025, credentials BuildVMGuest/GiteaPAT, etc.)
for daily use. Exposes -Rounds, -Parallelism, -BuildCommand, -RoundTimeoutMinutes.
In PS double-quoted strings, backslash does NOT escape '$'.
Only backtick does ('`$'). The previous '\' was being
expanded as a PS variable (undefined), triggering StrictMode error:
'Cannot retrieve variable \ because it has not been set.'
Replace '\$' with '`$' so the literal shell variable reference
\ is emitted correctly to the SSH command.
Test-Ns7zipBuild-Linux.ps1: e2e test for nsis7z plugin build on Linux VM
(confirmed PASS: nsis7z.dll 2272 KB in 03:09 2026-05-11)
Test-NsinnounpBuild-Linux.ps1: e2e test for nsinnounp plugin build on Linux VM
Exported functions:
- Invoke-SshCommand run a command on a Linux guest via ssh.exe (BatchMode=yes)
- Copy-SshItem scp a local path to a remote path (or vice versa)
- Test-SshReady test TCP:22 + ssh echo round-trip
Used by Wait-VMReady.ps1 -Transport SSH, Invoke-RemoteBuild.ps1 -GuestOS Linux,
Get-BuildArtifacts.ps1 -GuestOS Linux.
After Invoke-RemoteBuild closes its session, wsmprovhost.exe may take a moment
to release resources. Subsequent New-PSSession in Get-BuildArtifacts fails with
'WSMan service could not launch a host process'.
Add retry loop: 3 attempts, 10s backoff. Enough for WinRM to recover between
the build session and artifact collection session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
credential-store --file fails in WinRM on Windows (path escaping, no helper
binary resolution in non-interactive shell). Switch to http.extraHeader:
git -c credential.helper= -c 'http.extraHeader=Authorization: Basic <b64>' clone ...
Advantages:
- No credential helper binary needed
- No temp file (no cleanup required)
- Token never in URL, command display, or Write-Host output
- Works in WinRM non-interactive sessions (no TTY dependency)
- GCM still cleared by credential.helper= prefix
§1.5 compliant: Base64 is in git process argv (not logged by Write-Host).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Git Credential Manager (GCM) installed by Git for Windows tries to open
an interactive credential prompt or wincredman store — both fail in a
WinRM session (no TTY, no interactive desktop).
Fix:
- Set GIT_TERMINAL_PROMPT=0 to suppress all interactive git prompts
- Prepend '-c credential.helper=' (empty string) to reset the helper chain
before injecting our store-based helper (clearing GCM from the chain)
- Custom credential helper (store --file <tmp>) then appended only when PAT present
Without PAT (public repo): GCM still cleared, no prompt attempted, clean fail
if auth is actually required.
With PAT: store-based helper used exclusively, GCM never invoked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs:
1. Run-Test used '& $orchestrator @params' without Out-Host — orchestrator output
collected in function pipeline, mixed with returned hashtable → caller received
array instead of hashtable → .Success property lookup threw under StrictMode.
Fix: pipe to Out-Host.
2. RepoUrl was ssh://gitea-ci/... (SSH config alias) — alias exists only on
host SSH config, not in guest VM. Guest-side clone (§3.3) needs HTTP URL
directly reachable from inside the VM.
Fix: separate -HostRepoUrl / -GuestRepoUrl params, inject per mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create Test-E2E-Section3.3.ps1 for side-by-side measurement:
- Runs nsis-plugin-nsinnounp with default host-side clone (baseline)
- Runs same build with -UseGitClone flag (target: guest-side git)
- Displays elapsed time, artifact size, performance delta
Usage:
.\scripts\Test-E2E-Section3.3.ps1 # Full comparison
.\scripts\Test-E2E-Section3.3.ps1 -SkipBaseline # Guest-clone only
.\scripts\Test-E2E-Section3.3.ps1 -SkipGuestClone # Baseline only
Prerequisite: Template VM has Git installed (§6.6 Tier-1 Toolchain).
Also: Defer §3.5 (vCPU/RAM tuning) and §3.4 (pre-warm pool) to home lab
section — not required for initial CI operability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add guest-side clone mode to Invoke-RemoteBuild.ps1:
Changes:
1. Update DESCRIPTION: document Mode 1 (host) vs Mode 2 (guest) clone
2. Add params: CloneUrl, CloneBranch, CloneCommit, CloneSubmodules
3. Mode validation: enforce mutual exclusion (HostSourceDir XOR CloneUrl)
4. Conditional source prep:
- Mode 1 (default): compress host source, zip-transfer, expand in guest
- Mode 2 (§3.3): clone repo directly in guest via git
Guest clone implementation (§3.3):
- Clone with --depth 1, --branch specified, optionally --recurse-submodules
- Checkout specific commit if provided (handles shallow history edge case)
- PAT handling per §1.5: read from Credential Manager (placeholder),
passed as env var in WinRM session, cleaned up immediately after clone
- No PAT in argv, logs, or transcript — only in session env scope
Prerequisite: Git for Windows in template (§6.6, complete)
Testing: E2E with nsis-plugin-nsinnounp (-UseGitClone flag)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add -UseGitClone switch to Invoke-CIJob.ps1 + conditional logic:
Changes:
1. Add -UseGitClone [switch] param + docstring (§3.3 mode)
2. Modify Phase 1 to skip host clone when -UseGitClone active
3. Phase 5: Pass CloneUrl/CloneBranch/CloneCommit instead of HostSourceDir
when -UseGitClone enabled
4. Finally: Skip host cleanup when -UseGitClone (dir never created)
When -UseGitClone is active:
- Phase 1 (host clone) skipped — clone happens in VM instead
- Phase 5 passes git repo parameters to Invoke-RemoteBuild.ps1
- PAT (if needed) read in Invoke-RemoteBuild session (§1.5)
Prerequisite: Git for Windows + 7-Zip in template (§6.6, now complete)
Work in progress: Invoke-RemoteBuild.ps1 modifications pending (source transfer
conditional + guest-side git clone logic).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simone
Simone