scripts/install-ci-alias.sh installs `ci` (runs the orchestrator as ci-runner
via the prod venv) and `ci-me` (current user). Idempotent, marker-bounded;
supports --system (/etc/profile.d), --uninstall. setup-host-linux.sh gains a
step [7/7] that installs it system-wide, passing CI_PROD_PYTHON/CI_RUN_USER.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove _Common.psm1, _Transport.psm1 and their Pester test, plus their sole
consumer Measure-CIBenchmark.ps1 — all replaced by `bench measure`. Leaving
the script would dangle an Import-Module on a deleted .psm1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add -StaticIP/-Netmask/-Gateway params; injects guestinfo into cloned
VMX before start, mirroring job.py _inject_guestinfo_ip behaviour
- Add -GuestInfoOnly switch to Get-GuestIPAddress: skips vmrun
getGuestIPAddress fallback to prevent race where transient DHCP address
is returned before ci-static-ip.ps1 finishes reconfiguring the NIC
- Benchmark passes -GuestInfoOnly automatically when -StaticIP is set
- staticIP field added to benchmark.jsonl for DHCP vs static comparison
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Measure-CIBenchmark.ps1:
- Default params now Linux-aware via $IsWindows conditional
(TemplatePath, CloneBaseDir, VmrunPath, OutputDir)
- Add -GuestOS Auto/Windows/Linux param; Auto reads VMX guestOS field
(ubuntu-* → Linux, else Windows)
- Replace Test-NetConnection (Windows-only) with Test-TcpPort helper
using raw TcpClient async connect — works on PS5.1 and PS7/Linux
- Probe SSH/22 for Linux guests, WinRM/5986 for Windows guests
- Rename winrm→ready throughout ($t, JSONL field readySec, column header)
- Add guestOS field to JSONL output
_Common.psm1 / Invoke-VmrunBounded:
- Replace taskkill /F /T with platform-aware kill:
Windows → taskkill; Linux → Process.Kill(entireProcessTree) (.NET 5+)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All shims defaulted to F:\CI\python\venv\Scripts\python.exe when
CI_VENV_PYTHON was not set. On Linux host this caused instant failure.
Now uses $IsWindows (PS 6+ automatic var; absent on PS 5.1 = Windows)
to select /opt/ci/venv/bin/python on Linux. Windows paths unchanged.
Affects: Invoke-CIJob, Get-BuildArtifacts, Cleanup-OrphanedBuildVMs,
New-BuildVM, Remove-BuildVM, Wait-VMReady, Get-CIJobSummary,
Invoke-RemoteBuild, Watch-DiskSpace, Watch-RunnerHealth,
Set-CIGuestCredential, Test-CIGuestWinRM.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Passing $psExe as a second -ArgumentList item caused double-wrapping of
the string array, collapsing all args into one string when splatted.
Computing $IsWindows inside the worker avoids the extra param entirely.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
powershell.exe does not exist on Linux — child process launch failed
instantly (ExitCode null, Elapsed 00:00). Now selects powershell.exe
on Windows and pwsh on Linux via $IsWindows automatic variable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The blind -X -> --x shims forwarded PowerShell common parameters
(-ErrorAction, -Verbose, ...) to the Python CLI, e.g.
Measure-CIBenchmark.ps1 -> Remove-BuildVM.ps1 -ErrorAction
SilentlyContinue produced `ci_orchestrator vm remove --error-action`
-> "No such option" -> destroy failed (orphan VM left). Skip
common switches and consume value-taking ones. Applied to all 7
blind-loop shims; param-style shims (New-BuildVM, Invoke-RemoteBuild,
Get-BuildArtifacts) are not affected. PSSA clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove-Securely: suppress PSUseShouldProcessForStateChangingFunctions
(internal best-effort shred; ShouldProcess prompts are undesirable)
+ add [CmdletBinding()].
- Replace the two empty catch blocks (Set-CIGuestCredential.ps1,
Test-CIGuestWinRM.ps1) with Write-Verbose so PSAvoidUsingEmptyCatchBlock
is satisfied while keeping best-effort behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reproduces the orchestrator WinRM readiness probe as SYSTEM (the
act_runner account) and surfaces the real error the silent
is_ready() swallows: TCP reachability, guest hostname from the TLS
cert CN, and an auth x username-form matrix, with a classified
fix recommendation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stores a guest-VM credential into the LocalSystem keyring vault via a
transient SYSTEM scheduled task, using the production venv's keyring
(the backend ci_orchestrator reads). Fixes the per-user vault mismatch
that made act_runner (LocalSystem) fail with "Credential not found in
keyring" for cmdkey/UI-stored credentials.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
Each Linux BuildVM clone gets a new SSH host key for the same
NAT IP, causing 'host key has changed' errors with accept-new mode.
After IP detection, ssh-keygen -R removes the old entry from the
CI-local known_hosts file (F:\CI\State\known_hosts by default)
before Phase 4/5 make any SSH connection. The file is created if
it does not exist yet. Windows jobs are unaffected.
gitea/burnin-dummy/build.sh: bash equivalent of build.ps1 (sleep 30,
writes dist/build-info.txt). Needed because powershell is not present
on LinuxBuild2404 template.
scripts/Start-BurnInTest-Linux.ps1: burn-in wrapper for LinuxBuild2404
template (SnapshotName=BaseClean-Linux, GuestOS=Linux,
BuildCommand='bash build.sh'). Mirrors Start-BurnInTest.ps1.
gitea/burnin-dummy/build.ps1:
Fake build script to commit to the burnin-dummy repo.
Creates dist\build-info.txt (date, host, pid) after ~30s ping delay.
Invoke-RemoteBuild packages dist\ as artifacts.zip.
Start-BurnInTest.ps1:
Default BuildCommand updated from ping-only delay to build.ps1 invocation.
Run with -NoSkipArtifact to exercise the full artifact pipeline.
In PS 5.1, $(if (...) { '-SkipArtifact' }) inside a call produces the
literal string '-SkipArtifact' bound as a positional argument (-> IPList),
not interpreted as a switch name.
Fixed by building a hashtable and setting SkipArtifact = \True when needed.
PROBLEM: Invoke-RemoteBuild.ps1 was creating a placeholder artifact when the build
produced no dist/ directory. In production this hides build defects (build exits 0
but produces nothing = silent failure treated as success).
SOLUTION: Add explicit -SkipArtifact switch to the call chain:
Invoke-RemoteBuild.ps1:
- Add -SkipArtifact switch. When set: skips packaging entirely (no zip created).
- When absent (production default): missing GuestArtifactSource is a hard error
with a clear message pointing to -SkipArtifact as the opt-in for no-output builds.
Invoke-CIJob.ps1:
- Add -SkipArtifact switch. Passes it to Invoke-RemoteBuild.ps1.
- Phase 6 (Get-BuildArtifacts) is entirely skipped when set; logs 'skipped' event.
Test-CapacityBurnIn.ps1:
- Add -SkipArtifact switch. Adds '-SkipArtifact' flag to each child job argList.
Start-BurnInTest.ps1:
- Always passes -SkipArtifact by default (burn-in uses delay commands, no output).
- Add -NoSkipArtifact override switch for runs that do produce real artifacts.
- Remove -NoSkipArtifact escape from wrapper: default burn-in never needs artifacts.
Invoke-RemoteBuild.ps1: when BuildCommand succeeds (exit 0) but GuestArtifactSource
(default: dist) does not exist, instead of silently skipping packaging, now creates a
placeholder 'build-ok.txt' in the artifact output dir and zips it. This ensures
Get-BuildArtifacts.ps1 always finds artifacts.zip regardless of whether the build
produced real output files. Correct for burn-in, smoke, and delay-only build commands.
Start-BurnInTest.ps1: revert default BuildCommand to 'ping 127.0.0.1 -n 31 > nul'
(~30s delay, no artifact output) now that the pipeline handles missing dist gracefully.
The previous 'if not exist dist mkdir dist' workaround is no longer needed.
'exit 0' left C:\CI\build\dist non-existent -> packaging skipped -> Get-BuildArtifacts
could not find artifacts.zip -> all 4 jobs FAIL at artifact collection phase.
New default: 'if not exist dist mkdir dist & echo burnin_ok > dist\marker.txt'
This creates dist\marker.txt relative to C:\CI\build (GuestWorkDir), so the
packaging step finds it, creates artifacts.zip, and Get-BuildArtifacts succeeds.
Also updated docblock: clarify command runs via cmd /c; replace Start-Sleep example
(PS cmdlet, does not work via WinRM cmd) with ping delay equivalent.
scripts/Test-CapacityBurnIn.ps1 (new): N-way parallel burn-in harness using Start-Job
to launch N concurrent Invoke-CIJob.ps1 child processes for R rounds. After each round
asserts: all VMs destroyed, all jobs got distinct IPs, vm-start.lock gone, ip-leases
empty, wall time recorded. Params: -RepoUrl (mandatory), -IPList = @() (optional,
auto-detects via VMware Tools when empty), -GiteaCredentialTarget = 'GiteaPAT',
-Parallelism = 4, -Rounds = 3, -RoundTimeoutMinutes = 60, plus all Invoke-CIJob params.
scripts/Start-BurnInTest.ps1 (new): lab wrapper with all default params hardcoded
(template F:\CI\Templates\WinBuild2025, credentials BuildVMGuest/GiteaPAT, etc.)
for daily use. Exposes -Rounds, -Parallelism, -BuildCommand, -RoundTimeoutMinutes.
In PS double-quoted strings, backslash does NOT escape '$'.
Only backtick does ('`$'). The previous '\' was being
expanded as a PS variable (undefined), triggering StrictMode error:
'Cannot retrieve variable \ because it has not been set.'
Replace '\$' with '`$' so the literal shell variable reference
\ is emitted correctly to the SSH command.
Test-Ns7zipBuild-Linux.ps1: e2e test for nsis7z plugin build on Linux VM
(confirmed PASS: nsis7z.dll 2272 KB in 03:09 2026-05-11)
Test-NsinnounpBuild-Linux.ps1: e2e test for nsinnounp plugin build on Linux VM
Exported functions:
- Invoke-SshCommand run a command on a Linux guest via ssh.exe (BatchMode=yes)
- Copy-SshItem scp a local path to a remote path (or vice versa)
- Test-SshReady test TCP:22 + ssh echo round-trip
Used by Wait-VMReady.ps1 -Transport SSH, Invoke-RemoteBuild.ps1 -GuestOS Linux,
Get-BuildArtifacts.ps1 -GuestOS Linux.
After Invoke-RemoteBuild closes its session, wsmprovhost.exe may take a moment
to release resources. Subsequent New-PSSession in Get-BuildArtifacts fails with
'WSMan service could not launch a host process'.
Add retry loop: 3 attempts, 10s backoff. Enough for WinRM to recover between
the build session and artifact collection session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
credential-store --file fails in WinRM on Windows (path escaping, no helper
binary resolution in non-interactive shell). Switch to http.extraHeader:
git -c credential.helper= -c 'http.extraHeader=Authorization: Basic <b64>' clone ...
Advantages:
- No credential helper binary needed
- No temp file (no cleanup required)
- Token never in URL, command display, or Write-Host output
- Works in WinRM non-interactive sessions (no TTY dependency)
- GCM still cleared by credential.helper= prefix
§1.5 compliant: Base64 is in git process argv (not logged by Write-Host).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Git Credential Manager (GCM) installed by Git for Windows tries to open
an interactive credential prompt or wincredman store — both fail in a
WinRM session (no TTY, no interactive desktop).
Fix:
- Set GIT_TERMINAL_PROMPT=0 to suppress all interactive git prompts
- Prepend '-c credential.helper=' (empty string) to reset the helper chain
before injecting our store-based helper (clearing GCM from the chain)
- Custom credential helper (store --file <tmp>) then appended only when PAT present
Without PAT (public repo): GCM still cleared, no prompt attempted, clean fail
if auth is actually required.
With PAT: store-based helper used exclusively, GCM never invoked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simone
Simone