Stores a guest-VM credential into the LocalSystem keyring vault via a
transient SYSTEM scheduled task, using the production venv's keyring
(the backend ci_orchestrator reads). Fixes the per-user vault mismatch
that made act_runner (LocalSystem) fail with "Credential not found in
keyring" for cmdkey/UI-stored credentials.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
Each Linux BuildVM clone gets a new SSH host key for the same
NAT IP, causing 'host key has changed' errors with accept-new mode.
After IP detection, ssh-keygen -R removes the old entry from the
CI-local known_hosts file (F:\CI\State\known_hosts by default)
before Phase 4/5 make any SSH connection. The file is created if
it does not exist yet. Windows jobs are unaffected.
gitea/burnin-dummy/build.sh: bash equivalent of build.ps1 (sleep 30,
writes dist/build-info.txt). Needed because powershell is not present
on LinuxBuild2404 template.
scripts/Start-BurnInTest-Linux.ps1: burn-in wrapper for LinuxBuild2404
template (SnapshotName=BaseClean-Linux, GuestOS=Linux,
BuildCommand='bash build.sh'). Mirrors Start-BurnInTest.ps1.
gitea/burnin-dummy/build.ps1:
Fake build script to commit to the burnin-dummy repo.
Creates dist\build-info.txt (date, host, pid) after ~30s ping delay.
Invoke-RemoteBuild packages dist\ as artifacts.zip.
Start-BurnInTest.ps1:
Default BuildCommand updated from ping-only delay to build.ps1 invocation.
Run with -NoSkipArtifact to exercise the full artifact pipeline.
In PS 5.1, $(if (...) { '-SkipArtifact' }) inside a call produces the
literal string '-SkipArtifact' bound as a positional argument (-> IPList),
not interpreted as a switch name.
Fixed by building a hashtable and setting SkipArtifact = \True when needed.
PROBLEM: Invoke-RemoteBuild.ps1 was creating a placeholder artifact when the build
produced no dist/ directory. In production this hides build defects (build exits 0
but produces nothing = silent failure treated as success).
SOLUTION: Add explicit -SkipArtifact switch to the call chain:
Invoke-RemoteBuild.ps1:
- Add -SkipArtifact switch. When set: skips packaging entirely (no zip created).
- When absent (production default): missing GuestArtifactSource is a hard error
with a clear message pointing to -SkipArtifact as the opt-in for no-output builds.
Invoke-CIJob.ps1:
- Add -SkipArtifact switch. Passes it to Invoke-RemoteBuild.ps1.
- Phase 6 (Get-BuildArtifacts) is entirely skipped when set; logs 'skipped' event.
Test-CapacityBurnIn.ps1:
- Add -SkipArtifact switch. Adds '-SkipArtifact' flag to each child job argList.
Start-BurnInTest.ps1:
- Always passes -SkipArtifact by default (burn-in uses delay commands, no output).
- Add -NoSkipArtifact override switch for runs that do produce real artifacts.
- Remove -NoSkipArtifact escape from wrapper: default burn-in never needs artifacts.
Invoke-RemoteBuild.ps1: when BuildCommand succeeds (exit 0) but GuestArtifactSource
(default: dist) does not exist, instead of silently skipping packaging, now creates a
placeholder 'build-ok.txt' in the artifact output dir and zips it. This ensures
Get-BuildArtifacts.ps1 always finds artifacts.zip regardless of whether the build
produced real output files. Correct for burn-in, smoke, and delay-only build commands.
Start-BurnInTest.ps1: revert default BuildCommand to 'ping 127.0.0.1 -n 31 > nul'
(~30s delay, no artifact output) now that the pipeline handles missing dist gracefully.
The previous 'if not exist dist mkdir dist' workaround is no longer needed.
'exit 0' left C:\CI\build\dist non-existent -> packaging skipped -> Get-BuildArtifacts
could not find artifacts.zip -> all 4 jobs FAIL at artifact collection phase.
New default: 'if not exist dist mkdir dist & echo burnin_ok > dist\marker.txt'
This creates dist\marker.txt relative to C:\CI\build (GuestWorkDir), so the
packaging step finds it, creates artifacts.zip, and Get-BuildArtifacts succeeds.
Also updated docblock: clarify command runs via cmd /c; replace Start-Sleep example
(PS cmdlet, does not work via WinRM cmd) with ping delay equivalent.
scripts/Test-CapacityBurnIn.ps1 (new): N-way parallel burn-in harness using Start-Job
to launch N concurrent Invoke-CIJob.ps1 child processes for R rounds. After each round
asserts: all VMs destroyed, all jobs got distinct IPs, vm-start.lock gone, ip-leases
empty, wall time recorded. Params: -RepoUrl (mandatory), -IPList = @() (optional,
auto-detects via VMware Tools when empty), -GiteaCredentialTarget = 'GiteaPAT',
-Parallelism = 4, -Rounds = 3, -RoundTimeoutMinutes = 60, plus all Invoke-CIJob params.
scripts/Start-BurnInTest.ps1 (new): lab wrapper with all default params hardcoded
(template F:\CI\Templates\WinBuild2025, credentials BuildVMGuest/GiteaPAT, etc.)
for daily use. Exposes -Rounds, -Parallelism, -BuildCommand, -RoundTimeoutMinutes.
In PS double-quoted strings, backslash does NOT escape '$'.
Only backtick does ('`$'). The previous '\' was being
expanded as a PS variable (undefined), triggering StrictMode error:
'Cannot retrieve variable \ because it has not been set.'
Replace '\$' with '`$' so the literal shell variable reference
\ is emitted correctly to the SSH command.
Test-Ns7zipBuild-Linux.ps1: e2e test for nsis7z plugin build on Linux VM
(confirmed PASS: nsis7z.dll 2272 KB in 03:09 2026-05-11)
Test-NsinnounpBuild-Linux.ps1: e2e test for nsinnounp plugin build on Linux VM
Exported functions:
- Invoke-SshCommand run a command on a Linux guest via ssh.exe (BatchMode=yes)
- Copy-SshItem scp a local path to a remote path (or vice versa)
- Test-SshReady test TCP:22 + ssh echo round-trip
Used by Wait-VMReady.ps1 -Transport SSH, Invoke-RemoteBuild.ps1 -GuestOS Linux,
Get-BuildArtifacts.ps1 -GuestOS Linux.
After Invoke-RemoteBuild closes its session, wsmprovhost.exe may take a moment
to release resources. Subsequent New-PSSession in Get-BuildArtifacts fails with
'WSMan service could not launch a host process'.
Add retry loop: 3 attempts, 10s backoff. Enough for WinRM to recover between
the build session and artifact collection session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
credential-store --file fails in WinRM on Windows (path escaping, no helper
binary resolution in non-interactive shell). Switch to http.extraHeader:
git -c credential.helper= -c 'http.extraHeader=Authorization: Basic <b64>' clone ...
Advantages:
- No credential helper binary needed
- No temp file (no cleanup required)
- Token never in URL, command display, or Write-Host output
- Works in WinRM non-interactive sessions (no TTY dependency)
- GCM still cleared by credential.helper= prefix
§1.5 compliant: Base64 is in git process argv (not logged by Write-Host).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Git Credential Manager (GCM) installed by Git for Windows tries to open
an interactive credential prompt or wincredman store — both fail in a
WinRM session (no TTY, no interactive desktop).
Fix:
- Set GIT_TERMINAL_PROMPT=0 to suppress all interactive git prompts
- Prepend '-c credential.helper=' (empty string) to reset the helper chain
before injecting our store-based helper (clearing GCM from the chain)
- Custom credential helper (store --file <tmp>) then appended only when PAT present
Without PAT (public repo): GCM still cleared, no prompt attempted, clean fail
if auth is actually required.
With PAT: store-based helper used exclusively, GCM never invoked.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs:
1. Run-Test used '& $orchestrator @params' without Out-Host — orchestrator output
collected in function pipeline, mixed with returned hashtable → caller received
array instead of hashtable → .Success property lookup threw under StrictMode.
Fix: pipe to Out-Host.
2. RepoUrl was ssh://gitea-ci/... (SSH config alias) — alias exists only on
host SSH config, not in guest VM. Guest-side clone (§3.3) needs HTTP URL
directly reachable from inside the VM.
Fix: separate -HostRepoUrl / -GuestRepoUrl params, inject per mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create Test-E2E-Section3.3.ps1 for side-by-side measurement:
- Runs nsis-plugin-nsinnounp with default host-side clone (baseline)
- Runs same build with -UseGitClone flag (target: guest-side git)
- Displays elapsed time, artifact size, performance delta
Usage:
.\scripts\Test-E2E-Section3.3.ps1 # Full comparison
.\scripts\Test-E2E-Section3.3.ps1 -SkipBaseline # Guest-clone only
.\scripts\Test-E2E-Section3.3.ps1 -SkipGuestClone # Baseline only
Prerequisite: Template VM has Git installed (§6.6 Tier-1 Toolchain).
Also: Defer §3.5 (vCPU/RAM tuning) and §3.4 (pre-warm pool) to home lab
section — not required for initial CI operability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add guest-side clone mode to Invoke-RemoteBuild.ps1:
Changes:
1. Update DESCRIPTION: document Mode 1 (host) vs Mode 2 (guest) clone
2. Add params: CloneUrl, CloneBranch, CloneCommit, CloneSubmodules
3. Mode validation: enforce mutual exclusion (HostSourceDir XOR CloneUrl)
4. Conditional source prep:
- Mode 1 (default): compress host source, zip-transfer, expand in guest
- Mode 2 (§3.3): clone repo directly in guest via git
Guest clone implementation (§3.3):
- Clone with --depth 1, --branch specified, optionally --recurse-submodules
- Checkout specific commit if provided (handles shallow history edge case)
- PAT handling per §1.5: read from Credential Manager (placeholder),
passed as env var in WinRM session, cleaned up immediately after clone
- No PAT in argv, logs, or transcript — only in session env scope
Prerequisite: Git for Windows in template (§6.6, complete)
Testing: E2E with nsis-plugin-nsinnounp (-UseGitClone flag)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add -UseGitClone switch to Invoke-CIJob.ps1 + conditional logic:
Changes:
1. Add -UseGitClone [switch] param + docstring (§3.3 mode)
2. Modify Phase 1 to skip host clone when -UseGitClone active
3. Phase 5: Pass CloneUrl/CloneBranch/CloneCommit instead of HostSourceDir
when -UseGitClone enabled
4. Finally: Skip host cleanup when -UseGitClone (dir never created)
When -UseGitClone is active:
- Phase 1 (host clone) skipped — clone happens in VM instead
- Phase 5 passes git repo parameters to Invoke-RemoteBuild.ps1
- PAT (if needed) read in Invoke-RemoteBuild session (§1.5)
Prerequisite: Git for Windows + 7-Zip in template (§6.6, now complete)
Work in progress: Invoke-RemoteBuild.ps1 modifications pending (source transfer
conditional + guest-side git clone logic).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§3.2 — Performance: Implement 7-Zip compression for artifacts with fallback
to Compress-Archive for compatibility. Three call sites updated:
1. Host-side source compression: new Compress-BuildArtifact helper function
2. Guest-side custom build artifacts: inline 7-Zip check in scriptblock
3. Guest-side dotnet build output: inline 7-Zip check in scriptblock
Uses 7z.exe with -mmt=on (multi-threaded) and -mx1 (fast, low ratio impact).
Falls back to Compress-Archive if 7-Zip not found (until §6.6 template update).
Expected 10-20s speedup per build once 7-Zip installed in template.
Also defer all security items (§1.2/1.3/1.5/1.7/1.8) to home-lab section
since environment is isolated and excessive for current use case.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
Write-JobEvent appends one JSON line per phase transition (ts, jobId,
phase, status, data). Events emitted at: job.start, phase1.clone-repo
start/success, phase2-3b.vm-start start/success, phase4.wait-ready
start/success, phase5.build start/success, phase6.artifacts start/success,
job.success/failure (with elapsedSec and error string).
Silent no-op if $jsonLog is null (log dir setup failed). Errors in
Write-JobEvent are swallowed — logging never breaks the build.
Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.
§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
(source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
Optional -WebhookUrl for Discord/Gitea webhook notification.
Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).
§4.4 — Incident runbook (docs/RUNBOOK.md)
Four scenarios with symptom / triage commands / fix / escalation:
1. Runner offline in Gitea UI
2. All builds fail in Phase 2 (clone/start/IP)
3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
4. Template VMX corrupt after host crash (lock removal + backup restore)
Quick-reference table at the end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§2.1 — IP race with capacity>1 (Invoke-CIJob.ps1)
Phases 2-3b (clone+start+IP) now run inside a file-based exclusive mutex
(F:\CI\State\vm-start.lock via FileShare.None). Only one job at a time is
in the VM-start phase, preventing concurrent vmrun starts from racing for
the same DHCP lease. After IP is confirmed unique it is written to an IP
lease file (F:\CI\State\ip-leases\<IP>.lease). Lock is released immediately
so build phases run fully in parallel. Lease released in finally block.
Collision detection: if a lease file already exists for the detected IP,
the job fails fast with a clear error rather than silently sharing a WinRM target.
§2.5 — Versioned snapshot (Invoke-CIJob.ps1 + runner/config.yaml)
$SnapshotName now defaults to GITEA_CI_SNAPSHOT_NAME env var, falling back
to 'BaseClean'. config.yaml documents the commented-out variable for snapshot
refresh workflow (BaseClean_<yyyyMMdd> naming convention).
§2.3 — Artifact/log retention (scripts/Invoke-RetentionPolicy.ps1)
Purges per-job subdirs under F:\CI\Artifacts and F:\CI\Logs older than
RetentionDays (default 30). Switches to AggressiveRetentionDays (default 7)
when F: free space drops below MinFreeGB (default 50 GB). Also removes stale
IP lease files older than 12 hours (defensive cleanup for crashed jobs).
SupportsShouldProcess (-WhatIf) for dry-run.
§2.2 — Scheduled task registration (scripts/Register-CIScheduledTasks.ps1)
Registers two tasks under \CI\ task folder as SYSTEM/HighestPrivilege:
CI-CleanupOrphans — every 6h + AtStartup, -MaxAgeHours 4
CI-RetentionPolicy — daily 03:00 (+30min random) + AtStartup (+15min random)
Idempotent (-Force), SupportsShouldProcess (-WhatIf), validates script paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Test-WSMan in Windows PowerShell 5.1 does not support -SessionOption (WSManSessionOption),
causing "Cannot find a parameter matching the name 'SessionOption'" at runtime.
Prepare-WinBuild2025.ps1:
- Remove Test-WSMan connectivity check step entirely
- Open New-PSSession directly (handles -SkipCACheck via PSSessionOption)
- Wrap New-PSSession in try/catch with the same actionable error message
- Remove $wsmOpt (WSManSessionOption) — no longer needed
Wait-GuestWinRMReady: replace Test-WSMan probe with New-PSSession probe + Remove-PSSession
Wait-VMReady.ps1:
- Remove $wsmOpt entirely
- Replace Test-WSMan -UseSSL -SessionOption with Test-NetConnection -Port 5986
(TCP open on 5986 = HTTPS listener up = VM ready; credentials not available here)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simone
Simone