WSL on the host builds the 3 configs in parallel in ~103s (= one
config) — the parallel design is correct. The 4-vCPU CI VM ran 3x
make -j on 4 cores -> oversubscription thrash -> ~3x slower. Not
structural / not RAM. Bump guest-cpu to 8 (job.py patches the clone
VMX numvcpus); host is 10c/20t, this matrix is 2 VMs.
job.py already patches the clone VMX (numvcpus/memsize) via
--guest-cpu/--guest-memory-mb; expose them as action inputs
(default '0' = keep template). build-ns7zip.yml sets
guest-memory-mb: 8192 — the parallel MinGW cross-compile thrashes at
the 4 GB Linux template default (Windows template is 6 GB).
After a memory-heavy parallel build the guest can momentarily fail to
launch wsmprovhost (WSManFault 1018), which broke artifact collect
(now does Compress-Archive + fetch over WinRM). run() retries up to 4x
with exponential backoff on transient WSMan markers, dropping the
cached shell so a fresh one is opened once the guest recovers.
A bare `if: github.ref_type == 'tag'` on a needs-dependent job can
leave `release` stuck in "blocked" instead of skipped for
workflow_dispatch runs on Gitea. Use the explicit
${{ github.event_name == 'push' && github.ref_type == 'tag' }} form.
- SshTransport.run_streaming: incremental channel recv, forwards
stdout/stderr live (no more one-block-at-end). _linux_build uses it
with PYTHONUNBUFFERED so the guest python flushes promptly.
- _tail_lines split out for failure messages (no double print).
- Windows live streaming deferred (TODO 8.2) — pypsrp execute_ps
returns only at completion.
- build-ns7zip.yml: use-git-clone:'true' (in-guest), repo-url ->
https://gitea.emulab.it/... (guest-reachable; host SSH alias was
host-only). Requires GiteaPAT in the LocalSystem keyring or a public
nsis-plugin-ns7zip.
ns7zip now supports --dist (commit 57133b7 on nsis-plugin-ns7zip):
DLLs land in <repo>/dist/<config>. Build with `python build_plugin.py
--dist` and collect artifact-source 'dist'.
build_plugin.py is a dispatcher: it auto-detects host/toolset and
forwards extra args to the per-version script, which rejected
--final/--dist-dir. Both the Windows (legacy 2601_vs2026) and Linux
build scripts copy nsis7z.dll into <repo>/plugins/{x86-ansi,
x86-unicode,amd64-unicode}, not dist.
- build-command: just `python build_plugin.py` (python3 on Linux —
Ubuntu 24.04 has no `python`).
- artifact-source: 'plugins' (fixes the "dist missing" failure too).
Workflow now targets Simone/nsis-plugin-ns7zip. Renamed the file and
the workflow `name:`, and updated all references in active docs
(plans/archived and docs/archived left as frozen history).
On a non-zero build the guest stdout/stderr was written unflushed
then a ClickException raised, so the cause was lost (only
"build command failed (exit 1)" surfaced). _report_build_output now
emits the output with markers, flushes both streams, and returns the
last lines which are appended to the failure message — diagnosable
even if act_runner buffering drops the body. Applied to Linux + Windows.
PowerShell 5.1 drops a zero-length string when splatting to a native
exe, so an empty $repoCommit shifted flag/value pairing and click saw
the build command as an unexpected positional. Append --commit only
when a commit is set.
github.ref_name/github.sha refer to the workflow's own repo. When
repo-url points elsewhere (build-nsInnoUnp builds
nsis-plugin-nsinnounp) the workflow's branch does not exist there
("Remote branch ... not found in upstream origin"). New inputs:
- repo-branch: target branch (default: github.ref_name)
- repo-commit: target commit (default: github.sha only if repo-url
not overridden; cross-repo => empty = branch tip, no pin)
build-nsInnoUnp.yml sets repo-branch: main.
_host_clone hid all stderr ("exit 128" with no cause). Surface the
last stderr lines with scheme://user:pass@ credentials masked, so
failures (e.g. unresolved SSH alias under LocalSystem) are diagnosable.
repo-url is ssh://gitea-ci/... — an SSH alias defined only in the
host ~/.ssh/config. With the new in-guest-clone default the guest
tried to resolve `gitea-ci` and failed (name resolution). Set
use-git-clone:'false' so the host (which has the alias + SSH key)
performs the clone.
Short-form `Simone/local-ci-cd-system/...@main` resolved to
github.com (DEFAULT_ACTIONS_URL=github) -> "authentication required:
Repository not found". Use the full gitea.emulab.it URL like
self-test.yml.
The job command previously discarded --use-git-clone and --submodules
(`del`) and always in-guest cloned with submodules off — making both
action inputs dead no-ops (submodule repos silently broken).
- job.py: --use-git-clone/--host-clone and --submodules/--no-submodules
are real booleans defaulting to ON. Default = in-guest clone with
submodules. --host-clone clones on the host into a temp dir
(_host_clone) and transfers a zip; temp dir cleaned in finally.
clone_submodules now honoured (was hardcoded False).
- action.yml: use-git-clone / submodules default 'true'; forward the
explicit positive/negative flag so the Python default is overridden
deterministically.
- self-test.yml: add smoke-{windows,linux}-hostclone jobs covering the
host-side transport (default jobs already cover in-guest).
- tests: default-transport, host-clone, and --no-submodules coverage.
Artifact github.com dependency removed (fix 2 applied). Remaining:
actions/checkout@v4 in lint.yml. Recorded as deferred §8.1 with
revisit conditions (air-gapped host / total github.com independence).
actions/upload-artifact and download-artifact run code fetched from
github.com (and v4 is GHES-incompatible, v3 deprecated). The runner
host is single with a shared filesystem, so the orchestrator's
host-side collect dir is the artifact handoff:
- action.yml: replace the two upload-artifact steps with
host-path report steps; artifacts stay under F:\CI\Artifacts and
are aged by Invoke-RetentionPolicy.ps1.
- build-nsInnoUnp.yml: release job reads
F:\CI\Artifacts\{run_id}-{run_attempt}-{target} directly instead of
download-artifact; existing Gitea REST release-asset upload unchanged.
Tradeoff (accepted): artifacts are no longer downloadable from the
Gitea Actions run page for non-tag builds — only on host disk, and as
release assets for tagged builds.
Windows produced an intermediate C:\CI\output\artifacts.zip that
upload-artifact then re-zipped (zip-in-zip), while Linux deposited raw
files. Make both paths consistent: the build stages raw outputs into a
collect dir; collect zips it only as a transport archive, fetches, and
extracts into the host artifact dir; actions/upload-artifact produces
the single final archive.
- _windows_build: param artifact_zip -> output_dir; stage raw files
(in-place when source IS the collect dir, else wipe+copy), no
Compress-Archive; absolute artifact-source resolved.
- _windows_collect: zip guest dir as transport, fetch, extract (mirror
_linux_collect); guest_path is now the collect directory.
- job.py / build_run CLI: pass output_dir; rename --guest-artifact-zip
-> --guest-output-dir.
- Tests updated for the raw-file contract.
git clone --depth 1 --branch X only fetches the branch tip. When the
branch advances between workflow trigger and runner pickup, checking
out the pinned commit failed with "reference is not a tree: <sha>".
Fetch that exact commit (git fetch --depth 1 origin <sha>) before
checkout, on both Linux (SSH) and Windows (WinRM) paths.
Two bugs made the Linux smoke "succeed" with an empty artifact:
- artifact_source was always joined as workdir + '/' + src, so an
absolute source (/opt/ci/output) became /opt/ci/build//opt/ci/output
and was never found — only a non-fatal WARNING was logged.
- `rm -rf output_dir` ran unconditionally; when the build writes
directly into the collect dir (the smoke case: output == source),
it deleted the build output before collection.
Resolve absolute vs workdir-relative; when the source is the output
dir, collect in place without wiping; a missing/empty source is now a
hard failure instead of a silent empty artifact.
Reconcile docs with the end-to-end pipeline fixes:
- upload/download-artifact @v4 -> @v3 (Gitea GHES) in WORKFLOW-AUTHORING
and workflow-example.yml; add Common Mistakes rows (v4, action ref
form + DEFAULT_ACTIONS_URL, public action repo).
- BEST-PRACTICES / README / HOST-SETUP: guest credential must live in
the LocalSystem vault with a host-qualified username; document
Set-CIGuestCredential.ps1 / Test-CIGuestWinRM.ps1 and auth=ntlm.
- README / AGENTS / HOST-SETUP: production venv install is NON-editable
(LocalSystem); no CI workflow may install into it.
- HOST-SETUP: add DEFAULT_ACTIONS_URL=github + full-URL uses: + public
repo + @main requirements discovered during validation.
- Correct stale repo name local-ci-system -> local-ci-cd-system.
The Windows manual smoke used --guest-artifact-source
'C:\ci\workspace\artifact.txt' (a stale absolute path that does not
exist in the build flow; workdir is C:\CI\build). The build command
writes artifact.txt into the workdir, so the source must be relative
'artifact.txt' — matching the Linux example.
Two Windows-guest build bugs:
- The build command was wrapped in `cmd /c`, so PowerShell-authored
commands failed ("'New-Item' is not recognized"). Run it as PS
(seed LASTEXITCODE=0 so PS-only commands exit cleanly).
- artifact_source was always Join-Path'd with the workdir; an absolute
source (C:\CI\output) became C:\CI\build\C:\CI\output. Use the path
as-is when rooted, else Join-Path the workdir.
The python lint job set VENV_DIR=F:\CI\python\venv (the shared
orchestrator venv) and ran `pip install -e ".[dev]"`. Every lint run
repointed ci_orchestrator at the lint job's transient checkout; once
act_runner cleaned that dir, all later smoke/build jobs failed with
"No module named ci_orchestrator". Use a job-local venv under
github.workspace, rebuilt fresh each run.
Reproduces the orchestrator WinRM readiness probe as SYSTEM (the
act_runner account) and surfaces the real error the silent
is_ready() swallows: TCP reachability, guest hostname from the TLS
cert CN, and an auth x username-form matrix, with a classified
fix recommendation.
pypsrp defaults to auth='negotiate', which attempts Kerberos with no
domain and fails under SSPI with SEC_E_UNKNOWN_CREDENTIALS for the
workgroup ci_build account. Default to 'ntlm' (configurable). The
guest username must be host-qualified (WINBUILD-2025\ci_build) and
stored that way in the credential vault.
PowerShell 5.1 `Out-File -Encoding utf8` prepends a UTF-8 BOM. The
act_runner GITHUB_OUTPUT parser then read the key as "artifact-path",
so steps.invoke-ci.outputs.artifact-path resolved empty. Combined with
the trailing backslash, the upload path became "\" -> the F: drive root,
causing upload-artifact to ingest the entire drive (hundreds of GB of
unrelated files).
- Append outputs with UTF8Encoding($false) (no BOM).
- Throw if artifactPath is empty instead of emitting nothing.
- Drop the trailing backslash; use /*.log for the diagnostic glob.
act_runner runs as LocalSystem; `pip install -e .` writes a .pth to a
path LocalSystem can't resolve (N: user drive or a transient
F:\CI\RunnerWork job dir), causing "No module named ci_orchestrator".
Prescribe `pip install .` and warn about the cd target.
Stores a guest-VM credential into the LocalSystem keyring vault via a
transient SYSTEM scheduled task, using the production venv's keyring
(the backend ci_orchestrator reads). Fixes the per-user vault mismatch
that made act_runner (LocalSystem) fail with "Credential not found in
keyring" for cmdkey/UI-stored credentials.
Gitea act_runner reports as GHES; @actions/artifact v2 (upload/
download-artifact@v4) uses the Twirp API Gitea does not implement,
failing with GHESNotSupportedError. v3 uses the artifact protocol
Gitea supports.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12.
- Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps.
- Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
- Rinominato gitea/workflows/build-nsis.yml in build-nsInnoUnp.yml
- Aggiunto job release: scarica artifact Windows+Linux, crea release Gitea
via API REST, comprime e uploada gli asset come zip (richiede secret GITEA_TOKEN)
- Aggiornati tutti i riferimenti in README.md, TODO.md, plans/*.md
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
Each Ubuntu clone derived from the same snapshot shared an identical
machine-id, causing systemd-networkd to use the same DHCP client-id
and receive the same NAT IP triggering IP collision failures.
Step 7b truncates /etc/machine-id and removes /var/lib/dbus/machine-id
before the snapshot is taken. On first boot each clone generates a
fresh machine-id from /dev/urandom -> unique DHCP client-id -> unique IP.
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
Each Linux BuildVM clone gets a new SSH host key for the same
NAT IP, causing 'host key has changed' errors with accept-new mode.
After IP detection, ssh-keygen -R removes the old entry from the
CI-local known_hosts file (F:\CI\State\known_hosts by default)
before Phase 4/5 make any SSH connection. The file is created if
it does not exist yet. Windows jobs are unaffected.
gitea/burnin-dummy/build.sh: bash equivalent of build.ps1 (sleep 30,
writes dist/build-info.txt). Needed because powershell is not present
on LinuxBuild2404 template.
scripts/Start-BurnInTest-Linux.ps1: burn-in wrapper for LinuxBuild2404
template (SnapshotName=BaseClean-Linux, GuestOS=Linux,
BuildCommand='bash build.sh'). Mirrors Start-BurnInTest.ps1.
gitea/burnin-dummy/build.ps1:
Fake build script to commit to the burnin-dummy repo.
Creates dist\build-info.txt (date, host, pid) after ~30s ping delay.
Invoke-RemoteBuild packages dist\ as artifacts.zip.
Start-BurnInTest.ps1:
Default BuildCommand updated from ping-only delay to build.ps1 invocation.
Run with -NoSkipArtifact to exercise the full artifact pipeline.
In PS 5.1, $(if (...) { '-SkipArtifact' }) inside a call produces the
literal string '-SkipArtifact' bound as a positional argument (-> IPList),
not interpreted as a switch name.
Fixed by building a hashtable and setting SkipArtifact = \True when needed.
PROBLEM: Invoke-RemoteBuild.ps1 was creating a placeholder artifact when the build
produced no dist/ directory. In production this hides build defects (build exits 0
but produces nothing = silent failure treated as success).
SOLUTION: Add explicit -SkipArtifact switch to the call chain:
Invoke-RemoteBuild.ps1:
- Add -SkipArtifact switch. When set: skips packaging entirely (no zip created).
- When absent (production default): missing GuestArtifactSource is a hard error
with a clear message pointing to -SkipArtifact as the opt-in for no-output builds.
Invoke-CIJob.ps1:
- Add -SkipArtifact switch. Passes it to Invoke-RemoteBuild.ps1.
- Phase 6 (Get-BuildArtifacts) is entirely skipped when set; logs 'skipped' event.
Test-CapacityBurnIn.ps1:
- Add -SkipArtifact switch. Adds '-SkipArtifact' flag to each child job argList.
Start-BurnInTest.ps1:
- Always passes -SkipArtifact by default (burn-in uses delay commands, no output).
- Add -NoSkipArtifact override switch for runs that do produce real artifacts.
- Remove -NoSkipArtifact escape from wrapper: default burn-in never needs artifacts.
Invoke-RemoteBuild.ps1: when BuildCommand succeeds (exit 0) but GuestArtifactSource
(default: dist) does not exist, instead of silently skipping packaging, now creates a
placeholder 'build-ok.txt' in the artifact output dir and zips it. This ensures
Get-BuildArtifacts.ps1 always finds artifacts.zip regardless of whether the build
produced real output files. Correct for burn-in, smoke, and delay-only build commands.
Start-BurnInTest.ps1: revert default BuildCommand to 'ping 127.0.0.1 -n 31 > nul'
(~30s delay, no artifact output) now that the pipeline handles missing dist gracefully.
The previous 'if not exist dist mkdir dist' workaround is no longer needed.
'exit 0' left C:\CI\build\dist non-existent -> packaging skipped -> Get-BuildArtifacts
could not find artifacts.zip -> all 4 jobs FAIL at artifact collection phase.
New default: 'if not exist dist mkdir dist & echo burnin_ok > dist\marker.txt'
This creates dist\marker.txt relative to C:\CI\build (GuestWorkDir), so the
packaging step finds it, creates artifacts.zip, and Get-BuildArtifacts succeeds.
Also updated docblock: clarify command runs via cmd /c; replace Start-Sleep example
(PS cmdlet, does not work via WinRM cmd) with ping delay equivalent.
scripts/Test-CapacityBurnIn.ps1 (new): N-way parallel burn-in harness using Start-Job
to launch N concurrent Invoke-CIJob.ps1 child processes for R rounds. After each round
asserts: all VMs destroyed, all jobs got distinct IPs, vm-start.lock gone, ip-leases
empty, wall time recorded. Params: -RepoUrl (mandatory), -IPList = @() (optional,
auto-detects via VMware Tools when empty), -GiteaCredentialTarget = 'GiteaPAT',
-Parallelism = 4, -Rounds = 3, -RoundTimeoutMinutes = 60, plus all Invoke-CIJob params.
scripts/Start-BurnInTest.ps1 (new): lab wrapper with all default params hardcoded
(template F:\CI\Templates\WinBuild2025, credentials BuildVMGuest/GiteaPAT, etc.)
for daily use. Exposes -Rounds, -Parallelism, -BuildCommand, -RoundTimeoutMinutes.
Riscritta gitea/workflow-example.yml per usare la composite action
Simone/local-ci-system/.gitea/actions/local-ci-build@main invece della
chiamata diretta a Invoke-CIJob.ps1.
Commenti inline guidano la personalizzazione per ogni repo:
- build-command e artifact-source (unici campi obbligatori da adattare)
- submodules, repo-url, use-git-clone (opzionali)
- extra-guest-env-json per secret injection (chiavi di firma, GPG)
- artifact-name personalizzato (e.g. con ref_name per tag build)
Aggiunge workflow_dispatch per trigger manuale.
Rimossa logica di orchestrazione inline: zero duplicazione tra repo.
TODO.md: Gitea Workflow Integration chiuso (5/5).
Setup-WinBuild2025.ps1:
- New [switch] \ param (after \)
- Step 12 block wrapped with if (\) { skip-message } else { ... }
- Final gate Tier-2 Assert-Step wrapped with if (-not \)
Prepare-WinBuild2025.ps1:
- New [switch] \ param forwarded to Setup via setupArgs
Validate-SetupState.ps1:
- New [switch] \ param
- \ passed into both remote scriptblocks via -ArgumentList
- Tier-2 Chk calls guarded with if (-not \) in both blocks
Simone