idea-3- prefix was shared with idea-3-powershell-removal.md; rename the ESXi
idea to idea-4-esxi-support.md and update the ideas-overview link. Pure rename,
content unchanged.
All sections done: SSH keys, BuildVMGuest password, Gitea PAT,
OpenSSH Server, template powered-off verification, CI backup.
Fixed errata: .vmsn always present (not error indicator), backup
switched from Git tar to 7z LZMA2, removed scheduled-task section
(no CI-* tasks exist on this host).
User requirement (non-negotiable): the project must stay
first-class compatible with a Windows host permanently — Phase B
adds Linux *alongside* Windows, never replacing/degrading it.
- implementation-plan-A-B §2: categorical-constraint banner; B8
decommissioning cancelled (kept only as historical record);
master checklist B8 row updated.
- PhaseB-user-checklist: Passo 9 (B8) cancelled -> dual-host kept;
Passo 8 no longer a decommission gate; tracking table + closing
note updated.
build-ns7zip builds an EXTERNAL repo (nsis-plugin-ns7zip). The
`push: tags v*` trigger (inherited from when this workflow lived in
the plugin repo) made every CI-system tag — including the Phase A
marker v2.0.0-phaseA — auto-build and publish an ns7zip release on
the CI-system repo. Remove the tag trigger; manual dispatch only.
The release job's tag-gate `if` now never fires (skipped) — release
flow already validated with a throwaway tag.
Medium update: exec-summary status (A COMPLETATA, B ready), master
checklist A items [x], Phase A gating satisfied, §1 COMPLETATA banner
with real validation deviations (LocalSystem venv/keyring, github-free
artifacts, transport, shim common-params, deferred §8.1/§8.2), §7
A-side definition-of-done [x]. Fase B (§2/§3/§4) + risk matrix +
cronoprogramma left intact as the Phase B design reference alongside
PhaseB-user-checklist.md.
feature merged to main (753f47e), tag v2.0.0-phaseA pushed
(triggers build-ns7zip as final end-to-end validation of main).
All Passi 1-9 [x]. Residual non-blocking cleanup: delete the test
Gitea releases via UI (tags kept).
The blind -X -> --x shims forwarded PowerShell common parameters
(-ErrorAction, -Verbose, ...) to the Python CLI, e.g.
Measure-CIBenchmark.ps1 -> Remove-BuildVM.ps1 -ErrorAction
SilentlyContinue produced `ci_orchestrator vm remove --error-action`
-> "No such option" -> destroy failed (orphan VM left). Skip
common switches and consume value-taking ones. Applied to all 7
blind-loop shims; param-style shims (New-BuildVM, Invoke-RemoteBuild,
Get-BuildArtifacts) are not affected. PSSA clean.
Start-BurnInTest.ps1 (lab wrapper: burnin-dummy + build.ps1) — 4
concurrent x 3 rounds all PASS, clean VM/lock teardown each round.
Note the gotcha: raw Test-CapacityBurnIn.ps1 without -BuildCommand
defaults to dotnet and fails on the non-.NET dummy repo.
lint, self-test, build-ns7zip (Win+Linux) and the release job
(Gitea release + asset upload via REST API, auto token) all green.
Phase A Gitea end-to-end validated.
build-nsInnoUnp.yml shipped without the closing brace of the
asset-upload foreach (and no EOF newline) since its origin; the
release job had never run (tag-only) so the PS parse error
"MissingEndCurlyBrace" was latent until the test tag triggered it.
Close the loop, add per-asset + final Write-Host. PS parse verified.
- Remove-Securely: suppress PSUseShouldProcessForStateChangingFunctions
(internal best-effort shred; ShouldProcess prompts are undesirable)
+ add [CmdletBinding()].
- Replace the two empty catch blocks (Set-CIGuestCredential.ps1,
Test-CIGuestWinRM.ps1) with Write-Verbose so PSAvoidUsingEmptyCatchBlock
is satisfied while keeping best-effort behavior.
WSL on the host builds the 3 configs in parallel in ~103s (= one
config) — the parallel design is correct. The 4-vCPU CI VM ran 3x
make -j on 4 cores -> oversubscription thrash -> ~3x slower. Not
structural / not RAM. Bump guest-cpu to 8 (job.py patches the clone
VMX numvcpus); host is 10c/20t, this matrix is 2 VMs.
job.py already patches the clone VMX (numvcpus/memsize) via
--guest-cpu/--guest-memory-mb; expose them as action inputs
(default '0' = keep template). build-ns7zip.yml sets
guest-memory-mb: 8192 — the parallel MinGW cross-compile thrashes at
the 4 GB Linux template default (Windows template is 6 GB).
After a memory-heavy parallel build the guest can momentarily fail to
launch wsmprovhost (WSManFault 1018), which broke artifact collect
(now does Compress-Archive + fetch over WinRM). run() retries up to 4x
with exponential backoff on transient WSMan markers, dropping the
cached shell so a fresh one is opened once the guest recovers.
A bare `if: github.ref_type == 'tag'` on a needs-dependent job can
leave `release` stuck in "blocked" instead of skipped for
workflow_dispatch runs on Gitea. Use the explicit
${{ github.event_name == 'push' && github.ref_type == 'tag' }} form.
- SshTransport.run_streaming: incremental channel recv, forwards
stdout/stderr live (no more one-block-at-end). _linux_build uses it
with PYTHONUNBUFFERED so the guest python flushes promptly.
- _tail_lines split out for failure messages (no double print).
- Windows live streaming deferred (TODO 8.2) — pypsrp execute_ps
returns only at completion.
- build-ns7zip.yml: use-git-clone:'true' (in-guest), repo-url ->
https://gitea.emulab.it/... (guest-reachable; host SSH alias was
host-only). Requires GiteaPAT in the LocalSystem keyring or a public
nsis-plugin-ns7zip.
ns7zip now supports --dist (commit 57133b7 on nsis-plugin-ns7zip):
DLLs land in <repo>/dist/<config>. Build with `python build_plugin.py
--dist` and collect artifact-source 'dist'.
build_plugin.py is a dispatcher: it auto-detects host/toolset and
forwards extra args to the per-version script, which rejected
--final/--dist-dir. Both the Windows (legacy 2601_vs2026) and Linux
build scripts copy nsis7z.dll into <repo>/plugins/{x86-ansi,
x86-unicode,amd64-unicode}, not dist.
- build-command: just `python build_plugin.py` (python3 on Linux —
Ubuntu 24.04 has no `python`).
- artifact-source: 'plugins' (fixes the "dist missing" failure too).
Workflow now targets Simone/nsis-plugin-ns7zip. Renamed the file and
the workflow `name:`, and updated all references in active docs
(plans/archived and docs/archived left as frozen history).
On a non-zero build the guest stdout/stderr was written unflushed
then a ClickException raised, so the cause was lost (only
"build command failed (exit 1)" surfaced). _report_build_output now
emits the output with markers, flushes both streams, and returns the
last lines which are appended to the failure message — diagnosable
even if act_runner buffering drops the body. Applied to Linux + Windows.
PowerShell 5.1 drops a zero-length string when splatting to a native
exe, so an empty $repoCommit shifted flag/value pairing and click saw
the build command as an unexpected positional. Append --commit only
when a commit is set.
github.ref_name/github.sha refer to the workflow's own repo. When
repo-url points elsewhere (build-nsInnoUnp builds
nsis-plugin-nsinnounp) the workflow's branch does not exist there
("Remote branch ... not found in upstream origin"). New inputs:
- repo-branch: target branch (default: github.ref_name)
- repo-commit: target commit (default: github.sha only if repo-url
not overridden; cross-repo => empty = branch tip, no pin)
build-nsInnoUnp.yml sets repo-branch: main.
_host_clone hid all stderr ("exit 128" with no cause). Surface the
last stderr lines with scheme://user:pass@ credentials masked, so
failures (e.g. unresolved SSH alias under LocalSystem) are diagnosable.
repo-url is ssh://gitea-ci/... — an SSH alias defined only in the
host ~/.ssh/config. With the new in-guest-clone default the guest
tried to resolve `gitea-ci` and failed (name resolution). Set
use-git-clone:'false' so the host (which has the alias + SSH key)
performs the clone.
Short-form `Simone/local-ci-cd-system/...@main` resolved to
github.com (DEFAULT_ACTIONS_URL=github) -> "authentication required:
Repository not found". Use the full gitea.emulab.it URL like
self-test.yml.
The job command previously discarded --use-git-clone and --submodules
(`del`) and always in-guest cloned with submodules off — making both
action inputs dead no-ops (submodule repos silently broken).
- job.py: --use-git-clone/--host-clone and --submodules/--no-submodules
are real booleans defaulting to ON. Default = in-guest clone with
submodules. --host-clone clones on the host into a temp dir
(_host_clone) and transfers a zip; temp dir cleaned in finally.
clone_submodules now honoured (was hardcoded False).
- action.yml: use-git-clone / submodules default 'true'; forward the
explicit positive/negative flag so the Python default is overridden
deterministically.
- self-test.yml: add smoke-{windows,linux}-hostclone jobs covering the
host-side transport (default jobs already cover in-guest).
- tests: default-transport, host-clone, and --no-submodules coverage.
Artifact github.com dependency removed (fix 2 applied). Remaining:
actions/checkout@v4 in lint.yml. Recorded as deferred §8.1 with
revisit conditions (air-gapped host / total github.com independence).
actions/upload-artifact and download-artifact run code fetched from
github.com (and v4 is GHES-incompatible, v3 deprecated). The runner
host is single with a shared filesystem, so the orchestrator's
host-side collect dir is the artifact handoff:
- action.yml: replace the two upload-artifact steps with
host-path report steps; artifacts stay under F:\CI\Artifacts and
are aged by Invoke-RetentionPolicy.ps1.
- build-nsInnoUnp.yml: release job reads
F:\CI\Artifacts\{run_id}-{run_attempt}-{target} directly instead of
download-artifact; existing Gitea REST release-asset upload unchanged.
Tradeoff (accepted): artifacts are no longer downloadable from the
Gitea Actions run page for non-tag builds — only on host disk, and as
release assets for tagged builds.
Windows produced an intermediate C:\CI\output\artifacts.zip that
upload-artifact then re-zipped (zip-in-zip), while Linux deposited raw
files. Make both paths consistent: the build stages raw outputs into a
collect dir; collect zips it only as a transport archive, fetches, and
extracts into the host artifact dir; actions/upload-artifact produces
the single final archive.
- _windows_build: param artifact_zip -> output_dir; stage raw files
(in-place when source IS the collect dir, else wipe+copy), no
Compress-Archive; absolute artifact-source resolved.
- _windows_collect: zip guest dir as transport, fetch, extract (mirror
_linux_collect); guest_path is now the collect directory.
- job.py / build_run CLI: pass output_dir; rename --guest-artifact-zip
-> --guest-output-dir.
- Tests updated for the raw-file contract.
git clone --depth 1 --branch X only fetches the branch tip. When the
branch advances between workflow trigger and runner pickup, checking
out the pinned commit failed with "reference is not a tree: <sha>".
Fetch that exact commit (git fetch --depth 1 origin <sha>) before
checkout, on both Linux (SSH) and Windows (WinRM) paths.
Two bugs made the Linux smoke "succeed" with an empty artifact:
- artifact_source was always joined as workdir + '/' + src, so an
absolute source (/opt/ci/output) became /opt/ci/build//opt/ci/output
and was never found — only a non-fatal WARNING was logged.
- `rm -rf output_dir` ran unconditionally; when the build writes
directly into the collect dir (the smoke case: output == source),
it deleted the build output before collection.
Resolve absolute vs workdir-relative; when the source is the output
dir, collect in place without wiping; a missing/empty source is now a
hard failure instead of a silent empty artifact.
Reconcile docs with the end-to-end pipeline fixes:
- upload/download-artifact @v4 -> @v3 (Gitea GHES) in WORKFLOW-AUTHORING
and workflow-example.yml; add Common Mistakes rows (v4, action ref
form + DEFAULT_ACTIONS_URL, public action repo).
- BEST-PRACTICES / README / HOST-SETUP: guest credential must live in
the LocalSystem vault with a host-qualified username; document
Set-CIGuestCredential.ps1 / Test-CIGuestWinRM.ps1 and auth=ntlm.
- README / AGENTS / HOST-SETUP: production venv install is NON-editable
(LocalSystem); no CI workflow may install into it.
- HOST-SETUP: add DEFAULT_ACTIONS_URL=github + full-URL uses: + public
repo + @main requirements discovered during validation.
- Correct stale repo name local-ci-system -> local-ci-cd-system.
The Windows manual smoke used --guest-artifact-source
'C:\ci\workspace\artifact.txt' (a stale absolute path that does not
exist in the build flow; workdir is C:\CI\build). The build command
writes artifact.txt into the workdir, so the source must be relative
'artifact.txt' — matching the Linux example.
Two Windows-guest build bugs:
- The build command was wrapped in `cmd /c`, so PowerShell-authored
commands failed ("'New-Item' is not recognized"). Run it as PS
(seed LASTEXITCODE=0 so PS-only commands exit cleanly).
- artifact_source was always Join-Path'd with the workdir; an absolute
source (C:\CI\output) became C:\CI\build\C:\CI\output. Use the path
as-is when rooted, else Join-Path the workdir.
The python lint job set VENV_DIR=F:\CI\python\venv (the shared
orchestrator venv) and ran `pip install -e ".[dev]"`. Every lint run
repointed ci_orchestrator at the lint job's transient checkout; once
act_runner cleaned that dir, all later smoke/build jobs failed with
"No module named ci_orchestrator". Use a job-local venv under
github.workspace, rebuilt fresh each run.
Reproduces the orchestrator WinRM readiness probe as SYSTEM (the
act_runner account) and surfaces the real error the silent
is_ready() swallows: TCP reachability, guest hostname from the TLS
cert CN, and an auth x username-form matrix, with a classified
fix recommendation.
pypsrp defaults to auth='negotiate', which attempts Kerberos with no
domain and fails under SSPI with SEC_E_UNKNOWN_CREDENTIALS for the
workgroup ci_build account. Default to 'ntlm' (configurable). The
guest username must be host-qualified (WINBUILD-2025\ci_build) and
stored that way in the credential vault.
PowerShell 5.1 `Out-File -Encoding utf8` prepends a UTF-8 BOM. The
act_runner GITHUB_OUTPUT parser then read the key as "artifact-path",
so steps.invoke-ci.outputs.artifact-path resolved empty. Combined with
the trailing backslash, the upload path became "\" -> the F: drive root,
causing upload-artifact to ingest the entire drive (hundreds of GB of
unrelated files).
- Append outputs with UTF8Encoding($false) (no BOM).
- Throw if artifactPath is empty instead of emitting nothing.
- Drop the trailing backslash; use /*.log for the diagnostic glob.
act_runner runs as LocalSystem; `pip install -e .` writes a .pth to a
path LocalSystem can't resolve (N: user drive or a transient
F:\CI\RunnerWork job dir), causing "No module named ci_orchestrator".
Prescribe `pip install .` and warn about the cd target.
Runner fetches the composite action from @main; main still had
upload-artifact@v4 which fails on Gitea (GHESNotSupportedError).
Brings the v3 action.yml from feature/python-rewrite-and-linux-migration.
Stores a guest-VM credential into the LocalSystem keyring vault via a
transient SYSTEM scheduled task, using the production venv's keyring
(the backend ci_orchestrator reads). Fixes the per-user vault mismatch
that made act_runner (LocalSystem) fail with "Credential not found in
keyring" for cmdkey/UI-stored credentials.
Gitea act_runner reports as GHES; @actions/artifact v2 (upload/
download-artifact@v4) uses the Twirp API Gitea does not implement,
failing with GHESNotSupportedError. v3 uses the artifact protocol
Gitea supports.
Required so `uses: .../local-ci-build@main` resolves: the action only
existed on feature/python-rewrite-and-linux-migration. Brings just the
action.yml (host-side orchestrator unchanged) without merging the full
migration branch.
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
measured post-clone; added to result object and summary Format-Table
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12.
- Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps.
- Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
- Rinominato gitea/workflows/build-nsis.yml in build-nsInnoUnp.yml
- Aggiunto job release: scarica artifact Windows+Linux, crea release Gitea
via API REST, comprime e uploada gli asset come zip (richiede secret GITEA_TOKEN)
- Aggiornati tutti i riferimenti in README.md, TODO.md, plans/*.md
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.
This caused all 4 concurrent burn-in jobs to fail after ~15s with:
Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts
Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.
Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.
Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).
AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
Each Ubuntu clone derived from the same snapshot shared an identical
machine-id, causing systemd-networkd to use the same DHCP client-id
and receive the same NAT IP triggering IP collision failures.
Step 7b truncates /etc/machine-id and removes /var/lib/dbus/machine-id
before the snapshot is taken. On first boot each clone generates a
fresh machine-id from /dev/urandom -> unique DHCP client-id -> unique IP.
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
Each Linux BuildVM clone gets a new SSH host key for the same
NAT IP, causing 'host key has changed' errors with accept-new mode.
After IP detection, ssh-keygen -R removes the old entry from the
CI-local known_hosts file (F:\CI\State\known_hosts by default)
before Phase 4/5 make any SSH connection. The file is created if
it does not exist yet. Windows jobs are unaffected.
gitea/burnin-dummy/build.sh: bash equivalent of build.ps1 (sleep 30,
writes dist/build-info.txt). Needed because powershell is not present
on LinuxBuild2404 template.
scripts/Start-BurnInTest-Linux.ps1: burn-in wrapper for LinuxBuild2404
template (SnapshotName=BaseClean-Linux, GuestOS=Linux,
BuildCommand='bash build.sh'). Mirrors Start-BurnInTest.ps1.
Simone