Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c04a2f25aa31b4f25daa11872307695e7c73a8da
local-ci-cd-system/plans/final-master-plan.md
T
Simone ec13461862 docs(plans): add multi-agent technical review and final master plan 
- plans/gpt55-analysis.md: independent analysis by GPT-5.5
- plans/opus47-analysis.md: independent analysis by Opus 4.7
- plans/gpt55-review-of-opus.md: GPT-5.5 cross-review of Opus analysis
- plans/opus47-review-of-gpt55.md: Opus 4.7 cross-review of GPT-5.5 analysis
- plans/final-master-plan.md: consolidated master plan (8129 words)

Severity: 2 Critical, 12 High, 16 Medium, 10 Low, 6 Nice-to-have.
Top priorities: C2 secret logging in Invoke-RemoteBuild.ps1, C1 Linux
matrix template selection, H1 capacity=4 burn-in.
2026-05-11 23:47:06 +02:00

76 KiB
Raw Blame History

Final Master Plan — Local CI/CD System

Date: 2026-05-11 Status: consolidated technical plan derived from two independent reviews (plans/gpt55-analysis.md, plans/opus47-analysis.md) and their cross-reviews (plans/gpt55-review-of-opus.md, plans/opus47-review-of-gpt55.md). Source files were re-read where the two reviewers disagreed.

This is the single authoritative document for finishing v1.0 of the system. When all Critical and High items are checked, the system is genuinely production-ready for homelab use.

Executive Summary

The Local CI/CD System is a well-engineered, single-host homelab CI built on Gitea Actions, act_runner v1.0.2, VMware Workstation linked clones, Windows PowerShell 5.1, WinRM HTTPS for Windows guests and SSH for Linux guests. The architecture is the right shape for the constraint set: a thin orchestrator on the host, ephemeral linked clones for isolation, two transports for two guest families, and file-based state for everything else. Both reviewers independently graded the architecture A- and operational maturity B-/C+. That matches what the source shows.

The system is feature-complete and operator-usable, not yet unattended-daily-trustworthy. The gap is small in code volume but real in operational consequence. Five concrete defects can break a live build today: the Linux matrix leg of gitea/workflows/build-nsis.yml silently selects the Windows template; Linux ExtraGuestEnv secrets are logged in clear text; the composite action outputs metadata is malformed YAML; the Linux Mode 2 PAT injection writes credentials into a remote command string; Setup-Host.ps1 creates a directory layout that no longer matches the live system. None of these are architectural failures. Each is a 5-to-45-minute fix that the design otherwise supports cleanly.

The system makes one large unvalidated claim: capacity: 4 in runner/config.yaml. The IP-allocation mutex and lease design in scripts/Invoke-CIJob.ps1 is sound on paper, but the only evidence is single-job e2e runs. Four concurrent real builds have not been measured. This is the single most consequential outstanding question. The remediation is a burn-in, not a redesign.

Top 3 risks today, in order: (1) the Linux matrix leg builds against the Windows template until the action wiring is fixed; (2) any workflow that uses extra-guest-env-json for secrets is leaking those values to Gitea logs on Linux builds; (3) capacity: 4 is a documented capability the system has never proven it keeps. Everything else is smaller.

Current Project State

The optimistic TODO.md and the conservative docs/TEST-PLAN-v1.3-to-HEAD.md disagree. The honest state, reconciled against the source files, is below.

Area Genuinely Done Partially Done Unvalidated / Broken
Windows VM lifecycle Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy (scripts/Invoke-CIJob.ps1, scripts/New-BuildVM.ps1, scripts/Wait-VMReady.ps1, scripts/Invoke-RemoteBuild.ps1, scripts/Get-BuildArtifacts.ps1, scripts/Remove-BuildVM.ps1) — Real 4-way concurrency
Linux VM lifecycle Cloud-init deploy, SSH transport, IP via guestinfo.ci-ip (template/Deploy-LinuxBuild2404.ps1, scripts/_Transport.psm1) Generic validator (Windows-only) (template/Validate-DeployState.ps1) Matrix workflow Linux leg picks Windows template
Composite action Inputs forwarded via env vars, step outputs work, success and failure upload paths (gitea/actions/local-ci-build/action.yml) — Action-level outputs: block is malformed (duplicate key, wrong nesting); hardcoded N:\ script path
Concurrency File mutex + lease design (scripts/Invoke-CIJob.ps1 ~L303-L460) Stale lock auto-cleanup 4-way burn-in never run
Credential handling Windows Credential Manager target BuildVMGuest; PAT via Credential Manager target GiteaPAT; Windows Mode 2 uses http.extraHeader — Linux Mode 2 embeds PAT in URL; Linux ExtraGuestEnv echoed in build-command log
Maintenance Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks (scripts/Watch-DiskSpace.ps1, scripts/Watch-RunnerHealth.ps1, scripts/Invoke-RetentionPolicy.ps1, scripts/Cleanup-OrphanedBuildVMs.ps1, scripts/Register-CIScheduledTasks.ps1) Stale vm-start.lock removal missing —
Backup scripts/Backup-CITemplate.ps1 functional — Default TemplatePath points at obsolete WinBuild directory
Host bootstrap Runner install, NSSM service, credential prompts (Setup-Host.ps1) — Creates obsolete Templates\WinBuild, missing State, keys, Cache\pip, current template dirs
Observability JSONL per-job events, Event Log, optional webhook Phase-duration summary Cross-job analysis tool
Tests Argument-parsing Pester suite over fake vmrun.cmd (4 files in tests/) — Orchestrator state machine, real-VM smoke tests, lock concurrency
Documentation docs/ARCHITECTURE.md, docs/CI-FLOW.md, docs/RUNBOOK.md, docs/BEST-PRACTICES.md, docs/WORKFLOW-AUTHORING.md Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names Consolidated template-refresh runbook

The README is genuinely ahead of the evidence. The accurate label is feature-complete homelab CI awaiting stabilization pass. Everything in this plan exists to close that gap.

Realistic Goals

This system should aim to be:

  • A reliable single-host homelab CI/CD that runs builds inside ephemeral, isolated VMware Workstation linked clones, triggered from a self-hosted Gitea.
  • Operable by one person, from one workstation, against private repositories that they trust.
  • Capable of clean recovery from host reboot, VMware crash, hung vmrun, and stale lock files without manual intervention beyond what is documented.
  • Capable of producing repeatable Windows and Linux builds with cached toolchains, with artifacts collected and published to Gitea.
  • Observable enough that after a failed build the operator can diagnose whether it was their build script, the runner, the host, or the VM.

This system should NOT aim to be:

  • A multi-host or multi-tenant CI. Federation is explicitly out of scope.
  • A platform for running untrusted third-party code. The guest VMs are hardened only enough to be useful, not enough to safely execute hostile builds.
  • A general-purpose container orchestrator. Containers are deliberately not in the picture (see Docker / Container Notes).
  • A drop-in replacement for paid CI services like GitHub Actions cloud, Azure DevOps, or CircleCI. The threat model, secret handling, and audit story are all homelab-grade.
  • A PowerShell 7 project. The PS 5.1 mandate in AGENTS.md is a deliberate constraint; migration is not on the roadmap.

Anything in this plan that drifts beyond these bounds is rejected. Both reviewers tested clean here; no Kubernetes, no Prometheus, no service mesh, no enterprise vault leaked into the consolidated plan.

Recommended Architecture

The current architecture is the right one. Confirm it as the target shape rather than redesign it. The recommendations below are scoped, additive fixes inside the existing architecture.

VM lifecycle approach: keep as-is. Linked clones from a BaseClean snapshot, lifecycle owned by scripts/Invoke-CIJob.ps1 in a try/finally, sub-scripts pure and parameter-driven. Add a Invoke-VmrunBounded wrapper into scripts/_Common.psm1 and use it for the three lifecycle calls (start, stop, deleteVM). Do not bound list, readVariable, or getGuestIPAddress — those are short and the wrapper overhead is wasted there.

Transport layer organization: keep explicit Windows/Linux branches. Do not collapse WinRM and SSH into a generic transport interface. Branches in scripts/Invoke-RemoteBuild.ps1 and scripts/Get-BuildArtifacts.ps1 are debuggable at 2 AM in a way that an abstraction is not. Add three small helpers in scripts/_Common.psm1 instead: Invoke-VmrunBounded (timeouts), ConvertTo-SafeShellSingleQuotedString (Linux quoting), and Write-CIRedactedCommand (secret-aware log formatter).

IP allocation strategy: keep the lock+lease design. The Opus suggestion of a static IP pool with cloud-init/netsh injection is a sound design, but it is a phase-3 redesign justified only if the burn-in fails. Run the burn-in first. The current design is: serialize clone+start+IP-detect through F:\CI\State\vm-start.lock, read guestinfo.ci-ip as the primary channel with getGuestIPAddress fallback, write a per-IP lease file. After fixing stale-lock cleanup and verifying under four-way load, this is sufficient for capacity: 4.

Credential management: keep Windows Credential Manager + host-side SSH key file. No vault, no rotation service. The fixes are: (1) verify the act_runner service account actually has access to the Credential Manager target; (2) confirm F:\CI\keys\ci_linux has ACLs restricting access to the runner identity and Administrators; (3) document a manual rotation procedure in docs/RUNBOOK.md.

Module organization: keep scripts/_Common.psm1 and scripts/_Transport.psm1 split. Do not rename or merge. Add the three helpers listed above to _Common.psm1. Keep _Transport.psm1 as the SSH-specific helper despite the generic name; rename is churn.

Logging/observability design: keep JSONL + Event Log + webhook. Do not move to SQLite. Add a 40-line Get-CIJobSummary.ps1 over the existing JSONL files for cross-job analysis. Add a phase-duration table printed at job end inside scripts/Invoke-CIJob.ps1.

Recommended Stack

Canonical stack, after the fixes in this plan:

  • Windows 11 Pro host, i9-10900X, 64 GB RAM, NVMe SSD.
  • Windows PowerShell 5.1 only. No PowerShell 7 dependency in any production script.
  • VMware Workstation Pro, vmrun.exe at C:\Program Files (x86)\VMware\VMware Workstation\.
  • Built-in Windows 11 OpenSSH client (ssh.exe, scp.exe) for Linux guests.
  • act_runner v1.0.2 installed as a Windows service via NSSM, capacity 4.
  • Gitea at http://10.10.20.11:3100 / https://gitea.emulab.it, PAT stored in Credential Manager target GiteaPAT.
  • Templates: WinBuild2025 (primary Windows), WinBuild2022 (secondary Windows), LinuxBuild2404 (Ubuntu 24.04 cloud image).
  • WinRM HTTPS over port 5986 with Basic authentication and self-signed certificate. AllowUnencrypted=false.
  • SSH user ci_build on Linux guests with passwordless sudo, key at F:\CI\keys\ci_linux.

Add to the stack:

  • Pester v5 (already present in tests/).
  • PSScriptAnalyzer (already used by the lint workflow).

Remove from the stack:

  • runner/Install-Runner.ps1. Marked deprecated; superseded by Setup-Host.ps1. Delete the file.
  • The obsolete F:\CI\Templates\WinBuild directory if it still exists on the host. Replaced by WinBuild2025 / WinBuild2022.

Do not add:

  • Docker / Podman / container runtimes. See Docker / Container Notes.
  • PowerShell 7.
  • Prometheus, Grafana, Loki, OpenTelemetry collectors, or any external metrics stack.
  • SQLite or any other database engine.
  • HashiCorp Vault, Azure Key Vault, or any external secret store.

Critical Problems

These must be fixed before the system is trustworthy for unattended daily use. Severity Critical means the issue actively breaks production today.

C1. Linux matrix leg of build-nsis.yml selects the Windows template

Description: gitea/workflows/build-nsis.yml runs a matrix over [windows, linux] and dispatches on runs-on: ${{ matrix.target }}-build, but does NOT pass guest-os or template-path to the composite action. The action defaults template-path to empty, so scripts/Invoke-CIJob.ps1 reads $env:GITEA_CI_TEMPLATE_PATH — the Windows path. The orchestrator never reads GITEA_CI_LINUX_TEMPLATE_PATH. Result: the Linux leg clones WinBuild2025 and tries to SSH into a Windows VM (or via Auto-detect, into a Windows VM with WinRM credentials), which fails or produces nonsense.

Location: gitea/workflows/build-nsis.yml L20-L40; gitea/actions/local-ci-build/action.yml L77-L83; scripts/Invoke-CIJob.ps1 L108.

Impact: The advertised Windows+Linux matrix workflow is broken for the Linux leg. The only evidence of "Linux works" is direct invocation of Invoke-CIJob.ps1 with explicit -TemplatePath. End-to-end through the workflow has never worked for Linux.

Fix: Either (a) update the composite action to select between GITEA_CI_TEMPLATE_PATH and GITEA_CI_LINUX_TEMPLATE_PATH based on guest-os (or runner label inference) and similarly for snapshot name; or (b) update gitea/workflows/build-nsis.yml to pass guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }} plus an explicit template-path. Option (a) is the better one because it makes the action self-sufficient for any caller. Add an INPUT_GUEST_OS_RESOLVED step that selects the template-path env var, then pass it as -TemplatePath. Also set BaseClean-Linux as the snapshot name on Linux.

Estimated effort: 1 hour including a manual smoke test of both legs.

C2. Linux ExtraGuestEnv secrets are logged in clear text

Description: scripts/Invoke-RemoteBuild.ps1 Linux branch builds $buildCmd = "${envPrefix}cd '$GuestLinuxWorkDir' && $BuildCommand" where $envPrefix contains export KEY='value'; ... with secret values inlined. The next line emits Write-Host "[Invoke-RemoteBuild] Running build: $buildCmd". act_runner captures stdout and Gitea masks secrets.* by string match only; multi-byte transforms, base64 encoding, or substring overlap defeat the mask. The documented intended use case (extra-guest-env-json: '{"SIGN_PASS":"${{ secrets.SIGN_PASS }}"}') makes this an active leak path.

Location: scripts/Invoke-RemoteBuild.ps1 Linux build assembly, around the Write-Host immediately preceding Invoke-SshCommand for the build step.

Impact: Any homelab workflow currently using extra-guest-env-json for secrets has those secrets in Gitea job logs and in the local F:\CI\Logs\<jobId>\invoke-ci.log transcript.

Fix: Replace the Write-Host "Running build: $buildCmd" line with Write-Host "[Invoke-RemoteBuild] Running build (env vars redacted): cd '$GuestLinuxWorkDir' && <build-command>". Never log $envPrefix or any string containing it. Do the same redaction in any other location where $buildCmd or $envPrefix is logged.

Estimated effort: 15 minutes.

Secondary Problems

These are High priority items. Fix in the same stabilization pass as the Criticals.

H1. capacity: 4 is unvalidated under real concurrency

Description: runner/config.yaml sets capacity: 4, but the IP-allocation mutex in scripts/Invoke-CIJob.ps1 has only been exercised by single-job e2e runs (per TODO.md §2.1). The design is plausibly correct, but DHCP behavior under four concurrent vmrun start calls, native-tool concurrency safety, and the 4-minute serialization of the IP-acquire phase are unmeasured. Both reviewers flagged this; severity is HIGH (not CRITICAL) because the system can be safely operated at lower capacity while this is unproven.

Location: runner/config.yaml L17; scripts/Invoke-CIJob.ps1 L303-L460.

Impact: The "production-ready" claim in README.md rests on a capacity number that has not been measured. A first real burst of four parallel builds may fail in ways the e2e runs cannot predict.

Fix: Write scripts/Test-CapacityBurnIn.ps1. PowerShell 5.1, uses Start-Job to launch four parallel Invoke-CIJob.ps1 invocations with unique JobIds against a small no-op build. Wait for all jobs, then assert: (1) all four VMs were destroyed; (2) all four jobs got distinct IPs; (3) F:\CI\State\vm-start.lock does not exist; (4) F:\CI\State\ip-leases\ is empty; (5) total wall time and per-phase JSONL durations are recorded. Run three times. If all three pass, document the result in docs/RUNBOOK.md and keep capacity: 4. If any fail, set capacity: 2 until investigated.

Estimated effort: Half-day to write, run, and document.

H2. Composite action outputs: block is malformed YAML

Description: gitea/actions/local-ci-build/action.yml has artifact-path and a duplicate artifact-name under the inputs: map instead of a top-level outputs: block. Step-level outputs still work (the Out-File $env:GITHUB_OUTPUT lines drive the upload step inside the same composite), so the live workflow does upload artifacts. But the duplicate artifact-name key violates YAML 1.2 (most parsers silently overwrite with the later value) and the missing outputs: block means downstream workflows cannot read steps.local-ci-build.outputs.artifact-path.

Location: gitea/actions/local-ci-build/action.yml L96 and L132-L141.

Impact: Latent. Today's single consumer is unaffected. The first new consumer that tries to chain on the artifact path will fail mysteriously.

Fix: Insert a top-level outputs: mapping (sibling to inputs: and runs:) containing artifact-path and artifact-name. Delete the second artifact-name and the artifact-path from the inputs: map. Keep the inner Out-File $env:GITHUB_OUTPUT lines unchanged. Note: GPT-5.5 originally rated this CRITICAL; Opus rated it LOW. The final-plan severity is HIGH because impact is latent but the YAML is clearly malformed.

Estimated effort: 20 minutes.

H3. Linux Mode 2 PAT injected into clone URL

Description: scripts/Invoke-RemoteBuild.ps1 Linux Mode 2 builds https://user:pat@host/repo.git and exports it as GIT_CLONE_URL in the SSH command. The PAT becomes visible in /proc/<pid>/environ for the duration of git clone, in the host-side ssh.exe argv, and in any PowerShell error text that captures $envCloneCmd. The Windows path uses git -c http.extraHeader=Authorization: Basic <b64>, which is strictly safer.

Location: scripts/Invoke-RemoteBuild.ps1 Linux Mode 2 clone branch, around the GIT_CLONE_URL export.

Impact: PAT visibility on the guest filesystem and host process list. Bounded but real.

Fix: Mirror the Windows pattern. Base64-encode user:pat, build a single git -c credential.helper= -c http.extraHeader=Authorization: Basic <b64> clone <non-auth-url> <dest> command, invoke via Invoke-SshCommand. Drop the URL rewrite, drop the GIT_CLONE_URL export, drop the unset. Note: GPT-5.5 originally proposed an askpass helper with chmod 600 temp file; Opus proposed the http.extraHeader mirror. The simpler Opus pattern is adopted.

Estimated effort: 30 minutes including a clone test against a private repo.

H4. vmrun lifecycle calls have no timeout in production scripts

Description: scripts/_Common.psm1's Invoke-Vmrun checks $LASTEXITCODE but does not bound execution. scripts/Invoke-CIJob.ps1 and scripts/Remove-BuildVM.ps1 inherit this. A hung vmrun start or deleteVM waits on vmware-vmx.exe indefinitely, consuming the runner slot until act_runner's 2-hour job timeout fires. The finally cleanup cannot run while the native process is blocked.

Location: scripts/_Common.psm1 Invoke-Vmrun; callers in scripts/Invoke-CIJob.ps1 and scripts/Remove-BuildVM.ps1.

Impact: A single stuck VMware operation can pin one of the four runner slots for two hours.

Fix: Add Invoke-VmrunBounded to scripts/_Common.psm1. Use Start-Process -PassThru + Wait-Process -Timeout + Stop-Process on timeout. Return [pscustomobject]@{ExitCode; Output; TimedOut; ElapsedSeconds}. Replace Invoke-Vmrun calls for start, stop, and deleteVM only — keep the unbounded version for list, readVariable, getGuestIPAddress. Recommended timeouts: 180s for start, 60s for stop, 90s for deleteVM.

Estimated effort: 1 hour including a Pester test.

H5. Stale vm-start.lock not cleaned automatically

Description: The IP-allocation lock file at F:\CI\State\vm-start.lock is released correctly on normal exit. On host crash mid-lock, the file remains. On reboot the OS-level lock is gone (kernel released it), but the file's mere existence does not cause failure; the issue is the 10-minute retry loop and operator confusion. Neither scripts/Cleanup-OrphanedBuildVMs.ps1 nor scripts/Invoke-RetentionPolicy.ps1 removes it.

Location: scripts/Invoke-RetentionPolicy.ps1 and scripts/Cleanup-OrphanedBuildVMs.ps1.

Impact: First job after a crash gives a confusing message; manual Remove-Item cures it. Low operational severity, but high friction.

Fix: In both maintenance scripts, add a block that removes F:\CI\State\vm-start.lock if its LastWriteTime is older than 30 minutes. Do not write metadata into the lock file; the file's mere existence is sufficient signal. Log to console and JSONL.

Estimated effort: 15 minutes.

H6. Setup-Host directory list is stale

Description: Setup-Host.ps1 L118-L126 creates Templates\WinBuild and a handful of cache directories, but does NOT create Templates\WinBuild2025, Templates\WinBuild2022, Templates\LinuxBuild2404, State, State\ip-leases, keys, or Cache\pip. The orchestrator depends on F:\CI\State\ and F:\CI\keys\. A fresh host bootstrap leaves the system non-functional.

Location: Setup-Host.ps1 L118-L126.

Impact: Disaster recovery is broken. Rebuilding the host from scratch using the documented procedure does not produce a working system.

Fix: Update the directory array to: BuildVMs, Artifacts, Logs, Templates\WinBuild2025, Templates\WinBuild2022, Templates\LinuxBuild2404, State, State\ip-leases, keys, act_runner\logs, Cache\NuGet, Cache\pip, RunnerWork, ISO. Remove Templates\WinBuild. Add a comment block referencing AGENTS.md §Struttura directory host.

Estimated effort: 10 minutes.

H7. Backup-CITemplate default path is wrong

Description: scripts/Backup-CITemplate.ps1 L48 defaults TemplatePath to 'F:\CI\Templates\WinBuild'. That directory no longer exists. The default-argument backup silently fails or backs up an empty directory, then the operator refreshes the actual WinBuild2025 template, with no rollback.

Location: scripts/Backup-CITemplate.ps1 L48 and the help block.

Impact: An operator running .\Backup-CITemplate.ps1 before a template refresh believes they have a backup. They do not.

Fix: Change the default to 'F:\CI\Templates\WinBuild2025'. Add a -AllTemplates switch that iterates F:\CI\Templates\* and backs up each (with subfolder naming preserved in the backup destination). Update docs/HOST-SETUP.md and docs/RUNBOOK.md to reflect the new default. Update the help block.

Estimated effort: 30 minutes.

H8. act_runner service account vs SSH alias mismatch

Description: Setup-Host.ps1 L294-L306 writes the gitea-ci SSH alias to $env:USERPROFILE\.ssh\config — the interactive user's profile. The NSSM install at L264-L273 does not set an explicit service account, which on Windows commonly means LocalSystem. LocalSystem's home is C:\Windows\System32\config\systemprofile\, not the user profile. gitea/workflows/build-nsis.yml uses repo-url: 'ssh://gitea-ci/Simone/...'. If the service runs as LocalSystem, the alias resolution fails and every Mode 1 (host-side clone) attempt errors out.

Location: Setup-Host.ps1 L264-L273 (NSSM install), L294-L306 (SSH config write); gitea/workflows/build-nsis.yml L35.

Impact: Potentially blocks all Mode 1 host-side clone builds. Severity is HIGH-PENDING-VERIFICATION; first action is verification.

Fix: Step 1, verify: run Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName on the host and document the answer in docs/RUNBOOK.md. Step 2a, if StartName is LocalSystem: copy the SSH config and known-hosts to C:\Windows\System32\config\systemprofile\.ssh\ with proper ACLs. Step 2b alternative: create a dedicated ci_runner Windows user, reinstall NSSM with obj=.\ci_runner, store the SSH config under that user's profile. Step 3: add the verification check to template/Validate-DeployState.ps1 or a new Validate-HostState.ps1. Note: GPT-5.5 did not catch this. Opus did. Severity HIGH stands.

Estimated effort: 1 hour for verification + fix.

H9. ci-report-ip.service not asserted in deploy-state validator

Description: template/Validate-DeployState.ps1 is Windows-only: it checks WinRM, Defender, Firewall, UAC, etc. It does not check whether the Linux template's ci-report-ip.service (the systemd unit writing guestinfo.ci-ip) is enabled. The unit IS validated inside template/Install-CIToolchain-Linux2404.sh L203-L221 and template/Prepare-LinuxBuild2404.ps1 L361-L362, but not at the level of "is this template ready to be promoted to BaseClean-Linux." If a future refresh forgets to re-enable it, every Linux build silently falls back to the 120-second getGuestIPAddress timeout.

Location: template/Validate-DeployState.ps1.

Impact: Silent degradation of Linux build startup time after a faulty template refresh.

Fix: Add a Linux branch to Validate-DeployState.ps1 (or create a Validate-LinuxDeployState.ps1) that opens an SSH session and runs systemctl is-enabled ci-report-ip.service and test -x /usr/local/bin/ci-report-ip.sh. Fail with a clear error if either is missing.

Estimated effort: 45 minutes.

H10. No automated end-to-end smoke test

Description: The Pester suite in tests/ exercises argument parsing only via a fake vmrun.cmd. The "production-ready" claim rests on manual e2e runs (e2e-008, e2e-009, §3.3). There is no scripted, repeatable test that exercises real VM lifecycle and exits non-zero on failure. Both reviewers flagged this; severity is HIGH because regression risk without it is high after every change.

Location: tests/, gitea/workflows/.

Impact: Any change to orchestration scripts has no automated guard. Regressions are caught only by chance during the next manual run.

Fix: Create gitea/workflows/self-test.yml (workflow_dispatch trigger only). One job that calls the composite action against a minimal Write-Host 'ok' > out.txt build, asserts artifact upload, JSONL phase events, and clean teardown. Have it succeed-or-fail for Windows and a second job for Linux. Add scripts/Test-Smoke.ps1 that the workflow invokes; the script also runs locally. Note: this replaces the much larger Measure-CIBenchmark.ps1 wiring suggestion; the smoke test is sufficient and cheaper.

Estimated effort: Half-day.

H11. Composite action hardcodes the script root path

Description: gitea/actions/local-ci-build/action.yml L148 sets $ciScriptsDir = 'N:\Code\Workspace\Local-CI-CD-System\scripts'. The action is presented as reusable, but moving the repo breaks it. Opus caught this; GPT-5.5 missed it (it caught path drift in setup/backup only).

Location: gitea/actions/local-ci-build/action.yml L148.

Impact: Action is not actually reusable. Any second consumer or repo relocation breaks builds.

Fix: Replace with a runner env var. Add GITEA_CI_SCRIPT_ROOT: "N:\\Code\\Workspace\\Local-CI-CD-System\\scripts" to runner/config.yaml envs:. In the action, use $ciScriptsDir = $env:GITEA_CI_SCRIPT_ROOT. Do not use ${{ github.action_path }} — the action repo is not necessarily cloned into the consumer's workspace in this Gitea setup.

Estimated effort: 20 minutes.

H12. Measure-CIBenchmark uses a different IP discovery path than production

Description: scripts/Measure-CIBenchmark.ps1 L129-L146 polls Invoke-Vmrun -Operation getGuestIPAddress. scripts/Invoke-CIJob.ps1 L348-L389 primarily reads guestVar 'ci-ip' and uses getGuestIPAddress as fallback. The benchmark therefore measures a different code path than what production runs, and its phase timings cannot be used for capacity planning.

Location: scripts/Measure-CIBenchmark.ps1 L129-L146.

Impact: Benchmark data is misleading. Decisions to keep or change capacity: 4 based on it would be based on wrong measurements.

Fix: Extract IP-acquire into a function Get-GuestIPAddress in scripts/_Common.psm1 that takes the VMX path and credentials and implements the guestinfo.ci-ip-then-getGuestIPAddress flow used in production. Call it from both Invoke-CIJob.ps1 and Measure-CIBenchmark.ps1.

Estimated effort: 1 hour.

Complete Priority Checklist

Every item is concrete, references a file, and ends with an effort estimate.

Critical [ ]

High [ ]

Medium [ ]

Low [ ]

Nice to have [ ]

Detailed Operational Checklist

Execute in this order. Each task lists a prerequisite, an action, and a verification step.

Sprint 1: Validation

Goal: prove or downgrade unvalidated claims before any code changes go in.

Task 1.1 — Verify act_runner service account

  • Prerequisite: host PowerShell session as Administrator.
  • Action: Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName,State.
  • Verification: record StartName in docs/RUNBOOK.md under "Runner service identity". If LocalSystem, schedule H8 fix immediately.

Task 1.2 — Inventory current state-file presence

  • Prerequisite: nothing.
  • Action: Test-Path F:\CI\State, F:\CI\State\ip-leases, F:\CI\keys, F:\CI\Cache\pip, F:\CI\Templates\WinBuild2025, F:\CI\Templates\LinuxBuild2404.
  • Verification: each must return True. If any are missing, create manually for now and defer the Setup-Host.ps1 fix to Sprint 2.

Task 1.3 — Run 4-way concurrency burn-in (H1)

  • Prerequisite: write scripts/Test-CapacityBurnIn.ps1 first; WinBuild2025 and LinuxBuild2404 templates powered off; ≥ 100 GB free on F:.
  • Action: invoke the burn-in three times against a no-op build (e.g. Write-Host 'ok'); record per-run JSONL outputs.
  • Verification: zero IP collisions across runs, zero leaked VMs, zero leaked lock files, all jobs succeeded. If any fail, set runner/config.yaml capacity: 2 and create a follow-up investigation task.

Task 1.4 — Manually invoke Linux matrix leg to confirm C1 bug

  • Prerequisite: a test branch with a tag that triggers gitea/workflows/build-nsis.yml.
  • Action: push a tag, observe both legs.
  • Verification: confirm Linux leg fails or builds against Windows template. Document the failure mode in the C1 task notes.

Sprint 2: Critical Fixes

Goal: clear both Criticals plus the highest-impact Highs that touch production paths.

Task 2.1 — Fix Linux matrix template selection (C1)

  • Prerequisite: Sprint 1 task 1.4 done.
  • Action: edit gitea/actions/local-ci-build/action.yml so the PowerShell step resolves INPUT_GUEST_OS_RESOLVED from INPUT_GUEST_OS (with Auto honoring runs-on heuristic via $env:RUNNER_LABELS if Gitea exposes it; otherwise defaulting Auto to Windows) and selects between GITEA_CI_TEMPLATE_PATH and GITEA_CI_LINUX_TEMPLATE_PATH. Update gitea/workflows/build-nsis.yml matrix to pass guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }} as a belt-and-suspenders measure.
  • Verification: re-run the matrix workflow. Both legs succeed against the correct templates.

Task 2.2 — Redact Linux ExtraGuestEnv logging (C2)

  • Prerequisite: nothing.
  • Action: in scripts/Invoke-RemoteBuild.ps1 Linux branch, replace the build-command echo with the redacted form. Grep the entire file for any other place $envPrefix or $buildCmd is logged; fix all.
  • Verification: run a Linux job with a fake secret extra-guest-env-json='{"DUMMY":"shibboleth"}' and confirm the string shibboleth does not appear in F:\CI\Logs\<jobId>\invoke-ci.log or in act_runner output.

Task 2.3 — Fix composite action outputs YAML (H2)

  • Prerequisite: nothing.
  • Action: edit gitea/actions/local-ci-build/action.yml per H2 fix.
  • Verification: run gitea/workflows/build-nsis.yml; confirm artifact upload still works. Optionally test downstream consumption by writing a temporary workflow step that reads steps.local-ci-build.outputs.artifact-path.

Task 2.4 — Linux Mode 2 PAT to http.extraHeader (H3)

  • Prerequisite: a private Gitea repo with the PAT scope to clone.
  • Action: edit scripts/Invoke-RemoteBuild.ps1 Linux clone branch.
  • Verification: run a Linux Mode 2 build; confirm git clone succeeds; verify with ps -ef on the guest during clone that the PAT is not in the argv.

Task 2.5 — Fix Setup-Host directory list (H6)

  • Prerequisite: Sprint 1 task 1.2 captured what is missing.
  • Action: edit Setup-Host.ps1 L118 directory array.
  • Verification: on a test VM (not the production host), run Setup-Host.ps1 and confirm every directory in AGENTS.md §Struttura is created.

Task 2.6 — Fix Backup-CITemplate default (H7)

  • Prerequisite: nothing.
  • Action: edit scripts/Backup-CITemplate.ps1 default and help; add -AllTemplates switch.
  • Verification: run .\Backup-CITemplate.ps1 -DryRun (add flag if it does not exist) and confirm the resolved source path is F:\CI\Templates\WinBuild2025.

Sprint 3: High Priority Fixes

Goal: stabilize the rest of the High-tier items.

Task 3.1 — Add Invoke-VmrunBounded (H4)

  • Prerequisite: review existing bounded patterns in deploy scripts.
  • Action: add the function to scripts/_Common.psm1; wire start/stop/deleteVM callers in scripts/Invoke-CIJob.ps1 and scripts/Remove-BuildVM.ps1.
  • Verification: write a Pester test that injects a fake vmrun.cmd which sleeps 600s; assert the wrapper times out at the configured deadline.

Task 3.2 — Stale lock cleanup (H5)

Task 3.3 — Fix or relocate SSH alias for service account (H8)

  • Prerequisite: Sprint 1 task 1.1 result.
  • Action: per H8, either copy .ssh/ to systemprofile or change the service account.
  • Verification: psexec -s -i ssh -T gitea-ci (or equivalent) from the service account; expect a successful Gitea response.

Task 3.4 — Linux deploy validator check for ci-report-ip (H9)

  • Prerequisite: SSH key and IP of the LinuxBuild2404 template (boot it, capture, shut down).
  • Action: add the Linux branch to template/Validate-DeployState.ps1.
  • Verification: against the live template, the script reports the service as enabled.

Task 3.5 — Add smoke-test workflow (H10)

  • Prerequisite: H2 and C1 fixed (otherwise the workflow's own assertions fail).
  • Action: create gitea/workflows/self-test.yml and scripts/Test-Smoke.ps1.
  • Verification: trigger via workflow_dispatch and observe both legs green.

Task 3.6 — Replace hardcoded N:\ path (H11)

Task 3.7 — Benchmark IP path alignment (H12)

  • Prerequisite: H4 done (shared _Common.psm1 patterns ironed out).
  • Action: extract Get-GuestIPAddress into scripts/_Common.psm1; update both callers.
  • Verification: run Measure-CIBenchmark.ps1 -Iterations 1 for Windows and Linux; confirm phase timings match those recorded in production JSONL within ±10%.

Sprint 4: Operational Hardening

Goal: harden recovery, secret handling, and recoverability.

Task 4.1 — Pre-clone disk gate, JobId/ExtraGuestEnv validation, redacted command logger (Medium items grouped) — edit scripts/Invoke-CIJob.ps1 and scripts/Invoke-RemoteBuild.ps1; verify with deliberately bad inputs.

Task 4.2 — Failure-path diagnostics — add a Get-GuestDiagnostics helper invoked from the catch block of scripts/Invoke-CIJob.ps1. Skip on Phase 1/2 failures (no VM yet). Best-effort SSH/WinRM, no throw on failure.

Task 4.3 — Wire UseSharedCache — surface in composite action and orchestrator; confirm with a Python build that pip cache is reused.

Task 4.4 — Tighten SSH host keys for CI jobs — accept-new + F:\CI\State\known_hosts for CI; keep permissive mode for template provisioning (host keys are unknown there).

Task 4.5 — Ubuntu VMDK SHA256 verification — add to template/Deploy-LinuxBuild2404.ps1.

Task 4.6 — Consolidated template-refresh runbook — edit docs/RUNBOOK.md. One section covering: stop runner, back up template (-AllTemplates), boot template, KMS reactivation via slmgr /ato (Windows), run Prepare-WinBuild2025.ps1 or Prepare-LinuxBuild2404.ps1, shutdown, snapshot BaseClean_yyyyMMdd, run Validate-DeployState.ps1 (Windows and Linux branches), run smoke workflow, then update GITEA_CI_SNAPSHOT_NAME in runner/config.yaml, keep prior snapshot 7 days, rollback procedure.

Sprint 5: Quality and Optimization

Goal: polish.

Task 5.1 — Delete dead code — remove runner/Install-Runner.ps1; align Setup-Host help text.

Task 5.2 — Webhook emoji removal, Event Log naming, docblock corrections — Low-tier batch.

Task 5.3 — Update test plan — docs/TEST-PLAN-v1.3-to-HEAD.md — PS5.1 syntax, correct parameter names, current task and event log names.

Task 5.4 — Add Get-CIJobSummary.ps1 — nice-to-have summary tool.

Task 5.5 — Phase-duration summary in orchestrator — added at end of scripts/Invoke-CIJob.ps1 before final cleanup.

Task 5.6 — Begin SHA256 pinning — start with Git, 7-Zip, Python, .NET. Note in docs/BEST-PRACTICES.md that this is a homelab-acceptable, partial-coverage pin.

Recommended Refactors

Only refactors that genuinely simplify or fix something. No churn.

  1. Extract Get-GuestIPAddress into scripts/_Common.psm1. Removes the dual-implementation drift between Invoke-CIJob.ps1 and Measure-CIBenchmark.ps1. Single source of truth for the guestinfo.ci-ip-then-fallback flow. Justification: real bug avoidance, not aesthetic.

  2. Extract Invoke-VmrunBounded into scripts/_Common.psm1. Closes the H4 timeout gap and consolidates the pattern that already exists in some deploy scripts.

  3. Add Write-CIRedactedCommand and ConvertTo-SafeShellSingleQuotedString to scripts/_Common.psm1. The first standardizes secret-aware command logging across Windows and Linux paths. The second consolidates the -replace "'", "'\''" idiom used in scripts/Invoke-RemoteBuild.ps1. Both are 5-line functions.

  4. DO NOT extract a generic transport interface. Opus suggested Invoke-GuestCommand/Copy-GuestItem/Wait-GuestTransportReady. Both reviewers ultimately agreed this is the wrong move for a two-transport homelab. Keep explicit Windows/Linux branches in scripts/Invoke-RemoteBuild.ps1 and scripts/Get-BuildArtifacts.ps1.

  5. DO NOT introduce F:\CI\State\config.json as a central config. AGENTS.md and runner/config.yaml are already the two sources of truth. Adding a third creates drift, not reduces it. The remediation is to align the existing two, not invent a third.

  6. DO NOT migrate to PowerShell 7. The PS 5.1 constraint is deliberate, encoded in AGENTS.md, and consistently respected across the codebase. A migration is a multi-day churn that buys little.

  7. DO NOT replace JSONL with SQLite. The JSONL output is forward-compatible and trivially greppable. Get-CIJobSummary.ps1 over JSONL covers the cross-job analysis need without adding a database engine.

  8. DO NOT pursue a static IP pool unless the burn-in fails. The current lock+lease design after the stale-lock fix is sufficient for capacity: 4. Static IPs cost more than they save at this scale.

Security Improvements

Must (do these in Sprint 2 or 3):

  • Redact Linux ExtraGuestEnv secret logging (C2).
  • Replace Linux Mode 2 PAT URL with http.extraHeader (H3).
  • Verify and fix the act_runner SSH alias service-account issue (H8).
  • Add JobId pattern validation (Medium).
  • Add ExtraGuestEnv key pattern validation (Medium).
  • Run SSH StrictHostKeyChecking=accept-new with UserKnownHostsFile=F:\CI\State\known_hosts for CI jobs (Medium).

Could (Nice-to-have):

  • Begin SHA256 pinning of Tier-1 toolchain installers.
  • Verify ACLs on F:\CI\keys\ci_linux via a new Validate-HostState.ps1.
  • Add a one-paragraph note in docs/WORKFLOW-AUTHORING.md on Gitea log-masking limits (substring-only masking; base64 transforms defeat it).

Will NOT do:

  • Full PKI for WinRM. The self-signed lab certificate plus VMnet8 isolation is the right boundary.
  • External secret vault integration.
  • Enable Defender / Firewall / UAC inside the build VM. The threat model accepts this for trusted private code; documented in docs/BEST-PRACTICES.md.
  • Per-build credential rotation.
  • Artifact signing.

CI/CD Improvements

Concrete improvements to workflows and the composite action.

  1. Make the composite action self-sufficient for guest OS selection (C1 fix). After this, gitea/workflows/build-nsis.yml does not need to pass guest-os, template-path, or snapshot-name for the standard Windows/Linux matrix; the action infers from the runner label or accepts the explicit input.

  2. Add a top-level outputs: block so downstream consumers can chain (H2 fix).

  3. Add use-shared-cache input plumbed to Invoke-CIJob.ps1 -UseSharedCache (Medium fix). Forward to Invoke-RemoteBuild.ps1 so NuGet/pip caches actually work end-to-end.

  4. Add an on-demand gitea/workflows/self-test.yml (workflow_dispatch trigger only). One Windows job, one Linux job, both calling the composite action against a trivial build (Write-Host 'ok' > out.txt). Asserts artifact upload, JSONL events, clean teardown. Use as the regression gate for any action change.

  5. Add a Pester gitea/workflows/tests.yml that runs the existing tests on every push. PowerShell 5.1, Invoke-Pester -CI. This is the current missing "CI for the CI" piece.

  6. Keep the lint workflow gitea/workflows/lint.yml as is.

  7. Do NOT add scheduled builds, dependency-update workflows, or release workflows. Out of scope.

Docker / Container Notes

This project deliberately does not use Docker, Podman, or any container runtime. The decision is correct and should be preserved.

Rationale:

  • Windows builds require VS Build Tools, MSBuild, the full .NET SDK, and Windows-specific signing chains. Running those in Windows containers requires Windows Server host images, breaks isolation guarantees the project gets cheaply from VMware, and adds licensing complexity.
  • The project's isolation boundary is a VM, not a container. VMs offer strictly stronger isolation than containers (kernel separation, hardware-virtualized network stack, separate filesystem) at a cost the system can afford.
  • act_runner's container.enabled: false is the right setting for a system that uses real VMs.
  • The performance overhead of "spin up a VM" vs "spin up a container" is real (a clone takes 5-30s, a container ~1s), but the orchestrator amortizes this across a 2-hour build budget. The cost is acceptable.

When might containers enter the picture? Likely never for this scope. If the project ever needs to build small Linux-only services with no Windows leg, a side-by-side linux-build-container runner label could be added later. That is a phase-3+ idea, not a current task.

Monitoring and Logging Improvements

Concrete additions only. No metric server.

JSONL schema additions inside F:\CI\Logs\<jobId>\invoke-ci.jsonl (emitted by scripts/Invoke-CIJob.ps1):

  • Per-phase duration_ms calculated at phase end.
  • free_disk_gb_before and free_disk_gb_after for the job.
  • ip_detection_method field with values guestinfo or getGuestIPAddress.
  • vm_start_lock_wait_ms recording how long the orchestrator waited on vm-start.lock.
  • clone_dir_size_mb measured at teardown.

Event Log additions:

  • Standardize on CI-DiskSpaceAlert (EventId 1001), CI-RunnerHealth (1002 stopped / 1003 restarted / 1004 throttled).
  • Add CI-JobFailure source for failed orchestrator runs (EventId 2001) writing JobId and failed phase.

Watcher script additions:

  • scripts/Get-CIJobSummary.ps1 (nice-to-have): reads last N days of JSONL files, prints a table of JobId, started, elapsed, phase 1 duration, phase 5 duration, status.
  • scripts/Watch-RunnerHealth.ps1 (Low): respect F:\CI\State\runner-maintenance.flag to suppress restart attempts during template refresh.

Webhook payload changes:

  • Remove emoji prefixes. Use [WARN], [ERROR], [INFO] plain text.

Do NOT add:

  • Prometheus exporters.
  • Grafana dashboards.
  • OpenTelemetry collectors.
  • SQLite or any other database sink.
  • Discord/Slack rich-embed cards.

Backup and Recovery Improvements

The current backup story is roughly: scripts/Backup-CITemplate.ps1 (currently pointing at a stale path). Improvements:

  1. Fix the default path and add -AllTemplates switch (H7).
  2. Add to docs/RUNBOOK.md the consolidated template-refresh runbook (Sprint 4 task 4.6).
  3. Register a weekly scheduled task that runs .\Backup-CITemplate.ps1 -AllTemplates -KeepLast 3 and writes a log to F:\CI\Logs\backup\<date>.log. Update scripts/Register-CIScheduledTasks.ps1 to add this fifth scheduled task.
  4. Document recovery from a destroyed template: restore the latest .7z from the backup destination, validate snapshots exist via vmrun listSnapshots, run Validate-DeployState.ps1 (Windows + Linux), run the smoke workflow. If the smoke fails, restore the previous backup. Step-by-step in docs/RUNBOOK.md.
  5. Document the simple recovery path for a stale vm-start.lock and stale IP leases in docs/RUNBOOK.md — even though Sprint 3 task 3.2 makes this automatic, the operator-facing docs should still say "if you ever need to do this manually, the commands are X."

Do NOT add:

  • VM snapshot backup to remote storage (out of scope; 7z files on a second drive is sufficient).
  • Restore-time automated diff against a known-good template (over-engineered).

Runner and Build Optimization

Concrete, measurable.

  1. Wire use-shared-cache through the composite action and orchestrator (Medium item). Expected impact: ~15-30% faster Python and .NET builds that pull packages on every run, by avoiding the pip download and nuget restore over the WAN. Measurement: compare phase 5 duration JSONL before/after on a representative NSIS-plugin build.

  2. After burn-in, decide on capacity. If the burn-in shows that capacity 4 is solid, keep it. If it shows that capacity 2 is the safe ceiling under typical host CPU contention, set runner/config.yaml capacity: 2. Expected impact: avoidance of unreproducible build failures during peaks.

  3. Defender exclusion for F:\CI\BuildVMs\ on the host (Low). Cloning and start are I/O heavy on the parent VMDK chain; Defender scanning slows this. Use Set-MpPreference -ExclusionPath 'F:\CI\BuildVMs'. Expected impact: ~10% faster clone-to-IP times. Document the trade-off in docs/BEST-PRACTICES.md: malicious code reaching the host would persist faster, but the threat model already accepts trusted-builds-only.

  4. Linked-clone health monitoring (Nice-to-have): add per-job delta size to JSONL. If the average delta grows over months without a template refresh, that is a signal something is writing into the template chain. Not urgent; nice diagnostic.

  5. Do NOT pursue a warm clone pool. Both reviewers agreed: the savings (5-30s per job) do not justify the state-machine complexity for a 2-hour build budget.

  6. Do NOT pursue VMX CPU/RAM tuning per job unless the burn-in shows host saturation. Wait for evidence.

Repository Conventions

These are the codified rules. Where ambiguous, write them into AGENTS.md and docs/BEST-PRACTICES.md.

Style and linting (already largely in place):

  • PowerShell 5.1 only. Every script begins with #Requires -Version 5.1, Set-StrictMode -Version Latest, $ErrorActionPreference = 'Stop'.
  • 4-space indentation, no tabs.
  • Verb-Noun cmdlet naming. Get-, Set-, New-, Remove-, Invoke-, Test-, Wait-, Watch-.
  • Write-Host is acceptable and preferred over Write-Output in CI scripts (act_runner captures Write-Host cleanly; Write-Output pollutes function return values). Codified in PSScriptAnalyzerSettings.psd1.
  • No emoji in code, logs, webhook payloads, docs, or commit messages. Plain text markers like [WARN], [OK], [x], [ ].
  • No PS 7+ syntax: no ??, no ??=, no ternary, no &&/|| in PowerShell context, no ForEach-Object -Parallel, no Test-Json.

File organization:

  • Production scripts under scripts/. Template provisioning under template/. Tests under tests/. Documentation under docs/. Plans and reviews under plans/.
  • Module files _Common.psm1 and _Transport.psm1 stay in scripts/. Do not split further.
  • One script, one verb-noun purpose. Composite actions live under gitea/actions/<name>/action.yml.

Commit messages:

  • Imperative mood, present tense ("Add bounded vmrun wrapper", not "Added" or "Adding").
  • Subject line ≤72 characters. Body explains why, not what.
  • Reference the master plan checklist item by short title when relevant: e.g. Fix C1: Linux matrix template selection.

Branching:

  • main is the only long-lived branch.
  • Use short-lived feature branches fix/c1-linux-matrix, fix/h2-action-outputs, etc.
  • Squash merge or rebase; do not preserve micro-commits.

Testing requirements before merge:

  • After H10 is implemented: every PR runs the lint and Pester workflows. After the smoke-test workflow exists, run it for any change touching orchestration scripts.

Documentation updates:

Documentation Status

Per-document assessment.

Document Status Action
AGENTS.md Good. Reflects current paths, PS 5.1 rules, error patterns. Keep updated as new gotchas are discovered.
README.md Stale. Claims production-ready before stabilization pass completes. Replace "production-ready" with "feature-complete; stabilization pass in progress" until all Critical and High items are checked.
docs/ARCHITECTURE.md Mostly accurate; minor diagram rendering issue. Reformat the ASCII diagram box. Reflect H4 bounded-vmrun pattern in the lifecycle section.
docs/CI-FLOW.md Stale on partial-artifacts-on-failure. Still mentions getGuestIPAddress as primary IP detection. Update IP-detection section to reflect guestinfo.ci-ip primary. Reconcile failure-scenario table with whether H-level diagnostics are implemented.
docs/RUNBOOK.md Useful but incomplete. Add: consolidated template-refresh procedure; capacity burn-in results; recovery from stale lock; runner service-account identity.
docs/BEST-PRACTICES.md Good. Old WinBuild path references need fixing. Update all WinBuild to WinBuild2025. Add HGFS shared-folder cache poisoning note.
docs/WORKFLOW-AUTHORING.md Good. Add one-paragraph note on Gitea secret-mask substring limitation.
docs/HOST-SETUP.md Mostly accurate. Update directory list to match new Setup-Host.ps1. Add ACL guidance for F:\CI\keys\ci_linux.
docs/WINDOWS-TEMPLATE-SETUP.md Good. Cross-reference the consolidated template-refresh runbook in docs/RUNBOOK.md.
docs/LINUX-TEMPLATE-SETUP.md Good. Add Validate-DeployState.ps1 Linux branch usage.
docs/OPTIMIZATION.md Mostly accurate. Note Defender exclusion option and shared-cache wiring after H-Medium fixes.
docs/TEST-PLAN-v1.3-to-HEAD.md Stale. PS 7 syntax, wrong parameter names, wrong task/event names. Sprint 5 task 5.3: PS 5.1 syntax, -RepoUrl, current Event Log sources.
docs/Setup-GiteaSSH.md Untouched by reviewers; assume accurate. Verify against fixed service-account path from H8.
TODO.md Acting as a CHANGELOG. Sprint 5 task: move "done" sections into docs/CHANGELOG.md; keep TODO for genuinely outstanding work.
docs/archived/2026-05-10/ Historical. Leave as-is.

Final Completion Plan

Execute in this exact order. Each phase has a single success criterion that gates the next.

Phase 0 — Read this plan and commit it. Already done by the act of generating it.

Phase 1 — Validation (Sprint 1). Verify the act_runner service account, confirm the directory inventory, run the 4-way concurrency burn-in, manually trigger the broken Linux matrix to confirm C1. Output: a section appended to docs/RUNBOOK.md titled "Validation Results 2026-05-XX" documenting findings. Gate: burn-in passed three times (keep capacity: 4) or was set to 2 (with reasoning recorded).

Phase 2 — Critical Fixes (Sprint 2). Fix C1 and C2, plus H2/H3/H6/H7. Manually trigger gitea/workflows/build-nsis.yml for both matrix legs after each fix. Gate: both Critical items closed and [x] in this plan's Critical section; both legs green in the live matrix workflow.

Phase 3 — High Priority Fixes (Sprint 3). Complete the rest of the High items: H4 (bounded vmrun), H5 (stale lock), H8 (SSH alias fix), H9 (Linux validator), H10 (smoke-test workflow), H11 (script-root env var), H12 (benchmark alignment). Gate: smoke-test workflow green; all High items [x].

Phase 4 — Operational Hardening (Sprint 4). All Medium items including consolidated template-refresh runbook. Gate: documented end-to-end template refresh executed successfully against a non-production template clone.

Phase 5 — Polish (Sprint 5). Low items, nice-to-have summary tool, documentation alignment, TODO → CHANGELOG migration. Gate: README claim updated from "feature-complete; stabilization pass in progress" back to "production-ready for homelab".

Phase 6 — Tag v1.0. Create an annotated tag v1.0.0 on main once Phase 5 gate passes. Write a docs/CHANGELOG.md v1.0 entry summarizing the journey.

Phase 7 — Operate. Use the system. Track new findings in TODO.md. Run the burn-in any time act_runner capacity, host hardware, or template chains change materially.

When all Critical and High items are checked, this system is genuinely production-ready for homelab use.

Reference in New Issue View Git Blame Copy Permalink