Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1dbeec0f8ea34684deaacc854395dc97f9dce997
local-ci-cd-system/plans/final-master-plan.md
T
Simone 6cd6bff882 nice-to-have: implement all 6 tier items 
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
  manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
  artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
  override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
  to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
  GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
  7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
  measured post-clone; added to result object and summary Format-Table
2026-05-13 10:38:49 +02:00

733 lines
78 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Final Master Plan — Local CI/CD System
Date: 2026-05-11
Last updated: 2026-05-13
Status: Sprint 1 validation complete; Sprint 2 Critical + High code fixes applied (2026-05-12). Sprint 3 verification pass (2026-05-13): H9/H10/H12 confirmed already implemented; Medium items disk-gate/JobId-validation/phase-duration/ExtraGuestEnv-key-validation/WinRM-auth-check/Validate-SetupState confirmed done. Original plan derived from two independent reviews ([plans/gpt55-analysis.md](gpt55-analysis.md), [plans/opus47-analysis.md](opus47-analysis.md)) and their cross-reviews ([plans/gpt55-review-of-opus.md](gpt55-review-of-opus.md), [plans/opus47-review-of-gpt55.md](opus47-review-of-gpt55.md)). Source files were re-read where the two reviewers disagreed.
This is the single authoritative document for finishing v1.0 of the system. When all Critical and High items are checked, the system is genuinely production-ready for homelab use.
---
## Executive Summary
The Local CI/CD System is a well-engineered, single-host homelab CI built on Gitea Actions, act_runner v1.0.2, VMware Workstation linked clones, Windows PowerShell 5.1, WinRM HTTPS for Windows guests and SSH for Linux guests. The architecture is the right shape for the constraint set: a thin orchestrator on the host, ephemeral linked clones for isolation, two transports for two guest families, and file-based state for everything else. Both reviewers independently graded the architecture A- and operational maturity B-/C+. That matches what the source shows.
The system is *feature-complete* and *operator-usable*, not yet *unattended-daily-trustworthy*. The gap is small in code volume but real in operational consequence. Five concrete defects can break a live build today: the Linux matrix leg of [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) silently selects the Windows template; Linux `ExtraGuestEnv` secrets are logged in clear text; the composite action outputs metadata is malformed YAML; the Linux Mode 2 PAT injection writes credentials into a remote command string; [Setup-Host.ps1](../Setup-Host.ps1) creates a directory layout that no longer matches the live system. None of these are architectural failures. Each is a 5-to-45-minute fix that the design otherwise supports cleanly.
The system makes one large unvalidated claim: `capacity: 4` in [runner/config.yaml](../runner/config.yaml). The IP-allocation mutex and lease design in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) is sound on paper, but the only evidence is single-job e2e runs. Four concurrent real builds have not been measured. This is the single most consequential outstanding question. The remediation is a burn-in, not a redesign.
Top 3 risks today, in order: (1) the Linux matrix leg builds against the Windows template until the action wiring is fixed; (2) any workflow that uses `extra-guest-env-json` for secrets is leaking those values to Gitea logs on Linux builds; (3) `capacity: 4` is a documented capability the system has never proven it keeps. Everything else is smaller.
---
## Current Project State
The optimistic [TODO.md](../TODO.md) and the conservative [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) disagree. The honest state, reconciled against the source files, is below.
| Area | Genuinely Done | Partially Done | Unvalidated / Broken |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Windows VM lifecycle | Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1), [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1), [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1)) | — | Real 4-way concurrency |
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | **FIXED 2026-05-12**: Matrix workflow Linux leg now picks correct template (C1) |
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | **FIXED 2026-05-12**: `outputs:` block corrected (H2); hardcoded `N:\` replaced with env var (H11) |
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | **FIXED 2026-05-12**: Stale lock auto-cleanup added (H5) | 4-way burn-in never run |
| Credential handling | Windows Credential Manager target `BuildVMGuest`; PAT via Credential Manager target `GiteaPAT`; Windows Mode 2 uses `http.extraHeader` | — | **FIXED 2026-05-12**: Linux Mode 2 PAT now uses `http.extraHeader` (H3); `ExtraGuestEnv` logging redacted (C2) |
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | — | — |
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional; **FIXED 2026-05-12**: default `WinBuild2025`, `-AllTemplates` switch added (H7) | — | — |
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)); **FIXED 2026-05-12**: directory array updated (H6); `F:\CI\Cache\pip` created on live host | — | act_runner runs as LocalSystem — H8 SSH alias mismatch confirmed, not yet fixed |
| Observability | JSONL per-job events, Event Log, optional webhook | Phase-duration summary | Cross-job analysis tool |
| Tests | Argument-parsing Pester suite over fake `vmrun.cmd` (4 files in [tests/](../tests/)) | — | Orchestrator state machine, real-VM smoke tests, lock concurrency |
| Documentation | [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/RUNBOOK.md](../docs/RUNBOOK.md), [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md), [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names | Consolidated template-refresh runbook |
The README is genuinely ahead of the evidence. The accurate label is *feature-complete homelab CI awaiting stabilization pass*. Everything in this plan exists to close that gap.
---
## Realistic Goals
This system should aim to be:
- A reliable single-host homelab CI/CD that runs builds inside ephemeral, isolated VMware Workstation linked clones, triggered from a self-hosted Gitea.
- Operable by one person, from one workstation, against private repositories that they trust.
- Capable of clean recovery from host reboot, VMware crash, hung `vmrun`, and stale lock files without manual intervention beyond what is documented.
- Capable of producing repeatable Windows and Linux builds with cached toolchains, with artifacts collected and published to Gitea.
- Observable enough that after a failed build the operator can diagnose whether it was their build script, the runner, the host, or the VM.
This system should NOT aim to be:
- A multi-host or multi-tenant CI. Federation is explicitly out of scope.
- A platform for running untrusted third-party code. The guest VMs are hardened only enough to be useful, not enough to safely execute hostile builds.
- A general-purpose container orchestrator. Containers are deliberately not in the picture (see *Docker / Container Notes*).
- A drop-in replacement for paid CI services like GitHub Actions cloud, Azure DevOps, or CircleCI. The threat model, secret handling, and audit story are all homelab-grade.
- A PowerShell 7 project. The PS 5.1 mandate in [AGENTS.md](../AGENTS.md) is a deliberate constraint; migration is not on the roadmap.
Anything in this plan that drifts beyond these bounds is rejected. Both reviewers tested clean here; no Kubernetes, no Prometheus, no service mesh, no enterprise vault leaked into the consolidated plan.
---
## Recommended Architecture
The current architecture is the right one. Confirm it as the target shape rather than redesign it. The recommendations below are scoped, additive fixes inside the existing architecture.
**VM lifecycle approach: keep as-is.** Linked clones from a `BaseClean` snapshot, lifecycle owned by [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) in a try/finally, sub-scripts pure and parameter-driven. Add a `Invoke-VmrunBounded` wrapper into [scripts/_Common.psm1](../scripts/_Common.psm1) and use it for the three lifecycle calls (`start`, `stop`, `deleteVM`). Do not bound `list`, `readVariable`, or `getGuestIPAddress` — those are short and the wrapper overhead is wasted there.
**Transport layer organization: keep explicit Windows/Linux branches.** Do not collapse WinRM and SSH into a generic transport interface. Branches in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) and [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1) are debuggable at 2 AM in a way that an abstraction is not. Add three small helpers in [scripts/_Common.psm1](../scripts/_Common.psm1) instead: `Invoke-VmrunBounded` (timeouts), `ConvertTo-SafeShellSingleQuotedString` (Linux quoting), and `Write-CIRedactedCommand` (secret-aware log formatter).
**IP allocation strategy: keep the lock+lease design.** The Opus suggestion of a static IP pool with `cloud-init`/`netsh` injection is a sound design, but it is a phase-3 redesign justified only if the burn-in fails. Run the burn-in first. The current design is: serialize clone+start+IP-detect through `F:\CI\State\vm-start.lock`, read `guestinfo.ci-ip` as the primary channel with `getGuestIPAddress` fallback, write a per-IP lease file. After fixing stale-lock cleanup and verifying under four-way load, this is sufficient for `capacity: 4`.
**Credential management: keep Windows Credential Manager + host-side SSH key file.** No vault, no rotation service. The fixes are: (1) verify the act_runner service account actually has access to the Credential Manager target; (2) confirm `F:\CI\keys\ci_linux` has ACLs restricting access to the runner identity and Administrators; (3) document a manual rotation procedure in [docs/RUNBOOK.md](../docs/RUNBOOK.md).
**Module organization: keep [scripts/_Common.psm1](../scripts/_Common.psm1) and [scripts/_Transport.psm1](../scripts/_Transport.psm1) split.** Do not rename or merge. Add the three helpers listed above to `_Common.psm1`. Keep `_Transport.psm1` as the SSH-specific helper despite the generic name; rename is churn.
**Logging/observability design: keep JSONL + Event Log + webhook.** Do not move to SQLite. Add a 40-line `Get-CIJobSummary.ps1` over the existing JSONL files for cross-job analysis. Add a phase-duration table printed at job end inside [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1).
---
## Recommended Stack
Canonical stack, after the fixes in this plan:
- Windows 11 Pro host, i9-10900X, 64 GB RAM, NVMe SSD.
- Windows PowerShell 5.1 only. No PowerShell 7 dependency in any production script.
- VMware Workstation Pro, `vmrun.exe` at `C:\Program Files (x86)\VMware\VMware Workstation\`.
- Built-in Windows 11 OpenSSH client (`ssh.exe`, `scp.exe`) for Linux guests.
- act_runner v1.0.2 installed as a Windows service via NSSM, capacity 4.
- Gitea at `http://10.10.20.11:3100` / `https://gitea.emulab.it`, PAT stored in Credential Manager target `GiteaPAT`.
- Templates: `WinBuild2025` (primary Windows), `WinBuild2022` (secondary Windows), `LinuxBuild2404` (Ubuntu 24.04 cloud image).
- WinRM HTTPS over port 5986 with Basic authentication and self-signed certificate. `AllowUnencrypted=false`.
- SSH user `ci_build` on Linux guests with passwordless sudo, key at `F:\CI\keys\ci_linux`.
Add to the stack:
- `Pester` v5 (already present in tests/).
- `PSScriptAnalyzer` (already used by the lint workflow).
Remove from the stack:
- [runner/Install-Runner.ps1](../runner/Install-Runner.ps1). Marked deprecated; superseded by [Setup-Host.ps1](../Setup-Host.ps1). Delete the file.
- The obsolete `F:\CI\Templates\WinBuild` directory if it still exists on the host. Replaced by `WinBuild2025` / `WinBuild2022`.
Do not add:
- Docker / Podman / container runtimes. See *Docker / Container Notes*.
- PowerShell 7.
- Prometheus, Grafana, Loki, OpenTelemetry collectors, or any external metrics stack.
- SQLite or any other database engine.
- HashiCorp Vault, Azure Key Vault, or any external secret store.
---
## Critical Problems
These must be fixed before the system is trustworthy for unattended daily use. Severity *Critical* means the issue actively breaks production today.
### C1. Linux matrix leg of build-nsInnoUnp.yml selects the Windows template
**Description**: [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) runs a matrix over `[windows, linux]` and dispatches on `runs-on: ${{ matrix.target }}-build`, but does NOT pass `guest-os` or `template-path` to the composite action. The action defaults `template-path` to empty, so [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) reads `$env:GITEA_CI_TEMPLATE_PATH` — the Windows path. The orchestrator never reads `GITEA_CI_LINUX_TEMPLATE_PATH`. Result: the Linux leg clones `WinBuild2025` and tries to SSH into a Windows VM (or via Auto-detect, into a Windows VM with WinRM credentials), which fails or produces nonsense.
**Location**: [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) L20-L40; [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L77-L83; [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L108.
**Impact**: The advertised Windows+Linux matrix workflow is broken for the Linux leg. The only evidence of "Linux works" is direct invocation of `Invoke-CIJob.ps1` with explicit `-TemplatePath`. End-to-end through the workflow has never worked for Linux.
**Fix**: Either (a) update the composite action to select between `GITEA_CI_TEMPLATE_PATH` and `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` (or runner label inference) and similarly for snapshot name; or (b) update [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) to pass `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` plus an explicit `template-path`. Option (a) is the better one because it makes the action self-sufficient for any caller. Add an `INPUT_GUEST_OS_RESOLVED` step that selects the template-path env var, then pass it as `-TemplatePath`. Also set `BaseClean-Linux` as the snapshot name on Linux.
**Estimated effort**: 1 hour including a manual smoke test of both legs.
### C2. Linux ExtraGuestEnv secrets are logged in clear text
**Description**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch builds `$buildCmd = "${envPrefix}cd '$GuestLinuxWorkDir' && $BuildCommand"` where `$envPrefix` contains `export KEY='value'; ...` with secret values inlined. The next line emits `Write-Host "[Invoke-RemoteBuild] Running build: $buildCmd"`. act_runner captures stdout and Gitea masks `secrets.*` by string match only; multi-byte transforms, base64 encoding, or substring overlap defeat the mask. The documented intended use case (`extra-guest-env-json: '{"SIGN_PASS":"${{ secrets.SIGN_PASS }}"}'`) makes this an active leak path.
**Location**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux build assembly, around the `Write-Host` immediately preceding `Invoke-SshCommand` for the build step.
**Impact**: Any homelab workflow currently using `extra-guest-env-json` for secrets has those secrets in Gitea job logs and in the local `F:\CI\Logs\<jobId>\invoke-ci.log` transcript.
**Fix**: Replace the `Write-Host "Running build: $buildCmd"` line with `Write-Host "[Invoke-RemoteBuild] Running build (env vars redacted): cd '$GuestLinuxWorkDir' && <build-command>"`. Never log `$envPrefix` or any string containing it. Do the same redaction in any other location where `$buildCmd` or `$envPrefix` is logged.
**Estimated effort**: 15 minutes.
---
## Secondary Problems
These are *High* priority items. Fix in the same stabilization pass as the Criticals.
### H1. `capacity: 4` is unvalidated under real concurrency
**Description**: [runner/config.yaml](../runner/config.yaml) sets `capacity: 4`, but the IP-allocation mutex in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) has only been exercised by single-job e2e runs (per [TODO.md](../TODO.md) §2.1). The design is plausibly correct, but DHCP behavior under four concurrent `vmrun start` calls, native-tool concurrency safety, and the 4-minute serialization of the IP-acquire phase are unmeasured. Both reviewers flagged this; severity is HIGH (not CRITICAL) because the system can be safely operated at lower capacity while this is unproven.
**Location**: [runner/config.yaml](../runner/config.yaml) L17; [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L303-L460.
**Impact**: The "production-ready" claim in [README.md](../README.md) rests on a capacity number that has not been measured. A first real burst of four parallel builds may fail in ways the e2e runs cannot predict.
**Fix**: Write `scripts/Test-CapacityBurnIn.ps1`. PowerShell 5.1, uses `Start-Job` to launch four parallel `Invoke-CIJob.ps1` invocations with unique JobIds against a small no-op build. Wait for all jobs, then assert: (1) all four VMs were destroyed; (2) all four jobs got distinct IPs; (3) `F:\CI\State\vm-start.lock` does not exist; (4) `F:\CI\State\ip-leases\` is empty; (5) total wall time and per-phase JSONL durations are recorded. Run three times. If all three pass, document the result in [docs/RUNBOOK.md](../docs/RUNBOOK.md) and keep `capacity: 4`. If any fail, set `capacity: 2` until investigated.
**Estimated effort**: Half-day to write, run, and document.
### H2. Composite action `outputs:` block is malformed YAML
**Description**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) has `artifact-path` and a duplicate `artifact-name` under the `inputs:` map instead of a top-level `outputs:` block. Step-level outputs still work (the `Out-File $env:GITHUB_OUTPUT` lines drive the upload step inside the same composite), so the live workflow does upload artifacts. But the duplicate `artifact-name` key violates YAML 1.2 (most parsers silently overwrite with the later value) and the missing `outputs:` block means downstream workflows cannot read `steps.local-ci-build.outputs.artifact-path`.
**Location**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L96 and L132-L141.
**Impact**: Latent. Today's single consumer is unaffected. The first new consumer that tries to chain on the artifact path will fail mysteriously.
**Fix**: Insert a top-level `outputs:` mapping (sibling to `inputs:` and `runs:`) containing `artifact-path` and `artifact-name`. Delete the second `artifact-name` and the `artifact-path` from the `inputs:` map. Keep the inner `Out-File $env:GITHUB_OUTPUT` lines unchanged. Note: GPT-5.5 originally rated this CRITICAL; Opus rated it LOW. The final-plan severity is HIGH because impact is latent but the YAML is clearly malformed.
**Estimated effort**: 20 minutes.
### H3. Linux Mode 2 PAT injected into clone URL
**Description**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux Mode 2 builds `https://user:pat@host/repo.git` and exports it as `GIT_CLONE_URL` in the SSH command. The PAT becomes visible in `/proc/<pid>/environ` for the duration of `git clone`, in the host-side `ssh.exe` argv, and in any PowerShell error text that captures `$envCloneCmd`. The Windows path uses `git -c http.extraHeader=Authorization: Basic <b64>`, which is strictly safer.
**Location**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux Mode 2 clone branch, around the `GIT_CLONE_URL` export.
**Impact**: PAT visibility on the guest filesystem and host process list. Bounded but real.
**Fix**: Mirror the Windows pattern. Base64-encode `user:pat`, build a single `git -c credential.helper= -c http.extraHeader=Authorization: Basic <b64> clone <non-auth-url> <dest>` command, invoke via `Invoke-SshCommand`. Drop the URL rewrite, drop the `GIT_CLONE_URL` export, drop the `unset`. Note: GPT-5.5 originally proposed an askpass helper with `chmod 600` temp file; Opus proposed the `http.extraHeader` mirror. The simpler Opus pattern is adopted.
**Estimated effort**: 30 minutes including a clone test against a private repo.
### H4. `vmrun` lifecycle calls have no timeout in production scripts
**Description**: [scripts/_Common.psm1](../scripts/_Common.psm1)'s `Invoke-Vmrun` checks `$LASTEXITCODE` but does not bound execution. [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) inherit this. A hung `vmrun start` or `deleteVM` waits on `vmware-vmx.exe` indefinitely, consuming the runner slot until act_runner's 2-hour job timeout fires. The `finally` cleanup cannot run while the native process is blocked.
**Location**: [scripts/_Common.psm1](../scripts/_Common.psm1) `Invoke-Vmrun`; callers in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1).
**Impact**: A single stuck VMware operation can pin one of the four runner slots for two hours.
**Fix**: Add `Invoke-VmrunBounded` to [scripts/_Common.psm1](../scripts/_Common.psm1). Use `Start-Process -PassThru` + `Wait-Process -Timeout` + `Stop-Process` on timeout. Return `[pscustomobject]@{ExitCode; Output; TimedOut; ElapsedSeconds}`. Replace `Invoke-Vmrun` calls for `start`, `stop`, and `deleteVM` only — keep the unbounded version for `list`, `readVariable`, `getGuestIPAddress`. Recommended timeouts: 180s for start, 60s for stop, 90s for deleteVM.
**Estimated effort**: 1 hour including a Pester test.
### H5. Stale `vm-start.lock` not cleaned automatically
**Description**: The IP-allocation lock file at `F:\CI\State\vm-start.lock` is released correctly on normal exit. On host crash mid-lock, the file remains. On reboot the OS-level lock is gone (kernel released it), but the file's mere existence does not cause failure; the issue is the 10-minute retry loop and operator confusion. Neither [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) nor [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) removes it.
**Location**: [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1).
**Impact**: First job after a crash gives a confusing message; manual `Remove-Item` cures it. Low operational severity, but high friction.
**Fix**: In both maintenance scripts, add a block that removes `F:\CI\State\vm-start.lock` if its `LastWriteTime` is older than 30 minutes. Do not write metadata into the lock file; the file's mere existence is sufficient signal. Log to console and JSONL.
**Estimated effort**: 15 minutes.
### H6. Setup-Host directory list is stale
**Description**: [Setup-Host.ps1](../Setup-Host.ps1) L118-L126 creates `Templates\WinBuild` and a handful of cache directories, but does NOT create `Templates\WinBuild2025`, `Templates\WinBuild2022`, `Templates\LinuxBuild2404`, `State`, `State\ip-leases`, `keys`, or `Cache\pip`. The orchestrator depends on `F:\CI\State\` and `F:\CI\keys\`. A fresh host bootstrap leaves the system non-functional.
**Location**: [Setup-Host.ps1](../Setup-Host.ps1) L118-L126.
**Impact**: Disaster recovery is broken. Rebuilding the host from scratch using the documented procedure does not produce a working system.
**Fix**: Update the directory array to: `BuildVMs`, `Artifacts`, `Logs`, `Templates\WinBuild2025`, `Templates\WinBuild2022`, `Templates\LinuxBuild2404`, `State`, `State\ip-leases`, `keys`, `act_runner\logs`, `Cache\NuGet`, `Cache\pip`, `RunnerWork`, `ISO`. Remove `Templates\WinBuild`. Add a comment block referencing [AGENTS.md](../AGENTS.md) §Struttura directory host.
**Estimated effort**: 10 minutes.
### H7. Backup-CITemplate default path is wrong
**Description**: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 defaults `TemplatePath` to `'F:\CI\Templates\WinBuild'`. That directory no longer exists. The default-argument backup silently fails or backs up an empty directory, then the operator refreshes the actual `WinBuild2025` template, with no rollback.
**Location**: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 and the help block.
**Impact**: An operator running `.\Backup-CITemplate.ps1` before a template refresh believes they have a backup. They do not.
**Fix**: Change the default to `'F:\CI\Templates\WinBuild2025'`. Add a `-AllTemplates` switch that iterates `F:\CI\Templates\*` and backs up each (with subfolder naming preserved in the backup destination). Update [docs/HOST-SETUP.md](../docs/HOST-SETUP.md) and [docs/RUNBOOK.md](../docs/RUNBOOK.md) to reflect the new default. Update the help block.
**Estimated effort**: 30 minutes.
### H8. act_runner service account vs SSH alias mismatch
**Description**: [Setup-Host.ps1](../Setup-Host.ps1) L294-L306 writes the `gitea-ci` SSH alias to `$env:USERPROFILE\.ssh\config` — the interactive user's profile. The NSSM install at L264-L273 does not set an explicit service account, which on Windows commonly means LocalSystem. LocalSystem's home is `C:\Windows\System32\config\systemprofile\`, not the user profile. [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) uses `repo-url: 'ssh://gitea-ci/Simone/...'`. If the service runs as LocalSystem, the alias resolution fails and every Mode 1 (host-side clone) attempt errors out.
**Location**: [Setup-Host.ps1](../Setup-Host.ps1) L264-L273 (NSSM install), L294-L306 (SSH config write); [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) L35.
**Impact**: Potentially blocks all Mode 1 host-side clone builds. Severity is HIGH-PENDING-VERIFICATION; first action is verification.
**Fix**: Step 1, verify: run `Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName` on the host and document the answer in [docs/RUNBOOK.md](../docs/RUNBOOK.md). Step 2a, if `StartName` is `LocalSystem`: copy the SSH config and known-hosts to `C:\Windows\System32\config\systemprofile\.ssh\` with proper ACLs. Step 2b alternative: create a dedicated `ci_runner` Windows user, reinstall NSSM with `obj=.\ci_runner`, store the SSH config under that user's profile. Step 3: add the verification check to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) or a new `Validate-HostState.ps1`. Note: GPT-5.5 did not catch this. Opus did. Severity HIGH stands.
**Estimated effort**: 1 hour for verification + fix.
### H9. `ci-report-ip.service` not asserted in deploy-state validator
**Description**: [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) is Windows-only: it checks WinRM, Defender, Firewall, UAC, etc. It does not check whether the Linux template's `ci-report-ip.service` (the systemd unit writing `guestinfo.ci-ip`) is enabled. The unit IS validated inside [template/Install-CIToolchain-Linux2404.sh](../template/Install-CIToolchain-Linux2404.sh) L203-L221 and [template/Prepare-LinuxBuild2404.ps1](../template/Prepare-LinuxBuild2404.ps1) L361-L362, but not at the level of "is this template ready to be promoted to BaseClean-Linux." If a future refresh forgets to re-enable it, every Linux build silently falls back to the 120-second `getGuestIPAddress` timeout.
**Location**: [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
**Impact**: Silent degradation of Linux build startup time after a faulty template refresh.
**Fix**: Add a Linux branch to `Validate-DeployState.ps1` (or create a `Validate-LinuxDeployState.ps1`) that opens an SSH session and runs `systemctl is-enabled ci-report-ip.service` and `test -x /usr/local/bin/ci-report-ip.sh`. Fail with a clear error if either is missing.
**Estimated effort**: 45 minutes.
### H10. No automated end-to-end smoke test
**Description**: The Pester suite in [tests/](../tests/) exercises argument parsing only via a fake `vmrun.cmd`. The "production-ready" claim rests on manual e2e runs (`e2e-008`, `e2e-009`, `§3.3`). There is no scripted, repeatable test that exercises real VM lifecycle and exits non-zero on failure. Both reviewers flagged this; severity is HIGH because regression risk without it is high after every change.
**Location**: [tests/](../tests/), [gitea/workflows/](../gitea/workflows/).
**Impact**: Any change to orchestration scripts has no automated guard. Regressions are caught only by chance during the next manual run.
**Fix**: Create `gitea/workflows/self-test.yml` (workflow_dispatch trigger only). One job that calls the composite action against a minimal `Write-Host 'ok' > out.txt` build, asserts artifact upload, JSONL phase events, and clean teardown. Have it succeed-or-fail for Windows and a second job for Linux. Add `scripts/Test-Smoke.ps1` that the workflow invokes; the script also runs locally. Note: this replaces the much larger `Measure-CIBenchmark.ps1` wiring suggestion; the smoke test is sufficient and cheaper.
**Estimated effort**: Half-day.
### H11. Composite action hardcodes the script root path
**Description**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 sets `$ciScriptsDir = 'N:\Code\Workspace\Local-CI-CD-System\scripts'`. The action is presented as reusable, but moving the repo breaks it. Opus caught this; GPT-5.5 missed it (it caught path drift in setup/backup only).
**Location**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148.
**Impact**: Action is not actually reusable. Any second consumer or repo relocation breaks builds.
**Fix**: Replace with a runner env var. Add `GITEA_CI_SCRIPT_ROOT: "N:\\Code\\Workspace\\Local-CI-CD-System\\scripts"` to [runner/config.yaml](../runner/config.yaml) `envs:`. In the action, use `$ciScriptsDir = $env:GITEA_CI_SCRIPT_ROOT`. Do not use `${{ github.action_path }}` — the action repo is not necessarily cloned into the consumer's workspace in this Gitea setup.
**Estimated effort**: 20 minutes.
### H12. Measure-CIBenchmark uses a different IP discovery path than production
**Description**: [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) L129-L146 polls `Invoke-Vmrun -Operation getGuestIPAddress`. [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L348-L389 primarily reads `guestVar 'ci-ip'` and uses `getGuestIPAddress` as fallback. The benchmark therefore measures a different code path than what production runs, and its phase timings cannot be used for capacity planning.
**Location**: [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) L129-L146.
**Impact**: Benchmark data is misleading. Decisions to keep or change `capacity: 4` based on it would be based on wrong measurements.
**Fix**: Extract IP-acquire into a function `Get-GuestIPAddress` in [scripts/_Common.psm1](../scripts/_Common.psm1) that takes the VMX path and credentials and implements the `guestinfo.ci-ip`-then-`getGuestIPAddress` flow used in production. Call it from both `Invoke-CIJob.ps1` and `Measure-CIBenchmark.ps1`.
**Estimated effort**: 1 hour.
---
## Complete Priority Checklist
Every item is concrete, references a file, and ends with an effort estimate.
### Critical [x]
- [x] Fix Linux matrix template selection — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) + [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) — action selects `GITEA_CI_TEMPLATE_PATH` or `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` input or runner label, also resolves the correct `snapshot-name` (`BaseClean` or `BaseClean-Linux`) — done 2026-05-12.
- [x] Redact Linux ExtraGuestEnv secret logging — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch — replace `Write-Host "...$buildCmd"` with redacted version; never log `$envPrefix` — done 2026-05-12.
### High
- [x] done Run 4-way concurrency burn-in — [scripts/Test-CapacityBurnIn.ps1](../scripts/Test-CapacityBurnIn.ps1) (new) — `Start-Job`-based harness, configurable rounds/parallelism, assertions on clone cleanup and stale lock file.
- [x] Fix composite action outputs YAML — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) — add top-level `outputs:` block; remove duplicate `artifact-name` from `inputs:` — done 2026-05-12.
- [x] Replace Linux Mode 2 PAT URL with `http.extraHeader` — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch — base64-encode `user:pat`, single `git -c http.extraHeader=...` command — done 2026-05-12.
- [x] Add `Invoke-VmrunBounded` and use for lifecycle calls — [scripts/_Common.psm1](../scripts/_Common.psm1), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) — bound `start`, `stop`, `deleteVM` only — done 2026-05-12; Pester 18/18 pass.
- [x] Add stale `vm-start.lock` cleanup — [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) — remove file older than 30 minutes — done 2026-05-12.
- [x] Fix Setup-Host directory list — [Setup-Host.ps1](../Setup-Host.ps1) L118 — match [AGENTS.md](../AGENTS.md) §Struttura — done 2026-05-12.
- [x] Fix Backup-CITemplate default — [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 + help block — default `WinBuild2025`, add `-AllTemplates` — done 2026-05-12.
- [x] Verify and fix act_runner service-account SSH alias — [Setup-Host.ps1](../Setup-Host.ps1) — added Step 8: replicates `~\.ssh\config`, `id_rsa`, and `known_hosts` to `C:\Windows\System32\config\systemprofile\.ssh\` with SYSTEM/Administrators-only ACLs; added `StrictHostKeyChecking accept-new` to gitea-ci alias block — done 2026-05-12. Verification: run `Setup-Host.ps1` to apply, then test `ssh gitea-ci` from a SYSTEM-context shell.
- [x] Add `ci-report-ip.service` check to deploy-state validator — [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) — Linux branch via SSH — confirmed implemented 2026-05-13.
- [x] Add on-demand smoke-test workflow — [gitea/workflows/self-test.yml](../gitea/workflows/self-test.yml) + [scripts/Test-Smoke.ps1](../scripts/Test-Smoke.ps1) — confirmed implemented 2026-05-13.
- [x] Replace hardcoded `N:\` script path in composite action — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 + [runner/config.yaml](../runner/config.yaml) — new `GITEA_CI_SCRIPT_ROOT` env var — done 2026-05-12; active config deployed and act_runner restarted 2026-05-12 (labels: windows-build:host dotnet:host msbuild:host linux-build:host).
- [x] Align benchmark IP discovery with production — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) + [scripts/_Common.psm1](../scripts/_Common.psm1) — `Get-GuestIPAddress` already extracted and used by both — confirmed 2026-05-13.
### Medium [ ]
- [x] done Add pre-clone disk space gate — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) Phase 2 — confirmed implemented at L339 (2026-05-13).
- [x] done Add post-failure guest diagnostics collection — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) catch block — `Get-GuestDiagnostics` to `F:\CI\Artifacts\<jobId>\diagnostics\` — confirmed already implemented 2026-05-13.
- [x] done Wire `UseSharedCache` through composite action and orchestrator — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L133/L185/L269, [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L502 — confirmed 2026-05-13.
- [x] done Authenticate-not-just-TCP WinRM readiness — [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1) — optional Credential parameter, single `Invoke-Command { 'ready' }` after TCP open.
- [x] done Add Ubuntu cloud VMDK SHA256 verification — [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1) — `-VmdkSha256` param + `Get-FileHash` check at L387 — confirmed 2026-05-13.
- [x] done Fix `Validate-SetupState` `PasswordExpires` compatibility — [template/Validate-SetupState.ps1](../template/Validate-SetupState.ps1) — `(-not $u.PasswordExpires) -or ($u.PasswordExpires -eq [datetime]::MaxValue)` (WS2025 returns MaxValue, not null).
- [x] done Update [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — already uses `Start-Job` and `-RepoUrl` — confirmed 2026-05-13.
- [x] done Add `JobId` sanitization — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — `[ValidatePattern('^[A-Za-z0-9._-]+$')]` at L77 — confirmed 2026-05-13.
- [x] done Add `ExtraGuestEnv` key validation — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L184 — confirmed 2026-05-13.
- [x] done Add phase-duration summary at job end — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L541 — confirmed 2026-05-13.
- [x] done Add 90-minute job-duration webhook warning — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — `WebhookUrl` param + `Start-Job` fires `[WARNING]` after 5400s; cancelled in finally — done 2026-05-13.
- [x] done Document HGFS shared folder write semantics — [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) section 11 — write semantics, cache-poisoning risk, safe-use rules, invalidation — done 2026-05-13.
- [x] done Standardize Event Log source names — Watch-DiskSpace: `CI-DiskAlert`→`CI-DiskSpaceAlert`; Watch-RunnerHealth already `CI-RunnerHealth` — done 2026-05-13.
- [x] done Tighten SSH host-key handling for CI jobs — [scripts/_Transport.psm1](../scripts/_Transport.psm1) — `KnownHostsFile` param: when non-empty uses `accept-new` mode; when empty (default) uses permissive for provisioning — confirmed already implemented 2026-05-13.
- [x] done Consolidate template-refresh runbook — [docs/RUNBOOK.md](../docs/RUNBOOK.md) section 5 (5.1–5.9) — confirmed already fully implemented 2026-05-13.
### Low [ ]
- [x] done Delete deprecated runner installer — runner/Install-Runner.ps1 — file already absent from repo — confirmed 2026-05-13.
- [x] done Align Setup-Host password documentation — [Setup-Host.ps1](../Setup-Host.ps1) help block has `Default: CIBuild!ChangeMe2026` with explicit "CHANGE THIS before production use" note — sufficient for homelab — 2026-05-13.
- [x] done Remove emoji from webhook payloads — Watch-DiskSpace uses `[WARNING]`, Watch-RunnerHealth uses `[ALERT]`/`[WARNING]` — no emoji found — confirmed 2026-05-13.
- [x] done Fix Validate-* docblocks to reflect HTTPS/5986 — Validate-SetupState.ps1 L7: `HTTP/Basic` → `HTTPS/Basic, port 5986` — done 2026-05-13. (Validate-DeployState already correct.)
- [x] done Align benchmark JSON schema with test plan — [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) § 3.6 updated to match actual script output fields (`cloneSec`, `startSec`, `ipSec`, `winrmSec`, `destroySec`, `totalBootSec`) — done 2026-05-13.
- [x] done Add `TemplatePath` sanity check — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — `Write-Warning` if not under `F:\CI\Templates\` — done 2026-05-13.
- [x] done Add a maintenance flag for Watch-RunnerHealth — [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1) — skip restart if `F:\CI\State\runner-maintenance.flag` exists — done 2026-05-13.
- [x] done Add ACL validation script — [scripts/Validate-HostState.ps1](../scripts/Validate-HostState.ps1) — SSH key ACLs, Credential Manager target, CI dir structure, vmrun.exe check — done 2026-05-13.
- [x] done Move done sections of [TODO.md](../TODO.md) into a new [docs/CHANGELOG.md](../docs/CHANGELOG.md) — reverse-chronological log of all completed sprints — done 2026-05-13.
- [x] done Fix [docs/CI-FLOW.md](../docs/CI-FLOW.md) failure-scenario table — "Build fails inside VM" row updated: `Get-GuestDiagnostics` to `diagnostics\` folder — done 2026-05-13.
### Nice to have [x] done
- [x] Add `scripts/Get-CIJobSummary.ps1` — reads last N days of JSONL files, prints duration table — 1h.
- [x] Add artifact manifest with file hashes — [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1) — `manifest.json` with name, size, SHA256, commit, job ID — 1h.
- [x] Add per-job VMX CPU/RAM override — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) + composite action input — post-clone VMX edit, default off — 1h.
- [x] Add optional Gitea API runner-online check — [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1) — optional `-GiteaUrl` and PAT target — 1h.
- [x] Begin SHA256 pinning of toolchain installers — [template/Install-CIToolchain-WinBuild2025.ps1](../template/Install-CIToolchain-WinBuild2025.ps1) — fill hashes for Git, 7-Zip, Python, .NET, Node — 2h.
- [x] Add linked-clone delta size measurement — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) — record per-job delta growth — 30min.
---
## Detailed Operational Checklist
Execute in this order. Each task lists a prerequisite, an action, and a verification step.
### Sprint 1: Validation
Goal: prove or downgrade unvalidated claims before any code changes go in.
**Task 1.1 — Verify act_runner service account** [x] done 2026-05-12
- Result: StartName = **LocalSystem**. H8 fix is required. See [docs/RUNBOOK.md](../docs/RUNBOOK.md) — Runner service identity section to be updated.
**Task 1.2 — Inventory current state-file presence** [x] done 2026-05-12
- Result: `F:\CI\Cache\pip` was missing. Created manually during session. All other required dirs present.
**Task 1.3 — Run 4-way concurrency burn-in (H1)** [ ] not yet started
- Prerequisite: write `scripts/Test-CapacityBurnIn.ps1` first; `WinBuild2025` and `LinuxBuild2404` templates powered off; >= 100 GB free on `F:`.
- Action: invoke the burn-in three times against a no-op build (e.g. `Write-Host 'ok'`); record per-run JSONL outputs.
- Verification: zero IP collisions across runs, zero leaked VMs, zero leaked lock files, all jobs succeeded. If any fail, set `runner/config.yaml capacity: 2` and create a follow-up investigation task.
**Task 1.4 — Manually invoke Linux matrix leg to confirm C1 bug** [ ] smoke test pending
- C1 code fix applied 2026-05-12. A live workflow run with both legs is still needed to confirm end-to-end.
- Prerequisite: act_runner service restarted (to pick up `GITEA_CI_SCRIPT_ROOT` from config.yaml).
- Action: push a tag to a test branch, observe both legs.
- Verification: both legs succeed against the correct templates.
### Sprint 2: Critical Fixes
Goal: clear both Criticals plus the highest-impact Highs that touch production paths.
**Task 2.1 — Fix Linux matrix template selection (C1)** [x] done 2026-05-12
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): added `$resolvedGuestOS` / `$resolvedTemplatePath` / `$resolvedSnapshotName` resolution block; `Auto` infers from `$env:RUNNER_LABELS`.
- [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml): explicit `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` added as belt-and-suspenders.
- Live smoke test (Task 1.4) still required.
**Task 2.2 — Redact Linux ExtraGuestEnv logging (C2)** [x] done 2026-05-12
- [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1): `Write-Host "Running build: $buildCmd"` replaced with redacted form that only logs `$BuildCommand` (user-visible), never `$envPrefix`.
- Verification with a live job still recommended.
**Task 2.3 — Fix composite action outputs YAML (H2)** [x] done 2026-05-12
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): duplicate `artifact-name` under `inputs:` removed; proper top-level `outputs:` block added with both `artifact-path` and `artifact-name` mapped from step outputs.
**Task 2.4 — Linux Mode 2 PAT to http.extraHeader (H3)** [x] done 2026-05-12
- [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1): `GIT_CLONE_URL` env-var approach removed; replaced with `git -c credential.helper= -c http.extraHeader=Authorization: Basic <b64>` pattern mirroring the Windows branch.
- Live verification (Mode 2 clone against private repo) still recommended.
**Task 2.5 — Fix Setup-Host directory list (H6)** [x] done 2026-05-12
- [Setup-Host.ps1](../Setup-Host.ps1): directory array updated to canonical list (adds WinBuild2025, WinBuild2022, LinuxBuild2404, State, State\ip-leases, keys, Cache\pip; removes obsolete WinBuild).
**Task 2.6 — Fix Backup-CITemplate default (H7)** [x] done 2026-05-12
- [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1): default `TemplatePath` changed to `F:\CI\Templates\WinBuild2025`; `-AllTemplates` switch added (iterates all template subdirs with per-template naming and pruning).
### Sprint 3: High Priority Fixes
Goal: stabilize the rest of the High-tier items.
**Task 3.1 — Add `Invoke-VmrunBounded` (H4)** [x] done 2026-05-12
- `Invoke-VmrunBounded` added to [scripts/_Common.psm1](../scripts/_Common.psm1) using `[System.Diagnostics.Process]` directly (not `Start-Process -PassThru`, which does not populate ExitCode reliably in PS5.1). Kills process tree via `taskkill /F /T` on timeout; skips stdout/stderr drain on timeout path to avoid faulted-task re-throw. Returns `{ExitCode; Output; TimedOut; ElapsedSeconds}`.
- Wired in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): `vmrun start` uses 180s timeout.
- Wired in [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1): `vmrun stop soft/hard` (60s each), `vmrun deleteVM` (90s).
- Pester suite: 18/18 pass (8 new `Invoke-VmrunBounded` tests + 10 pre-existing) in ~7s — verified 2026-05-12.
**Task 3.2 — Stale lock cleanup (H5)** [x] done 2026-05-12
- [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1): both now remove `F:\CI\State\vm-start.lock` if LastWriteTime is older than 30 minutes.
**Task 3.3 — Fix or relocate SSH alias for service account (H8)** [x] done 2026-05-12
- [Setup-Host.ps1](../Setup-Host.ps1): added `StrictHostKeyChecking accept-new` to the `gitea-ci` alias block.
- [Setup-Host.ps1](../Setup-Host.ps1): added Step 8 that replicates `~\.ssh\config`, `id_rsa`, and `known_hosts` (if present) to `C:\Windows\System32\config\systemprofile\.ssh\` (LocalSystem's SSH home). Sets ACLs to SYSTEM/Administrators-only on the private key.
- To apply: re-run `Setup-Host.ps1` (or run Step 8 standalone) elevated. Key must exist at `~\.ssh\id_rsa`.
- Verification: `psexec -s -i cmd /c ssh -T gitea-ci` should return `"Hi <user>! You've successfully authenticated..."` from Gitea.
**Task 3.4 — Linux deploy validator check for ci-report-ip (H9)** [x] done 2026-05-12
- `-GuestOS Linux` branch added to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
- Checks via `ssh.exe`: `systemctl is-enabled ci-report-ip.service` and `test -x /usr/local/bin/ci-report-ip.sh`.
- Windows branch unchanged. New params: `-GuestOS`, `-SshKeyPath`, `-SshUser`.
- Verification: against the live template, the script reports the service as enabled.
**Task 3.5 — Add smoke-test workflow (H10)** [x] done 2026-05-12
- [gitea/workflows/self-test.yml](../gitea/workflows/self-test.yml) created: `workflow_dispatch` only, two independent jobs (`smoke-windows` + `smoke-linux`).
- [scripts/Test-Smoke.ps1](../scripts/Test-Smoke.ps1) created: invokes `Invoke-CIJob.ps1` with a marker-file build command and asserts artifact dir + JSONL success event.
- Prerequisite: H2 and C1 fixed (both done). Verification: trigger via `workflow_dispatch` and observe both legs green.
**Task 3.6 — Replace hardcoded N:\ path (H11)** [x] done 2026-05-12
- [runner/config.yaml](../runner/config.yaml): `GITEA_CI_SCRIPT_ROOT` env var added.
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): `$ciScriptsDir = $env:GITEA_CI_SCRIPT_ROOT` used in place of hardcoded path.
- Active config deployed to `F:\CI\act_runner\config.yaml` and act_runner restarted 2026-05-12. Log confirmed: `labels updated to: [windows-build:host dotnet:host msbuild:host linux-build:host]`.
**Task 3.7 — Benchmark IP path alignment (H12)** [x] done 2026-05-12
- `Get-GuestIPAddress` extracted into [scripts/_Common.psm1](../scripts/_Common.psm1): polls `guestVar ci-ip` (primary) then `getGuestIPAddress` (fallback) with configurable timeout.
- [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): Phase 3b replaced with `Get-GuestIPAddress` call (eliminates ~50 lines of duplicated polling).
- [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1): IP phase now calls `Get-GuestIPAddress` (was using `getGuestIPAddress`-only fallback, missing `ci-ip` primary).
- Verification: run `Measure-CIBenchmark.ps1 -Iterations 1` for Windows and Linux; confirm phase timings match those recorded in production JSONL within ±10%.
### Sprint 4: Operational Hardening
Goal: harden recovery, secret handling, and recoverability.
[x] done **Task 4.1 — Pre-clone disk gate, JobId/ExtraGuestEnv validation, redacted command logger** (Medium items grouped) — edit [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1); verify with deliberately bad inputs.
[x] done **Task 4.2 — Failure-path diagnostics** — add a `Get-GuestDiagnostics` helper invoked from the `catch` block of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1). Skip on Phase 1/2 failures (no VM yet). Best-effort SSH/WinRM, no throw on failure.
[x] done **Task 4.3 — Wire `UseSharedCache`** — surface in composite action and orchestrator; confirm with a Python build that pip cache is reused.
[x] done **Task 4.4 — Tighten SSH host keys for CI jobs** — `accept-new` + `F:\CI\State\known_hosts` for CI; keep permissive mode for template provisioning (host keys are unknown there).
[x] done **Task 4.5 — Ubuntu VMDK SHA256 verification** — add to [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1).
[x] done **Task 4.6 — Consolidated template-refresh runbook** — edit [docs/RUNBOOK.md](../docs/RUNBOOK.md). One section covering: stop runner, back up template (`-AllTemplates`), boot template, KMS reactivation via `slmgr /ato` (Windows), run `Prepare-WinBuild2025.ps1` or `Prepare-LinuxBuild2404.ps1`, shutdown, snapshot `BaseClean_yyyyMMdd`, run `Validate-DeployState.ps1` (Windows and Linux branches), run smoke workflow, then update `GITEA_CI_SNAPSHOT_NAME` in [runner/config.yaml](../runner/config.yaml), keep prior snapshot 7 days, rollback procedure.
### Sprint 5: Quality and Optimization
Goal: polish.
**Task 5.1 — Delete dead code** — [x] done — removed [runner/Install-Runner.ps1](../runner/Install-Runner.ps1) (DEPRECATED dead code).
**Task 5.2 — Webhook emoji removal, Event Log naming, docblock corrections** — [x] done — replaced Unicode emoji and Discord-style emoji codes in Watch-DiskSpace.ps1, Watch-RunnerHealth.ps1, Test-E2E-Section3.3.ps1 with plain-text prefixes ([WARNING], [ALERT], [WARN]).
**Task 5.3 — Update test plan** — [x] done — [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — replaced ForEach-Object -Parallel with Start-Job pattern; replaced -RepositoryUrl with -RepoUrl (3 occurrences).
**Task 5.4 — Add `Get-CIJobSummary.ps1`** — [x] done — new [scripts/Get-CIJobSummary.ps1](../scripts/Get-CIJobSummary.ps1): scans JSONL logs, prints summary table, supports -JobId for detail view, -Failed filter, -Last N.
**Task 5.5 — Phase-duration summary in orchestrator** — [x] done — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): phaseTimings list tracks Phase 1-6 elapsed seconds; printed as table before SUCCESS/FAILURE banner.
**Task 5.6 — Begin SHA256 pinning** — [x] done — [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) Section 10 added: pinning policy, priority table (Git, 7-Zip, Python, .NET, Ubuntu VMDK), PS5.1 implementation pattern, update guidance.
---
## Recommended Refactors
Only refactors that genuinely simplify or fix something. No churn.
1. **Extract `Get-GuestIPAddress` into [scripts/_Common.psm1](../scripts/_Common.psm1)**. Removes the dual-implementation drift between `Invoke-CIJob.ps1` and `Measure-CIBenchmark.ps1`. Single source of truth for the `guestinfo.ci-ip`-then-fallback flow. Justification: real bug avoidance, not aesthetic.
2. **Extract `Invoke-VmrunBounded` into [scripts/_Common.psm1](../scripts/_Common.psm1)**. Closes the H4 timeout gap and consolidates the pattern that already exists in some deploy scripts.
3. **Add `Write-CIRedactedCommand` and `ConvertTo-SafeShellSingleQuotedString` to [scripts/_Common.psm1](../scripts/_Common.psm1)**. The first standardizes secret-aware command logging across Windows and Linux paths. The second consolidates the `-replace "'", "'\''"` idiom used in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1). Both are 5-line functions.
4. **DO NOT extract a generic transport interface**. Opus suggested `Invoke-GuestCommand`/`Copy-GuestItem`/`Wait-GuestTransportReady`. Both reviewers ultimately agreed this is the wrong move for a two-transport homelab. Keep explicit Windows/Linux branches in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) and [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1).
5. **DO NOT introduce `F:\CI\State\config.json` as a central config**. [AGENTS.md](../AGENTS.md) and [runner/config.yaml](../runner/config.yaml) are already the two sources of truth. Adding a third creates drift, not reduces it. The remediation is to align the existing two, not invent a third.
6. **DO NOT migrate to PowerShell 7**. The PS 5.1 constraint is deliberate, encoded in [AGENTS.md](../AGENTS.md), and consistently respected across the codebase. A migration is a multi-day churn that buys little.
7. **DO NOT replace JSONL with SQLite**. The JSONL output is forward-compatible and trivially greppable. `Get-CIJobSummary.ps1` over JSONL covers the cross-job analysis need without adding a database engine.
8. **DO NOT pursue a static IP pool unless the burn-in fails**. The current lock+lease design after the stale-lock fix is sufficient for `capacity: 4`. Static IPs cost more than they save at this scale.
---
## Security Improvements
Must (do these in Sprint 2 or 3):
- Redact Linux `ExtraGuestEnv` secret logging (C2).
- Replace Linux Mode 2 PAT URL with `http.extraHeader` (H3).
- Verify and fix the act_runner SSH alias service-account issue (H8).
- Add `JobId` pattern validation (Medium).
- Add `ExtraGuestEnv` key pattern validation (Medium).
- Run SSH `StrictHostKeyChecking=accept-new` with `UserKnownHostsFile=F:\CI\State\known_hosts` for CI jobs (Medium).
Could (Nice-to-have):
- Begin SHA256 pinning of Tier-1 toolchain installers.
- Verify ACLs on `F:\CI\keys\ci_linux` via a new `Validate-HostState.ps1`.
- Add a one-paragraph note in [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) on Gitea log-masking limits (substring-only masking; base64 transforms defeat it).
Will NOT do:
- Full PKI for WinRM. The self-signed lab certificate plus VMnet8 isolation is the right boundary.
- External secret vault integration.
- Enable Defender / Firewall / UAC inside the build VM. The threat model accepts this for trusted private code; documented in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md).
- Per-build credential rotation.
- Artifact signing.
---
## CI/CD Improvements
Concrete improvements to workflows and the composite action.
1. **Make the composite action self-sufficient for guest OS selection** (C1 fix). After this, [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) does not need to pass `guest-os`, `template-path`, or `snapshot-name` for the standard Windows/Linux matrix; the action infers from the runner label or accepts the explicit input.
2. **Add a top-level `outputs:` block** so downstream consumers can chain (H2 fix).
3. **Add `use-shared-cache` input** plumbed to `Invoke-CIJob.ps1 -UseSharedCache` (Medium fix). Forward to `Invoke-RemoteBuild.ps1` so NuGet/pip caches actually work end-to-end.
4. **Add an on-demand `gitea/workflows/self-test.yml`** (`workflow_dispatch` trigger only). One Windows job, one Linux job, both calling the composite action against a trivial build (`Write-Host 'ok' > out.txt`). Asserts artifact upload, JSONL events, clean teardown. Use as the regression gate for any action change.
5. **Add a Pester `gitea/workflows/tests.yml`** that runs the existing tests on every push. PowerShell 5.1, `Invoke-Pester -CI`. This is the current missing "CI for the CI" piece.
6. **Keep the lint workflow** [gitea/workflows/lint.yml](../gitea/workflows/lint.yml) as is.
7. **Do NOT add scheduled builds, dependency-update workflows, or release workflows.** Out of scope.
---
## Docker / Container Notes
This project deliberately does not use Docker, Podman, or any container runtime. The decision is correct and should be preserved.
Rationale:
- Windows builds require VS Build Tools, MSBuild, the full .NET SDK, and Windows-specific signing chains. Running those in Windows containers requires Windows Server host images, breaks isolation guarantees the project gets cheaply from VMware, and adds licensing complexity.
- The project's isolation boundary is a VM, not a container. VMs offer strictly stronger isolation than containers (kernel separation, hardware-virtualized network stack, separate filesystem) at a cost the system can afford.
- act_runner's `container.enabled: false` is the right setting for a system that uses real VMs.
- The performance overhead of "spin up a VM" vs "spin up a container" is real (a clone takes 5-30s, a container ~1s), but the orchestrator amortizes this across a 2-hour build budget. The cost is acceptable.
When might containers enter the picture? Likely never for this scope. If the project ever needs to build small Linux-only services with no Windows leg, a side-by-side `linux-build-container` runner label could be added later. That is a phase-3+ idea, not a current task.
---
## Monitoring and Logging Improvements
Concrete additions only. No metric server.
JSONL schema additions inside `F:\CI\Logs\<jobId>\invoke-ci.jsonl` (emitted by [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1)):
- Per-phase `duration_ms` calculated at phase end.
- `free_disk_gb_before` and `free_disk_gb_after` for the job.
- `ip_detection_method` field with values `guestinfo` or `getGuestIPAddress`.
- `vm_start_lock_wait_ms` recording how long the orchestrator waited on `vm-start.lock`.
- `clone_dir_size_mb` measured at teardown.
Event Log additions:
- Standardize on `CI-DiskSpaceAlert` (EventId 1001), `CI-RunnerHealth` (1002 stopped / 1003 restarted / 1004 throttled).
- Add `CI-JobFailure` source for failed orchestrator runs (EventId 2001) writing JobId and failed phase.
Watcher script additions:
- `scripts/Get-CIJobSummary.ps1` (nice-to-have): reads last N days of JSONL files, prints a table of JobId, started, elapsed, phase 1 duration, phase 5 duration, status.
- `scripts/Watch-RunnerHealth.ps1` (Low): respect `F:\CI\State\runner-maintenance.flag` to suppress restart attempts during template refresh.
Webhook payload changes:
- Remove emoji prefixes. Use `[WARN]`, `[ERROR]`, `[INFO]` plain text.
Do NOT add:
- Prometheus exporters.
- Grafana dashboards.
- OpenTelemetry collectors.
- SQLite or any other database sink.
- Discord/Slack rich-embed cards.
---
## Backup and Recovery Improvements
The current backup story is roughly: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) (currently pointing at a stale path). Improvements:
1. Fix the default path and add `-AllTemplates` switch (H7).
2. Add to [docs/RUNBOOK.md](../docs/RUNBOOK.md) the consolidated template-refresh runbook (Sprint 4 task 4.6).
3. Register a weekly scheduled task that runs `.\Backup-CITemplate.ps1 -AllTemplates -KeepLast 3` and writes a log to `F:\CI\Logs\backup\<date>.log`. Update [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1) to add this fifth scheduled task.
4. Document recovery from a destroyed template: restore the latest `.7z` from the backup destination, validate snapshots exist via `vmrun listSnapshots`, run `Validate-DeployState.ps1` (Windows + Linux), run the smoke workflow. If the smoke fails, restore the previous backup. Step-by-step in [docs/RUNBOOK.md](../docs/RUNBOOK.md).
5. Document the simple recovery path for a stale `vm-start.lock` and stale IP leases in [docs/RUNBOOK.md](../docs/RUNBOOK.md) — even though Sprint 3 task 3.2 makes this automatic, the operator-facing docs should still say "if you ever need to do this manually, the commands are X."
Do NOT add:
- VM snapshot backup to remote storage (out of scope; 7z files on a second drive is sufficient).
- Restore-time automated diff against a known-good template (over-engineered).
---
## Runner and Build Optimization
Concrete, measurable.
1. **Wire `use-shared-cache`** through the composite action and orchestrator (Medium item). Expected impact: ~15-30% faster Python and .NET builds that pull packages on every run, by avoiding the `pip download` and `nuget restore` over the WAN. Measurement: compare `phase 5 duration` JSONL before/after on a representative NSIS-plugin build.
2. **After burn-in, decide on `capacity`**. If the burn-in shows that capacity 4 is solid, keep it. If it shows that capacity 2 is the safe ceiling under typical host CPU contention, set `runner/config.yaml capacity: 2`. Expected impact: avoidance of unreproducible build failures during peaks.
3. **Defender exclusion for `F:\CI\BuildVMs\`** on the host (Low). Cloning and start are I/O heavy on the parent VMDK chain; Defender scanning slows this. Use `Set-MpPreference -ExclusionPath 'F:\CI\BuildVMs'`. Expected impact: ~10% faster clone-to-IP times. Document the trade-off in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md): malicious code reaching the host would persist faster, but the threat model already accepts trusted-builds-only.
4. **Linked-clone health monitoring** (Nice-to-have): add per-job delta size to JSONL. If the average delta grows over months without a template refresh, that is a signal something is writing into the template chain. Not urgent; nice diagnostic.
5. **Do NOT pursue a warm clone pool**. Both reviewers agreed: the savings (5-30s per job) do not justify the state-machine complexity for a 2-hour build budget.
6. **Do NOT pursue VMX CPU/RAM tuning per job** unless the burn-in shows host saturation. Wait for evidence.
---
## Repository Conventions
These are the codified rules. Where ambiguous, write them into [AGENTS.md](../AGENTS.md) and [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md).
**Style and linting** (already largely in place):
- PowerShell 5.1 only. Every script begins with `#Requires -Version 5.1`, `Set-StrictMode -Version Latest`, `$ErrorActionPreference = 'Stop'`.
- 4-space indentation, no tabs.
- Verb-Noun cmdlet naming. `Get-`, `Set-`, `New-`, `Remove-`, `Invoke-`, `Test-`, `Wait-`, `Watch-`.
- `Write-Host` is acceptable and preferred over `Write-Output` in CI scripts (act_runner captures `Write-Host` cleanly; `Write-Output` pollutes function return values). Codified in [PSScriptAnalyzerSettings.psd1](../PSScriptAnalyzerSettings.psd1).
- No emoji in code, logs, webhook payloads, docs, or commit messages. Plain text markers like `[WARN]`, `[OK]`, `[x]`, `[ ]`.
- No PS 7+ syntax: no `??`, no `??=`, no ternary, no `&&`/`||` in PowerShell context, no `ForEach-Object -Parallel`, no `Test-Json`.
**File organization**:
- Production scripts under `scripts/`. Template provisioning under `template/`. Tests under `tests/`. Documentation under `docs/`. Plans and reviews under `plans/`.
- Module files `_Common.psm1` and `_Transport.psm1` stay in `scripts/`. Do not split further.
- One script, one verb-noun purpose. Composite actions live under `gitea/actions/<name>/action.yml`.
**Commit messages**:
- Imperative mood, present tense ("Add bounded vmrun wrapper", not "Added" or "Adding").
- Subject line ≤72 characters. Body explains *why*, not *what*.
- Reference the master plan checklist item by short title when relevant: e.g. `Fix C1: Linux matrix template selection`.
**Branching**:
- `main` is the only long-lived branch.
- Use short-lived feature branches `fix/c1-linux-matrix`, `fix/h2-action-outputs`, etc.
- Squash merge or rebase; do not preserve micro-commits.
**Testing requirements before merge**:
- After H10 is implemented: every PR runs the lint and Pester workflows. After the smoke-test workflow exists, run it for any change touching orchestration scripts.
**Documentation updates**:
- A change to a script that changes its parameters or behavior must update [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), or [docs/RUNBOOK.md](../docs/RUNBOOK.md) in the same commit.
- A change to [runner/config.yaml](../runner/config.yaml) or directory layout must update [AGENTS.md](../AGENTS.md) and [Setup-Host.ps1](../Setup-Host.ps1).
---
## Documentation Status
Per-document assessment.
| Document | Status | Action |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| [AGENTS.md](../AGENTS.md) | Good. Reflects current paths, PS 5.1 rules, error patterns. | Keep updated as new gotchas are discovered. |
| [README.md](../README.md) | Stale. Claims production-ready before stabilization pass completes. | Replace "production-ready" with "feature-complete; stabilization pass in progress" until all Critical and High items are checked. |
| [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md) | Mostly accurate; minor diagram rendering issue. | Reformat the ASCII diagram box. Reflect H4 bounded-vmrun pattern in the lifecycle section. |
| [docs/CI-FLOW.md](../docs/CI-FLOW.md) | Stale on partial-artifacts-on-failure. Still mentions `getGuestIPAddress` as primary IP detection. | Update IP-detection section to reflect `guestinfo.ci-ip` primary. Reconcile failure-scenario table with whether H-level diagnostics are implemented. |
| [docs/RUNBOOK.md](../docs/RUNBOOK.md) | Useful but incomplete. | Add: consolidated template-refresh procedure; capacity burn-in results; recovery from stale lock; runner service-account identity. |
| [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) | Good. Old `WinBuild` path references need fixing. | Update all `WinBuild` to `WinBuild2025`. Add HGFS shared-folder cache poisoning note. |
| [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Good. | Add one-paragraph note on Gitea secret-mask substring limitation. |
| [docs/HOST-SETUP.md](../docs/HOST-SETUP.md) | Mostly accurate. | Update directory list to match new [Setup-Host.ps1](../Setup-Host.ps1). Add ACL guidance for `F:\CI\keys\ci_linux`. |
| [docs/WINDOWS-TEMPLATE-SETUP.md](../docs/WINDOWS-TEMPLATE-SETUP.md) | Good. | Cross-reference the consolidated template-refresh runbook in [docs/RUNBOOK.md](../docs/RUNBOOK.md). |
| [docs/LINUX-TEMPLATE-SETUP.md](../docs/LINUX-TEMPLATE-SETUP.md) | Good. | Add `Validate-DeployState.ps1` Linux branch usage. |
| [docs/OPTIMIZATION.md](../docs/OPTIMIZATION.md) | Mostly accurate. | Note Defender exclusion option and shared-cache wiring after H-Medium fixes. |
| [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) | Stale. PS 7 syntax, wrong parameter names, wrong task/event names. | Sprint 5 task 5.3: PS 5.1 syntax, `-RepoUrl`, current Event Log sources. |
| [docs/Setup-GiteaSSH.md](../docs/Setup-GiteaSSH.md) | Untouched by reviewers; assume accurate. | Verify against fixed service-account path from H8. |
| [TODO.md](../TODO.md) | Acting as a CHANGELOG. | Sprint 5 task: move "done" sections into `docs/CHANGELOG.md`; keep TODO for genuinely outstanding work. |
| [docs/archived/2026-05-10/](../docs/archived/2026-05-10/) | Historical. | Leave as-is. |
---
## Final Completion Plan
Execute in this exact order. Each phase has a single success criterion that gates the next.
**Phase 0 — Read this plan and commit it.** Already done by the act of generating it.
**Phase 1 — Validation (Sprint 1).** Verify the act_runner service account, confirm the directory inventory, run the 4-way concurrency burn-in, manually trigger the broken Linux matrix to confirm C1. Output: a section appended to [docs/RUNBOOK.md](../docs/RUNBOOK.md) titled "Validation Results 2026-05-XX" documenting findings. Gate: burn-in passed three times (keep `capacity: 4`) or was set to 2 (with reasoning recorded).
**Phase 2 — Critical Fixes (Sprint 2).** Fix C1 and C2, plus H2/H3/H6/H7. Manually trigger [gitea/workflows/build-nsInnoUnp.yml](../gitea/workflows/build-nsInnoUnp.yml) for both matrix legs after each fix. Gate: both Critical items closed and `[x]` in this plan's Critical section; both legs green in the live matrix workflow.
**Phase 3 — High Priority Fixes (Sprint 3).** Complete the rest of the High items: H4 (bounded vmrun), H5 (stale lock), H8 (SSH alias fix), H9 (Linux validator), H10 (smoke-test workflow), H11 (script-root env var), H12 (benchmark alignment). Gate: smoke-test workflow green; all High items `[x]`.
**Phase 4 — Operational Hardening (Sprint 4).** All Medium items including consolidated template-refresh runbook. Gate: documented end-to-end template refresh executed successfully against a non-production template clone.
**Phase 5 — Polish (Sprint 5).** Low items, nice-to-have summary tool, documentation alignment, TODO → CHANGELOG migration. Gate: README claim updated from "feature-complete; stabilization pass in progress" back to "production-ready for homelab".
**Phase 6 — Tag v1.0.** Create an annotated tag `v1.0.0` on `main` once Phase 5 gate passes. Write a `docs/CHANGELOG.md` v1.0 entry summarizing the journey.
**Phase 7 — Operate.** Use the system. Track new findings in `TODO.md`. Run the burn-in any time act_runner capacity, host hardware, or template chains change materially.
When all Critical and High items are checked, this system is genuinely production-ready for homelab use.
Reference in New Issue View Git Blame Copy Permalink