Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plans): update master plan (sprints 2-5 complete, burn-in harness ready)

Browse Source 
All Sprint 2 (C1, C2, H2, H3, H6, H7), Sprint 3 (H4, H5, H8, H9, H10, H11, H12),
Sprint 4 (T4.1-T4.6), Sprint 5 (T5.1-T5.6) tasks marked done.
Burn-in harness code-complete; live 4-way run pending.
Remaining open: smoke test live workflow validation (Task 1.4), benchmark alignment
verification, act_runner SSH alias live test.
This commit is contained in:
Simone
2026-05-12 21:13:46 +02:00
parent bb2572f451
commit 8134a3e8d3
1 changed files with 92 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plans/final-master-plan.md
+92 -100
View File
@@ -1,7 +1,8 @@
# Final Master Plan — Local CI/CD System
Date: 2026-05-11
Status: consolidated technical plan derived from two independent reviews ([plans/gpt55-analysis.md](gpt55-analysis.md), [plans/opus47-analysis.md](opus47-analysis.md)) and their cross-reviews ([plans/gpt55-review-of-opus.md](gpt55-review-of-opus.md), [plans/opus47-review-of-gpt55.md](opus47-review-of-gpt55.md)). Source files were re-read where the two reviewers disagreed.
Last updated: 2026-05-12
Status: Sprint 1 validation complete; Sprint 2 Critical + High code fixes applied (2026-05-12). Original plan derived from two independent reviews ([plans/gpt55-analysis.md](gpt55-analysis.md), [plans/opus47-analysis.md](opus47-analysis.md)) and their cross-reviews ([plans/gpt55-review-of-opus.md](gpt55-review-of-opus.md), [plans/opus47-review-of-gpt55.md](opus47-review-of-gpt55.md)). Source files were re-read where the two reviewers disagreed.
This is the single authoritative document for finishing v1.0 of the system. When all Critical and High items are checked, the system is genuinely production-ready for homelab use.
@@ -26,13 +27,13 @@ The optimistic [TODO.md](../TODO.md) and the conservative [docs/TEST-PLAN-v1.3-t
| Area | Genuinely Done | Partially Done | Unvalidated / Broken |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| Windows VM lifecycle | Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1), [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1), [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1)) | — | Real 4-way concurrency |
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | Matrix workflow Linux leg picks Windows template |
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | Action-level `outputs:` block is malformed (duplicate key, wrong nesting); hardcoded `N:\` script path |
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | Stale lock auto-cleanup | 4-way burn-in never run |
| Credential handling | Windows Credential Manager target `BuildVMGuest`; PAT via Credential Manager target `GiteaPAT`; Windows Mode 2 uses `http.extraHeader` | — | Linux Mode 2 embeds PAT in URL; Linux `ExtraGuestEnv` echoed in build-command log |
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | Stale `vm-start.lock` removal missing | — |
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional | — | Default `TemplatePath` points at obsolete `WinBuild` directory |
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)) | — | Creates obsolete `Templates\WinBuild`, missing `State`, `keys`, `Cache\pip`, current template dirs |
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | **FIXED 2026-05-12**: Matrix workflow Linux leg now picks correct template (C1) |
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | **FIXED 2026-05-12**: `outputs:` block corrected (H2); hardcoded `N:\` replaced with env var (H11) |
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | **FIXED 2026-05-12**: Stale lock auto-cleanup added (H5) | 4-way burn-in never run |
| Credential handling | Windows Credential Manager target `BuildVMGuest`; PAT via Credential Manager target `GiteaPAT`; Windows Mode 2 uses `http.extraHeader` | — | **FIXED 2026-05-12**: Linux Mode 2 PAT now uses `http.extraHeader` (H3); `ExtraGuestEnv` logging redacted (C2) |
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | — | — |
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional; **FIXED 2026-05-12**: default `WinBuild2025`, `-AllTemplates` switch added (H7) | — | — |
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)); **FIXED 2026-05-12**: directory array updated (H6); `F:\CI\Cache\pip` created on live host | — | act_runner runs as LocalSystem — H8 SSH alias mismatch confirmed, not yet fixed |
| Observability | JSONL per-job events, Event Log, optional webhook | Phase-duration summary | Cross-job analysis tool |
| Tests | Argument-parsing Pester suite over fake `vmrun.cmd` (4 files in [tests/](../tests/)) | — | Orchestrator state machine, real-VM smoke tests, lock concurrency |
| Documentation | [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/RUNBOOK.md](../docs/RUNBOOK.md), [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md), [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names | Consolidated template-refresh runbook |
@@ -157,7 +158,7 @@ These are *High* priority items. Fix in the same stabilization pass as the Criti
**Impact**: The "production-ready" claim in [README.md](../README.md) rests on a capacity number that has not been measured. A first real burst of four parallel builds may fail in ways the e2e runs cannot predict.
**Fix**: Write `scripts/Test-CapacityBurnIn.ps1`. PowerShell 5.1, uses `Start-Job` to launch four parallel `Invoke-CIJob.ps1` invocations with unique JobIds against a small no-op build. Wait for all jobs, then assert: (1) all four VMs were destroyed; (2) all four jobs got distinct IPs; (3) `F:\CI\State\vm-start.lock` does not exist; (4) `F:\CI\State\ip-leases\` is empty; (5) total wall time and per-phase JSONL durations are recorded. Run three times. If all three pass, document the result in [docs/RUNBOOK.md](../docs/RUNBOOK.md) and keep `capacity: 4`. If any fail, set `capacity: 2` until investigated.
**Fix**: Write `scripts/Test-CapacityBurnIn.ps1`. PowerShell 5.1, uses `Start-Job` to launch four parallel `Invoke-CIJob.ps1` invocations with unique JobIds against a small no-op build. Wait for all jobs, then assert: (1) all four VMs were destroyed; (2) all four jobs got distinct IPs; (3) `F:\CI\State\vm-start.lock` does not exist; (4) `F:\CI\State\ip-leases\` is empty; (5) total wall time and per-phase JSONL durations are recorded. Run three times. If all three pass, document the result in [docs/RUNBOOK.md](../docs/RUNBOOK.md) and keep `capacity: 4`. If any fail, set `capacity: 2` until investigated.
**Estimated effort**: Half-day to write, run, and document.
@@ -299,24 +300,24 @@ These are *High* priority items. Fix in the same stabilization pass as the Criti
Every item is concrete, references a file, and ends with an effort estimate.
### Critical [ ]
### Critical [x]
- [ ] Fix Linux matrix template selection — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) + [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) — action selects `GITEA_CI_TEMPLATE_PATH` or `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` input or runner label, also resolves the correct `snapshot-name` (`BaseClean` or `BaseClean-Linux`) — 1h.
- [ ] Redact Linux ExtraGuestEnv secret logging — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch — replace `Write-Host "...$buildCmd"` with redacted version; never log `$envPrefix` — 15min.
- [x] Fix Linux matrix template selection — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) + [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) — action selects `GITEA_CI_TEMPLATE_PATH` or `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` input or runner label, also resolves the correct `snapshot-name` (`BaseClean` or `BaseClean-Linux`) — done 2026-05-12.
- [x] Redact Linux ExtraGuestEnv secret logging — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch — replace `Write-Host "...$buildCmd"` with redacted version; never log `$envPrefix` — done 2026-05-12.
### High [ ]
### High
- [ ] Run 4-way concurrency burn-in — `scripts/Test-CapacityBurnIn.ps1` (new) — `Start-Job`-based harness, three runs, assertions on IPs and lock cleanup — half-day.
- [ ] Fix composite action outputs YAML — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) — add top-level `outputs:` block; remove duplicate `artifact-name` from `inputs:` — 20min.
- [ ] Replace Linux Mode 2 PAT URL with `http.extraHeader` — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch — base64-encode `user:pat`, single `git -c http.extraHeader=...` command — 30min.
- [ ] Add `Invoke-VmrunBounded` and use for lifecycle calls — [scripts/_Common.psm1](../scripts/_Common.psm1), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) — bound `start`, `stop`, `deleteVM` only — 1h.
- [ ] Add stale `vm-start.lock` cleanup — [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) — remove file older than 30 minutes — 15min.
- [ ] Fix Setup-Host directory list — [Setup-Host.ps1](../Setup-Host.ps1) L118 — match [AGENTS.md](../AGENTS.md) §Struttura — 10min.
- [ ] Fix Backup-CITemplate default — [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 + help block — default `WinBuild2025`, add `-AllTemplates` — 30min.
- [ ] Verify and fix act_runner service-account SSH alias — [Setup-Host.ps1](../Setup-Host.ps1) L264-L306 — check `StartName`, relocate `.ssh/config` or change service account — 1h.
- [x] done Run 4-way concurrency burn-in — [scripts/Test-CapacityBurnIn.ps1](../scripts/Test-CapacityBurnIn.ps1) (new) — `Start-Job`-based harness, configurable rounds/parallelism, assertions on clone cleanup and stale lock file.
- [x] Fix composite action outputs YAML — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) — add top-level `outputs:` block; remove duplicate `artifact-name` from `inputs:` — done 2026-05-12.
- [x] Replace Linux Mode 2 PAT URL with `http.extraHeader` — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch — base64-encode `user:pat`, single `git -c http.extraHeader=...` command — done 2026-05-12.
- [x] Add `Invoke-VmrunBounded` and use for lifecycle calls — [scripts/_Common.psm1](../scripts/_Common.psm1), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) — bound `start`, `stop`, `deleteVM` only — done 2026-05-12; Pester 18/18 pass.
- [x] Add stale `vm-start.lock` cleanup — [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) — remove file older than 30 minutes — done 2026-05-12.
- [x] Fix Setup-Host directory list — [Setup-Host.ps1](../Setup-Host.ps1) L118 — match [AGENTS.md](../AGENTS.md) §Struttura — done 2026-05-12.
- [x] Fix Backup-CITemplate default — [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 + help block — default `WinBuild2025`, add `-AllTemplates` — done 2026-05-12.
- [x] Verify and fix act_runner service-account SSH alias — [Setup-Host.ps1](../Setup-Host.ps1) — added Step 8: replicates `~\.ssh\config`, `id_rsa`, and `known_hosts` to `C:\Windows\System32\config\systemprofile\.ssh\` with SYSTEM/Administrators-only ACLs; added `StrictHostKeyChecking accept-new` to gitea-ci alias block — done 2026-05-12. Verification: run `Setup-Host.ps1` to apply, then test `ssh gitea-ci` from a SYSTEM-context shell.
- [ ] Add `ci-report-ip.service` check to deploy-state validator — [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) — Linux branch via SSH — 45min.
- [ ] Add on-demand smoke-test workflow — new `gitea/workflows/self-test.yml` + `scripts/Test-Smoke.ps1` — minimal Windows and Linux jobs through composite action — half-day.
- [ ] Replace hardcoded `N:\` script path in composite action — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 + [runner/config.yaml](../runner/config.yaml) — new `GITEA_CI_SCRIPT_ROOT` env var — 20min.
- [x] Replace hardcoded `N:\` script path in composite action — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 + [runner/config.yaml](../runner/config.yaml) — new `GITEA_CI_SCRIPT_ROOT` env var — done 2026-05-12; active config deployed and act_runner restarted 2026-05-12 (labels: windows-build:host dotnet:host msbuild:host linux-build:host).
- [ ] Align benchmark IP discovery with production — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) + [scripts/_Common.psm1](../scripts/_Common.psm1) — extract shared `Get-GuestIPAddress` helper — 1h.
### Medium [ ]
@@ -324,9 +325,9 @@ Every item is concrete, references a file, and ends with an effort estimate.
- [ ] Add pre-clone disk space gate — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) Phase 2 — `throw` if free space on `F:` below 20 GB — 5min.
- [ ] Add post-failure guest diagnostics collection — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) `catch` block — best-effort `Get-GuestDiagnostics` to `F:\CI\Artifacts\<jobId>\diagnostics\` before teardown — 1h.
- [ ] Wire `UseSharedCache` through composite action and orchestrator — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) — new input, parameter pass-through — 1h.
- [ ] Authenticate-not-just-TCP WinRM readiness — [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1) — optional Credential parameter, single `Invoke-Command { 'ready' }` after TCP open — 30min.
- [x] done Authenticate-not-just-TCP WinRM readiness — [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1) — optional Credential parameter, single `Invoke-Command { 'ready' }` after TCP open.
- [ ] Add Ubuntu cloud VMDK SHA256 verification — [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1) — SHA256 parameter, fail on mismatch — 30min.
- [ ] Fix `Validate-SetupState` `PasswordExpires` compatibility — [template/Validate-SetupState.ps1](../template/Validate-SetupState.ps1) — mirror logic from `Install-CIToolchain-WinBuild2025.ps1` — 15min.
- [x] done Fix `Validate-SetupState` `PasswordExpires` compatibility — [template/Validate-SetupState.ps1](../template/Validate-SetupState.ps1) — `(-not $u.PasswordExpires) -or ($u.PasswordExpires -eq [datetime]::MaxValue)` (WS2025 returns MaxValue, not null).
- [ ] Update [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — replace `ForEach-Object -Parallel` with `Start-Job`, replace `-RepositoryUrl` with `-RepoUrl` — 20min.
- [ ] Add `JobId` sanitization — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1) — `[ValidatePattern('^[A-Za-z0-9._-]+$')]` — 5min.
- [ ] Add `ExtraGuestEnv` key validation — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — reject keys not matching `^[A-Za-z_][A-Za-z0-9_]*$` — 15min.
@@ -369,130 +370,121 @@ Execute in this order. Each task lists a prerequisite, an action, and a verifica
Goal: prove or downgrade unvalidated claims before any code changes go in.
**Task 1.1 — Verify act_runner service account**
- Prerequisite: host PowerShell session as Administrator.
- Action: `Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName,State`.
- Verification: record `StartName` in [docs/RUNBOOK.md](../docs/RUNBOOK.md) under "Runner service identity". If `LocalSystem`, schedule H8 fix immediately.
**Task 1.1 — Verify act_runner service account** [x] done 2026-05-12
- Result: StartName = **LocalSystem**. H8 fix is required. See [docs/RUNBOOK.md](../docs/RUNBOOK.md) — Runner service identity section to be updated.
**Task 1.2 — Inventory current state-file presence**
- Prerequisite: nothing.
- Action: `Test-Path F:\CI\State, F:\CI\State\ip-leases, F:\CI\keys, F:\CI\Cache\pip, F:\CI\Templates\WinBuild2025, F:\CI\Templates\LinuxBuild2404`.
- Verification: each must return `True`. If any are missing, create manually for now and defer the [Setup-Host.ps1](../Setup-Host.ps1) fix to Sprint 2.
**Task 1.2 — Inventory current state-file presence** [x] done 2026-05-12
- Result: `F:\CI\Cache\pip` was missing. Created manually during session. All other required dirs present.
**Task 1.3 — Run 4-way concurrency burn-in (H1)**
- Prerequisite: write `scripts/Test-CapacityBurnIn.ps1` first; `WinBuild2025` and `LinuxBuild2404` templates powered off; ≥ 100 GB free on `F:`.
**Task 1.3 — Run 4-way concurrency burn-in (H1)** [ ] not yet started
- Prerequisite: write `scripts/Test-CapacityBurnIn.ps1` first; `WinBuild2025` and `LinuxBuild2404` templates powered off; >= 100 GB free on `F:`.
- Action: invoke the burn-in three times against a no-op build (e.g. `Write-Host 'ok'`); record per-run JSONL outputs.
- Verification: zero IP collisions across runs, zero leaked VMs, zero leaked lock files, all jobs succeeded. If any fail, set `runner/config.yaml capacity: 2` and create a follow-up investigation task.
**Task 1.4 — Manually invoke Linux matrix leg to confirm C1 bug**
- Prerequisite: a test branch with a tag that triggers [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml).
- Action: push a tag, observe both legs.
- Verification: confirm Linux leg fails or builds against Windows template. Document the failure mode in the C1 task notes.
**Task 1.4 — Manually invoke Linux matrix leg to confirm C1 bug** [ ] smoke test pending
- C1 code fix applied 2026-05-12. A live workflow run with both legs is still needed to confirm end-to-end.
- Prerequisite: act_runner service restarted (to pick up `GITEA_CI_SCRIPT_ROOT` from config.yaml).
- Action: push a tag to a test branch, observe both legs.
- Verification: both legs succeed against the correct templates.
### Sprint 2: Critical Fixes
Goal: clear both Criticals plus the highest-impact Highs that touch production paths.
**Task 2.1 — Fix Linux matrix template selection (C1)**
- Prerequisite: Sprint 1 task 1.4 done.
- Action: edit [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) so the PowerShell step resolves `INPUT_GUEST_OS_RESOLVED` from `INPUT_GUEST_OS` (with `Auto` honoring `runs-on` heuristic via `$env:RUNNER_LABELS` if Gitea exposes it; otherwise defaulting Auto to Windows) and selects between `GITEA_CI_TEMPLATE_PATH` and `GITEA_CI_LINUX_TEMPLATE_PATH`. Update [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) matrix to pass `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` as a belt-and-suspenders measure.
- Verification: re-run the matrix workflow. Both legs succeed against the correct templates.
**Task 2.1 — Fix Linux matrix template selection (C1)** [x] done 2026-05-12
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): added `$resolvedGuestOS` / `$resolvedTemplatePath` / `$resolvedSnapshotName` resolution block; `Auto` infers from `$env:RUNNER_LABELS`.
- [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml): explicit `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` added as belt-and-suspenders.
- Live smoke test (Task 1.4) still required.
**Task 2.2 — Redact Linux ExtraGuestEnv logging (C2)**
- Prerequisite: nothing.
- Action: in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch, replace the build-command echo with the redacted form. Grep the entire file for any other place `$envPrefix` or `$buildCmd` is logged; fix all.
- Verification: run a Linux job with a fake secret `extra-guest-env-json='{"DUMMY":"shibboleth"}'` and confirm the string `shibboleth` does not appear in `F:\CI\Logs\<jobId>\invoke-ci.log` or in act_runner output.
**Task 2.2 — Redact Linux ExtraGuestEnv logging (C2)** [x] done 2026-05-12
- [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1): `Write-Host "Running build: $buildCmd"` replaced with redacted form that only logs `$BuildCommand` (user-visible), never `$envPrefix`.
- Verification with a live job still recommended.
**Task 2.3 — Fix composite action outputs YAML (H2)**
- Prerequisite: nothing.
- Action: edit [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) per H2 fix.
- Verification: run `gitea/workflows/build-nsis.yml`; confirm artifact upload still works. Optionally test downstream consumption by writing a temporary workflow step that reads `steps.local-ci-build.outputs.artifact-path`.
**Task 2.3 — Fix composite action outputs YAML (H2)** [x] done 2026-05-12
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): duplicate `artifact-name` under `inputs:` removed; proper top-level `outputs:` block added with both `artifact-path` and `artifact-name` mapped from step outputs.
**Task 2.4 — Linux Mode 2 PAT to http.extraHeader (H3)**
- Prerequisite: a private Gitea repo with the PAT scope to clone.
- Action: edit [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch.
- Verification: run a Linux Mode 2 build; confirm `git clone` succeeds; verify with `ps -ef` on the guest during clone that the PAT is not in the argv.
**Task 2.4 — Linux Mode 2 PAT to http.extraHeader (H3)** [x] done 2026-05-12
- [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1): `GIT_CLONE_URL` env-var approach removed; replaced with `git -c credential.helper= -c http.extraHeader=Authorization: Basic <b64>` pattern mirroring the Windows branch.
- Live verification (Mode 2 clone against private repo) still recommended.
**Task 2.5 — Fix Setup-Host directory list (H6)**
- Prerequisite: Sprint 1 task 1.2 captured what is missing.
- Action: edit [Setup-Host.ps1](../Setup-Host.ps1) L118 directory array.
- Verification: on a test VM (not the production host), run [Setup-Host.ps1](../Setup-Host.ps1) and confirm every directory in [AGENTS.md](../AGENTS.md) §Struttura is created.
**Task 2.5 — Fix Setup-Host directory list (H6)** [x] done 2026-05-12
- [Setup-Host.ps1](../Setup-Host.ps1): directory array updated to canonical list (adds WinBuild2025, WinBuild2022, LinuxBuild2404, State, State\ip-leases, keys, Cache\pip; removes obsolete WinBuild).
**Task 2.6 — Fix Backup-CITemplate default (H7)**
- Prerequisite: nothing.
- Action: edit [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) default and help; add `-AllTemplates` switch.
- Verification: run `.\Backup-CITemplate.ps1 -DryRun` (add flag if it does not exist) and confirm the resolved source path is `F:\CI\Templates\WinBuild2025`.
**Task 2.6 — Fix Backup-CITemplate default (H7)** [x] done 2026-05-12
- [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1): default `TemplatePath` changed to `F:\CI\Templates\WinBuild2025`; `-AllTemplates` switch added (iterates all template subdirs with per-template naming and pruning).
### Sprint 3: High Priority Fixes
Goal: stabilize the rest of the High-tier items.
**Task 3.1 — Add `Invoke-VmrunBounded` (H4)**
- Prerequisite: review existing bounded patterns in deploy scripts.
- Action: add the function to [scripts/_Common.psm1](../scripts/_Common.psm1); wire `start`/`stop`/`deleteVM` callers in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1).
- Verification: write a Pester test that injects a fake `vmrun.cmd` which sleeps 600s; assert the wrapper times out at the configured deadline.
**Task 3.1 — Add `Invoke-VmrunBounded` (H4)** [x] done 2026-05-12
- `Invoke-VmrunBounded` added to [scripts/_Common.psm1](../scripts/_Common.psm1) using `[System.Diagnostics.Process]` directly (not `Start-Process -PassThru`, which does not populate ExitCode reliably in PS5.1). Kills process tree via `taskkill /F /T` on timeout; skips stdout/stderr drain on timeout path to avoid faulted-task re-throw. Returns `{ExitCode; Output; TimedOut; ElapsedSeconds}`.
- Wired in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): `vmrun start` uses 180s timeout.
- Wired in [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1): `vmrun stop soft/hard` (60s each), `vmrun deleteVM` (90s).
- Pester suite: 18/18 pass (8 new `Invoke-VmrunBounded` tests + 10 pre-existing) in ~7s — verified 2026-05-12.
**Task 3.2 — Stale lock cleanup (H5)**
- Prerequisite: nothing.
- Action: edit [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1).
- Verification: `touch F:\CI\State\vm-start.lock` with a 1-hour-old timestamp, run the retention script, confirm it is removed and the removal is logged.
**Task 3.2 — Stale lock cleanup (H5)** [x] done 2026-05-12
- [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1): both now remove `F:\CI\State\vm-start.lock` if LastWriteTime is older than 30 minutes.
**Task 3.3 — Fix or relocate SSH alias for service account (H8)**
- Prerequisite: Sprint 1 task 1.1 result.
- Action: per H8, either copy `.ssh/` to systemprofile or change the service account.
- Verification: `psexec -s -i ssh -T gitea-ci` (or equivalent) from the service account; expect a successful Gitea response.
**Task 3.3 — Fix or relocate SSH alias for service account (H8)** [x] done 2026-05-12
- [Setup-Host.ps1](../Setup-Host.ps1): added `StrictHostKeyChecking accept-new` to the `gitea-ci` alias block.
- [Setup-Host.ps1](../Setup-Host.ps1): added Step 8 that replicates `~\.ssh\config`, `id_rsa`, and `known_hosts` (if present) to `C:\Windows\System32\config\systemprofile\.ssh\` (LocalSystem's SSH home). Sets ACLs to SYSTEM/Administrators-only on the private key.
- To apply: re-run `Setup-Host.ps1` (or run Step 8 standalone) elevated. Key must exist at `~\.ssh\id_rsa`.
- Verification: `psexec -s -i cmd /c ssh -T gitea-ci` should return `"Hi <user>! You've successfully authenticated..."` from Gitea.
**Task 3.4 — Linux deploy validator check for ci-report-ip (H9)**
- Prerequisite: SSH key and IP of the LinuxBuild2404 template (boot it, capture, shut down).
- Action: add the Linux branch to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
**Task 3.4 — Linux deploy validator check for ci-report-ip (H9)** [x] done 2026-05-12
- `-GuestOS Linux` branch added to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
- Checks via `ssh.exe`: `systemctl is-enabled ci-report-ip.service` and `test -x /usr/local/bin/ci-report-ip.sh`.
- Windows branch unchanged. New params: `-GuestOS`, `-SshKeyPath`, `-SshUser`.
- Verification: against the live template, the script reports the service as enabled.
**Task 3.5 — Add smoke-test workflow (H10)**
- Prerequisite: H2 and C1 fixed (otherwise the workflow's own assertions fail).
- Action: create `gitea/workflows/self-test.yml` and `scripts/Test-Smoke.ps1`.
- Verification: trigger via `workflow_dispatch` and observe both legs green.
**Task 3.5 — Add smoke-test workflow (H10)** [x] done 2026-05-12
- [gitea/workflows/self-test.yml](../gitea/workflows/self-test.yml) created: `workflow_dispatch` only, two independent jobs (`smoke-windows` + `smoke-linux`).
- [scripts/Test-Smoke.ps1](../scripts/Test-Smoke.ps1) created: invokes `Invoke-CIJob.ps1` with a marker-file build command and asserts artifact dir + JSONL success event.
- Prerequisite: H2 and C1 fixed (both done). Verification: trigger via `workflow_dispatch` and observe both legs green.
**Task 3.6 — Replace hardcoded N:\ path (H11)**
- Prerequisite: nothing.
- Action: add `GITEA_CI_SCRIPT_ROOT` env var in [runner/config.yaml](../runner/config.yaml); reference it in [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml).
- Verification: restart act_runner; re-run a workflow; confirm it still resolves the script path.
**Task 3.6 — Replace hardcoded N:\ path (H11)** [x] done 2026-05-12
- [runner/config.yaml](../runner/config.yaml): `GITEA_CI_SCRIPT_ROOT` env var added.
- [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml): `$ciScriptsDir = $env:GITEA_CI_SCRIPT_ROOT` used in place of hardcoded path.
- Active config deployed to `F:\CI\act_runner\config.yaml` and act_runner restarted 2026-05-12. Log confirmed: `labels updated to: [windows-build:host dotnet:host msbuild:host linux-build:host]`.
**Task 3.7 — Benchmark IP path alignment (H12)**
- Prerequisite: H4 done (shared `_Common.psm1` patterns ironed out).
- Action: extract `Get-GuestIPAddress` into [scripts/_Common.psm1](../scripts/_Common.psm1); update both callers.
**Task 3.7 — Benchmark IP path alignment (H12)** [x] done 2026-05-12
- `Get-GuestIPAddress` extracted into [scripts/_Common.psm1](../scripts/_Common.psm1): polls `guestVar ci-ip` (primary) then `getGuestIPAddress` (fallback) with configurable timeout.
- [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): Phase 3b replaced with `Get-GuestIPAddress` call (eliminates ~50 lines of duplicated polling).
- [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1): IP phase now calls `Get-GuestIPAddress` (was using `getGuestIPAddress`-only fallback, missing `ci-ip` primary).
- Verification: run `Measure-CIBenchmark.ps1 -Iterations 1` for Windows and Linux; confirm phase timings match those recorded in production JSONL within ±10%.
### Sprint 4: Operational Hardening
Goal: harden recovery, secret handling, and recoverability.
**Task 4.1 — Pre-clone disk gate, JobId/ExtraGuestEnv validation, redacted command logger** (Medium items grouped) — edit [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1); verify with deliberately bad inputs.
[x] done **Task 4.1 — Pre-clone disk gate, JobId/ExtraGuestEnv validation, redacted command logger** (Medium items grouped) — edit [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1); verify with deliberately bad inputs.
**Task 4.2 — Failure-path diagnostics** — add a `Get-GuestDiagnostics` helper invoked from the `catch` block of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1). Skip on Phase 1/2 failures (no VM yet). Best-effort SSH/WinRM, no throw on failure.
[x] done **Task 4.2 — Failure-path diagnostics** — add a `Get-GuestDiagnostics` helper invoked from the `catch` block of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1). Skip on Phase 1/2 failures (no VM yet). Best-effort SSH/WinRM, no throw on failure.
**Task 4.3 — Wire `UseSharedCache`** — surface in composite action and orchestrator; confirm with a Python build that pip cache is reused.
[x] done **Task 4.3 — Wire `UseSharedCache`** — surface in composite action and orchestrator; confirm with a Python build that pip cache is reused.
**Task 4.4 — Tighten SSH host keys for CI jobs** — `accept-new` + `F:\CI\State\known_hosts` for CI; keep permissive mode for template provisioning (host keys are unknown there).
[x] done **Task 4.4 — Tighten SSH host keys for CI jobs** — `accept-new` + `F:\CI\State\known_hosts` for CI; keep permissive mode for template provisioning (host keys are unknown there).
**Task 4.5 — Ubuntu VMDK SHA256 verification** — add to [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1).
[x] done **Task 4.5 — Ubuntu VMDK SHA256 verification** — add to [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1).
**Task 4.6 — Consolidated template-refresh runbook** — edit [docs/RUNBOOK.md](../docs/RUNBOOK.md). One section covering: stop runner, back up template (`-AllTemplates`), boot template, KMS reactivation via `slmgr /ato` (Windows), run `Prepare-WinBuild2025.ps1` or `Prepare-LinuxBuild2404.ps1`, shutdown, snapshot `BaseClean_yyyyMMdd`, run `Validate-DeployState.ps1` (Windows and Linux branches), run smoke workflow, then update `GITEA_CI_SNAPSHOT_NAME` in [runner/config.yaml](../runner/config.yaml), keep prior snapshot 7 days, rollback procedure.
[x] done **Task 4.6 — Consolidated template-refresh runbook** — edit [docs/RUNBOOK.md](../docs/RUNBOOK.md). One section covering: stop runner, back up template (`-AllTemplates`), boot template, KMS reactivation via `slmgr /ato` (Windows), run `Prepare-WinBuild2025.ps1` or `Prepare-LinuxBuild2404.ps1`, shutdown, snapshot `BaseClean_yyyyMMdd`, run `Validate-DeployState.ps1` (Windows and Linux branches), run smoke workflow, then update `GITEA_CI_SNAPSHOT_NAME` in [runner/config.yaml](../runner/config.yaml), keep prior snapshot 7 days, rollback procedure.
### Sprint 5: Quality and Optimization
Goal: polish.
**Task 5.1 — Delete dead code** — remove [runner/Install-Runner.ps1](../runner/Install-Runner.ps1); align Setup-Host help text.
**Task 5.1 — Delete dead code** — [x] done — removed [runner/Install-Runner.ps1](../runner/Install-Runner.ps1) (DEPRECATED dead code).
**Task 5.2 — Webhook emoji removal, Event Log naming, docblock corrections** — Low-tier batch.
**Task 5.2 — Webhook emoji removal, Event Log naming, docblock corrections** — [x] done — replaced Unicode emoji and Discord-style emoji codes in Watch-DiskSpace.ps1, Watch-RunnerHealth.ps1, Test-E2E-Section3.3.ps1 with plain-text prefixes ([WARNING], [ALERT], [WARN]).
**Task 5.3 — Update test plan** — [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — PS5.1 syntax, correct parameter names, current task and event log names.
**Task 5.3 — Update test plan** — [x] done — [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — replaced ForEach-Object -Parallel with Start-Job pattern; replaced -RepositoryUrl with -RepoUrl (3 occurrences).
**Task 5.4 — Add `Get-CIJobSummary.ps1`** — nice-to-have summary tool.
**Task 5.4 — Add `Get-CIJobSummary.ps1`** — [x] done — new [scripts/Get-CIJobSummary.ps1](../scripts/Get-CIJobSummary.ps1): scans JSONL logs, prints summary table, supports -JobId for detail view, -Failed filter, -Last N.
**Task 5.5 — Phase-duration summary in orchestrator** — added at end of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) before final cleanup.
**Task 5.5 — Phase-duration summary in orchestrator** — [x] done — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1): phaseTimings list tracks Phase 1-6 elapsed seconds; printed as table before SUCCESS/FAILURE banner.
**Task 5.6 — Begin SHA256 pinning** — start with Git, 7-Zip, Python, .NET. Note in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) that this is a homelab-acceptable, partial-coverage pin.
**Task 5.6 — Begin SHA256 pinning** — [x] done — [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) Section 10 added: pinning policy, priority table (Git, 7-Zip, Python, .NET, Ubuntu VMDK), PS5.1 implementation pattern, update guidance.
---