docs(plans): add multi-agent technical review and final master plan
- plans/gpt55-analysis.md: independent analysis by GPT-5.5 - plans/opus47-analysis.md: independent analysis by Opus 4.7 - plans/gpt55-review-of-opus.md: GPT-5.5 cross-review of Opus analysis - plans/opus47-review-of-gpt55.md: Opus 4.7 cross-review of GPT-5.5 analysis - plans/final-master-plan.md: consolidated master plan (8129 words) Severity: 2 Critical, 12 High, 16 Medium, 10 Low, 6 Nice-to-have. Top priorities: C2 secret logging in Invoke-RemoteBuild.ps1, C1 Linux matrix template selection, H1 capacity=4 burn-in.
This commit is contained in:
Simone
@@ -0,0 +1,740 @@
|
||||
# Final Master Plan — Local CI/CD System
|
||||
|
||||
Date: 2026-05-11
|
||||
Status: consolidated technical plan derived from two independent reviews ([plans/gpt55-analysis.md](gpt55-analysis.md), [plans/opus47-analysis.md](opus47-analysis.md)) and their cross-reviews ([plans/gpt55-review-of-opus.md](gpt55-review-of-opus.md), [plans/opus47-review-of-gpt55.md](opus47-review-of-gpt55.md)). Source files were re-read where the two reviewers disagreed.
|
||||
|
||||
This is the single authoritative document for finishing v1.0 of the system. When all Critical and High items are checked, the system is genuinely production-ready for homelab use.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The Local CI/CD System is a well-engineered, single-host homelab CI built on Gitea Actions, act_runner v1.0.2, VMware Workstation linked clones, Windows PowerShell 5.1, WinRM HTTPS for Windows guests and SSH for Linux guests. The architecture is the right shape for the constraint set: a thin orchestrator on the host, ephemeral linked clones for isolation, two transports for two guest families, and file-based state for everything else. Both reviewers independently graded the architecture A- and operational maturity B-/C+. That matches what the source shows.
|
||||
|
||||
The system is *feature-complete* and *operator-usable*, not yet *unattended-daily-trustworthy*. The gap is small in code volume but real in operational consequence. Five concrete defects can break a live build today: the Linux matrix leg of [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) silently selects the Windows template; Linux `ExtraGuestEnv` secrets are logged in clear text; the composite action outputs metadata is malformed YAML; the Linux Mode 2 PAT injection writes credentials into a remote command string; [Setup-Host.ps1](../Setup-Host.ps1) creates a directory layout that no longer matches the live system. None of these are architectural failures. Each is a 5-to-45-minute fix that the design otherwise supports cleanly.
|
||||
|
||||
The system makes one large unvalidated claim: `capacity: 4` in [runner/config.yaml](../runner/config.yaml). The IP-allocation mutex and lease design in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) is sound on paper, but the only evidence is single-job e2e runs. Four concurrent real builds have not been measured. This is the single most consequential outstanding question. The remediation is a burn-in, not a redesign.
|
||||
|
||||
Top 3 risks today, in order: (1) the Linux matrix leg builds against the Windows template until the action wiring is fixed; (2) any workflow that uses `extra-guest-env-json` for secrets is leaking those values to Gitea logs on Linux builds; (3) `capacity: 4` is a documented capability the system has never proven it keeps. Everything else is smaller.
|
||||
|
||||
---
|
||||
|
||||
## Current Project State
|
||||
|
||||
The optimistic [TODO.md](../TODO.md) and the conservative [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) disagree. The honest state, reconciled against the source files, is below.
|
||||
|
||||
| Area | Genuinely Done | Partially Done | Unvalidated / Broken |
|
||||
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
|
||||
| Windows VM lifecycle | Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1), [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1), [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1)) | — | Real 4-way concurrency |
|
||||
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | Matrix workflow Linux leg picks Windows template |
|
||||
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | Action-level `outputs:` block is malformed (duplicate key, wrong nesting); hardcoded `N:\` script path |
|
||||
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | Stale lock auto-cleanup | 4-way burn-in never run |
|
||||
| Credential handling | Windows Credential Manager target `BuildVMGuest`; PAT via Credential Manager target `GiteaPAT`; Windows Mode 2 uses `http.extraHeader` | — | Linux Mode 2 embeds PAT in URL; Linux `ExtraGuestEnv` echoed in build-command log |
|
||||
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | Stale `vm-start.lock` removal missing | — |
|
||||
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional | — | Default `TemplatePath` points at obsolete `WinBuild` directory |
|
||||
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)) | — | Creates obsolete `Templates\WinBuild`, missing `State`, `keys`, `Cache\pip`, current template dirs |
|
||||
| Observability | JSONL per-job events, Event Log, optional webhook | Phase-duration summary | Cross-job analysis tool |
|
||||
| Tests | Argument-parsing Pester suite over fake `vmrun.cmd` (4 files in [tests/](../tests/)) | — | Orchestrator state machine, real-VM smoke tests, lock concurrency |
|
||||
| Documentation | [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/RUNBOOK.md](../docs/RUNBOOK.md), [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md), [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names | Consolidated template-refresh runbook |
|
||||
|
||||
The README is genuinely ahead of the evidence. The accurate label is *feature-complete homelab CI awaiting stabilization pass*. Everything in this plan exists to close that gap.
|
||||
|
||||
---
|
||||
|
||||
## Realistic Goals
|
||||
|
||||
This system should aim to be:
|
||||
|
||||
- A reliable single-host homelab CI/CD that runs builds inside ephemeral, isolated VMware Workstation linked clones, triggered from a self-hosted Gitea.
|
||||
- Operable by one person, from one workstation, against private repositories that they trust.
|
||||
- Capable of clean recovery from host reboot, VMware crash, hung `vmrun`, and stale lock files without manual intervention beyond what is documented.
|
||||
- Capable of producing repeatable Windows and Linux builds with cached toolchains, with artifacts collected and published to Gitea.
|
||||
- Observable enough that after a failed build the operator can diagnose whether it was their build script, the runner, the host, or the VM.
|
||||
|
||||
This system should NOT aim to be:
|
||||
|
||||
- A multi-host or multi-tenant CI. Federation is explicitly out of scope.
|
||||
- A platform for running untrusted third-party code. The guest VMs are hardened only enough to be useful, not enough to safely execute hostile builds.
|
||||
- A general-purpose container orchestrator. Containers are deliberately not in the picture (see *Docker / Container Notes*).
|
||||
- A drop-in replacement for paid CI services like GitHub Actions cloud, Azure DevOps, or CircleCI. The threat model, secret handling, and audit story are all homelab-grade.
|
||||
- A PowerShell 7 project. The PS 5.1 mandate in [AGENTS.md](../AGENTS.md) is a deliberate constraint; migration is not on the roadmap.
|
||||
|
||||
Anything in this plan that drifts beyond these bounds is rejected. Both reviewers tested clean here; no Kubernetes, no Prometheus, no service mesh, no enterprise vault leaked into the consolidated plan.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
The current architecture is the right one. Confirm it as the target shape rather than redesign it. The recommendations below are scoped, additive fixes inside the existing architecture.
|
||||
|
||||
**VM lifecycle approach: keep as-is.** Linked clones from a `BaseClean` snapshot, lifecycle owned by [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) in a try/finally, sub-scripts pure and parameter-driven. Add a `Invoke-VmrunBounded` wrapper into [scripts/_Common.psm1](../scripts/_Common.psm1) and use it for the three lifecycle calls (`start`, `stop`, `deleteVM`). Do not bound `list`, `readVariable`, or `getGuestIPAddress` — those are short and the wrapper overhead is wasted there.
|
||||
|
||||
**Transport layer organization: keep explicit Windows/Linux branches.** Do not collapse WinRM and SSH into a generic transport interface. Branches in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) and [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1) are debuggable at 2 AM in a way that an abstraction is not. Add three small helpers in [scripts/_Common.psm1](../scripts/_Common.psm1) instead: `Invoke-VmrunBounded` (timeouts), `ConvertTo-SafeShellSingleQuotedString` (Linux quoting), and `Write-CIRedactedCommand` (secret-aware log formatter).
|
||||
|
||||
**IP allocation strategy: keep the lock+lease design.** The Opus suggestion of a static IP pool with `cloud-init`/`netsh` injection is a sound design, but it is a phase-3 redesign justified only if the burn-in fails. Run the burn-in first. The current design is: serialize clone+start+IP-detect through `F:\CI\State\vm-start.lock`, read `guestinfo.ci-ip` as the primary channel with `getGuestIPAddress` fallback, write a per-IP lease file. After fixing stale-lock cleanup and verifying under four-way load, this is sufficient for `capacity: 4`.
|
||||
|
||||
**Credential management: keep Windows Credential Manager + host-side SSH key file.** No vault, no rotation service. The fixes are: (1) verify the act_runner service account actually has access to the Credential Manager target; (2) confirm `F:\CI\keys\ci_linux` has ACLs restricting access to the runner identity and Administrators; (3) document a manual rotation procedure in [docs/RUNBOOK.md](../docs/RUNBOOK.md).
|
||||
|
||||
**Module organization: keep [scripts/_Common.psm1](../scripts/_Common.psm1) and [scripts/_Transport.psm1](../scripts/_Transport.psm1) split.** Do not rename or merge. Add the three helpers listed above to `_Common.psm1`. Keep `_Transport.psm1` as the SSH-specific helper despite the generic name; rename is churn.
|
||||
|
||||
**Logging/observability design: keep JSONL + Event Log + webhook.** Do not move to SQLite. Add a 40-line `Get-CIJobSummary.ps1` over the existing JSONL files for cross-job analysis. Add a phase-duration table printed at job end inside [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1).
|
||||
|
||||
---
|
||||
|
||||
## Recommended Stack
|
||||
|
||||
Canonical stack, after the fixes in this plan:
|
||||
|
||||
- Windows 11 Pro host, i9-10900X, 64 GB RAM, NVMe SSD.
|
||||
- Windows PowerShell 5.1 only. No PowerShell 7 dependency in any production script.
|
||||
- VMware Workstation Pro, `vmrun.exe` at `C:\Program Files (x86)\VMware\VMware Workstation\`.
|
||||
- Built-in Windows 11 OpenSSH client (`ssh.exe`, `scp.exe`) for Linux guests.
|
||||
- act_runner v1.0.2 installed as a Windows service via NSSM, capacity 4.
|
||||
- Gitea at `http://10.10.20.11:3100` / `https://gitea.emulab.it`, PAT stored in Credential Manager target `GiteaPAT`.
|
||||
- Templates: `WinBuild2025` (primary Windows), `WinBuild2022` (secondary Windows), `LinuxBuild2404` (Ubuntu 24.04 cloud image).
|
||||
- WinRM HTTPS over port 5986 with Basic authentication and self-signed certificate. `AllowUnencrypted=false`.
|
||||
- SSH user `ci_build` on Linux guests with passwordless sudo, key at `F:\CI\keys\ci_linux`.
|
||||
|
||||
Add to the stack:
|
||||
|
||||
- `Pester` v5 (already present in tests/).
|
||||
- `PSScriptAnalyzer` (already used by the lint workflow).
|
||||
|
||||
Remove from the stack:
|
||||
|
||||
- [runner/Install-Runner.ps1](../runner/Install-Runner.ps1). Marked deprecated; superseded by [Setup-Host.ps1](../Setup-Host.ps1). Delete the file.
|
||||
- The obsolete `F:\CI\Templates\WinBuild` directory if it still exists on the host. Replaced by `WinBuild2025` / `WinBuild2022`.
|
||||
|
||||
Do not add:
|
||||
|
||||
- Docker / Podman / container runtimes. See *Docker / Container Notes*.
|
||||
- PowerShell 7.
|
||||
- Prometheus, Grafana, Loki, OpenTelemetry collectors, or any external metrics stack.
|
||||
- SQLite or any other database engine.
|
||||
- HashiCorp Vault, Azure Key Vault, or any external secret store.
|
||||
|
||||
---
|
||||
|
||||
## Critical Problems
|
||||
|
||||
These must be fixed before the system is trustworthy for unattended daily use. Severity *Critical* means the issue actively breaks production today.
|
||||
|
||||
### C1. Linux matrix leg of build-nsis.yml selects the Windows template
|
||||
|
||||
**Description**: [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) runs a matrix over `[windows, linux]` and dispatches on `runs-on: ${{ matrix.target }}-build`, but does NOT pass `guest-os` or `template-path` to the composite action. The action defaults `template-path` to empty, so [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) reads `$env:GITEA_CI_TEMPLATE_PATH` — the Windows path. The orchestrator never reads `GITEA_CI_LINUX_TEMPLATE_PATH`. Result: the Linux leg clones `WinBuild2025` and tries to SSH into a Windows VM (or via Auto-detect, into a Windows VM with WinRM credentials), which fails or produces nonsense.
|
||||
|
||||
**Location**: [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) L20-L40; [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L77-L83; [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L108.
|
||||
|
||||
**Impact**: The advertised Windows+Linux matrix workflow is broken for the Linux leg. The only evidence of "Linux works" is direct invocation of `Invoke-CIJob.ps1` with explicit `-TemplatePath`. End-to-end through the workflow has never worked for Linux.
|
||||
|
||||
**Fix**: Either (a) update the composite action to select between `GITEA_CI_TEMPLATE_PATH` and `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` (or runner label inference) and similarly for snapshot name; or (b) update [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) to pass `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` plus an explicit `template-path`. Option (a) is the better one because it makes the action self-sufficient for any caller. Add an `INPUT_GUEST_OS_RESOLVED` step that selects the template-path env var, then pass it as `-TemplatePath`. Also set `BaseClean-Linux` as the snapshot name on Linux.
|
||||
|
||||
**Estimated effort**: 1 hour including a manual smoke test of both legs.
|
||||
|
||||
### C2. Linux ExtraGuestEnv secrets are logged in clear text
|
||||
|
||||
**Description**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch builds `$buildCmd = "${envPrefix}cd '$GuestLinuxWorkDir' && $BuildCommand"` where `$envPrefix` contains `export KEY='value'; ...` with secret values inlined. The next line emits `Write-Host "[Invoke-RemoteBuild] Running build: $buildCmd"`. act_runner captures stdout and Gitea masks `secrets.*` by string match only; multi-byte transforms, base64 encoding, or substring overlap defeat the mask. The documented intended use case (`extra-guest-env-json: '{"SIGN_PASS":"${{ secrets.SIGN_PASS }}"}'`) makes this an active leak path.
|
||||
|
||||
**Location**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux build assembly, around the `Write-Host` immediately preceding `Invoke-SshCommand` for the build step.
|
||||
|
||||
**Impact**: Any homelab workflow currently using `extra-guest-env-json` for secrets has those secrets in Gitea job logs and in the local `F:\CI\Logs\<jobId>\invoke-ci.log` transcript.
|
||||
|
||||
**Fix**: Replace the `Write-Host "Running build: $buildCmd"` line with `Write-Host "[Invoke-RemoteBuild] Running build (env vars redacted): cd '$GuestLinuxWorkDir' && <build-command>"`. Never log `$envPrefix` or any string containing it. Do the same redaction in any other location where `$buildCmd` or `$envPrefix` is logged.
|
||||
|
||||
**Estimated effort**: 15 minutes.
|
||||
|
||||
---
|
||||
|
||||
## Secondary Problems
|
||||
|
||||
These are *High* priority items. Fix in the same stabilization pass as the Criticals.
|
||||
|
||||
### H1. `capacity: 4` is unvalidated under real concurrency
|
||||
|
||||
**Description**: [runner/config.yaml](../runner/config.yaml) sets `capacity: 4`, but the IP-allocation mutex in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) has only been exercised by single-job e2e runs (per [TODO.md](../TODO.md) §2.1). The design is plausibly correct, but DHCP behavior under four concurrent `vmrun start` calls, native-tool concurrency safety, and the 4-minute serialization of the IP-acquire phase are unmeasured. Both reviewers flagged this; severity is HIGH (not CRITICAL) because the system can be safely operated at lower capacity while this is unproven.
|
||||
|
||||
**Location**: [runner/config.yaml](../runner/config.yaml) L17; [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L303-L460.
|
||||
|
||||
**Impact**: The "production-ready" claim in [README.md](../README.md) rests on a capacity number that has not been measured. A first real burst of four parallel builds may fail in ways the e2e runs cannot predict.
|
||||
|
||||
**Fix**: Write `scripts/Test-CapacityBurnIn.ps1`. PowerShell 5.1, uses `Start-Job` to launch four parallel `Invoke-CIJob.ps1` invocations with unique JobIds against a small no-op build. Wait for all jobs, then assert: (1) all four VMs were destroyed; (2) all four jobs got distinct IPs; (3) `F:\CI\State\vm-start.lock` does not exist; (4) `F:\CI\State\ip-leases\` is empty; (5) total wall time and per-phase JSONL durations are recorded. Run three times. If all three pass, document the result in [docs/RUNBOOK.md](../docs/RUNBOOK.md) and keep `capacity: 4`. If any fail, set `capacity: 2` until investigated.
|
||||
|
||||
**Estimated effort**: Half-day to write, run, and document.
|
||||
|
||||
### H2. Composite action `outputs:` block is malformed YAML
|
||||
|
||||
**Description**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) has `artifact-path` and a duplicate `artifact-name` under the `inputs:` map instead of a top-level `outputs:` block. Step-level outputs still work (the `Out-File $env:GITHUB_OUTPUT` lines drive the upload step inside the same composite), so the live workflow does upload artifacts. But the duplicate `artifact-name` key violates YAML 1.2 (most parsers silently overwrite with the later value) and the missing `outputs:` block means downstream workflows cannot read `steps.local-ci-build.outputs.artifact-path`.
|
||||
|
||||
**Location**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L96 and L132-L141.
|
||||
|
||||
**Impact**: Latent. Today's single consumer is unaffected. The first new consumer that tries to chain on the artifact path will fail mysteriously.
|
||||
|
||||
**Fix**: Insert a top-level `outputs:` mapping (sibling to `inputs:` and `runs:`) containing `artifact-path` and `artifact-name`. Delete the second `artifact-name` and the `artifact-path` from the `inputs:` map. Keep the inner `Out-File $env:GITHUB_OUTPUT` lines unchanged. Note: GPT-5.5 originally rated this CRITICAL; Opus rated it LOW. The final-plan severity is HIGH because impact is latent but the YAML is clearly malformed.
|
||||
|
||||
**Estimated effort**: 20 minutes.
|
||||
|
||||
### H3. Linux Mode 2 PAT injected into clone URL
|
||||
|
||||
**Description**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux Mode 2 builds `https://user:pat@host/repo.git` and exports it as `GIT_CLONE_URL` in the SSH command. The PAT becomes visible in `/proc/<pid>/environ` for the duration of `git clone`, in the host-side `ssh.exe` argv, and in any PowerShell error text that captures `$envCloneCmd`. The Windows path uses `git -c http.extraHeader=Authorization: Basic <b64>`, which is strictly safer.
|
||||
|
||||
**Location**: [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux Mode 2 clone branch, around the `GIT_CLONE_URL` export.
|
||||
|
||||
**Impact**: PAT visibility on the guest filesystem and host process list. Bounded but real.
|
||||
|
||||
**Fix**: Mirror the Windows pattern. Base64-encode `user:pat`, build a single `git -c credential.helper= -c http.extraHeader=Authorization: Basic <b64> clone <non-auth-url> <dest>` command, invoke via `Invoke-SshCommand`. Drop the URL rewrite, drop the `GIT_CLONE_URL` export, drop the `unset`. Note: GPT-5.5 originally proposed an askpass helper with `chmod 600` temp file; Opus proposed the `http.extraHeader` mirror. The simpler Opus pattern is adopted.
|
||||
|
||||
**Estimated effort**: 30 minutes including a clone test against a private repo.
|
||||
|
||||
### H4. `vmrun` lifecycle calls have no timeout in production scripts
|
||||
|
||||
**Description**: [scripts/_Common.psm1](../scripts/_Common.psm1)'s `Invoke-Vmrun` checks `$LASTEXITCODE` but does not bound execution. [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) inherit this. A hung `vmrun start` or `deleteVM` waits on `vmware-vmx.exe` indefinitely, consuming the runner slot until act_runner's 2-hour job timeout fires. The `finally` cleanup cannot run while the native process is blocked.
|
||||
|
||||
**Location**: [scripts/_Common.psm1](../scripts/_Common.psm1) `Invoke-Vmrun`; callers in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1).
|
||||
|
||||
**Impact**: A single stuck VMware operation can pin one of the four runner slots for two hours.
|
||||
|
||||
**Fix**: Add `Invoke-VmrunBounded` to [scripts/_Common.psm1](../scripts/_Common.psm1). Use `Start-Process -PassThru` + `Wait-Process -Timeout` + `Stop-Process` on timeout. Return `[pscustomobject]@{ExitCode; Output; TimedOut; ElapsedSeconds}`. Replace `Invoke-Vmrun` calls for `start`, `stop`, and `deleteVM` only — keep the unbounded version for `list`, `readVariable`, `getGuestIPAddress`. Recommended timeouts: 180s for start, 60s for stop, 90s for deleteVM.
|
||||
|
||||
**Estimated effort**: 1 hour including a Pester test.
|
||||
|
||||
### H5. Stale `vm-start.lock` not cleaned automatically
|
||||
|
||||
**Description**: The IP-allocation lock file at `F:\CI\State\vm-start.lock` is released correctly on normal exit. On host crash mid-lock, the file remains. On reboot the OS-level lock is gone (kernel released it), but the file's mere existence does not cause failure; the issue is the 10-minute retry loop and operator confusion. Neither [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) nor [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) removes it.
|
||||
|
||||
**Location**: [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1).
|
||||
|
||||
**Impact**: First job after a crash gives a confusing message; manual `Remove-Item` cures it. Low operational severity, but high friction.
|
||||
|
||||
**Fix**: In both maintenance scripts, add a block that removes `F:\CI\State\vm-start.lock` if its `LastWriteTime` is older than 30 minutes. Do not write metadata into the lock file; the file's mere existence is sufficient signal. Log to console and JSONL.
|
||||
|
||||
**Estimated effort**: 15 minutes.
|
||||
|
||||
### H6. Setup-Host directory list is stale
|
||||
|
||||
**Description**: [Setup-Host.ps1](../Setup-Host.ps1) L118-L126 creates `Templates\WinBuild` and a handful of cache directories, but does NOT create `Templates\WinBuild2025`, `Templates\WinBuild2022`, `Templates\LinuxBuild2404`, `State`, `State\ip-leases`, `keys`, or `Cache\pip`. The orchestrator depends on `F:\CI\State\` and `F:\CI\keys\`. A fresh host bootstrap leaves the system non-functional.
|
||||
|
||||
**Location**: [Setup-Host.ps1](../Setup-Host.ps1) L118-L126.
|
||||
|
||||
**Impact**: Disaster recovery is broken. Rebuilding the host from scratch using the documented procedure does not produce a working system.
|
||||
|
||||
**Fix**: Update the directory array to: `BuildVMs`, `Artifacts`, `Logs`, `Templates\WinBuild2025`, `Templates\WinBuild2022`, `Templates\LinuxBuild2404`, `State`, `State\ip-leases`, `keys`, `act_runner\logs`, `Cache\NuGet`, `Cache\pip`, `RunnerWork`, `ISO`. Remove `Templates\WinBuild`. Add a comment block referencing [AGENTS.md](../AGENTS.md) §Struttura directory host.
|
||||
|
||||
**Estimated effort**: 10 minutes.
|
||||
|
||||
### H7. Backup-CITemplate default path is wrong
|
||||
|
||||
**Description**: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 defaults `TemplatePath` to `'F:\CI\Templates\WinBuild'`. That directory no longer exists. The default-argument backup silently fails or backs up an empty directory, then the operator refreshes the actual `WinBuild2025` template, with no rollback.
|
||||
|
||||
**Location**: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 and the help block.
|
||||
|
||||
**Impact**: An operator running `.\Backup-CITemplate.ps1` before a template refresh believes they have a backup. They do not.
|
||||
|
||||
**Fix**: Change the default to `'F:\CI\Templates\WinBuild2025'`. Add a `-AllTemplates` switch that iterates `F:\CI\Templates\*` and backs up each (with subfolder naming preserved in the backup destination). Update [docs/HOST-SETUP.md](../docs/HOST-SETUP.md) and [docs/RUNBOOK.md](../docs/RUNBOOK.md) to reflect the new default. Update the help block.
|
||||
|
||||
**Estimated effort**: 30 minutes.
|
||||
|
||||
### H8. act_runner service account vs SSH alias mismatch
|
||||
|
||||
**Description**: [Setup-Host.ps1](../Setup-Host.ps1) L294-L306 writes the `gitea-ci` SSH alias to `$env:USERPROFILE\.ssh\config` — the interactive user's profile. The NSSM install at L264-L273 does not set an explicit service account, which on Windows commonly means LocalSystem. LocalSystem's home is `C:\Windows\System32\config\systemprofile\`, not the user profile. [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) uses `repo-url: 'ssh://gitea-ci/Simone/...'`. If the service runs as LocalSystem, the alias resolution fails and every Mode 1 (host-side clone) attempt errors out.
|
||||
|
||||
**Location**: [Setup-Host.ps1](../Setup-Host.ps1) L264-L273 (NSSM install), L294-L306 (SSH config write); [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) L35.
|
||||
|
||||
**Impact**: Potentially blocks all Mode 1 host-side clone builds. Severity is HIGH-PENDING-VERIFICATION; first action is verification.
|
||||
|
||||
**Fix**: Step 1, verify: run `Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName` on the host and document the answer in [docs/RUNBOOK.md](../docs/RUNBOOK.md). Step 2a, if `StartName` is `LocalSystem`: copy the SSH config and known-hosts to `C:\Windows\System32\config\systemprofile\.ssh\` with proper ACLs. Step 2b alternative: create a dedicated `ci_runner` Windows user, reinstall NSSM with `obj=.\ci_runner`, store the SSH config under that user's profile. Step 3: add the verification check to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) or a new `Validate-HostState.ps1`. Note: GPT-5.5 did not catch this. Opus did. Severity HIGH stands.
|
||||
|
||||
**Estimated effort**: 1 hour for verification + fix.
|
||||
|
||||
### H9. `ci-report-ip.service` not asserted in deploy-state validator
|
||||
|
||||
**Description**: [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) is Windows-only: it checks WinRM, Defender, Firewall, UAC, etc. It does not check whether the Linux template's `ci-report-ip.service` (the systemd unit writing `guestinfo.ci-ip`) is enabled. The unit IS validated inside [template/Install-CIToolchain-Linux2404.sh](../template/Install-CIToolchain-Linux2404.sh) L203-L221 and [template/Prepare-LinuxBuild2404.ps1](../template/Prepare-LinuxBuild2404.ps1) L361-L362, but not at the level of "is this template ready to be promoted to BaseClean-Linux." If a future refresh forgets to re-enable it, every Linux build silently falls back to the 120-second `getGuestIPAddress` timeout.
|
||||
|
||||
**Location**: [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
|
||||
|
||||
**Impact**: Silent degradation of Linux build startup time after a faulty template refresh.
|
||||
|
||||
**Fix**: Add a Linux branch to `Validate-DeployState.ps1` (or create a `Validate-LinuxDeployState.ps1`) that opens an SSH session and runs `systemctl is-enabled ci-report-ip.service` and `test -x /usr/local/bin/ci-report-ip.sh`. Fail with a clear error if either is missing.
|
||||
|
||||
**Estimated effort**: 45 minutes.
|
||||
|
||||
### H10. No automated end-to-end smoke test
|
||||
|
||||
**Description**: The Pester suite in [tests/](../tests/) exercises argument parsing only via a fake `vmrun.cmd`. The "production-ready" claim rests on manual e2e runs (`e2e-008`, `e2e-009`, `§3.3`). There is no scripted, repeatable test that exercises real VM lifecycle and exits non-zero on failure. Both reviewers flagged this; severity is HIGH because regression risk without it is high after every change.
|
||||
|
||||
**Location**: [tests/](../tests/), [gitea/workflows/](../gitea/workflows/).
|
||||
|
||||
**Impact**: Any change to orchestration scripts has no automated guard. Regressions are caught only by chance during the next manual run.
|
||||
|
||||
**Fix**: Create `gitea/workflows/self-test.yml` (workflow_dispatch trigger only). One job that calls the composite action against a minimal `Write-Host 'ok' > out.txt` build, asserts artifact upload, JSONL phase events, and clean teardown. Have it succeed-or-fail for Windows and a second job for Linux. Add `scripts/Test-Smoke.ps1` that the workflow invokes; the script also runs locally. Note: this replaces the much larger `Measure-CIBenchmark.ps1` wiring suggestion; the smoke test is sufficient and cheaper.
|
||||
|
||||
**Estimated effort**: Half-day.
|
||||
|
||||
### H11. Composite action hardcodes the script root path
|
||||
|
||||
**Description**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 sets `$ciScriptsDir = 'N:\Code\Workspace\Local-CI-CD-System\scripts'`. The action is presented as reusable, but moving the repo breaks it. Opus caught this; GPT-5.5 missed it (it caught path drift in setup/backup only).
|
||||
|
||||
**Location**: [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148.
|
||||
|
||||
**Impact**: Action is not actually reusable. Any second consumer or repo relocation breaks builds.
|
||||
|
||||
**Fix**: Replace with a runner env var. Add `GITEA_CI_SCRIPT_ROOT: "N:\\Code\\Workspace\\Local-CI-CD-System\\scripts"` to [runner/config.yaml](../runner/config.yaml) `envs:`. In the action, use `$ciScriptsDir = $env:GITEA_CI_SCRIPT_ROOT`. Do not use `${{ github.action_path }}` — the action repo is not necessarily cloned into the consumer's workspace in this Gitea setup.
|
||||
|
||||
**Estimated effort**: 20 minutes.
|
||||
|
||||
### H12. Measure-CIBenchmark uses a different IP discovery path than production
|
||||
|
||||
**Description**: [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) L129-L146 polls `Invoke-Vmrun -Operation getGuestIPAddress`. [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) L348-L389 primarily reads `guestVar 'ci-ip'` and uses `getGuestIPAddress` as fallback. The benchmark therefore measures a different code path than what production runs, and its phase timings cannot be used for capacity planning.
|
||||
|
||||
**Location**: [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) L129-L146.
|
||||
|
||||
**Impact**: Benchmark data is misleading. Decisions to keep or change `capacity: 4` based on it would be based on wrong measurements.
|
||||
|
||||
**Fix**: Extract IP-acquire into a function `Get-GuestIPAddress` in [scripts/_Common.psm1](../scripts/_Common.psm1) that takes the VMX path and credentials and implements the `guestinfo.ci-ip`-then-`getGuestIPAddress` flow used in production. Call it from both `Invoke-CIJob.ps1` and `Measure-CIBenchmark.ps1`.
|
||||
|
||||
**Estimated effort**: 1 hour.
|
||||
|
||||
---
|
||||
|
||||
## Complete Priority Checklist
|
||||
|
||||
Every item is concrete, references a file, and ends with an effort estimate.
|
||||
|
||||
### Critical [ ]
|
||||
|
||||
- [ ] Fix Linux matrix template selection — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) + [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) — action selects `GITEA_CI_TEMPLATE_PATH` or `GITEA_CI_LINUX_TEMPLATE_PATH` based on `guest-os` input or runner label, also resolves the correct `snapshot-name` (`BaseClean` or `BaseClean-Linux`) — 1h.
|
||||
- [ ] Redact Linux ExtraGuestEnv secret logging — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch — replace `Write-Host "...$buildCmd"` with redacted version; never log `$envPrefix` — 15min.
|
||||
|
||||
### High [ ]
|
||||
|
||||
- [ ] Run 4-way concurrency burn-in — `scripts/Test-CapacityBurnIn.ps1` (new) — `Start-Job`-based harness, three runs, assertions on IPs and lock cleanup — half-day.
|
||||
- [ ] Fix composite action outputs YAML — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) — add top-level `outputs:` block; remove duplicate `artifact-name` from `inputs:` — 20min.
|
||||
- [ ] Replace Linux Mode 2 PAT URL with `http.extraHeader` — [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch — base64-encode `user:pat`, single `git -c http.extraHeader=...` command — 30min.
|
||||
- [ ] Add `Invoke-VmrunBounded` and use for lifecycle calls — [scripts/_Common.psm1](../scripts/_Common.psm1), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1) — bound `start`, `stop`, `deleteVM` only — 1h.
|
||||
- [ ] Add stale `vm-start.lock` cleanup — [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1) — remove file older than 30 minutes — 15min.
|
||||
- [ ] Fix Setup-Host directory list — [Setup-Host.ps1](../Setup-Host.ps1) L118 — match [AGENTS.md](../AGENTS.md) §Struttura — 10min.
|
||||
- [ ] Fix Backup-CITemplate default — [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) L48 + help block — default `WinBuild2025`, add `-AllTemplates` — 30min.
|
||||
- [ ] Verify and fix act_runner service-account SSH alias — [Setup-Host.ps1](../Setup-Host.ps1) L264-L306 — check `StartName`, relocate `.ssh/config` or change service account — 1h.
|
||||
- [ ] Add `ci-report-ip.service` check to deploy-state validator — [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1) — Linux branch via SSH — 45min.
|
||||
- [ ] Add on-demand smoke-test workflow — new `gitea/workflows/self-test.yml` + `scripts/Test-Smoke.ps1` — minimal Windows and Linux jobs through composite action — half-day.
|
||||
- [ ] Replace hardcoded `N:\` script path in composite action — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) L148 + [runner/config.yaml](../runner/config.yaml) — new `GITEA_CI_SCRIPT_ROOT` env var — 20min.
|
||||
- [ ] Align benchmark IP discovery with production — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) + [scripts/_Common.psm1](../scripts/_Common.psm1) — extract shared `Get-GuestIPAddress` helper — 1h.
|
||||
|
||||
### Medium [ ]
|
||||
|
||||
- [ ] Add pre-clone disk space gate — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) Phase 2 — `throw` if free space on `F:` below 20 GB — 5min.
|
||||
- [ ] Add post-failure guest diagnostics collection — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) `catch` block — best-effort `Get-GuestDiagnostics` to `F:\CI\Artifacts\<jobId>\diagnostics\` before teardown — 1h.
|
||||
- [ ] Wire `UseSharedCache` through composite action and orchestrator — [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml), [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) — new input, parameter pass-through — 1h.
|
||||
- [ ] Authenticate-not-just-TCP WinRM readiness — [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1) — optional Credential parameter, single `Invoke-Command { 'ready' }` after TCP open — 30min.
|
||||
- [ ] Add Ubuntu cloud VMDK SHA256 verification — [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1) — SHA256 parameter, fail on mismatch — 30min.
|
||||
- [ ] Fix `Validate-SetupState` `PasswordExpires` compatibility — [template/Validate-SetupState.ps1](../template/Validate-SetupState.ps1) — mirror logic from `Install-CIToolchain-WinBuild2025.ps1` — 15min.
|
||||
- [ ] Update [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — replace `ForEach-Object -Parallel` with `Start-Job`, replace `-RepositoryUrl` with `-RepoUrl` — 20min.
|
||||
- [ ] Add `JobId` sanitization — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1) — `[ValidatePattern('^[A-Za-z0-9._-]+$')]` — 5min.
|
||||
- [ ] Add `ExtraGuestEnv` key validation — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — reject keys not matching `^[A-Za-z_][A-Za-z0-9_]*$` — 15min.
|
||||
- [ ] Add phase-duration summary at job end — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — table printed from in-memory timestamps — 30min.
|
||||
- [ ] Add 90-minute job-duration webhook warning — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — background `Start-Job` posts once — 30min.
|
||||
- [ ] Document HGFS shared folder write semantics — [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) — explicit note on cache-poisoning risk — 15min.
|
||||
- [ ] Standardize Event Log source names — [scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1), [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — pick `CI-DiskSpaceAlert`, `CI-RunnerHealth`, propagate — 30min.
|
||||
- [ ] Tighten SSH host-key handling for CI jobs — [scripts/_Transport.psm1](../scripts/_Transport.psm1) — `StrictHostKeyChecking=accept-new`, `UserKnownHostsFile=F:\CI\State\known_hosts` for CI-job calls; leave permissive mode for template provisioning — 1h.
|
||||
- [ ] Consolidate template-refresh runbook — [docs/RUNBOOK.md](../docs/RUNBOOK.md) — single section: backup → refresh → validate → smoke → snapshot → update `runner/config.yaml` → keep previous snapshot 7 days → rollback procedure — 2h.
|
||||
|
||||
### Low [ ]
|
||||
|
||||
- [ ] Delete deprecated runner installer — [runner/Install-Runner.ps1](../runner/Install-Runner.ps1) — remove the file — 5min.
|
||||
- [ ] Align Setup-Host password documentation with behavior — [Setup-Host.ps1](../Setup-Host.ps1) help block — describe prompt fallback, remove ghost default `'CIBuild!ChangeMe2026'` — 5min.
|
||||
- [ ] Remove emoji from webhook payloads — [scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1) — `[WARN]`, `[ERROR]` prefixes — 5min.
|
||||
- [ ] Fix Validate-* docblocks to reflect HTTPS/5986 — [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1), [template/Validate-SetupState.ps1](../template/Validate-SetupState.ps1) — update HTTP/Basic to HTTPS/Basic — 10min.
|
||||
- [ ] Align benchmark JSON schema with test plan — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) or [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — pick one and update the other — 30min.
|
||||
- [ ] Add `TemplatePath` sanity check — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) — warn if not under `F:\CI\Templates\` — 10min.
|
||||
- [ ] Add a maintenance flag for Watch-RunnerHealth — [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1) — skip restart if `F:\CI\State\runner-maintenance.flag` exists — 10min.
|
||||
- [ ] Add ACL validation script — new `scripts/Validate-HostState.ps1` — assert ACLs on `F:\CI\keys\ci_linux` and Credential Manager target accessible — 30min.
|
||||
- [ ] Move done sections of [TODO.md](../TODO.md) into a new `docs/CHANGELOG.md` — 1h.
|
||||
- [ ] Fix [docs/CI-FLOW.md](../docs/CI-FLOW.md) failure-scenario table — change "partial artifacts collected" to "artifacts collected only after successful build packaging" or implement H-level diagnostics; if H-level diagnostics added, leave docs as-is — 15min.
|
||||
|
||||
### Nice to have [ ]
|
||||
|
||||
- [ ] Add `scripts/Get-CIJobSummary.ps1` — reads last N days of JSONL files, prints duration table — 1h.
|
||||
- [ ] Add artifact manifest with file hashes — [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1) — `manifest.json` with name, size, SHA256, commit, job ID — 1h.
|
||||
- [ ] Add per-job VMX CPU/RAM override — [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) + composite action input — post-clone VMX edit, default off — 1h.
|
||||
- [ ] Add optional Gitea API runner-online check — [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1) — optional `-GiteaUrl` and PAT target — 1h.
|
||||
- [ ] Begin SHA256 pinning of toolchain installers — [template/Install-CIToolchain-WinBuild2025.ps1](../template/Install-CIToolchain-WinBuild2025.ps1) — fill hashes for Git, 7-Zip, Python, .NET, Node — 2h.
|
||||
- [ ] Add linked-clone delta size measurement — [scripts/Measure-CIBenchmark.ps1](../scripts/Measure-CIBenchmark.ps1) — record per-job delta growth — 30min.
|
||||
|
||||
---
|
||||
|
||||
## Detailed Operational Checklist
|
||||
|
||||
Execute in this order. Each task lists a prerequisite, an action, and a verification step.
|
||||
|
||||
### Sprint 1: Validation
|
||||
|
||||
Goal: prove or downgrade unvalidated claims before any code changes go in.
|
||||
|
||||
**Task 1.1 — Verify act_runner service account**
|
||||
- Prerequisite: host PowerShell session as Administrator.
|
||||
- Action: `Get-CimInstance Win32_Service -Filter "Name='act_runner'" | Select-Object Name,StartName,State`.
|
||||
- Verification: record `StartName` in [docs/RUNBOOK.md](../docs/RUNBOOK.md) under "Runner service identity". If `LocalSystem`, schedule H8 fix immediately.
|
||||
|
||||
**Task 1.2 — Inventory current state-file presence**
|
||||
- Prerequisite: nothing.
|
||||
- Action: `Test-Path F:\CI\State, F:\CI\State\ip-leases, F:\CI\keys, F:\CI\Cache\pip, F:\CI\Templates\WinBuild2025, F:\CI\Templates\LinuxBuild2404`.
|
||||
- Verification: each must return `True`. If any are missing, create manually for now and defer the [Setup-Host.ps1](../Setup-Host.ps1) fix to Sprint 2.
|
||||
|
||||
**Task 1.3 — Run 4-way concurrency burn-in (H1)**
|
||||
- Prerequisite: write `scripts/Test-CapacityBurnIn.ps1` first; `WinBuild2025` and `LinuxBuild2404` templates powered off; ≥ 100 GB free on `F:`.
|
||||
- Action: invoke the burn-in three times against a no-op build (e.g. `Write-Host 'ok'`); record per-run JSONL outputs.
|
||||
- Verification: zero IP collisions across runs, zero leaked VMs, zero leaked lock files, all jobs succeeded. If any fail, set `runner/config.yaml capacity: 2` and create a follow-up investigation task.
|
||||
|
||||
**Task 1.4 — Manually invoke Linux matrix leg to confirm C1 bug**
|
||||
- Prerequisite: a test branch with a tag that triggers [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml).
|
||||
- Action: push a tag, observe both legs.
|
||||
- Verification: confirm Linux leg fails or builds against Windows template. Document the failure mode in the C1 task notes.
|
||||
|
||||
### Sprint 2: Critical Fixes
|
||||
|
||||
Goal: clear both Criticals plus the highest-impact Highs that touch production paths.
|
||||
|
||||
**Task 2.1 — Fix Linux matrix template selection (C1)**
|
||||
- Prerequisite: Sprint 1 task 1.4 done.
|
||||
- Action: edit [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) so the PowerShell step resolves `INPUT_GUEST_OS_RESOLVED` from `INPUT_GUEST_OS` (with `Auto` honoring `runs-on` heuristic via `$env:RUNNER_LABELS` if Gitea exposes it; otherwise defaulting Auto to Windows) and selects between `GITEA_CI_TEMPLATE_PATH` and `GITEA_CI_LINUX_TEMPLATE_PATH`. Update [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) matrix to pass `guest-os: ${{ matrix.target == 'linux' && 'Linux' || 'Windows' }}` as a belt-and-suspenders measure.
|
||||
- Verification: re-run the matrix workflow. Both legs succeed against the correct templates.
|
||||
|
||||
**Task 2.2 — Redact Linux ExtraGuestEnv logging (C2)**
|
||||
- Prerequisite: nothing.
|
||||
- Action: in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux branch, replace the build-command echo with the redacted form. Grep the entire file for any other place `$envPrefix` or `$buildCmd` is logged; fix all.
|
||||
- Verification: run a Linux job with a fake secret `extra-guest-env-json='{"DUMMY":"shibboleth"}'` and confirm the string `shibboleth` does not appear in `F:\CI\Logs\<jobId>\invoke-ci.log` or in act_runner output.
|
||||
|
||||
**Task 2.3 — Fix composite action outputs YAML (H2)**
|
||||
- Prerequisite: nothing.
|
||||
- Action: edit [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml) per H2 fix.
|
||||
- Verification: run `gitea/workflows/build-nsis.yml`; confirm artifact upload still works. Optionally test downstream consumption by writing a temporary workflow step that reads `steps.local-ci-build.outputs.artifact-path`.
|
||||
|
||||
**Task 2.4 — Linux Mode 2 PAT to http.extraHeader (H3)**
|
||||
- Prerequisite: a private Gitea repo with the PAT scope to clone.
|
||||
- Action: edit [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) Linux clone branch.
|
||||
- Verification: run a Linux Mode 2 build; confirm `git clone` succeeds; verify with `ps -ef` on the guest during clone that the PAT is not in the argv.
|
||||
|
||||
**Task 2.5 — Fix Setup-Host directory list (H6)**
|
||||
- Prerequisite: Sprint 1 task 1.2 captured what is missing.
|
||||
- Action: edit [Setup-Host.ps1](../Setup-Host.ps1) L118 directory array.
|
||||
- Verification: on a test VM (not the production host), run [Setup-Host.ps1](../Setup-Host.ps1) and confirm every directory in [AGENTS.md](../AGENTS.md) §Struttura is created.
|
||||
|
||||
**Task 2.6 — Fix Backup-CITemplate default (H7)**
|
||||
- Prerequisite: nothing.
|
||||
- Action: edit [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) default and help; add `-AllTemplates` switch.
|
||||
- Verification: run `.\Backup-CITemplate.ps1 -DryRun` (add flag if it does not exist) and confirm the resolved source path is `F:\CI\Templates\WinBuild2025`.
|
||||
|
||||
### Sprint 3: High Priority Fixes
|
||||
|
||||
Goal: stabilize the rest of the High-tier items.
|
||||
|
||||
**Task 3.1 — Add `Invoke-VmrunBounded` (H4)**
|
||||
- Prerequisite: review existing bounded patterns in deploy scripts.
|
||||
- Action: add the function to [scripts/_Common.psm1](../scripts/_Common.psm1); wire `start`/`stop`/`deleteVM` callers in [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1).
|
||||
- Verification: write a Pester test that injects a fake `vmrun.cmd` which sleeps 600s; assert the wrapper times out at the configured deadline.
|
||||
|
||||
**Task 3.2 — Stale lock cleanup (H5)**
|
||||
- Prerequisite: nothing.
|
||||
- Action: edit [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1) and [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1).
|
||||
- Verification: `touch F:\CI\State\vm-start.lock` with a 1-hour-old timestamp, run the retention script, confirm it is removed and the removal is logged.
|
||||
|
||||
**Task 3.3 — Fix or relocate SSH alias for service account (H8)**
|
||||
- Prerequisite: Sprint 1 task 1.1 result.
|
||||
- Action: per H8, either copy `.ssh/` to systemprofile or change the service account.
|
||||
- Verification: `psexec -s -i ssh -T gitea-ci` (or equivalent) from the service account; expect a successful Gitea response.
|
||||
|
||||
**Task 3.4 — Linux deploy validator check for ci-report-ip (H9)**
|
||||
- Prerequisite: SSH key and IP of the LinuxBuild2404 template (boot it, capture, shut down).
|
||||
- Action: add the Linux branch to [template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1).
|
||||
- Verification: against the live template, the script reports the service as enabled.
|
||||
|
||||
**Task 3.5 — Add smoke-test workflow (H10)**
|
||||
- Prerequisite: H2 and C1 fixed (otherwise the workflow's own assertions fail).
|
||||
- Action: create `gitea/workflows/self-test.yml` and `scripts/Test-Smoke.ps1`.
|
||||
- Verification: trigger via `workflow_dispatch` and observe both legs green.
|
||||
|
||||
**Task 3.6 — Replace hardcoded N:\ path (H11)**
|
||||
- Prerequisite: nothing.
|
||||
- Action: add `GITEA_CI_SCRIPT_ROOT` env var in [runner/config.yaml](../runner/config.yaml); reference it in [gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml).
|
||||
- Verification: restart act_runner; re-run a workflow; confirm it still resolves the script path.
|
||||
|
||||
**Task 3.7 — Benchmark IP path alignment (H12)**
|
||||
- Prerequisite: H4 done (shared `_Common.psm1` patterns ironed out).
|
||||
- Action: extract `Get-GuestIPAddress` into [scripts/_Common.psm1](../scripts/_Common.psm1); update both callers.
|
||||
- Verification: run `Measure-CIBenchmark.ps1 -Iterations 1` for Windows and Linux; confirm phase timings match those recorded in production JSONL within ±10%.
|
||||
|
||||
### Sprint 4: Operational Hardening
|
||||
|
||||
Goal: harden recovery, secret handling, and recoverability.
|
||||
|
||||
**Task 4.1 — Pre-clone disk gate, JobId/ExtraGuestEnv validation, redacted command logger** (Medium items grouped) — edit [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) and [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1); verify with deliberately bad inputs.
|
||||
|
||||
**Task 4.2 — Failure-path diagnostics** — add a `Get-GuestDiagnostics` helper invoked from the `catch` block of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1). Skip on Phase 1/2 failures (no VM yet). Best-effort SSH/WinRM, no throw on failure.
|
||||
|
||||
**Task 4.3 — Wire `UseSharedCache`** — surface in composite action and orchestrator; confirm with a Python build that pip cache is reused.
|
||||
|
||||
**Task 4.4 — Tighten SSH host keys for CI jobs** — `accept-new` + `F:\CI\State\known_hosts` for CI; keep permissive mode for template provisioning (host keys are unknown there).
|
||||
|
||||
**Task 4.5 — Ubuntu VMDK SHA256 verification** — add to [template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1).
|
||||
|
||||
**Task 4.6 — Consolidated template-refresh runbook** — edit [docs/RUNBOOK.md](../docs/RUNBOOK.md). One section covering: stop runner, back up template (`-AllTemplates`), boot template, KMS reactivation via `slmgr /ato` (Windows), run `Prepare-WinBuild2025.ps1` or `Prepare-LinuxBuild2404.ps1`, shutdown, snapshot `BaseClean_yyyyMMdd`, run `Validate-DeployState.ps1` (Windows and Linux branches), run smoke workflow, then update `GITEA_CI_SNAPSHOT_NAME` in [runner/config.yaml](../runner/config.yaml), keep prior snapshot 7 days, rollback procedure.
|
||||
|
||||
### Sprint 5: Quality and Optimization
|
||||
|
||||
Goal: polish.
|
||||
|
||||
**Task 5.1 — Delete dead code** — remove [runner/Install-Runner.ps1](../runner/Install-Runner.ps1); align Setup-Host help text.
|
||||
|
||||
**Task 5.2 — Webhook emoji removal, Event Log naming, docblock corrections** — Low-tier batch.
|
||||
|
||||
**Task 5.3 — Update test plan** — [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) — PS5.1 syntax, correct parameter names, current task and event log names.
|
||||
|
||||
**Task 5.4 — Add `Get-CIJobSummary.ps1`** — nice-to-have summary tool.
|
||||
|
||||
**Task 5.5 — Phase-duration summary in orchestrator** — added at end of [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) before final cleanup.
|
||||
|
||||
**Task 5.6 — Begin SHA256 pinning** — start with Git, 7-Zip, Python, .NET. Note in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) that this is a homelab-acceptable, partial-coverage pin.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Refactors
|
||||
|
||||
Only refactors that genuinely simplify or fix something. No churn.
|
||||
|
||||
1. **Extract `Get-GuestIPAddress` into [scripts/_Common.psm1](../scripts/_Common.psm1)**. Removes the dual-implementation drift between `Invoke-CIJob.ps1` and `Measure-CIBenchmark.ps1`. Single source of truth for the `guestinfo.ci-ip`-then-fallback flow. Justification: real bug avoidance, not aesthetic.
|
||||
|
||||
2. **Extract `Invoke-VmrunBounded` into [scripts/_Common.psm1](../scripts/_Common.psm1)**. Closes the H4 timeout gap and consolidates the pattern that already exists in some deploy scripts.
|
||||
|
||||
3. **Add `Write-CIRedactedCommand` and `ConvertTo-SafeShellSingleQuotedString` to [scripts/_Common.psm1](../scripts/_Common.psm1)**. The first standardizes secret-aware command logging across Windows and Linux paths. The second consolidates the `-replace "'", "'\''"` idiom used in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1). Both are 5-line functions.
|
||||
|
||||
4. **DO NOT extract a generic transport interface**. Opus suggested `Invoke-GuestCommand`/`Copy-GuestItem`/`Wait-GuestTransportReady`. Both reviewers ultimately agreed this is the wrong move for a two-transport homelab. Keep explicit Windows/Linux branches in [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1) and [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1).
|
||||
|
||||
5. **DO NOT introduce `F:\CI\State\config.json` as a central config**. [AGENTS.md](../AGENTS.md) and [runner/config.yaml](../runner/config.yaml) are already the two sources of truth. Adding a third creates drift, not reduces it. The remediation is to align the existing two, not invent a third.
|
||||
|
||||
6. **DO NOT migrate to PowerShell 7**. The PS 5.1 constraint is deliberate, encoded in [AGENTS.md](../AGENTS.md), and consistently respected across the codebase. A migration is a multi-day churn that buys little.
|
||||
|
||||
7. **DO NOT replace JSONL with SQLite**. The JSONL output is forward-compatible and trivially greppable. `Get-CIJobSummary.ps1` over JSONL covers the cross-job analysis need without adding a database engine.
|
||||
|
||||
8. **DO NOT pursue a static IP pool unless the burn-in fails**. The current lock+lease design after the stale-lock fix is sufficient for `capacity: 4`. Static IPs cost more than they save at this scale.
|
||||
|
||||
---
|
||||
|
||||
## Security Improvements
|
||||
|
||||
Must (do these in Sprint 2 or 3):
|
||||
|
||||
- Redact Linux `ExtraGuestEnv` secret logging (C2).
|
||||
- Replace Linux Mode 2 PAT URL with `http.extraHeader` (H3).
|
||||
- Verify and fix the act_runner SSH alias service-account issue (H8).
|
||||
- Add `JobId` pattern validation (Medium).
|
||||
- Add `ExtraGuestEnv` key pattern validation (Medium).
|
||||
- Run SSH `StrictHostKeyChecking=accept-new` with `UserKnownHostsFile=F:\CI\State\known_hosts` for CI jobs (Medium).
|
||||
|
||||
Could (Nice-to-have):
|
||||
|
||||
- Begin SHA256 pinning of Tier-1 toolchain installers.
|
||||
- Verify ACLs on `F:\CI\keys\ci_linux` via a new `Validate-HostState.ps1`.
|
||||
- Add a one-paragraph note in [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) on Gitea log-masking limits (substring-only masking; base64 transforms defeat it).
|
||||
|
||||
Will NOT do:
|
||||
|
||||
- Full PKI for WinRM. The self-signed lab certificate plus VMnet8 isolation is the right boundary.
|
||||
- External secret vault integration.
|
||||
- Enable Defender / Firewall / UAC inside the build VM. The threat model accepts this for trusted private code; documented in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md).
|
||||
- Per-build credential rotation.
|
||||
- Artifact signing.
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Improvements
|
||||
|
||||
Concrete improvements to workflows and the composite action.
|
||||
|
||||
1. **Make the composite action self-sufficient for guest OS selection** (C1 fix). After this, [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) does not need to pass `guest-os`, `template-path`, or `snapshot-name` for the standard Windows/Linux matrix; the action infers from the runner label or accepts the explicit input.
|
||||
|
||||
2. **Add a top-level `outputs:` block** so downstream consumers can chain (H2 fix).
|
||||
|
||||
3. **Add `use-shared-cache` input** plumbed to `Invoke-CIJob.ps1 -UseSharedCache` (Medium fix). Forward to `Invoke-RemoteBuild.ps1` so NuGet/pip caches actually work end-to-end.
|
||||
|
||||
4. **Add an on-demand `gitea/workflows/self-test.yml`** (`workflow_dispatch` trigger only). One Windows job, one Linux job, both calling the composite action against a trivial build (`Write-Host 'ok' > out.txt`). Asserts artifact upload, JSONL events, clean teardown. Use as the regression gate for any action change.
|
||||
|
||||
5. **Add a Pester `gitea/workflows/tests.yml`** that runs the existing tests on every push. PowerShell 5.1, `Invoke-Pester -CI`. This is the current missing "CI for the CI" piece.
|
||||
|
||||
6. **Keep the lint workflow** [gitea/workflows/lint.yml](../gitea/workflows/lint.yml) as is.
|
||||
|
||||
7. **Do NOT add scheduled builds, dependency-update workflows, or release workflows.** Out of scope.
|
||||
|
||||
---
|
||||
|
||||
## Docker / Container Notes
|
||||
|
||||
This project deliberately does not use Docker, Podman, or any container runtime. The decision is correct and should be preserved.
|
||||
|
||||
Rationale:
|
||||
|
||||
- Windows builds require VS Build Tools, MSBuild, the full .NET SDK, and Windows-specific signing chains. Running those in Windows containers requires Windows Server host images, breaks isolation guarantees the project gets cheaply from VMware, and adds licensing complexity.
|
||||
- The project's isolation boundary is a VM, not a container. VMs offer strictly stronger isolation than containers (kernel separation, hardware-virtualized network stack, separate filesystem) at a cost the system can afford.
|
||||
- act_runner's `container.enabled: false` is the right setting for a system that uses real VMs.
|
||||
- The performance overhead of "spin up a VM" vs "spin up a container" is real (a clone takes 5-30s, a container ~1s), but the orchestrator amortizes this across a 2-hour build budget. The cost is acceptable.
|
||||
|
||||
When might containers enter the picture? Likely never for this scope. If the project ever needs to build small Linux-only services with no Windows leg, a side-by-side `linux-build-container` runner label could be added later. That is a phase-3+ idea, not a current task.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring and Logging Improvements
|
||||
|
||||
Concrete additions only. No metric server.
|
||||
|
||||
JSONL schema additions inside `F:\CI\Logs\<jobId>\invoke-ci.jsonl` (emitted by [scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1)):
|
||||
|
||||
- Per-phase `duration_ms` calculated at phase end.
|
||||
- `free_disk_gb_before` and `free_disk_gb_after` for the job.
|
||||
- `ip_detection_method` field with values `guestinfo` or `getGuestIPAddress`.
|
||||
- `vm_start_lock_wait_ms` recording how long the orchestrator waited on `vm-start.lock`.
|
||||
- `clone_dir_size_mb` measured at teardown.
|
||||
|
||||
Event Log additions:
|
||||
|
||||
- Standardize on `CI-DiskSpaceAlert` (EventId 1001), `CI-RunnerHealth` (1002 stopped / 1003 restarted / 1004 throttled).
|
||||
- Add `CI-JobFailure` source for failed orchestrator runs (EventId 2001) writing JobId and failed phase.
|
||||
|
||||
Watcher script additions:
|
||||
|
||||
- `scripts/Get-CIJobSummary.ps1` (nice-to-have): reads last N days of JSONL files, prints a table of JobId, started, elapsed, phase 1 duration, phase 5 duration, status.
|
||||
- `scripts/Watch-RunnerHealth.ps1` (Low): respect `F:\CI\State\runner-maintenance.flag` to suppress restart attempts during template refresh.
|
||||
|
||||
Webhook payload changes:
|
||||
|
||||
- Remove emoji prefixes. Use `[WARN]`, `[ERROR]`, `[INFO]` plain text.
|
||||
|
||||
Do NOT add:
|
||||
|
||||
- Prometheus exporters.
|
||||
- Grafana dashboards.
|
||||
- OpenTelemetry collectors.
|
||||
- SQLite or any other database sink.
|
||||
- Discord/Slack rich-embed cards.
|
||||
|
||||
---
|
||||
|
||||
## Backup and Recovery Improvements
|
||||
|
||||
The current backup story is roughly: [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) (currently pointing at a stale path). Improvements:
|
||||
|
||||
1. Fix the default path and add `-AllTemplates` switch (H7).
|
||||
2. Add to [docs/RUNBOOK.md](../docs/RUNBOOK.md) the consolidated template-refresh runbook (Sprint 4 task 4.6).
|
||||
3. Register a weekly scheduled task that runs `.\Backup-CITemplate.ps1 -AllTemplates -KeepLast 3` and writes a log to `F:\CI\Logs\backup\<date>.log`. Update [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1) to add this fifth scheduled task.
|
||||
4. Document recovery from a destroyed template: restore the latest `.7z` from the backup destination, validate snapshots exist via `vmrun listSnapshots`, run `Validate-DeployState.ps1` (Windows + Linux), run the smoke workflow. If the smoke fails, restore the previous backup. Step-by-step in [docs/RUNBOOK.md](../docs/RUNBOOK.md).
|
||||
5. Document the simple recovery path for a stale `vm-start.lock` and stale IP leases in [docs/RUNBOOK.md](../docs/RUNBOOK.md) — even though Sprint 3 task 3.2 makes this automatic, the operator-facing docs should still say "if you ever need to do this manually, the commands are X."
|
||||
|
||||
Do NOT add:
|
||||
|
||||
- VM snapshot backup to remote storage (out of scope; 7z files on a second drive is sufficient).
|
||||
- Restore-time automated diff against a known-good template (over-engineered).
|
||||
|
||||
---
|
||||
|
||||
## Runner and Build Optimization
|
||||
|
||||
Concrete, measurable.
|
||||
|
||||
1. **Wire `use-shared-cache`** through the composite action and orchestrator (Medium item). Expected impact: ~15-30% faster Python and .NET builds that pull packages on every run, by avoiding the `pip download` and `nuget restore` over the WAN. Measurement: compare `phase 5 duration` JSONL before/after on a representative NSIS-plugin build.
|
||||
|
||||
2. **After burn-in, decide on `capacity`**. If the burn-in shows that capacity 4 is solid, keep it. If it shows that capacity 2 is the safe ceiling under typical host CPU contention, set `runner/config.yaml capacity: 2`. Expected impact: avoidance of unreproducible build failures during peaks.
|
||||
|
||||
3. **Defender exclusion for `F:\CI\BuildVMs\`** on the host (Low). Cloning and start are I/O heavy on the parent VMDK chain; Defender scanning slows this. Use `Set-MpPreference -ExclusionPath 'F:\CI\BuildVMs'`. Expected impact: ~10% faster clone-to-IP times. Document the trade-off in [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md): malicious code reaching the host would persist faster, but the threat model already accepts trusted-builds-only.
|
||||
|
||||
4. **Linked-clone health monitoring** (Nice-to-have): add per-job delta size to JSONL. If the average delta grows over months without a template refresh, that is a signal something is writing into the template chain. Not urgent; nice diagnostic.
|
||||
|
||||
5. **Do NOT pursue a warm clone pool**. Both reviewers agreed: the savings (5-30s per job) do not justify the state-machine complexity for a 2-hour build budget.
|
||||
|
||||
6. **Do NOT pursue VMX CPU/RAM tuning per job** unless the burn-in shows host saturation. Wait for evidence.
|
||||
|
||||
---
|
||||
|
||||
## Repository Conventions
|
||||
|
||||
These are the codified rules. Where ambiguous, write them into [AGENTS.md](../AGENTS.md) and [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md).
|
||||
|
||||
**Style and linting** (already largely in place):
|
||||
|
||||
- PowerShell 5.1 only. Every script begins with `#Requires -Version 5.1`, `Set-StrictMode -Version Latest`, `$ErrorActionPreference = 'Stop'`.
|
||||
- 4-space indentation, no tabs.
|
||||
- Verb-Noun cmdlet naming. `Get-`, `Set-`, `New-`, `Remove-`, `Invoke-`, `Test-`, `Wait-`, `Watch-`.
|
||||
- `Write-Host` is acceptable and preferred over `Write-Output` in CI scripts (act_runner captures `Write-Host` cleanly; `Write-Output` pollutes function return values). Codified in [PSScriptAnalyzerSettings.psd1](../PSScriptAnalyzerSettings.psd1).
|
||||
- No emoji in code, logs, webhook payloads, docs, or commit messages. Plain text markers like `[WARN]`, `[OK]`, `[x]`, `[ ]`.
|
||||
- No PS 7+ syntax: no `??`, no `??=`, no ternary, no `&&`/`||` in PowerShell context, no `ForEach-Object -Parallel`, no `Test-Json`.
|
||||
|
||||
**File organization**:
|
||||
|
||||
- Production scripts under `scripts/`. Template provisioning under `template/`. Tests under `tests/`. Documentation under `docs/`. Plans and reviews under `plans/`.
|
||||
- Module files `_Common.psm1` and `_Transport.psm1` stay in `scripts/`. Do not split further.
|
||||
- One script, one verb-noun purpose. Composite actions live under `gitea/actions/<name>/action.yml`.
|
||||
|
||||
**Commit messages**:
|
||||
|
||||
- Imperative mood, present tense ("Add bounded vmrun wrapper", not "Added" or "Adding").
|
||||
- Subject line ≤72 characters. Body explains *why*, not *what*.
|
||||
- Reference the master plan checklist item by short title when relevant: e.g. `Fix C1: Linux matrix template selection`.
|
||||
|
||||
**Branching**:
|
||||
|
||||
- `main` is the only long-lived branch.
|
||||
- Use short-lived feature branches `fix/c1-linux-matrix`, `fix/h2-action-outputs`, etc.
|
||||
- Squash merge or rebase; do not preserve micro-commits.
|
||||
|
||||
**Testing requirements before merge**:
|
||||
|
||||
- After H10 is implemented: every PR runs the lint and Pester workflows. After the smoke-test workflow exists, run it for any change touching orchestration scripts.
|
||||
|
||||
**Documentation updates**:
|
||||
|
||||
- A change to a script that changes its parameters or behavior must update [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), or [docs/RUNBOOK.md](../docs/RUNBOOK.md) in the same commit.
|
||||
- A change to [runner/config.yaml](../runner/config.yaml) or directory layout must update [AGENTS.md](../AGENTS.md) and [Setup-Host.ps1](../Setup-Host.ps1).
|
||||
|
||||
---
|
||||
|
||||
## Documentation Status
|
||||
|
||||
Per-document assessment.
|
||||
|
||||
| Document | Status | Action |
|
||||
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [AGENTS.md](../AGENTS.md) | Good. Reflects current paths, PS 5.1 rules, error patterns. | Keep updated as new gotchas are discovered. |
|
||||
| [README.md](../README.md) | Stale. Claims production-ready before stabilization pass completes. | Replace "production-ready" with "feature-complete; stabilization pass in progress" until all Critical and High items are checked. |
|
||||
| [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md) | Mostly accurate; minor diagram rendering issue. | Reformat the ASCII diagram box. Reflect H4 bounded-vmrun pattern in the lifecycle section. |
|
||||
| [docs/CI-FLOW.md](../docs/CI-FLOW.md) | Stale on partial-artifacts-on-failure. Still mentions `getGuestIPAddress` as primary IP detection. | Update IP-detection section to reflect `guestinfo.ci-ip` primary. Reconcile failure-scenario table with whether H-level diagnostics are implemented. |
|
||||
| [docs/RUNBOOK.md](../docs/RUNBOOK.md) | Useful but incomplete. | Add: consolidated template-refresh procedure; capacity burn-in results; recovery from stale lock; runner service-account identity. |
|
||||
| [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md) | Good. Old `WinBuild` path references need fixing. | Update all `WinBuild` to `WinBuild2025`. Add HGFS shared-folder cache poisoning note. |
|
||||
| [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Good. | Add one-paragraph note on Gitea secret-mask substring limitation. |
|
||||
| [docs/HOST-SETUP.md](../docs/HOST-SETUP.md) | Mostly accurate. | Update directory list to match new [Setup-Host.ps1](../Setup-Host.ps1). Add ACL guidance for `F:\CI\keys\ci_linux`. |
|
||||
| [docs/WINDOWS-TEMPLATE-SETUP.md](../docs/WINDOWS-TEMPLATE-SETUP.md) | Good. | Cross-reference the consolidated template-refresh runbook in [docs/RUNBOOK.md](../docs/RUNBOOK.md). |
|
||||
| [docs/LINUX-TEMPLATE-SETUP.md](../docs/LINUX-TEMPLATE-SETUP.md) | Good. | Add `Validate-DeployState.ps1` Linux branch usage. |
|
||||
| [docs/OPTIMIZATION.md](../docs/OPTIMIZATION.md) | Mostly accurate. | Note Defender exclusion option and shared-cache wiring after H-Medium fixes. |
|
||||
| [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) | Stale. PS 7 syntax, wrong parameter names, wrong task/event names. | Sprint 5 task 5.3: PS 5.1 syntax, `-RepoUrl`, current Event Log sources. |
|
||||
| [docs/Setup-GiteaSSH.md](../docs/Setup-GiteaSSH.md) | Untouched by reviewers; assume accurate. | Verify against fixed service-account path from H8. |
|
||||
| [TODO.md](../TODO.md) | Acting as a CHANGELOG. | Sprint 5 task: move "done" sections into `docs/CHANGELOG.md`; keep TODO for genuinely outstanding work. |
|
||||
| [docs/archived/2026-05-10/](../docs/archived/2026-05-10/) | Historical. | Leave as-is. |
|
||||
|
||||
---
|
||||
|
||||
## Final Completion Plan
|
||||
|
||||
Execute in this exact order. Each phase has a single success criterion that gates the next.
|
||||
|
||||
**Phase 0 — Read this plan and commit it.** Already done by the act of generating it.
|
||||
|
||||
**Phase 1 — Validation (Sprint 1).** Verify the act_runner service account, confirm the directory inventory, run the 4-way concurrency burn-in, manually trigger the broken Linux matrix to confirm C1. Output: a section appended to [docs/RUNBOOK.md](../docs/RUNBOOK.md) titled "Validation Results 2026-05-XX" documenting findings. Gate: burn-in passed three times (keep `capacity: 4`) or was set to 2 (with reasoning recorded).
|
||||
|
||||
**Phase 2 — Critical Fixes (Sprint 2).** Fix C1 and C2, plus H2/H3/H6/H7. Manually trigger [gitea/workflows/build-nsis.yml](../gitea/workflows/build-nsis.yml) for both matrix legs after each fix. Gate: both Critical items closed and `[x]` in this plan's Critical section; both legs green in the live matrix workflow.
|
||||
|
||||
**Phase 3 — High Priority Fixes (Sprint 3).** Complete the rest of the High items: H4 (bounded vmrun), H5 (stale lock), H8 (SSH alias fix), H9 (Linux validator), H10 (smoke-test workflow), H11 (script-root env var), H12 (benchmark alignment). Gate: smoke-test workflow green; all High items `[x]`.
|
||||
|
||||
**Phase 4 — Operational Hardening (Sprint 4).** All Medium items including consolidated template-refresh runbook. Gate: documented end-to-end template refresh executed successfully against a non-production template clone.
|
||||
|
||||
**Phase 5 — Polish (Sprint 5).** Low items, nice-to-have summary tool, documentation alignment, TODO → CHANGELOG migration. Gate: README claim updated from "feature-complete; stabilization pass in progress" back to "production-ready for homelab".
|
||||
|
||||
**Phase 6 — Tag v1.0.** Create an annotated tag `v1.0.0` on `main` once Phase 5 gate passes. Write a `docs/CHANGELOG.md` v1.0 entry summarizing the journey.
|
||||
|
||||
**Phase 7 — Operate.** Use the system. Track new findings in `TODO.md`. Run the burn-in any time act_runner capacity, host hardware, or template chains change materially.
|
||||
|
||||
When all Critical and High items are checked, this system is genuinely production-ready for homelab use.
|
||||
Reference in New Issue
Block a user