Refactor documentation and analysis in final master plan, opus47 analysis, and opus47 review of GPT-5.5
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12. - Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps. - Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
This commit is contained in:
Simone
+12
-12
@@ -24,19 +24,19 @@ Top 3 risks today, in order: (1) the Linux matrix leg builds against the Windows
|
||||
|
||||
The optimistic [TODO.md](../TODO.md) and the conservative [docs/TEST-PLAN-v1.3-to-HEAD.md](../docs/TEST-PLAN-v1.3-to-HEAD.md) disagree. The honest state, reconciled against the source files, is below.
|
||||
|
||||
| Area | Genuinely Done | Partially Done | Unvalidated / Broken |
|
||||
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
|
||||
| Windows VM lifecycle | Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1), [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1), [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1)) | — | Real 4-way concurrency |
|
||||
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | **FIXED 2026-05-12**: Matrix workflow Linux leg now picks correct template (C1) |
|
||||
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | **FIXED 2026-05-12**: `outputs:` block corrected (H2); hardcoded `N:\` replaced with env var (H11) |
|
||||
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | **FIXED 2026-05-12**: Stale lock auto-cleanup added (H5) | 4-way burn-in never run |
|
||||
| Area | Genuinely Done | Partially Done | Unvalidated / Broken |
|
||||
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
|
||||
| Windows VM lifecycle | Clone, start, IP detect, WinRM HTTPS, build, artifacts, destroy ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1), [scripts/New-BuildVM.ps1](../scripts/New-BuildVM.ps1), [scripts/Wait-VMReady.ps1](../scripts/Wait-VMReady.ps1), [scripts/Invoke-RemoteBuild.ps1](../scripts/Invoke-RemoteBuild.ps1), [scripts/Get-BuildArtifacts.ps1](../scripts/Get-BuildArtifacts.ps1), [scripts/Remove-BuildVM.ps1](../scripts/Remove-BuildVM.ps1)) | — | Real 4-way concurrency |
|
||||
| Linux VM lifecycle | Cloud-init deploy, SSH transport, IP via `guestinfo.ci-ip` ([template/Deploy-LinuxBuild2404.ps1](../template/Deploy-LinuxBuild2404.ps1), [scripts/_Transport.psm1](../scripts/_Transport.psm1)) | Generic validator (Windows-only) ([template/Validate-DeployState.ps1](../template/Validate-DeployState.ps1)) | **FIXED 2026-05-12**: Matrix workflow Linux leg now picks correct template (C1) |
|
||||
| Composite action | Inputs forwarded via env vars, step outputs work, success and failure upload paths ([gitea/actions/local-ci-build/action.yml](../gitea/actions/local-ci-build/action.yml)) | — | **FIXED 2026-05-12**: `outputs:` block corrected (H2); hardcoded `N:\` replaced with env var (H11) |
|
||||
| Concurrency | File mutex + lease design ([scripts/Invoke-CIJob.ps1](../scripts/Invoke-CIJob.ps1) ~L303-L460) | **FIXED 2026-05-12**: Stale lock auto-cleanup added (H5) | 4-way burn-in never run |
|
||||
| Credential handling | Windows Credential Manager target `BuildVMGuest`; PAT via Credential Manager target `GiteaPAT`; Windows Mode 2 uses `http.extraHeader` | — | **FIXED 2026-05-12**: Linux Mode 2 PAT now uses `http.extraHeader` (H3); `ExtraGuestEnv` logging redacted (C2) |
|
||||
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | — | — |
|
||||
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional; **FIXED 2026-05-12**: default `WinBuild2025`, `-AllTemplates` switch added (H7) | — | — |
|
||||
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)); **FIXED 2026-05-12**: directory array updated (H6); `F:\CI\Cache\pip` created on live host | — | act_runner runs as LocalSystem — H8 SSH alias mismatch confirmed, not yet fixed |
|
||||
| Observability | JSONL per-job events, Event Log, optional webhook | Phase-duration summary | Cross-job analysis tool |
|
||||
| Tests | Argument-parsing Pester suite over fake `vmrun.cmd` (4 files in [tests/](../tests/)) | — | Orchestrator state machine, real-VM smoke tests, lock concurrency |
|
||||
| Documentation | [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/RUNBOOK.md](../docs/RUNBOOK.md), [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md), [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names | Consolidated template-refresh runbook |
|
||||
| Maintenance | Disk watcher, runner health watcher, retention, orphan cleanup, scheduled tasks ([scripts/Watch-DiskSpace.ps1](../scripts/Watch-DiskSpace.ps1), [scripts/Watch-RunnerHealth.ps1](../scripts/Watch-RunnerHealth.ps1), [scripts/Invoke-RetentionPolicy.ps1](../scripts/Invoke-RetentionPolicy.ps1), [scripts/Cleanup-OrphanedBuildVMs.ps1](../scripts/Cleanup-OrphanedBuildVMs.ps1), [scripts/Register-CIScheduledTasks.ps1](../scripts/Register-CIScheduledTasks.ps1)) | — | — |
|
||||
| Backup | [scripts/Backup-CITemplate.ps1](../scripts/Backup-CITemplate.ps1) functional; **FIXED 2026-05-12**: default `WinBuild2025`, `-AllTemplates` switch added (H7) | — | — |
|
||||
| Host bootstrap | Runner install, NSSM service, credential prompts ([Setup-Host.ps1](../Setup-Host.ps1)); **FIXED 2026-05-12**: directory array updated (H6); `F:\CI\Cache\pip` created on live host | — | act_runner runs as LocalSystem — H8 SSH alias mismatch confirmed, not yet fixed |
|
||||
| Observability | JSONL per-job events, Event Log, optional webhook | Phase-duration summary | Cross-job analysis tool |
|
||||
| Tests | Argument-parsing Pester suite over fake `vmrun.cmd` (4 files in [tests/](../tests/)) | — | Orchestrator state machine, real-VM smoke tests, lock concurrency |
|
||||
| Documentation | [docs/ARCHITECTURE.md](../docs/ARCHITECTURE.md), [docs/CI-FLOW.md](../docs/CI-FLOW.md), [docs/RUNBOOK.md](../docs/RUNBOOK.md), [docs/BEST-PRACTICES.md](../docs/BEST-PRACTICES.md), [docs/WORKFLOW-AUTHORING.md](../docs/WORKFLOW-AUTHORING.md) | Drift: README claims production-ready; CI-FLOW claims partial-artifacts-on-failure; test plan uses PS7 syntax and wrong param names | Consolidated template-refresh runbook |
|
||||
|
||||
The README is genuinely ahead of the evidence. The accurate label is *feature-complete homelab CI awaiting stabilization pass*. Everything in this plan exists to close that gap.
|
||||
|
||||
|
||||
+84
-84
@@ -30,32 +30,32 @@ The rest of this document expands on these and 30+ smaller findings.
|
||||
|
||||
The system declares itself "production-ready" in `README.md` line 1. Measured against what a reasonable definition of that phrase requires, here is the state:
|
||||
|
||||
| Capability | Status | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Ephemeral VM lifecycle (clone, start, destroy) | Complete | `scripts/New-BuildVM.ps1`, `Remove-BuildVM.ps1`, `Invoke-CIJob.ps1` |
|
||||
| Windows guest (WinRM HTTPS) | Complete | `Invoke-RemoteBuild.ps1`, `Get-BuildArtifacts.ps1` Windows branch |
|
||||
| Linux guest (SSH) | Complete | `_Transport.psm1`, Linux branches in build/artifact scripts |
|
||||
| Auto-detect guest OS from VMX | Complete | `Invoke-CIJob.ps1` `guestOS` regex |
|
||||
| Host-clone mode + zip transfer (Mode 1) | Complete | Default path in `Invoke-RemoteBuild.ps1` |
|
||||
| In-VM `git clone` (Mode 2) | Complete | `-UseGitClone`, PAT via `http.extraHeader` |
|
||||
| Credential storage (CredentialManager) | Complete | `BuildVMGuest`, `GiteaPAT` |
|
||||
| Reusable composite action | Complete | `gitea/actions/local-ci-build/action.yml` |
|
||||
| Build matrix Windows + Linux | Complete | `gitea/workflows/build-nsInnoUnp.yml` |
|
||||
| Shared NuGet/pip cache via HGFS | Complete | `Set-TemplateSharedFolders.ps1`, `-UseSharedCache` |
|
||||
| Structured JSONL logs | Complete | `Write-JobEvent` in `Invoke-CIJob.ps1` |
|
||||
| Scheduled maintenance | Complete | `Register-CIScheduledTasks.ps1` (4 tasks) |
|
||||
| Disk-space + runner-health watchdogs | Complete | `Watch-DiskSpace.ps1`, `Watch-RunnerHealth.ps1` |
|
||||
| Retention policy with aggressive fallback | Complete | `Invoke-RetentionPolicy.ps1` |
|
||||
| Orphan cleanup | Complete | `Cleanup-OrphanedBuildVMs.ps1` |
|
||||
| Template backup before refresh | Complete | `Backup-CITemplate.ps1` |
|
||||
| Benchmark harness | Complete | `Measure-CIBenchmark.ps1` |
|
||||
| Secret injection into guest (`ExtraGuestEnv`) | Complete | Forwarded host → action → `Invoke-CIJob` → `Invoke-RemoteBuild` |
|
||||
| Pester unit tests | Stub-level | `tests/*.Tests.ps1`, all use fake `vmrun.cmd` |
|
||||
| Automated integration test | **Missing** | No automated harness exercises real VMs |
|
||||
| Concurrency burn-in | **Missing** | `TODO.md §2.1` admits "race non riprodotta in e2e-008/009" |
|
||||
| Multi-host federation | Not in scope | `TODO.md §6.3` deferred |
|
||||
| Template snapshot refresh procedure | Partial | `Backup-CITemplate.ps1` exists; refresh procedure not codified |
|
||||
| Setup-Host post-conditions documented | Partial | `Setup-Host.ps1` does the work but `Validate-DeployState.ps1` covers templates, not host |
|
||||
| Capability | Status | Evidence |
|
||||
| ---------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------- |
|
||||
| Ephemeral VM lifecycle (clone, start, destroy) | Complete | `scripts/New-BuildVM.ps1`, `Remove-BuildVM.ps1`, `Invoke-CIJob.ps1` |
|
||||
| Windows guest (WinRM HTTPS) | Complete | `Invoke-RemoteBuild.ps1`, `Get-BuildArtifacts.ps1` Windows branch |
|
||||
| Linux guest (SSH) | Complete | `_Transport.psm1`, Linux branches in build/artifact scripts |
|
||||
| Auto-detect guest OS from VMX | Complete | `Invoke-CIJob.ps1` `guestOS` regex |
|
||||
| Host-clone mode + zip transfer (Mode 1) | Complete | Default path in `Invoke-RemoteBuild.ps1` |
|
||||
| In-VM `git clone` (Mode 2) | Complete | `-UseGitClone`, PAT via `http.extraHeader` |
|
||||
| Credential storage (CredentialManager) | Complete | `BuildVMGuest`, `GiteaPAT` |
|
||||
| Reusable composite action | Complete | `gitea/actions/local-ci-build/action.yml` |
|
||||
| Build matrix Windows + Linux | Complete | `gitea/workflows/build-nsInnoUnp.yml` |
|
||||
| Shared NuGet/pip cache via HGFS | Complete | `Set-TemplateSharedFolders.ps1`, `-UseSharedCache` |
|
||||
| Structured JSONL logs | Complete | `Write-JobEvent` in `Invoke-CIJob.ps1` |
|
||||
| Scheduled maintenance | Complete | `Register-CIScheduledTasks.ps1` (4 tasks) |
|
||||
| Disk-space + runner-health watchdogs | Complete | `Watch-DiskSpace.ps1`, `Watch-RunnerHealth.ps1` |
|
||||
| Retention policy with aggressive fallback | Complete | `Invoke-RetentionPolicy.ps1` |
|
||||
| Orphan cleanup | Complete | `Cleanup-OrphanedBuildVMs.ps1` |
|
||||
| Template backup before refresh | Complete | `Backup-CITemplate.ps1` |
|
||||
| Benchmark harness | Complete | `Measure-CIBenchmark.ps1` |
|
||||
| Secret injection into guest (`ExtraGuestEnv`) | Complete | Forwarded host → action → `Invoke-CIJob` → `Invoke-RemoteBuild` |
|
||||
| Pester unit tests | Stub-level | `tests/*.Tests.ps1`, all use fake `vmrun.cmd` |
|
||||
| Automated integration test | **Missing** | No automated harness exercises real VMs |
|
||||
| Concurrency burn-in | **Missing** | `TODO.md §2.1` admits "race non riprodotta in e2e-008/009" |
|
||||
| Multi-host federation | Not in scope | `TODO.md §6.3` deferred |
|
||||
| Template snapshot refresh procedure | Partial | `Backup-CITemplate.ps1` exists; refresh procedure not codified |
|
||||
| Setup-Host post-conditions documented | Partial | `Setup-Host.ps1` does the work but `Validate-DeployState.ps1` covers templates, not host |
|
||||
|
||||
Overall completion against the original scope: **~85%**. The remaining 15% is concentrated in test coverage and concurrency validation — exactly the parts that justify the "production-ready" label.
|
||||
|
||||
@@ -284,18 +284,18 @@ Currently `[string] $GuestPassword = ''`, which triggers a prompt — fine. But
|
||||
|
||||
### 5.7 OWASP Top 10 Coverage (in scope)
|
||||
|
||||
| Risk | Status |
|
||||
| --- | --- |
|
||||
| A01 Broken Access Control | Local host only. Acceptable. |
|
||||
| A02 Cryptographic Failures | Self-signed TLS for lab use. Acceptable. PAT base64 in command line is the only real concern (§5.2). |
|
||||
| A03 Injection | `BuildCommand` is by-design caller-controlled. JSON env injection parses safely via `ConvertFrom-Json`. SSH commands built from string concatenation in `Invoke-RemoteBuild.ps1` Linux branch — `$GuestLinuxWorkDir`, `$CloneBranch`, etc. are not user-supplied in the trigger path but a workflow author could pass `'$(rm -rf /)'` in `submodules` etc. The `single-quote escape` on `ExtraGuestEnv` values is correct; other interpolated strings are not escaped. **MEDIUM** — see §6.7. |
|
||||
| A04 Insecure Design | Acceptable for homelab. |
|
||||
| A05 Security Misconfiguration | Defender/Firewall/UAC off, documented in BEST-PRACTICES. Acceptable for isolated lab. |
|
||||
| A06 Vulnerable Components | `TODO.md §1.3 SHA256 pinning` was deferred. Toolchain installer downloads (Tier-1/Tier-2) use only placeholders. **MEDIUM** — see §6.8. |
|
||||
| A07 Auth Failures | Shared `BuildVMGuest` account — see §5.4. |
|
||||
| A08 Software/Data Integrity | No artifact signing. **NICE-TO-HAVE** for a homelab. |
|
||||
| A09 Logging Failures | Strong (`invoke-ci.jsonl`, Event Log, webhook). Good. |
|
||||
| A10 SSRF | Not applicable. |
|
||||
| Risk | Status |
|
||||
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| A01 Broken Access Control | Local host only. Acceptable. |
|
||||
| A02 Cryptographic Failures | Self-signed TLS for lab use. Acceptable. PAT base64 in command line is the only real concern (§5.2). |
|
||||
| A03 Injection | `BuildCommand` is by-design caller-controlled. JSON env injection parses safely via `ConvertFrom-Json`. SSH commands built from string concatenation in `Invoke-RemoteBuild.ps1` Linux branch — `$GuestLinuxWorkDir`, `$CloneBranch`, etc. are not user-supplied in the trigger path but a workflow author could pass `'$(rm -rf /)'` in `submodules` etc. The `single-quote escape` on `ExtraGuestEnv` values is correct; other interpolated strings are not escaped. **MEDIUM** — see §6.7. |
|
||||
| A04 Insecure Design | Acceptable for homelab. |
|
||||
| A05 Security Misconfiguration | Defender/Firewall/UAC off, documented in BEST-PRACTICES. Acceptable for isolated lab. |
|
||||
| A06 Vulnerable Components | `TODO.md §1.3 SHA256 pinning` was deferred. Toolchain installer downloads (Tier-1/Tier-2) use only placeholders. **MEDIUM** — see §6.8. |
|
||||
| A07 Auth Failures | Shared `BuildVMGuest` account — see §5.4. |
|
||||
| A08 Software/Data Integrity | No artifact signing. **NICE-TO-HAVE** for a homelab. |
|
||||
| A09 Logging Failures | Strong (`invoke-ci.jsonl`, Event Log, webhook). Good. |
|
||||
| A10 SSRF | Not applicable. |
|
||||
|
||||
---
|
||||
|
||||
@@ -370,21 +370,21 @@ If a workflow author somehow gets `$GuestLinuxWorkDir` to contain `' && rm -rf /
|
||||
|
||||
## 7. Operational Gaps
|
||||
|
||||
| Gap | Severity | Notes |
|
||||
| --- | --- | --- |
|
||||
| No documented template-refresh runbook | HIGH | `Backup-CITemplate.ps1` exists, but the procedure (Win KMS reactivation, snapshot rebuild, validation) is not codified. |
|
||||
| No automated end-to-end smoke test | HIGH | The "production-ready" claim rests on `e2e-008`/`e2e-009` manual runs. |
|
||||
| `capacity: 4` never validated | HIGH | See §6.1. |
|
||||
| No CI for the CI | HIGH | PSScriptAnalyzer lint workflow exists; no Pester run on commit; no integration test on commit. |
|
||||
| No alerting on stuck jobs | MEDIUM | A job that hangs at 1 h 50 min (just under the 2 h runner timeout) is invisible. Watchdog only checks the runner service, not job duration. |
|
||||
| No central log aggregation | MEDIUM | JSONL files are per-job. Cross-job analysis requires manual grep. A `Get-CIJobStats.ps1` summarizer would help. |
|
||||
| No quota/limit on PR builds | MEDIUM | A misbehaving workflow could fork the orchestrator. Concurrency lock prevents IP collisions but not disk fill. `MaxAgeHours 4` provides eventual cleanup. |
|
||||
| No multi-host fan-out | NICE-TO-HAVE | `§6.3` deferred. Single point of failure is the workstation. Acceptable for homelab. |
|
||||
| No remote-access procedure for debugging a stuck VM | LOW | `Cleanup-OrphanedBuildVMs.ps1` will eventually destroy a stuck VM. Manual inspection between job and cleanup requires knowing the clone path. |
|
||||
| No host hardware monitoring | LOW | CPU/RAM/disk-IO not measured. `Measure-CIBenchmark.ps1` is phase-time only. |
|
||||
| `BaseClean` snapshot age not tracked | LOW | Windows KMS lease is 180 days. After that, builds will silently degrade (activation issues affecting `dotnet`). |
|
||||
| `runner/Install-Runner.ps1` deprecated but present | LOW | Delete it. |
|
||||
| `Setup-Host.ps1` password documentation mismatch | LOW | See §4.5. |
|
||||
| Gap | Severity | Notes |
|
||||
| --------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| No documented template-refresh runbook | HIGH | `Backup-CITemplate.ps1` exists, but the procedure (Win KMS reactivation, snapshot rebuild, validation) is not codified. |
|
||||
| No automated end-to-end smoke test | HIGH | The "production-ready" claim rests on `e2e-008`/`e2e-009` manual runs. |
|
||||
| `capacity: 4` never validated | HIGH | See §6.1. |
|
||||
| No CI for the CI | HIGH | PSScriptAnalyzer lint workflow exists; no Pester run on commit; no integration test on commit. |
|
||||
| No alerting on stuck jobs | MEDIUM | A job that hangs at 1 h 50 min (just under the 2 h runner timeout) is invisible. Watchdog only checks the runner service, not job duration. |
|
||||
| No central log aggregation | MEDIUM | JSONL files are per-job. Cross-job analysis requires manual grep. A `Get-CIJobStats.ps1` summarizer would help. |
|
||||
| No quota/limit on PR builds | MEDIUM | A misbehaving workflow could fork the orchestrator. Concurrency lock prevents IP collisions but not disk fill. `MaxAgeHours 4` provides eventual cleanup. |
|
||||
| No multi-host fan-out | NICE-TO-HAVE | `§6.3` deferred. Single point of failure is the workstation. Acceptable for homelab. |
|
||||
| No remote-access procedure for debugging a stuck VM | LOW | `Cleanup-OrphanedBuildVMs.ps1` will eventually destroy a stuck VM. Manual inspection between job and cleanup requires knowing the clone path. |
|
||||
| No host hardware monitoring | LOW | CPU/RAM/disk-IO not measured. `Measure-CIBenchmark.ps1` is phase-time only. |
|
||||
| `BaseClean` snapshot age not tracked | LOW | Windows KMS lease is 180 days. After that, builds will silently degrade (activation issues affecting `dotnet`). |
|
||||
| `runner/Install-Runner.ps1` deprecated but present | LOW | Delete it. |
|
||||
| `Setup-Host.ps1` password documentation mismatch | LOW | See §4.5. |
|
||||
|
||||
---
|
||||
|
||||
@@ -436,15 +436,15 @@ Four Pester v5 files exist in `tests/`:
|
||||
|
||||
Per `TODO.md §5.1`, they all use a fake `vmrun.cmd` driven by `$env:FAKE_VMRUN_EXIT`. What this means in practice:
|
||||
|
||||
| Tested | Not Tested |
|
||||
| --- | --- |
|
||||
| Argument construction for `vmrun clone` | Whether `vmrun clone` actually clones |
|
||||
| `Invoke-Vmrun` return shape | Whether `Invoke-Vmrun` works with real `vmrun.exe` |
|
||||
| Parameter validation regexes | Whether `Test-NetConnection -Port 5986` works |
|
||||
| Path resolution helpers | Whether `Start-Sleep` polling deadlines fire correctly |
|
||||
| Throw-on-error paths via fake exit codes | The actual failure modes (`vmware-vmx.exe` file lock, DHCP timeout, etc.) |
|
||||
| WinRM session options object shape | Whether WinRM actually accepts self-signed cert |
|
||||
| Phase 1 readiness loop (with fake `vmrun list` output) | Whether real VM boot timing fits the deadline |
|
||||
| Tested | Not Tested |
|
||||
| ------------------------------------------------------ | ------------------------------------------------------------------------- |
|
||||
| Argument construction for `vmrun clone` | Whether `vmrun clone` actually clones |
|
||||
| `Invoke-Vmrun` return shape | Whether `Invoke-Vmrun` works with real `vmrun.exe` |
|
||||
| Parameter validation regexes | Whether `Test-NetConnection -Port 5986` works |
|
||||
| Path resolution helpers | Whether `Start-Sleep` polling deadlines fire correctly |
|
||||
| Throw-on-error paths via fake exit codes | The actual failure modes (`vmware-vmx.exe` file lock, DHCP timeout, etc.) |
|
||||
| WinRM session options object shape | Whether WinRM actually accepts self-signed cert |
|
||||
| Phase 1 readiness loop (with fake `vmrun list` output) | Whether real VM boot timing fits the deadline |
|
||||
|
||||
This is meaningful coverage — the Pester suite catches regressions in argument shape, parameter validation, return types, and exit-code interpretation. It is NOT integration coverage. Calling this "production-ready" testing is overstating it.
|
||||
|
||||
@@ -464,18 +464,18 @@ Estimated test coverage (lines exercised by Pester / lines of production code):
|
||||
|
||||
This is the area I am most positive about. The system genuinely has better observability than most paid CI systems give you:
|
||||
|
||||
| Channel | Purpose | Quality |
|
||||
| --- | --- | --- |
|
||||
| `F:\CI\Logs\$JobId\invoke-ci.log` | Per-job transcript | High |
|
||||
| `F:\CI\Logs\$JobId\invoke-ci.jsonl` | Per-job structured phase events | High |
|
||||
| `F:\CI\Logs\benchmark.jsonl` | Cross-job benchmark trend | High |
|
||||
| Windows Event Log `CI-DiskAlert` (EventId 1001) | Disk alert | Medium |
|
||||
| Windows Event Log `CI-RunnerHealth` (EventIds 1002/1003/1004) | Runner state changes | High |
|
||||
| Webhook (Discord/Gitea-compatible) | Real-time notification | High |
|
||||
| `F:\CI\State\runner-restart-log.json` | Rolling 1-hour restart cooldown | High |
|
||||
| `F:\CI\State\ip-leases\<ip>.lease` | Active IP allocations | Medium |
|
||||
| Task Scheduler history | Scheduled-task health | Medium |
|
||||
| PSScriptAnalyzer lint workflow output | Code health | Medium |
|
||||
| Channel | Purpose | Quality |
|
||||
| ------------------------------------------------------------- | ------------------------------- | ------- |
|
||||
| `F:\CI\Logs\$JobId\invoke-ci.log` | Per-job transcript | High |
|
||||
| `F:\CI\Logs\$JobId\invoke-ci.jsonl` | Per-job structured phase events | High |
|
||||
| `F:\CI\Logs\benchmark.jsonl` | Cross-job benchmark trend | High |
|
||||
| Windows Event Log `CI-DiskAlert` (EventId 1001) | Disk alert | Medium |
|
||||
| Windows Event Log `CI-RunnerHealth` (EventIds 1002/1003/1004) | Runner state changes | High |
|
||||
| Webhook (Discord/Gitea-compatible) | Real-time notification | High |
|
||||
| `F:\CI\State\runner-restart-log.json` | Rolling 1-hour restart cooldown | High |
|
||||
| `F:\CI\State\ip-leases\<ip>.lease` | Active IP allocations | Medium |
|
||||
| Task Scheduler history | Scheduled-task health | Medium |
|
||||
| PSScriptAnalyzer lint workflow output | Code health | Medium |
|
||||
|
||||
What is missing:
|
||||
|
||||
@@ -737,17 +737,17 @@ What stops me from agreeing with the `README.md` claim of "production-ready":
|
||||
|
||||
Honest grade:
|
||||
|
||||
| Dimension | Grade |
|
||||
| --- | --- |
|
||||
| Architecture | A- |
|
||||
| Code Quality | B+ |
|
||||
| Security (in scope) | B |
|
||||
| Dimension | Grade |
|
||||
| ---------------------- | ------------------------------------ |
|
||||
| Architecture | A- |
|
||||
| Code Quality | B+ |
|
||||
| Security (in scope) | B |
|
||||
| Reliability (in scope) | B- (would be A- with §6.1 validated) |
|
||||
| Test Coverage | C |
|
||||
| Observability | A- |
|
||||
| Documentation | B+ |
|
||||
| Operational Readiness | B- |
|
||||
| Overall | **B+** |
|
||||
| Test Coverage | C |
|
||||
| Observability | A- |
|
||||
| Documentation | B+ |
|
||||
| Operational Readiness | B- |
|
||||
| Overall | **B+** |
|
||||
|
||||
**Production-ready for the homelab use case it was built for**, with the caveats above. **Not production-ready by any commercial standard**, but it was never built to be, and that is fine.
|
||||
|
||||
|
||||
@@ -241,26 +241,26 @@ The other GPT-5.5 recommendations respect PS 5.1, VMware Workstation single-host
|
||||
|
||||
## 7. Priority Validation
|
||||
|
||||
| GPT-5.5 Severity | Item | My Verdict | Rationale |
|
||||
| --- | --- | --- | --- |
|
||||
| CRITICAL | Composite action outputs structure | Should be HIGH | Step-level outputs work today; only caller-facing action interface is broken. Latent, not active. |
|
||||
| CRITICAL | Linux extra guest env secret logging | CRITICAL (concur) | Active leak path. |
|
||||
| CRITICAL | Linux Mode 2 PAT in SSH command | HIGH-CRITICAL boundary | PAT visibility limited to host argv + guest /proc. Real but bounded. I would put HIGH; CRITICAL is defensible. |
|
||||
| HIGH | vmrun unbounded calls | HIGH (concur) | |
|
||||
| HIGH | Stale vm-start.lock | HIGH (concur) | |
|
||||
| HIGH | Setup-Host stale dirs | HIGH (concur) | |
|
||||
| HIGH | Backup default path | HIGH (concur) | |
|
||||
| HIGH | Failure-path diagnostics | Should be MEDIUM | Quality-of-life, not operational-trust. |
|
||||
| HIGH | Test plan PS7 / wrong params | Should be MEDIUM | Documentation-only. |
|
||||
| HIGH | Shared cache not wired | Should be MEDIUM | Documented feature missing, performance impact only. |
|
||||
| MEDIUM | JobId not sanitized | Should be LOW | Local-only attack surface in single-tenant homelab. |
|
||||
| MEDIUM | Wait-VMReady TCP-only check | MEDIUM (concur) | |
|
||||
| MEDIUM | CI-FLOW doc claims partial artifacts | Should be LOW | Pure docs. |
|
||||
| MEDIUM | Validate-SetupState PasswordExpires | MEDIUM (concur) | Validation failure mode. |
|
||||
| MEDIUM | Ubuntu VMDK no hash | MEDIUM (concur) | |
|
||||
| MEDIUM | Installer hash placeholders | MEDIUM (concur for homelab) | |
|
||||
| MEDIUM | Event Log / task name drift | Should be LOW | Tests fail not scripts. |
|
||||
| MEDIUM | _Transport.psm1 host-key handling | MEDIUM (concur) | |
|
||||
| GPT-5.5 Severity | Item | My Verdict | Rationale |
|
||||
| ---------------- | ------------------------------------ | --------------------------- | -------------------------------------------------------------------------------------------------------------- |
|
||||
| CRITICAL | Composite action outputs structure | Should be HIGH | Step-level outputs work today; only caller-facing action interface is broken. Latent, not active. |
|
||||
| CRITICAL | Linux extra guest env secret logging | CRITICAL (concur) | Active leak path. |
|
||||
| CRITICAL | Linux Mode 2 PAT in SSH command | HIGH-CRITICAL boundary | PAT visibility limited to host argv + guest /proc. Real but bounded. I would put HIGH; CRITICAL is defensible. |
|
||||
| HIGH | vmrun unbounded calls | HIGH (concur) | |
|
||||
| HIGH | Stale vm-start.lock | HIGH (concur) | |
|
||||
| HIGH | Setup-Host stale dirs | HIGH (concur) | |
|
||||
| HIGH | Backup default path | HIGH (concur) | |
|
||||
| HIGH | Failure-path diagnostics | Should be MEDIUM | Quality-of-life, not operational-trust. |
|
||||
| HIGH | Test plan PS7 / wrong params | Should be MEDIUM | Documentation-only. |
|
||||
| HIGH | Shared cache not wired | Should be MEDIUM | Documented feature missing, performance impact only. |
|
||||
| MEDIUM | JobId not sanitized | Should be LOW | Local-only attack surface in single-tenant homelab. |
|
||||
| MEDIUM | Wait-VMReady TCP-only check | MEDIUM (concur) | |
|
||||
| MEDIUM | CI-FLOW doc claims partial artifacts | Should be LOW | Pure docs. |
|
||||
| MEDIUM | Validate-SetupState PasswordExpires | MEDIUM (concur) | Validation failure mode. |
|
||||
| MEDIUM | Ubuntu VMDK no hash | MEDIUM (concur) | |
|
||||
| MEDIUM | Installer hash placeholders | MEDIUM (concur for homelab) | |
|
||||
| MEDIUM | Event Log / task name drift | Should be LOW | Tests fail not scripts. |
|
||||
| MEDIUM | _Transport.psm1 host-key handling | MEDIUM (concur) | |
|
||||
|
||||
Items I would add at HIGH that GPT-5.5 missed: `act_runner` SYSTEM-vs-user SSH alias mismatch (verifiable, runtime-reachable), `ci-report-ip.service` missing from `Validate-DeployState.ps1` (silent 120 s timeout), `capacity: 4` burn-in not run (concurrency claim unmeasured).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user