Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor documentation and analysis in final master plan, opus47 analysis, and opus47 review of GPT-5.5

Browse Source 
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12.
- Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps.
- Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
This commit is contained in:
Simone
2026-05-13 00:08:19 +02:00
parent 5d1b47e943
commit 3592dcab78
13 changed files with 350 additions and 343 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plans/opus47-analysis.md
+84 -84
View File
@@ -30,32 +30,32 @@ The rest of this document expands on these and 30+ smaller findings.
The system declares itself "production-ready" in `README.md` line 1. Measured against what a reasonable definition of that phrase requires, here is the state:
| Capability | Status | Evidence |
| --- | --- | --- |
| Ephemeral VM lifecycle (clone, start, destroy) | Complete | `scripts/New-BuildVM.ps1`, `Remove-BuildVM.ps1`, `Invoke-CIJob.ps1` |
| Windows guest (WinRM HTTPS) | Complete | `Invoke-RemoteBuild.ps1`, `Get-BuildArtifacts.ps1` Windows branch |
| Linux guest (SSH) | Complete | `_Transport.psm1`, Linux branches in build/artifact scripts |
| Auto-detect guest OS from VMX | Complete | `Invoke-CIJob.ps1` `guestOS` regex |
| Host-clone mode + zip transfer (Mode 1) | Complete | Default path in `Invoke-RemoteBuild.ps1` |
| In-VM `git clone` (Mode 2) | Complete | `-UseGitClone`, PAT via `http.extraHeader` |
| Credential storage (CredentialManager) | Complete | `BuildVMGuest`, `GiteaPAT` |
| Reusable composite action | Complete | `gitea/actions/local-ci-build/action.yml` |
| Build matrix Windows + Linux | Complete | `gitea/workflows/build-nsInnoUnp.yml` |
| Shared NuGet/pip cache via HGFS | Complete | `Set-TemplateSharedFolders.ps1`, `-UseSharedCache` |
| Structured JSONL logs | Complete | `Write-JobEvent` in `Invoke-CIJob.ps1` |
| Scheduled maintenance | Complete | `Register-CIScheduledTasks.ps1` (4 tasks) |
| Disk-space + runner-health watchdogs | Complete | `Watch-DiskSpace.ps1`, `Watch-RunnerHealth.ps1` |
| Retention policy with aggressive fallback | Complete | `Invoke-RetentionPolicy.ps1` |
| Orphan cleanup | Complete | `Cleanup-OrphanedBuildVMs.ps1` |
| Template backup before refresh | Complete | `Backup-CITemplate.ps1` |
| Benchmark harness | Complete | `Measure-CIBenchmark.ps1` |
| Secret injection into guest (`ExtraGuestEnv`) | Complete | Forwarded host → action → `Invoke-CIJob` → `Invoke-RemoteBuild` |
| Pester unit tests | Stub-level | `tests/*.Tests.ps1`, all use fake `vmrun.cmd` |
| Automated integration test | **Missing** | No automated harness exercises real VMs |
| Concurrency burn-in | **Missing** | `TODO.md §2.1` admits "race non riprodotta in e2e-008/009" |
| Multi-host federation | Not in scope | `TODO.md §6.3` deferred |
| Template snapshot refresh procedure | Partial | `Backup-CITemplate.ps1` exists; refresh procedure not codified |
| Setup-Host post-conditions documented | Partial | `Setup-Host.ps1` does the work but `Validate-DeployState.ps1` covers templates, not host |
| Capability | Status | Evidence |
| ---------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------- |
| Ephemeral VM lifecycle (clone, start, destroy) | Complete | `scripts/New-BuildVM.ps1`, `Remove-BuildVM.ps1`, `Invoke-CIJob.ps1` |
| Windows guest (WinRM HTTPS) | Complete | `Invoke-RemoteBuild.ps1`, `Get-BuildArtifacts.ps1` Windows branch |
| Linux guest (SSH) | Complete | `_Transport.psm1`, Linux branches in build/artifact scripts |
| Auto-detect guest OS from VMX | Complete | `Invoke-CIJob.ps1` `guestOS` regex |
| Host-clone mode + zip transfer (Mode 1) | Complete | Default path in `Invoke-RemoteBuild.ps1` |
| In-VM `git clone` (Mode 2) | Complete | `-UseGitClone`, PAT via `http.extraHeader` |
| Credential storage (CredentialManager) | Complete | `BuildVMGuest`, `GiteaPAT` |
| Reusable composite action | Complete | `gitea/actions/local-ci-build/action.yml` |
| Build matrix Windows + Linux | Complete | `gitea/workflows/build-nsInnoUnp.yml` |
| Shared NuGet/pip cache via HGFS | Complete | `Set-TemplateSharedFolders.ps1`, `-UseSharedCache` |
| Structured JSONL logs | Complete | `Write-JobEvent` in `Invoke-CIJob.ps1` |
| Scheduled maintenance | Complete | `Register-CIScheduledTasks.ps1` (4 tasks) |
| Disk-space + runner-health watchdogs | Complete | `Watch-DiskSpace.ps1`, `Watch-RunnerHealth.ps1` |
| Retention policy with aggressive fallback | Complete | `Invoke-RetentionPolicy.ps1` |
| Orphan cleanup | Complete | `Cleanup-OrphanedBuildVMs.ps1` |
| Template backup before refresh | Complete | `Backup-CITemplate.ps1` |
| Benchmark harness | Complete | `Measure-CIBenchmark.ps1` |
| Secret injection into guest (`ExtraGuestEnv`) | Complete | Forwarded host → action → `Invoke-CIJob` → `Invoke-RemoteBuild` |
| Pester unit tests | Stub-level | `tests/*.Tests.ps1`, all use fake `vmrun.cmd` |
| Automated integration test | **Missing** | No automated harness exercises real VMs |
| Concurrency burn-in | **Missing** | `TODO.md §2.1` admits "race non riprodotta in e2e-008/009" |
| Multi-host federation | Not in scope | `TODO.md §6.3` deferred |
| Template snapshot refresh procedure | Partial | `Backup-CITemplate.ps1` exists; refresh procedure not codified |
| Setup-Host post-conditions documented | Partial | `Setup-Host.ps1` does the work but `Validate-DeployState.ps1` covers templates, not host |
Overall completion against the original scope: **~85%**. The remaining 15% is concentrated in test coverage and concurrency validation — exactly the parts that justify the "production-ready" label.
@@ -284,18 +284,18 @@ Currently `[string] $GuestPassword = ''`, which triggers a prompt — fine. But
### 5.7 OWASP Top 10 Coverage (in scope)
| Risk | Status |
| --- | --- |
| A01 Broken Access Control | Local host only. Acceptable. |
| A02 Cryptographic Failures | Self-signed TLS for lab use. Acceptable. PAT base64 in command line is the only real concern (§5.2). |
| A03 Injection | `BuildCommand` is by-design caller-controlled. JSON env injection parses safely via `ConvertFrom-Json`. SSH commands built from string concatenation in `Invoke-RemoteBuild.ps1` Linux branch — `$GuestLinuxWorkDir`, `$CloneBranch`, etc. are not user-supplied in the trigger path but a workflow author could pass `'$(rm -rf /)'` in `submodules` etc. The `single-quote escape` on `ExtraGuestEnv` values is correct; other interpolated strings are not escaped. **MEDIUM** — see §6.7. |
| A04 Insecure Design | Acceptable for homelab. |
| A05 Security Misconfiguration | Defender/Firewall/UAC off, documented in BEST-PRACTICES. Acceptable for isolated lab. |
| A06 Vulnerable Components | `TODO.md §1.3 SHA256 pinning` was deferred. Toolchain installer downloads (Tier-1/Tier-2) use only placeholders. **MEDIUM** — see §6.8. |
| A07 Auth Failures | Shared `BuildVMGuest` account — see §5.4. |
| A08 Software/Data Integrity | No artifact signing. **NICE-TO-HAVE** for a homelab. |
| A09 Logging Failures | Strong (`invoke-ci.jsonl`, Event Log, webhook). Good. |
| A10 SSRF | Not applicable. |
| Risk | Status |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A01 Broken Access Control | Local host only. Acceptable. |
| A02 Cryptographic Failures | Self-signed TLS for lab use. Acceptable. PAT base64 in command line is the only real concern (§5.2). |
| A03 Injection | `BuildCommand` is by-design caller-controlled. JSON env injection parses safely via `ConvertFrom-Json`. SSH commands built from string concatenation in `Invoke-RemoteBuild.ps1` Linux branch — `$GuestLinuxWorkDir`, `$CloneBranch`, etc. are not user-supplied in the trigger path but a workflow author could pass `'$(rm -rf /)'` in `submodules` etc. The `single-quote escape` on `ExtraGuestEnv` values is correct; other interpolated strings are not escaped. **MEDIUM** — see §6.7. |
| A04 Insecure Design | Acceptable for homelab. |
| A05 Security Misconfiguration | Defender/Firewall/UAC off, documented in BEST-PRACTICES. Acceptable for isolated lab. |
| A06 Vulnerable Components | `TODO.md §1.3 SHA256 pinning` was deferred. Toolchain installer downloads (Tier-1/Tier-2) use only placeholders. **MEDIUM** — see §6.8. |
| A07 Auth Failures | Shared `BuildVMGuest` account — see §5.4. |
| A08 Software/Data Integrity | No artifact signing. **NICE-TO-HAVE** for a homelab. |
| A09 Logging Failures | Strong (`invoke-ci.jsonl`, Event Log, webhook). Good. |
| A10 SSRF | Not applicable. |
---
@@ -370,21 +370,21 @@ If a workflow author somehow gets `$GuestLinuxWorkDir` to contain `' && rm -rf /
## 7. Operational Gaps
| Gap | Severity | Notes |
| --- | --- | --- |
| No documented template-refresh runbook | HIGH | `Backup-CITemplate.ps1` exists, but the procedure (Win KMS reactivation, snapshot rebuild, validation) is not codified. |
| No automated end-to-end smoke test | HIGH | The "production-ready" claim rests on `e2e-008`/`e2e-009` manual runs. |
| `capacity: 4` never validated | HIGH | See §6.1. |
| No CI for the CI | HIGH | PSScriptAnalyzer lint workflow exists; no Pester run on commit; no integration test on commit. |
| No alerting on stuck jobs | MEDIUM | A job that hangs at 1 h 50 min (just under the 2 h runner timeout) is invisible. Watchdog only checks the runner service, not job duration. |
| No central log aggregation | MEDIUM | JSONL files are per-job. Cross-job analysis requires manual grep. A `Get-CIJobStats.ps1` summarizer would help. |
| No quota/limit on PR builds | MEDIUM | A misbehaving workflow could fork the orchestrator. Concurrency lock prevents IP collisions but not disk fill. `MaxAgeHours 4` provides eventual cleanup. |
| No multi-host fan-out | NICE-TO-HAVE | `§6.3` deferred. Single point of failure is the workstation. Acceptable for homelab. |
| No remote-access procedure for debugging a stuck VM | LOW | `Cleanup-OrphanedBuildVMs.ps1` will eventually destroy a stuck VM. Manual inspection between job and cleanup requires knowing the clone path. |
| No host hardware monitoring | LOW | CPU/RAM/disk-IO not measured. `Measure-CIBenchmark.ps1` is phase-time only. |
| `BaseClean` snapshot age not tracked | LOW | Windows KMS lease is 180 days. After that, builds will silently degrade (activation issues affecting `dotnet`). |
| `runner/Install-Runner.ps1` deprecated but present | LOW | Delete it. |
| `Setup-Host.ps1` password documentation mismatch | LOW | See §4.5. |
| Gap | Severity | Notes |
| --------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No documented template-refresh runbook | HIGH | `Backup-CITemplate.ps1` exists, but the procedure (Win KMS reactivation, snapshot rebuild, validation) is not codified. |
| No automated end-to-end smoke test | HIGH | The "production-ready" claim rests on `e2e-008`/`e2e-009` manual runs. |
| `capacity: 4` never validated | HIGH | See §6.1. |
| No CI for the CI | HIGH | PSScriptAnalyzer lint workflow exists; no Pester run on commit; no integration test on commit. |
| No alerting on stuck jobs | MEDIUM | A job that hangs at 1 h 50 min (just under the 2 h runner timeout) is invisible. Watchdog only checks the runner service, not job duration. |
| No central log aggregation | MEDIUM | JSONL files are per-job. Cross-job analysis requires manual grep. A `Get-CIJobStats.ps1` summarizer would help. |
| No quota/limit on PR builds | MEDIUM | A misbehaving workflow could fork the orchestrator. Concurrency lock prevents IP collisions but not disk fill. `MaxAgeHours 4` provides eventual cleanup. |
| No multi-host fan-out | NICE-TO-HAVE | `§6.3` deferred. Single point of failure is the workstation. Acceptable for homelab. |
| No remote-access procedure for debugging a stuck VM | LOW | `Cleanup-OrphanedBuildVMs.ps1` will eventually destroy a stuck VM. Manual inspection between job and cleanup requires knowing the clone path. |
| No host hardware monitoring | LOW | CPU/RAM/disk-IO not measured. `Measure-CIBenchmark.ps1` is phase-time only. |
| `BaseClean` snapshot age not tracked | LOW | Windows KMS lease is 180 days. After that, builds will silently degrade (activation issues affecting `dotnet`). |
| `runner/Install-Runner.ps1` deprecated but present | LOW | Delete it. |
| `Setup-Host.ps1` password documentation mismatch | LOW | See §4.5. |
---
@@ -436,15 +436,15 @@ Four Pester v5 files exist in `tests/`:
Per `TODO.md §5.1`, they all use a fake `vmrun.cmd` driven by `$env:FAKE_VMRUN_EXIT`. What this means in practice:
| Tested | Not Tested |
| --- | --- |
| Argument construction for `vmrun clone` | Whether `vmrun clone` actually clones |
| `Invoke-Vmrun` return shape | Whether `Invoke-Vmrun` works with real `vmrun.exe` |
| Parameter validation regexes | Whether `Test-NetConnection -Port 5986` works |
| Path resolution helpers | Whether `Start-Sleep` polling deadlines fire correctly |
| Throw-on-error paths via fake exit codes | The actual failure modes (`vmware-vmx.exe` file lock, DHCP timeout, etc.) |
| WinRM session options object shape | Whether WinRM actually accepts self-signed cert |
| Phase 1 readiness loop (with fake `vmrun list` output) | Whether real VM boot timing fits the deadline |
| Tested | Not Tested |
| ------------------------------------------------------ | ------------------------------------------------------------------------- |
| Argument construction for `vmrun clone` | Whether `vmrun clone` actually clones |
| `Invoke-Vmrun` return shape | Whether `Invoke-Vmrun` works with real `vmrun.exe` |
| Parameter validation regexes | Whether `Test-NetConnection -Port 5986` works |
| Path resolution helpers | Whether `Start-Sleep` polling deadlines fire correctly |
| Throw-on-error paths via fake exit codes | The actual failure modes (`vmware-vmx.exe` file lock, DHCP timeout, etc.) |
| WinRM session options object shape | Whether WinRM actually accepts self-signed cert |
| Phase 1 readiness loop (with fake `vmrun list` output) | Whether real VM boot timing fits the deadline |
This is meaningful coverage — the Pester suite catches regressions in argument shape, parameter validation, return types, and exit-code interpretation. It is NOT integration coverage. Calling this "production-ready" testing is overstating it.
@@ -464,18 +464,18 @@ Estimated test coverage (lines exercised by Pester / lines of production code):
This is the area I am most positive about. The system genuinely has better observability than most paid CI systems give you:
| Channel | Purpose | Quality |
| --- | --- | --- |
| `F:\CI\Logs\$JobId\invoke-ci.log` | Per-job transcript | High |
| `F:\CI\Logs\$JobId\invoke-ci.jsonl` | Per-job structured phase events | High |
| `F:\CI\Logs\benchmark.jsonl` | Cross-job benchmark trend | High |
| Windows Event Log `CI-DiskAlert` (EventId 1001) | Disk alert | Medium |
| Windows Event Log `CI-RunnerHealth` (EventIds 1002/1003/1004) | Runner state changes | High |
| Webhook (Discord/Gitea-compatible) | Real-time notification | High |
| `F:\CI\State\runner-restart-log.json` | Rolling 1-hour restart cooldown | High |
| `F:\CI\State\ip-leases\<ip>.lease` | Active IP allocations | Medium |
| Task Scheduler history | Scheduled-task health | Medium |
| PSScriptAnalyzer lint workflow output | Code health | Medium |
| Channel | Purpose | Quality |
| ------------------------------------------------------------- | ------------------------------- | ------- |
| `F:\CI\Logs\$JobId\invoke-ci.log` | Per-job transcript | High |
| `F:\CI\Logs\$JobId\invoke-ci.jsonl` | Per-job structured phase events | High |
| `F:\CI\Logs\benchmark.jsonl` | Cross-job benchmark trend | High |
| Windows Event Log `CI-DiskAlert` (EventId 1001) | Disk alert | Medium |
| Windows Event Log `CI-RunnerHealth` (EventIds 1002/1003/1004) | Runner state changes | High |
| Webhook (Discord/Gitea-compatible) | Real-time notification | High |
| `F:\CI\State\runner-restart-log.json` | Rolling 1-hour restart cooldown | High |
| `F:\CI\State\ip-leases\<ip>.lease` | Active IP allocations | Medium |
| Task Scheduler history | Scheduled-task health | Medium |
| PSScriptAnalyzer lint workflow output | Code health | Medium |
What is missing:
@@ -737,17 +737,17 @@ What stops me from agreeing with the `README.md` claim of "production-ready":
Honest grade:
| Dimension | Grade |
| --- | --- |
| Architecture | A- |
| Code Quality | B+ |
| Security (in scope) | B |
| Dimension | Grade |
| ---------------------- | ------------------------------------ |
| Architecture | A- |
| Code Quality | B+ |
| Security (in scope) | B |
| Reliability (in scope) | B- (would be A- with §6.1 validated) |
| Test Coverage | C |
| Observability | A- |
| Documentation | B+ |
| Operational Readiness | B- |
| Overall | **B+** |
| Test Coverage | C |
| Observability | A- |
| Documentation | B+ |
| Operational Readiness | B- |
| Overall | **B+** |
**Production-ready for the homelab use case it was built for**, with the caveats above. **Not production-ready by any commercial standard**, but it was never built to be, and that is fine.