feat(sprint3): §4.1/4.3/4.4 — JSONL logging, disk alert, runbook

§4.1 — Structured JSONL log (Invoke-CIJob.ps1) Each job now emits a parallel invoke-ci.jsonl alongside the transcript. Write-JobEvent appends one JSON line per phase transition (ts, jobId, phase, status, data). Events emitted at: job.start, phase1.clone-repo start/success, phase2-3b.vm-start start/success, phase4.wait-ready start/success, phase5.build start/success, phase6.artifacts start/success, job.success/failure (with elapsedSec and error string). Silent no-op if $jsonLog is null (log dir setup failed). Errors in Write-JobEvent are swallowed — logging never breaks the build. Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json. §4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1) Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks). Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log (source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run. Optional -WebhookUrl for Discord/Gitea webhook notification. Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min). §4.4 — Incident runbook (docs/RUNBOOK.md) Four scenarios with symptom / triage commands / fix / escalation: 1. Runner offline in Gitea UI 2. All builds fail in Phase 2 (clone/start/IP) 3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks) 4. Template VMX corrupt after host crash (lock removal + backup restore) Quick-reference table at the end. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 19:07:48 +02:00
parent bc894a8058
commit 1962c977ba
5 changed files with 394 additions and 6 deletions
@@ -0,0 +1,212 @@
+# CI System Runbook
+
+Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.
+
+---
+
+## 1. Runner offline in Gitea UI
+
+**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline.
+Queued jobs stay pending indefinitely.
+
+**Triage**:
+```powershell
+# Check service state
+Get-Service act_runner
+
+# Last 50 lines of runner log
+Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50
+
+# Check registration file is intact
+Test-Path 'F:\CI\act_runner\.runner'
+```
+
+**Fix**:
+```powershell
+# Restart the service
+Restart-Service act_runner
+
+# Verify it came back online (wait ~10s then check Gitea UI)
+Get-Service act_runner | Select-Object Status, StartType
+
+# If service won't start, check NSSM log
+& 'C:\nssm\nssm.exe' status act_runner
+```
+
+If the `.runner` registration file is missing or corrupt, re-register:
+```powershell
+cd F:\CI\act_runner
+.\act_runner.exe register --no-interactive `
+    --instance http://10.10.20.11:3100 `
+    --token <token-from-gitea-admin> `
+    --name local-windows-runner `
+    --labels "windows-build:host,dotnet:host,msbuild:host"
+```
+
+**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`.
+
+---
+
+## 2. All builds fail in Phase 2 (VM clone / start)
+
+**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like:
+- `vmrun clone failed`
+- `vmrun start failed`
+- `Template VMX not found`
+- `Could not detect VM IP address`
+
+**Triage**:
+```powershell
+# List all running VMs
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list
+
+# Check template VMX exists and is accessible
+Test-Path 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx'
+
+# Check for orphaned clones that may be consuming disk
+Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime
+
+# Check disk free space
+Get-PSDrive F | Select-Object Name, Free, Used
+
+# Check for a stuck vm-start lock from a crashed job
+Test-Path 'F:\CI\State\vm-start.lock'
+```
+
+**Fix** — by root cause:
+
+*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`.
+
+*Parent VMDK locked* (VMware left a lock file after host crash):
+```powershell
+# Stop all VMs
+& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx' hard
+# Delete lock files
+Remove-Item 'F:\CI\Templates\WinBuild\*.lck' -Recurse -Force -ErrorAction SilentlyContinue
+```
+
+*Snapshot missing* (`BaseClean` was deleted or renamed):
+```powershell
+# List snapshots on template VM
+& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx'
+# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name
+```
+
+*Disk full* (clone delta files need space):
+```powershell
+# Emergency cleanup — remove all orphaned clones
+& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
+# Then run retention
+& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3
+```
+
+*Stale vm-start lock* (from a job that crashed without cleanup):
+```powershell
+Remove-Item 'F:\CI\State\vm-start.lock' -Force
+Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force
+```
+
+**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.
+
+---
+
+## 3. Builds are slow
+
+**Symptom**: jobs that previously completed in ~3 min now take 8+ min.
+Phase durations visible in `F:\CI\Logs\<jobId>\invoke-ci.jsonl`.
+
+**Triage**:
+```powershell
+# Check disk free space (below 50 GB = fragmented writes)
+Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}
+
+# Check active VM CPU usage (Task Manager or:)
+Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending
+
+# Check VMnet8 NAT adapter status
+Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }
+
+# Parse JSONL for per-phase durations (requires jq or manual inspection)
+# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
+Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status
+```
+
+**Fix** — by root cause:
+
+*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK.
+
+*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2.
+
+*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running.
+
+*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`.
+
+**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).
+
+---
+
+## 4. Template VMX corrupt after host crash
+
+**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state.
+
+**Triage**:
+```powershell
+# Try starting the template directly in VMware Workstation UI
+# If it reports "configuration file error" or "disk lock", proceed below.
+
+# Check for lock files
+Get-ChildItem 'F:\CI\Templates\WinBuild\' -Recurse -Filter '*.lck'
+
+# Check if backup exists
+Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5
+```
+
+**Fix**:
+
+*Lock files only* (common after hard shutdown):
+```powershell
+# Ensure no VMware processes are running
+Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
+# Remove locks
+Remove-Item 'F:\CI\Templates\WinBuild\*.lck' -Recurse -Force
+# Test clone
+& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx'
+```
+
+*VMX or VMDK truly corrupt — restore from backup*:
+```powershell
+# Stop all CI activity first
+Stop-Service act_runner
+
+# Identify latest backup
+$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
+Write-Host "Restoring from: $($latest.FullName)"
+
+# Replace template directory
+Remove-Item 'F:\CI\Templates\WinBuild\' -Recurse -Force
+Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild\' -Recurse
+
+# Restart runner
+Start-Service act_runner
+```
+
+*No backup exists*: must re-provision the template from scratch.
+Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare).
+Estimated time: 2-4 hours including Windows Update.
+
+**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.
+
+---
+
+## Quick Reference
+
+| Symptom                        | First command                                                          |
+| ------------------------------ | ---------------------------------------------------------------------- |
+| Runner offline                 | `Get-Service act_runner`, then `Restart-Service act_runner`            |
+| Phase 2 clone fails            | `Test-Path F:\CI\Templates\WinBuild\CI-WinBuild.vmx`                  |
+| Disk full                      | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1`       |
+| Stale lock                     | `Remove-Item F:\CI\State\vm-start.lock`                               |
+| Slow builds                    | Check `invoke-ci.jsonl` phase timestamps; check disk I/O              |
+| Template corrupt               | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\`    |
+| Snapshot missing               | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME`           |
+| IP collision                   | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity`          |