local-ci-cd-system/docs/RUNBOOK.md

# CI System Runbook

Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.

---

## 1. Runner offline in Gitea UI

**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline.
Queued jobs stay pending indefinitely.

**Triage**:
```powershell
# Check service state
Get-Service act_runner

# Last 50 lines of runner log
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50

# Check registration file is intact
Test-Path 'F:\CI\act_runner\.runner'
```

**Fix**:
```powershell
# Restart the service
Restart-Service act_runner

# Verify it came back online (wait ~10s then check Gitea UI)
Get-Service act_runner | Select-Object Status, StartType

# If service won't start, check NSSM log
& 'C:\nssm\nssm.exe' status act_runner
```

If the `.runner` registration file is missing or corrupt, re-register:
```powershell
cd F:\CI\act_runner
.\act_runner.exe register --no-interactive `
    --instance http://10.10.20.11:3100 `
    --token <token-from-gitea-admin> `
    --name local-windows-runner `
    --labels "windows-build:host,dotnet:host,msbuild:host"
```

**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`.

---

## 2. All builds fail in Phase 2 (VM clone / start)

**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like:
- `vmrun clone failed`
- `vmrun start failed`
- `Template VMX not found`
- `Could not detect VM IP address`

**Triage**:
```powershell
# List all running VMs
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list

# Check template VMX exists and is accessible
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

# Check for orphaned clones that may be consuming disk
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime

# Check disk free space
Get-PSDrive F | Select-Object Name, Free, Used

# Check for a stuck vm-start lock from a crashed job
Test-Path 'F:\CI\State\vm-start.lock'
```

**Fix** — by root cause:

*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`.

*Parent VMDK locked* (VMware left a lock file after host crash):
```powershell
# Stop all VMs
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
# Delete lock files
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue
```

*Snapshot missing* (`BaseClean` was deleted or renamed):
```powershell
# List snapshots on template VM
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name
```

*Disk full* (clone delta files need space):
```powershell
# Emergency cleanup — remove all orphaned clones
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
# Then run retention
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3
```

*Stale vm-start lock* (from a job that crashed without cleanup):
```powershell
Remove-Item 'F:\CI\State\vm-start.lock' -Force
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force
```

**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.

---

## 3. Builds are slow

**Symptom**: jobs that previously completed in ~3 min now take 8+ min.
Phase durations visible in `F:\CI\Logs\<jobId>\invoke-ci.jsonl`.

**Triage**:
```powershell
# Check disk free space (below 50 GB = fragmented writes)
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}

# Check active VM CPU usage (Task Manager or:)
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending

# Check VMnet8 NAT adapter status
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }

# Parse JSONL for per-phase durations (requires jq or manual inspection)
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status
```

**Fix** — by root cause:

*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK.

*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2.

*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running.

*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`.

**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).

---

## 4. Template VMX corrupt after host crash

**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state.

**Triage**:
```powershell
# Try starting the template directly in VMware Workstation UI
# If it reports "configuration file error" or "disk lock", proceed below.

# Check for lock files
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'

# Check if backup exists
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5
```

**Fix**:

*Lock files only* (common after hard shutdown):
```powershell
# Ensure no VMware processes are running
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
# Remove locks
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
# Test clone
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
```

*VMX or VMDK truly corrupt — restore from backup*:
```powershell
# Stop all CI activity first
Stop-Service act_runner

# Identify latest backup
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Write-Host "Restoring from: $($latest.FullName)"

# Replace template directory
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse

# Restart runner
Start-Service act_runner
```

*No backup exists*: must re-provision the template from scratch.
Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare).
Estimated time: 2-4 hours including Windows Update.

**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.

---

## Quick Reference

| Symptom                        | First command                                                          |
| ------------------------------ | ---------------------------------------------------------------------- |
| Runner offline                 | `Get-Service act_runner`, then `Restart-Service act_runner`            |
| Phase 2 clone fails            | `Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx`                  |
| Disk full                      | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1`       |
| Stale lock                     | `Remove-Item F:\CI\State\vm-start.lock`                               |
| Slow builds                    | Check `invoke-ci.jsonl` phase timestamps; check disk I/O              |
| Template corrupt               | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\`    |
| Snapshot missing               | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME`           |
| IP collision                   | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity`          |