local-ci-cd-system/docs/RUNBOOK.md

CI System Runbook

Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.

---

1. Runner offline in Gitea UI

Symptom: http://10.10.20.11:3100/admin/runners shows local-windows-runner as offline. Queued jobs stay pending indefinitely.

Triage:

# Check service state
Get-Service act_runner

# Last 50 lines of runner log
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50

# Check registration file is intact
Test-Path 'F:\CI\act_runner\.runner'

Fix:

# Restart the service
Restart-Service act_runner

# Verify it came back online (wait ~10s then check Gitea UI)
Get-Service act_runner | Select-Object Status, StartType

# If service won't start, check NSSM log
& 'C:\nssm\nssm.exe' status act_runner

If the .runner registration file is missing or corrupt, re-register:

cd F:\CI\act_runner
.\act_runner.exe register --no-interactive `
    --instance http://10.10.20.11:3100 `
    --token <token-from-gitea-admin> `
    --name local-windows-runner `
    --labels "windows-build:host,dotnet:host,msbuild:host"

Escalation: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for act_runner errors and inspect F:\CI\act_runner\logs\.

---

2. All builds fail in Phase 2 (VM clone / start)

Symptom: Invoke-CIJob.ps1 fails at Phase 2 with errors like:

• vmrun clone failed
• vmrun start failed
• Template VMX not found
• Could not detect VM IP address

Triage:

# List all running VMs
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list

# Check template VMX exists and is accessible
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

# Check for orphaned clones that may be consuming disk
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime

# Check disk free space
Get-PSDrive F | Select-Object Name, Free, Used

# Check for a stuck vm-start lock from a crashed job
Test-Path 'F:\CI\State\vm-start.lock'

Fix — by root cause:

Template VMX missing/moved: check GITEA_CI_TEMPLATE_PATH in F:\CI\act_runner\config.yaml.

Parent VMDK locked (VMware left a lock file after host crash):

# Stop all VMs
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
# Delete lock files
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue

Snapshot missing (BaseClean was deleted or renamed):

# List snapshots on template VM
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name

Disk full (clone delta files need space):

# Emergency cleanup — remove all orphaned clones
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
# Then run retention
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3

Stale vm-start lock (from a job that crashed without cleanup):

Remove-Item 'F:\CI\State\vm-start.lock' -Force
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force

Escalation: If vmrun clone fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.

---

3. Builds are slow

Symptom: jobs that previously completed in ~3 min now take 8+ min. Phase durations visible in F:\CI\Logs\<jobId>\invoke-ci.jsonl.

Triage:

# Check disk free space (below 50 GB = fragmented writes)
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}

# Check active VM CPU usage (Task Manager or:)
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending

# Check VMnet8 NAT adapter status
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }

# Parse JSONL for per-phase durations (requires jq or manual inspection)
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status

Fix — by root cause:

Low disk space → fragmented VMDKs: run retention policy, then consider vmware-vdiskmanager -d to defragment the template VMDK.

High vmware-vmx CPU with many VMs: reduce capacity in config.yaml from 4 to 2.

VMnet8 NAT bottleneck (slow pip/nuget downloads inside VM): check Services.msc → VMware NAT Service is running.

NVMe saturation: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce capacity: 2.

Escalation: Use invoke-ci.jsonl to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).

---

4. Template VMX corrupt after host crash

Symptom: After an unclean host shutdown, vmrun clone or vmrun start on the template fails. VMware Workstation shows the template in an error state.

Triage:

# Try starting the template directly in VMware Workstation UI
# If it reports "configuration file error" or "disk lock", proceed below.

# Check for lock files
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'

# Check if backup exists
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5

Fix:

Lock files only (common after hard shutdown):

# Ensure no VMware processes are running
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
# Remove locks
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
# Test clone
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

VMX or VMDK truly corrupt — restore from backup:

# Stop all CI activity first
Stop-Service act_runner

# Identify latest backup
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Write-Host "Restoring from: $($latest.FullName)"

# Replace template directory
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse

# Restart runner
Start-Service act_runner

No backup exists: must re-provision the template from scratch. Follow docs/WINDOWS-TEMPLATE-SETUP.md → Fase A (Deploy) → Fase B (Prepare). Estimated time: 2-4 hours including Windows Update.

Escalation: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.

---

Quick Reference

Symptom	First command
Runner offline	Get-Service act_runner, then Restart-Service act_runner
Phase 2 clone fails	Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx
Disk full	Get-PSDrive F | Select Free; run Invoke-RetentionPolicy.ps1
Stale lock	Remove-Item F:\CI\State\vm-start.lock
Slow builds	Check invoke-ci.jsonl phase timestamps; check disk I/O
Template corrupt	Remove *.lck files; if persistent, restore from F:\CI\Backups\
Snapshot missing	vmrun listSnapshots <vmx>; update GITEA_CI_SNAPSHOT_NAME
IP collision	Remove-Item F:\CI\State\ip-leases\*.lease; lower capacity

---

5. Template Refresh Procedure

Use this procedure when the template OS needs updated packages, toolchain upgrades, or a new snapshot. Run on the host with an elevated PowerShell 5.1 session.

5.1 Pre-flight

# Stop the runner so no CI jobs start during the refresh
Stop-Service act_runner

# Verify no clone VMs are running
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
# Expected: "Total running VMs: 0"

# Backup the existing template (keeps last 3 by default)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates

5.2 Boot the template

Windows (WinBuild2025):

$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

Linux (LinuxBuild2404):

$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

5.3 Apply updates inside the template

Windows — connect via WinRM or open the VMware console, then run the Prepare script from the host:

$vmxWin   = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
$credTgt  = 'BuildVMGuest'    # Windows Credential Manager target
$cred     = Get-StoredCredential -Target $credTgt   # requires CredentialManager module
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
    -VMXPath    $vmxWin `
    -Credential $cred

Linux — SSH into the template and run the toolchain script:

$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
# Get IP (wait for VMware Tools if needed)
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
# Apply updates
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
    -VMXPath  $vmxLin `
    -SshKeyPath 'F:\CI\keys\ci_linux'

Alternatively, run Install-CIToolchain-WinBuild2025.ps1 / Install-CIToolchain-Linux2404.sh manually inside the guest to apply only toolchain changes without the full Prepare script.

5.4 Shut down and snapshot

# Shut down gracefully (wait up to 120 s)
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'   # or Linux vmx
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft

# Name: BaseClean_yyyyMMdd  (keeps old name for rollback reference)
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
Write-Host "Snapshot created: $snapshotName"

Confirm no .vmem / .vmsn files exist before snapshotting (see AGENTS.md item 9):

Get-ChildItem (Split-Path $vmx) -Filter '*.vmem'  # must be empty

5.5 Validate

# Run the validation script
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
    -VMXPath $vmx -SnapshotName $snapshotName

For Linux, also run a quick SSH smoke-test from the host:

Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
    -Command 'gcc --version && cmake --version' -PassThru
$result.Output

5.6 Run a smoke workflow

Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI. Confirm the job uses the new snapshot and completes successfully.

5.7 Promote the new snapshot

Update GITEA_CI_SNAPSHOT_NAME in runner/config.yaml and redeploy:

# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'

# Deploy config and restart runner
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner

5.8 Retain old snapshot 7 days, then delete

Keep the previous BaseClean_* snapshot for 7 days as a rollback point:

# List existing snapshots
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx

# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME

5.9 Rollback procedure

If a smoke-test failure is discovered after promotion:

# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
# (or set it back to 'BaseClean' for the permanent base)
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner
# The prior snapshot is still in the template — jobs will use it immediately.

---

6. Windows host pre-migration baseline (reference for B7)

Recorded 2026-05-17 — Measure-CIBenchmark.ps1 × 4 iterations, Python orchestrator post-Phase-A, Windows 11 + VMware Workstation Pro, template WinBuild2025 / snapshot BaseClean.

Iter	Clone (s)	Start (s)	IP acquire (s)	WinRM (s)	Destroy (s)	Boot total (s)
1	0.63	1.75	66.57	0.01	4.81	68.96
2	0.63	1.89	20.21	0.01	6.39	22.74
3	0.62	1.72	85.07	0.01	4.50	87.42
4	0.61	1.72	60.97	0.01	4.20	63.31
avg	0.62	1.77	58.20	0.01	4.98	60.61

Key finding: IP-acquire phase dominates total time and is highly variable (20–85 s) due to VMware Tools guest IP detection latency. Clone/Start/WinRM are negligible and stable.

B7 comparison guidance (tolerance ±20%):

Metric	Windows baseline	±20% range
Clone	0.62 s	0.50–0.74 s
Start	1.77 s	1.42–2.12 s
Destroy	4.98 s	3.98–5.98 s
Boot total (avg)	60.6 s	48.5–72.7 s

IP-acquire variance on Windows (σ ≈ 26 s) means boot-total comparison requires ≥10 samples on Linux to be meaningful. If Linux avg boot total exceeds 72.7 s, open an issue in TODO.md with per-phase breakdown before declaring B7 failed — check whether IP-acquire increased or non-IP phases regressed.

---

7. Linux host post-migration baseline (B7 result)

Recorded 2026-05-24 — Measure-CIBenchmark.ps1 × 4 iterations, Linux Mint host + VMware Workstation Pro Linux, template WinBuild2025 / snapshot BaseClean. Ready column = WinRM/5986 TCP probe.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.42	1.89	53.06	0.03	4.55	55.40
2	0.40	1.89	129.76	0.00	4.96	132.05
3	0.52	2.81	176.83	0.00	5.67	180.16
4	0.40	1.90	39.17	0.00	4.51	41.47
avg	0.44	2.12	99.71	0.01	4.92	102.27

Phase verdict vs Windows baseline (±20%):

Metric	Windows	Linux avg	In range?
Clone	0.62 s	0.44 s	✓ (faster)
Start	1.77 s	2.12 s	✓ (at upper edge)
Destroy	4.98 s	4.92 s	✓
IP avg	58.2 s	99.7 s	✗ outside — IP variance (39–177 s)
Ready	0.01 s	0.01 s	✓

Key finding: Clone/Start/Ready/Destroy within ±20%. IP-acquire dominates and is highly variable on Linux host (σ ≈ 57 s, range 39–177 s) — wider than Windows (σ ≈ 26 s). This is VMware Tools DHCP/guestinfo reporting latency, not a regression in orchestrator logic. With 4 samples the avg is not stable; additional runs may close the gap. No non-IP phase regressed.

---

8. Static IP baseline — WinBuild2025 with ip_pool (B8 result)

Recorded 2026-05-25 — Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 4, Linux Mint host, template WinBuild2025 / snapshot BaseClean. guestinfo.ip-assignment injected into cloned VMX before start; ci-static-ip.ps1 scheduled task applies IP at boot and writes back guestinfo.ci-ip. IP column = time until guestinfo.ci-ip readable via vmrun readVariable (-GuestInfoOnly mode — DHCP fallback disabled). Ready column = WinRM/5986 TCP probe after IP known.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.41	1.86	21.80	0.00	9.75	24.07
2	0.40	1.88	21.74	0.00	9.81	24.02
3	0.40	1.90	21.75	0.00	12.26	24.05
4	0.40	1.89	21.77	0.00	9.77	24.06
avg	0.40	1.88	21.77	0.00	10.40	24.05

Three-way comparison — B6 Windows DHCP / B7 Linux DHCP / B8 Linux static IP:

Metric	B6 Win DHCP avg	B7 Lin DHCP avg	B8 Lin static avg	B8 vs B6	B8 vs B7
Clone	0.62 s	0.44 s	0.40 s	−35%	≈ same
Start	1.77 s	2.12 s	1.88 s	+6%	≈ same
IP acquire	58.2 s	99.7 s	21.8 s	−63%	−78%
Ready	0.01 s	0.01 s	0.00 s	≈ same	≈ same
Boot total	60.6 s	102.3 s	24.1 s	−60%	−76%
IP σ	~26 s	~57 s	<0.03 s	deterministic	deterministic

Key findings:

• Static IP beats even the Windows DHCP baseline by 60% on boot total.
• IP acquire drops from 58 s (Win) / 100 s (Linux) DHCP to a deterministic 21.8 s — variance eliminated entirely (σ < 0.03 s).
• Ready = 0 s: WinRM is already listening on the static IP by the time guestinfo.ci-ip is written — no additional TCP probe wait.
• ci-static-ip.ps1 startup latency (~21.8 s) is the new floor; it reflects Windows boot + Task Scheduler + NIC reconfiguration time.
• Clone/Start/Ready unchanged across all three baselines — static IP has no side effects on non-IP phases.
• Destroy is slower than B6/B7 (~10 s vs ~5 s) — likely disk pressure or clone state at test time, unrelated to static IP.

---

9. Concurrent capacity burn-in (B7 — 4 × 10)

Recorded 2026-06-07 — Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10, Linux Mint host + VMware Workstation Pro Linux, run as ci-runner, static IP pool (192.168.79.201–204). Repo Simone/burnin-dummy. Each "round" is the wall-clock time for 4 concurrent end-to-end jobs (clone → boot → IP → transport → build → artifacts → destroy). This is the first concurrent capacity burn-in (the §6–§8 baselines are single-job Measure-CIBenchmark).

Template	Snapshot	Build cmd	Jobs PASS	Round time (min–max)	Round avg
WinBuild2025	BaseClean	build.ps1	40 / 40	68–81 s	~78.6 s
LinuxBuild2404	BaseClean-Linux	build.sh	40 / 40	70–71 s	~70.2 s

Result: OVERALL PASS — 80/80 jobs, 20/20 rounds, zero orphaned clones, zero leaked IP leases between rounds.

Key findings:

• Per-round wall time is near-deterministic (Win σ ≈ 1.3 s, Linux σ ≈ 0.4 s), confirming the static-IP pool eliminates the DHCP IP-acquire variance that dominated the §6/§7 single-job baselines (σ 26–57 s).
• 4-way concurrency adds no contention penalty: round time ≈ single-job boot total (§8 static ≈ 24 s) plus build + serialized destroy, well within the 4-IP pool capacity.
• Linux rounds (~70 s) slightly faster and tighter than Windows (~79 s).
• IP-pool release-on-completion works under concurrency — pool returned to all-free (null) after every round.

Operational note (pool not auto-reconciled): ip-pool.json is not reconciled against live clones at startup. A job killed mid-flight (SIGKILL, crash) leaks its lease permanently; with only 4 IPs for parallelism 4, a single leak exhausts the pool and every subsequent round fails fast with IP pool exhausted after 60s. Recovery: stop all jobs, confirm build-vms/ empty, then reset the pool:

sudo -u ci-runner /opt/ci/venv/bin/python -c \
  "import pathlib; pathlib.Path('/var/lib/ci/ip-pool.json').write_text('{}\n')"

See TODO.md (IP-pool auto-reconciliation) for the proposed fix.