local-ci-cd-system/docs/RUNBOOK.md

CI System Runbook

Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.

---

1. Runner offline in Gitea UI

Symptom: http://10.10.20.11:3100/admin/runners shows local-windows-runner as offline. Queued jobs stay pending indefinitely.

Triage:

# Check service state
Get-Service act_runner

# Last 50 lines of runner log
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50

# Check registration file is intact
Test-Path 'F:\CI\act_runner\.runner'

Fix:

# Restart the service
Restart-Service act_runner

# Verify it came back online (wait ~10s then check Gitea UI)
Get-Service act_runner | Select-Object Status, StartType

# If service won't start, check NSSM log
& 'C:\nssm\nssm.exe' status act_runner

If the .runner registration file is missing or corrupt, re-register:

cd F:\CI\act_runner
.\act_runner.exe register --no-interactive `
    --instance http://10.10.20.11:3100 `
    --token <token-from-gitea-admin> `
    --name local-windows-runner `
    --labels "windows-build:host,dotnet:host,msbuild:host"

Escalation: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for act_runner errors and inspect F:\CI\act_runner\logs\.

---

2. All builds fail in Phase 2 (VM clone / start)

Symptom: Invoke-CIJob.ps1 fails at Phase 2 with errors like:

• vmrun clone failed
• vmrun start failed
• Template VMX not found
• Could not detect VM IP address

Triage:

# List all running VMs
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list

# Check template VMX exists and is accessible
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

# Check for orphaned clones that may be consuming disk
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime

# Check disk free space
Get-PSDrive F | Select-Object Name, Free, Used

# Check for a stuck vm-start lock from a crashed job
Test-Path 'F:\CI\State\vm-start.lock'

Fix — by root cause:

Template VMX missing/moved: check GITEA_CI_TEMPLATE_PATH in F:\CI\act_runner\config.yaml.

Parent VMDK locked (VMware left a lock file after host crash):

# Stop all VMs
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
# Delete lock files
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue

Snapshot missing (BaseClean was deleted or renamed):

# List snapshots on template VM
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name

Disk full (clone delta files need space):

# Emergency cleanup — remove all orphaned clones
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
# Then run retention
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3

Stale vm-start lock (from a job that crashed without cleanup):

Remove-Item 'F:\CI\State\vm-start.lock' -Force
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force

Escalation: If vmrun clone fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.

---

3. Builds are slow

Symptom: jobs that previously completed in ~3 min now take 8+ min. Phase durations visible in F:\CI\Logs\<jobId>\invoke-ci.jsonl.

Triage:

# Check disk free space (below 50 GB = fragmented writes)
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}

# Check active VM CPU usage (Task Manager or:)
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending

# Check VMnet8 NAT adapter status
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }

# Parse JSONL for per-phase durations (requires jq or manual inspection)
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status

Fix — by root cause:

Low disk space → fragmented VMDKs: run retention policy, then consider vmware-vdiskmanager -d to defragment the template VMDK.

High vmware-vmx CPU with many VMs: reduce capacity in config.yaml from 4 to 2.

VMnet8 NAT bottleneck (slow pip/nuget downloads inside VM): check Services.msc → VMware NAT Service is running.

NVMe saturation: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce capacity: 2.

Escalation: Use invoke-ci.jsonl to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).

---

4. Template VMX corrupt after host crash

Symptom: After an unclean host shutdown, vmrun clone or vmrun start on the template fails. VMware Workstation shows the template in an error state.

Triage:

# Try starting the template directly in VMware Workstation UI
# If it reports "configuration file error" or "disk lock", proceed below.

# Check for lock files
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'

# Check if backup exists
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5

Fix:

Lock files only (common after hard shutdown):

# Ensure no VMware processes are running
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
# Remove locks
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
# Test clone
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

VMX or VMDK truly corrupt — restore from backup:

# Stop all CI activity first
Stop-Service act_runner

# Identify latest backup
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Write-Host "Restoring from: $($latest.FullName)"

# Replace template directory
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse

# Restart runner
Start-Service act_runner

No backup exists: must re-provision the template from scratch. Follow docs/WINDOWS-TEMPLATE-SETUP.md → Fase A (Deploy) → Fase B (Prepare). Estimated time: 2-4 hours including Windows Update.

Escalation: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.

---

Quick Reference

Symptom	First command
Runner offline	Get-Service act_runner, then Restart-Service act_runner
Phase 2 clone fails	Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx
Disk full	Get-PSDrive F | Select Free; run Invoke-RetentionPolicy.ps1
Stale lock	Remove-Item F:\CI\State\vm-start.lock
Slow builds	Check invoke-ci.jsonl phase timestamps; check disk I/O
Template corrupt	Remove *.lck files; if persistent, restore from F:\CI\Backups\
Snapshot missing	vmrun listSnapshots <vmx>; update GITEA_CI_SNAPSHOT_NAME
IP collision	Remove-Item F:\CI\State\ip-leases\*.lease; lower capacity

---

5. Template Refresh Procedure

Use this procedure when the template OS needs updated packages, toolchain upgrades, or a new snapshot. Run on the host with an elevated PowerShell 5.1 session.

5.1 Pre-flight

# Stop the runner so no CI jobs start during the refresh
Stop-Service act_runner

# Verify no clone VMs are running
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
# Expected: "Total running VMs: 0"

# Backup the existing template (keeps last 3 by default)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates

5.2 Boot the template

Windows (WinBuild2025):

$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

Linux (LinuxBuild2404):

$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

5.3 Apply updates inside the template

Windows — connect via WinRM or open the VMware console, then run the Prepare script from the host:

$vmxWin   = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
$credTgt  = 'BuildVMGuest'    # Windows Credential Manager target
$cred     = Get-StoredCredential -Target $credTgt   # requires CredentialManager module
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
    -VMXPath    $vmxWin `
    -Credential $cred

Linux — SSH into the template and run the toolchain script:

$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
# Get IP (wait for VMware Tools if needed)
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
# Apply updates
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
    -VMXPath  $vmxLin `
    -SshKeyPath 'F:\CI\keys\ci_linux'

Alternatively, run Install-CIToolchain-WinBuild2025.ps1 / Install-CIToolchain-Linux2404.sh manually inside the guest to apply only toolchain changes without the full Prepare script.

5.4 Shut down and snapshot

# Shut down gracefully (wait up to 120 s)
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'   # or Linux vmx
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft

# Name: BaseClean_yyyyMMdd  (keeps old name for rollback reference)
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
Write-Host "Snapshot created: $snapshotName"

Confirm no .vmem / .vmsn files exist before snapshotting (see AGENTS.md item 9):

Get-ChildItem (Split-Path $vmx) -Filter '*.vmem'  # must be empty

5.5 Validate

# Run the validation script
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
    -VMXPath $vmx -SnapshotName $snapshotName

For Linux, also run a quick SSH smoke-test from the host:

Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
    -Command 'gcc --version && cmake --version' -PassThru
$result.Output

5.6 Run a smoke workflow

Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI. Confirm the job uses the new snapshot and completes successfully.

5.7 Promote the new snapshot

Update GITEA_CI_SNAPSHOT_NAME in runner/config.yaml and redeploy:

# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'

# Deploy config and restart runner
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner

5.8 Retain old snapshot 7 days, then delete

Keep the previous BaseClean_* snapshot for 7 days as a rollback point:

# List existing snapshots
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx

# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME

5.9 Rollback procedure

If a smoke-test failure is discovered after promotion:

# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
# (or set it back to 'BaseClean' for the permanent base)
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner
# The prior snapshot is still in the template — jobs will use it immediately.

---

6. Windows host pre-migration baseline (reference for B7)

Recorded 2026-05-17 — Measure-CIBenchmark.ps1 × 4 iterations, Python orchestrator post-Phase-A, Windows 11 + VMware Workstation Pro, template WinBuild2025 / snapshot BaseClean.

Iter	Clone (s)	Start (s)	IP acquire (s)	WinRM (s)	Destroy (s)	Boot total (s)
1	0.63	1.75	66.57	0.01	4.81	68.96
2	0.63	1.89	20.21	0.01	6.39	22.74
3	0.62	1.72	85.07	0.01	4.50	87.42
4	0.61	1.72	60.97	0.01	4.20	63.31
avg	0.62	1.77	58.20	0.01	4.98	60.61

Key finding: IP-acquire phase dominates total time and is highly variable (20–85 s) due to VMware Tools guest IP detection latency. Clone/Start/WinRM are negligible and stable.

B7 comparison guidance (tolerance ±20%):

Metric	Windows baseline	±20% range
Clone	0.62 s	0.50–0.74 s
Start	1.77 s	1.42–2.12 s
Destroy	4.98 s	3.98–5.98 s
Boot total (avg)	60.6 s	48.5–72.7 s

IP-acquire variance on Windows (σ ≈ 26 s) means boot-total comparison requires ≥10 samples on Linux to be meaningful. If Linux avg boot total exceeds 72.7 s, open an issue in TODO.md with per-phase breakdown before declaring B7 failed — check whether IP-acquire increased or non-IP phases regressed.

---

7. Linux host post-migration baseline (B7 result)

Recorded 2026-05-24 — Measure-CIBenchmark.ps1 × 4 iterations, Linux Mint host + VMware Workstation Pro Linux, template WinBuild2025 / snapshot BaseClean. Ready column = WinRM/5986 TCP probe.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.42	1.89	53.06	0.03	4.55	55.40
2	0.40	1.89	129.76	0.00	4.96	132.05
3	0.52	2.81	176.83	0.00	5.67	180.16
4	0.40	1.90	39.17	0.00	4.51	41.47
avg	0.44	2.12	99.71	0.01	4.92	102.27

Phase verdict vs Windows baseline (±20%):

Metric	Windows	Linux avg	In range?
Clone	0.62 s	0.44 s	✓ (faster)
Start	1.77 s	2.12 s	✓ (at upper edge)
Destroy	4.98 s	4.92 s	✓
IP avg	58.2 s	99.7 s	✗ outside — IP variance (39–177 s)
Ready	0.01 s	0.01 s	✓

Key finding: Clone/Start/Ready/Destroy within ±20%. IP-acquire dominates and is highly variable on Linux host (σ ≈ 57 s, range 39–177 s) — wider than Windows (σ ≈ 26 s). This is VMware Tools DHCP/guestinfo reporting latency, not a regression in orchestrator logic. With 4 samples the avg is not stable; additional runs may close the gap. No non-IP phase regressed.

---

8. Static IP baseline — WinBuild2025 with ip_pool (B8 result)

Recorded 2026-05-25 — Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 4, Linux Mint host, template WinBuild2025 / snapshot BaseClean. guestinfo.ip-assignment injected into cloned VMX before start; ci-static-ip.ps1 scheduled task applies IP at boot and writes back guestinfo.ci-ip. IP column = time until guestinfo.ci-ip readable via vmrun readVariable (-GuestInfoOnly mode — DHCP fallback disabled). Ready column = WinRM/5986 TCP probe after IP known.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.41	1.86	21.80	0.00	9.75	24.07
2	0.40	1.88	21.74	0.00	9.81	24.02
3	0.40	1.90	21.75	0.00	12.26	24.05
4	0.40	1.89	21.77	0.00	9.77	24.06
avg	0.40	1.88	21.77	0.00	10.40	24.05

Three-way comparison — B6 Windows DHCP / B7 Linux DHCP / B8 Linux static IP:

Metric	B6 Win DHCP avg	B7 Lin DHCP avg	B8 Lin static avg	B8 vs B6	B8 vs B7
Clone	0.62 s	0.44 s	0.40 s	−35%	≈ same
Start	1.77 s	2.12 s	1.88 s	+6%	≈ same
IP acquire	58.2 s	99.7 s	21.8 s	−63%	−78%
Ready	0.01 s	0.01 s	0.00 s	≈ same	≈ same
Boot total	60.6 s	102.3 s	24.1 s	−60%	−76%
IP σ	~26 s	~57 s	<0.03 s	deterministic	deterministic

Key findings:

• Static IP beats even the Windows DHCP baseline by 60% on boot total.
• IP acquire drops from 58 s (Win) / 100 s (Linux) DHCP to a deterministic 21.8 s — variance eliminated entirely (σ < 0.03 s).
• Ready = 0 s: WinRM is already listening on the static IP by the time guestinfo.ci-ip is written — no additional TCP probe wait.
• ci-static-ip.ps1 startup latency (~21.8 s) is the new floor; it reflects Windows boot + Task Scheduler + NIC reconfiguration time.
• Clone/Start/Ready unchanged across all three baselines — static IP has no side effects on non-IP phases.
• Destroy is slower than B6/B7 (~10 s vs ~5 s) — likely disk pressure or clone state at test time, unrelated to static IP.

---

9. Concurrent capacity burn-in (B7 — 4 × 10)

Recorded 2026-06-07 — Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10, Linux Mint host + VMware Workstation Pro Linux, run as ci-runner, static IP pool (192.168.79.201–204). Repo Simone/burnin-dummy. Each "round" is the wall-clock time for 4 concurrent end-to-end jobs (clone → boot → IP → transport → build → artifacts → destroy). This is the first concurrent capacity burn-in (the §6–§8 baselines are single-job Measure-CIBenchmark).

Template	Snapshot	Build cmd	Jobs PASS	Round time (min–max)	Round avg
WinBuild2025	BaseClean	build.ps1	40 / 40	68–81 s	~78.6 s
LinuxBuild2404	BaseClean-Linux	build.sh	40 / 40	70–71 s	~70.2 s

Result: OVERALL PASS — 80/80 jobs, 20/20 rounds, zero orphaned clones, zero leaked IP leases between rounds.

Key findings:

• Per-round wall time is near-deterministic (Win σ ≈ 1.3 s, Linux σ ≈ 0.4 s), confirming the static-IP pool eliminates the DHCP IP-acquire variance that dominated the §6/§7 single-job baselines (σ 26–57 s).
• 4-way concurrency adds no contention penalty: round time ≈ single-job boot total (§8 static ≈ 24 s) plus build + serialized destroy, well within the 4-IP pool capacity.
• Linux rounds (~70 s) slightly faster and tighter than Windows (~79 s).
• IP-pool release-on-completion works under concurrency — pool returned to all-free (null) after every round.

Operational note (pool not auto-reconciled): ip-pool.json is not reconciled against live clones at startup. A job killed mid-flight (SIGKILL, crash) leaks its lease permanently; with only 4 IPs for parallelism 4, a single leak exhausts the pool and every subsequent round fails fast with IP pool exhausted after 60s. Recovery: stop all jobs, confirm build-vms/ empty, then reset the pool:

sudo -u ci-runner /opt/ci/venv/bin/python -c \
  "import pathlib; pathlib.Path('/var/lib/ci/ip-pool.json').write_text('{}\n')"

See TODO.md (IP-pool auto-reconciliation) for the proposed fix.

---

10. Linux guest single-job baseline — LinuxBuild2404 (B7 follow-up)

Recorded 2026-06-07 — Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10, Linux Mint host + VMware Workstation Pro Linux, run as ci-runner, template LinuxBuild2404 / snapshot BaseClean-Linux, DHCP (no static IP — the guest reads its DHCP lease and publishes it via guestinfo.ci-ip; the Windows-only ci-static-ip.ps1 task does not apply). Ready = SSH/22 probe.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1–10	0.22	1.14–1.16	7.44–7.48	0.00–0.01	4.90–4.99	8.80–8.86
avg	0.22	1.14	7.45	0.00	4.94	8.82

Key finding: the Linux guest is near-deterministic even on DHCP (σ ≈ 0.01 s on IP-acquire) — no static-IP injection needed. IP-acquire is 7.45 s vs Windows-guest 58 s (DHCP, §6) / 21.8 s (static, §8). Boot total 8.82 s — ~2.7× faster than the Windows static-IP guest.

Guest-vs-guest comparison (Linux host, single-job):

Metric	Win guest static (§8)	Lin guest DHCP (§10)	Lin advantage
Clone	0.40 s	0.22 s	−45%
Start	1.88 s	1.14 s	−39%
IP acquire	21.77 s	7.45 s	−66%
Ready	0.00 s	0.00 s	≈ same
Destroy	10.40 s	4.94 s	−53%
Boot total	24.05 s	8.82 s	−63%

Caveat: different IP modes (Win static-IP task vs Lin DHCP+guestinfo) — the comparison reflects the as-deployed path for each guest, not an IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such in-guest agent.