local-ci-cd-system/docs/RUNBOOK.md

# CI System Runbook

Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.

---

## 1. Runner offline in Gitea UI

**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline.
Queued jobs stay pending indefinitely.

**Triage**:
```powershell
# Check service state
Get-Service act_runner

# Last 50 lines of runner log
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50

# Check registration file is intact
Test-Path 'F:\CI\act_runner\.runner'
```

**Fix**:
```powershell
# Restart the service
Restart-Service act_runner

# Verify it came back online (wait ~10s then check Gitea UI)
Get-Service act_runner | Select-Object Status, StartType

# If service won't start, check NSSM log
& 'C:\nssm\nssm.exe' status act_runner
```

If the `.runner` registration file is missing or corrupt, re-register:
```powershell
cd F:\CI\act_runner
.\act_runner.exe register --no-interactive `
    --instance http://10.10.20.11:3100 `
    --token <token-from-gitea-admin> `
    --name local-windows-runner `
    --labels "windows-build:host,dotnet:host,msbuild:host"
```

**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`.

---

## 2. All builds fail in Phase 2 (VM clone / start)

**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like:
- `vmrun clone failed`
- `vmrun start failed`
- `Template VMX not found`
- `Could not detect VM IP address`

**Triage**:
```powershell
# List all running VMs
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list

# Check template VMX exists and is accessible
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

# Check for orphaned clones that may be consuming disk
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime

# Check disk free space
Get-PSDrive F | Select-Object Name, Free, Used

# Check for a stuck vm-start lock from a crashed job
Test-Path 'F:\CI\State\vm-start.lock'
```

**Fix** — by root cause:

*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`.

*Parent VMDK locked* (VMware left a lock file after host crash):
```powershell
# Stop all VMs
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
# Delete lock files
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue
```

*Snapshot missing* (`BaseClean` was deleted or renamed):
```powershell
# List snapshots on template VM
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name
```

*Disk full* (clone delta files need space):
```powershell
# Emergency cleanup — remove all orphaned clones
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
# Then run retention
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3
```

*Stale vm-start lock* (from a job that crashed without cleanup):
```powershell
Remove-Item 'F:\CI\State\vm-start.lock' -Force
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force
```

**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.

---

## 3. Builds are slow

**Symptom**: jobs that previously completed in ~3 min now take 8+ min.
Phase durations visible in `F:\CI\Logs\<jobId>\invoke-ci.jsonl`.

**Triage**:
```powershell
# Check disk free space (below 50 GB = fragmented writes)
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}

# Check active VM CPU usage (Task Manager or:)
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending

# Check VMnet8 NAT adapter status
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }

# Parse JSONL for per-phase durations (requires jq or manual inspection)
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status
```

**Fix** — by root cause:

*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK.

*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2.

*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running.

*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`.

**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).

---

## 4. Template VMX corrupt after host crash

**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state.

**Triage**:
```powershell
# Try starting the template directly in VMware Workstation UI
# If it reports "configuration file error" or "disk lock", proceed below.

# Check for lock files
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'

# Check if backup exists
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5
```

**Fix**:

*Lock files only* (common after hard shutdown):
```powershell
# Ensure no VMware processes are running
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
# Remove locks
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
# Test clone
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
```

*VMX or VMDK truly corrupt — restore from backup*:
```powershell
# Stop all CI activity first
Stop-Service act_runner

# Identify latest backup
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Write-Host "Restoring from: $($latest.FullName)"

# Replace template directory
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse

# Restart runner
Start-Service act_runner
```

*No backup exists*: must re-provision the template from scratch.
Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare).
Estimated time: 2-4 hours including Windows Update.

**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.

---

## Windows host baseline

**Data**: 2026-05-17 (Phase A closure — commit `36913ab6`)

**Hardware**: Intel i9-10900X, 64 GB RAM, NVMe SSD

**Versioni**:
- `ci_orchestrator`: v2.0.0-phaseA (SHA `b4ca7f3`), Python 3.13.3
- `act_runner`: v1.0.2

**Benchmark infra — VM lifecycle** (`Measure-CIBenchmark.ps1`, 4 iter, template Windows):

| Phase     | iter 1 | iter 2 | iter 3 | iter 4 | **media** |
| --------- | ------ | ------ | ------ | ------ | --------- |
| Clone (s) | 0.63   | 0.63   | 0.62   | 0.61   | **0.62**  |
| Start (s) | 1.75   | 1.89   | 1.72   | 1.72   | **1.77**  |
| IP (s)    | 66.57  | 20.21  | 85.07  | 60.97  | **58.2**  |
| WinRM (s) | 0.01   | 0.01   | 0.01   | 0.01   | **0.01**  |
| Destroy(s)| 4.81   | 6.39   | 4.50   | 4.20   | **4.98**  |
| Boot tot  | 68.96  | 22.74  | 87.42  | 63.31  | **60.6**  |

> Fase IP (20–85 s) = costo dominante e variabile (detect IP via VMware Tools). Normale, non bloccante.

**Tempo medio per job** (smoke `self-test.yml`, Gitea Actions, Passo 6 Phase A):

| Template       | Transport             | Wall time |
| -------------- | --------------------- | --------- |
| WinBuild2025   | in-guest git clone    | 26 s      |
| WinBuild2025   | host-side clone + zip | 28 s      |
| LinuxBuild2404 | in-guest git clone    | 51 s      |
| LinuxBuild2404 | host-side clone + zip | 46 s      |
| **media Win**  |                       | **27 s**  |
| **media Linux**|                       | **49 s**  |

**Success rate**: 100 % — burn-in 12/12 (3 round × 4 job concorrenti, Passo 7 Phase A).

> Questi valori sono il baseline di riferimento per il criterio B7 ("tempo medio entro ±20% baseline Windows"). Margini: Win ≤ 32 s, Linux ≤ 59 s.

---

## Quick Reference

| Symptom             | First command                                                      |
| ------------------- | ------------------------------------------------------------------ |
| Runner offline      | `Get-Service act_runner`, then `Restart-Service act_runner`        |
| Phase 2 clone fails | `Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx`          |
| Disk full           | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1`   |
| Stale lock          | `Remove-Item F:\CI\State\vm-start.lock`                            |
| Slow builds         | Check `invoke-ci.jsonl` phase timestamps; check disk I/O           |
| Template corrupt    | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` |
| Snapshot missing    | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME`       |
| IP collision        | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity`      |

---

## 5. Template Refresh Procedure

Use this procedure when the template OS needs updated packages, toolchain upgrades, or
a new snapshot.  Run on the **host** with an elevated PowerShell 5.1 session.

### 5.1 Pre-flight

```powershell
# Stop the runner so no CI jobs start during the refresh
Stop-Service act_runner

# Verify no clone VMs are running
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
# Expected: "Total running VMs: 0"

# Backup the existing template (keeps last 3 by default)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates
```

### 5.2 Boot the template

**Windows (WinBuild2025)**:
```powershell
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
```

**Linux (LinuxBuild2404)**:
```powershell
$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
```

### 5.3 Apply updates inside the template

**Windows** — connect via WinRM or open the VMware console, then run the Prepare
script from the host:
```powershell
$vmxWin   = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
$credTgt  = 'BuildVMGuest'    # Windows Credential Manager target
$cred     = Get-StoredCredential -Target $credTgt   # requires CredentialManager module
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
    -VMXPath    $vmxWin `
    -Credential $cred
```

**Linux** — SSH into the template and run the toolchain script:
```powershell
$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
# Get IP (wait for VMware Tools if needed)
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
# Apply updates
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
    -VMXPath  $vmxLin `
    -SshKeyPath 'F:\CI\keys\ci_linux'
```

Alternatively, run `Install-CIToolchain-WinBuild2025.ps1` / `Install-CIToolchain-Linux2404.sh`
manually inside the guest to apply only toolchain changes without the full Prepare script.

### 5.4 Shut down and snapshot

```powershell
# Shut down gracefully (wait up to 120 s)
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'   # or Linux vmx
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft

# Name: BaseClean_yyyyMMdd  (keeps old name for rollback reference)
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
Write-Host "Snapshot created: $snapshotName"
```

Confirm no `.vmem` / `.vmsn` files exist before snapshotting (see AGENTS.md item 9):
```powershell
Get-ChildItem (Split-Path $vmx) -Filter '*.vmem'  # must be empty
```

### 5.5 Validate

```powershell
# Run the validation script
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
    -VMXPath $vmx -SnapshotName $snapshotName
```

For Linux, also run a quick SSH smoke-test from the host:
```powershell
Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
    -Command 'gcc --version && cmake --version' -PassThru
$result.Output
```

### 5.6 Run a smoke workflow

Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI.
Confirm the job uses the new snapshot and completes successfully.

### 5.7 Promote the new snapshot

Update `GITEA_CI_SNAPSHOT_NAME` in `runner/config.yaml` and redeploy:
```powershell
# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'

# Deploy config and restart runner
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner
```

### 5.8 Retain old snapshot 7 days, then delete

Keep the previous `BaseClean_*` snapshot for 7 days as a rollback point:
```powershell
# List existing snapshots
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx

# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME
```

### 5.9 Rollback procedure

If a smoke-test failure is discovered after promotion:
```powershell
# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
# (or set it back to 'BaseClean' for the permanent base)
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner
# The prior snapshot is still in the template — jobs will use it immediately.
```

---

## 6. Windows host pre-migration baseline (reference for B7)

Recorded 2026-05-17 — `Measure-CIBenchmark.ps1 × 4 iterations`, Python
orchestrator post-Phase-A, Windows 11 + VMware Workstation Pro,
template `WinBuild2025` / snapshot `BaseClean`.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | WinRM (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1       | 0.63      | 1.75      | 66.57          | 0.01      | 4.81        | 68.96          |
| 2       | 0.63      | 1.89      | 20.21          | 0.01      | 6.39        | 22.74          |
| 3       | 0.62      | 1.72      | 85.07          | 0.01      | 4.50        | 87.42          |
| 4       | 0.61      | 1.72      | 60.97          | 0.01      | 4.20        | 63.31          |
| **avg** | **0.62**  | **1.77**  | **58.20**      | **0.01**  | **4.98**    | **60.61**      |

**Key finding**: IP-acquire phase dominates total time and is highly variable
(20–85 s) due to VMware Tools guest IP detection latency. Clone/Start/WinRM
are negligible and stable.

**B7 comparison guidance** (tolerance ±20%):

| Metric           | Windows baseline | ±20% range  |
| ---------------- | ---------------- | ----------- |
| Clone            | 0.62 s           | 0.50–0.74 s |
| Start            | 1.77 s           | 1.42–2.12 s |
| Destroy          | 4.98 s           | 3.98–5.98 s |
| Boot total (avg) | 60.6 s           | 48.5–72.7 s |

IP-acquire variance on Windows (σ ≈ 26 s) means boot-total comparison
requires ≥10 samples on Linux to be meaningful. If Linux avg boot total
exceeds 72.7 s, open an issue in `TODO.md` with per-phase breakdown before
declaring B7 failed — check whether IP-acquire increased or non-IP phases
regressed.

---

## 7. Linux host post-migration baseline (B7 result)

Recorded 2026-05-24 — `Measure-CIBenchmark.ps1 × 4 iterations`, Linux Mint
host + VMware Workstation Pro Linux, template `WinBuild2025` / snapshot
`BaseClean`. Ready column = WinRM/5986 TCP probe.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1       | 0.42      | 1.89      | 53.06          | 0.03      | 4.55        | 55.40          |
| 2       | 0.40      | 1.89      | 129.76         | 0.00      | 4.96        | 132.05         |
| 3       | 0.52      | 2.81      | 176.83         | 0.00      | 5.67        | 180.16         |
| 4       | 0.40      | 1.90      | 39.17          | 0.00      | 4.51        | 41.47          |
| **avg** | **0.44**  | **2.12**  | **99.71**      | **0.01**  | **4.92**    | **102.27**     |

**Phase verdict vs Windows baseline (±20%):**

| Metric  | Windows | Linux avg | In range?                          |
| ------- | ------- | --------- | ---------------------------------- |
| Clone   | 0.62 s  | 0.44 s    | ✓ (faster)                         |
| Start   | 1.77 s  | 2.12 s    | ✓ (at upper edge)                  |
| Destroy | 4.98 s  | 4.92 s    | ✓                                  |
| IP avg  | 58.2 s  | 99.7 s    | ✗ outside — IP variance (39–177 s) |
| Ready   | 0.01 s  | 0.01 s    | ✓                                  |

**Key finding**: Clone/Start/Ready/Destroy within ±20%. IP-acquire dominates
and is highly variable on Linux host (σ ≈ 57 s, range 39–177 s) — wider than
Windows (σ ≈ 26 s). This is VMware Tools DHCP/guestinfo reporting latency,
not a regression in orchestrator logic. With 4 samples the avg is not stable;
additional runs may close the gap. No non-IP phase regressed.

---

## 8. Static IP baseline — WinBuild2025 with ip_pool (B8 result)

Recorded 2026-05-25 — `Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 4`,
Linux Mint host, template `WinBuild2025` / snapshot `BaseClean`.
`guestinfo.ip-assignment` injected into cloned VMX before start;
`ci-static-ip.ps1` scheduled task applies IP at boot and writes back
`guestinfo.ci-ip`. IP column = time until `guestinfo.ci-ip` readable via
`vmrun readVariable` (`-GuestInfoOnly` mode — DHCP fallback disabled).
Ready column = WinRM/5986 TCP probe after IP known.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1       | 0.41      | 1.86      | 21.80          | 0.00      | 9.75        | 24.07          |
| 2       | 0.40      | 1.88      | 21.74          | 0.00      | 9.81        | 24.02          |
| 3       | 0.40      | 1.90      | 21.75          | 0.00      | 12.26       | 24.05          |
| 4       | 0.40      | 1.89      | 21.77          | 0.00      | 9.77        | 24.06          |
| **avg** | **0.40**  | **1.88**  | **21.77**      | **0.00**  | **10.40**   | **24.05**      |

**Three-way comparison — B6 Windows DHCP / B7 Linux DHCP / B8 Linux static IP:**

| Metric     | B6 Win DHCP avg | B7 Lin DHCP avg | B8 Lin static avg | B8 vs B6         | B8 vs B7         |
| ---------- | --------------- | --------------- | ----------------- | ---------------- | ---------------- |
| Clone      | 0.62 s          | 0.44 s          | 0.40 s            | −35%             | ≈ same           |
| Start      | 1.77 s          | 2.12 s          | 1.88 s            | +6%              | ≈ same           |
| IP acquire | 58.2 s          | 99.7 s          | 21.8 s            | **−63%**         | **−78%**         |
| Ready      | 0.01 s          | 0.01 s          | 0.00 s            | ≈ same           | ≈ same           |
| Boot total | 60.6 s          | 102.3 s         | 24.1 s            | **−60%**         | **−76%**         |
| IP σ       | ~26 s           | ~57 s           | <0.03 s           | **deterministic**| **deterministic**|

**Key findings**:
- Static IP beats even the Windows DHCP baseline by 60% on boot total.
- IP acquire drops from 58 s (Win) / 100 s (Linux) DHCP to a deterministic
  21.8 s — variance eliminated entirely (σ < 0.03 s).
- Ready = 0 s: WinRM is already listening on the static IP by the time
  `guestinfo.ci-ip` is written — no additional TCP probe wait.
- `ci-static-ip.ps1` startup latency (~21.8 s) is the new floor; it reflects
  Windows boot + Task Scheduler + NIC reconfiguration time.
- Clone/Start/Ready unchanged across all three baselines — static IP has no
  side effects on non-IP phases.
- Destroy is slower than B6/B7 (~10 s vs ~5 s) — likely disk pressure or
  clone state at test time, unrelated to static IP.

---

## 9. Concurrent capacity burn-in (B7 — 4 × 10)

Recorded 2026-06-07 — `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10`,
Linux Mint host + VMware Workstation Pro Linux, run as `ci-runner`, static IP
pool (`192.168.79.201–204`). Repo `Simone/burnin-dummy`. Each "round" is the
wall-clock time for 4 concurrent end-to-end jobs (clone → boot → IP →
transport → build → artifacts → destroy). This is the first concurrent
capacity burn-in (the §6–§8 baselines are single-job `Measure-CIBenchmark`).

| Template          | Snapshot         | Build cmd       | Jobs PASS | Round time (min–max) | Round avg |
| ----------------- | ---------------- | --------------- | --------- | -------------------- | --------- |
| WinBuild2025      | `BaseClean`      | `build.ps1`     | 40 / 40   | 68–81 s              | ~78.6 s   |
| LinuxBuild2404    | `BaseClean-Linux`| `build.sh`      | 40 / 40   | 70–71 s              | ~70.2 s   |

**Result: OVERALL PASS** — 80/80 jobs, 20/20 rounds, zero orphaned clones,
zero leaked IP leases between rounds.

**Key findings**:
- Per-round wall time is near-deterministic (Win σ ≈ 1.3 s, Linux σ ≈ 0.4 s),
  confirming the static-IP pool eliminates the DHCP IP-acquire variance that
  dominated the §6/§7 single-job baselines (σ 26–57 s).
- 4-way concurrency adds no contention penalty: round time ≈ single-job boot
  total (§8 static ≈ 24 s) plus build + serialized destroy, well within the
  4-IP pool capacity.
- Linux rounds (~70 s) slightly faster and tighter than Windows (~79 s).
- IP-pool release-on-completion works under concurrency — pool returned to
  all-free (`null`) after every round.

**Operational note (pool not auto-reconciled)**: `ip-pool.json` is *not*
reconciled against live clones at startup. A job killed mid-flight (SIGKILL,
crash) leaks its lease permanently; with only 4 IPs for parallelism 4, a
single leak exhausts the pool and every subsequent round fails fast with
`IP pool exhausted after 60s`. Recovery: stop all jobs, confirm
`build-vms/` empty, then reset the pool:

```bash
sudo -u ci-runner /opt/ci/venv/bin/python -c \
  "import pathlib; pathlib.Path('/var/lib/ci/ip-pool.json').write_text('{}\n')"
```

See `TODO.md` (IP-pool auto-reconciliation) for the proposed fix.

---

## 10. Linux guest single-job baseline — LinuxBuild2404 (B7 follow-up)

Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10`,
Linux Mint host + VMware Workstation Pro Linux, run as `ci-runner`, template
`LinuxBuild2404` / snapshot `BaseClean-Linux`, **DHCP** (no static IP — the
guest reads its DHCP lease and publishes it via `guestinfo.ci-ip`; the
Windows-only `ci-static-ip.ps1` task does not apply). Ready = SSH/22 probe.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1–10    | 0.22      | 1.14–1.16 | 7.44–7.48      | 0.00–0.01 | 4.90–4.99   | 8.80–8.86      |
| **avg** | **0.22**  | **1.14**  | **7.45**       | **0.00**  | **4.94**    | **8.82**       |

**Key finding**: the Linux guest is near-deterministic *even on DHCP*
(σ ≈ 0.01 s on IP-acquire) — no static-IP injection needed. IP-acquire is
7.45 s vs Windows-guest 58 s (DHCP, §6) / 21.8 s (static, §8). Boot total
8.82 s — ~2.7× faster than the Windows static-IP guest.

**Guest-vs-guest comparison (Linux host, single-job):**

| Metric     | Win guest static (§8) | Lin guest DHCP (§10) | Lin advantage |
| ---------- | --------------------- | -------------------- | ------------- |
| Clone      | 0.40 s                | 0.22 s               | −45%          |
| Start      | 1.88 s                | 1.14 s               | −39%          |
| IP acquire | 21.77 s               | 7.45 s               | **−66%**      |
| Ready      | 0.00 s                | 0.00 s               | ≈ same        |
| Destroy    | 10.40 s               | 4.94 s               | −53%          |
| Boot total | 24.05 s               | 8.82 s               | **−63%**      |

Caveat: different IP modes (Win static-IP task vs Lin DHCP+guestinfo) — the
comparison reflects the *as-deployed* path for each guest, not an
IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated
by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such
in-guest agent.

---

## 11. Static IP baseline — WinBuild2025 on Windows host (pairs §8)

Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200
-Iterations 10`, **Windows 11 host** + VMware Workstation Pro, template
`WinBuild2025` / snapshot `BaseClean`, `guestinfo.ip-assignment` injected
before start; the in-guest `ci-static-ip.ps1` task applies the IP and writes
back `guestinfo.ci-ip`. IP column = time to `guestinfo.ci-ip` readable
(`-GuestInfoOnly`). Ready = WinRM/5986 TCP probe. This is the Windows-host
counterpart to the Linux-host §8.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1       | 0.70      | 1.71      | 26.65          | 0.45      | 10.47       | 29.51          |
| 2       | 0.62      | 1.75      | 24.29          | 2.01      | 24.29       | 28.67          |
| 3       | 0.63      | 1.76      | 21.83          | 6.01      | 24.32       | 30.23          |
| 4       | 0.62      | 1.57      | 21.92          | 6.02      | 10.80       | 30.13          |
| 5       | 0.62      | 1.56      | 24.24          | 2.02      | 24.21       | 28.44          |
| 6       | 0.62      | 1.55      | 21.94          | 6.01      | 24.28       | 30.12          |
| 7       | 0.63      | 1.55      | 21.85          | 6.01      | 24.28       | 30.04          |
| 8       | 0.64      | 1.74      | 21.73          | 6.01      | 11.25       | 30.12          |
| 9       | 0.62      | 1.57      | 21.90          | 6.01      | 24.28       | 30.10          |
| 10      | 0.62      | 1.74      | 21.72          | 6.02      | 10.76       | 30.10          |
| **avg** | **0.63**  | **1.65**  | **22.81**      | **4.66**  | **18.89**   | **29.75**      |

**Host comparison — §8 (Linux host) vs §11 (Windows host), Win guest static:**

| Metric     | §8 Linux host | §11 Windows host | Δ (Win vs Lin)            |
| ---------- | ------------- | ---------------- | ------------------------- |
| Clone      | 0.40 s        | 0.63 s           | +58 % (NTFS linked clone) |
| Start      | 1.88 s        | 1.65 s           | −12 %                     |
| IP acquire | 21.77 s       | 22.81 s          | +5 % (≈ same)             |
| Ready      | 0.00 s        | 4.66 s           | +4.7 s                    |
| Destroy    | 10.40 s       | 18.89 s          | +82 % (bimodal ~11/24 s)  |
| Boot total | 24.05 s       | 29.75 s          | +24 %                     |

**Key findings**:
- The **static-IP floor (~21.8 s) is host-independent** — exactly as the plan
  predicted. It is set by Windows boot + Task Scheduler + NIC reconfig inside
  the guest, identical across both hosts (Win 22.81 s vs Lin 21.77 s, +5 %).
- **Ready ≠ 0 on the Windows host** (avg 4.66 s, mostly ~6 s) where it was 0 on
  Linux. After `ci-static-ip.ps1` rewrites the NIC, WinRM/5986 needs a few
  seconds to re-bind on the new address before the TCP probe succeeds; on the
  Linux host WinRM was already listening when `ci-ip` was written.
- Clone is slower on NTFS (+58 %); destroy is slower and bimodal (~11 s vs
  ~24 s) — the ~24 s cases hit the 10 s graceful-stop timeout before deleteVM.
- Net boot-total penalty of the Windows host at constant guest+mode: **+24 %**.

**Pre-req gotcha (template parity)**: the first §11 attempt timed out at 300 s
every iteration — `guestinfo.ci-ip` never appeared, guest stayed on DHCP.
Cause: the `F:\CI\Templates\WinBuild2025` `BaseClean` snapshot did **not**
contain the in-guest `ci-static-ip` agent (added to the Linux-host template
during B8/§8, never propagated to the Windows-host copy). Host-side guestinfo
injection works, but with no in-guest consumer the static IP is never applied.
Fixed by copying the Linux-host template set onto the Windows host (template
parity), after which all 10 iterations passed. If §11 ever regresses to 300 s
timeouts, confirm `C:\CI\ci-static-ip.ps1` + the `CI-StaticIp` startup task
exist inside a clone (see [§13 op-notes](#13-concurrent-capacity-burn-in-on-windows-host-pairs-9)).

---

## 12. Linux guest single-job baseline — Windows host (pairs §10)

Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10`,
**Windows 11 host** + VMware Workstation Pro, template `LinuxBuild2404` /
snapshot `BaseClean-Linux`, **DHCP** (guest publishes its lease via
`guestinfo.ci-ip`). Ready = SSH/22 probe. Windows-host counterpart to §10.

| Iter    | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1–10    | 0.62–0.69 | 1.37–1.81 | 11.26–11.42    | 0.00–0.02 | 6.02–6.21   | 13.43–13.83    |
| **avg** | **0.64**  | **1.50**  | **11.36**      | **0.00**  | **6.13**    | **13.51**      |

**Host comparison — §10 (Linux host) vs §12 (Windows host), Lin guest DHCP:**

| Metric     | §10 Linux host | §12 Windows host | Δ (Win vs Lin)             |
| ---------- | -------------- | ---------------- | -------------------------- |
| Clone      | 0.22 s         | 0.64 s           | +191 % (NTFS linked clone) |
| Start      | 1.14 s         | 1.50 s           | +32 %                      |
| IP acquire | 7.45 s         | 11.36 s          | +52 %                      |
| Ready      | 0.00 s         | 0.00 s           | ≈ same                     |
| Destroy    | 4.94 s         | 6.13 s           | +24 %                      |
| Boot total | 8.82 s         | 13.51 s          | **+53 %**                  |

**Key findings**:
- The Linux guest stays near-deterministic on the Windows host too
  (IP σ ≈ 0.05 s) — no static-IP agent needed; DHCP+guestinfo is stable.
- The **host penalty is larger for the Linux guest (+53 % boot total) than for
  the Windows guest (+24 %, §11)** because the Linux guest's boot is so fast
  (8.82 s) that fixed Windows-host overheads — NTFS clone (+0.4 s) and the
  vmnet8/NAT DHCP+guestinfo round-trip (+3.9 s on IP-acquire) — are a larger
  *fraction* of a small total. In absolute terms the host adds ~4.7 s either way.
- IP-acquire delta (7.45 → 11.36 s) isolates the **host networking effect**
  (Windows vmnet8 NAT + VMware Tools guestinfo path) on an otherwise identical
  guest.

---

## 13. Concurrent capacity burn-in on Windows host (pairs §9)

Recorded 2026-06-07 — `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10`,
**Windows 11 host** + VMware Workstation Pro, static IP pool
`192.168.79.201–204` (added to `F:\CI\config.toml` `[ip_pool]` for parity with
the Linux-host §9). Repo `Simone/burnin-dummy`. Windows guests use the static
pool; Linux guests skip it (DHCP+guestinfo). Windows-host counterpart to §9.

| Template       | Snapshot          | Build cmd   | Jobs PASS | Rounds OK | Round time (PASS rounds) | Round avg (PASS) |
| -------------- | ----------------- | ----------- | --------- | --------- | ------------------------ | ---------------- |
| WinBuild2025   | `BaseClean`       | `build.ps1` | 36 / 40   | 7 / 10    | 88–93 s                  | ~91.6 s          |
| LinuxBuild2404 | `BaseClean-Linux` | `build.sh`  | 40 / 40   | 10 / 10   | 88–112 s                 | ~96.4 s          |

**Result: PARTIAL** — Linux 40/40 (OVERALL PASS); Windows 36/40 (OVERALL FAIL,
3 rounds with transient WinRM faults). 76/80 jobs overall.

**Host comparison vs §9 (Linux host):**

| Guest | §9 Linux host    | §13 Windows host             | Δ round avg            |
| ----- | ---------------- | ---------------------------- | ---------------------- |
| Win   | 40/40, ~78.6 s   | 36/40, ~91.6 s (PASS rounds) | +16 % + WinRM instability |
| Lin   | 40/40, ~70.2 s   | 40/40, ~96.4 s               | +37 %                  |

**Key findings**:
- **Linux-guest concurrency is stable on the Windows host** (40/40, 10/10) just
  as on the Linux host — SSH transport showed no faults. Round avg +37 % vs §9,
  consistent with the §12 single-job host penalty.
- **Windows-guest concurrency is *not* robust on the Windows host.** Rounds 1–5,
  9–10 were clean (~91 s); rounds 6–8 each lost 1–2 jobs to **transient WinRM
  faults** under 4× load and ballooned to 269 / 266 / 622 s (30 s connect
  timeouts + pypsrp retries). Two failure signatures:
    - `ConnectTimeoutError` to 5986 (WinRM listener briefly unreachable);
    - `WSManFault 2150858843` "the shell was not found on the server" (the WinRM
      shell was recycled mid-build).
  Failures **self-recovered**: each failed job released its IP slot, no clones
  were orphaned, the IP pool returned to all-free, and rounds 9–10 were clean
  again. So this is bursty WinRM contention, not a monotonic collapse or a leak.
- Likely cause: vCPU oversubscription (4 VMs × 4 vCPU = 16 vCPU) plus the higher
  host-OS overhead of Windows-host VMware tips concurrent WinRM into timeouts;
  RAM was not the constraint (13.8 GB free of 63.7 GB during the run). The Linux
  host on the same hardware sustained 40/40 (§9). **Recommendation for sustained
  4× Windows-guest concurrency on the Windows host**: lower per-VM vCPU, reduce
  parallelism to 3, or raise WinRM/pypsrp connect timeouts + add a job-level
  retry. Tracked in `TODO.md`.

**Operational notes (Windows-host benchmark prerequisites)** — discovered while
running §11–§13; required for any Windows-host orchestration:
1. **Template parity** — the Windows-host templates must carry the same in-guest
   agents as the Linux-host templates (`ci-static-ip.ps1` + `CI-StaticIp`
   startup task for Windows guests; `ci-report-ip` for Linux). Without the
   static-IP agent, static-IP jobs hang for the full IP timeout (see §11).
2. **Production venv must be current** — `F:\CI\python\venv` predated the
   `ip_pool` feature; `[ip_pool]` in config is silently ignored by a stale venv.
   Re-run `F:\CI\python\venv\Scripts\python.exe -m pip install .` (non-editable)
   after pulling new code, then verify `import ci_orchestrator.ip_pool` succeeds.
3. **Set `CI_VENV_PYTHON`** before invoking `Test-CapacityBurnIn.ps1` /
   `Invoke-CIJob.ps1` interactively. The shim's auto-detect branch dereferences
   `$IsWindows`, which is undefined under Windows PowerShell 5.1 + `StrictMode`
   and throws `VariableIsUndefined`; setting `CI_VENV_PYTHON` skips that branch.
   Production sets it via the runner (`CI_PYTHON_LAUNCHER`). *Note (Phase C): on
   the Linux host this is moot — burn-in now runs via `bench run` (below), which
   never touches the shim.*

### Phase C — Python equivalents (run these on the Linux host; no `pwsh`)

The manual `.ps1` procedures above are superseded by native CLI commands. Invoke
as the service user (`sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ...`):

| Old PowerShell (Linux host)                         | Python command                                   |
| --------------------------------------------------- | ------------------------------------------------ |
| `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10` | `bench run --concurrency 4 --rounds 10 --guest-os linux` |
| `Measure-CIBenchmark.ps1 -Iterations N`             | `bench measure --iterations N --guest-os linux`  |
| `Test-Smoke.ps1 -GuestOS Linux`                     | `smoke run --guest-os linux`                     |
| `Validate-HostState.ps1` (host checks)              | `validate host`                                  |
| `Test-CIGuestWinRM.ps1` (transport diag)            | `validate guest --host <IP> [--winrm\|--ssh]`    |
| `Set-CIGuestCredential.ps1`                          | `creds set --target BuildVMGuest --user <u> --password-stdin` |

`bench run` writes a JSON report to `CI_ARTIFACTS/bench/`; `bench measure` appends
to `benchmark.jsonl` (same fields as before, so the §6–§13 trend lines stay
comparable). The benchmark records in §6–§13 above are kept as historical
provenance — they were captured with the `.ps1` tools before the cutover.

---

## 14. Host × guest × IP-mode matrix (complete)

With §11–§13 the Linux-host / Windows-host comparison is symmetric:

| Host \ (Guest, mode)  | Win/DHCP | Win/static | Lin/DHCP | burn-in 4×10                |
| --------------------- | -------- | ---------- | -------- | --------------------------- |
| **Linux** (boot/avg)  | §7 102 s | §8 24.1 s  | §10 8.8 s | §9 Win 78.6 s / Lin 70.2 s  |
| **Windows** (boot/avg)| §6 60.6 s| §11 29.8 s | §12 13.5 s| §13 Win ~91.6 s / Lin 96.4 s |

Read **down a column** = host effect at constant guest+mode; **across a row** =
guest/IP-mode effect at constant host. Headline: the Windows host adds **+24 %**
(Win-guest static) to **+53 %** (Lin-guest DHCP) on single-job boot-to-ready,
the static-IP floor (~22 s) is host-independent, and Windows-guest WinRM is the
only path that loses jobs under sustained 4× concurrency (Linux/SSH is 40/40 on
both hosts).

> Note on §6 vs §7: §6 (Win host DHCP) 60.6 s is *faster* than §7 (Lin host
> DHCP) 102 s, the reverse of every other column. Both are 4-iteration DHCP
> baselines whose IP-acquire is dominated by high-variance VMware-Tools polling
> (σ 26–57 s); the 10-iteration static (§8/§11) and Linux (§10/§12) rows are the
> reliable host-effect signal.

---

## 15. Dual-boot operation (Linux ⇄ Windows host)

The CI machine is **dual-boot** on one piece of hardware: Linux Mint and
Windows 11 never run at the same time. Each OS carries a full CI stack
(orchestrator + runner + templates under its own root). Booting Linux brings
up the Linux runner; booting Windows brings up the Windows runner. Templates
and the `burnin-dummy` repo are kept at **template parity** across both roots.

| | Linux host | Windows host |
| --- | --- | --- |
| CI root | `/var/lib/ci/` | `F:\CI\` |
| Runner | `act-runner.service` (systemd) | `actions-runner` (Windows service / NSSM) |
| Transport to guests | WinRM (Win guest) / SSH (Linux guest) | same |
| vmrun | `/usr/bin/vmrun` | `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe` |

### Current GRUB behaviour

`GRUB_DEFAULT=0`, `GRUB_TIMEOUT=0`, `GRUB_TIMEOUT_STYLE=hidden` — the machine
boots **straight into Linux Mint with no menu**. Boot entries present:
`Linux Mint 22.3 Cinnamon` (default) and
`Windows Boot Manager (on /dev/nvme0n1p1)`.

### Switch Linux → Windows

With the hidden/zero-timeout GRUB, hold **Esc** (or **Shift**) during early
boot to reveal the GRUB menu, then pick `Windows Boot Manager`. For a planned,
unattended switch, set a one-shot next-boot target instead:

```bash
# Requires GRUB_DEFAULT=saved (see "Optional GRUB tweak" below).
sudo grub-reboot "Windows Boot Manager (on /dev/nvme0n1p1)"
sudo reboot
```

Before rebooting out of Linux: confirm no job is mid-flight
(`sudo -u ci-runner ls /var/lib/ci/build-vms/` empty) and optionally pause the
Linux runner in the Gitea UI so queued jobs wait for the Windows runner.

### Switch Windows → Linux

Reboot; with `GRUB_DEFAULT=0` the machine returns to Linux automatically (no
key needed). If `grub-reboot` / `GRUB_DEFAULT=saved` is in use, Linux is the
saved default after a normal Linux shutdown.

### Post-boot verification (either OS)

Linux:
```bash
systemctl is-active act-runner        # → active
systemctl --failed                    # → 0 loaded units
systemctl list-timers 'ci-*'          # all ci-* timers scheduled
# then confirm the runner shows online in the Gitea UI (Admin → Runners)
```

Windows (PowerShell):
```powershell
Get-Service actions-runner            # Status Running
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list   # vmrun OK
# then confirm the runner shows online in the Gitea UI
```

### Optional GRUB tweak for everyday dual-boot

To get a short pick menu and remember the last choice (handy if Windows is used
often), edit `/etc/default/grub`:

```bash
GRUB_DEFAULT=saved
GRUB_SAVEDEFAULT=true
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
```

```bash
sudo update-grub
```

This is **optional** — the default boots-to-Linux behaviour is intentional so
the machine comes back as the Linux CI host after any unattended reboot
(power-loss, kernel update). Leave it as-is unless Windows becomes the daily
driver.