local-ci-cd-system/docs/RUNBOOK.md

CI System Runbook

Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.

---

1. Runner offline in Gitea UI

Symptom: http://10.10.20.11:3100/admin/runners shows local-windows-runner as offline. Queued jobs stay pending indefinitely.

Triage:

# Check service state
Get-Service act_runner

# Last 50 lines of runner log
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50

# Check registration file is intact
Test-Path 'F:\CI\act_runner\.runner'

Fix:

# Restart the service
Restart-Service act_runner

# Verify it came back online (wait ~10s then check Gitea UI)
Get-Service act_runner | Select-Object Status, StartType

# If service won't start, check NSSM log
& 'C:\nssm\nssm.exe' status act_runner

If the .runner registration file is missing or corrupt, re-register:

cd F:\CI\act_runner
.\act_runner.exe register --no-interactive `
    --instance http://10.10.20.11:3100 `
    --token <token-from-gitea-admin> `
    --name local-windows-runner `
    --labels "windows-build:host,dotnet:host,msbuild:host"

Escalation: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for act_runner errors and inspect F:\CI\act_runner\logs\.

---

2. All builds fail in Phase 2 (VM clone / start)

Symptom: Invoke-CIJob.ps1 fails at Phase 2 with errors like:

• vmrun clone failed
• vmrun start failed
• Template VMX not found
• Could not detect VM IP address

Triage:

# List all running VMs
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list

# Check template VMX exists and is accessible
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

# Check for orphaned clones that may be consuming disk
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime

# Check disk free space
Get-PSDrive F | Select-Object Name, Free, Used

# Check for a stuck vm-start lock from a crashed job
Test-Path 'F:\CI\State\vm-start.lock'

Fix — by root cause:

Template VMX missing/moved: check GITEA_CI_TEMPLATE_PATH in F:\CI\act_runner\config.yaml.

Parent VMDK locked (VMware left a lock file after host crash):

# Stop all VMs
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
# Delete lock files
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue

Snapshot missing (BaseClean was deleted or renamed):

# List snapshots on template VM
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name

Disk full (clone delta files need space):

# Emergency cleanup — remove all orphaned clones
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
# Then run retention
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3

Stale vm-start lock (from a job that crashed without cleanup):

Remove-Item 'F:\CI\State\vm-start.lock' -Force
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force

Escalation: If vmrun clone fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.

---

3. Builds are slow

Symptom: jobs that previously completed in ~3 min now take 8+ min. Phase durations visible in F:\CI\Logs\<jobId>\invoke-ci.jsonl.

Triage:

# Check disk free space (below 50 GB = fragmented writes)
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}

# Check active VM CPU usage (Task Manager or:)
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending

# Check VMnet8 NAT adapter status
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }

# Parse JSONL for per-phase durations (requires jq or manual inspection)
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status

Fix — by root cause:

Low disk space → fragmented VMDKs: run retention policy, then consider vmware-vdiskmanager -d to defragment the template VMDK.

High vmware-vmx CPU with many VMs: reduce capacity in config.yaml from 4 to 2.

VMnet8 NAT bottleneck (slow pip/nuget downloads inside VM): check Services.msc → VMware NAT Service is running.

NVMe saturation: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce capacity: 2.

Escalation: Use invoke-ci.jsonl to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).

---

4. Template VMX corrupt after host crash

Symptom: After an unclean host shutdown, vmrun clone or vmrun start on the template fails. VMware Workstation shows the template in an error state.

Triage:

# Try starting the template directly in VMware Workstation UI
# If it reports "configuration file error" or "disk lock", proceed below.

# Check for lock files
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'

# Check if backup exists
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5

Fix:

Lock files only (common after hard shutdown):

# Ensure no VMware processes are running
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
# Remove locks
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
# Test clone
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'

VMX or VMDK truly corrupt — restore from backup:

# Stop all CI activity first
Stop-Service act_runner

# Identify latest backup
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Write-Host "Restoring from: $($latest.FullName)"

# Replace template directory
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse

# Restart runner
Start-Service act_runner

No backup exists: must re-provision the template from scratch. Follow docs/WINDOWS-TEMPLATE-SETUP.md → Fase A (Deploy) → Fase B (Prepare). Estimated time: 2-4 hours including Windows Update.

Escalation: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.

---

Windows host baseline

Data: 2026-05-17 (Phase A closure — commit 36913ab6)

Hardware: Intel i9-10900X, 64 GB RAM, NVMe SSD

Versioni:

• ci_orchestrator: v2.0.0-phaseA (SHA b4ca7f3), Python 3.13.3
• act_runner: v1.0.2

Benchmark infra — VM lifecycle (Measure-CIBenchmark.ps1, 4 iter, template Windows):

Phase	iter 1	iter 2	iter 3	iter 4	media
Clone (s)	0.63	0.63	0.62	0.61	0.62
Start (s)	1.75	1.89	1.72	1.72	1.77
IP (s)	66.57	20.21	85.07	60.97	58.2
WinRM (s)	0.01	0.01	0.01	0.01	0.01
Destroy(s)	4.81	6.39	4.50	4.20	4.98
Boot tot	68.96	22.74	87.42	63.31	60.6

Fase IP (20–85 s) = costo dominante e variabile (detect IP via VMware Tools). Normale, non bloccante.

Tempo medio per job (smoke self-test.yml, Gitea Actions, Passo 6 Phase A):

Template	Transport	Wall time
WinBuild2025	in-guest git clone	26 s
WinBuild2025	host-side clone + zip	28 s
LinuxBuild2404	in-guest git clone	51 s
LinuxBuild2404	host-side clone + zip	46 s
media Win		27 s
media Linux		49 s

Success rate: 100 % — burn-in 12/12 (3 round × 4 job concorrenti, Passo 7 Phase A).

Questi valori sono il baseline di riferimento per il criterio B7 ("tempo medio entro ±20% baseline Windows"). Margini: Win ≤ 32 s, Linux ≤ 59 s.

---

Quick Reference

Symptom	First command
Runner offline	Get-Service act_runner, then Restart-Service act_runner
Phase 2 clone fails	Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx
Disk full	Get-PSDrive F | Select Free; run Invoke-RetentionPolicy.ps1
Stale lock	Remove-Item F:\CI\State\vm-start.lock
Slow builds	Check invoke-ci.jsonl phase timestamps; check disk I/O
Template corrupt	Remove *.lck files; if persistent, restore from F:\CI\Backups\
Snapshot missing	vmrun listSnapshots <vmx>; update GITEA_CI_SNAPSHOT_NAME
IP collision	Remove-Item F:\CI\State\ip-leases\*.lease; lower capacity

---

5. Template Refresh Procedure

Use this procedure when the template OS needs updated packages, toolchain upgrades, or a new snapshot. Run on the host with an elevated PowerShell 5.1 session.

5.1 Pre-flight

# Stop the runner so no CI jobs start during the refresh
Stop-Service act_runner

# Verify no clone VMs are running
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
# Expected: "Total running VMs: 0"

# Backup the existing template (keeps last 3 by default)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates

5.2 Boot the template

Windows (WinBuild2025):

$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

Linux (LinuxBuild2404):

$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui

5.3 Apply updates inside the template

Windows — connect via WinRM or open the VMware console, then run the Prepare script from the host:

$vmxWin   = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
$credTgt  = 'BuildVMGuest'    # Windows Credential Manager target
$cred     = Get-StoredCredential -Target $credTgt   # requires CredentialManager module
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
    -VMXPath    $vmxWin `
    -Credential $cred

Linux — SSH into the template and run the toolchain script:

$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
# Get IP (wait for VMware Tools if needed)
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
# Apply updates
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
    -VMXPath  $vmxLin `
    -SshKeyPath 'F:\CI\keys\ci_linux'

Alternatively, run Install-CIToolchain-WinBuild2025.ps1 / Install-CIToolchain-Linux2404.sh manually inside the guest to apply only toolchain changes without the full Prepare script.

5.4 Shut down and snapshot

# Shut down gracefully (wait up to 120 s)
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'   # or Linux vmx
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft

# Name: BaseClean_yyyyMMdd  (keeps old name for rollback reference)
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
Write-Host "Snapshot created: $snapshotName"

Confirm no .vmem / .vmsn files exist before snapshotting (see AGENTS.md item 9):

Get-ChildItem (Split-Path $vmx) -Filter '*.vmem'  # must be empty

5.5 Validate

# Run the validation script
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
    -VMXPath $vmx -SnapshotName $snapshotName

For Linux, also run a quick SSH smoke-test from the host:

Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
    -Command 'gcc --version && cmake --version' -PassThru
$result.Output

5.6 Run a smoke workflow

Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI. Confirm the job uses the new snapshot and completes successfully.

5.7 Promote the new snapshot

Update GITEA_CI_SNAPSHOT_NAME in runner/config.yaml and redeploy:

# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'

# Deploy config and restart runner
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner

5.8 Retain old snapshot 7 days, then delete

Keep the previous BaseClean_* snapshot for 7 days as a rollback point:

# List existing snapshots
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx

# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME

5.9 Rollback procedure

If a smoke-test failure is discovered after promotion:

# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
# (or set it back to 'BaseClean' for the permanent base)
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
          'F:\CI\act_runner\config.yaml' -Force
Restart-Service act_runner
# The prior snapshot is still in the template — jobs will use it immediately.

---

6. Windows host pre-migration baseline (reference for B7)

Recorded 2026-05-17 — Measure-CIBenchmark.ps1 × 4 iterations, Python orchestrator post-Phase-A, Windows 11 + VMware Workstation Pro, template WinBuild2025 / snapshot BaseClean.

Iter	Clone (s)	Start (s)	IP acquire (s)	WinRM (s)	Destroy (s)	Boot total (s)
1	0.63	1.75	66.57	0.01	4.81	68.96
2	0.63	1.89	20.21	0.01	6.39	22.74
3	0.62	1.72	85.07	0.01	4.50	87.42
4	0.61	1.72	60.97	0.01	4.20	63.31
avg	0.62	1.77	58.20	0.01	4.98	60.61

Key finding: IP-acquire phase dominates total time and is highly variable (20–85 s) due to VMware Tools guest IP detection latency. Clone/Start/WinRM are negligible and stable.

B7 comparison guidance (tolerance ±20%):

Metric	Windows baseline	±20% range
Clone	0.62 s	0.50–0.74 s
Start	1.77 s	1.42–2.12 s
Destroy	4.98 s	3.98–5.98 s
Boot total (avg)	60.6 s	48.5–72.7 s

IP-acquire variance on Windows (σ ≈ 26 s) means boot-total comparison requires ≥10 samples on Linux to be meaningful. If Linux avg boot total exceeds 72.7 s, open an issue in TODO.md with per-phase breakdown before declaring B7 failed — check whether IP-acquire increased or non-IP phases regressed.

---

7. Linux host post-migration baseline (B7 result)

Recorded 2026-05-24 — Measure-CIBenchmark.ps1 × 4 iterations, Linux Mint host + VMware Workstation Pro Linux, template WinBuild2025 / snapshot BaseClean. Ready column = WinRM/5986 TCP probe.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.42	1.89	53.06	0.03	4.55	55.40
2	0.40	1.89	129.76	0.00	4.96	132.05
3	0.52	2.81	176.83	0.00	5.67	180.16
4	0.40	1.90	39.17	0.00	4.51	41.47
avg	0.44	2.12	99.71	0.01	4.92	102.27

Phase verdict vs Windows baseline (±20%):

Metric	Windows	Linux avg	In range?
Clone	0.62 s	0.44 s	✓ (faster)
Start	1.77 s	2.12 s	✓ (at upper edge)
Destroy	4.98 s	4.92 s	✓
IP avg	58.2 s	99.7 s	✗ outside — IP variance (39–177 s)
Ready	0.01 s	0.01 s	✓

Key finding: Clone/Start/Ready/Destroy within ±20%. IP-acquire dominates and is highly variable on Linux host (σ ≈ 57 s, range 39–177 s) — wider than Windows (σ ≈ 26 s). This is VMware Tools DHCP/guestinfo reporting latency, not a regression in orchestrator logic. With 4 samples the avg is not stable; additional runs may close the gap. No non-IP phase regressed.

---

8. Static IP baseline — WinBuild2025 with ip_pool (B8 result)

Recorded 2026-05-25 — Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 4, Linux Mint host, template WinBuild2025 / snapshot BaseClean. guestinfo.ip-assignment injected into cloned VMX before start; ci-static-ip.ps1 scheduled task applies IP at boot and writes back guestinfo.ci-ip. IP column = time until guestinfo.ci-ip readable via vmrun readVariable (-GuestInfoOnly mode — DHCP fallback disabled). Ready column = WinRM/5986 TCP probe after IP known.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.41	1.86	21.80	0.00	9.75	24.07
2	0.40	1.88	21.74	0.00	9.81	24.02
3	0.40	1.90	21.75	0.00	12.26	24.05
4	0.40	1.89	21.77	0.00	9.77	24.06
avg	0.40	1.88	21.77	0.00	10.40	24.05

Three-way comparison — B6 Windows DHCP / B7 Linux DHCP / B8 Linux static IP:

Metric	B6 Win DHCP avg	B7 Lin DHCP avg	B8 Lin static avg	B8 vs B6	B8 vs B7
Clone	0.62 s	0.44 s	0.40 s	−35%	≈ same
Start	1.77 s	2.12 s	1.88 s	+6%	≈ same
IP acquire	58.2 s	99.7 s	21.8 s	−63%	−78%
Ready	0.01 s	0.01 s	0.00 s	≈ same	≈ same
Boot total	60.6 s	102.3 s	24.1 s	−60%	−76%
IP σ	~26 s	~57 s	<0.03 s	deterministic	deterministic

Key findings:

• Static IP beats even the Windows DHCP baseline by 60% on boot total.
• IP acquire drops from 58 s (Win) / 100 s (Linux) DHCP to a deterministic 21.8 s — variance eliminated entirely (σ < 0.03 s).
• Ready = 0 s: WinRM is already listening on the static IP by the time guestinfo.ci-ip is written — no additional TCP probe wait.
• ci-static-ip.ps1 startup latency (~21.8 s) is the new floor; it reflects Windows boot + Task Scheduler + NIC reconfiguration time.
• Clone/Start/Ready unchanged across all three baselines — static IP has no side effects on non-IP phases.
• Destroy is slower than B6/B7 (~10 s vs ~5 s) — likely disk pressure or clone state at test time, unrelated to static IP.

---

9. Concurrent capacity burn-in (B7 — 4 × 10)

Recorded 2026-06-07 — Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10, Linux Mint host + VMware Workstation Pro Linux, run as ci-runner, static IP pool (192.168.79.201–204). Repo Simone/burnin-dummy. Each "round" is the wall-clock time for 4 concurrent end-to-end jobs (clone → boot → IP → transport → build → artifacts → destroy). This is the first concurrent capacity burn-in (the §6–§8 baselines are single-job Measure-CIBenchmark).

Template	Snapshot	Build cmd	Jobs PASS	Round time (min–max)	Round avg
WinBuild2025	BaseClean	build.ps1	40 / 40	68–81 s	~78.6 s
LinuxBuild2404	BaseClean-Linux	build.sh	40 / 40	70–71 s	~70.2 s

Result: OVERALL PASS — 80/80 jobs, 20/20 rounds, zero orphaned clones, zero leaked IP leases between rounds.

Key findings:

• Per-round wall time is near-deterministic (Win σ ≈ 1.3 s, Linux σ ≈ 0.4 s), confirming the static-IP pool eliminates the DHCP IP-acquire variance that dominated the §6/§7 single-job baselines (σ 26–57 s).
• 4-way concurrency adds no contention penalty: round time ≈ single-job boot total (§8 static ≈ 24 s) plus build + serialized destroy, well within the 4-IP pool capacity.
• Linux rounds (~70 s) slightly faster and tighter than Windows (~79 s).
• IP-pool release-on-completion works under concurrency — pool returned to all-free (null) after every round.

Operational note (pool not auto-reconciled): ip-pool.json is not reconciled against live clones at startup. A job killed mid-flight (SIGKILL, crash) leaks its lease permanently; with only 4 IPs for parallelism 4, a single leak exhausts the pool and every subsequent round fails fast with IP pool exhausted after 60s. Recovery: stop all jobs, confirm build-vms/ empty, then reset the pool:

sudo -u ci-runner /opt/ci/venv/bin/python -c \
  "import pathlib; pathlib.Path('/var/lib/ci/ip-pool.json').write_text('{}\n')"

See TODO.md (IP-pool auto-reconciliation) for the proposed fix.

---

10. Linux guest single-job baseline — LinuxBuild2404 (B7 follow-up)

Recorded 2026-06-07 — Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10, Linux Mint host + VMware Workstation Pro Linux, run as ci-runner, template LinuxBuild2404 / snapshot BaseClean-Linux, DHCP (no static IP — the guest reads its DHCP lease and publishes it via guestinfo.ci-ip; the Windows-only ci-static-ip.ps1 task does not apply). Ready = SSH/22 probe.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1–10	0.22	1.14–1.16	7.44–7.48	0.00–0.01	4.90–4.99	8.80–8.86
avg	0.22	1.14	7.45	0.00	4.94	8.82

Key finding: the Linux guest is near-deterministic even on DHCP (σ ≈ 0.01 s on IP-acquire) — no static-IP injection needed. IP-acquire is 7.45 s vs Windows-guest 58 s (DHCP, §6) / 21.8 s (static, §8). Boot total 8.82 s — ~2.7× faster than the Windows static-IP guest.

Guest-vs-guest comparison (Linux host, single-job):

Metric	Win guest static (§8)	Lin guest DHCP (§10)	Lin advantage
Clone	0.40 s	0.22 s	−45%
Start	1.88 s	1.14 s	−39%
IP acquire	21.77 s	7.45 s	−66%
Ready	0.00 s	0.00 s	≈ same
Destroy	10.40 s	4.94 s	−53%
Boot total	24.05 s	8.82 s	−63%

Caveat: different IP modes (Win static-IP task vs Lin DHCP+guestinfo) — the comparison reflects the as-deployed path for each guest, not an IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such in-guest agent.

---

11. Static IP baseline — WinBuild2025 on Windows host (pairs §8)

Recorded 2026-06-07 — Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 10, Windows 11 host + VMware Workstation Pro, template WinBuild2025 / snapshot BaseClean, guestinfo.ip-assignment injected before start; the in-guest ci-static-ip.ps1 task applies the IP and writes back guestinfo.ci-ip. IP column = time to guestinfo.ci-ip readable (-GuestInfoOnly). Ready = WinRM/5986 TCP probe. This is the Windows-host counterpart to the Linux-host §8.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1	0.70	1.71	26.65	0.45	10.47	29.51
2	0.62	1.75	24.29	2.01	24.29	28.67
3	0.63	1.76	21.83	6.01	24.32	30.23
4	0.62	1.57	21.92	6.02	10.80	30.13
5	0.62	1.56	24.24	2.02	24.21	28.44
6	0.62	1.55	21.94	6.01	24.28	30.12
7	0.63	1.55	21.85	6.01	24.28	30.04
8	0.64	1.74	21.73	6.01	11.25	30.12
9	0.62	1.57	21.90	6.01	24.28	30.10
10	0.62	1.74	21.72	6.02	10.76	30.10
avg	0.63	1.65	22.81	4.66	18.89	29.75

Host comparison — §8 (Linux host) vs §11 (Windows host), Win guest static:

Metric	§8 Linux host	§11 Windows host	Δ (Win vs Lin)
Clone	0.40 s	0.63 s	+58 % (NTFS linked clone)
Start	1.88 s	1.65 s	−12 %
IP acquire	21.77 s	22.81 s	+5 % (≈ same)
Ready	0.00 s	4.66 s	+4.7 s
Destroy	10.40 s	18.89 s	+82 % (bimodal ~11/24 s)
Boot total	24.05 s	29.75 s	+24 %

Key findings:

• The static-IP floor (~21.8 s) is host-independent — exactly as the plan predicted. It is set by Windows boot + Task Scheduler + NIC reconfig inside the guest, identical across both hosts (Win 22.81 s vs Lin 21.77 s, +5 %).
• Ready ≠ 0 on the Windows host (avg 4.66 s, mostly ~6 s) where it was 0 on Linux. After ci-static-ip.ps1 rewrites the NIC, WinRM/5986 needs a few seconds to re-bind on the new address before the TCP probe succeeds; on the Linux host WinRM was already listening when ci-ip was written.
• Clone is slower on NTFS (+58 %); destroy is slower and bimodal (~11 s vs ~24 s) — the ~24 s cases hit the 10 s graceful-stop timeout before deleteVM.
• Net boot-total penalty of the Windows host at constant guest+mode: +24 %.

Pre-req gotcha (template parity): the first §11 attempt timed out at 300 s every iteration — guestinfo.ci-ip never appeared, guest stayed on DHCP. Cause: the F:\CI\Templates\WinBuild2025 BaseClean snapshot did not contain the in-guest ci-static-ip agent (added to the Linux-host template during B8/§8, never propagated to the Windows-host copy). Host-side guestinfo injection works, but with no in-guest consumer the static IP is never applied. Fixed by copying the Linux-host template set onto the Windows host (template parity), after which all 10 iterations passed. If §11 ever regresses to 300 s timeouts, confirm C:\CI\ci-static-ip.ps1 + the CI-StaticIp startup task exist inside a clone (see §13 op-notes).

---

12. Linux guest single-job baseline — Windows host (pairs §10)

Recorded 2026-06-07 — Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10, Windows 11 host + VMware Workstation Pro, template LinuxBuild2404 / snapshot BaseClean-Linux, DHCP (guest publishes its lease via guestinfo.ci-ip). Ready = SSH/22 probe. Windows-host counterpart to §10.

Iter	Clone (s)	Start (s)	IP acquire (s)	Ready (s)	Destroy (s)	Boot total (s)
1–10	0.62–0.69	1.37–1.81	11.26–11.42	0.00–0.02	6.02–6.21	13.43–13.83
avg	0.64	1.50	11.36	0.00	6.13	13.51

Host comparison — §10 (Linux host) vs §12 (Windows host), Lin guest DHCP:

Metric	§10 Linux host	§12 Windows host	Δ (Win vs Lin)
Clone	0.22 s	0.64 s	+191 % (NTFS linked clone)
Start	1.14 s	1.50 s	+32 %
IP acquire	7.45 s	11.36 s	+52 %
Ready	0.00 s	0.00 s	≈ same
Destroy	4.94 s	6.13 s	+24 %
Boot total	8.82 s	13.51 s	+53 %

Key findings:

• The Linux guest stays near-deterministic on the Windows host too (IP σ ≈ 0.05 s) — no static-IP agent needed; DHCP+guestinfo is stable.
• The host penalty is larger for the Linux guest (+53 % boot total) than for the Windows guest (+24 %, §11) because the Linux guest's boot is so fast (8.82 s) that fixed Windows-host overheads — NTFS clone (+0.4 s) and the vmnet8/NAT DHCP+guestinfo round-trip (+3.9 s on IP-acquire) — are a larger fraction of a small total. In absolute terms the host adds ~4.7 s either way.
• IP-acquire delta (7.45 → 11.36 s) isolates the host networking effect (Windows vmnet8 NAT + VMware Tools guestinfo path) on an otherwise identical guest.

---

13. Concurrent capacity burn-in on Windows host (pairs §9)

Recorded 2026-06-07 — Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10, Windows 11 host + VMware Workstation Pro, static IP pool 192.168.79.201–204 (added to F:\CI\config.toml [ip_pool] for parity with the Linux-host §9). Repo Simone/burnin-dummy. Windows guests use the static pool; Linux guests skip it (DHCP+guestinfo). Windows-host counterpart to §9.

Template	Snapshot	Build cmd	Jobs PASS	Rounds OK	Round time (PASS rounds)	Round avg (PASS)
WinBuild2025	BaseClean	build.ps1	36 / 40	7 / 10	88–93 s	~91.6 s
LinuxBuild2404	BaseClean-Linux	build.sh	40 / 40	10 / 10	88–112 s	~96.4 s

Result: PARTIAL — Linux 40/40 (OVERALL PASS); Windows 36/40 (OVERALL FAIL, 3 rounds with transient WinRM faults). 76/80 jobs overall.

Host comparison vs §9 (Linux host):

Guest	§9 Linux host	§13 Windows host	Δ round avg
Win	40/40, ~78.6 s	36/40, ~91.6 s (PASS rounds)	+16 % + WinRM instability
Lin	40/40, ~70.2 s	40/40, ~96.4 s	+37 %

Key findings:

• Linux-guest concurrency is stable on the Windows host (40/40, 10/10) just as on the Linux host — SSH transport showed no faults. Round avg +37 % vs §9, consistent with the §12 single-job host penalty.
• Windows-guest concurrency is not robust on the Windows host. Rounds 1–5, 9–10 were clean (~91 s); rounds 6–8 each lost 1–2 jobs to transient WinRM faults under 4× load and ballooned to 269 / 266 / 622 s (30 s connect timeouts + pypsrp retries). Two failure signatures:
  • ConnectTimeoutError to 5986 (WinRM listener briefly unreachable);
  • WSManFault 2150858843 "the shell was not found on the server" (the WinRM shell was recycled mid-build). Failures self-recovered: each failed job released its IP slot, no clones were orphaned, the IP pool returned to all-free, and rounds 9–10 were clean again. So this is bursty WinRM contention, not a monotonic collapse or a leak.
• Likely cause: vCPU oversubscription (4 VMs × 4 vCPU = 16 vCPU) plus the higher host-OS overhead of Windows-host VMware tips concurrent WinRM into timeouts; RAM was not the constraint (13.8 GB free of 63.7 GB during the run). The Linux host on the same hardware sustained 40/40 (§9). Recommendation for sustained 4× Windows-guest concurrency on the Windows host: lower per-VM vCPU, reduce parallelism to 3, or raise WinRM/pypsrp connect timeouts + add a job-level retry. Tracked in TODO.md.

Operational notes (Windows-host benchmark prerequisites) — discovered while running §11–§13; required for any Windows-host orchestration:

1. Template parity — the Windows-host templates must carry the same in-guest agents as the Linux-host templates (ci-static-ip.ps1 + CI-StaticIp startup task for Windows guests; ci-report-ip for Linux). Without the static-IP agent, static-IP jobs hang for the full IP timeout (see §11).
2. Production venv must be current — F:\CI\python\venv predated the ip_pool feature; [ip_pool] in config is silently ignored by a stale venv. Re-run F:\CI\python\venv\Scripts\python.exe -m pip install . (non-editable) after pulling new code, then verify import ci_orchestrator.ip_pool succeeds.
3. Set CI_VENV_PYTHON before invoking Test-CapacityBurnIn.ps1 / Invoke-CIJob.ps1 interactively. The shim's auto-detect branch dereferences $IsWindows, which is undefined under Windows PowerShell 5.1 + StrictMode and throws VariableIsUndefined; setting CI_VENV_PYTHON skips that branch. Production sets it via the runner (CI_PYTHON_LAUNCHER). A code fix to guard the $IsWindows reference is tracked in TODO.md.

---

14. Host × guest × IP-mode matrix (complete)

With §11–§13 the Linux-host / Windows-host comparison is symmetric:

Host \ (Guest, mode)	Win/DHCP	Win/static	Lin/DHCP	burn-in 4×10
Linux (boot/avg)	§7 102 s	§8 24.1 s	§10 8.8 s	§9 Win 78.6 s / Lin 70.2 s
Windows (boot/avg)	§6 60.6 s	§11 29.8 s	§12 13.5 s	§13 Win ~91.6 s / Lin 96.4 s

Read down a column = host effect at constant guest+mode; across a row = guest/IP-mode effect at constant host. Headline: the Windows host adds +24 % (Win-guest static) to +53 % (Lin-guest DHCP) on single-job boot-to-ready, the static-IP floor (~22 s) is host-independent, and Windows-guest WinRM is the only path that loses jobs under sustained 4× concurrency (Linux/SSH is 40/40 on both hosts).

Note on §6 vs §7: §6 (Win host DHCP) 60.6 s is faster than §7 (Lin host DHCP) 102 s, the reverse of every other column. Both are 4-iteration DHCP baselines whose IP-acquire is dominated by high-variance VMware-Tools polling (σ 26–57 s); the 10-iteration static (§8/§11) and Linux (§10/§12) rows are the reliable host-effect signal.