Simone
aadab2d49b
§10: single-job Measure-CIBenchmark on LinuxBuild2404 (DHCP, 10 iter) on the Linux host — boot total 8.82s avg, near-deterministic even on DHCP (IP-acquire 7.45s, σ≈0.01). Guest-vs-guest table: Linux guest ~63% faster boot total than the Windows static-IP guest (§8) on the same host. plans/benchmark-windows-host.md: dual-boot plan to collect the symmetric Windows-host numbers (§11 Win static, §12 Linux DHCP, §13 concurrent burn-in) so the host×guest×IP-mode matrix is complete and Linux-vs-Windows is a column-for-column comparison. Includes Windows paths, Credential Manager prereqs, and the BaseClean-Linux snapshot caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
542 lines
23 KiB
Markdown
542 lines
23 KiB
Markdown
# CI System Runbook
|
||
|
||
Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.
|
||
|
||
---
|
||
|
||
## 1. Runner offline in Gitea UI
|
||
|
||
**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline.
|
||
Queued jobs stay pending indefinitely.
|
||
|
||
**Triage**:
|
||
```powershell
|
||
# Check service state
|
||
Get-Service act_runner
|
||
|
||
# Last 50 lines of runner log
|
||
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50
|
||
|
||
# Check registration file is intact
|
||
Test-Path 'F:\CI\act_runner\.runner'
|
||
```
|
||
|
||
**Fix**:
|
||
```powershell
|
||
# Restart the service
|
||
Restart-Service act_runner
|
||
|
||
# Verify it came back online (wait ~10s then check Gitea UI)
|
||
Get-Service act_runner | Select-Object Status, StartType
|
||
|
||
# If service won't start, check NSSM log
|
||
& 'C:\nssm\nssm.exe' status act_runner
|
||
```
|
||
|
||
If the `.runner` registration file is missing or corrupt, re-register:
|
||
```powershell
|
||
cd F:\CI\act_runner
|
||
.\act_runner.exe register --no-interactive `
|
||
--instance http://10.10.20.11:3100 `
|
||
--token <token-from-gitea-admin> `
|
||
--name local-windows-runner `
|
||
--labels "windows-build:host,dotnet:host,msbuild:host"
|
||
```
|
||
|
||
**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`.
|
||
|
||
---
|
||
|
||
## 2. All builds fail in Phase 2 (VM clone / start)
|
||
|
||
**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like:
|
||
- `vmrun clone failed`
|
||
- `vmrun start failed`
|
||
- `Template VMX not found`
|
||
- `Could not detect VM IP address`
|
||
|
||
**Triage**:
|
||
```powershell
|
||
# List all running VMs
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list
|
||
|
||
# Check template VMX exists and is accessible
|
||
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||
|
||
# Check for orphaned clones that may be consuming disk
|
||
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime
|
||
|
||
# Check disk free space
|
||
Get-PSDrive F | Select-Object Name, Free, Used
|
||
|
||
# Check for a stuck vm-start lock from a crashed job
|
||
Test-Path 'F:\CI\State\vm-start.lock'
|
||
```
|
||
|
||
**Fix** — by root cause:
|
||
|
||
*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`.
|
||
|
||
*Parent VMDK locked* (VMware left a lock file after host crash):
|
||
```powershell
|
||
# Stop all VMs
|
||
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
|
||
# Delete lock files
|
||
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue
|
||
```
|
||
|
||
*Snapshot missing* (`BaseClean` was deleted or renamed):
|
||
```powershell
|
||
# List snapshots on template VM
|
||
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name
|
||
```
|
||
|
||
*Disk full* (clone delta files need space):
|
||
```powershell
|
||
# Emergency cleanup — remove all orphaned clones
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
|
||
# Then run retention
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3
|
||
```
|
||
|
||
*Stale vm-start lock* (from a job that crashed without cleanup):
|
||
```powershell
|
||
Remove-Item 'F:\CI\State\vm-start.lock' -Force
|
||
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force
|
||
```
|
||
|
||
**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.
|
||
|
||
---
|
||
|
||
## 3. Builds are slow
|
||
|
||
**Symptom**: jobs that previously completed in ~3 min now take 8+ min.
|
||
Phase durations visible in `F:\CI\Logs\<jobId>\invoke-ci.jsonl`.
|
||
|
||
**Triage**:
|
||
```powershell
|
||
# Check disk free space (below 50 GB = fragmented writes)
|
||
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}
|
||
|
||
# Check active VM CPU usage (Task Manager or:)
|
||
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending
|
||
|
||
# Check VMnet8 NAT adapter status
|
||
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }
|
||
|
||
# Parse JSONL for per-phase durations (requires jq or manual inspection)
|
||
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
|
||
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status
|
||
```
|
||
|
||
**Fix** — by root cause:
|
||
|
||
*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK.
|
||
|
||
*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2.
|
||
|
||
*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running.
|
||
|
||
*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`.
|
||
|
||
**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).
|
||
|
||
---
|
||
|
||
## 4. Template VMX corrupt after host crash
|
||
|
||
**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state.
|
||
|
||
**Triage**:
|
||
```powershell
|
||
# Try starting the template directly in VMware Workstation UI
|
||
# If it reports "configuration file error" or "disk lock", proceed below.
|
||
|
||
# Check for lock files
|
||
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'
|
||
|
||
# Check if backup exists
|
||
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5
|
||
```
|
||
|
||
**Fix**:
|
||
|
||
*Lock files only* (common after hard shutdown):
|
||
```powershell
|
||
# Ensure no VMware processes are running
|
||
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
|
||
# Remove locks
|
||
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
|
||
# Test clone
|
||
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||
```
|
||
|
||
*VMX or VMDK truly corrupt — restore from backup*:
|
||
```powershell
|
||
# Stop all CI activity first
|
||
Stop-Service act_runner
|
||
|
||
# Identify latest backup
|
||
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
|
||
Write-Host "Restoring from: $($latest.FullName)"
|
||
|
||
# Replace template directory
|
||
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
|
||
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse
|
||
|
||
# Restart runner
|
||
Start-Service act_runner
|
||
```
|
||
|
||
*No backup exists*: must re-provision the template from scratch.
|
||
Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare).
|
||
Estimated time: 2-4 hours including Windows Update.
|
||
|
||
**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.
|
||
|
||
---
|
||
|
||
## Quick Reference
|
||
|
||
| Symptom | First command |
|
||
| ------------------- | ------------------------------------------------------------------ |
|
||
| Runner offline | `Get-Service act_runner`, then `Restart-Service act_runner` |
|
||
| Phase 2 clone fails | `Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` |
|
||
| Disk full | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1` |
|
||
| Stale lock | `Remove-Item F:\CI\State\vm-start.lock` |
|
||
| Slow builds | Check `invoke-ci.jsonl` phase timestamps; check disk I/O |
|
||
| Template corrupt | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` |
|
||
| Snapshot missing | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME` |
|
||
| IP collision | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity` |
|
||
|
||
---
|
||
|
||
## 5. Template Refresh Procedure
|
||
|
||
Use this procedure when the template OS needs updated packages, toolchain upgrades, or
|
||
a new snapshot. Run on the **host** with an elevated PowerShell 5.1 session.
|
||
|
||
### 5.1 Pre-flight
|
||
|
||
```powershell
|
||
# Stop the runner so no CI jobs start during the refresh
|
||
Stop-Service act_runner
|
||
|
||
# Verify no clone VMs are running
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
|
||
# Expected: "Total running VMs: 0"
|
||
|
||
# Backup the existing template (keeps last 3 by default)
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates
|
||
```
|
||
|
||
### 5.2 Boot the template
|
||
|
||
**Windows (WinBuild2025)**:
|
||
```powershell
|
||
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
||
```
|
||
|
||
**Linux (LinuxBuild2404)**:
|
||
```powershell
|
||
$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
||
```
|
||
|
||
### 5.3 Apply updates inside the template
|
||
|
||
**Windows** — connect via WinRM or open the VMware console, then run the Prepare
|
||
script from the host:
|
||
```powershell
|
||
$vmxWin = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||
$credTgt = 'BuildVMGuest' # Windows Credential Manager target
|
||
$cred = Get-StoredCredential -Target $credTgt # requires CredentialManager module
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
|
||
-VMXPath $vmxWin `
|
||
-Credential $cred
|
||
```
|
||
|
||
**Linux** — SSH into the template and run the toolchain script:
|
||
```powershell
|
||
$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
||
# Get IP (wait for VMware Tools if needed)
|
||
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
|
||
# Apply updates
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
|
||
-VMXPath $vmxLin `
|
||
-SshKeyPath 'F:\CI\keys\ci_linux'
|
||
```
|
||
|
||
Alternatively, run `Install-CIToolchain-WinBuild2025.ps1` / `Install-CIToolchain-Linux2404.sh`
|
||
manually inside the guest to apply only toolchain changes without the full Prepare script.
|
||
|
||
### 5.4 Shut down and snapshot
|
||
|
||
```powershell
|
||
# Shut down gracefully (wait up to 120 s)
|
||
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' # or Linux vmx
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft
|
||
|
||
# Name: BaseClean_yyyyMMdd (keeps old name for rollback reference)
|
||
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
|
||
Write-Host "Snapshot created: $snapshotName"
|
||
```
|
||
|
||
Confirm no `.vmem` / `.vmsn` files exist before snapshotting (see AGENTS.md item 9):
|
||
```powershell
|
||
Get-ChildItem (Split-Path $vmx) -Filter '*.vmem' # must be empty
|
||
```
|
||
|
||
### 5.5 Validate
|
||
|
||
```powershell
|
||
# Run the validation script
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
|
||
-VMXPath $vmx -SnapshotName $snapshotName
|
||
```
|
||
|
||
For Linux, also run a quick SSH smoke-test from the host:
|
||
```powershell
|
||
Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
|
||
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
|
||
-Command 'gcc --version && cmake --version' -PassThru
|
||
$result.Output
|
||
```
|
||
|
||
### 5.6 Run a smoke workflow
|
||
|
||
Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI.
|
||
Confirm the job uses the new snapshot and completes successfully.
|
||
|
||
### 5.7 Promote the new snapshot
|
||
|
||
Update `GITEA_CI_SNAPSHOT_NAME` in `runner/config.yaml` and redeploy:
|
||
```powershell
|
||
# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
|
||
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'
|
||
|
||
# Deploy config and restart runner
|
||
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
||
'F:\CI\act_runner\config.yaml' -Force
|
||
Restart-Service act_runner
|
||
```
|
||
|
||
### 5.8 Retain old snapshot 7 days, then delete
|
||
|
||
Keep the previous `BaseClean_*` snapshot for 7 days as a rollback point:
|
||
```powershell
|
||
# List existing snapshots
|
||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx
|
||
|
||
# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
|
||
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME
|
||
```
|
||
|
||
### 5.9 Rollback procedure
|
||
|
||
If a smoke-test failure is discovered after promotion:
|
||
```powershell
|
||
# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
|
||
# (or set it back to 'BaseClean' for the permanent base)
|
||
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
||
'F:\CI\act_runner\config.yaml' -Force
|
||
Restart-Service act_runner
|
||
# The prior snapshot is still in the template — jobs will use it immediately.
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Windows host pre-migration baseline (reference for B7)
|
||
|
||
Recorded 2026-05-17 — `Measure-CIBenchmark.ps1 × 4 iterations`, Python
|
||
orchestrator post-Phase-A, Windows 11 + VMware Workstation Pro,
|
||
template `WinBuild2025` / snapshot `BaseClean`.
|
||
|
||
| Iter | Clone (s) | Start (s) | IP acquire (s) | WinRM (s) | Destroy (s) | Boot total (s) |
|
||
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
|
||
| 1 | 0.63 | 1.75 | 66.57 | 0.01 | 4.81 | 68.96 |
|
||
| 2 | 0.63 | 1.89 | 20.21 | 0.01 | 6.39 | 22.74 |
|
||
| 3 | 0.62 | 1.72 | 85.07 | 0.01 | 4.50 | 87.42 |
|
||
| 4 | 0.61 | 1.72 | 60.97 | 0.01 | 4.20 | 63.31 |
|
||
| **avg** | **0.62** | **1.77** | **58.20** | **0.01** | **4.98** | **60.61** |
|
||
|
||
**Key finding**: IP-acquire phase dominates total time and is highly variable
|
||
(20–85 s) due to VMware Tools guest IP detection latency. Clone/Start/WinRM
|
||
are negligible and stable.
|
||
|
||
**B7 comparison guidance** (tolerance ±20%):
|
||
|
||
| Metric | Windows baseline | ±20% range |
|
||
| ---------------- | ---------------- | ----------- |
|
||
| Clone | 0.62 s | 0.50–0.74 s |
|
||
| Start | 1.77 s | 1.42–2.12 s |
|
||
| Destroy | 4.98 s | 3.98–5.98 s |
|
||
| Boot total (avg) | 60.6 s | 48.5–72.7 s |
|
||
|
||
IP-acquire variance on Windows (σ ≈ 26 s) means boot-total comparison
|
||
requires ≥10 samples on Linux to be meaningful. If Linux avg boot total
|
||
exceeds 72.7 s, open an issue in `TODO.md` with per-phase breakdown before
|
||
declaring B7 failed — check whether IP-acquire increased or non-IP phases
|
||
regressed.
|
||
|
||
---
|
||
|
||
## 7. Linux host post-migration baseline (B7 result)
|
||
|
||
Recorded 2026-05-24 — `Measure-CIBenchmark.ps1 × 4 iterations`, Linux Mint
|
||
host + VMware Workstation Pro Linux, template `WinBuild2025` / snapshot
|
||
`BaseClean`. Ready column = WinRM/5986 TCP probe.
|
||
|
||
| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
|
||
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
|
||
| 1 | 0.42 | 1.89 | 53.06 | 0.03 | 4.55 | 55.40 |
|
||
| 2 | 0.40 | 1.89 | 129.76 | 0.00 | 4.96 | 132.05 |
|
||
| 3 | 0.52 | 2.81 | 176.83 | 0.00 | 5.67 | 180.16 |
|
||
| 4 | 0.40 | 1.90 | 39.17 | 0.00 | 4.51 | 41.47 |
|
||
| **avg** | **0.44** | **2.12** | **99.71** | **0.01** | **4.92** | **102.27** |
|
||
|
||
**Phase verdict vs Windows baseline (±20%):**
|
||
|
||
| Metric | Windows | Linux avg | In range? |
|
||
| ------- | ------- | --------- | ---------------------------------- |
|
||
| Clone | 0.62 s | 0.44 s | ✓ (faster) |
|
||
| Start | 1.77 s | 2.12 s | ✓ (at upper edge) |
|
||
| Destroy | 4.98 s | 4.92 s | ✓ |
|
||
| IP avg | 58.2 s | 99.7 s | ✗ outside — IP variance (39–177 s) |
|
||
| Ready | 0.01 s | 0.01 s | ✓ |
|
||
|
||
**Key finding**: Clone/Start/Ready/Destroy within ±20%. IP-acquire dominates
|
||
and is highly variable on Linux host (σ ≈ 57 s, range 39–177 s) — wider than
|
||
Windows (σ ≈ 26 s). This is VMware Tools DHCP/guestinfo reporting latency,
|
||
not a regression in orchestrator logic. With 4 samples the avg is not stable;
|
||
additional runs may close the gap. No non-IP phase regressed.
|
||
|
||
---
|
||
|
||
## 8. Static IP baseline — WinBuild2025 with ip_pool (B8 result)
|
||
|
||
Recorded 2026-05-25 — `Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 -Iterations 4`,
|
||
Linux Mint host, template `WinBuild2025` / snapshot `BaseClean`.
|
||
`guestinfo.ip-assignment` injected into cloned VMX before start;
|
||
`ci-static-ip.ps1` scheduled task applies IP at boot and writes back
|
||
`guestinfo.ci-ip`. IP column = time until `guestinfo.ci-ip` readable via
|
||
`vmrun readVariable` (`-GuestInfoOnly` mode — DHCP fallback disabled).
|
||
Ready column = WinRM/5986 TCP probe after IP known.
|
||
|
||
| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
|
||
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
|
||
| 1 | 0.41 | 1.86 | 21.80 | 0.00 | 9.75 | 24.07 |
|
||
| 2 | 0.40 | 1.88 | 21.74 | 0.00 | 9.81 | 24.02 |
|
||
| 3 | 0.40 | 1.90 | 21.75 | 0.00 | 12.26 | 24.05 |
|
||
| 4 | 0.40 | 1.89 | 21.77 | 0.00 | 9.77 | 24.06 |
|
||
| **avg** | **0.40** | **1.88** | **21.77** | **0.00** | **10.40** | **24.05** |
|
||
|
||
**Three-way comparison — B6 Windows DHCP / B7 Linux DHCP / B8 Linux static IP:**
|
||
|
||
| Metric | B6 Win DHCP avg | B7 Lin DHCP avg | B8 Lin static avg | B8 vs B6 | B8 vs B7 |
|
||
| ---------- | --------------- | --------------- | ----------------- | ---------------- | ---------------- |
|
||
| Clone | 0.62 s | 0.44 s | 0.40 s | −35% | ≈ same |
|
||
| Start | 1.77 s | 2.12 s | 1.88 s | +6% | ≈ same |
|
||
| IP acquire | 58.2 s | 99.7 s | 21.8 s | **−63%** | **−78%** |
|
||
| Ready | 0.01 s | 0.01 s | 0.00 s | ≈ same | ≈ same |
|
||
| Boot total | 60.6 s | 102.3 s | 24.1 s | **−60%** | **−76%** |
|
||
| IP σ | ~26 s | ~57 s | <0.03 s | **deterministic**| **deterministic**|
|
||
|
||
**Key findings**:
|
||
- Static IP beats even the Windows DHCP baseline by 60% on boot total.
|
||
- IP acquire drops from 58 s (Win) / 100 s (Linux) DHCP to a deterministic
|
||
21.8 s — variance eliminated entirely (σ < 0.03 s).
|
||
- Ready = 0 s: WinRM is already listening on the static IP by the time
|
||
`guestinfo.ci-ip` is written — no additional TCP probe wait.
|
||
- `ci-static-ip.ps1` startup latency (~21.8 s) is the new floor; it reflects
|
||
Windows boot + Task Scheduler + NIC reconfiguration time.
|
||
- Clone/Start/Ready unchanged across all three baselines — static IP has no
|
||
side effects on non-IP phases.
|
||
- Destroy is slower than B6/B7 (~10 s vs ~5 s) — likely disk pressure or
|
||
clone state at test time, unrelated to static IP.
|
||
|
||
---
|
||
|
||
## 9. Concurrent capacity burn-in (B7 — 4 × 10)
|
||
|
||
Recorded 2026-06-07 — `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10`,
|
||
Linux Mint host + VMware Workstation Pro Linux, run as `ci-runner`, static IP
|
||
pool (`192.168.79.201–204`). Repo `Simone/burnin-dummy`. Each "round" is the
|
||
wall-clock time for 4 concurrent end-to-end jobs (clone → boot → IP →
|
||
transport → build → artifacts → destroy). This is the first concurrent
|
||
capacity burn-in (the §6–§8 baselines are single-job `Measure-CIBenchmark`).
|
||
|
||
| Template | Snapshot | Build cmd | Jobs PASS | Round time (min–max) | Round avg |
|
||
| ----------------- | ---------------- | --------------- | --------- | -------------------- | --------- |
|
||
| WinBuild2025 | `BaseClean` | `build.ps1` | 40 / 40 | 68–81 s | ~78.6 s |
|
||
| LinuxBuild2404 | `BaseClean-Linux`| `build.sh` | 40 / 40 | 70–71 s | ~70.2 s |
|
||
|
||
**Result: OVERALL PASS** — 80/80 jobs, 20/20 rounds, zero orphaned clones,
|
||
zero leaked IP leases between rounds.
|
||
|
||
**Key findings**:
|
||
- Per-round wall time is near-deterministic (Win σ ≈ 1.3 s, Linux σ ≈ 0.4 s),
|
||
confirming the static-IP pool eliminates the DHCP IP-acquire variance that
|
||
dominated the §6/§7 single-job baselines (σ 26–57 s).
|
||
- 4-way concurrency adds no contention penalty: round time ≈ single-job boot
|
||
total (§8 static ≈ 24 s) plus build + serialized destroy, well within the
|
||
4-IP pool capacity.
|
||
- Linux rounds (~70 s) slightly faster and tighter than Windows (~79 s).
|
||
- IP-pool release-on-completion works under concurrency — pool returned to
|
||
all-free (`null`) after every round.
|
||
|
||
**Operational note (pool not auto-reconciled)**: `ip-pool.json` is *not*
|
||
reconciled against live clones at startup. A job killed mid-flight (SIGKILL,
|
||
crash) leaks its lease permanently; with only 4 IPs for parallelism 4, a
|
||
single leak exhausts the pool and every subsequent round fails fast with
|
||
`IP pool exhausted after 60s`. Recovery: stop all jobs, confirm
|
||
`build-vms/` empty, then reset the pool:
|
||
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -c \
|
||
"import pathlib; pathlib.Path('/var/lib/ci/ip-pool.json').write_text('{}\n')"
|
||
```
|
||
|
||
See `TODO.md` (IP-pool auto-reconciliation) for the proposed fix.
|
||
|
||
---
|
||
|
||
## 10. Linux guest single-job baseline — LinuxBuild2404 (B7 follow-up)
|
||
|
||
Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10`,
|
||
Linux Mint host + VMware Workstation Pro Linux, run as `ci-runner`, template
|
||
`LinuxBuild2404` / snapshot `BaseClean-Linux`, **DHCP** (no static IP — the
|
||
guest reads its DHCP lease and publishes it via `guestinfo.ci-ip`; the
|
||
Windows-only `ci-static-ip.ps1` task does not apply). Ready = SSH/22 probe.
|
||
|
||
| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
|
||
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
|
||
| 1–10 | 0.22 | 1.14–1.16 | 7.44–7.48 | 0.00–0.01 | 4.90–4.99 | 8.80–8.86 |
|
||
| **avg** | **0.22** | **1.14** | **7.45** | **0.00** | **4.94** | **8.82** |
|
||
|
||
**Key finding**: the Linux guest is near-deterministic *even on DHCP*
|
||
(σ ≈ 0.01 s on IP-acquire) — no static-IP injection needed. IP-acquire is
|
||
7.45 s vs Windows-guest 58 s (DHCP, §6) / 21.8 s (static, §8). Boot total
|
||
8.82 s — ~2.7× faster than the Windows static-IP guest.
|
||
|
||
**Guest-vs-guest comparison (Linux host, single-job):**
|
||
|
||
| Metric | Win guest static (§8) | Lin guest DHCP (§10) | Lin advantage |
|
||
| ---------- | --------------------- | -------------------- | ------------- |
|
||
| Clone | 0.40 s | 0.22 s | −45% |
|
||
| Start | 1.88 s | 1.14 s | −39% |
|
||
| IP acquire | 21.77 s | 7.45 s | **−66%** |
|
||
| Ready | 0.00 s | 0.00 s | ≈ same |
|
||
| Destroy | 10.40 s | 4.94 s | −53% |
|
||
| Boot total | 24.05 s | 8.82 s | **−63%** |
|
||
|
||
Caveat: different IP modes (Win static-IP task vs Lin DHCP+guestinfo) — the
|
||
comparison reflects the *as-deployed* path for each guest, not an
|
||
IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated
|
||
by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such
|
||
in-guest agent.
|