diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 2860d62..6d2f7c3 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -579,3 +579,188 @@ comparison reflects the *as-deployed* path for each guest, not an IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such in-guest agent. + +--- + +## 11. Static IP baseline — WinBuild2025 on Windows host (pairs §8) + +Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200 +-Iterations 10`, **Windows 11 host** + VMware Workstation Pro, template +`WinBuild2025` / snapshot `BaseClean`, `guestinfo.ip-assignment` injected +before start; the in-guest `ci-static-ip.ps1` task applies the IP and writes +back `guestinfo.ci-ip`. IP column = time to `guestinfo.ci-ip` readable +(`-GuestInfoOnly`). Ready = WinRM/5986 TCP probe. This is the Windows-host +counterpart to the Linux-host §8. + +| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) | +| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- | +| 1 | 0.70 | 1.71 | 26.65 | 0.45 | 10.47 | 29.51 | +| 2 | 0.62 | 1.75 | 24.29 | 2.01 | 24.29 | 28.67 | +| 3 | 0.63 | 1.76 | 21.83 | 6.01 | 24.32 | 30.23 | +| 4 | 0.62 | 1.57 | 21.92 | 6.02 | 10.80 | 30.13 | +| 5 | 0.62 | 1.56 | 24.24 | 2.02 | 24.21 | 28.44 | +| 6 | 0.62 | 1.55 | 21.94 | 6.01 | 24.28 | 30.12 | +| 7 | 0.63 | 1.55 | 21.85 | 6.01 | 24.28 | 30.04 | +| 8 | 0.64 | 1.74 | 21.73 | 6.01 | 11.25 | 30.12 | +| 9 | 0.62 | 1.57 | 21.90 | 6.01 | 24.28 | 30.10 | +| 10 | 0.62 | 1.74 | 21.72 | 6.02 | 10.76 | 30.10 | +| **avg** | **0.63** | **1.65** | **22.81** | **4.66** | **18.89** | **29.75** | + +**Host comparison — §8 (Linux host) vs §11 (Windows host), Win guest static:** + +| Metric | §8 Linux host | §11 Windows host | Δ (Win vs Lin) | +| ---------- | ------------- | ---------------- | ------------------------- | +| Clone | 0.40 s | 0.63 s | +58 % (NTFS linked clone) | +| Start | 1.88 s | 1.65 s | −12 % | +| IP acquire | 21.77 s | 22.81 s | +5 % (≈ same) | +| Ready | 0.00 s | 4.66 s | +4.7 s | +| Destroy | 10.40 s | 18.89 s | +82 % (bimodal ~11/24 s) | +| Boot total | 24.05 s | 29.75 s | +24 % | + +**Key findings**: +- The **static-IP floor (~21.8 s) is host-independent** — exactly as the plan + predicted. It is set by Windows boot + Task Scheduler + NIC reconfig inside + the guest, identical across both hosts (Win 22.81 s vs Lin 21.77 s, +5 %). +- **Ready ≠ 0 on the Windows host** (avg 4.66 s, mostly ~6 s) where it was 0 on + Linux. After `ci-static-ip.ps1` rewrites the NIC, WinRM/5986 needs a few + seconds to re-bind on the new address before the TCP probe succeeds; on the + Linux host WinRM was already listening when `ci-ip` was written. +- Clone is slower on NTFS (+58 %); destroy is slower and bimodal (~11 s vs + ~24 s) — the ~24 s cases hit the 10 s graceful-stop timeout before deleteVM. +- Net boot-total penalty of the Windows host at constant guest+mode: **+24 %**. + +**Pre-req gotcha (template parity)**: the first §11 attempt timed out at 300 s +every iteration — `guestinfo.ci-ip` never appeared, guest stayed on DHCP. +Cause: the `F:\CI\Templates\WinBuild2025` `BaseClean` snapshot did **not** +contain the in-guest `ci-static-ip` agent (added to the Linux-host template +during B8/§8, never propagated to the Windows-host copy). Host-side guestinfo +injection works, but with no in-guest consumer the static IP is never applied. +Fixed by copying the Linux-host template set onto the Windows host (template +parity), after which all 10 iterations passed. If §11 ever regresses to 300 s +timeouts, confirm `C:\CI\ci-static-ip.ps1` + the `CI-StaticIp` startup task +exist inside a clone (see [§13 op-notes](#13-concurrent-capacity-burn-in-on-windows-host-pairs-9)). + +--- + +## 12. Linux guest single-job baseline — Windows host (pairs §10) + +Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10`, +**Windows 11 host** + VMware Workstation Pro, template `LinuxBuild2404` / +snapshot `BaseClean-Linux`, **DHCP** (guest publishes its lease via +`guestinfo.ci-ip`). Ready = SSH/22 probe. Windows-host counterpart to §10. + +| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) | +| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- | +| 1–10 | 0.62–0.69 | 1.37–1.81 | 11.26–11.42 | 0.00–0.02 | 6.02–6.21 | 13.43–13.83 | +| **avg** | **0.64** | **1.50** | **11.36** | **0.00** | **6.13** | **13.51** | + +**Host comparison — §10 (Linux host) vs §12 (Windows host), Lin guest DHCP:** + +| Metric | §10 Linux host | §12 Windows host | Δ (Win vs Lin) | +| ---------- | -------------- | ---------------- | -------------------------- | +| Clone | 0.22 s | 0.64 s | +191 % (NTFS linked clone) | +| Start | 1.14 s | 1.50 s | +32 % | +| IP acquire | 7.45 s | 11.36 s | +52 % | +| Ready | 0.00 s | 0.00 s | ≈ same | +| Destroy | 4.94 s | 6.13 s | +24 % | +| Boot total | 8.82 s | 13.51 s | **+53 %** | + +**Key findings**: +- The Linux guest stays near-deterministic on the Windows host too + (IP σ ≈ 0.05 s) — no static-IP agent needed; DHCP+guestinfo is stable. +- The **host penalty is larger for the Linux guest (+53 % boot total) than for + the Windows guest (+24 %, §11)** because the Linux guest's boot is so fast + (8.82 s) that fixed Windows-host overheads — NTFS clone (+0.4 s) and the + vmnet8/NAT DHCP+guestinfo round-trip (+3.9 s on IP-acquire) — are a larger + *fraction* of a small total. In absolute terms the host adds ~4.7 s either way. +- IP-acquire delta (7.45 → 11.36 s) isolates the **host networking effect** + (Windows vmnet8 NAT + VMware Tools guestinfo path) on an otherwise identical + guest. + +--- + +## 13. Concurrent capacity burn-in on Windows host (pairs §9) + +Recorded 2026-06-07 — `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10`, +**Windows 11 host** + VMware Workstation Pro, static IP pool +`192.168.79.201–204` (added to `F:\CI\config.toml` `[ip_pool]` for parity with +the Linux-host §9). Repo `Simone/burnin-dummy`. Windows guests use the static +pool; Linux guests skip it (DHCP+guestinfo). Windows-host counterpart to §9. + +| Template | Snapshot | Build cmd | Jobs PASS | Rounds OK | Round time (PASS rounds) | Round avg (PASS) | +| -------------- | ----------------- | ----------- | --------- | --------- | ------------------------ | ---------------- | +| WinBuild2025 | `BaseClean` | `build.ps1` | 36 / 40 | 7 / 10 | 88–93 s | ~91.6 s | +| LinuxBuild2404 | `BaseClean-Linux` | `build.sh` | 40 / 40 | 10 / 10 | 88–112 s | ~96.4 s | + +**Result: PARTIAL** — Linux 40/40 (OVERALL PASS); Windows 36/40 (OVERALL FAIL, +3 rounds with transient WinRM faults). 76/80 jobs overall. + +**Host comparison vs §9 (Linux host):** + +| Guest | §9 Linux host | §13 Windows host | Δ round avg | +| ----- | ---------------- | ---------------------------- | ---------------------- | +| Win | 40/40, ~78.6 s | 36/40, ~91.6 s (PASS rounds) | +16 % + WinRM instability | +| Lin | 40/40, ~70.2 s | 40/40, ~96.4 s | +37 % | + +**Key findings**: +- **Linux-guest concurrency is stable on the Windows host** (40/40, 10/10) just + as on the Linux host — SSH transport showed no faults. Round avg +37 % vs §9, + consistent with the §12 single-job host penalty. +- **Windows-guest concurrency is *not* robust on the Windows host.** Rounds 1–5, + 9–10 were clean (~91 s); rounds 6–8 each lost 1–2 jobs to **transient WinRM + faults** under 4× load and ballooned to 269 / 266 / 622 s (30 s connect + timeouts + pypsrp retries). Two failure signatures: + - `ConnectTimeoutError` to 5986 (WinRM listener briefly unreachable); + - `WSManFault 2150858843` "the shell was not found on the server" (the WinRM + shell was recycled mid-build). + Failures **self-recovered**: each failed job released its IP slot, no clones + were orphaned, the IP pool returned to all-free, and rounds 9–10 were clean + again. So this is bursty WinRM contention, not a monotonic collapse or a leak. +- Likely cause: vCPU oversubscription (4 VMs × 4 vCPU = 16 vCPU) plus the higher + host-OS overhead of Windows-host VMware tips concurrent WinRM into timeouts; + RAM was not the constraint (13.8 GB free of 63.7 GB during the run). The Linux + host on the same hardware sustained 40/40 (§9). **Recommendation for sustained + 4× Windows-guest concurrency on the Windows host**: lower per-VM vCPU, reduce + parallelism to 3, or raise WinRM/pypsrp connect timeouts + add a job-level + retry. Tracked in `TODO.md`. + +**Operational notes (Windows-host benchmark prerequisites)** — discovered while +running §11–§13; required for any Windows-host orchestration: +1. **Template parity** — the Windows-host templates must carry the same in-guest + agents as the Linux-host templates (`ci-static-ip.ps1` + `CI-StaticIp` + startup task for Windows guests; `ci-report-ip` for Linux). Without the + static-IP agent, static-IP jobs hang for the full IP timeout (see §11). +2. **Production venv must be current** — `F:\CI\python\venv` predated the + `ip_pool` feature; `[ip_pool]` in config is silently ignored by a stale venv. + Re-run `F:\CI\python\venv\Scripts\python.exe -m pip install .` (non-editable) + after pulling new code, then verify `import ci_orchestrator.ip_pool` succeeds. +3. **Set `CI_VENV_PYTHON`** before invoking `Test-CapacityBurnIn.ps1` / + `Invoke-CIJob.ps1` interactively. The shim's auto-detect branch dereferences + `$IsWindows`, which is undefined under Windows PowerShell 5.1 + `StrictMode` + and throws `VariableIsUndefined`; setting `CI_VENV_PYTHON` skips that branch. + Production sets it via the runner (`CI_PYTHON_LAUNCHER`). A code fix to guard + the `$IsWindows` reference is tracked in `TODO.md`. + +--- + +## 14. Host × guest × IP-mode matrix (complete) + +With §11–§13 the Linux-host / Windows-host comparison is symmetric: + +| Host \ (Guest, mode) | Win/DHCP | Win/static | Lin/DHCP | burn-in 4×10 | +| --------------------- | -------- | ---------- | -------- | --------------------------- | +| **Linux** (boot/avg) | §7 102 s | §8 24.1 s | §10 8.8 s | §9 Win 78.6 s / Lin 70.2 s | +| **Windows** (boot/avg)| §6 60.6 s| §11 29.8 s | §12 13.5 s| §13 Win ~91.6 s / Lin 96.4 s | + +Read **down a column** = host effect at constant guest+mode; **across a row** = +guest/IP-mode effect at constant host. Headline: the Windows host adds **+24 %** +(Win-guest static) to **+53 %** (Lin-guest DHCP) on single-job boot-to-ready, +the static-IP floor (~22 s) is host-independent, and Windows-guest WinRM is the +only path that loses jobs under sustained 4× concurrency (Linux/SSH is 40/40 on +both hosts). + +> Note on §6 vs §7: §6 (Win host DHCP) 60.6 s is *faster* than §7 (Lin host +> DHCP) 102 s, the reverse of every other column. Both are 4-iteration DHCP +> baselines whose IP-acquire is dominated by high-variance VMware-Tools polling +> (σ 26–57 s); the 10-iteration static (§8/§11) and Linux (§10/§12) rows are the +> reliable host-effect signal. diff --git a/plans/benchmark-windows-host.md b/plans/benchmark-windows-host.md index e2a91c9..071988d 100644 --- a/plans/benchmark-windows-host.md +++ b/plans/benchmark-windows-host.md @@ -15,21 +15,21 @@ ## 1. Cosa esiste già e cosa manca -| RUNBOOK § | Host | Guest | IP mode | Stato | -| --------- | ----- | -------------- | ------- | ---------- | -| §6 | Win | WinBuild2025 | DHCP | ✓ 2026-05-17 | -| §7 | Linux | WinBuild2025 | DHCP | ✓ | -| §8 | Linux | WinBuild2025 | static | ✓ | -| §9 | Linux | Win + Linux | burn-in (static pool) | ✓ | -| §10 | Linux | LinuxBuild2404 | DHCP | ✓ | +| RUNBOOK § | Host | Guest | IP mode | Stato | +| --------- | ----- | -------------- | --------------------- | ------------ | +| §6 | Win | WinBuild2025 | DHCP | ✓ 2026-05-17 | +| §7 | Linux | WinBuild2025 | DHCP | ✓ | +| §8 | Linux | WinBuild2025 | static | ✓ | +| §9 | Linux | Win + Linux | burn-in (static pool) | ✓ | +| §10 | Linux | LinuxBuild2404 | DHCP | ✓ | **Mancanti sul Windows host** (target di questo plan): -| Nuovo § | Host | Guest | IP mode | Pair-with | -| ------- | ---- | -------------- | ------- | --------- | -| §11 | Win | WinBuild2025 | static | §8 | -| §12 | Win | LinuxBuild2404 | DHCP | §10 | -| §13 | Win | Win + Linux | burn-in 4×10 | §9 | +| Nuovo § | Host | Guest | IP mode | Pair-with | +| ------- | ---- | -------------- | ------------ | --------- | +| §11 | Win | WinBuild2025 | static | §8 | +| §12 | Win | LinuxBuild2404 | DHCP | §10 | +| §13 | Win | Win + Linux | burn-in 4×10 | §9 | §6 copre già Win×Win×DHCP. Opzionale: ri-eseguirlo nella stessa sessione per coerenza (stesso disco/fragmentazione del momento). @@ -38,24 +38,23 @@ coerenza (stesso disco/fragmentazione del momento). ## 2. Prerequisiti sul Windows host -- [ ] Boot in Windows; verificare VMware Workstation + `vmrun.exe` in +- [x] Boot in Windows; verificare VMware Workstation + `vmrun.exe` in `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe`. -- [ ] Template fully powered-off con snapshot integri: +- [x] Template fully powered-off con snapshot integri: ```powershell & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws listSnapshots 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx' ``` - Attesi: `BaseClean` e `BaseClean-Linux`. -- [ ] Credenziali in **Windows Credential Manager** (non keyring file): - target `BuildVMGuest` (guest WinRM) e `GiteaPAT` (clone in-VM). - Verifica: `cmdkey /list | findstr /i "BuildVMGuest GiteaPAT"`. -- [ ] Chiave SSH guest Linux presente: `F:\CI\keys\ci_linux` (per §12/§13 + Attesi: `BaseClean` e `BaseClean-Linux`. ✓ (anche `PostInstall` su Win) +- [x] Credenziali in **Windows Credential Manager** (non keyring file): + target `BuildVMGuest` (guest WinRM, user `ci_build`) e `GiteaPAT` + (user `Simone`). Verificato. +- [x] Chiave SSH guest Linux presente: `F:\CI\keys\ci_linux` (per §12/§13 lato LinuxBuild2404). -- [ ] Gitea raggiungibile (`http://10.10.20.11:3100`) e repo - `Simone/burnin-dummy` con `build.ps1` + `build.sh` (per §13). -- [ ] `F:\CI\BuildVMs\` privo di cloni orfani e `F:\CI\ip-pool.json` - (se esiste) pulito — stesso caveat lease stale del Linux host - (RUNBOOK §9 / TODO §2.8). Reset se serve: +- [x] Gitea raggiungibile (`http://10.10.20.11:3100`, HTTP 200) e repo + `Simone/burnin-dummy` con `build.ps1` + `build.sh` (verificato via API). +- [x] `F:\CI\BuildVMs\` privo di cloni orfani e `F:\CI\ip-pool.json` assente + (pulito). Reset se serve: ```powershell Set-Content -Path 'F:\CI\ip-pool.json' -Value '{}' -NoNewline ``` @@ -71,10 +70,13 @@ coerenza (stesso disco/fragmentazione del momento). -StaticIP 192.168.79.200 -Netmask 255.255.255.0 ` -Iterations 10 ``` -- [ ] 10 iter completate, tabella per-fase stampata. -- [ ] Confronto vs §8 (Linux host static): atteso clone/start simili, - IP-acquire dominato da `ci-static-ip.ps1` (~21.8 s floor, indipendente - dall'host). Delta host atteso piccolo sulle fasi non-IP. +- [x] 10 iter completate, tabella per-fase stampata. **Boot total avg 29.75 s.** +- [x] Confronto vs §8 (Linux host static): IP-floor confermato host-independent + (22.81 s vs 21.77 s, +5 %). Penalità host: clone +58 % (NTFS), Ready +4.7 s + (WinRM re-bind dopo NIC reconfig), destroy +82 % (bimodale), boot +24 %. + Vedi RUNBOOK §11. **Nota**: primo tentativo fallito (timeout 300 s/iter) — + template Windows privo dell'agent `ci-static-ip`; risolto copiando i + template dal Linux host (parità template). ## 4. §12 — LinuxBuild2404 DHCP (pair §10) @@ -85,10 +87,10 @@ coerenza (stesso disco/fragmentazione del momento). -GuestOS Linux ` -Iterations 10 ``` -- [ ] 10 iter completate. -- [ ] Confronto vs §10 (Linux host, Lin guest DHCP, boot total 8.82 s): - isola l'**effetto host** sul guest Linux. Se Win host ≫ 8.82 s, - l'overhead è host-side (NTFS / NAT vmnet8 Windows), non guest. +- [x] 10 iter completate. **Boot total avg 13.51 s.** +- [x] Confronto vs §10 (Linux host, 8.82 s): **+53 %** host-side — clone +191 % + (NTFS), IP-acquire +52 % (NAT vmnet8 Windows + guestinfo). Guest Linux resta + deterministico (IP σ ≈ 0.05 s). Vedi RUNBOOK §12. ## 5. §13 — Concurrent burn-in 4×10 (pair §9) @@ -115,9 +117,15 @@ coerenza (stesso disco/fragmentazione del momento). -GuestOS Linux ` -BuildCommand 'bash build.sh' ``` -- [ ] Entrambi 40/40 PASS, 10/10 round. -- [ ] Round avg vs §9 (Linux host: Win ~78.6 s, Lin ~70.2 s): delta = effetto - host sotto concorrenza 4×. +- [~] **Linux 40/40 PASS (10/10 round). Windows 36/40 (7/10 round).** I 4 fail + Windows = fault WinRM transitori (round 6–8: connect-timeout 5986 + + WSManFault "shell not found") sotto 4× concorrenza; auto-recuperati (slot IP + rilasciati, 0 cloni orfani, round 9–10 di nuovo puliti). Linux/SSH 0 fault. +- [x] Round avg vs §9 (Win ~78.6 s, Lin ~70.2 s): Windows host Win ~91.6 s + (round PASS) **+16 %** + instabilità WinRM; Lin 96.4 s **+37 %**. + Vedi RUNBOOK §13. **Pre-req scoperti**: (1) parità template, (2) prod venv + F:\CI\python\venv stale (pre-`ip_pool`) → reinstallato `pip install .`, + (3) `CI_VENV_PYTHON` da settare (bug shim `$IsWindows` su PS 5.1+StrictMode). --- @@ -125,10 +133,9 @@ coerenza (stesso disco/fragmentazione del momento). - Tutti i run appendono a `F:\CI\Logs\benchmark.jsonl` (single-job) e stampano summary (burn-in). -- [ ] Copiare/annotare i risultati e aggiungere a `docs/RUNBOOK.md` le - sezioni **§11–§13** con le stesse tabelle di §8/§9/§10 e una riga - finale di **confronto host Linux-vs-Windows** per ciascuna coppia. -- [ ] Pulizia post-run: `F:\CI\BuildVMs\` vuoto, artifact `burnin-*` +- [x] Risultati aggiunti a `docs/RUNBOOK.md` §11–§13 (+ §14 matrice completa), + con confronto host Linux-vs-Windows per ciascuna coppia. +- [x] Pulizia post-run: `F:\CI\BuildVMs\` vuoto, artifact `burnin-*` rimossi, `F:\CI\ip-pool.json` reset. ---