Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(benchmark): record Windows-host §11–§14 + plan closeout

Browse Source 
§11 Win-guest static (boot 29.75s, +24% vs Linux-host §8; static-IP floor
~22s confirmed host-independent), §12 Lin-guest DHCP (13.51s, +53%), §13
burn-in 4×10 (Win 36/40 — transient WinRM faults under 4× concurrency,
self-recovered; Lin 40/40). §14 adds the full host×guest×IP-mode matrix.
Plan checkboxes filled with results and the template-parity pre-req note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Simone
2026-06-07 14:06:13 +02:00
parent ab0c13007e
commit 66cdd4866f
2 changed files with 231 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/RUNBOOK.md
+185
View File
@@ -579,3 +579,188 @@ comparison reflects the *as-deployed* path for each guest, not an
IP-mode-controlled A/B. The Win guest's static-IP floor (~21.8 s) is dominated
by Windows boot + Task Scheduler + NIC reconfig; the Linux guest needs no such
in-guest agent.
---
## 11. Static IP baseline — WinBuild2025 on Windows host (pairs §8)
Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -StaticIP 192.168.79.200
-Iterations 10`, **Windows 11 host** + VMware Workstation Pro, template
`WinBuild2025` / snapshot `BaseClean`, `guestinfo.ip-assignment` injected
before start; the in-guest `ci-static-ip.ps1` task applies the IP and writes
back `guestinfo.ci-ip`. IP column = time to `guestinfo.ci-ip` readable
(`-GuestInfoOnly`). Ready = WinRM/5986 TCP probe. This is the Windows-host
counterpart to the Linux-host §8.
| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1 | 0.70 | 1.71 | 26.65 | 0.45 | 10.47 | 29.51 |
| 2 | 0.62 | 1.75 | 24.29 | 2.01 | 24.29 | 28.67 |
| 3 | 0.63 | 1.76 | 21.83 | 6.01 | 24.32 | 30.23 |
| 4 | 0.62 | 1.57 | 21.92 | 6.02 | 10.80 | 30.13 |
| 5 | 0.62 | 1.56 | 24.24 | 2.02 | 24.21 | 28.44 |
| 6 | 0.62 | 1.55 | 21.94 | 6.01 | 24.28 | 30.12 |
| 7 | 0.63 | 1.55 | 21.85 | 6.01 | 24.28 | 30.04 |
| 8 | 0.64 | 1.74 | 21.73 | 6.01 | 11.25 | 30.12 |
| 9 | 0.62 | 1.57 | 21.90 | 6.01 | 24.28 | 30.10 |
| 10 | 0.62 | 1.74 | 21.72 | 6.02 | 10.76 | 30.10 |
| **avg** | **0.63** | **1.65** | **22.81** | **4.66** | **18.89** | **29.75** |
**Host comparison — §8 (Linux host) vs §11 (Windows host), Win guest static:**
| Metric | §8 Linux host | §11 Windows host | Δ (Win vs Lin) |
| ---------- | ------------- | ---------------- | ------------------------- |
| Clone | 0.40 s | 0.63 s | +58 % (NTFS linked clone) |
| Start | 1.88 s | 1.65 s | −12 % |
| IP acquire | 21.77 s | 22.81 s | +5 % (≈ same) |
| Ready | 0.00 s | 4.66 s | +4.7 s |
| Destroy | 10.40 s | 18.89 s | +82 % (bimodal ~11/24 s) |
| Boot total | 24.05 s | 29.75 s | +24 % |
**Key findings**:
- The **static-IP floor (~21.8 s) is host-independent** — exactly as the plan
predicted. It is set by Windows boot + Task Scheduler + NIC reconfig inside
the guest, identical across both hosts (Win 22.81 s vs Lin 21.77 s, +5 %).
- **Ready ≠ 0 on the Windows host** (avg 4.66 s, mostly ~6 s) where it was 0 on
Linux. After `ci-static-ip.ps1` rewrites the NIC, WinRM/5986 needs a few
seconds to re-bind on the new address before the TCP probe succeeds; on the
Linux host WinRM was already listening when `ci-ip` was written.
- Clone is slower on NTFS (+58 %); destroy is slower and bimodal (~11 s vs
~24 s) — the ~24 s cases hit the 10 s graceful-stop timeout before deleteVM.
- Net boot-total penalty of the Windows host at constant guest+mode: **+24 %**.
**Pre-req gotcha (template parity)**: the first §11 attempt timed out at 300 s
every iteration — `guestinfo.ci-ip` never appeared, guest stayed on DHCP.
Cause: the `F:\CI\Templates\WinBuild2025` `BaseClean` snapshot did **not**
contain the in-guest `ci-static-ip` agent (added to the Linux-host template
during B8/§8, never propagated to the Windows-host copy). Host-side guestinfo
injection works, but with no in-guest consumer the static IP is never applied.
Fixed by copying the Linux-host template set onto the Windows host (template
parity), after which all 10 iterations passed. If §11 ever regresses to 300 s
timeouts, confirm `C:\CI\ci-static-ip.ps1` + the `CI-StaticIp` startup task
exist inside a clone (see [§13 op-notes](#13-concurrent-capacity-burn-in-on-windows-host-pairs-9)).
---
## 12. Linux guest single-job baseline — Windows host (pairs §10)
Recorded 2026-06-07 — `Measure-CIBenchmark.ps1 -GuestOS Linux -Iterations 10`,
**Windows 11 host** + VMware Workstation Pro, template `LinuxBuild2404` /
snapshot `BaseClean-Linux`, **DHCP** (guest publishes its lease via
`guestinfo.ci-ip`). Ready = SSH/22 probe. Windows-host counterpart to §10.
| Iter | Clone (s) | Start (s) | IP acquire (s) | Ready (s) | Destroy (s) | Boot total (s) |
| ------- | --------- | --------- | -------------- | --------- | ----------- | -------------- |
| 1–10 | 0.62–0.69 | 1.37–1.81 | 11.26–11.42 | 0.00–0.02 | 6.02–6.21 | 13.43–13.83 |
| **avg** | **0.64** | **1.50** | **11.36** | **0.00** | **6.13** | **13.51** |
**Host comparison — §10 (Linux host) vs §12 (Windows host), Lin guest DHCP:**
| Metric | §10 Linux host | §12 Windows host | Δ (Win vs Lin) |
| ---------- | -------------- | ---------------- | -------------------------- |
| Clone | 0.22 s | 0.64 s | +191 % (NTFS linked clone) |
| Start | 1.14 s | 1.50 s | +32 % |
| IP acquire | 7.45 s | 11.36 s | +52 % |
| Ready | 0.00 s | 0.00 s | ≈ same |
| Destroy | 4.94 s | 6.13 s | +24 % |
| Boot total | 8.82 s | 13.51 s | **+53 %** |
**Key findings**:
- The Linux guest stays near-deterministic on the Windows host too
(IP σ ≈ 0.05 s) — no static-IP agent needed; DHCP+guestinfo is stable.
- The **host penalty is larger for the Linux guest (+53 % boot total) than for
the Windows guest (+24 %, §11)** because the Linux guest's boot is so fast
(8.82 s) that fixed Windows-host overheads — NTFS clone (+0.4 s) and the
vmnet8/NAT DHCP+guestinfo round-trip (+3.9 s on IP-acquire) — are a larger
*fraction* of a small total. In absolute terms the host adds ~4.7 s either way.
- IP-acquire delta (7.45 → 11.36 s) isolates the **host networking effect**
(Windows vmnet8 NAT + VMware Tools guestinfo path) on an otherwise identical
guest.
---
## 13. Concurrent capacity burn-in on Windows host (pairs §9)
Recorded 2026-06-07 — `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10`,
**Windows 11 host** + VMware Workstation Pro, static IP pool
`192.168.79.201–204` (added to `F:\CI\config.toml` `[ip_pool]` for parity with
the Linux-host §9). Repo `Simone/burnin-dummy`. Windows guests use the static
pool; Linux guests skip it (DHCP+guestinfo). Windows-host counterpart to §9.
| Template | Snapshot | Build cmd | Jobs PASS | Rounds OK | Round time (PASS rounds) | Round avg (PASS) |
| -------------- | ----------------- | ----------- | --------- | --------- | ------------------------ | ---------------- |
| WinBuild2025 | `BaseClean` | `build.ps1` | 36 / 40 | 7 / 10 | 88–93 s | ~91.6 s |
| LinuxBuild2404 | `BaseClean-Linux` | `build.sh` | 40 / 40 | 10 / 10 | 88–112 s | ~96.4 s |
**Result: PARTIAL** — Linux 40/40 (OVERALL PASS); Windows 36/40 (OVERALL FAIL,
3 rounds with transient WinRM faults). 76/80 jobs overall.
**Host comparison vs §9 (Linux host):**
| Guest | §9 Linux host | §13 Windows host | Δ round avg |
| ----- | ---------------- | ---------------------------- | ---------------------- |
| Win | 40/40, ~78.6 s | 36/40, ~91.6 s (PASS rounds) | +16 % + WinRM instability |
| Lin | 40/40, ~70.2 s | 40/40, ~96.4 s | +37 % |
**Key findings**:
- **Linux-guest concurrency is stable on the Windows host** (40/40, 10/10) just
as on the Linux host — SSH transport showed no faults. Round avg +37 % vs §9,
consistent with the §12 single-job host penalty.
- **Windows-guest concurrency is *not* robust on the Windows host.** Rounds 1–5,
9–10 were clean (~91 s); rounds 6–8 each lost 1–2 jobs to **transient WinRM
faults** under 4× load and ballooned to 269 / 266 / 622 s (30 s connect
timeouts + pypsrp retries). Two failure signatures:
- `ConnectTimeoutError` to 5986 (WinRM listener briefly unreachable);
- `WSManFault 2150858843` "the shell was not found on the server" (the WinRM
shell was recycled mid-build).
Failures **self-recovered**: each failed job released its IP slot, no clones
were orphaned, the IP pool returned to all-free, and rounds 9–10 were clean
again. So this is bursty WinRM contention, not a monotonic collapse or a leak.
- Likely cause: vCPU oversubscription (4 VMs × 4 vCPU = 16 vCPU) plus the higher
host-OS overhead of Windows-host VMware tips concurrent WinRM into timeouts;
RAM was not the constraint (13.8 GB free of 63.7 GB during the run). The Linux
host on the same hardware sustained 40/40 (§9). **Recommendation for sustained
4× Windows-guest concurrency on the Windows host**: lower per-VM vCPU, reduce
parallelism to 3, or raise WinRM/pypsrp connect timeouts + add a job-level
retry. Tracked in `TODO.md`.
**Operational notes (Windows-host benchmark prerequisites)** — discovered while
running §11–§13; required for any Windows-host orchestration:
1. **Template parity** — the Windows-host templates must carry the same in-guest
agents as the Linux-host templates (`ci-static-ip.ps1` + `CI-StaticIp`
startup task for Windows guests; `ci-report-ip` for Linux). Without the
static-IP agent, static-IP jobs hang for the full IP timeout (see §11).
2. **Production venv must be current** — `F:\CI\python\venv` predated the
`ip_pool` feature; `[ip_pool]` in config is silently ignored by a stale venv.
Re-run `F:\CI\python\venv\Scripts\python.exe -m pip install .` (non-editable)
after pulling new code, then verify `import ci_orchestrator.ip_pool` succeeds.
3. **Set `CI_VENV_PYTHON`** before invoking `Test-CapacityBurnIn.ps1` /
`Invoke-CIJob.ps1` interactively. The shim's auto-detect branch dereferences
`$IsWindows`, which is undefined under Windows PowerShell 5.1 + `StrictMode`
and throws `VariableIsUndefined`; setting `CI_VENV_PYTHON` skips that branch.
Production sets it via the runner (`CI_PYTHON_LAUNCHER`). A code fix to guard
the `$IsWindows` reference is tracked in `TODO.md`.
---
## 14. Host × guest × IP-mode matrix (complete)
With §11–§13 the Linux-host / Windows-host comparison is symmetric:
| Host \ (Guest, mode) | Win/DHCP | Win/static | Lin/DHCP | burn-in 4×10 |
| --------------------- | -------- | ---------- | -------- | --------------------------- |
| **Linux** (boot/avg) | §7 102 s | §8 24.1 s | §10 8.8 s | §9 Win 78.6 s / Lin 70.2 s |
| **Windows** (boot/avg)| §6 60.6 s| §11 29.8 s | §12 13.5 s| §13 Win ~91.6 s / Lin 96.4 s |
Read **down a column** = host effect at constant guest+mode; **across a row** =
guest/IP-mode effect at constant host. Headline: the Windows host adds **+24 %**
(Win-guest static) to **+53 %** (Lin-guest DHCP) on single-job boot-to-ready,
the static-IP floor (~22 s) is host-independent, and Windows-guest WinRM is the
only path that loses jobs under sustained 4× concurrency (Linux/SSH is 40/40 on
both hosts).
> Note on §6 vs §7: §6 (Win host DHCP) 60.6 s is *faster* than §7 (Lin host
> DHCP) 102 s, the reverse of every other column. Both are 4-iteration DHCP
> baselines whose IP-acquire is dominated by high-variance VMware-Tools polling
> (σ 26–57 s); the 10-iteration static (§8/§11) and Linux (§10/§12) rows are the
> reliable host-effect signal.
plans/benchmark-windows-host.md
+36 -29
View File
@@ -16,7 +16,7 @@
## 1. Cosa esiste già e cosa manca
| RUNBOOK § | Host | Guest | IP mode | Stato |
| --------- | ----- | -------------- | ------- | ---------- |
| --------- | ----- | -------------- | --------------------- | ------------ |
| §6 | Win | WinBuild2025 | DHCP | ✓ 2026-05-17 |
| §7 | Linux | WinBuild2025 | DHCP | ✓ |
| §8 | Linux | WinBuild2025 | static | ✓ |
@@ -26,7 +26,7 @@
**Mancanti sul Windows host** (target di questo plan):
| Nuovo § | Host | Guest | IP mode | Pair-with |
| ------- | ---- | -------------- | ------- | --------- |
| ------- | ---- | -------------- | ------------ | --------- |
| §11 | Win | WinBuild2025 | static | §8 |
| §12 | Win | LinuxBuild2404 | DHCP | §10 |
| §13 | Win | Win + Linux | burn-in 4×10 | §9 |
@@ -38,24 +38,23 @@ coerenza (stesso disco/fragmentazione del momento).
## 2. Prerequisiti sul Windows host
- [ ] Boot in Windows; verificare VMware Workstation + `vmrun.exe` in
- [x] Boot in Windows; verificare VMware Workstation + `vmrun.exe` in
`C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe`.
- [ ] Template fully powered-off con snapshot integri:
- [x] Template fully powered-off con snapshot integri:
```powershell
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws listSnapshots 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
```
Attesi: `BaseClean` e `BaseClean-Linux`.
- [ ] Credenziali in **Windows Credential Manager** (non keyring file):
target `BuildVMGuest` (guest WinRM) e `GiteaPAT` (clone in-VM).
Verifica: `cmdkey /list | findstr /i "BuildVMGuest GiteaPAT"`.
- [ ] Chiave SSH guest Linux presente: `F:\CI\keys\ci_linux` (per §12/§13
Attesi: `BaseClean` e `BaseClean-Linux`. ✓ (anche `PostInstall` su Win)
- [x] Credenziali in **Windows Credential Manager** (non keyring file):
target `BuildVMGuest` (guest WinRM, user `ci_build`) e `GiteaPAT`
(user `Simone`). Verificato.
- [x] Chiave SSH guest Linux presente: `F:\CI\keys\ci_linux` (per §12/§13
lato LinuxBuild2404).
- [ ] Gitea raggiungibile (`http://10.10.20.11:3100`) e repo
`Simone/burnin-dummy` con `build.ps1` + `build.sh` (per §13).
- [ ] `F:\CI\BuildVMs\` privo di cloni orfani e `F:\CI\ip-pool.json`
(se esiste) pulito — stesso caveat lease stale del Linux host
(RUNBOOK §9 / TODO §2.8). Reset se serve:
- [x] Gitea raggiungibile (`http://10.10.20.11:3100`, HTTP 200) e repo
`Simone/burnin-dummy` con `build.ps1` + `build.sh` (verificato via API).
- [x] `F:\CI\BuildVMs\` privo di cloni orfani e `F:\CI\ip-pool.json` assente
(pulito). Reset se serve:
```powershell
Set-Content -Path 'F:\CI\ip-pool.json' -Value '{}' -NoNewline
```
@@ -71,10 +70,13 @@ coerenza (stesso disco/fragmentazione del momento).
-StaticIP 192.168.79.200 -Netmask 255.255.255.0 `
-Iterations 10
```
- [ ] 10 iter completate, tabella per-fase stampata.
- [ ] Confronto vs §8 (Linux host static): atteso clone/start simili,
IP-acquire dominato da `ci-static-ip.ps1` (~21.8 s floor, indipendente
dall'host). Delta host atteso piccolo sulle fasi non-IP.
- [x] 10 iter completate, tabella per-fase stampata. **Boot total avg 29.75 s.**
- [x] Confronto vs §8 (Linux host static): IP-floor confermato host-independent
(22.81 s vs 21.77 s, +5 %). Penalità host: clone +58 % (NTFS), Ready +4.7 s
(WinRM re-bind dopo NIC reconfig), destroy +82 % (bimodale), boot +24 %.
Vedi RUNBOOK §11. **Nota**: primo tentativo fallito (timeout 300 s/iter) —
template Windows privo dell'agent `ci-static-ip`; risolto copiando i
template dal Linux host (parità template).
## 4. §12 — LinuxBuild2404 DHCP (pair §10)
@@ -85,10 +87,10 @@ coerenza (stesso disco/fragmentazione del momento).
-GuestOS Linux `
-Iterations 10
```
- [ ] 10 iter completate.
- [ ] Confronto vs §10 (Linux host, Lin guest DHCP, boot total 8.82 s):
isola l'**effetto host** sul guest Linux. Se Win host ≫ 8.82 s,
l'overhead è host-side (NTFS / NAT vmnet8 Windows), non guest.
- [x] 10 iter completate. **Boot total avg 13.51 s.**
- [x] Confronto vs §10 (Linux host, 8.82 s): **+53 %** host-side — clone +191 %
(NTFS), IP-acquire +52 % (NAT vmnet8 Windows + guestinfo). Guest Linux resta
deterministico (IP σ ≈ 0.05 s). Vedi RUNBOOK §12.
## 5. §13 — Concurrent burn-in 4×10 (pair §9)
@@ -115,9 +117,15 @@ coerenza (stesso disco/fragmentazione del momento).
-GuestOS Linux `
-BuildCommand 'bash build.sh'
```
- [ ] Entrambi 40/40 PASS, 10/10 round.
- [ ] Round avg vs §9 (Linux host: Win ~78.6 s, Lin ~70.2 s): delta = effetto
host sotto concorrenza 4×.
- [~] **Linux 40/40 PASS (10/10 round). Windows 36/40 (7/10 round).** I 4 fail
Windows = fault WinRM transitori (round 6–8: connect-timeout 5986 +
WSManFault "shell not found") sotto 4× concorrenza; auto-recuperati (slot IP
rilasciati, 0 cloni orfani, round 9–10 di nuovo puliti). Linux/SSH 0 fault.
- [x] Round avg vs §9 (Win ~78.6 s, Lin ~70.2 s): Windows host Win ~91.6 s
(round PASS) **+16 %** + instabilità WinRM; Lin 96.4 s **+37 %**.
Vedi RUNBOOK §13. **Pre-req scoperti**: (1) parità template, (2) prod venv
F:\CI\python\venv stale (pre-`ip_pool`) → reinstallato `pip install .`,
(3) `CI_VENV_PYTHON` da settare (bug shim `$IsWindows` su PS 5.1+StrictMode).
---
@@ -125,10 +133,9 @@ coerenza (stesso disco/fragmentazione del momento).
- Tutti i run appendono a `F:\CI\Logs\benchmark.jsonl` (single-job) e
stampano summary (burn-in).
- [ ] Copiare/annotare i risultati e aggiungere a `docs/RUNBOOK.md` le
sezioni **§11–§13** con le stesse tabelle di §8/§9/§10 e una riga
finale di **confronto host Linux-vs-Windows** per ciascuna coppia.
- [ ] Pulizia post-run: `F:\CI\BuildVMs\` vuoto, artifact `burnin-*`
- [x] Risultati aggiunti a `docs/RUNBOOK.md` §11–§13 (+ §14 matrice completa),
con confronto host Linux-vs-Windows per ciascuna coppia.
- [x] Pulizia post-run: `F:\CI\BuildVMs\` vuoto, artifact `burnin-*`
rimossi, `F:\CI\ip-pool.json` reset.
---