test(e2e): Add §3.3 comparison script (host-side vs guest-clone)

Create Test-E2E-Section3.3.ps1 for side-by-side measurement:
- Runs nsis-plugin-nsinnounp with default host-side clone (baseline)
- Runs same build with -UseGitClone flag (target: guest-side git)
- Displays elapsed time, artifact size, performance delta

Usage:
  .\scripts\Test-E2E-Section3.3.ps1               # Full comparison
  .\scripts\Test-E2E-Section3.3.ps1 -SkipBaseline  # Guest-clone only
  .\scripts\Test-E2E-Section3.3.ps1 -SkipGuestClone # Baseline only

Prerequisite: Template VM has Git installed (§6.6 Tier-1 Toolchain).

Also: Defer §3.5 (vCPU/RAM tuning) and §3.4 (pre-warm pool) to home lab
section — not required for initial CI operability.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Simone
2026-05-10 21:00:03 +02:00
parent 021ce9718f
commit 13ecd823dc
3 changed files with 247 additions and 7 deletions
+19 -6
View File
@@ -31,7 +31,7 @@ _Last updated: 2026-05-10 (post Sprint 7: §1.6/§3.2 done, test plan created)_
| [ ] | Gitea Workflow Integration | 4 | 1 | manca solo §P3 generalizzazione workflow | | [ ] | Gitea Workflow Integration | 4 | 1 | manca solo §P3 generalizzazione workflow |
| [x] | §1 Sicurezza & hardening | 3 | 0 | 1.1, 1.4, 1.6 done; 1.2/1.3/1.5/1.7/1.8 deferred (home lab) | | [x] | §1 Sicurezza & hardening | 3 | 0 | 1.1, 1.4, 1.6 done; 1.2/1.3/1.5/1.7/1.8 deferred (home lab) |
| [x] | §2 Affidabilità & resilienza | 7 | 0 | tutto done (2.1-2.7) | | [x] | §2 Affidabilità & resilienza | 7 | 0 | tutto done (2.1-2.7) |
| [ ] | §3 Performance & throughput | 4 | 1 | 3.1/3.2/3.3/3.6 done; 3.5 backlog; 3.4 deferred | | [ ] | §3 Performance & throughput | 4 | 1 | 3.1/3.2/3.3/3.6 done; 3.5/3.4 deferred (home lab optimization) |
| [x] | §4 Osservabilità & manutenzione | 3 | 0 | 4.1/4.3/4.4 done (4.2 Prometheus, 4.5 Dashboard rimossi) | | [x] | §4 Osservabilità & manutenzione | 3 | 0 | 4.1/4.3/4.4 done (4.2 Prometheus, 4.5 Dashboard rimossi) |
| [x] | §5 Qualità codice & test | 5 | 0 | 5.1/5.2/5.3/5.4/5.5 done | | [x] | §5 Qualità codice & test | 5 | 0 | 5.1/5.2/5.3/5.4/5.5 done |
| [ ] | §6 Scalabilità & estensioni | 1 | 5 | 6.6 (Git+7-Zip) done; 6.1 (Linux), 6.2-6.5 (estensioni) TODO | | [ ] | §6 Scalabilità & estensioni | 1 | 5 | 6.6 (Git+7-Zip) done; 6.1 (Linux), 6.2-6.5 (estensioni) TODO |
@@ -40,12 +40,12 @@ _Last updated: 2026-05-10 (post Sprint 7: §1.6/§3.2 done, test plan created)_
| [ ] | Linux Build VM (§6.1) | 0 | 4 | bozza in `docs/LINUX-TEMPLATE-SETUP.md` | | [ ] | Linux Build VM (§6.1) | 0 | 4 | bozza in `docs/LINUX-TEMPLATE-SETUP.md` |
**Prossimi passi consigliati** (vedi anche "Suggerimento di sequencing" in fondo): **Prossimi passi consigliati** (vedi anche "Suggerimento di sequencing" in fondo):
1. **E2E Test §3.3** — eseguire nsis-plugin-nsinnounp con `-UseGitClone` flag (misurare vs default) 1. **E2E Test §3.3** — eseguire `scripts/Test-E2E-Section3.3.ps1` (nsis-plugin-nsinnounp con `-UseGitClone` flag)
2. **Test Plan** — eseguire completo `docs/TEST-PLAN-v1.3-to-HEAD.md` (25+ Pester tests, validazione features) 2. **Test Plan** — eseguire completo `docs/TEST-PLAN-v1.3-to-HEAD.md` (25+ Pester tests, validazione features)
3. **§6.6 expansion** — Toolchain Tier-2 (PowerShell 7, NSIS, CMake, Node.js, WiX, gh, Sysinternals, vcpkg) 3. **§6.6 expansion** — Toolchain Tier-2 (PowerShell 7, NSIS, CMake, Node.js, WiX, gh, Sysinternals, vcpkg)
4. **§3.5** — vCPU/RAM tuning per workload (P2, estendere params a Invoke-CIJob) 4. **Security**Rifiori se condizioni cambiano; vedi sezione "Deferred (Home Lab)"
5. **§3.4** — Pre-warm pool di cloni (P3, deferred — profiling-dependent optimization) 5. **§3.5** — vCPU/RAM tuning per workload (P2, deferred — home lab optimization)
6. **Security**Rifiori se condizioni cambiano; vedi sezione "Deferred (Home Lab)" 6. **§3.4** — Pre-warm pool di cloni (P3, deferred — profiling-dependent optimization)
--- ---
## Setup & Infrastructure ## Setup & Infrastructure
@@ -339,7 +339,7 @@ File: [scripts/Invoke-CIJob.ps1](scripts/Invoke-CIJob.ps1), [scripts/Invoke-Remo
**Beneficio**: Elimina host-zip-transfer overhead (rilevante per repo > 200 MB con submoduli). **Beneficio**: Elimina host-zip-transfer overhead (rilevante per repo > 200 MB con submoduli).
**Prossimo**: E2E test con nsis-plugin-nsinnounp (-UseGitClone flag) per misurare impatto. **Prossimo**: E2E test con nsis-plugin-nsinnounp (-UseGitClone flag) per misurare impatto.
### 3.5 [P2] [ ] vCPU/RAM tuning per workload ### 3.5 [P2] [ ] vCPU/RAM tuning per workload — DEFERRED (home lab)
File: [docs/OPTIMIZATION.md:82-89](docs/OPTIMIZATION.md), VMX template. File: [docs/OPTIMIZATION.md:82-89](docs/OPTIMIZATION.md), VMX template.
`numvcpus=4`, `memsize=6144` è ragionevole ma non ottimo per ogni build. Per `numvcpus=4`, `memsize=6144` è ragionevole ma non ottimo per ogni build. Per
@@ -349,6 +349,8 @@ Considerare:
- VMX pinning: 6 vCPU per build C++ pesanti, 2 vCPU per build .NET piccoli. - VMX pinning: 6 vCPU per build C++ pesanti, 2 vCPU per build .NET piccoli.
- Esporre `-VMCpu` / `-VMMemory` a `Invoke-CIJob.ps1` con override via workflow. - Esporre `-VMCpu` / `-VMMemory` a `Invoke-CIJob.ps1` con override via workflow.
**Deferred**: Non necessario per home lab; priorità bassa fino a che performance non è bottleneck.
### 3.6 [P2] [x] Benchmark baseline — COMPLETATO 2026-05-10 ### 3.6 [P2] [x] Benchmark baseline — COMPLETATO 2026-05-10
File: [scripts/Measure-CIBenchmark.ps1](scripts/Measure-CIBenchmark.ps1). File: [scripts/Measure-CIBenchmark.ps1](scripts/Measure-CIBenchmark.ps1).
@@ -768,6 +770,17 @@ monitora il pool e ricrea cloni dopo uso.
**Rifiori quando**: Profiling mostra che New-BuildVM+Wait-VMReady è bottleneck >30% nel **Rifiori quando**: Profiling mostra che New-BuildVM+Wait-VMReady è bottleneck >30% nel
tempo totale di job. Con build >2 minuti, overhead stimato ~40% (rilevante solo per 1-min builds). tempo totale di job. Con build >2 minuti, overhead stimato ~40% (rilevante solo per 1-min builds).
### 3.5 [P2] [ ] vCPU/RAM tuning per workload
File: [docs/OPTIMIZATION.md](docs/OPTIMIZATION.md), VMX template config.
Esporre parametri `-VMCpu` / `-VMMemory` a `Invoke-CIJob.ps1` per permettere override
da workflow. Attualmente fisso a `numvcpus=4, memsize=6144` — subottimo per build leggeri
(waste) e pesanti (throttle).
**Stato attuale**: Progettato, non implementato. Template supporta la modifica.
**Rifiori quando**: Performance profiling mostra contentione CPU/RAM è bottleneck per la
workload target. Per home lab, 4 vCPU/6 GB è ragionevole compromesso.
--- ---
## Suggerimento di sequencing ## Suggerimento di sequencing
+1 -1
View File
@@ -24,7 +24,7 @@ Organized by sprint area with pre-requisites, test steps, and pass/fail criteria
- [x] Template VM `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` exists + BaseClean snapshot present (verified 2026-05-10) - [x] Template VM `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` exists + BaseClean snapshot present (verified 2026-05-10)
- [x] `F:\CI\` directory structure intact (BuildVMs, Artifacts, Logs, Cache, State) (verified 2026-05-10) - [x] `F:\CI\` directory structure intact (BuildVMs, Artifacts, Logs, Cache, State) (verified 2026-05-10)
- [x] act_runner service running (`Get-Service act_runner | Select Status`) (verified 2026-05-10) - [x] act_runner service running (`Get-Service act_runner | Select Status`) (verified 2026-05-10)
- [ ] Gitea available at `http://10.10.20.11:3100` or `https://gitea.emulab.it` - [x] Gitea available at `http://10.10.20.11:3100` or `https://gitea.emulab.it`
- [x] PowerShell ≥5.1, Pester v5 installed (`Install-Module Pester -MinimumVersion 5.0 -Force`) (Pester 5.7.1 installed 2026-05-10) - [x] PowerShell ≥5.1, Pester v5 installed (`Install-Module Pester -MinimumVersion 5.0 -Force`) (Pester 5.7.1 installed 2026-05-10)
- [x] CredentialManager module installed (`Install-Module CredentialManager`) (verified 2026-05-10) - [x] CredentialManager module installed (`Install-Module CredentialManager`) (verified 2026-05-10)
- [x] **[NEW 2026-05-10]** Template includes §6.6 Tier-1 Toolchain: Git for Windows + 7-Zip (Step 11 in Setup-WinBuild2025.ps1) - [x] **[NEW 2026-05-10]** Template includes §6.6 Tier-1 Toolchain: Git for Windows + 7-Zip (Step 11 in Setup-WinBuild2025.ps1)
+227
View File
@@ -0,0 +1,227 @@
#Requires -Version 5.1
<#
.SYNOPSIS
E2E test for §3.3 In-VM Git Clone feature.
.DESCRIPTION
Runs nsis-plugin-nsinnounp build twice:
1. Without -UseGitClone (baseline: host-side zip-transfer)
2. With -UseGitClone (target: guest-side git clone)
Measures elapsed time, artifact size, and logs comparison.
Prerequisite: Template VM must have Git for Windows installed (§6.6).
.PARAMETER BaselineJobId
Job ID for baseline run (host-side clone).
Default: e2e-baseline-<yyyyMMdd_HHmmss>
.PARAMETER GuestCloneJobId
Job ID for guest-side clone run.
Default: e2e-guestclone-<yyyyMMdd_HHmmss>
.PARAMETER Branch
Branch to build. Default: main
.PARAMETER TemplatePath
Path to template VMX. Default: env:GITEA_CI_TEMPLATE_PATH or
F:\CI\Templates\WinBuild2025\WinBuild2025.vmx
.PARAMETER SkipBaseline
If set, skip baseline run and only run with -UseGitClone.
.PARAMETER SkipGuestClone
If set, skip guest-clone run and only run baseline.
.EXAMPLE
# Full comparison (both runs):
.\Test-E2E-Section3.3.ps1
# Only test guest-clone mode (assume baseline already exists):
.\Test-E2E-Section3.3.ps1 -SkipBaseline
# Only test baseline:
.\Test-E2E-Section3.3.ps1 -SkipGuestClone
#>
[CmdletBinding()]
param(
[string] $BaselineJobId = "e2e-baseline-$(Get-Date -Format 'yyyyMMdd_HHmmss')",
[string] $GuestCloneJobId = "e2e-guestclone-$(Get-Date -Format 'yyyyMMdd_HHmmss')",
[string] $Branch = 'main',
[string] $TemplatePath = $(
if ($env:GITEA_CI_TEMPLATE_PATH) { $env:GITEA_CI_TEMPLATE_PATH }
else { 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' }
),
[switch] $SkipBaseline,
[switch] $SkipGuestClone
)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$scriptDir = Split-Path $MyInvocation.MyCommand.Path -Parent
$orchestrator = Join-Path $scriptDir 'Invoke-CIJob.ps1'
# Common parameters for both runs
$commonParams = @{
Branch = $Branch
TemplatePath = $TemplatePath
Submodules = $true
BuildCommand = 'python build_plugin.py --final --dist-dir dist'
GuestArtifactSource = 'dist'
GuestCredentialTarget = 'BuildVMGuest'
RepoUrl = 'ssh://gitea-ci/Simone/nsis-plugin-nsinnounp.git'
}
function Format-Elapsed {
param([TimeSpan] $ts)
if ($ts.TotalHours -ge 1) {
return $ts.ToString('hh\:mm\:ss')
} else {
return $ts.ToString('mm\:ss\.ff')
}
}
function Run-Test {
param(
[string] $JobId,
[string] $Mode,
[switch] $UseGitClone,
[hashtable] $CommonParams
)
Write-Host ""
Write-Host "────────────────────────────────────────────────────────" -ForegroundColor Cyan
Write-Host " $Mode" -ForegroundColor Cyan
Write-Host "────────────────────────────────────────────────────────" -ForegroundColor Cyan
Write-Host " JobId : $JobId"
Write-Host " Branch : $Branch"
Write-Host " Mode : $(if ($UseGitClone) { 'Guest-side clone (-UseGitClone)' } else { 'Host-side zip-transfer (baseline)' })"
Write-Host ""
$params = $CommonParams.Clone()
$params['JobId'] = $JobId
if ($UseGitClone) {
$params['UseGitClone'] = $true
}
$startTime = Get-Date
$exitCode = 0
try {
& $orchestrator @params
$exitCode = $LASTEXITCODE
}
catch {
Write-Host "[Test] Orchestrator threw: $_" -ForegroundColor Red
$exitCode = 1
}
$elapsed = (Get-Date) - $startTime
if ($exitCode -ne 0) {
Write-Host "[Test] FAIL — exit code $exitCode after $(Format-Elapsed $elapsed)" -ForegroundColor Red
return @{ Success = $false; Elapsed = $elapsed; JobId = $JobId; Mode = $Mode }
}
$artifactDir = "F:\CI\Artifacts\$JobId"
$artifactZip = Join-Path $artifactDir 'artifacts.zip'
if (-not (Test-Path $artifactZip)) {
Write-Host "[Test] FAIL — artifact not found: $artifactZip" -ForegroundColor Red
return @{ Success = $false; Elapsed = $elapsed; JobId = $JobId; Mode = $Mode }
}
$zipSize = [math]::Round((Get-Item $artifactZip).Length / 1MB, 2)
Write-Host "[Test] PASS — $(Format-Elapsed $elapsed)" -ForegroundColor Green
Write-Host " Artifact: $zipSize MB"
Write-Host " Logs : $artifactDir"
return @{ Success = $true; Elapsed = $elapsed; Size = $zipSize; JobId = $JobId; Mode = $Mode }
}
# ────────────────────────────────────────────────────────────────────────────
# Test runs
# ────────────────────────────────────────────────────────────────────────────
Write-Host ""
Write-Host "╔════════════════════════════════════════════════════════╗" -ForegroundColor Yellow
Write-Host "║ §3.3 E2E Test: In-VM Git Clone vs Host-side Zip ║" -ForegroundColor Yellow
Write-Host "╚════════════════════════════════════════════════════════╝" -ForegroundColor Yellow
$baselineResult = $null
$guestCloneResult = $null
if (-not $SkipBaseline) {
$baselineResult = Run-Test -JobId $BaselineJobId -Mode "BASELINE (host-side zip)" -CommonParams $commonParams
}
if (-not $SkipGuestClone) {
$guestCloneResult = Run-Test -JobId $GuestCloneJobId -Mode "GUEST CLONE (-UseGitClone)" -UseGitClone -CommonParams $commonParams
}
# ────────────────────────────────────────────────────────────────────────────
# Summary
# ────────────────────────────────────────────────────────────────────────────
Write-Host ""
Write-Host "════════════════════════════════════════════════════════" -ForegroundColor Cyan
Write-Host " SUMMARY" -ForegroundColor Cyan
Write-Host "════════════════════════════════════════════════════════" -ForegroundColor Cyan
if ($baselineResult -and $guestCloneResult) {
if ($baselineResult.Success -and $guestCloneResult.Success) {
$baselineTime = $baselineResult.Elapsed.TotalSeconds
$guestTime = $guestCloneResult.Elapsed.TotalSeconds
$diff = $baselineTime - $guestTime
$pctChange = [math]::Round(($diff / $baselineTime) * 100, 1)
Write-Host ""
Write-Host "Baseline (host-side): $(Format-Elapsed $baselineResult.Elapsed) / $($baselineResult.Size) MB"
Write-Host "Guest clone (-UseGit): $(Format-Elapsed $guestCloneResult.Elapsed) / $($guestCloneResult.Size) MB"
Write-Host ""
Write-Host "Difference: $([math]::Round($diff, 1)) sec ($pctChange%)" -ForegroundColor $(if ($diff -gt 0) { 'Green' } else { 'Red' })
Write-Host ""
if ($diff -gt 0) {
Write-Host "✓ §3.3 guest-clone is FASTER by $(Format-Elapsed ([TimeSpan]::FromSeconds($diff)))" -ForegroundColor Green
}
else {
Write-Host "⚠ Host-side is faster by $(Format-Elapsed ([TimeSpan]::FromSeconds(-$diff)))" -ForegroundColor Yellow
}
}
else {
Write-Host ""
Write-Host "One or both tests failed. Review logs above." -ForegroundColor Red
Write-Host ""
if ($baselineResult.Success) {
Write-Host " Baseline: ✓ PASS"
}
else {
Write-Host " Baseline: ✗ FAIL"
}
if ($guestCloneResult.Success) {
Write-Host " Guest: ✓ PASS"
}
else {
Write-Host " Guest: ✗ FAIL"
}
}
}
elseif ($baselineResult) {
Write-Host ""
Write-Host "Baseline (host-side): $(if ($baselineResult.Success) { '✓ PASS' } else { '✗ FAIL' })" -ForegroundColor $(if ($baselineResult.Success) { 'Green' } else { 'Red' })
if ($baselineResult.Success) {
Write-Host " Time: $(Format-Elapsed $baselineResult.Elapsed)"
}
}
elseif ($guestCloneResult) {
Write-Host ""
Write-Host "Guest clone (-UseGit): $(if ($guestCloneResult.Success) { '✓ PASS' } else { '✗ FAIL' })" -ForegroundColor $(if ($guestCloneResult.Success) { 'Green' } else { 'Red' })
if ($guestCloneResult.Success) {
Write-Host " Time: $(Format-Elapsed $guestCloneResult.Elapsed)"
}
}
Write-Host ""
Write-Host "Logs: F:\CI\Logs\$BaselineJobId or $GuestCloneJobId" -ForegroundColor Gray
Write-Host "════════════════════════════════════════════════════════" -ForegroundColor Cyan