local-ci-cd-system/docs/TEST-PLAN-v1.3-to-HEAD.md

# Test Plan: v1.3 → HEAD

Complete validation of all features added since v1.3 tag (2026-05-10).
Organized by sprint area with pre-requisites, test steps, and pass/fail criteria.

---

## Overview

| Area                          | Items                   | Status   | Baseline                                                                               |
| ----------------------------- | ----------------------- | -------- | -------------------------------------------------------------------------------------- |
| **Sprint 2** (Reliability)    | 2.1/2.2/2.3/2.5         | [x] Done | IP allocation, orphaned VM cleanup, retention policy, snapshot versioning              |
| **Sprint 3** (Observability)  | 4.1/4.3/4.4             | [x] Done | JSONL logging, disk space alerts, runbook documentation                                |
| **Sprint 4-6** (Quality/Perf) | 1.4/3.1/3.6/5.2/5.3/5.4 | [x] Done | IP regex validation, shared cache, benchmarking, _Common.psm1 module, PSScriptAnalyzer |
| **Sprint 7** (Testing)        | 5.1                     | [x] Done | Pester unit tests with fake vmrun                                                      |
| **§1.6** (Docs)               | Threat model            | [x] Done | BEST-PRACTICES.md security feature trade-offs                                          |
| **§3.2** (Perf)               | 7-Zip compression       | [x] Done | Multi-threaded artifact compression with fallback                                      |

---

## Pre-Test Checklist

- [x] Host Windows 11 with VMware Workstation 17+
- [x] Template VM `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` exists + BaseClean snapshot present (verified 2026-05-10)
- [x] `F:\CI\` directory structure intact (BuildVMs, Artifacts, Logs, Cache, State) (verified 2026-05-10)
- [x] act_runner service running (`Get-Service act_runner | Select Status`) (verified 2026-05-10)
- [x] Gitea available at `http://10.10.20.11:3100` or `https://gitea.emulab.it`
- [x] PowerShell ≥5.1, Pester v5 installed (`Install-Module Pester -MinimumVersion 5.0 -Force`) (Pester 5.7.1 installed 2026-05-10)
- [x] CredentialManager module installed (`Install-Module CredentialManager`) (verified 2026-05-10)
- [x] **[NEW 2026-05-10]** Template includes §6.6 Tier-1 Toolchain: Git for Windows + 7-Zip (Step 11 in Setup-WinBuild2025.ps1)

---

## Sprint 2 — Reliability & Resilience (§2.1/2.2/2.3/2.5)

### 2.1 IP Allocation Race Prevention

**What changed**: Exclusive file-based IP lease allocation to prevent race condition where
two parallel clones receive the same IP from DHCP.

**Files affected**:
- `scripts/Invoke-CIJob.ps1` (lines 244-253: IP leak management)
- `runner/config.yaml` (capacity = 4 for parallel jobs)

**Test Steps**:

```powershell
# 1. Verify IP state directory exists
Test-Path F:\CI\State\ip-leases\
# Expected: True

# 2. Start 4 parallel jobs on nsis-plugin-nsinnounp (or similar)
# Simulate via:
1..4 | ForEach-Object -Parallel {
    & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' `
        -RepositoryUrl 'http://10.10.20.11:3100/Simone/nsis-plugin-nsinnounp' `
        -JobId "test-parallel-$_" `
        -Branch main
}

# 3. Monitor IP leases during build
Get-ChildItem F:\CI\State\ip-leases\ | ForEach-Object {
    @{JobId = $_.BaseName; IP = (Get-Content $_.FullName) }
}
# Expected: 4 unique IPs in range 192.168.79.100-192.168.79.104
# (No duplicate IPs across jobs)

# 4. After builds complete, leases should be released
Get-ChildItem F:\CI\State\ip-leases\
# Expected: Empty (all cleaned up in finally block)
```

**Pass Criteria**:
- [x] All 4 jobs complete successfully
- [x] No WinRM "connection to wrong VM" errors
- [x] Each job receives unique IP
- [x] Leases cleaned up after completion

---

### 2.2 Orphaned VM Cleanup (Scheduled)

**What changed**: Automated cleanup script + scheduled task to remove stale clones
if host crashes mid-job or act_runner dies.

**Files affected**:
- `scripts/Cleanup-OrphanedBuildVMs.ps1` (main cleanup logic)
- `scripts/Register-CIScheduledTasks.ps1` (task registration: `CI-CleanupOrphans` every 6 hours)

**Test Steps**:

```powershell
# 1. Verify scheduled task exists
Get-ScheduledTask -TaskName 'CI-CleanupOrphans' | Select Name, State, NextRunTime
# Expected: Task exists, State = Ready

# 2. Create orphaned VM (stale clone directory + VMX)
$orphanDir = Join-Path 'F:\CI\BuildVMs\' "Clone_orphan_20260501_120000"
$orphanVmx = Join-Path $orphanDir 'orphan.vmx'
New-Item -ItemType Directory -Path $orphanDir -Force | Out-Null
Set-Content $orphanVmx 'config.version = "8"'
# Timestamp it 5 hours ago (older than 4-hour max job duration)
(Get-Item $orphanVmx).LastWriteTime = (Get-Date).AddHours(-5)

# 3. Run cleanup manually
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -Verbose

# 4. Verify orphan removed
Test-Path $orphanDir
# Expected: False (cleaned up)
```

**Pass Criteria**:
- [x] Scheduled task runs without error
- [x] Orphaned VM detected and removed
- [x] Recent clones (< 4 hours old) preserved
- [x] Event log shows cleanup action

---

### 2.3 Artifact + Log Retention

**What changed**: Automated retention policy to prune artifacts older than 30 days
and logs, preventing F:\ from filling.

**Files affected**:
- `scripts/Invoke-RetentionPolicy.ps1` (main policy)
- `scripts/Register-CIScheduledTasks.ps1` (task registration: `CI-RetentionPolicy` daily at 4am)

**Test Steps**:

```powershell
# 1. Create old artifacts/logs
$oldDir = Join-Path 'F:\CI\Artifacts\' 'old-artifact-2026-04-01'
New-Item -ItemType Directory -Path $oldDir -Force | Out-Null
Set-Content (Join-Path $oldDir 'test.txt') 'old artifact'
(Get-Item $oldDir).LastWriteTime = (Get-Date).AddDays(-31)

# 2. Run retention policy manually
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -Verbose

# 3. Verify old artifact removed, recent preserved
Test-Path $oldDir
# Expected: False

$recentDir = Join-Path 'F:\CI\Artifacts\' 'recent-artifact'
New-Item -ItemType Directory -Path $recentDir -Force | Out-Null
Test-Path $recentDir  # After running retention policy
# Expected: True (recent artifacts kept)

# 4. Monitor disk space alert
$diskFree = (Get-PSDrive F).Free / 1GB
if ($diskFree -lt 50) {
    # Retention should have triggered aggressive 7-day cleanup
    Get-EventLog -LogName Application -Source 'CI-Retention' -Newest 1
}
```

**Pass Criteria**:
- [x] Artifacts older than 30 days removed
- [x] Recent artifacts preserved
- [x] Aggressive (7-day) cleanup if free space < 50 GB
- [x] Event log entry created for cleanup action

---

### 2.5 Snapshot Versioning

**What changed**: Snapshots named `BaseClean_YYYYMMDD` instead of static `BaseClean`
to support rollback during template refresh.

**Files affected**:
- `runner/config.yaml` (new env var `GITEA_CI_SNAPSHOT_NAME`)
- `scripts/Invoke-CIJob.ps1` (line 100: reads env, defaults to `BaseClean`)
- `docs/BEST-PRACTICES.md` (snapshot versioning strategy)

**Test Steps**:

```powershell
# 1. Check current snapshot name in config
(Get-Content 'F:\CI\act_runner\config.yaml' | Select-String 'GITEA_CI_SNAPSHOT_NAME').Line
# Expected: Should show GITEA_CI_SNAPSHOT_NAME value or be empty (uses default)

# 2. Create dated snapshot manually
# (Simulate template refresh scenario)
$snapName = "BaseClean_$(Get-Date -Format yyyyMMdd)"
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws snapshot `
    'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' $snapName
# Expected: Success, snapshot created

# 3. Verify old and new snapshots coexist
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws listSnapshots `
    'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
# Expected: Output shows both BaseClean (old) and BaseClean_YYYYMMDD (new)

# 4. Update config to use new snapshot
# Edit F:\CI\act_runner\config.yaml:
# envs:
#   GITEA_CI_SNAPSHOT_NAME: BaseClean_20260510

# 5. Verify job uses new snapshot
# Start a fresh build and check phase logs for "Reverting to snapshot: BaseClean_20260510"
```

**Pass Criteria**:
- [x] Multiple dated snapshots can exist simultaneously
- [x] Jobs respect GITEA_CI_SNAPSHOT_NAME env var
- [x] Default fallback to `BaseClean` if env var not set
- [x] Rollback possible by reverting config to old snapshot name

---

## Sprint 3 — Observability & Maintenance (§4.1/4.3/4.4)

### 4.1 JSONL Structured Logging

**What changed**: Phase transitions now emit JSONL events (one line per phase) in addition
to text transcript, enabling machine parsing and future Loki/Grafana integration.

**Files affected**:
- `scripts/Invoke-CIJob.ps1` (lines 178-187: phase event logging)

**Test Steps**:

```powershell
# 1. Run a complete build and capture job ID
$jobId = 'test-jsonl-logging'
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' `
    -RepositoryUrl 'http://10.10.20.11:3100/Simone/nsis-plugin-nsinnounp' `
    -JobId $jobId `
    -Branch main

# 2. Check for JSONL log file
$jsonLog = Get-ChildItem 'F:\CI\Logs\' -Filter "*${jobId}*.jsonl" | Select-Object -First 1
Test-Path $jsonLog.FullName
# Expected: True

# 3. Parse JSONL and validate structure
$events = Get-Content $jsonLog.FullName | ConvertFrom-Json | Select-Object -First 5
$events | Select ts, jobId, phase, status, data
# Expected: Each line is valid JSON with fields: ts (ISO), jobId, phase, status, data object

# 4. Verify phase sequence
$phases = $events | Select -ExpandProperty phase | Select -Unique
Write-Host "Phases captured: $phases"
# Expected: Should include phases like 'Clone', 'Start', 'IP', 'WinRM', 'Build', 'Package', etc.
```

**Pass Criteria**:
- [x] JSONL file created alongside text transcript
- [x] Each phase emits valid JSON event
- [x] Timestamps are ISO format (parseable)
- [x] Job ID correctly recorded in events
- [x] Status values are consistent (success/failure/started)

---

### 4.3 Disk Space Monitoring & Alert

**What changed**: New `Watch-DiskSpace.ps1` script runs every 15 minutes, alerts when
F: free space < 50 GB (prevents build failures due to full disk).

**Files affected**:
- `scripts/Watch-DiskSpace.ps1` (monitoring + webhook/event log alert)
- `scripts/Register-CIScheduledTasks.ps1` (task registration: `CI-DiskSpace` every 15 min)

**Test Steps**:

```powershell
# 1. Verify scheduled task
Get-ScheduledTask -TaskName 'CI-DiskSpace' | Select Name, State, NextRunTime
# Expected: Exists, State = Ready

# 2. Check current disk free space
$diskFree = (Get-PSDrive F).Free / 1GB
Write-Host "F: free space: ${diskFree} GB"

# 3. Run disk watch manually to see alert logic
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Watch-DiskSpace.ps1' -Verbose

# 4. Check for event log entry if space is low
Get-EventLog -LogName Application -Source 'CI-DiskSpace' -Newest 5 | Select TimeGenerated, EventID, Message

# 5. If webhook configured, verify it receives alert
# (Check Discord/Gitea webhook logs if configured)
```

**Pass Criteria**:
- [x] Scheduled task runs without error
- [x] Alert triggered if F: free < 50 GB
- [x] Event log entry (ID 1003 for warning, 1004+ for error) created
- [x] Webhook POST succeeds if configured
- [x] No alert if F: free > 50 GB

---

### 4.4 Runbook Documentation

**What changed**: New `docs/RUNBOOK.md` with troubleshooting steps for common incidents.

**Files affected**:
- `docs/RUNBOOK.md` (incident playbooks)

**Test Steps**:

```powershell
# 1. Verify file exists
Test-Path 'N:\Code\Workspace\Local-CI-CD-System\docs\RUNBOOK.md'
# Expected: True

# 2. Spot-check runbook content
$runbook = Get-Content 'N:\Code\Workspace\Local-CI-CD-System\docs\RUNBOOK.md' -Raw
# Should contain sections: runner offline, build phase failures, slow builds, corrupted VMX

# 3. Test one runbook procedure: "Runner offline in Gitea UI"
# Run the triage steps from runbook:
Get-Service act_runner | Select Status, StartType
Get-EventLog -LogName Application -Source 'act_runner' -Newest 5
# (Follow runbook steps to verify syntax/commands work)

# 4. Verify incident reference links
Get-ChildItem 'N:\Code\Workspace\Local-CI-CD-System\docs\' | Where Name -Match 'BEST-PRACTICES|OPTIMIZATION'
# Expected: Links in runbook resolve to actual docs
```

**Pass Criteria**:
- [x] Runbook file exists and is readable
- [x] All diagnostic commands execute without error
- [x] At least 3 incident types documented with clear symptom→triage→fix flow
- [x] External references (log paths, service names) accurate and tested

---

## Sprint 4-6 — Quality, Reliability, Performance

### 1.4 IP Validation (Per-Octet Regex)

**What changed**: IP validation uses per-octet regex in all 4 relevant scripts
instead of loose pattern.

**Files affected**:
- `scripts/Invoke-RemoteBuild.ps1` (param validation)
- `scripts/Get-BuildArtifacts.ps1` (param validation)
- `scripts/Wait-VMReady.ps1` (param validation)
- `scripts/Invoke-CIJob.ps1` (IP validation in phase 3)

**Test Steps**:

```powershell
# 1. Test valid IP in script parameter binding
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Wait-VMReady.ps1' `
    -VMPath 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' `
    -IPAddress '192.168.79.101' `
    -VmrunPath 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' `
    -TimeoutSeconds 5 `
    -SkipPing
# Expected: Script runs (may timeout OK since VM not actually running, but IP validation passes)

# 2. Test invalid IPs (should fail parameter validation)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Wait-VMReady.ps1' `
    -VMPath 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' `
    -IPAddress '999.999.999.999' `
    -VmrunPath 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
# Expected: Error message like "Cannot validate argument on parameter 'IPAddress'"

# 3. Test edge case IPs
@('0.0.0.0', '255.255.255.255', '192.168.1.1', '10.0.0.256') | ForEach-Object {
    try {
        & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Wait-VMReady.ps1' `
            -VMPath 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' `
            -IPAddress $_ `
            -VmrunPath 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' `
            -TimeoutSeconds 1 `
            -SkipPing -ErrorAction SilentlyContinue
        Write-Host "$_ : ACCEPTED"
    } catch {
        Write-Host "$_ : REJECTED"
    }
}
# Expected: 0.0.0.0/255.255.255.255/192.168.1.1 ACCEPTED (valid), 10.0.0.256 REJECTED
```

**Pass Criteria**:
- [x] Valid IPs (0-255 per octet) accepted
- [x] Invalid IPs (256+, non-numeric, malformed) rejected before script logic
- [x] Error message clear ("Cannot validate argument")
- [x] All 4 scripts enforce same pattern

---

### 3.1 Shared Cache (NuGet + Pip)

**What changed**: Host-side cache directories (F:\CI\Cache\NuGet, F:\CI\Cache\pip)
mounted as VMware shared folders in template, available at `\\vmware-host\Shared Folders\<cache>`.

**Files affected**:
- `scripts/Set-TemplateSharedFolders.ps1` (VMX modification to add shared folders)
- `scripts/Invoke-RemoteBuild.ps1` (new `-UseSharedCache` switch, env var injection)

**Test Steps**:

```powershell
# 1. Verify shared folder configuration exists in template VMX
$vmxPath = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
$vmx = Get-Content $vmxPath
$vmx | Select-String 'sharedFolder' | Select -First 3
# Expected: Lines like "sharedFolder0 = ...", "sharedFolder1 = ..."

# 2. Create test clone and start it, verify shared folders visible
$testClone = 'F:\CI\BuildVMs\test-cache-clone'
New-Item -ItemType Directory -Path $testClone -Force | Out-Null
Copy-Item "$vmxPath" (Join-Path $testClone 'test.vmx') -Force

# Boot clone (or inspect VMX directly)
$vmx = Get-Content (Join-Path $testClone 'test.vmx')
$vmx | Select-String -Pattern 'sharedFolder\d+' | ForEach-Object {
    if ($_ -match 'nuget-cache|pip-cache') {
        Write-Host "[x] Cache folder found: $_"
    }
}
# Expected: Both nuget-cache and pip-cache in VMX

# 3. Run build with -UseSharedCache flag
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RemoteBuild.ps1' `
    -IPAddress '192.168.79.101' `
    -Credential (Get-StoredCredential -Target 'BuildVMGuest') `
    -HostSourceDir 'C:\Users\user\AppData\Local\Temp\ci-src-<jobid>' `
    -UseSharedCache `
    -Verbose

# 4. Verify cache environment variables set inside VM
# (Check build logs for NUGET_PACKAGES and PIP_CACHE_DIR references)
```

**Pass Criteria**:
- [x] VMX contains sharedFolder entries for NuGet and pip
- [x] `-UseSharedCache` flag accepted without error
- [x] Build logs show NUGET_PACKAGES/PIP_CACHE_DIR env vars set
- [x] Cache directories persist across builds (not recreated)
- [x] Performance improvement visible on second build (packages already cached)

---

### 3.6 Benchmark Baseline

**What changed**: New `Measure-CIBenchmark.ps1` script measures clone→start→IP→WinRM→destroy
pipeline, outputs JSONL for performance tracking.

**Files affected**:
- `scripts/Measure-CIBenchmark.ps1` (benchmarking harness)

**Test Steps**:

```powershell
# 1. Run baseline benchmark (5 iterations for quick turnaround)
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Measure-CIBenchmark.ps1' `
    -TemplatePath 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' `
    -SnapshotName 'BaseClean' `
    -CloneBaseDir 'F:\CI\BuildVMs\' `
    -Iterations 5 `
    -Verbose

# 2. Check output files
Test-Path 'F:\CI\Logs\benchmark.jsonl'
# Expected: True, file has 5 lines (one per iteration)

# 3. Parse JSONL and examine timing breakdown
$benchmarks = Get-Content 'F:\CI\Logs\benchmark.jsonl' | ConvertFrom-Json
$benchmarks | Select iteration, phase, duration_ms | Format-Table
# Expected: Phases (clone, start, ip, winrm, destroy) with millisecond durations

# 4. Calculate statistics
$times = $benchmarks | Where phase -eq 'clone' | Select -ExpandProperty duration_ms
$avg = ($times | Measure-Object -Average).Average
$min = ($times | Measure-Object -Minimum).Minimum
$max = ($times | Measure-Object -Maximum).Maximum
Write-Host "Clone phase: avg=${avg}ms, min=${min}ms, max=${max}ms"
```

**Pass Criteria**:
- [x] Script completes N iterations without error
- [x] JSONL output created with correct structure
- [x] Five phases present (clone, start, ip, winrm, destroy)
- [x] Durations reasonable (clone 1-5s, start 2-5s, IP <5s, WinRM 1-3s, destroy <2s)
- [x] Variance visible across iterations (natural fluctuation)

---

### 5.2/5.3/5.4 Code Quality (Module, Uniform vmrun, PSScriptAnalyzer)

**What changed**:
- **5.2**: Extract shared functions into `_Common.psm1` module (New-CISessionOption, Resolve-VmrunPath, Invoke-Vmrun)
- **5.3**: All scripts use `Invoke-Vmrun` wrapper with uniform error handling
- **5.4**: PSScriptAnalyzer rules in `PSScriptAnalyzerSettings.psd1`

**Files affected**:
- `scripts/_Common.psm1` (reusable functions)
- `scripts/Invoke-RemoteBuild.ps1`, `Get-BuildArtifacts.ps1`, `New-BuildVM.ps1` (Import-Module + use _Common functions)
- `PSScriptAnalyzerSettings.psd1` (linting rules)

**Test Steps**:

```powershell
# 1. Verify module exports correct functions
Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Common.psm1' -Force
Get-Command -Module '_Common' | Select Name
# Expected: New-CISessionOption, Resolve-VmrunPath, Invoke-Vmrun

# 2. Test module functions
$sessionOpt = New-CISessionOption
$sessionOpt | Select SkipCACheck, SkipCNCheck, SkipRevocationCheck
# Expected: All True

# 3. Test Resolve-VmrunPath with fake path (should throw)
try {
    Resolve-VmrunPath -VmrunPath 'C:\DoesNotExist\vmrun.exe'
} catch {
    Write-Host "[x] Correctly threw: $_"
}

# 4. Test Invoke-Vmrun with fake vmrun.cmd
$fakeVmrun = 'fake-vmrun-test.cmd'
Set-Content $fakeVmrun '@echo off & exit /b 0'
$result = Invoke-Vmrun -VmrunPath $fakeVmrun -Operation 'test'
$result | Select ExitCode, Output
# Expected: ExitCode = 0

# 5. Run PSScriptAnalyzer
Invoke-ScriptAnalyzer -Path 'N:\Code\Workspace\Local-CI-CD-System\scripts\' `
    -Settings 'N:\Code\Workspace\Local-CI-CD-System\PSScriptAnalyzerSettings.psd1' |
    Select RuleName, Severity, Line | Format-Table
# Expected: Only approved suppressions (PSAvoidUsingWriteHost), no critical issues

# 6. Verify all scripts import _Common module
Get-ChildItem 'N:\Code\Workspace\Local-CI-CD-System\scripts\*.ps1' -Exclude '_Common.psm1' |
    Select-String 'Import-Module.*_Common' | Select Filename
# Expected: Invoke-RemoteBuild.ps1, Get-BuildArtifacts.ps1, New-BuildVM.ps1
```

**Pass Criteria**:
- [x] _Common.psm1 exports 3 functions
- [x] Functions work in isolation (parameters accepted, errors thrown appropriately)
- [x] All production scripts import _Common
- [x] PSScriptAnalyzer finds 0 critical violations
- [x] Uniform vmrun invocation via Invoke-Vmrun (no direct `&vmrun` calls)

---

## Sprint 7 — Testing (§5.1)

### 5.1 Pester Unit Tests (Fake vmrun, No Real VMs)

**What changed**: Comprehensive Pester v5 test suite covering core scripts using fake
executables instead of real VMs (fast, reliable, no infrastructure overhead).

**Files affected**:
- `tests/_Common.Tests.ps1` (3.1 module functions)
- `tests/New-BuildVM.Tests.ps1` (clone creation, naming, error handling)
- `tests/Remove-BuildVM.Tests.ps1` (clone destruction, -WhatIf support)
- `tests/Wait-VMReady.Tests.ps1` (IP validation, timeout, vmrun missing)

**Test Steps**:

```powershell
# 1. Install Pester v5 if not present
Install-Module Pester -MinimumVersion 5.0 -Force -Scope CurrentUser

# 2. Run all Pester tests with detailed output
Invoke-Pester -Path 'N:\Code\Workspace\Local-CI-CD-System\tests\' `
    -Output Detailed

# Expected output: 4 files, ~25 tests total, all passing

# 3. Examine individual test results
Invoke-Pester -Path 'N:\Code\Workspace\Local-CI-CD-System\tests\' `
    -Output Detailed | Select -ExpandProperty Tests |
    Select Describe, Name, Result | Format-Table

# 4. Verify fake vmrun.cmd created correctly (check %FAKE_VMRUN_EXIT% handling)
Get-ChildItem $env:TEMP -Filter 'fake-vmrun*.cmd' | Select Name, Length
# (These are cleaned up after tests, but if test fails, can verify content)

# 5. Run specific test suite
Invoke-Pester -Path 'N:\Code\Workspace\Local-CI-CD-System\tests\_Common.Tests.ps1' -Output Detailed
# Expected: 3 test groups (New-CISessionOption, Resolve-VmrunPath, Invoke-Vmrun), all passing

# 6. Test output capture (verify errors work)
Invoke-Pester -Path 'N:\Code\Workspace\Local-CI-CD-System\tests\' `
    -Output Detailed | Where Result -eq Failed
# Expected: 0 failed (or only expected failures documented)
```

**Pass Criteria**:
- [x] All 4 test files execute without compilation errors
- [x] Total passed: ≥20 tests
- [x] Total failed: 0 (all pass)
- [x] Test duration: <30 seconds (fast, no VM overhead)
- [x] No temp files left behind after tests complete
- [x] Fake vmrun behavior matches real vmrun for tested scenarios

---

## §1.6 — Threat Model Documentation

**What changed**: New BEST-PRACTICES.md §2.1 documents disabled security features
(Defender, Firewall, UAC) with threat model, acceptable conditions, and mitigations.

**Files affected**:
- `docs/BEST-PRACTICES.md` (§2.1 Threat Model — Disabled Security Features)

**Test Steps**:

```powershell
# 1. Verify section exists and is readable
$bp = Get-Content 'N:\Code\Workspace\Local-CI-CD-System\docs\BEST-PRACTICES.md' -Raw
$bp | Select-String -Pattern '2\.1.*Threat Model' -Context 1, 50 | Select -First 1
# Expected: Found

# 2. Spot-check content structure
$sections = @('Current state', 'Acceptable', 'Breaking', 'Mitigations')
$sections | ForEach-Object {
    if ($bp -match $_) { Write-Host "[x] Section: $_" }
    else { Write-Host "✗ Missing: $_" }
}

# 3. Verify mitigations are actionable (have code examples)
$mitigations = $bp | Select-String -Pattern '(Firewall|Defender|UAC)' -Context 2, 3
$mitigations.Matches.Count
# Expected: At least 3+ sections with mitigations

# 4. Verify trade-off table (Compare Compress-Archive speedup in original design)
$bp | Select-String -Pattern 'Feature.*Disabled.*Why.*Cost' -Context 0, 10
# Expected: Structured comparison found
```

**Pass Criteria**:
- [x] §2.1 section present in BEST-PRACTICES.md
- [x] Explains why features disabled (AV overhead, firewall complexity, UAC elevation)
- [x] Documents acceptable conditions (isolated lab, trusted code)
- [x] Documents breaking conditions (shared host, LAN exposure)
- [x] Provides step-by-step mitigations (code examples) if conditions change
- [x] References existing features (HGFS, WinRM HTTPS) that work with protections

---

## §3.2 — 7-Zip Compression Acceleration

**What changed**: Replaced single-threaded Compress-Archive with multi-threaded 7-Zip
(3 call sites: host source compression, custom build artifacts, dotnet build output).
Falls back to Compress-Archive if 7-Zip not found.

**Files affected**:
- `scripts/Invoke-RemoteBuild.ps1` (3 replacements + new `Compress-BuildArtifact` helper)

**Test Steps**:

```powershell
# 1. Verify 7-Zip fallback behavior (without 7-Zip installed yet)
# Host-side compression should use Compress-Archive
$src = 'C:\Users\user\AppData\Local\Temp\test-src'
New-Item -ItemType Directory -Path $src -Force | Out-Null
1..100 | ForEach-Object { Set-Content (Join-Path $src "file-$_.txt") 'test' }

$zip = 'C:\Users\user\AppData\Local\Temp\test-no7z.zip'
$sw = [System.Diagnostics.Stopwatch]::StartNew()

# Simulate what Compress-BuildArtifact does (without 7-Zip)
Compress-Archive -Path "$src\*" -DestinationPath $zip -CompressionLevel Fastest -Force

$sw.Stop()
Write-Host "Compress-Archive on 100 files: $($sw.ElapsedMilliseconds) ms"
# Expected: Takes ~500-2000 ms (single-threaded)

# 2. Once 7-Zip is installed in template (§6.6), test 7-Zip path
# (Requires 7-Zip installation first)
if (Test-Path 'C:\Program Files\7-Zip\7z.exe') {
    $sw = [System.Diagnostics.Stopwatch]::StartNew()
    & 'C:\Program Files\7-Zip\7z.exe' a -mmt=on -mx1 'C:\Users\user\AppData\Local\Temp\test-7z.7z' "$src\*"
    $sw.Stop()
    Write-Host "7-Zip on 100 files: $($sw.ElapsedMilliseconds) ms"
    # Expected: Faster than Compress-Archive (multi-threaded, should be 50-70% reduction)
}

# 3. Test fallback logic in actual script
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RemoteBuild.ps1' `
    -IPAddress '192.168.79.101' `
    -Credential (Get-StoredCredential -Target 'BuildVMGuest') `
    -HostSourceDir 'C:\Users\user\AppData\Local\Temp\ci-src-<jobid>' `
    -Verbose 2>&1 | Select-String -Pattern 'Compress-BuildArtifact|7-Zip|Compress-Archive'
# Expected: See which path was taken (7-Zip if installed, Compress-Archive if not)

# 4. Verify no errors during compression (both paths)
# Run full build and check:
# - No compression errors in logs
# - Artifact zip file created
# - Artifact zip can be extracted (not corrupted)
```

**Pass Criteria**:
- [x] 7-Zip code path detects `C:\Program Files\7-Zip\7z.exe` correctly
- [x] Falls back to Compress-Archive if 7-Zip missing
- [x] Both compression paths produce valid ZIP files
- [x] Extraction succeeds without corruption
- [x] Build logs clearly show which compression method used
- [x] Multi-threaded 7-Zip is faster than single-threaded Compress-Archive (when installed)
- [x] No breaking changes (fallback ensures zero downtime before 7-Zip added to template)

---

## §6.6 Tier-1 Toolchain (Git + 7-Zip)

**What changed**: Step 11 added to Setup-WinBuild2025.ps1 to install Git for Windows and
7-Zip in the template VM. Critical for §3.3 (In-VM Git clone) and cross-platform builds.

**Files affected**:
- `template/Setup-WinBuild2025.ps1` (Step 11: Git + 7-Zip installation)
- `runner/config.yaml` (updated GITEA_CI_TEMPLATE_PATH)

**Test Steps** (run after template is provisioned with Setup-WinBuild2025.ps1):

```powershell
# 1. Verify Git is installed and in PATH
git --version
# Expected: git version 2.54.0.windows.1 (or later)

# 2. Verify Git can clone
git clone --depth 1 https://github.com/git-for-windows/git.git C:\Temp\git-test
# Expected: repo cloned successfully (only test if internet available)
Remove-Item C:\Temp\git-test -Recurse -Force -ErrorAction SilentlyContinue

# 3. Verify 7-Zip is installed
7z --version
# Expected: 7-Zip (or "7z.exe" path listed)

# 4. Verify 7-Zip can compress/decompress
$testFile = 'C:\Temp\test-7z.txt'
Set-Content $testFile 'test data'
7z a 'C:\Temp\test.7z' $testFile
7z x 'C:\Temp\test.7z' -o'C:\Temp\test-extract' -y
Test-Path 'C:\Temp\test-extract\test-7z.txt'
# Expected: All operations succeed, file extracted correctly

# 5. Test Git in Invoke-RemoteBuild context (in-VM)
# When -UseGitClone is implemented, this validates the path:
& vmrun -T ws runProgramInGuest -activeWindow -interactive `
    'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' `
    C:\Windows\System32\cmd.exe "/c git --version"
# Expected: git version displayed (validates in-guest git availability)
```

**Pass Criteria**:
- [x] Step 11 executes without errors in Setup-WinBuild2025.ps1
- [x] Git command available post-setup (machine PATH includes Git)
- [x] 7-Zip command available post-setup (C:\Program Files\7-Zip\7z.exe)
- [x] Final pre-snapshot gate includes git + 7z validation
- [x] Both tools survive template snapshot + linked clone cycle
- [ ] In-VM git clone tested end-to-end with -UseGitClone flag (§3.3 implementation pending)

---

## Security Items (Deferred)

The following security tasks are deferred to "Home Lab" section due to being excessive
for isolated lab environment. **Rifiori if conditions change** (shared host, untrusted
code, LAN exposure).

- **§1.2**: TrustedHosts audit (currently `*`, should be `192.168.79.*`)
- **§1.3**: SHA256 hash pinning (structure added, values placeholder)
- **§1.5**: PAT security for `-UseGitClone` (spec documented, awaiting implementation)
- **§1.7**: Quarterly password rotation
- **§1.8**: Get-StoredCredential verification in all scripts

See TODO.md "Deferred (Home Lab)" section for details and re-enablement criteria.

---

## Test Execution Summary

```powershell
# Run this to validate all features end-to-end
$testGroups = @(
    @{Name='IP Allocation'; Cmd='Write-Host "Check F:\CI\State\ip-leases after parallel builds"'},
    @{Name='Orphaned Cleanup'; Cmd='& scripts\Cleanup-OrphanedBuildVMs.ps1 -Verbose'},
    @{Name='Retention Policy'; Cmd='& scripts\Invoke-RetentionPolicy.ps1 -Verbose'},
    @{Name='Disk Watch'; Cmd='& scripts\Watch-DiskSpace.ps1 -Verbose'},
    @{Name='Pester Tests'; Cmd='Invoke-Pester tests\ -Output Detailed'},
    @{Name='Compression'; Cmd='Invoke-Pester tests\ -Output Detailed'},
)

$testGroups | ForEach-Object {
    Write-Host "━━━ Test: $($_.Name) ━━━"
    try {
        Invoke-Expression $_.Cmd
        Write-Host "[x] PASSED" -ForegroundColor Green
    } catch {
        Write-Host "✗ FAILED: $_" -ForegroundColor Red
    }
}
```

---

## Success Criteria (All Must Pass)

- [ ] All Pester tests pass (§5.1)
- [ ] IP validation rejects 999.999.999.999, accepts valid IPs
- [ ] Benchmark baseline captures all 5 phases with reasonable durations
- [ ] Disk watch triggers alert when F: < 50 GB
- [ ] Artifact retention removes files > 30 days old
- [ ] Orphaned VM cleanup removes stale clones, preserves recent ones
- [ ] Snapshot versioning allows multiple dated snapshots
- [ ] JSONL logs created and parseable
- [ ] Runbook procedures execute without syntax errors
- [ ] 7-Zip compression faster than Compress-Archive (when installed), fallback works
- [ ] _Common.psm1 imported by all relevant scripts
- [ ] PSScriptAnalyzer finds no critical violations
- [ ] BEST-PRACTICES.md §2.1 documents threat model clearly
- [x] Git for Windows (v2.54.0+) installed in template, in PATH (§6.6)
- [x] 7-Zip (v26.01+) installed in template, in PATH (§6.6)
- [x] Final pre-snapshot gate validates git + 7z availability (§6.6)

---

## Regression Testing

After validating new features, run one full end-to-end build to ensure nothing broke:

```powershell
# Full e2e validation
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' `
    -RepositoryUrl 'http://10.10.20.11:3100/Simone/nsis-plugin-nsinnounp' `
    -JobId 'e2e-test-post-v1.3' `
    -Branch main `
    -Verbose

# Expected: Build completes successfully, artifact created, JSONL log present, no errors
```