local-ci-cd-system/docs/OPTIMIZATION.md

Optimization Strategies

1. Linked Clone Disk Layout

Goal: Minimize clone creation time and I/O contention between concurrent VMs.

Recommended NVMe Partition Layout

NVMe SSD (e.g. 2TB)
├── C:\  — Windows host OS, VMware Workstation installation
├── F:\CI\
│   ├── Templates\      — Template VM VMX + base VMDK (parent snapshot)
│   ├── BuildVMs\       — Ephemeral linked clone VMs (delta VMDKs)
│   ├── Artifacts\      — Collected build artifacts (per job)
│   ├── Cache\          — NuGet / npm cache (see §4)
│   └── RunnerWork\     — act_runner workspace (checkout, step scripts)

Why separate directories matter:

• Template VMDK reads (CoW base) and clone delta writes happen simultaneously
• Keeping them on the same fast NVMe avoids I/O stalls; a separate spinning disk for the clone directory would bottleneck clone creation
• Artifacts and cache dirs have sequential I/O patterns; they can share space

---

2. Snapshot Strategy

Tiered Snapshot Model

Tier 0: Base OS Install (Windows only, no tools)
  └── Snapshot: "OSBase" — rarely updated (yearly or on major Windows updates)

Tier 1: Build Toolchain
  └── Snapshot: "BaseClean" ← CLONE SOURCE (weekly refresh)
       Includes: VS Build Tools 2026, .NET SDK 10.0.203, Python 3.13.3, Git, WinRM config
       Requirement: ALL linked clones must reference this exact snapshot

Refresh Schedule

Snapshot	Refresh Frequency	Trigger
OSBase	Quarterly	Windows cumulative update
BaseClean	Weekly/Monthly	.NET SDK patch, security update, VS update

Note: Windows Server 2025 KMS lease = 180 giorni. Prima della scadenza: boot template su VMnet8 (NAT) → slmgr /ato → spegni → nuovo snapshot BaseClean.

---

3. Parallel Build Capacity

RAM Budget (i9-10900X · 64 GB)

Component	RAM Usage
Windows host OS	~4 GB
VMware Workstation	~0.5 GB
Gitea server	~0.5–1 GB
act_runner service	~100 MB
Each build VM (idle)	~2–3 GB
Each build VM (active MSBuild/Python)	~6–8 GB
Headroom target (20%)	~13 GB

Available for VMs: 64 - 4 - 0.5 - 1 - 0.1 - 13 = ~45 GB
Max VMs at peak:   45 / 8 = ~5.6 → safe limit = 4

Recommendation: capacity: 4 in config.yaml.

CPU Budget (i9-10900X: 10 cores / 20 threads)

• Template VM: 4 vCPU configurati
• Con 4 VM parallele: 4 × 4 = 16 thread, lascia 4 per host OS / Gitea / runner
• build_plugin.py usa get_optimal_thread_count() che rileva i core della VM e divide per il numero di build parallele (fix TRK0002)

VM Configuration (template VMX)

numvcpus = "4"
cpuid.coresPerSocket = "2"
memsize = "6144"          # 6 GB RAM per VM
scsi0.virtualDev = "pvscsi"
ethernet0.virtualDev = "vmxnet3"

---

4. NuGet / Package Cache on Host

Problem: Each build VM does a fresh dotnet restore, re-downloading NuGet packages every time.

Solution: Map a host-side NuGet cache directory as a shared folder into each VM.

Setup (host side)

F:\CI\Cache\NuGet\   ← shared folder on host

VMware Shared Folder Configuration (in template VMX)

Add to WinBuild.vmx before taking the BaseClean snapshot:

sharedFolder0.present = "TRUE"
sharedFolder0.enabled = "TRUE"
sharedFolder0.readAccess = "TRUE"
sharedFolder0.writeAccess = "TRUE"
sharedFolder0.hostPath = "F:\\CI\\Cache\\NuGet"
sharedFolder0.guestName = "nuget-cache"
sharedFolder.maxNum = "1"

Inside the VM, the shared folder appears as \\vmware-host\Shared Folders\nuget-cache.

NuGet Cache Redirect (in Invoke-RemoteBuild.ps1)

Add to the remote build ScriptBlock:

$env:NUGET_PACKAGES = '\\vmware-host\Shared Folders\nuget-cache'
dotnet restore --packages $env:NUGET_PACKAGES

Result: First build per package downloads once; subsequent builds read from the host cache. Cache is shared across all concurrent VMs (NuGet packages are safe for concurrent reads).

---

5. Clone Pre-Warming

For latency-sensitive pipelines, keep N pre-warmed clones ready in "booted and idle" state.

Tradeoff: Saves ~45–90 seconds of startup per job, but clones remain running (consuming RAM) until used.

Implementation sketch:

# Pre-warmer job (separate scheduled task on host)
# Runs between CI jobs to maintain a pool of warm VMs
$poolSize = 2  # Keep 2 VMs warm at all times

# Check current running clones
$runningClones = Get-ChildItem F:\CI\WarmPool\ -Filter *.vmx
if ($runningClones.Count -lt $poolSize) {
    # Create and start new clone in warm pool
    $vmx = .\New-BuildVM.ps1 -TemplatePath $template -CloneBaseDir F:\CI\WarmPool -JobId "warm-$(Get-Random)"
    vmrun -T ws start $vmx nogui
}

Note: Pre-warming is an advanced optimization. Start without it and add only if CI overhead (startup time) is a demonstrated bottleneck.

---

6. Artifact Storage Management

Build artifacts accumulate quickly. Automate cleanup:

# Scheduled task: run daily
# Remove artifact dirs older than 30 days
Get-ChildItem 'F:\CI\Artifacts' -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } |
    Remove-Item -Recurse -Force

# Remove orphaned clone directories (VMs that were not properly deleted)
Get-ChildItem 'F:\CI\BuildVMs' -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-4) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter *.vmx | Select-Object -First 1
        if ($vmx) {
            vmrun -T ws stop $vmx.FullName hard 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force
    }

---

7. NUMA & CPU Affinity (Advanced)

The i9-10900X is a single NUMA node CPU (no NUMA effects). CPU pinning is not needed.

However, you can set CPU affinity per VM group to reduce vCPU migration overhead:

# Optional: pin act_runner process to cores 0-3 (leave build VMs on 4-19)
$runnerPid = (Get-Process act_runner).Id
Start-Process -FilePath 'cmd' -ArgumentList "/c start /affinity 0xF /b /wait powershell" -NoNewWindow

For most workloads, Windows scheduler handles this well — skip affinity unless profiling shows issues.