chore: initial commit local CI/CD system (e2e verified, production-ready)
Sistema CI locale basato su VMware Workstation + Gitea Actions + act_runner. Testato e2e con nsis-plugin-nsinnounp (MSBuild + Python, 4 configurazioni parallele). - scripts/: Invoke-CIJob, Invoke-RemoteBuild, New/Remove-BuildVM, Wait-VMReady, Get-BuildArtifacts - runner/: act_runner config (windows-build label, capacity 4) - gitea/workflows/: build-nsis.yml (template per progetti MSBuild/Python) - template/: script di provisioning template VM (VS BuildTools 2026, .NET SDK 10, Python 3.13) - docs/: ARCHITECTURE, CI-FLOW, OPTIMIZATION, BEST-PRACTICES, Setup-GiteaSSH Verificato: e2e-009 SUCCESS in 02:09, cleanup automatico VM confermato.
This commit is contained in:
Simone
@@ -0,0 +1,197 @@
|
||||
# Optimization Strategies
|
||||
|
||||
## 1. Linked Clone Disk Layout
|
||||
|
||||
**Goal:** Minimize clone creation time and I/O contention between concurrent VMs.
|
||||
|
||||
### Recommended NVMe Partition Layout
|
||||
|
||||
```
|
||||
NVMe SSD (e.g. 2TB)
|
||||
├── C:\ — Windows host OS, VMware Workstation installation
|
||||
├── F:\CI\
|
||||
│ ├── Templates\ — Template VM VMX + base VMDK (parent snapshot)
|
||||
│ ├── BuildVMs\ — Ephemeral linked clone VMs (delta VMDKs)
|
||||
│ ├── Artifacts\ — Collected build artifacts (per job)
|
||||
│ ├── Cache\ — NuGet / npm cache (see §4)
|
||||
│ └── RunnerWork\ — act_runner workspace (checkout, step scripts)
|
||||
```
|
||||
|
||||
**Why separate directories matter:**
|
||||
- Template VMDK reads (CoW base) and clone delta writes happen simultaneously
|
||||
- Keeping them on the same fast NVMe avoids I/O stalls; a separate spinning disk for
|
||||
the clone directory would bottleneck clone creation
|
||||
- Artifacts and cache dirs have sequential I/O patterns; they can share space
|
||||
|
||||
---
|
||||
|
||||
## 2. Snapshot Strategy
|
||||
|
||||
### Tiered Snapshot Model
|
||||
|
||||
```
|
||||
Tier 0: Base OS Install (Windows only, no tools)
|
||||
└── Snapshot: "OSBase" — rarely updated (yearly or on major Windows updates)
|
||||
|
||||
Tier 1: Build Toolchain
|
||||
└── Snapshot: "BaseClean" ← CLONE SOURCE (weekly refresh)
|
||||
Includes: VS Build Tools 2022, .NET SDK, Git, WinRM config
|
||||
Requirement: ALL linked clones must reference this exact snapshot
|
||||
```
|
||||
|
||||
### Refresh Schedule
|
||||
|
||||
| Snapshot | Refresh Frequency | Trigger |
|
||||
|----------|------------------|-------|
|
||||
| `OSBase` | Quarterly | Windows cumulative update |
|
||||
| `BaseClean` | Weekly/Monthly | .NET SDK patch, security update, VS update |
|
||||
|
||||
> **Note:** Windows Server 2025 KMS lease = 180 giorni. Prima della scadenza:
|
||||
> boot template su VMnet8 (NAT) → `slmgr /ato` → spegni → nuovo snapshot `BaseClean`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Parallel Build Capacity
|
||||
|
||||
### RAM Budget (i9-10900X · 64 GB)
|
||||
|
||||
| Component | RAM Usage |
|
||||
|-----------|----------|
|
||||
| Windows host OS | ~4 GB |
|
||||
| VMware Workstation | ~0.5 GB |
|
||||
| Gitea server | ~0.5–1 GB |
|
||||
| act_runner service | ~100 MB |
|
||||
| Each build VM (idle) | ~2–3 GB |
|
||||
| Each build VM (active MSBuild/Python) | ~6–8 GB |
|
||||
| **Headroom target (20%)** | ~13 GB |
|
||||
|
||||
```
|
||||
Available for VMs: 64 - 4 - 0.5 - 1 - 0.1 - 13 = ~45 GB
|
||||
Max VMs at peak: 45 / 8 = ~5.6 → safe limit = 4
|
||||
```
|
||||
|
||||
**Recommendation:** `capacity: 4` in `config.yaml`.
|
||||
|
||||
### CPU Budget (i9-10900X: 10 cores / 20 threads)
|
||||
|
||||
- Template VM: 4 vCPU configurati
|
||||
- Con 4 VM parallele: 4 × 4 = 16 thread, lascia 4 per host OS / Gitea / runner
|
||||
- `build_plugin.py` usa `get_optimal_thread_count()` che rileva i core della VM e divide per il numero di build parallele (fix TRK0002)
|
||||
|
||||
### VM Configuration (template VMX)
|
||||
|
||||
```ini
|
||||
numvcpus = "4"
|
||||
cpuid.coresPerSocket = "2"
|
||||
memsize = "6144" # 6 GB RAM per VM
|
||||
scsi0.virtualDev = "pvscsi"
|
||||
ethernet0.virtualDev = "vmxnet3"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. NuGet / Package Cache on Host
|
||||
|
||||
**Problem:** Each build VM does a fresh `dotnet restore`, re-downloading NuGet packages every time.
|
||||
|
||||
**Solution:** Map a host-side NuGet cache directory as a shared folder into each VM.
|
||||
|
||||
### Setup (host side)
|
||||
|
||||
```
|
||||
F:\CI\Cache\NuGet\ ← shared folder on host
|
||||
```
|
||||
|
||||
### VMware Shared Folder Configuration (in template VMX)
|
||||
|
||||
Add to `WinBuild.vmx` before taking the `BaseClean` snapshot:
|
||||
```ini
|
||||
sharedFolder0.present = "TRUE"
|
||||
sharedFolder0.enabled = "TRUE"
|
||||
sharedFolder0.readAccess = "TRUE"
|
||||
sharedFolder0.writeAccess = "TRUE"
|
||||
sharedFolder0.hostPath = "F:\\CI\\Cache\\NuGet"
|
||||
sharedFolder0.guestName = "nuget-cache"
|
||||
sharedFolder.maxNum = "1"
|
||||
```
|
||||
|
||||
Inside the VM, the shared folder appears as `\\vmware-host\Shared Folders\nuget-cache`.
|
||||
|
||||
### NuGet Cache Redirect (in `Invoke-RemoteBuild.ps1`)
|
||||
|
||||
Add to the remote build ScriptBlock:
|
||||
```powershell
|
||||
$env:NUGET_PACKAGES = '\\vmware-host\Shared Folders\nuget-cache'
|
||||
dotnet restore --packages $env:NUGET_PACKAGES
|
||||
```
|
||||
|
||||
**Result:** First build per package downloads once; subsequent builds read from the host cache.
|
||||
Cache is shared across all concurrent VMs (NuGet packages are safe for concurrent reads).
|
||||
|
||||
---
|
||||
|
||||
## 5. Clone Pre-Warming
|
||||
|
||||
For latency-sensitive pipelines, keep N pre-warmed clones ready in "booted and idle" state.
|
||||
|
||||
**Tradeoff:** Saves ~45–90 seconds of startup per job, but clones remain running (consuming RAM) until used.
|
||||
|
||||
**Implementation sketch:**
|
||||
|
||||
```powershell
|
||||
# Pre-warmer job (separate scheduled task on host)
|
||||
# Runs between CI jobs to maintain a pool of warm VMs
|
||||
$poolSize = 2 # Keep 2 VMs warm at all times
|
||||
|
||||
# Check current running clones
|
||||
$runningClones = Get-ChildItem F:\CI\WarmPool\ -Filter *.vmx
|
||||
if ($runningClones.Count -lt $poolSize) {
|
||||
# Create and start new clone in warm pool
|
||||
$vmx = .\New-BuildVM.ps1 -TemplatePath $template -CloneBaseDir F:\CI\WarmPool -JobId "warm-$(Get-Random)"
|
||||
vmrun -T ws start $vmx nogui
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** Pre-warming is an advanced optimization. Start without it and add
|
||||
> only if CI overhead (startup time) is a demonstrated bottleneck.
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifact Storage Management
|
||||
|
||||
Build artifacts accumulate quickly. Automate cleanup:
|
||||
|
||||
```powershell
|
||||
# Scheduled task: run daily
|
||||
# Remove artifact dirs older than 30 days
|
||||
Get-ChildItem 'F:\CI\Artifacts' -Directory |
|
||||
Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } |
|
||||
Remove-Item -Recurse -Force
|
||||
|
||||
# Remove orphaned clone directories (VMs that were not properly deleted)
|
||||
Get-ChildItem 'F:\CI\BuildVMs' -Directory |
|
||||
Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-4) } |
|
||||
ForEach-Object {
|
||||
$vmx = Get-ChildItem $_.FullName -Filter *.vmx | Select-Object -First 1
|
||||
if ($vmx) {
|
||||
vmrun -T ws stop $vmx.FullName hard 2>$null
|
||||
}
|
||||
Remove-Item $_.FullName -Recurse -Force
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. NUMA & CPU Affinity (Advanced)
|
||||
|
||||
The i9-10900X is a single NUMA node CPU (no NUMA effects). CPU pinning is not needed.
|
||||
|
||||
However, you can set CPU affinity per VM group to reduce vCPU migration overhead:
|
||||
|
||||
```powershell
|
||||
# Optional: pin act_runner process to cores 0-3 (leave build VMs on 4-19)
|
||||
$runnerPid = (Get-Process act_runner).Id
|
||||
Start-Process -FilePath 'cmd' -ArgumentList "/c start /affinity 0xF /b /wait powershell" -NoNewWindow
|
||||
```
|
||||
|
||||
For most workloads, Windows scheduler handles this well — skip affinity unless profiling shows issues.
|
||||
Reference in New Issue
Block a user