chore: initial commit local CI/CD system (e2e verified, production-ready)
Sistema CI locale basato su VMware Workstation + Gitea Actions + act_runner. Testato e2e con nsis-plugin-nsinnounp (MSBuild + Python, 4 configurazioni parallele). - scripts/: Invoke-CIJob, Invoke-RemoteBuild, New/Remove-BuildVM, Wait-VMReady, Get-BuildArtifacts - runner/: act_runner config (windows-build label, capacity 4) - gitea/workflows/: build-nsis.yml (template per progetti MSBuild/Python) - template/: script di provisioning template VM (VS BuildTools 2026, .NET SDK 10, Python 3.13) - docs/: ARCHITECTURE, CI-FLOW, OPTIMIZATION, BEST-PRACTICES, Setup-GiteaSSH Verificato: e2e-009 SUCCESS in 02:09, cleanup automatico VM confermato.
This commit is contained in:
Simone
@@ -0,0 +1,193 @@
|
||||
# Architecture — Local CI/CD System
|
||||
|
||||
## Overview
|
||||
|
||||
A fully local, isolated CI/CD pipeline running on a Windows host using:
|
||||
- **Gitea** as the self-hosted Git server + CI platform
|
||||
- **act_runner** as the job executor (runs on the Windows host)
|
||||
- **VMware Workstation** for ephemeral build VMs (linked clones)
|
||||
- **WinRM / PowerShell Remoting** for communication with build VMs
|
||||
- **MSBuild / dotnet CLI** executing inside the guest VM only
|
||||
|
||||
The runner host **never executes build tools directly**. Its only role is orchestration: creating, driving, and destroying VMs.
|
||||
|
||||
---
|
||||
|
||||
## System Diagram
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ DEVELOPER WORKSTATION │
|
||||
│ │
|
||||
│ git push → Gitea (self-hosted, LAN) │
|
||||
└───────────────────────────┬─────────────────────────────────────────────┘
|
||||
│ HTTP/HTTPS (Gitea Actions API)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ WINDOWS HOST (i9-10900X · 64 GB RAM · NVMe SSD) │
|
||||
│ │
|
||||
│ ┌───────────────────┐ polls job queue ┌──────────────────┐ │
|
||||
│ │ Gitea Server │◄────────────────────────►│ act_runner │ │
|
||||
│ │ (Actions API) │ reports status/logs │ (Windows svc) │ │
|
||||
│ └───────────────────┘ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ calls Invoke-CIJob.ps1 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Orchestrator Scripts (PowerShell) │ │
|
||||
│ │ │ │
|
||||
│ │ New-BuildVM.ps1 ──► vmrun.exe (linked clone from template) │ │
|
||||
│ │ Wait-VMReady.ps1 ─► Test-WSMan poll loop │ │
|
||||
│ │ Invoke-RemoteBuild.ps1 ► PSSession + Invoke-Command │ │
|
||||
│ │ Get-BuildArtifacts.ps1 ► Copy-Item -FromSession │ │
|
||||
│ │ Remove-BuildVM.ps1 ───► vmrun stop + deleteVM │ │
|
||||
│ └───────────────────────────────┬───────────────────────────────────┘ │
|
||||
│ │ vmrun.exe (VMware CLI) │
|
||||
│ ▼ │
|
||||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ VMware Workstation │ │
|
||||
│ │ │ │
|
||||
│ │ Template VM ──snapshot "BaseClean"──┐ │ │
|
||||
│ │ ├── Clone_Job_001.vmx │ │
|
||||
│ │ ├── Clone_Job_002.vmx │ │
|
||||
│ │ └── Clone_Job_003.vmx │ │
|
||||
│ └───────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ WinRM :5985 (192.168.11.0/24) │
|
||||
└──────────────────────────────────┼──────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────────────┼──────────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
|
||||
│ Build VM #1 │ │ Build VM #2 │ │ Build VM #N │
|
||||
│ (ephemeral) │ │ (ephemeral) │ │ (ephemeral) │
|
||||
│ │ │ │ │ │
|
||||
│ VS Build Tools │ │ VS Build Tools │ │ VS Build Tools │
|
||||
│ .NET SDK 8 │ │ .NET SDK 8 │ │ .NET SDK 8 │
|
||||
│ Git │ │ Git │ │ Git │
|
||||
│ WinRM enabled │ │ WinRM enabled │ │ WinRM enabled │
|
||||
│ │ │ │ │ │
|
||||
│ > git clone │ │ > git clone │ │ > git clone │
|
||||
│ > dotnet build │ │ > dotnet build │ │ > dotnet build │
|
||||
│ > artifacts │ │ > artifacts │ │ > artifacts │
|
||||
│ │ │ │ │ │
|
||||
│ [DESTROYED] │ │ [DESTROYED] │ │ [DESTROYED] │
|
||||
└──────────────────┘ └──────────────────┘ └──────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Component Descriptions
|
||||
|
||||
### Gitea (Self-hosted Git + CI)
|
||||
- Hosts Git repositories
|
||||
- Provides Gitea Actions (GitHub Actions-compatible YAML workflow engine)
|
||||
- Manages job queue and reports build status to commits/PRs
|
||||
- Stores build artifacts uploaded by act_runner
|
||||
|
||||
### act_runner (Job Executor)
|
||||
- Runs as a Windows service on the host
|
||||
- Polls Gitea Actions API for pending jobs
|
||||
- Executes `.gitea/workflows/*.yml` steps
|
||||
- **Configured with `container.enabled: false`** — does NOT use Docker
|
||||
- Each step runs as a PowerShell command on the host (orchestration only)
|
||||
- Supports `capacity: N` for concurrent job execution (N parallel VMs)
|
||||
|
||||
### VMware Workstation + vmrun CLI
|
||||
- Manages the template VM and all ephemeral clones
|
||||
- `vmrun.exe` is the command-line interface for all VM lifecycle operations
|
||||
- **Linked clones** are used: each build VM is a delta clone from a frozen template snapshot
|
||||
- Clone creation: ~5–15 seconds (vs 60–120 seconds for full clone)
|
||||
- Disk footprint: ~2–5 GB per clone (vs 40–80 GB for full clone)
|
||||
- Tradeoff: template snapshot must remain intact
|
||||
|
||||
### WinRM / PowerShell Remoting
|
||||
- Default transport for remote build execution inside VMs
|
||||
- Port: **5985** (HTTP, in-lab use); migrate to 5986/HTTPS for hardened setups
|
||||
- Authentication: **Basic** (unencrypted — acceptable for isolated lab)
|
||||
- Used for:
|
||||
- Transferring source: `Compress-Archive` on host → WinRM copy → `Expand-Archive` in guest
|
||||
- Running build commands (`Invoke-Command`)
|
||||
- Packaging artifacts (`Compress-Archive` in guest) and collecting via `Copy-Item -FromSession`
|
||||
|
||||
### Build Toolchain (inside VM only)
|
||||
- Visual Studio Build Tools **2026** (MSBuild 18.5.4 / toolset v145)
|
||||
- .NET SDK **10.0.203**
|
||||
- Python **3.13.3** (per build script personalizzati come `build_plugin.py`)
|
||||
- Git (per repo clone inside VM)
|
||||
- NuGet CLI (optional, dotnet restore covers most cases)
|
||||
|
||||
---
|
||||
|
||||
## Network Topology
|
||||
|
||||
```
|
||||
Host NIC (LAN) → 192.168.1.x → Gitea server (10.10.20.11:3100), developer machines
|
||||
VMware VMnet8 (NAT) → 192.168.79.0/24 → Build VMs (DHCP, internet access via NAT)
|
||||
|
||||
Build VMs:
|
||||
Clone_Job_001 → 192.168.79.131
|
||||
Clone_Job_002 → 192.168.79.132
|
||||
...
|
||||
Clone_Job_N → 192.168.79.1xx
|
||||
|
||||
WinRM port 5985 (HTTP) on each VM IP.
|
||||
VMs have internet access via NAT — required for pip/nuget during build.
|
||||
```
|
||||
|
||||
> **Note:** VMnet8 NAT gives VMs internet access. The host detects the VM IP via
|
||||
> `vmrun getGuestIPAddress` after boot — no static IP assignment needed.
|
||||
|
||||
---
|
||||
|
||||
## VM Lifecycle
|
||||
|
||||
```
|
||||
1. TEMPLATE (static, locked)
|
||||
└─ Snapshot: "BaseClean" ───────────────────────┐
|
||||
│
|
||||
2. CLONE CREATION │
|
||||
vmrun clone <template.vmx> <clone.vmx> linked ──►│
|
||||
Duration: ~5–15 seconds │
|
||||
│
|
||||
3. START │
|
||||
vmrun start <clone.vmx> nogui │
|
||||
Duration: ~20–40 seconds boot │
|
||||
│
|
||||
4. READINESS CHECK │
|
||||
Poll: getState=running → ping → Test-WSMan │
|
||||
Duration: ~30–90 seconds total │
|
||||
│
|
||||
5. BUILD │
|
||||
Copy script → Invoke-Command → collect artifacts │
|
||||
Duration: varies (seconds to minutes) │
|
||||
│
|
||||
6. DESTROY │
|
||||
vmrun stop hard -> vmrun deleteVM → Remove-Item │
|
||||
Duration: ~5–10 seconds │
|
||||
Result: zero disk footprint remaining ▼
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resource Limits & Parallel Capacity
|
||||
|
||||
| Resource | Host Total | Per VM (est.) | Max VMs |
|
||||
| --------------- | ---------------------- | --------------- | ---------------- |
|
||||
| RAM | 64 GB | ~6–8 GB | 8–10 |
|
||||
| CPU | 20 threads (i9-10900X) | ~4 vCPU | 5 |
|
||||
| NVMe IOPS | High | Clone delta I/O | ~6–8 |
|
||||
| **Recommended** | — | — | **4 concurrent** |
|
||||
|
||||
`act_runner` `capacity: 4` is the conservative default. Raise to 6 after benchmarking.
|
||||
|
||||
---
|
||||
|
||||
## Security Boundaries
|
||||
|
||||
- Build VMs are on VMnet8 NAT (192.168.79.0/24) — host reaches VMs, VMs have internet access
|
||||
- WinRM credentials are stored in Windows Credential Manager on host (not hardcoded)
|
||||
- Each VM is fully destroyed after the job — no state carries between builds
|
||||
- Runner host does not expose any ports to the build VM network
|
||||
- Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) — no git clone inside VM
|
||||
@@ -0,0 +1,281 @@
|
||||
# Best Practices — Stability, Security & Operations
|
||||
|
||||
## 1. Credential Management
|
||||
|
||||
### Do NOT store credentials in scripts or config files
|
||||
|
||||
The scripts use `-GuestCredentialTarget` (a Windows Credential Manager target name)
|
||||
rather than plaintext username/password parameters. Store credentials once:
|
||||
|
||||
```powershell
|
||||
# Run on host (once, before first CI job)
|
||||
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:YourStrongPassword
|
||||
```
|
||||
|
||||
Retrieve in scripts via the `CredentialManager` PowerShell module:
|
||||
|
||||
```powershell
|
||||
Install-Module CredentialManager -Scope CurrentUser
|
||||
$cred = Get-StoredCredential -Target 'BuildVMGuest'
|
||||
```
|
||||
|
||||
### Rotate credentials quarterly
|
||||
|
||||
1. Update password in the template VM (requires rebuilding `BaseClean` snapshot)
|
||||
2. Update Windows Credential Manager on the host:
|
||||
```
|
||||
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:NewPassword
|
||||
```
|
||||
3. No script changes required — they reference the target name, not the password.
|
||||
|
||||
---
|
||||
|
||||
## 2. WinRM Security
|
||||
|
||||
### Current setup (HTTP / port 5985)
|
||||
|
||||
Acceptable for an **isolated lab network** where:
|
||||
- Build VMs are on a dedicated host-only VMware network (192.168.11.0/24 — VMnet11 dedicated CI)
|
||||
- No external traffic reaches port 5985
|
||||
- Credentials are managed via Windows Credential Manager
|
||||
|
||||
### Recommended upgrade: WinRM over HTTPS (port 5986)
|
||||
|
||||
```powershell
|
||||
# Inside the template VM (before taking BaseClean snapshot):
|
||||
|
||||
# Create self-signed certificate
|
||||
$cert = New-SelfSignedCertificate `
|
||||
-Subject "CN=$(hostname)" `
|
||||
-CertStoreLocation Cert:\LocalMachine\My `
|
||||
-KeyUsage DigitalSignature, KeyEncipherment `
|
||||
-KeyAlgorithm RSA `
|
||||
-KeyLength 2048
|
||||
|
||||
# Create HTTPS WinRM listener
|
||||
New-WSManInstance WinRM/Config/Listener `
|
||||
-SelectorSet @{ Transport = 'HTTPS'; Address = '*' } `
|
||||
-ValueSet @{ Hostname = $(hostname); CertificateThumbprint = $cert.Thumbprint }
|
||||
|
||||
# Allow port 5986
|
||||
New-NetFirewallRule -Name 'WinRM-HTTPS-CI' -DisplayName 'WinRM HTTPS CI' `
|
||||
-Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow
|
||||
```
|
||||
|
||||
```powershell
|
||||
# On the HOST (in scripts), use HTTPS session options:
|
||||
$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
|
||||
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL `
|
||||
-Credential $cred -SessionOption $sessionOptions
|
||||
```
|
||||
|
||||
> `-SkipCACheck` is acceptable for a self-signed cert in an isolated lab. Do NOT
|
||||
> use this against externally accessible machines.
|
||||
|
||||
---
|
||||
|
||||
## 3. act_runner Service Stability
|
||||
|
||||
### Windows Service Recovery Policy
|
||||
|
||||
The `Install-Runner.ps1` script configures automatic service restart on failure:
|
||||
- Restart after 1st failure: 5 seconds
|
||||
- Restart after 2nd failure: 10 seconds
|
||||
- Restart after subsequent: 30 seconds
|
||||
|
||||
Verify in Services → act_runner → Properties → Recovery tab.
|
||||
|
||||
### Monitor the service
|
||||
|
||||
```powershell
|
||||
# Check service status
|
||||
Get-Service act_runner | Select-Object Status, StartType
|
||||
|
||||
# View last 50 log lines
|
||||
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List
|
||||
|
||||
# Restart if needed
|
||||
Restart-Service act_runner
|
||||
```
|
||||
|
||||
### Scheduled health check (optional)
|
||||
|
||||
Create a scheduled task that verifies the runner appears "Online" in Gitea via API:
|
||||
|
||||
```powershell
|
||||
# Check runner status via Gitea API every 15 minutes
|
||||
$response = Invoke-RestMethod `
|
||||
-Uri "http://gitea.local/api/v1/admin/runners" `
|
||||
-Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }
|
||||
|
||||
$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
|
||||
if (-not $runnerOnline) {
|
||||
# Send alert (email, webhook, etc.) or restart service
|
||||
Restart-Service act_runner
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Template VM Integrity
|
||||
|
||||
The "BaseClean" snapshot is the foundation of every build. If it is corrupted,
|
||||
**all builds fail immediately**.
|
||||
|
||||
### Protection measures
|
||||
|
||||
1. **Never power on the template VM for reasons other than planned maintenance.**
|
||||
Configure VMware Workstation to prevent accidental starts: right-click → Settings →
|
||||
Options → Advanced → disable "Allow background snapshots".
|
||||
|
||||
2. **Backup the parent VMDK before any template changes:**
|
||||
```powershell
|
||||
# Before any template maintenance
|
||||
$templateDir = 'F:\CI\Templates\WinBuild'
|
||||
$backupDir = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
|
||||
Copy-Item $templateDir $backupDir -Recurse
|
||||
```
|
||||
|
||||
3. **Keep a list of all current linked clones** before refreshing the snapshot.
|
||||
If any clone exists when you modify the parent, it may break.
|
||||
Check: `vmrun list` — should return no build VMs during maintenance window.
|
||||
|
||||
4. **Version the snapshot name** to make rollback easy:
|
||||
Instead of reusing "BaseClean", name snapshots `BaseClean_20260101`.
|
||||
Update `config.yaml` `envs.GITEA_CI_SNAPSHOT_NAME` when rotating.
|
||||
|
||||
---
|
||||
|
||||
## 5. Orphaned VM Cleanup
|
||||
|
||||
If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be
|
||||
destroyed. Run this cleanup script on host startup or as a daily scheduled task:
|
||||
|
||||
```powershell
|
||||
# Cleanup-OrphanedBuildVMs.ps1
|
||||
$vmrun = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
|
||||
$cloneBase = 'F:\CI\BuildVMs'
|
||||
$maxAgeHours = 4 # No job should run longer than 4 hours
|
||||
|
||||
Get-ChildItem $cloneBase -Directory |
|
||||
Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
|
||||
ForEach-Object {
|
||||
$vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
|
||||
if ($vmx) {
|
||||
Write-Host "Cleaning orphan: $($vmx.FullName)"
|
||||
& $vmrun -T ws stop $vmx.FullName hard 2>$null
|
||||
& $vmrun -T ws deleteVM $vmx.FullName 2>$null
|
||||
}
|
||||
Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Gitea Repository Configuration
|
||||
|
||||
### Required repository settings for workflows to run
|
||||
|
||||
1. Enable Actions for the repository: Settings → Repository → Actions → Enable
|
||||
2. Add secrets if needed: Settings → Secrets and Variables → Actions
|
||||
3. Protect main branch: Settings → Branches → Branch protection rules
|
||||
|
||||
### Workflow file location
|
||||
|
||||
Workflows **must** be at `.gitea/workflows/*.yml` (not `.github/workflows/`).
|
||||
|
||||
```
|
||||
your-repo/
|
||||
└── .gitea/
|
||||
└── workflows/
|
||||
└── build.yml ← copy from gitea/workflow-example.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Logging & Observability
|
||||
|
||||
### act_runner logs
|
||||
|
||||
The runner daemon writes to stdout (captured by the Windows service manager).
|
||||
Increase verbosity for debugging:
|
||||
|
||||
```yaml
|
||||
# runner/config.yaml
|
||||
log:
|
||||
level: debug # change from "info" to "debug"
|
||||
format: text
|
||||
```
|
||||
|
||||
### Per-job build logs
|
||||
|
||||
`Invoke-CIJob.ps1` outputs timestamped phase banners. act_runner captures all
|
||||
stdout/stderr and uploads it to Gitea Actions → job log viewer.
|
||||
|
||||
For persistent local logs:
|
||||
|
||||
```powershell
|
||||
# In your workflow YAML, redirect output to a log file:
|
||||
- name: Build in ephemeral VM
|
||||
shell: pwsh
|
||||
run: |
|
||||
.\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"
|
||||
```
|
||||
|
||||
### Windows Event Log
|
||||
|
||||
act_runner (when installed as a service) writes events to Windows Event Log →
|
||||
Application source "act_runner". Check with:
|
||||
|
||||
```powershell
|
||||
Get-EventLog -LogName Application -Source '*runner*' -Newest 20
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Network Isolation Verification
|
||||
|
||||
After setting up the host-only VMware network, verify that build VMs cannot
|
||||
reach the internet (important for supply-chain security):
|
||||
|
||||
```powershell
|
||||
# From inside a build VM via WinRM:
|
||||
Invoke-Command -Session $session -ScriptBlock {
|
||||
$result = Test-Connection 8.8.8.8 -Count 1 -Quiet
|
||||
if ($result) {
|
||||
Write-Warning "VM has internet access — check VMware network adapter type"
|
||||
} else {
|
||||
Write-Host "VM correctly isolated (no internet)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Build VMs should only reach:
|
||||
- The host (for WinRM connection)
|
||||
- Gitea server (for git clone, if reachable via host-only network)
|
||||
- NuGet cache share (host-side shared folder)
|
||||
|
||||
---
|
||||
|
||||
## 9. Updating the Build Toolchain
|
||||
|
||||
When a new .NET SDK or VS Build Tools version is released:
|
||||
|
||||
1. **During a maintenance window** (no CI jobs running):
|
||||
```
|
||||
vmrun list ← must be empty
|
||||
```
|
||||
2. Boot the template VM
|
||||
3. Run updates:
|
||||
```powershell
|
||||
# Update .NET SDK
|
||||
& "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0
|
||||
|
||||
# Update VS Build Tools via Visual Studio Installer
|
||||
"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
|
||||
```
|
||||
4. Verify tools work (run a test build manually)
|
||||
5. Shut down VM
|
||||
6. Take new snapshot: `BaseClean_$(Get-Date -Format yyyyMMdd)`
|
||||
7. Update `SnapshotName` in `runner/config.yaml`
|
||||
8. Delete the old snapshot after confirming new one works for 1 week
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
# CI Pipeline Flow — Step by Step
|
||||
|
||||
## Trigger to Artifact: Complete Pipeline Walk-Through
|
||||
|
||||
---
|
||||
|
||||
### Step 0 — Prerequisites (one-time setup)
|
||||
|
||||
Before any CI job runs, the following must be in place:
|
||||
|
||||
| Component | State |
|
||||
| ------------------ | ----------------------------------------------------------------------- |
|
||||
| Gitea server | Running, accessible on LAN |
|
||||
| act_runner | Registered, running as Windows service on host |
|
||||
| Template VM | Provisioned, snapshot "BaseClean" taken, VM powered off |
|
||||
| WinRM | Enabled on template VM (inherits to all clones) |
|
||||
| vmrun.exe | Present at `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe` |
|
||||
| `F:\CI\BuildVMs\` | Directory exists and writable |
|
||||
| `F:\CI\Artifacts\` | Directory exists and writable |
|
||||
|
||||
---
|
||||
|
||||
### Step 1 — Developer Pushes Code
|
||||
|
||||
```
|
||||
Developer → git push origin main
|
||||
```
|
||||
|
||||
- Git push lands on Gitea server
|
||||
- Gitea evaluates `.gitea/workflows/build.yml` in the repository
|
||||
- If the workflow trigger matches (e.g. `on: push` to `main`), Gitea enqueues a new **Actions job**
|
||||
- Job is tagged with the label defined in the workflow (`runs-on: windows-build`)
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — act_runner Picks Up the Job
|
||||
|
||||
```
|
||||
act_runner (host) ─polling─► Gitea Actions API
|
||||
◄─job data─
|
||||
```
|
||||
|
||||
- act_runner polls the Gitea API approximately every 5–10 seconds
|
||||
- Detects a pending job matching its labels (`windows-build`)
|
||||
- Downloads the workflow YAML and job parameters
|
||||
- If `capacity` slots are available, starts job execution immediately
|
||||
- If all slots are busy, job waits in Gitea queue until a slot frees
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — act_runner Executes Workflow Steps
|
||||
|
||||
The workflow YAML defines steps that run as PowerShell commands **on the host**.
|
||||
The first substantive step calls `Invoke-CIJob.ps1`:
|
||||
|
||||
```yaml
|
||||
- name: Build in ephemeral VM
|
||||
shell: pwsh
|
||||
run: |
|
||||
.\scripts\Invoke-CIJob.ps1 `
|
||||
-JobId "${{ github.run_id }}" `
|
||||
-RepoUrl "${{ env.GITEA_REPO_URL }}" `
|
||||
-Branch "${{ github.ref_name }}" `
|
||||
-Commit "${{ github.sha }}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Invoke-CIJob.ps1: Clone VM
|
||||
|
||||
```
|
||||
New-BuildVM.ps1
|
||||
vmrun.exe -T ws clone <template.vmx> <clone.vmx> linked -snapshot "BaseClean"
|
||||
```
|
||||
|
||||
- Generates unique clone name: `Clone_{JobId}_{yyyyMMdd_HHmmss}`
|
||||
- Creates clone directory under `F:\CI\BuildVMs\`
|
||||
- **Linked clone** creation takes ~5–15 seconds
|
||||
- The template snapshot "BaseClean" is **not modified**
|
||||
- Returns the full path to the new `.vmx` file
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Start VM
|
||||
|
||||
```
|
||||
vmrun.exe -T ws start <clone.vmx> nogui
|
||||
```
|
||||
|
||||
- Starts the cloned VM in headless (no GUI) mode
|
||||
- VM boots from the "BaseClean" state: clean OS, build tools installed, WinRM enabled
|
||||
- Boot takes ~20–40 seconds
|
||||
|
||||
---
|
||||
|
||||
### Step 6 — Wait for VM Readiness
|
||||
|
||||
```
|
||||
Wait-VMReady.ps1 -VMPath <clone.vmx> -IPAddress <dynamic-NAT-ip> -TimeoutSeconds 300
|
||||
```
|
||||
|
||||
IP is discovered dynamically via `vmrun getGuestIPAddress` (VMnet8 NAT, subnet 192.168.79.0/24).
|
||||
No hardcoded IPs — each clone gets a DHCP-assigned address from VMware NAT.
|
||||
|
||||
Three-phase readiness check (retried every 5 seconds, up to 300s timeout):
|
||||
|
||||
```
|
||||
Phase 1: vmrun getState → must return "running"
|
||||
Phase 2: Test-Connection (ICMP ping) → must succeed
|
||||
Phase 3: Test-WSMan → WinRM listener must respond
|
||||
```
|
||||
|
||||
- Total wait is typically **30–90 seconds** after `vmrun start`
|
||||
- If timeout exceeded, VM is destroyed and job fails with a clear error message
|
||||
- On success, IP and credentials are ready for PSSession
|
||||
|
||||
---
|
||||
|
||||
### Step 7 — Remote Build Execution
|
||||
|
||||
```
|
||||
Invoke-RemoteBuild.ps1
|
||||
-IPAddress <dynamic-NAT-ip> # from vmrun getGuestIPAddress
|
||||
-Credential (from Windows Credential Manager)
|
||||
-BuildCommand 'python build_plugin.py --final --dist-dir dist'
|
||||
-GuestArtifactSource 'dist'
|
||||
```
|
||||
|
||||
Sequence inside the PSSession:
|
||||
|
||||
```
|
||||
1. New-PSSession → open WinRM session to VM
|
||||
2. Compress-Archive (host) → copy zip to C:\CI\build.zip (guest) → Expand-Archive to C:\CI\build\
|
||||
(source code è già clonato sull'host in Phase 1 — non si fa git clone nella VM)
|
||||
3. Invoke-Expression "$BuildCommand" in C:\CI\build\
|
||||
Esempio: python build_plugin.py --final --dist-dir dist
|
||||
a. Build 4 configurazioni MSBuild in parallelo (x86-unicode, x64-unicode, cli-x86, cli-x64)
|
||||
b. Raccoglie artifact in C:\CI\build\dist\
|
||||
c. Cleanup build dir (--final)
|
||||
4. Compress-Archive C:\CI\build\$GuestArtifactSource C:\CI\output\artifacts.zip
|
||||
5. Cattura exit code; se non-zero il job fallisce con errore chiaro
|
||||
6. Remove-PSSession
|
||||
```
|
||||
|
||||
- Output standard e stderr dalla build sono streamati all'host e catturati da act_runner per i log Gitea Actions
|
||||
- Se la build fallisce (exit code ≠ 0), il controllo passa al blocco `finally`
|
||||
|
||||
---
|
||||
|
||||
### Step 8 — Collect Artifacts
|
||||
|
||||
```
|
||||
Get-BuildArtifacts.ps1
|
||||
-IPAddress <dynamic-NAT-ip>
|
||||
-Credential (same as above)
|
||||
-GuestArtifactPath C:\CI\output\artifacts.zip
|
||||
-HostArtifactDir F:\CI\Artifacts\{JobId}\
|
||||
```
|
||||
|
||||
- Apre PSSession e copia `C:\CI\output\artifacts.zip` dall'host
|
||||
- Valida che il file esista e sia non-vuoto
|
||||
- Il zip contiene la struttura `dist/` con tutti gli artifact di build
|
||||
|
||||
---
|
||||
|
||||
### Step 9 — Destroy VM
|
||||
|
||||
```
|
||||
Remove-BuildVM.ps1 -VMPath <clone.vmx>
|
||||
```
|
||||
|
||||
Sequence:
|
||||
|
||||
```
|
||||
1. vmrun stop <clone.vmx> soft (graceful shutdown attempt)
|
||||
2. Wait 5 seconds
|
||||
3. vmrun getState → if still "running":
|
||||
vmrun stop <clone.vmx> hard (force off)
|
||||
4. vmrun deleteVM <clone.vmx> (removes VMX + VMDK delta files)
|
||||
5. Remove-Item <clone dir> -Recurse (removes any leftover files)
|
||||
```
|
||||
|
||||
- VM is gone from disk within ~5–10 seconds
|
||||
- No state, no temp files, no credentials remain in the clone
|
||||
- Template VM and "BaseClean" snapshot are unaffected
|
||||
|
||||
> **Note:** This step runs in a `finally` block inside `Invoke-CIJob.ps1`, so it
|
||||
> executes even if Steps 6–8 failed. A build VM is **never** left running.
|
||||
|
||||
---
|
||||
|
||||
### Step 10 — Report Status & Upload Artifacts
|
||||
|
||||
```
|
||||
act_runner ─► Gitea Actions API (job status: success / failure)
|
||||
act_runner ─► Gitea artifact store (uploads F:\CI\Artifacts\{JobId}\*)
|
||||
```
|
||||
|
||||
- act_runner reads the exit code from `Invoke-CIJob.ps1`
|
||||
- Reports `success` or `failure` to Gitea
|
||||
- Uploads artifacts using `actions/upload-artifact` step in the workflow YAML
|
||||
- Build log (stdout from all steps) is stored in Gitea Actions UI
|
||||
- Commit/PR status badge is updated
|
||||
|
||||
---
|
||||
|
||||
## Parallel Build Flow
|
||||
|
||||
When multiple pushes happen concurrently (or multiple PRs are open):
|
||||
|
||||
```
|
||||
act_runner (capacity: 4)
|
||||
├── Job #1 ──► Clone_Job_001 (192.168.79.x) → build → destroy
|
||||
├── Job #2 ──► Clone_Job_002 (192.168.79.x) → build → destroy
|
||||
├── Job #3 ──► Clone_Job_003 (192.168.79.x) → build → destroy
|
||||
└── Job #4 ──► Clone_Job_004 (192.168.79.x) → build → destroy
|
||||
(Job #5 waits in Gitea queue)
|
||||
```
|
||||
|
||||
Note: each clone gets a dynamic IP from VMware NAT DHCP (192.168.79.0/24). IP is discovered
|
||||
automatically via `vmrun getGuestIPAddress` — no static assignment required.
|
||||
|
||||
- Each job creates its own independently named clone
|
||||
- Clones share the parent "BaseClean" snapshot (read-only CoW)
|
||||
- No filesystem or process overlap between concurrent builds
|
||||
- act_runner enforces the `capacity` limit
|
||||
|
||||
---
|
||||
|
||||
## Failure Scenarios
|
||||
|
||||
| Failure Point | Behavior |
|
||||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| VM clone fails | `Invoke-CIJob.ps1` throws; no VM to destroy; job fails immediately |
|
||||
| VM never reaches ready state (timeout) | `Wait-VMReady.ps1` throws; `finally` destroys the (running) VM |
|
||||
| Build fails inside VM | `Invoke-RemoteBuild.ps1` throws; `finally` collects partial artifacts (if any), destroys VM |
|
||||
| Artifact collection fails | Job marked failed; VM is still destroyed in `finally` |
|
||||
| Host machine loses power mid-job | `act_runner` resumes on reboot, marks in-flight job as failed; orphan VMs must be cleaned up manually (see TODO.md) |
|
||||
|
||||
---
|
||||
|
||||
## Timing Summary (typical build)
|
||||
|
||||
| Phase | Typical Duration |
|
||||
| ---------------------------- | -------------------- |
|
||||
| Gitea → act_runner poll | 5–10 s |
|
||||
| Linked clone creation | 5–15 s |
|
||||
| VM boot | 20–40 s |
|
||||
| WinRM readiness | 10–30 s (after boot) |
|
||||
| git clone (from local Gitea) | 2–10 s |
|
||||
| dotnet restore + build | project-dependent |
|
||||
| Artifact collection | 2–10 s |
|
||||
| VM destruction | 5–10 s |
|
||||
| **Overhead total** | **~60–120 s** |
|
||||
@@ -0,0 +1,197 @@
|
||||
# Optimization Strategies
|
||||
|
||||
## 1. Linked Clone Disk Layout
|
||||
|
||||
**Goal:** Minimize clone creation time and I/O contention between concurrent VMs.
|
||||
|
||||
### Recommended NVMe Partition Layout
|
||||
|
||||
```
|
||||
NVMe SSD (e.g. 2TB)
|
||||
├── C:\ — Windows host OS, VMware Workstation installation
|
||||
├── F:\CI\
|
||||
│ ├── Templates\ — Template VM VMX + base VMDK (parent snapshot)
|
||||
│ ├── BuildVMs\ — Ephemeral linked clone VMs (delta VMDKs)
|
||||
│ ├── Artifacts\ — Collected build artifacts (per job)
|
||||
│ ├── Cache\ — NuGet / npm cache (see §4)
|
||||
│ └── RunnerWork\ — act_runner workspace (checkout, step scripts)
|
||||
```
|
||||
|
||||
**Why separate directories matter:**
|
||||
- Template VMDK reads (CoW base) and clone delta writes happen simultaneously
|
||||
- Keeping them on the same fast NVMe avoids I/O stalls; a separate spinning disk for
|
||||
the clone directory would bottleneck clone creation
|
||||
- Artifacts and cache dirs have sequential I/O patterns; they can share space
|
||||
|
||||
---
|
||||
|
||||
## 2. Snapshot Strategy
|
||||
|
||||
### Tiered Snapshot Model
|
||||
|
||||
```
|
||||
Tier 0: Base OS Install (Windows only, no tools)
|
||||
└── Snapshot: "OSBase" — rarely updated (yearly or on major Windows updates)
|
||||
|
||||
Tier 1: Build Toolchain
|
||||
└── Snapshot: "BaseClean" ← CLONE SOURCE (weekly refresh)
|
||||
Includes: VS Build Tools 2022, .NET SDK, Git, WinRM config
|
||||
Requirement: ALL linked clones must reference this exact snapshot
|
||||
```
|
||||
|
||||
### Refresh Schedule
|
||||
|
||||
| Snapshot | Refresh Frequency | Trigger |
|
||||
|----------|------------------|-------|
|
||||
| `OSBase` | Quarterly | Windows cumulative update |
|
||||
| `BaseClean` | Weekly/Monthly | .NET SDK patch, security update, VS update |
|
||||
|
||||
> **Note:** Windows Server 2025 KMS lease = 180 giorni. Prima della scadenza:
|
||||
> boot template su VMnet8 (NAT) → `slmgr /ato` → spegni → nuovo snapshot `BaseClean`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Parallel Build Capacity
|
||||
|
||||
### RAM Budget (i9-10900X · 64 GB)
|
||||
|
||||
| Component | RAM Usage |
|
||||
|-----------|----------|
|
||||
| Windows host OS | ~4 GB |
|
||||
| VMware Workstation | ~0.5 GB |
|
||||
| Gitea server | ~0.5–1 GB |
|
||||
| act_runner service | ~100 MB |
|
||||
| Each build VM (idle) | ~2–3 GB |
|
||||
| Each build VM (active MSBuild/Python) | ~6–8 GB |
|
||||
| **Headroom target (20%)** | ~13 GB |
|
||||
|
||||
```
|
||||
Available for VMs: 64 - 4 - 0.5 - 1 - 0.1 - 13 = ~45 GB
|
||||
Max VMs at peak: 45 / 8 = ~5.6 → safe limit = 4
|
||||
```
|
||||
|
||||
**Recommendation:** `capacity: 4` in `config.yaml`.
|
||||
|
||||
### CPU Budget (i9-10900X: 10 cores / 20 threads)
|
||||
|
||||
- Template VM: 4 vCPU configurati
|
||||
- Con 4 VM parallele: 4 × 4 = 16 thread, lascia 4 per host OS / Gitea / runner
|
||||
- `build_plugin.py` usa `get_optimal_thread_count()` che rileva i core della VM e divide per il numero di build parallele (fix TRK0002)
|
||||
|
||||
### VM Configuration (template VMX)
|
||||
|
||||
```ini
|
||||
numvcpus = "4"
|
||||
cpuid.coresPerSocket = "2"
|
||||
memsize = "6144" # 6 GB RAM per VM
|
||||
scsi0.virtualDev = "pvscsi"
|
||||
ethernet0.virtualDev = "vmxnet3"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. NuGet / Package Cache on Host
|
||||
|
||||
**Problem:** Each build VM does a fresh `dotnet restore`, re-downloading NuGet packages every time.
|
||||
|
||||
**Solution:** Map a host-side NuGet cache directory as a shared folder into each VM.
|
||||
|
||||
### Setup (host side)
|
||||
|
||||
```
|
||||
F:\CI\Cache\NuGet\ ← shared folder on host
|
||||
```
|
||||
|
||||
### VMware Shared Folder Configuration (in template VMX)
|
||||
|
||||
Add to `WinBuild.vmx` before taking the `BaseClean` snapshot:
|
||||
```ini
|
||||
sharedFolder0.present = "TRUE"
|
||||
sharedFolder0.enabled = "TRUE"
|
||||
sharedFolder0.readAccess = "TRUE"
|
||||
sharedFolder0.writeAccess = "TRUE"
|
||||
sharedFolder0.hostPath = "F:\\CI\\Cache\\NuGet"
|
||||
sharedFolder0.guestName = "nuget-cache"
|
||||
sharedFolder.maxNum = "1"
|
||||
```
|
||||
|
||||
Inside the VM, the shared folder appears as `\\vmware-host\Shared Folders\nuget-cache`.
|
||||
|
||||
### NuGet Cache Redirect (in `Invoke-RemoteBuild.ps1`)
|
||||
|
||||
Add to the remote build ScriptBlock:
|
||||
```powershell
|
||||
$env:NUGET_PACKAGES = '\\vmware-host\Shared Folders\nuget-cache'
|
||||
dotnet restore --packages $env:NUGET_PACKAGES
|
||||
```
|
||||
|
||||
**Result:** First build per package downloads once; subsequent builds read from the host cache.
|
||||
Cache is shared across all concurrent VMs (NuGet packages are safe for concurrent reads).
|
||||
|
||||
---
|
||||
|
||||
## 5. Clone Pre-Warming
|
||||
|
||||
For latency-sensitive pipelines, keep N pre-warmed clones ready in "booted and idle" state.
|
||||
|
||||
**Tradeoff:** Saves ~45–90 seconds of startup per job, but clones remain running (consuming RAM) until used.
|
||||
|
||||
**Implementation sketch:**
|
||||
|
||||
```powershell
|
||||
# Pre-warmer job (separate scheduled task on host)
|
||||
# Runs between CI jobs to maintain a pool of warm VMs
|
||||
$poolSize = 2 # Keep 2 VMs warm at all times
|
||||
|
||||
# Check current running clones
|
||||
$runningClones = Get-ChildItem F:\CI\WarmPool\ -Filter *.vmx
|
||||
if ($runningClones.Count -lt $poolSize) {
|
||||
# Create and start new clone in warm pool
|
||||
$vmx = .\New-BuildVM.ps1 -TemplatePath $template -CloneBaseDir F:\CI\WarmPool -JobId "warm-$(Get-Random)"
|
||||
vmrun -T ws start $vmx nogui
|
||||
}
|
||||
```
|
||||
|
||||
> **Note:** Pre-warming is an advanced optimization. Start without it and add
|
||||
> only if CI overhead (startup time) is a demonstrated bottleneck.
|
||||
|
||||
---
|
||||
|
||||
## 6. Artifact Storage Management
|
||||
|
||||
Build artifacts accumulate quickly. Automate cleanup:
|
||||
|
||||
```powershell
|
||||
# Scheduled task: run daily
|
||||
# Remove artifact dirs older than 30 days
|
||||
Get-ChildItem 'F:\CI\Artifacts' -Directory |
|
||||
Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-30) } |
|
||||
Remove-Item -Recurse -Force
|
||||
|
||||
# Remove orphaned clone directories (VMs that were not properly deleted)
|
||||
Get-ChildItem 'F:\CI\BuildVMs' -Directory |
|
||||
Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-4) } |
|
||||
ForEach-Object {
|
||||
$vmx = Get-ChildItem $_.FullName -Filter *.vmx | Select-Object -First 1
|
||||
if ($vmx) {
|
||||
vmrun -T ws stop $vmx.FullName hard 2>$null
|
||||
}
|
||||
Remove-Item $_.FullName -Recurse -Force
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. NUMA & CPU Affinity (Advanced)
|
||||
|
||||
The i9-10900X is a single NUMA node CPU (no NUMA effects). CPU pinning is not needed.
|
||||
|
||||
However, you can set CPU affinity per VM group to reduce vCPU migration overhead:
|
||||
|
||||
```powershell
|
||||
# Optional: pin act_runner process to cores 0-3 (leave build VMs on 4-19)
|
||||
$runnerPid = (Get-Process act_runner).Id
|
||||
Start-Process -FilePath 'cmd' -ArgumentList "/c start /affinity 0xF /b /wait powershell" -NoNewWindow
|
||||
```
|
||||
|
||||
For most workloads, Windows scheduler handles this well — skip affinity unless profiling shows issues.
|
||||
@@ -0,0 +1,101 @@
|
||||
# Configurazione accesso SSH a Gitea
|
||||
|
||||
**Target**: clone da `git@gitea.emulab.it:Simone/nsis-plugin-nsinnounp.git`
|
||||
**Gitea**: `http://10.10.20.11:3100` / `https://gitea.emulab.it`
|
||||
|
||||
---
|
||||
|
||||
## 1. Genera la chiave SSH sull'host Windows
|
||||
|
||||
```powershell
|
||||
# Da PowerShell normale (non admin necessario)
|
||||
ssh-keygen -t ed25519 -C "ci-build@WS1-W11" -f "$env:USERPROFILE\.ssh\id_ci_gitea"
|
||||
```
|
||||
|
||||
- Passphrase: **lascia vuota** (il CI gira non-interattivo)
|
||||
- Produce due file:
|
||||
- `~\.ssh\id_ci_gitea` — chiave privata (non condividere mai)
|
||||
- `~\.ssh\id_ci_gitea.pub` — chiave pubblica da aggiungere a Gitea
|
||||
|
||||
---
|
||||
|
||||
## 2. Aggiungi la chiave pubblica a Gitea
|
||||
|
||||
1. Apri `http://10.10.20.11:3100` → Login come **Simone**
|
||||
2. Menu utente (in alto a destra) → **Settings** → **SSH / GPG Keys**
|
||||
3. Clicca **Add Key**
|
||||
4. Incolla il contenuto di `~\.ssh\id_ci_gitea.pub`:
|
||||
|
||||
```powershell
|
||||
Get-Content "$env:USERPROFILE\.ssh\id_ci_gitea.pub" | clip
|
||||
```
|
||||
|
||||
5. **Key Name**: `WS1-CI-Host`
|
||||
6. Salva → **Add Key**
|
||||
|
||||
---
|
||||
|
||||
## 3. Configura SSH client sull'host
|
||||
|
||||
> **Attenzione**: `10.10.20.11` è già usato nel file `~\.ssh\config` con `User root` (entry `HomeAssistant`).
|
||||
> Non usare `Host 10.10.20.11` direttamente — usare un alias dedicato `gitea-ci`.
|
||||
|
||||
Aggiungi in fondo a `~\.ssh\config`:
|
||||
|
||||
```powershell
|
||||
$entry = @"
|
||||
|
||||
Host gitea-ci
|
||||
HostName 10.10.20.11
|
||||
Port 2222
|
||||
User git
|
||||
IdentityFile $env:USERPROFILE\.ssh\id_ci_gitea
|
||||
IdentitiesOnly yes
|
||||
"@
|
||||
Add-Content "$env:USERPROFILE\.ssh\config" $entry
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Verifica connessione
|
||||
|
||||
```powershell
|
||||
ssh -T gitea-ci
|
||||
```
|
||||
|
||||
Risposta attesa:
|
||||
```
|
||||
Hi there, Simone! You've successfully authenticated with the key named ci-build@WS1-W11, but Gitea does not provide shell access.
|
||||
```
|
||||
|
||||
> **Nota**: exit code 1 è **normale** — Gitea non fornisce shell, ma l'autenticazione è avvenuta.
|
||||
|
||||
---
|
||||
|
||||
## 5. ~~Trova la porta SSH di Gitea~~
|
||||
|
||||
**Porta confermata: 2222** (verificata con `Test-NetConnection -ComputerName 10.10.20.11 -Port 2222`).
|
||||
|
||||
---
|
||||
|
||||
## 6. URL SSH da usare in Invoke-CIJob.ps1
|
||||
|
||||
Usare l'alias `gitea-ci` definito in `~\.ssh\config`:
|
||||
|
||||
```powershell
|
||||
-RepoUrl 'ssh://gitea-ci/Simone/nsis-plugin-nsinnounp.git'
|
||||
```
|
||||
|
||||
Oppure con IP/porta espliciti (equivalente):
|
||||
|
||||
```powershell
|
||||
-RepoUrl 'ssh://git@10.10.20.11:2222/Simone/nsis-plugin-nsinnounp.git'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Aggiungi host a known_hosts (evita prompt interattivo)
|
||||
|
||||
```powershell
|
||||
ssh-keyscan -p 2222 10.10.20.11 >> "$env:USERPROFILE\.ssh\known_hosts"
|
||||
```
|
||||
Reference in New Issue
Block a user