local-ci-cd-system/docs/BEST-PRACTICES.md

Best Practices — Stability, Security & Operations

1. Credential Management

Do NOT store credentials in scripts or config files

The scripts use -GuestCredentialTarget (a Windows Credential Manager target name) rather than plaintext username/password parameters. Store credentials once:

# Run on host (once, before first CI job)
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:YourStrongPassword

Retrieve in scripts via the CredentialManager PowerShell module:

Install-Module CredentialManager -Scope CurrentUser
$cred = Get-StoredCredential -Target 'BuildVMGuest'

Rotate credentials quarterly

1. Update password in the template VM (requires rebuilding BaseClean snapshot)
2. Update Windows Credential Manager on the host:

   cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:NewPassword
   

3. No script changes required — they reference the target name, not the password.

---

2. WinRM Security — HTTPS/5986 (implementato 2026-05-10)

Setup attuale (HTTPS / port 5986)

Deploy-WinBuild2025.ps1 post-install.ps1 crea un certificato self-signed e configura il listener HTTPS/5986 prima dello snapshot BaseClean. AllowUnencrypted=false.

• Build VMs su VMnet8 NAT (192.168.79.0/24) — accesso solo dall'host
• Port 5986 firewall rule ristretta a RemoteAddress '192.168.79.0/24'
• Credentials via Windows Credential Manager (target BuildVMGuest)

Tutti gli script host usano:

$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL -Authentication Basic `
    -Credential $cred -SessionOption $sessionOptions

-SkipCACheck/-SkipCNCheck sono accettabili per un cert self-signed in lab isolato. Non usare contro macchine accessibili dall'esterno — usare una CA trusted in quel caso.

---

2.1. Threat Model — Disabled Security Features

Current state: Defender, Firewall, and UAC disabled

The template VM disables Windows Defender, Windows Firewall, and User Account Control (UAC). This is intentional — not a bug, not an oversight. Each has tradeoffs:

Feature	Disabled?	Why	Cost if enabled
Windows Defender	Yes	Real-time AV scanning blocks .NET compilation, Python wheels, and npm installs	5–10 min per build overhead; false positives on dev tools
Windows Firewall	Yes	Blocks inbound WinRM even with rules; requires Domain/Home profile tuning	Complex rules; fragile across OS updates
UAC (LocalAccountTokenFilterPolicy)	Yes	Prevents non-elevated WinRM scripts from running builds	Requires built-in Administrator account; WinRM behaves like a user with limited rights

When this threat model is acceptable

Current threat model is safe if ALL of these are true:

1. Isolated lab environment — Build VMs exist only on VMnet8 (NAT), not on host LAN.
2. No shared resources — Host is not shared with untrusted users or concurrent CI systems.
3. Trusted source code — Code being built is from trusted repositories (internal team only).
4. No external access — VMnet8 is not bridged or exposed to corporate LAN or internet.
5. Act_runner is trusted — The act_runner service token cannot be used to access host resources outside the isolated network.

If all conditions hold, the attack surface is limited to:

• Network eavesdropping on 192.168.79.0/24 (mitigated: WinRM is HTTPS)
• Code injection via malicious commits (mitigated: code review process)
• Privilege escalation from VM to host (mitigated: VMs are ephemeral; no persistence)

When the model breaks down

Do NOT use this configuration if:

• ❌ Third-party code builds — Running untrusted vendor code (open-source projects, third-party libraries with build scripts)
• ❌ Shared build machine — Other teams or processes share the host CPU/storage
• ❌ LAN-exposed network — VMnet8 is bridged to corporate LAN or internet
• ❌ Host resource sharing — Build VMs can access host shares, USB drives, or external storage
• ❌ Long-lived VMs — VMs are not destroyed after each build (antivirus blind spot for persistence)

In these scenarios, disabled AV and firewall create unacceptable risk.

Mitigations if constraints change

If you must run in a less-isolated environment, re-enable protections with cost awareness:

Option 1: Re-enable Firewall only (lowest cost)

# In template VM via WinRM, before taking BaseClean snapshot:
Set-NetFirewallProfile -Profile Domain, Public, Private -Enabled $true
# Add inbound rule for WinRM listener
New-NetFirewallRule -Name "WinRM-HTTPS" `
    -DisplayName "Windows Remote Management (HTTPS)" `
    -Direction Inbound `
    -LocalPort 5986 `
    -Protocol TCP `
    -Action Allow

Cost: 30–60 seconds per build (firewall rule evaluation + logging).
Benefit: Blocks outbound malware callbacks if VM is compromised.

Option 2: Re-enable Defender with exclusions (moderate cost)

# In template VM, enable Defender but exclude build directories:
Enable-MpComputerDefault   # Re-enable Defender
Add-MpPreference -ExclusionPath @(
    'C:\Build',
    'C:\Users\ci_build\AppData\Local\Microsoft\dotnet',
    'C:\Users\ci_build\AppData\Roaming\npm'
) -Force
# Reduce scanning aggressiveness:
Set-MpPreference -DisableRealtimeMonitoring $false -DisableBehaviorMonitoring $true

Cost: 2–5 min per build (initial scan; exclusions help but don't eliminate overhead).
Benefit: Detects known malware uploaded in build artifacts.

Option 3: Enable UAC for elevated builds only (requires refactor)

# NOT RECOMMENDED without major refactoring.
# WinRM remote commands run as non-elevated user; builds fail.
# Requires either:
#   - Running WinRM as built-in Administrator (security anti-pattern)
#   - Adding explicit runas prompts (breaks automation)
#   - Using Windows Task Scheduler instead of WinRM (complexity)

Audit and sign-off

Before deploying to production or a shared host:

1. Document the decision: Update this section with current date and approver name.
2. Test the mitigations: Create test clone, enable firewall/AV, measure build time overhead.
3. Establish monitoring: Run Watch-RunnerHealth.ps1 continuously; alert on service restarts.
4. Plan rotation: Schedule quarterly credential rotation (see §1 Credential Management).

---

3. act_runner Service Stability

Windows Service Recovery Policy

The Install-Runner.ps1 script configures automatic service restart on failure:

• Restart after 1st failure: 5 seconds
• Restart after 2nd failure: 10 seconds
• Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

Monitor the service

# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner

Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}

---

4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted, all builds fail immediately.

Protection measures

1. Never power on the template VM for reasons other than planned maintenance. Configure VMware Workstation to prevent accidental starts: right-click → Settings → Options → Advanced → disable "Allow background snapshots".

2. Backup the parent VMDK before any template changes:

   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   

3. Keep a list of all current linked clones before refreshing the snapshot. If any clone exists when you modify the parent, it may break. Check: vmrun list — should return no build VMs during maintenance window.

4. Version the snapshot name to make rollback easy: Instead of reusing "BaseClean", name snapshots BaseClean_20260101. Update config.yaml envs.GITEA_CI_SNAPSHOT_NAME when rotating.

---

5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be destroyed. Run this cleanup script on host startup or as a daily scheduled task:

# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }

---

6. Gitea Repository Configuration

Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

Workflow file location

Workflows must be at .gitea/workflows/*.yml (not .github/workflows/).

your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml

---

7. Logging & Observability

act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager). Increase verbosity for debugging:

# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text

Per-job build logs

Invoke-CIJob.ps1 outputs timestamped phase banners. act_runner captures all stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"

Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log → Application source "act_runner". Check with:

Get-EventLog -LogName Application -Source '*runner*' -Newest 20

---

8. Network Topology Verification

Build VMs run on VMnet8 (NAT) — they have internet access, which is required for pip/nuget package downloads at build time. Verify the expected topology:

# From inside a build VM via WinRM — confirm NAT internet is reachable:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Host "VM has NAT internet access — expected for pip/nuget builds."
    } else {
        Write-Warning "VM cannot reach internet — pip/nuget installs will fail. Check VMware NAT service."
    }
}

Build VMs can reach:

• The host via VMnet8 gateway (WinRM HTTPS on port 5986)
• Internet via VMware NAT (for pip, nuget, npm at build time)
• Gitea server if on LAN reachable via NAT gateway

Supply-chain note: Source code is always injected by the host via WinRM zip transfer — never cloned inside the VM using a PAT. This keeps credentials off the VM even though the VM has outbound internet access.

---

9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. During a maintenance window (no CI jobs running):

   vmrun list   ← must be empty
   

2. Boot the template VM
3. Run updates:

   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0
   
   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   

4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: BaseClean_$(Get-Date -Format yyyyMMdd)
7. Update SnapshotName in runner/config.yaml
8. Delete the old snapshot after confirming new one works for 1 week

---

10. SHA256 Pinning for Tier-1 Toolchain Installers

Homelab policy: partial-coverage pinning is acceptable. Pin only the installers that are re-downloaded as part of template provisioning (not installers already cached in the ISO or in F:\CI\ISO\).

Priority targets (descending risk):

Installer	Script	Where to pin
Git for Windows .exe	template/Install-CIToolchain-WinBuild2025.ps1	-sha256 param or Get-FileHash check after download
7-Zip .msi / .exe	template/Install-CIToolchain-WinBuild2025.ps1	same
Python .exe	template/Install-CIToolchain-WinBuild2025.ps1	same
.NET SDK install script	template/Install-CIToolchain-WinBuild2025.ps1	HTTPS only; hash less critical
Ubuntu cloud VMDK	template/Deploy-LinuxBuild2404.ps1	already implemented via -VmdkSha256 parameter

Implementation pattern (PS 5.1):

# After downloading $installerPath:
if ($ExpectedSha256 -ne '') {
    $actual = (Get-FileHash -Path $installerPath -Algorithm SHA256).Hash
    if ($actual -ne $ExpectedSha256.ToUpper()) {
        throw "SHA256 mismatch for $installerPath.`n  Expected: $ExpectedSha256`n  Actual:   $actual"
    }
    Write-Host "[Install] SHA256 OK: $installerPath"
}

Pin values must be updated each time a new installer version is adopted. Store the expected hash in the script's parameter default or in a companion .sha256 sidecar file next to the cached installer in F:\CI\ISO\.