local-ci-cd-system/docs/BEST-PRACTICES.md

Best Practices — Stability, Security & Operations

1. Credential Management

Do NOT store credentials in scripts or config files

The guest VM credential is referenced by target name (BuildVMGuest, GITEA_CI_GUEST_CRED_TARGET) and read by the Python orchestrator via the keyring library — never as plaintext parameters.

Critical: store it in the LocalSystem vault, not your user vault. act_runner runs as the LocalSystem service account. Windows Credential Manager / keyring vaults are per-user. A credential added with cmdkey or the Credential Manager UI from an interactive admin session lands in your vault and is invisible to the runner, which then fails with Credential 'BuildVMGuest' not found in keyring.

Username must be host-qualified. The guest is a workgroup machine; NTLM rejects a bare ci_build with SEC_E_UNKNOWN_CREDENTIALS. Store it as WINBUILD-2025\ci_build (the guest computer name, i.e. the WinRM TLS certificate CN). The WinRM transport forces auth='ntlm' for the same reason (Negotiate→Kerberos is meaningless without a domain).

Store (or rotate) the credential with the helper, which writes into the SYSTEM vault via the production venv's keyring (run elevated):

.\scripts\Set-CIGuestCredential.ps1 -UserName 'WINBUILD-2025\ci_build'

It prompts for the password securely, writes it to the SYSTEM vault, and verifies the read-back as SYSTEM. Diagnose WinRM reachability/auth with .\scripts\Test-CIGuestWinRM.ps1 -IpAddress <guest-ip>.

Rotate credentials quarterly

1. Update the password in the template VM (rebuild the BaseClean snapshot).
2. Re-run Set-CIGuestCredential.ps1 -UserName 'WINBUILD-2025\ci_build' with the new password.
3. No code changes required — the orchestrator references the target name.

---

2. WinRM Security — HTTPS/5986 (implementato 2026-05-10)

Setup attuale (HTTPS / port 5986)

Deploy-WinBuild2025.ps1 post-install.ps1 crea un certificato self-signed e configura il listener HTTPS/5986 prima dello snapshot BaseClean. AllowUnencrypted=false.

• Build VMs su VMnet8 NAT (192.168.79.0/24) — accesso solo dall'host
• Port 5986 firewall rule ristretta a RemoteAddress '192.168.79.0/24'
• Credentials via Windows Credential Manager (target BuildVMGuest)

Tutti gli script host usano:

$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL -Authentication Basic `
    -Credential $cred -SessionOption $sessionOptions

-SkipCACheck/-SkipCNCheck sono accettabili per un cert self-signed in lab isolato. Non usare contro macchine accessibili dall'esterno — usare una CA trusted in quel caso.

---

2.1. Threat Model — Disabled Security Features

Current state: Defender, Firewall, and UAC disabled

The template VM disables Windows Defender, Windows Firewall, and User Account Control (UAC). This is intentional — not a bug, not an oversight. Each has tradeoffs:

Feature	Disabled?	Why	Cost if enabled
Windows Defender	Yes	Real-time AV scanning blocks .NET compilation, Python wheels, and npm installs	5–10 min per build overhead; false positives on dev tools
Windows Firewall	Yes	Blocks inbound WinRM even with rules; requires Domain/Home profile tuning	Complex rules; fragile across OS updates
UAC (LocalAccountTokenFilterPolicy)	Yes	Prevents non-elevated WinRM scripts from running builds	Requires built-in Administrator account; WinRM behaves like a user with limited rights

When this threat model is acceptable

Current threat model is safe if ALL of these are true:

1. Isolated lab environment — Build VMs exist only on VMnet8 (NAT), not on host LAN.
2. No shared resources — Host is not shared with untrusted users or concurrent CI systems.
3. Trusted source code — Code being built is from trusted repositories (internal team only).
4. No external access — VMnet8 is not bridged or exposed to corporate LAN or internet.
5. Act_runner is trusted — The act_runner service token cannot be used to access host resources outside the isolated network.

If all conditions hold, the attack surface is limited to:

• Network eavesdropping on 192.168.79.0/24 (mitigated: WinRM is HTTPS)
• Code injection via malicious commits (mitigated: code review process)
• Privilege escalation from VM to host (mitigated: VMs are ephemeral; no persistence)

When the model breaks down

Do NOT use this configuration if:

• ❌ Third-party code builds — Running untrusted vendor code (open-source projects, third-party libraries with build scripts)
• ❌ Shared build machine — Other teams or processes share the host CPU/storage
• ❌ LAN-exposed network — VMnet8 is bridged to corporate LAN or internet
• ❌ Host resource sharing — Build VMs can access host shares, USB drives, or external storage
• ❌ Long-lived VMs — VMs are not destroyed after each build (antivirus blind spot for persistence)

In these scenarios, disabled AV and firewall create unacceptable risk.

Mitigations if constraints change

If you must run in a less-isolated environment, re-enable protections with cost awareness:

Option 1: Re-enable Firewall only (lowest cost)

# In template VM via WinRM, before taking BaseClean snapshot:
Set-NetFirewallProfile -Profile Domain, Public, Private -Enabled $true
# Add inbound rule for WinRM listener
New-NetFirewallRule -Name "WinRM-HTTPS" `
    -DisplayName "Windows Remote Management (HTTPS)" `
    -Direction Inbound `
    -LocalPort 5986 `
    -Protocol TCP `
    -Action Allow

Cost: 30–60 seconds per build (firewall rule evaluation + logging).
Benefit: Blocks outbound malware callbacks if VM is compromised.

Option 2: Re-enable Defender with exclusions (moderate cost)

# In template VM, enable Defender but exclude build directories:
Enable-MpComputerDefault   # Re-enable Defender
Add-MpPreference -ExclusionPath @(
    'C:\Build',
    'C:\Users\ci_build\AppData\Local\Microsoft\dotnet',
    'C:\Users\ci_build\AppData\Roaming\npm'
) -Force
# Reduce scanning aggressiveness:
Set-MpPreference -DisableRealtimeMonitoring $false -DisableBehaviorMonitoring $true

Cost: 2–5 min per build (initial scan; exclusions help but don't eliminate overhead).
Benefit: Detects known malware uploaded in build artifacts.

Option 3: Enable UAC for elevated builds only (requires refactor)

# NOT RECOMMENDED without major refactoring.
# WinRM remote commands run as non-elevated user; builds fail.
# Requires either:
#   - Running WinRM as built-in Administrator (security anti-pattern)
#   - Adding explicit runas prompts (breaks automation)
#   - Using Windows Task Scheduler instead of WinRM (complexity)

Audit and sign-off

Before deploying to production or a shared host:

1. Document the decision: Update this section with current date and approver name.
2. Test the mitigations: Create test clone, enable firewall/AV, measure build time overhead.
3. Establish monitoring: Run Watch-RunnerHealth.ps1 continuously; alert on service restarts.
4. Plan rotation: Schedule quarterly credential rotation (see §1 Credential Management).

---

3. act_runner Service Stability

Windows Service Recovery Policy

The Install-Runner.ps1 script configures automatic service restart on failure:

• Restart after 1st failure: 5 seconds
• Restart after 2nd failure: 10 seconds
• Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

Monitor the service

# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner

Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}

---

4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted, all builds fail immediately.

Protection measures

1. Never power on the template VM for reasons other than planned maintenance. Configure VMware Workstation to prevent accidental starts: right-click → Settings → Options → Advanced → disable "Allow background snapshots".

2. Backup the parent VMDK before any template changes:

   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   

3. Keep a list of all current linked clones before refreshing the snapshot. If any clone exists when you modify the parent, it may break. Check: vmrun list — should return no build VMs during maintenance window.

4. Version the snapshot name to make rollback easy: Instead of reusing "BaseClean", name snapshots BaseClean_20260101. Update config.yaml envs.GITEA_CI_SNAPSHOT_NAME when rotating.

---

5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be destroyed. Run this cleanup script on host startup or as a daily scheduled task:

# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }

---

6. Gitea Repository Configuration

Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

Workflow file location

Workflows must be at .gitea/workflows/*.yml (not .github/workflows/).

your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml

---

7. Logging & Observability

act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager). Increase verbosity for debugging:

# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text

Per-job build logs

Invoke-CIJob.ps1 outputs timestamped phase banners. act_runner captures all stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"

Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log → Application source "act_runner". Check with:

Get-EventLog -LogName Application -Source '*runner*' -Newest 20

---

8. Network Topology Verification

Build VMs run on VMnet8 (NAT) — they have internet access, which is required for pip/nuget package downloads at build time. Verify the expected topology:

# From inside a build VM via WinRM — confirm NAT internet is reachable:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Host "VM has NAT internet access — expected for pip/nuget builds."
    } else {
        Write-Warning "VM cannot reach internet — pip/nuget installs will fail. Check VMware NAT service."
    }
}

Build VMs can reach:

• The host via VMnet8 gateway (WinRM HTTPS on port 5986)
• Internet via VMware NAT (for pip, nuget, npm at build time)
• Gitea server if on LAN reachable via NAT gateway

Supply-chain note: Source code is always injected by the host via WinRM zip transfer — never cloned inside the VM using a PAT. This keeps credentials off the VM even though the VM has outbound internet access.

---

9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. During a maintenance window (no CI jobs running):

   vmrun list   ← must be empty
   

2. Boot the template VM
3. Run updates:

   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0
   
   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   

4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: BaseClean_$(Get-Date -Format yyyyMMdd)
7. Update SnapshotName in runner/config.yaml
8. Delete the old snapshot after confirming new one works for 1 week

---

10. SHA256 Pinning for Tier-1 Toolchain Installers

Homelab policy: partial-coverage pinning is acceptable. Pin only the installers that are re-downloaded as part of template provisioning (not installers already cached in the ISO or in F:\CI\ISO\).

Priority targets (descending risk):

Installer	Script	Where to pin
Git for Windows .exe	template/Install-CIToolchain-WinBuild2025.ps1	-sha256 param or Get-FileHash check after download
7-Zip .msi / .exe	template/Install-CIToolchain-WinBuild2025.ps1	same
Python .exe	template/Install-CIToolchain-WinBuild2025.ps1	same
.NET SDK install script	template/Install-CIToolchain-WinBuild2025.ps1	HTTPS only; hash less critical
Ubuntu cloud VMDK	template/Deploy-LinuxBuild2404.ps1	already implemented via -VmdkSha256 parameter

Implementation pattern (PS 5.1):

# After downloading $installerPath:
if ($ExpectedSha256 -ne '') {
    $actual = (Get-FileHash -Path $installerPath -Algorithm SHA256).Hash
    if ($actual -ne $ExpectedSha256.ToUpper()) {
        throw "SHA256 mismatch for $installerPath.`n  Expected: $ExpectedSha256`n  Actual:   $actual"
    }
    Write-Host "[Install] SHA256 OK: $installerPath"
}

Pin values must be updated each time a new installer version is adopted. Store the expected hash in the script's parameter default or in a companion .sha256 sidecar file next to the cached installer in F:\CI\ISO\.

---

11. VMware Shared Folders (HGFS) — Write Semantics and Cache-Poisoning Risk

What UseSharedCache does

When Invoke-CIJob.ps1 -UseSharedCache is set, the composite action enables VMware shared folders on the clone VM. The host-side folders (NuGet, pip) are mounted read-write inside the guest:

Host path	Guest path (Windows)	Guest path (Linux)
F:\CI\Cache\NuGet	\\vmware-host\Shared Folders\ci-nuget-cache	/mnt/hgfs/ci-nuget-cache
F:\CI\Cache\pip	\\vmware-host\Shared Folders\ci-pip-cache	/mnt/hgfs/ci-pip-cache

The guest build command reads from (and writes to) these folders directly. This avoids re-downloading packages on every build at the cost of shared state.

Write semantics — what you must understand

Writes from any guest are immediately visible on the host and in any other concurrently running guest that has the same shared folder mounted.

Consequences:

1. A compromised or malicious build script can overwrite packages in the cache. The next build that pulls from the same cache will use the poisoned package.
2. Two concurrent builds writing to the same cache entry (e.g. the same pip wheel filename) will corrupt each other. NuGet and pip use hash-verified filenames so collisions are rare, but not impossible for mutable packages.
3. Host-side antivirus exclusions for F:\CI\Cache\ must be maintained if AV is re-enabled (see section 2.1 Threat Model).

Safe-use rules

• Use UseSharedCache only for trusted source code (internal team repos). Do NOT enable for workflows that build third-party or unknown code.
• Treat cache poisoning as a real threat if any of the following is true:
  • The built repo has external contributors
  • The repo has git submodule paths pointing to external forks
  • The build script runs pip install / nuget restore with unpinned versions
• When in doubt, omit -UseSharedCache (the default). Package downloads add 1-3 minutes per build but eliminate the shared-state risk entirely.

Cache invalidation

There is no automatic cache invalidation. To force a clean state:

# Remove all cached NuGet packages
Remove-Item 'F:\CI\Cache\NuGet\*' -Recurse -Force

# Remove all cached pip wheels
Remove-Item 'F:\CI\Cache\pip\*' -Recurse -Force

After any template toolchain upgrade, clear the cache to avoid stale package metadata. The next build repopulates it.