local-ci-cd-system/docs/BEST-PRACTICES.md

Best Practices — Stability, Security & Operations

1. Credential Management

Do NOT store credentials in scripts or config files

The scripts use -GuestCredentialTarget (a Windows Credential Manager target name) rather than plaintext username/password parameters. Store credentials once:

# Run on host (once, before first CI job)
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:YourStrongPassword

Retrieve in scripts via the CredentialManager PowerShell module:

Install-Module CredentialManager -Scope CurrentUser
$cred = Get-StoredCredential -Target 'BuildVMGuest'

Rotate credentials quarterly

1. Update password in the template VM (requires rebuilding BaseClean snapshot)
2. Update Windows Credential Manager on the host:

   cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:NewPassword
   

3. No script changes required — they reference the target name, not the password.

---

2. WinRM Security

Current setup (HTTP / port 5985)

Acceptable for an isolated lab network where:

• Build VMs are on a dedicated host-only VMware network (192.168.11.0/24 — VMnet11 dedicated CI)
• No external traffic reaches port 5985
• Credentials are managed via Windows Credential Manager

Recommended upgrade: WinRM over HTTPS (port 5986)

# Inside the template VM (before taking BaseClean snapshot):

# Create self-signed certificate
$cert = New-SelfSignedCertificate `
    -Subject    "CN=$(hostname)" `
    -CertStoreLocation Cert:\LocalMachine\My `
    -KeyUsage   DigitalSignature, KeyEncipherment `
    -KeyAlgorithm RSA `
    -KeyLength  2048

# Create HTTPS WinRM listener
New-WSManInstance WinRM/Config/Listener `
    -SelectorSet @{ Transport = 'HTTPS'; Address = '*' } `
    -ValueSet    @{ Hostname = $(hostname); CertificateThumbprint = $cert.Thumbprint }

# Allow port 5986
New-NetFirewallRule -Name 'WinRM-HTTPS-CI' -DisplayName 'WinRM HTTPS CI' `
    -Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow

# On the HOST (in scripts), use HTTPS session options:
$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL `
    -Credential $cred -SessionOption $sessionOptions

-SkipCACheck is acceptable for a self-signed cert in an isolated lab. Do NOT use this against externally accessible machines.

---

3. act_runner Service Stability

Windows Service Recovery Policy

The Install-Runner.ps1 script configures automatic service restart on failure:

• Restart after 1st failure: 5 seconds
• Restart after 2nd failure: 10 seconds
• Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

Monitor the service

# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner

Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}

---

4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted, all builds fail immediately.

Protection measures

1. Never power on the template VM for reasons other than planned maintenance. Configure VMware Workstation to prevent accidental starts: right-click → Settings → Options → Advanced → disable "Allow background snapshots".

2. Backup the parent VMDK before any template changes:

   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   

3. Keep a list of all current linked clones before refreshing the snapshot. If any clone exists when you modify the parent, it may break. Check: vmrun list — should return no build VMs during maintenance window.

4. Version the snapshot name to make rollback easy: Instead of reusing "BaseClean", name snapshots BaseClean_20260101. Update config.yaml envs.GITEA_CI_SNAPSHOT_NAME when rotating.

---

5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be destroyed. Run this cleanup script on host startup or as a daily scheduled task:

# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }

---

6. Gitea Repository Configuration

Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

Workflow file location

Workflows must be at .gitea/workflows/*.yml (not .github/workflows/).

your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml

---

7. Logging & Observability

act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager). Increase verbosity for debugging:

# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text

Per-job build logs

Invoke-CIJob.ps1 outputs timestamped phase banners. act_runner captures all stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"

Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log → Application source "act_runner". Check with:

Get-EventLog -LogName Application -Source '*runner*' -Newest 20

---

8. Network Isolation Verification

After setting up the host-only VMware network, verify that build VMs cannot reach the internet (important for supply-chain security):

# From inside a build VM via WinRM:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Warning "VM has internet access — check VMware network adapter type"
    } else {
        Write-Host "VM correctly isolated (no internet)"
    }
}

Build VMs should only reach:

• The host (for WinRM connection)
• Gitea server (for git clone, if reachable via host-only network)
• NuGet cache share (host-side shared folder)

---

9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. During a maintenance window (no CI jobs running):

   vmrun list   ← must be empty
   

2. Boot the template VM
3. Run updates:

   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0
   
   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   

4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: BaseClean_$(Get-Date -Format yyyyMMdd)
7. Update SnapshotName in runner/config.yaml
8. Delete the old snapshot after confirming new one works for 1 week