local-ci-cd-system/docs/BEST-PRACTICES.md

# Best Practices — Stability, Security & Operations

## 1. Credential Management

### Do NOT store credentials in scripts or config files

The scripts use `-GuestCredentialTarget` (a Windows Credential Manager target name)
rather than plaintext username/password parameters. Store credentials once:

```powershell
# Run on host (once, before first CI job)
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:YourStrongPassword
```

Retrieve in scripts via the `CredentialManager` PowerShell module:

```powershell
Install-Module CredentialManager -Scope CurrentUser
$cred = Get-StoredCredential -Target 'BuildVMGuest'
```

### Rotate credentials quarterly

1. Update password in the template VM (requires rebuilding `BaseClean` snapshot)
2. Update Windows Credential Manager on the host:
   ```
   cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:NewPassword
   ```
3. No script changes required — they reference the target name, not the password.

---

## 2. WinRM Security

### Current setup (HTTP / port 5985)

Acceptable for an **isolated lab network** where:
- Build VMs are on a dedicated host-only VMware network (192.168.11.0/24 — VMnet11 dedicated CI)
- No external traffic reaches port 5985
- Credentials are managed via Windows Credential Manager

### Recommended upgrade: WinRM over HTTPS (port 5986)

```powershell
# Inside the template VM (before taking BaseClean snapshot):

# Create self-signed certificate
$cert = New-SelfSignedCertificate `
    -Subject    "CN=$(hostname)" `
    -CertStoreLocation Cert:\LocalMachine\My `
    -KeyUsage   DigitalSignature, KeyEncipherment `
    -KeyAlgorithm RSA `
    -KeyLength  2048

# Create HTTPS WinRM listener
New-WSManInstance WinRM/Config/Listener `
    -SelectorSet @{ Transport = 'HTTPS'; Address = '*' } `
    -ValueSet    @{ Hostname = $(hostname); CertificateThumbprint = $cert.Thumbprint }

# Allow port 5986
New-NetFirewallRule -Name 'WinRM-HTTPS-CI' -DisplayName 'WinRM HTTPS CI' `
    -Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow
```

```powershell
# On the HOST (in scripts), use HTTPS session options:
$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL `
    -Credential $cred -SessionOption $sessionOptions
```

> `-SkipCACheck` is acceptable for a self-signed cert in an isolated lab. Do NOT
> use this against externally accessible machines.

---

## 3. act_runner Service Stability

### Windows Service Recovery Policy

The `Install-Runner.ps1` script configures automatic service restart on failure:
- Restart after 1st failure: 5 seconds
- Restart after 2nd failure: 10 seconds
- Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

### Monitor the service

```powershell
# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner
```

### Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

```powershell
# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}
```

---

## 4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted,
**all builds fail immediately**.

### Protection measures

1. **Never power on the template VM for reasons other than planned maintenance.**
   Configure VMware Workstation to prevent accidental starts: right-click → Settings →
   Options → Advanced → disable "Allow background snapshots".

2. **Backup the parent VMDK before any template changes:**
   ```powershell
   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   ```

3. **Keep a list of all current linked clones** before refreshing the snapshot.
   If any clone exists when you modify the parent, it may break.
   Check: `vmrun list` — should return no build VMs during maintenance window.

4. **Version the snapshot name** to make rollback easy:
   Instead of reusing "BaseClean", name snapshots `BaseClean_20260101`.
   Update `config.yaml` `envs.GITEA_CI_SNAPSHOT_NAME` when rotating.

---

## 5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be
destroyed. Run this cleanup script on host startup or as a daily scheduled task:

```powershell
# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }
```

---

## 6. Gitea Repository Configuration

### Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

### Workflow file location

Workflows **must** be at `.gitea/workflows/*.yml` (not `.github/workflows/`).

```
your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml
```

---

## 7. Logging & Observability

### act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager).
Increase verbosity for debugging:

```yaml
# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text
```

### Per-job build logs

`Invoke-CIJob.ps1` outputs timestamped phase banners. act_runner captures all
stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

```powershell
# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"
```

### Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log →
Application source "act_runner". Check with:

```powershell
Get-EventLog -LogName Application -Source '*runner*' -Newest 20
```

---

## 8. Network Isolation Verification

After setting up the host-only VMware network, verify that build VMs cannot
reach the internet (important for supply-chain security):

```powershell
# From inside a build VM via WinRM:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Warning "VM has internet access — check VMware network adapter type"
    } else {
        Write-Host "VM correctly isolated (no internet)"
    }
}
```

Build VMs should only reach:
- The host (for WinRM connection)
- Gitea server (for git clone, if reachable via host-only network)
- NuGet cache share (host-side shared folder)

---

## 9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. **During a maintenance window** (no CI jobs running):
   ```
   vmrun list   ← must be empty
   ```
2. Boot the template VM
3. Run updates:
   ```powershell
   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0

   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   ```
4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: `BaseClean_$(Get-Date -Format yyyyMMdd)`
7. Update `SnapshotName` in `runner/config.yaml`
8. Delete the old snapshot after confirming new one works for 1 week