local-ci-cd-system/docs/BEST-PRACTICES.md

# Best Practices — Stability, Security & Operations

## 1. Credential Management

### Do NOT store credentials in scripts or config files

The scripts use `-GuestCredentialTarget` (a Windows Credential Manager target name)
rather than plaintext username/password parameters. Store credentials once:

```powershell
# Run on host (once, before first CI job)
cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:YourStrongPassword
```

Retrieve in scripts via the `CredentialManager` PowerShell module:

```powershell
Install-Module CredentialManager -Scope CurrentUser
$cred = Get-StoredCredential -Target 'BuildVMGuest'
```

### Rotate credentials quarterly

1. Update password in the template VM (requires rebuilding `BaseClean` snapshot)
2. Update Windows Credential Manager on the host:
   ```
   cmdkey /generic:BuildVMGuest /user:MACHINENAME\ci_build /pass:NewPassword
   ```
3. No script changes required — they reference the target name, not the password.

---

## 2. WinRM Security — HTTPS/5986 (implementato 2026-05-10)

### Setup attuale (HTTPS / port 5986)

`Deploy-WinBuild2025.ps1` post-install.ps1 crea un certificato self-signed e configura
il listener HTTPS/5986 **prima** dello snapshot `BaseClean`. `AllowUnencrypted=false`.

- Build VMs su VMnet8 NAT (192.168.79.0/24) — accesso solo dall'host
- Port 5986 firewall rule ristretta a `RemoteAddress '192.168.79.0/24'`
- Credentials via Windows Credential Manager (target `BuildVMGuest`)

Tutti gli script host usano:
```powershell
$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL -Authentication Basic `
    -Credential $cred -SessionOption $sessionOptions
```

> `-SkipCACheck`/`-SkipCNCheck` sono accettabili per un cert self-signed in lab isolato.
> Non usare contro macchine accessibili dall'esterno — usare una CA trusted in quel caso.

---

## 2.1. Threat Model — Disabled Security Features

### Current state: Defender, Firewall, and UAC disabled

The template VM disables Windows Defender, Windows Firewall, and User Account Control (UAC).
This is **intentional** — not a bug, not an oversight. Each has tradeoffs:

| Feature                                 | Disabled? | Why                                                                            | Cost if enabled                                                                        |
| --------------------------------------- | --------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| **Windows Defender**                    | Yes       | Real-time AV scanning blocks .NET compilation, Python wheels, and npm installs | 5–10 min per build overhead; false positives on dev tools                              |
| **Windows Firewall**                    | Yes       | Blocks inbound WinRM even with rules; requires Domain/Home profile tuning      | Complex rules; fragile across OS updates                                               |
| **UAC (LocalAccountTokenFilterPolicy)** | Yes       | Prevents non-elevated WinRM scripts from running builds                        | Requires built-in Administrator account; WinRM behaves like a user with limited rights |

### When this threat model is acceptable

Current threat model is **safe** if **ALL** of these are true:

1. **Isolated lab environment** — Build VMs exist only on VMnet8 (NAT), not on host LAN.
2. **No shared resources** — Host is not shared with untrusted users or concurrent CI systems.
3. **Trusted source code** — Code being built is from trusted repositories (internal team only).
4. **No external access** — VMnet8 is not bridged or exposed to corporate LAN or internet.
5. **Act_runner is trusted** — The act_runner service token cannot be used to access host resources outside the isolated network.

If all conditions hold, the attack surface is limited to:
- Network eavesdropping on 192.168.79.0/24 (mitigated: WinRM is HTTPS)
- Code injection via malicious commits (mitigated: code review process)
- Privilege escalation from VM to host (mitigated: VMs are ephemeral; no persistence)

### When the model breaks down

**Do NOT use this configuration if:**

- ❌ **Third-party code builds** — Running untrusted vendor code (open-source projects, third-party libraries with build scripts)
- ❌ **Shared build machine** — Other teams or processes share the host CPU/storage
- ❌ **LAN-exposed network** — VMnet8 is bridged to corporate LAN or internet
- ❌ **Host resource sharing** — Build VMs can access host shares, USB drives, or external storage
- ❌ **Long-lived VMs** — VMs are not destroyed after each build (antivirus blind spot for persistence)

In these scenarios, disabled AV and firewall create **unacceptable risk**.

### Mitigations if constraints change

If you must run in a less-isolated environment, re-enable protections **with cost awareness**:

#### Option 1: Re-enable Firewall only (lowest cost)
```powershell
# In template VM via WinRM, before taking BaseClean snapshot:
Set-NetFirewallProfile -Profile Domain, Public, Private -Enabled $true
# Add inbound rule for WinRM listener
New-NetFirewallRule -Name "WinRM-HTTPS" `
    -DisplayName "Windows Remote Management (HTTPS)" `
    -Direction Inbound `
    -LocalPort 5986 `
    -Protocol TCP `
    -Action Allow
```
**Cost:** 30–60 seconds per build (firewall rule evaluation + logging).  
**Benefit:** Blocks outbound malware callbacks if VM is compromised.

#### Option 2: Re-enable Defender with exclusions (moderate cost)
```powershell
# In template VM, enable Defender but exclude build directories:
Enable-MpComputerDefault   # Re-enable Defender
Add-MpPreference -ExclusionPath @(
    'C:\Build',
    'C:\Users\ci_build\AppData\Local\Microsoft\dotnet',
    'C:\Users\ci_build\AppData\Roaming\npm'
) -Force
# Reduce scanning aggressiveness:
Set-MpPreference -DisableRealtimeMonitoring $false -DisableBehaviorMonitoring $true
```
**Cost:** 2–5 min per build (initial scan; exclusions help but don't eliminate overhead).  
**Benefit:** Detects known malware uploaded in build artifacts.

#### Option 3: Enable UAC for elevated builds only (requires refactor)
```powershell
# NOT RECOMMENDED without major refactoring.
# WinRM remote commands run as non-elevated user; builds fail.
# Requires either:
#   - Running WinRM as built-in Administrator (security anti-pattern)
#   - Adding explicit runas prompts (breaks automation)
#   - Using Windows Task Scheduler instead of WinRM (complexity)
```

### Audit and sign-off

Before deploying to production or a shared host:

1. **Document the decision:** Update this section with current date and approver name.
2. **Test the mitigations:** Create test clone, enable firewall/AV, measure build time overhead.
3. **Establish monitoring:** Run Watch-RunnerHealth.ps1 continuously; alert on service restarts.
4. **Plan rotation:** Schedule quarterly credential rotation (see §1 Credential Management).

---

## 3. act_runner Service Stability

### Windows Service Recovery Policy

The `Install-Runner.ps1` script configures automatic service restart on failure:
- Restart after 1st failure: 5 seconds
- Restart after 2nd failure: 10 seconds
- Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

### Monitor the service

```powershell
# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner
```

### Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

```powershell
# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}
```

---

## 4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted,
**all builds fail immediately**.

### Protection measures

1. **Never power on the template VM for reasons other than planned maintenance.**
   Configure VMware Workstation to prevent accidental starts: right-click → Settings →
   Options → Advanced → disable "Allow background snapshots".

2. **Backup the parent VMDK before any template changes:**
   ```powershell
   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   ```

3. **Keep a list of all current linked clones** before refreshing the snapshot.
   If any clone exists when you modify the parent, it may break.
   Check: `vmrun list` — should return no build VMs during maintenance window.

4. **Version the snapshot name** to make rollback easy:
   Instead of reusing "BaseClean", name snapshots `BaseClean_20260101`.
   Update `config.yaml` `envs.GITEA_CI_SNAPSHOT_NAME` when rotating.

---

## 5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be
destroyed. Run this cleanup script on host startup or as a daily scheduled task:

```powershell
# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }
```

---

## 6. Gitea Repository Configuration

### Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

### Workflow file location

Workflows **must** be at `.gitea/workflows/*.yml` (not `.github/workflows/`).

```
your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml
```

---

## 7. Logging & Observability

### act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager).
Increase verbosity for debugging:

```yaml
# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text
```

### Per-job build logs

`Invoke-CIJob.ps1` outputs timestamped phase banners. act_runner captures all
stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

```powershell
# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"
```

### Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log →
Application source "act_runner". Check with:

```powershell
Get-EventLog -LogName Application -Source '*runner*' -Newest 20
```

---

## 8. Network Topology Verification

Build VMs run on **VMnet8 (NAT)** — they have internet access, which is required
for pip/nuget package downloads at build time. Verify the expected topology:

```powershell
# From inside a build VM via WinRM — confirm NAT internet is reachable:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Host "VM has NAT internet access — expected for pip/nuget builds."
    } else {
        Write-Warning "VM cannot reach internet — pip/nuget installs will fail. Check VMware NAT service."
    }
}
```

Build VMs can reach:
- The host via VMnet8 gateway (WinRM HTTPS on port 5986)
- Internet via VMware NAT (for pip, nuget, npm at build time)
- Gitea server if on LAN reachable via NAT gateway

**Supply-chain note:** Source code is always injected by the host via WinRM zip
transfer — never cloned inside the VM using a PAT. This keeps credentials off
the VM even though the VM has outbound internet access.

---

## 9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. **During a maintenance window** (no CI jobs running):
   ```
   vmrun list   ← must be empty
   ```
2. Boot the template VM
3. Run updates:
   ```powershell
   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0

   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   ```
4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: `BaseClean_$(Get-Date -Format yyyyMMdd)`
7. Update `SnapshotName` in `runner/config.yaml`
8. Delete the old snapshot after confirming new one works for 1 week

---

## 10. SHA256 Pinning for Tier-1 Toolchain Installers

**Homelab policy**: partial-coverage pinning is acceptable. Pin only the installers
that are re-downloaded as part of template provisioning (not installers already
cached in the ISO or in `F:\CI\ISO\`).

Priority targets (descending risk):

| Installer               | Script                                          | Where to pin                                           |
| ----------------------- | ----------------------------------------------- | ------------------------------------------------------ |
| Git for Windows `.exe`  | `template/Install-CIToolchain-WinBuild2025.ps1` | `-sha256` param or `Get-FileHash` check after download |
| 7-Zip `.msi` / `.exe`   | `template/Install-CIToolchain-WinBuild2025.ps1` | same                                                   |
| Python `.exe`           | `template/Install-CIToolchain-WinBuild2025.ps1` | same                                                   |
| .NET SDK install script | `template/Install-CIToolchain-WinBuild2025.ps1` | HTTPS only; hash less critical                         |
| Ubuntu cloud VMDK       | `template/Deploy-LinuxBuild2404.ps1`            | already implemented via `-VmdkSha256` parameter        |

**Implementation pattern** (PS 5.1):

```powershell
# After downloading $installerPath:
if ($ExpectedSha256 -ne '') {
    $actual = (Get-FileHash -Path $installerPath -Algorithm SHA256).Hash
    if ($actual -ne $ExpectedSha256.ToUpper()) {
        throw "SHA256 mismatch for $installerPath.`n  Expected: $ExpectedSha256`n  Actual:   $actual"
    }
    Write-Host "[Install] SHA256 OK: $installerPath"
}
```

Pin values must be updated each time a new installer version is adopted.
Store the expected hash in the script's parameter default or in a companion
`.sha256` sidecar file next to the cached installer in `F:\CI\ISO\`.