local-ci-cd-system/docs/BEST-PRACTICES.md

# Best Practices — Stability, Security & Operations

## 1. Credential Management

### Do NOT store credentials in scripts or config files

The guest VM credential is referenced by target name (`BuildVMGuest`,
`GITEA_CI_GUEST_CRED_TARGET`) and read by the Python orchestrator via the
`keyring` library — never as plaintext parameters.

> **Critical: store it in the LocalSystem vault, not your user vault.**
> act_runner runs as the `LocalSystem` service account. Windows Credential
> Manager / keyring vaults are **per-user**. A credential added with
> `cmdkey` or the Credential Manager UI from an interactive admin session
> lands in *your* vault and is invisible to the runner, which then fails
> with `Credential 'BuildVMGuest' not found in keyring`.

> **Username must be host-qualified.** The guest is a workgroup machine;
> NTLM rejects a bare `ci_build` with `SEC_E_UNKNOWN_CREDENTIALS`. Store
> it as `WINBUILD-2025\ci_build` (the guest computer name, i.e. the WinRM
> TLS certificate CN). The WinRM transport forces `auth='ntlm'` for the
> same reason (Negotiate→Kerberos is meaningless without a domain).

Store (or rotate) the credential with the helper, which writes into the
SYSTEM vault via the production venv's `keyring` (run elevated):

```powershell
.\scripts\Set-CIGuestCredential.ps1 -UserName 'WINBUILD-2025\ci_build'
```

It prompts for the password securely, writes it to the SYSTEM vault, and
verifies the read-back as SYSTEM. Diagnose WinRM reachability/auth with
`.\scripts\Test-CIGuestWinRM.ps1 -IpAddress <guest-ip>`.

### Rotate credentials quarterly

1. Update the password in the template VM (rebuild the `BaseClean` snapshot).
2. Re-run `Set-CIGuestCredential.ps1 -UserName 'WINBUILD-2025\ci_build'`
   with the new password.
3. No code changes required — the orchestrator references the target name.

---

## 2. WinRM Security — HTTPS/5986 (implementato 2026-05-10)

### Setup attuale (HTTPS / port 5986)

`Deploy-WinBuild2025.ps1` post-install.ps1 crea un certificato self-signed e configura
il listener HTTPS/5986 **prima** dello snapshot `BaseClean`. `AllowUnencrypted=false`.

- Build VMs su VMnet8 NAT (192.168.79.0/24) — accesso solo dall'host
- Port 5986 firewall rule ristretta a `RemoteAddress '192.168.79.0/24'`
- Credentials via Windows Credential Manager (target `BuildVMGuest`)

Tutti gli script host usano:
```powershell
$sessionOptions = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$session = New-PSSession -ComputerName $ip -Port 5986 -UseSSL -Authentication Basic `
    -Credential $cred -SessionOption $sessionOptions
```

> `-SkipCACheck`/`-SkipCNCheck` sono accettabili per un cert self-signed in lab isolato.
> Non usare contro macchine accessibili dall'esterno — usare una CA trusted in quel caso.

---

## 2.1. Threat Model — Disabled Security Features

### Current state: Defender, Firewall, and UAC disabled

The template VM disables Windows Defender, Windows Firewall, and User Account Control (UAC).
This is **intentional** — not a bug, not an oversight. Each has tradeoffs:

| Feature                                 | Disabled? | Why                                                                            | Cost if enabled                                                                        |
| --------------------------------------- | --------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| **Windows Defender**                    | Yes       | Real-time AV scanning blocks .NET compilation, Python wheels, and npm installs | 5–10 min per build overhead; false positives on dev tools                              |
| **Windows Firewall**                    | Yes       | Blocks inbound WinRM even with rules; requires Domain/Home profile tuning      | Complex rules; fragile across OS updates                                               |
| **UAC (LocalAccountTokenFilterPolicy)** | Yes       | Prevents non-elevated WinRM scripts from running builds                        | Requires built-in Administrator account; WinRM behaves like a user with limited rights |

### When this threat model is acceptable

Current threat model is **safe** if **ALL** of these are true:

1. **Isolated lab environment** — Build VMs exist only on VMnet8 (NAT), not on host LAN.
2. **No shared resources** — Host is not shared with untrusted users or concurrent CI systems.
3. **Trusted source code** — Code being built is from trusted repositories (internal team only).
4. **No external access** — VMnet8 is not bridged or exposed to corporate LAN or internet.
5. **Act_runner is trusted** — The act_runner service token cannot be used to access host resources outside the isolated network.

If all conditions hold, the attack surface is limited to:
- Network eavesdropping on 192.168.79.0/24 (mitigated: WinRM is HTTPS)
- Code injection via malicious commits (mitigated: code review process)
- Privilege escalation from VM to host (mitigated: VMs are ephemeral; no persistence)

### When the model breaks down

**Do NOT use this configuration if:**

- ❌ **Third-party code builds** — Running untrusted vendor code (open-source projects, third-party libraries with build scripts)
- ❌ **Shared build machine** — Other teams or processes share the host CPU/storage
- ❌ **LAN-exposed network** — VMnet8 is bridged to corporate LAN or internet
- ❌ **Host resource sharing** — Build VMs can access host shares, USB drives, or external storage
- ❌ **Long-lived VMs** — VMs are not destroyed after each build (antivirus blind spot for persistence)

In these scenarios, disabled AV and firewall create **unacceptable risk**.

### Mitigations if constraints change

If you must run in a less-isolated environment, re-enable protections **with cost awareness**:

#### Option 1: Re-enable Firewall only (lowest cost)
```powershell
# In template VM via WinRM, before taking BaseClean snapshot:
Set-NetFirewallProfile -Profile Domain, Public, Private -Enabled $true
# Add inbound rule for WinRM listener
New-NetFirewallRule -Name "WinRM-HTTPS" `
    -DisplayName "Windows Remote Management (HTTPS)" `
    -Direction Inbound `
    -LocalPort 5986 `
    -Protocol TCP `
    -Action Allow
```
**Cost:** 30–60 seconds per build (firewall rule evaluation + logging).  
**Benefit:** Blocks outbound malware callbacks if VM is compromised.

#### Option 2: Re-enable Defender with exclusions (moderate cost)
```powershell
# In template VM, enable Defender but exclude build directories:
Enable-MpComputerDefault   # Re-enable Defender
Add-MpPreference -ExclusionPath @(
    'C:\Build',
    'C:\Users\ci_build\AppData\Local\Microsoft\dotnet',
    'C:\Users\ci_build\AppData\Roaming\npm'
) -Force
# Reduce scanning aggressiveness:
Set-MpPreference -DisableRealtimeMonitoring $false -DisableBehaviorMonitoring $true
```
**Cost:** 2–5 min per build (initial scan; exclusions help but don't eliminate overhead).  
**Benefit:** Detects known malware uploaded in build artifacts.

#### Option 3: Enable UAC for elevated builds only (requires refactor)
```powershell
# NOT RECOMMENDED without major refactoring.
# WinRM remote commands run as non-elevated user; builds fail.
# Requires either:
#   - Running WinRM as built-in Administrator (security anti-pattern)
#   - Adding explicit runas prompts (breaks automation)
#   - Using Windows Task Scheduler instead of WinRM (complexity)
```

### Audit and sign-off

Before deploying to production or a shared host:

1. **Document the decision:** Update this section with current date and approver name.
2. **Test the mitigations:** Create test clone, enable firewall/AV, measure build time overhead.
3. **Establish monitoring:** Run Watch-RunnerHealth.ps1 continuously; alert on service restarts.
4. **Plan rotation:** Schedule quarterly credential rotation (see §1 Credential Management).

---

## 3. act_runner Service Stability

### Windows Service Recovery Policy

The `Install-Runner.ps1` script configures automatic service restart on failure:
- Restart after 1st failure: 5 seconds
- Restart after 2nd failure: 10 seconds
- Restart after subsequent: 30 seconds

Verify in Services → act_runner → Properties → Recovery tab.

### Monitor the service

```powershell
# Check service status
Get-Service act_runner | Select-Object Status, StartType

# View last 50 log lines
Get-EventLog -LogName Application -Source act_runner -Newest 50 | Format-List

# Restart if needed
Restart-Service act_runner
```

### Scheduled health check (optional)

Create a scheduled task that verifies the runner appears "Online" in Gitea via API:

```powershell
# Check runner status via Gitea API every 15 minutes
$response = Invoke-RestMethod `
    -Uri     "http://gitea.local/api/v1/admin/runners" `
    -Headers @{ Authorization = "token $env:GITEA_API_TOKEN" }

$runnerOnline = $response | Where-Object { $_.name -eq 'local-windows-runner' -and $_.status -eq 'online' }
if (-not $runnerOnline) {
    # Send alert (email, webhook, etc.) or restart service
    Restart-Service act_runner
}
```

---

## 4. Template VM Integrity

The "BaseClean" snapshot is the foundation of every build. If it is corrupted,
**all builds fail immediately**.

### Protection measures

1. **Never power on the template VM for reasons other than planned maintenance.**
   Configure VMware Workstation to prevent accidental starts: right-click → Settings →
   Options → Advanced → disable "Allow background snapshots".

2. **Backup the parent VMDK before any template changes:**
   ```powershell
   # Before any template maintenance
   $templateDir = 'F:\CI\Templates\WinBuild'
   $backupDir   = "F:\CI\Backups\Template_$(Get-Date -Format yyyyMMdd)"
   Copy-Item $templateDir $backupDir -Recurse
   ```

3. **Keep a list of all current linked clones** before refreshing the snapshot.
   If any clone exists when you modify the parent, it may break.
   Check: `vmrun list` — should return no build VMs during maintenance window.

4. **Version the snapshot name** to make rollback easy:
   Instead of reusing "BaseClean", name snapshots `BaseClean_20260101`.
   Update `config.yaml` `envs.GITEA_CI_SNAPSHOT_NAME` when rotating.

---

## 5. Orphaned VM Cleanup

If the host loses power mid-job or act_runner crashes, ephemeral VMs may not be
destroyed. Run this cleanup script on host startup or as a daily scheduled task:

```powershell
# Cleanup-OrphanedBuildVMs.ps1
$vmrun      = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe'
$cloneBase  = 'F:\CI\BuildVMs'
$maxAgeHours = 4   # No job should run longer than 4 hours

Get-ChildItem $cloneBase -Directory |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddHours(-$maxAgeHours) } |
    ForEach-Object {
        $vmx = Get-ChildItem $_.FullName -Filter '*.vmx' | Select-Object -First 1
        if ($vmx) {
            Write-Host "Cleaning orphan: $($vmx.FullName)"
            & $vmrun -T ws stop $vmx.FullName hard 2>$null
            & $vmrun -T ws deleteVM $vmx.FullName 2>$null
        }
        Remove-Item $_.FullName -Recurse -Force -ErrorAction SilentlyContinue
    }
```

---

## 6. Gitea Repository Configuration

### Required repository settings for workflows to run

1. Enable Actions for the repository: Settings → Repository → Actions → Enable
2. Add secrets if needed: Settings → Secrets and Variables → Actions
3. Protect main branch: Settings → Branches → Branch protection rules

### Workflow file location

Workflows **must** be at `.gitea/workflows/*.yml` (not `.github/workflows/`).

```
your-repo/
└── .gitea/
    └── workflows/
        └── build.yml    ← copy from gitea/workflow-example.yml
```

---

## 7. Logging & Observability

### act_runner logs

The runner daemon writes to stdout (captured by the Windows service manager).
Increase verbosity for debugging:

```yaml
# runner/config.yaml
log:
  level: debug    # change from "info" to "debug"
  format: text
```

### Per-job build logs

`Invoke-CIJob.ps1` outputs timestamped phase banners. act_runner captures all
stdout/stderr and uploads it to Gitea Actions → job log viewer.

For persistent local logs:

```powershell
# In your workflow YAML, redirect output to a log file:
- name: Build in ephemeral VM
  shell: pwsh
  run: |
    .\scripts\Invoke-CIJob.ps1 ... *>&1 | Tee-Object -FilePath "F:\CI\Logs\job-${{ github.run_id }}.log"
```

### Windows Event Log

act_runner (when installed as a service) writes events to Windows Event Log →
Application source "act_runner". Check with:

```powershell
Get-EventLog -LogName Application -Source '*runner*' -Newest 20
```

---

## 8. Network Topology Verification

Build VMs run on **VMnet8 (NAT)** — they have internet access, which is required
for pip/nuget package downloads at build time. Verify the expected topology:

```powershell
# From inside a build VM via WinRM — confirm NAT internet is reachable:
Invoke-Command -Session $session -ScriptBlock {
    $result = Test-Connection 8.8.8.8 -Count 1 -Quiet
    if ($result) {
        Write-Host "VM has NAT internet access — expected for pip/nuget builds."
    } else {
        Write-Warning "VM cannot reach internet — pip/nuget installs will fail. Check VMware NAT service."
    }
}
```

Build VMs can reach:
- The host via VMnet8 gateway (WinRM HTTPS on port 5986)
- Internet via VMware NAT (for pip, nuget, npm at build time)
- Gitea server if on LAN reachable via NAT gateway

**Supply-chain note:** Source code is always injected by the host via WinRM zip
transfer — never cloned inside the VM using a PAT. This keeps credentials off
the VM even though the VM has outbound internet access.

---

## 9. Updating the Build Toolchain

When a new .NET SDK or VS Build Tools version is released:

1. **During a maintenance window** (no CI jobs running):
   ```
   vmrun list   ← must be empty
   ```
2. Boot the template VM
3. Run updates:
   ```powershell
   # Update .NET SDK
   & "C:\Users\ci_build\AppData\Local\Microsoft\dotnet\dotnet-install.ps1" -Channel 8.0

   # Update VS Build Tools via Visual Studio Installer
   "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vs_installer.exe" update --quiet --norestart
   ```
4. Verify tools work (run a test build manually)
5. Shut down VM
6. Take new snapshot: `BaseClean_$(Get-Date -Format yyyyMMdd)`
7. Update `SnapshotName` in `runner/config.yaml`
8. Delete the old snapshot after confirming new one works for 1 week

---

## 10. SHA256 Pinning for Tier-1 Toolchain Installers

**Homelab policy**: partial-coverage pinning is acceptable. Pin only the installers
that are re-downloaded as part of template provisioning (not installers already
cached in the ISO or in `F:\CI\ISO\`).

Priority targets (descending risk):

| Installer               | Script                                          | Where to pin                                           |
| ----------------------- | ----------------------------------------------- | ------------------------------------------------------ |
| Git for Windows `.exe`  | `template/Install-CIToolchain-WinBuild2025.ps1` | `-sha256` param or `Get-FileHash` check after download |
| 7-Zip `.msi` / `.exe`   | `template/Install-CIToolchain-WinBuild2025.ps1` | same                                                   |
| Python `.exe`           | `template/Install-CIToolchain-WinBuild2025.ps1` | same                                                   |
| .NET SDK install script | `template/Install-CIToolchain-WinBuild2025.ps1` | HTTPS only; hash less critical                         |
| Ubuntu cloud VMDK       | `template/Deploy-LinuxBuild2404.ps1`            | already implemented via `-VmdkSha256` parameter        |

**Implementation pattern** (PS 5.1):

```powershell
# After downloading $installerPath:
if ($ExpectedSha256 -ne '') {
    $actual = (Get-FileHash -Path $installerPath -Algorithm SHA256).Hash
    if ($actual -ne $ExpectedSha256.ToUpper()) {
        throw "SHA256 mismatch for $installerPath.`n  Expected: $ExpectedSha256`n  Actual:   $actual"
    }
    Write-Host "[Install] SHA256 OK: $installerPath"
}
```

Pin values must be updated each time a new installer version is adopted.
Store the expected hash in the script's parameter default or in a companion
`.sha256` sidecar file next to the cached installer in `F:\CI\ISO\`.

---

## 11. VMware Shared Folders (HGFS) — Write Semantics and Cache-Poisoning Risk

### What UseSharedCache does

When `Invoke-CIJob.ps1 -UseSharedCache` is set, the composite action enables
VMware shared folders on the clone VM. The host-side folders (NuGet, pip) are
mounted read-write inside the guest:

| Host path              | Guest path (Windows)             | Guest path (Linux)        |
| ---------------------- | -------------------------------- | ------------------------- |
| `F:\CI\Cache\NuGet`    | `\\vmware-host\Shared Folders\ci-nuget-cache` | `/mnt/hgfs/ci-nuget-cache` |
| `F:\CI\Cache\pip`      | `\\vmware-host\Shared Folders\ci-pip-cache`   | `/mnt/hgfs/ci-pip-cache`  |

The guest build command reads from (and writes to) these folders directly. This
avoids re-downloading packages on every build at the cost of shared state.

### Write semantics — what you must understand

**Writes from any guest are immediately visible on the host and in any other
concurrently running guest that has the same shared folder mounted.**

Consequences:
1. A compromised or malicious build script can overwrite packages in the cache.
   The next build that pulls from the same cache will use the poisoned package.
2. Two concurrent builds writing to the same cache entry (e.g. the same pip
   wheel filename) will corrupt each other. NuGet and pip use hash-verified
   filenames so collisions are rare, but not impossible for mutable packages.
3. Host-side antivirus exclusions for `F:\CI\Cache\` must be maintained if AV
   is re-enabled (see section 2.1 Threat Model).

### Safe-use rules

- Use `UseSharedCache` **only** for trusted source code (internal team repos).
  Do NOT enable for workflows that build third-party or unknown code.
- Treat cache poisoning as a real threat if any of the following is true:
  - The built repo has external contributors
  - The repo has `git submodule` paths pointing to external forks
  - The build script runs `pip install` / `nuget restore` with unpinned versions
- When in doubt, omit `-UseSharedCache` (the default). Package downloads add
  1-3 minutes per build but eliminate the shared-state risk entirely.

### Cache invalidation

There is no automatic cache invalidation. To force a clean state:
```powershell
# Remove all cached NuGet packages
Remove-Item 'F:\CI\Cache\NuGet\*' -Recurse -Force

# Remove all cached pip wheels
Remove-Item 'F:\CI\Cache\pip\*' -Recurse -Force
```

After any template toolchain upgrade, clear the cache to avoid stale package
metadata. The next build repopulates it.