Simone
3592dcab78
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12. - Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps. - Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
352 lines
12 KiB
Markdown
352 lines
12 KiB
Markdown
# CI System Runbook
|
|
|
|
Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation.
|
|
|
|
---
|
|
|
|
## 1. Runner offline in Gitea UI
|
|
|
|
**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline.
|
|
Queued jobs stay pending indefinitely.
|
|
|
|
**Triage**:
|
|
```powershell
|
|
# Check service state
|
|
Get-Service act_runner
|
|
|
|
# Last 50 lines of runner log
|
|
Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50
|
|
|
|
# Check registration file is intact
|
|
Test-Path 'F:\CI\act_runner\.runner'
|
|
```
|
|
|
|
**Fix**:
|
|
```powershell
|
|
# Restart the service
|
|
Restart-Service act_runner
|
|
|
|
# Verify it came back online (wait ~10s then check Gitea UI)
|
|
Get-Service act_runner | Select-Object Status, StartType
|
|
|
|
# If service won't start, check NSSM log
|
|
& 'C:\nssm\nssm.exe' status act_runner
|
|
```
|
|
|
|
If the `.runner` registration file is missing or corrupt, re-register:
|
|
```powershell
|
|
cd F:\CI\act_runner
|
|
.\act_runner.exe register --no-interactive `
|
|
--instance http://10.10.20.11:3100 `
|
|
--token <token-from-gitea-admin> `
|
|
--name local-windows-runner `
|
|
--labels "windows-build:host,dotnet:host,msbuild:host"
|
|
```
|
|
|
|
**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`.
|
|
|
|
---
|
|
|
|
## 2. All builds fail in Phase 2 (VM clone / start)
|
|
|
|
**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like:
|
|
- `vmrun clone failed`
|
|
- `vmrun start failed`
|
|
- `Template VMX not found`
|
|
- `Could not detect VM IP address`
|
|
|
|
**Triage**:
|
|
```powershell
|
|
# List all running VMs
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list
|
|
|
|
# Check template VMX exists and is accessible
|
|
Test-Path 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
|
|
|
# Check for orphaned clones that may be consuming disk
|
|
Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime
|
|
|
|
# Check disk free space
|
|
Get-PSDrive F | Select-Object Name, Free, Used
|
|
|
|
# Check for a stuck vm-start lock from a crashed job
|
|
Test-Path 'F:\CI\State\vm-start.lock'
|
|
```
|
|
|
|
**Fix** — by root cause:
|
|
|
|
*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`.
|
|
|
|
*Parent VMDK locked* (VMware left a lock file after host crash):
|
|
```powershell
|
|
# Stop all VMs
|
|
& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' hard
|
|
# Delete lock files
|
|
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force -ErrorAction SilentlyContinue
|
|
```
|
|
|
|
*Snapshot missing* (`BaseClean` was deleted or renamed):
|
|
```powershell
|
|
# List snapshots on template VM
|
|
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
|
# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name
|
|
```
|
|
|
|
*Disk full* (clone delta files need space):
|
|
```powershell
|
|
# Emergency cleanup — remove all orphaned clones
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0
|
|
# Then run retention
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3
|
|
```
|
|
|
|
*Stale vm-start lock* (from a job that crashed without cleanup):
|
|
```powershell
|
|
Remove-Item 'F:\CI\State\vm-start.lock' -Force
|
|
Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force
|
|
```
|
|
|
|
**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed.
|
|
|
|
---
|
|
|
|
## 3. Builds are slow
|
|
|
|
**Symptom**: jobs that previously completed in ~3 min now take 8+ min.
|
|
Phase durations visible in `F:\CI\Logs\<jobId>\invoke-ci.jsonl`.
|
|
|
|
**Triage**:
|
|
```powershell
|
|
# Check disk free space (below 50 GB = fragmented writes)
|
|
Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}
|
|
|
|
# Check active VM CPU usage (Task Manager or:)
|
|
Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending
|
|
|
|
# Check VMnet8 NAT adapter status
|
|
Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' }
|
|
|
|
# Parse JSONL for per-phase durations (requires jq or manual inspection)
|
|
# Each phase has a 'start' and 'success' event — diff the 'ts' fields.
|
|
Get-Content 'F:\CI\Logs\<jobId>\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status
|
|
```
|
|
|
|
**Fix** — by root cause:
|
|
|
|
*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK.
|
|
|
|
*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2.
|
|
|
|
*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running.
|
|
|
|
*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`.
|
|
|
|
**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem).
|
|
|
|
---
|
|
|
|
## 4. Template VMX corrupt after host crash
|
|
|
|
**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state.
|
|
|
|
**Triage**:
|
|
```powershell
|
|
# Try starting the template directly in VMware Workstation UI
|
|
# If it reports "configuration file error" or "disk lock", proceed below.
|
|
|
|
# Check for lock files
|
|
Get-ChildItem 'F:\CI\Templates\WinBuild2025\' -Recurse -Filter '*.lck'
|
|
|
|
# Check if backup exists
|
|
Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5
|
|
```
|
|
|
|
**Fix**:
|
|
|
|
*Lock files only* (common after hard shutdown):
|
|
```powershell
|
|
# Ensure no VMware processes are running
|
|
Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force
|
|
# Remove locks
|
|
Remove-Item 'F:\CI\Templates\WinBuild2025\*.lck' -Recurse -Force
|
|
# Test clone
|
|
& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
|
```
|
|
|
|
*VMX or VMDK truly corrupt — restore from backup*:
|
|
```powershell
|
|
# Stop all CI activity first
|
|
Stop-Service act_runner
|
|
|
|
# Identify latest backup
|
|
$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
|
|
Write-Host "Restoring from: $($latest.FullName)"
|
|
|
|
# Replace template directory
|
|
Remove-Item 'F:\CI\Templates\WinBuild2025\' -Recurse -Force
|
|
Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild2025\' -Recurse
|
|
|
|
# Restart runner
|
|
Start-Service act_runner
|
|
```
|
|
|
|
*No backup exists*: must re-provision the template from scratch.
|
|
Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare).
|
|
Estimated time: 2-4 hours including Windows Update.
|
|
|
|
**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact.
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
| Symptom | First command |
|
|
| ------------------- | ------------------------------------------------------------------ |
|
|
| Runner offline | `Get-Service act_runner`, then `Restart-Service act_runner` |
|
|
| Phase 2 clone fails | `Test-Path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` |
|
|
| Disk full | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1` |
|
|
| Stale lock | `Remove-Item F:\CI\State\vm-start.lock` |
|
|
| Slow builds | Check `invoke-ci.jsonl` phase timestamps; check disk I/O |
|
|
| Template corrupt | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` |
|
|
| Snapshot missing | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME` |
|
|
| IP collision | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity` |
|
|
|
|
---
|
|
|
|
## 5. Template Refresh Procedure
|
|
|
|
Use this procedure when the template OS needs updated packages, toolchain upgrades, or
|
|
a new snapshot. Run on the **host** with an elevated PowerShell 5.1 session.
|
|
|
|
### 5.1 Pre-flight
|
|
|
|
```powershell
|
|
# Stop the runner so no CI jobs start during the refresh
|
|
Stop-Service act_runner
|
|
|
|
# Verify no clone VMs are running
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
|
|
# Expected: "Total running VMs: 0"
|
|
|
|
# Backup the existing template (keeps last 3 by default)
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates
|
|
```
|
|
|
|
### 5.2 Boot the template
|
|
|
|
**Windows (WinBuild2025)**:
|
|
```powershell
|
|
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
|
```
|
|
|
|
**Linux (LinuxBuild2404)**:
|
|
```powershell
|
|
$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
|
```
|
|
|
|
### 5.3 Apply updates inside the template
|
|
|
|
**Windows** — connect via WinRM or open the VMware console, then run the Prepare
|
|
script from the host:
|
|
```powershell
|
|
$vmxWin = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
|
$credTgt = 'BuildVMGuest' # Windows Credential Manager target
|
|
$cred = Get-StoredCredential -Target $credTgt # requires CredentialManager module
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
|
|
-VMXPath $vmxWin `
|
|
-Credential $cred
|
|
```
|
|
|
|
**Linux** — SSH into the template and run the toolchain script:
|
|
```powershell
|
|
$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
|
# Get IP (wait for VMware Tools if needed)
|
|
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
|
|
# Apply updates
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
|
|
-VMXPath $vmxLin `
|
|
-SshKeyPath 'F:\CI\keys\ci_linux'
|
|
```
|
|
|
|
Alternatively, run `Install-CIToolchain-WinBuild2025.ps1` / `Install-CIToolchain-Linux2404.sh`
|
|
manually inside the guest to apply only toolchain changes without the full Prepare script.
|
|
|
|
### 5.4 Shut down and snapshot
|
|
|
|
```powershell
|
|
# Shut down gracefully (wait up to 120 s)
|
|
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' # or Linux vmx
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft
|
|
|
|
# Name: BaseClean_yyyyMMdd (keeps old name for rollback reference)
|
|
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
|
|
Write-Host "Snapshot created: $snapshotName"
|
|
```
|
|
|
|
Confirm no `.vmem` / `.vmsn` files exist before snapshotting (see AGENTS.md item 9):
|
|
```powershell
|
|
Get-ChildItem (Split-Path $vmx) -Filter '*.vmem' # must be empty
|
|
```
|
|
|
|
### 5.5 Validate
|
|
|
|
```powershell
|
|
# Run the validation script
|
|
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
|
|
-VMXPath $vmx -SnapshotName $snapshotName
|
|
```
|
|
|
|
For Linux, also run a quick SSH smoke-test from the host:
|
|
```powershell
|
|
Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
|
|
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
|
|
-Command 'gcc --version && cmake --version' -PassThru
|
|
$result.Output
|
|
```
|
|
|
|
### 5.6 Run a smoke workflow
|
|
|
|
Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI.
|
|
Confirm the job uses the new snapshot and completes successfully.
|
|
|
|
### 5.7 Promote the new snapshot
|
|
|
|
Update `GITEA_CI_SNAPSHOT_NAME` in `runner/config.yaml` and redeploy:
|
|
```powershell
|
|
# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
|
|
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'
|
|
|
|
# Deploy config and restart runner
|
|
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
|
'F:\CI\act_runner\config.yaml' -Force
|
|
Restart-Service act_runner
|
|
```
|
|
|
|
### 5.8 Retain old snapshot 7 days, then delete
|
|
|
|
Keep the previous `BaseClean_*` snapshot for 7 days as a rollback point:
|
|
```powershell
|
|
# List existing snapshots
|
|
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx
|
|
|
|
# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
|
|
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME
|
|
```
|
|
|
|
### 5.9 Rollback procedure
|
|
|
|
If a smoke-test failure is discovered after promotion:
|
|
```powershell
|
|
# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
|
|
# (or set it back to 'BaseClean' for the permanent base)
|
|
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
|
'F:\CI\act_runner\config.yaml' -Force
|
|
Restart-Service act_runner
|
|
# The prior snapshot is still in the template — jobs will use it immediately.
|
|
```
|
|
|
|
|