1962c977ba
§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
Write-JobEvent appends one JSON line per phase transition (ts, jobId,
phase, status, data). Events emitted at: job.start, phase1.clone-repo
start/success, phase2-3b.vm-start start/success, phase4.wait-ready
start/success, phase5.build start/success, phase6.artifacts start/success,
job.success/failure (with elapsedSec and error string).
Silent no-op if $jsonLog is null (log dir setup failed). Errors in
Write-JobEvent are swallowed — logging never breaks the build.
Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.
§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
(source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
Optional -WebhookUrl for Discord/Gitea webhook notification.
Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).
§4.4 — Incident runbook (docs/RUNBOOK.md)
Four scenarios with symptom / triage commands / fix / escalation:
1. Runner offline in Gitea UI
2. All builds fail in Phase 2 (clone/start/IP)
3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
4. Template VMX corrupt after host crash (lock removal + backup restore)
Quick-reference table at the end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
159 lines
7.0 KiB
PowerShell
159 lines
7.0 KiB
PowerShell
#Requires -Version 5.1
|
|
#Requires -RunAsAdministrator
|
|
<#
|
|
.SYNOPSIS
|
|
Registers Windows Scheduled Tasks for CI system maintenance.
|
|
|
|
.DESCRIPTION
|
|
Creates or updates two tasks under the \CI\ task folder:
|
|
|
|
CI-CleanupOrphans — Runs Cleanup-OrphanedBuildVMs.ps1 every 6 hours
|
|
and at host startup. Destroys stale build VMs
|
|
(those older than -MaxAgeHours, default 4) that
|
|
were not cleaned up after a crash or timeout.
|
|
|
|
CI-RetentionPolicy — Runs Invoke-RetentionPolicy.ps1 daily at 3:00 AM
|
|
and at host startup (with a 15-min random delay).
|
|
Purges old artifact and log directories per the
|
|
configured retention window.
|
|
|
|
Both tasks run as SYSTEM with highest privilege. Idempotent: safe to re-run
|
|
after updating the scripts — existing tasks are overwritten (-Force).
|
|
|
|
.PARAMETER ScriptRoot
|
|
Directory containing the CI scripts. Default: N:\Code\Workspace\Local-CI-CD-System\scripts
|
|
|
|
.PARAMETER MaxAgeHours
|
|
Passed to Cleanup-OrphanedBuildVMs.ps1. VMs older than this are treated as
|
|
orphaned. Must exceed the longest expected build duration. Default: 4
|
|
|
|
.EXAMPLE
|
|
# Register (or update) tasks — run from an elevated PowerShell session
|
|
.\Register-CIScheduledTasks.ps1
|
|
|
|
# Preview what would be registered
|
|
.\Register-CIScheduledTasks.ps1 -WhatIf
|
|
#>
|
|
[CmdletBinding(SupportsShouldProcess)]
|
|
param(
|
|
[string] $ScriptRoot = 'N:\Code\Workspace\Local-CI-CD-System\scripts',
|
|
|
|
[ValidateRange(1, 168)]
|
|
[int] $MaxAgeHours = 4
|
|
)
|
|
|
|
Set-StrictMode -Version Latest
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
$psExe = "$env:SystemRoot\System32\WindowsPowerShell\v1.0\powershell.exe"
|
|
$taskPath = '\CI\'
|
|
$principal = New-ScheduledTaskPrincipal -UserId 'SYSTEM' -RunLevel Highest
|
|
|
|
# Ensure task folder exists (Register-ScheduledTask creates it, but be explicit)
|
|
$schedulerService = New-Object -ComObject 'Schedule.Service'
|
|
$schedulerService.Connect()
|
|
$rootFolder = $schedulerService.GetFolder('\')
|
|
try { $rootFolder.GetFolder('CI') | Out-Null }
|
|
catch { $rootFolder.CreateFolder('CI') | Out-Null }
|
|
[System.Runtime.InteropServices.Marshal]::ReleaseComObject($schedulerService) | Out-Null
|
|
|
|
# ── Task 1: CI-CleanupOrphans ─────────────────────────────────────────────────
|
|
$cleanupScript = Join-Path $ScriptRoot 'Cleanup-OrphanedBuildVMs.ps1'
|
|
if (-not (Test-Path $cleanupScript -PathType Leaf)) {
|
|
Write-Warning "Script not found — skipping CI-CleanupOrphans: $cleanupScript"
|
|
} else {
|
|
$cleanupAction = New-ScheduledTaskAction `
|
|
-Execute $psExe `
|
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$cleanupScript`" -MaxAgeHours $MaxAgeHours"
|
|
|
|
# Repeat every 6 hours indefinitely (Once trigger + RepetitionInterval)
|
|
$repeat6h = New-ScheduledTaskTrigger -Once -At '00:00' `
|
|
-RepetitionInterval (New-TimeSpan -Hours 6)
|
|
$atStartup = New-ScheduledTaskTrigger -AtStartup
|
|
|
|
$cleanupSettings = New-ScheduledTaskSettingsSet `
|
|
-ExecutionTimeLimit (New-TimeSpan -Hours 1) `
|
|
-MultipleInstances IgnoreNew `
|
|
-StartWhenAvailable
|
|
|
|
if ($PSCmdlet.ShouldProcess('CI-CleanupOrphans', 'Register/update scheduled task')) {
|
|
Register-ScheduledTask `
|
|
-TaskName 'CI-CleanupOrphans' `
|
|
-TaskPath $taskPath `
|
|
-Action $cleanupAction `
|
|
-Trigger @($repeat6h, $atStartup) `
|
|
-Principal $principal `
|
|
-Settings $cleanupSettings `
|
|
-Description "Destroys orphaned CI build VMs older than ${MaxAgeHours}h (Local CI/CD System)" `
|
|
-Force | Out-Null
|
|
Write-Host "[Register] CI-CleanupOrphans registered (every 6h + at startup, -MaxAgeHours $MaxAgeHours)." -ForegroundColor Green
|
|
}
|
|
}
|
|
|
|
# ── Task 2: CI-RetentionPolicy ────────────────────────────────────────────────
|
|
$retentionScript = Join-Path $ScriptRoot 'Invoke-RetentionPolicy.ps1'
|
|
if (-not (Test-Path $retentionScript -PathType Leaf)) {
|
|
Write-Warning "Script not found — skipping CI-RetentionPolicy: $retentionScript"
|
|
} else {
|
|
$retentionAction = New-ScheduledTaskAction `
|
|
-Execute $psExe `
|
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$retentionScript`""
|
|
|
|
$daily3am = New-ScheduledTaskTrigger -Daily -At '03:00' `
|
|
-RandomDelay (New-TimeSpan -Minutes 30)
|
|
$atStartup2 = New-ScheduledTaskTrigger -AtStartup `
|
|
-RandomDelay (New-TimeSpan -Minutes 15)
|
|
|
|
$retentionSettings = New-ScheduledTaskSettingsSet `
|
|
-ExecutionTimeLimit (New-TimeSpan -Hours 1) `
|
|
-MultipleInstances IgnoreNew `
|
|
-StartWhenAvailable
|
|
|
|
if ($PSCmdlet.ShouldProcess('CI-RetentionPolicy', 'Register/update scheduled task')) {
|
|
Register-ScheduledTask `
|
|
-TaskName 'CI-RetentionPolicy' `
|
|
-TaskPath $taskPath `
|
|
-Action $retentionAction `
|
|
-Trigger @($daily3am, $atStartup2) `
|
|
-Principal $principal `
|
|
-Settings $retentionSettings `
|
|
-Description 'Purges old CI artifacts, logs, and stale IP leases (Local CI/CD System)' `
|
|
-Force | Out-Null
|
|
Write-Host "[Register] CI-RetentionPolicy registered (daily 03:00 + at startup)." -ForegroundColor Green
|
|
}
|
|
}
|
|
|
|
# ── Task 3: CI-DiskSpaceAlert ─────────────────────────────────────────────────
|
|
$diskScript = Join-Path $ScriptRoot 'Watch-DiskSpace.ps1'
|
|
if (-not (Test-Path $diskScript -PathType Leaf)) {
|
|
Write-Warning "Script not found — skipping CI-DiskSpaceAlert: $diskScript"
|
|
} else {
|
|
$diskAction = New-ScheduledTaskAction `
|
|
-Execute $psExe `
|
|
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$diskScript`""
|
|
|
|
# Every 15 minutes indefinitely
|
|
$disk15min = New-ScheduledTaskTrigger -Once -At '00:00' `
|
|
-RepetitionInterval (New-TimeSpan -Minutes 15)
|
|
|
|
$diskSettings = New-ScheduledTaskSettingsSet `
|
|
-ExecutionTimeLimit (New-TimeSpan -Minutes 5) `
|
|
-MultipleInstances IgnoreNew `
|
|
-StartWhenAvailable
|
|
|
|
if ($PSCmdlet.ShouldProcess('CI-DiskSpaceAlert', 'Register/update scheduled task')) {
|
|
Register-ScheduledTask `
|
|
-TaskName 'CI-DiskSpaceAlert' `
|
|
-TaskPath $taskPath `
|
|
-Action $diskAction `
|
|
-Trigger $disk15min `
|
|
-Principal $principal `
|
|
-Settings $diskSettings `
|
|
-Description 'Alerts via Event Log when CI drive free space drops below 50 GB (Local CI/CD System)' `
|
|
-Force | Out-Null
|
|
Write-Host "[Register] CI-DiskSpaceAlert registered (every 15 min)." -ForegroundColor Green
|
|
}
|
|
}
|
|
|
|
Write-Host "`n[Register] Done. Verify with: Get-ScheduledTask -TaskPath '\CI\' | Format-Table TaskName, State" -ForegroundColor Cyan
|