feat(sprint3): §4.1/4.3/4.4 — JSONL logging, disk alert, runbook
§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
Write-JobEvent appends one JSON line per phase transition (ts, jobId,
phase, status, data). Events emitted at: job.start, phase1.clone-repo
start/success, phase2-3b.vm-start start/success, phase4.wait-ready
start/success, phase5.build start/success, phase6.artifacts start/success,
job.success/failure (with elapsedSec and error string).
Silent no-op if $jsonLog is null (log dir setup failed). Errors in
Write-JobEvent are swallowed — logging never breaks the build.
Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.
§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
(source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
Optional -WebhookUrl for Discord/Gitea webhook notification.
Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).
§4.4 — Incident runbook (docs/RUNBOOK.md)
Four scenarios with symptom / triage commands / fix / escalation:
1. Runner offline in Gitea UI
2. All builds fail in Phase 2 (clone/start/IP)
3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
4. Template VMX corrupt after host crash (lock removal + backup restore)
Quick-reference table at the end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -123,4 +123,36 @@ if (-not (Test-Path $retentionScript -PathType Leaf)) {
|
||||
}
|
||||
}
|
||||
|
||||
# ── Task 3: CI-DiskSpaceAlert ─────────────────────────────────────────────────
|
||||
$diskScript = Join-Path $ScriptRoot 'Watch-DiskSpace.ps1'
|
||||
if (-not (Test-Path $diskScript -PathType Leaf)) {
|
||||
Write-Warning "Script not found — skipping CI-DiskSpaceAlert: $diskScript"
|
||||
} else {
|
||||
$diskAction = New-ScheduledTaskAction `
|
||||
-Execute $psExe `
|
||||
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$diskScript`""
|
||||
|
||||
# Every 15 minutes indefinitely
|
||||
$disk15min = New-ScheduledTaskTrigger -Once -At '00:00' `
|
||||
-RepetitionInterval (New-TimeSpan -Minutes 15)
|
||||
|
||||
$diskSettings = New-ScheduledTaskSettingsSet `
|
||||
-ExecutionTimeLimit (New-TimeSpan -Minutes 5) `
|
||||
-MultipleInstances IgnoreNew `
|
||||
-StartWhenAvailable
|
||||
|
||||
if ($PSCmdlet.ShouldProcess('CI-DiskSpaceAlert', 'Register/update scheduled task')) {
|
||||
Register-ScheduledTask `
|
||||
-TaskName 'CI-DiskSpaceAlert' `
|
||||
-TaskPath $taskPath `
|
||||
-Action $diskAction `
|
||||
-Trigger $disk15min `
|
||||
-Principal $principal `
|
||||
-Settings $diskSettings `
|
||||
-Description 'Alerts via Event Log when CI drive free space drops below 50 GB (Local CI/CD System)' `
|
||||
-Force | Out-Null
|
||||
Write-Host "[Register] CI-DiskSpaceAlert registered (every 15 min)." -ForegroundColor Green
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "`n[Register] Done. Verify with: Get-ScheduledTask -TaskPath '\CI\' | Format-Table TaskName, State" -ForegroundColor Cyan
|
||||
|
||||
Reference in New Issue
Block a user