feat(sprint3): §4.1/4.3/4.4 — JSONL logging, disk alert, runbook

§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
  Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
  Write-JobEvent appends one JSON line per phase transition (ts, jobId,
  phase, status, data). Events emitted at: job.start, phase1.clone-repo
  start/success, phase2-3b.vm-start start/success, phase4.wait-ready
  start/success, phase5.build start/success, phase6.artifacts start/success,
  job.success/failure (with elapsedSec and error string).
  Silent no-op if $jsonLog is null (log dir setup failed). Errors in
  Write-JobEvent are swallowed — logging never breaks the build.
  Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.

§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
  Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
  Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
  (source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
  Optional -WebhookUrl for Discord/Gitea webhook notification.
  Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).

§4.4 — Incident runbook (docs/RUNBOOK.md)
  Four scenarios with symptom / triage commands / fix / escalation:
    1. Runner offline in Gitea UI
    2. All builds fail in Phase 2 (clone/start/IP)
    3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
    4. Template VMX corrupt after host crash (lock removal + backup restore)
  Quick-reference table at the end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Simone
2026-05-10 19:07:48 +02:00
parent bc894a8058
commit 1962c977ba
5 changed files with 394 additions and 6 deletions
+32
View File
@@ -123,4 +123,36 @@ if (-not (Test-Path $retentionScript -PathType Leaf)) {
}
}
# ── Task 3: CI-DiskSpaceAlert ─────────────────────────────────────────────────
$diskScript = Join-Path $ScriptRoot 'Watch-DiskSpace.ps1'
if (-not (Test-Path $diskScript -PathType Leaf)) {
Write-Warning "Script not found — skipping CI-DiskSpaceAlert: $diskScript"
} else {
$diskAction = New-ScheduledTaskAction `
-Execute $psExe `
-Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$diskScript`""
# Every 15 minutes indefinitely
$disk15min = New-ScheduledTaskTrigger -Once -At '00:00' `
-RepetitionInterval (New-TimeSpan -Minutes 15)
$diskSettings = New-ScheduledTaskSettingsSet `
-ExecutionTimeLimit (New-TimeSpan -Minutes 5) `
-MultipleInstances IgnoreNew `
-StartWhenAvailable
if ($PSCmdlet.ShouldProcess('CI-DiskSpaceAlert', 'Register/update scheduled task')) {
Register-ScheduledTask `
-TaskName 'CI-DiskSpaceAlert' `
-TaskPath $taskPath `
-Action $diskAction `
-Trigger $disk15min `
-Principal $principal `
-Settings $diskSettings `
-Description 'Alerts via Event Log when CI drive free space drops below 50 GB (Local CI/CD System)' `
-Force | Out-Null
Write-Host "[Register] CI-DiskSpaceAlert registered (every 15 min)." -ForegroundColor Green
}
}
Write-Host "`n[Register] Done. Verify with: Get-ScheduledTask -TaskPath '\CI\' | Format-Table TaskName, State" -ForegroundColor Cyan