feat(sprint3): §4.1/4.3/4.4 — JSONL logging, disk alert, runbook
§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
Write-JobEvent appends one JSON line per phase transition (ts, jobId,
phase, status, data). Events emitted at: job.start, phase1.clone-repo
start/success, phase2-3b.vm-start start/success, phase4.wait-ready
start/success, phase5.build start/success, phase6.artifacts start/success,
job.success/failure (with elapsedSec and error string).
Silent no-op if $jsonLog is null (log dir setup failed). Errors in
Write-JobEvent are swallowed — logging never breaks the build.
Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.
§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
(source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
Optional -WebhookUrl for Discord/Gitea webhook notification.
Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).
§4.4 — Incident runbook (docs/RUNBOOK.md)
Four scenarios with symptom / triage commands / fix / escalation:
1. Runner offline in Gitea UI
2. All builds fail in Phase 2 (clone/start/IP)
3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
4. Template VMX corrupt after host crash (lock removal + backup restore)
Quick-reference table at the end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
#Requires -Version 5.1
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Checks CI host drive free space and alerts via Windows Event Log (and optionally a webhook).
|
||||
|
||||
.DESCRIPTION
|
||||
Intended to run as a scheduled task every 15 minutes (registered by
|
||||
Register-CIScheduledTasks.ps1). If the monitored drive is below MinFreeGB,
|
||||
writes a Warning entry to the Windows Application Event Log under source
|
||||
'CI-DiskAlert' (Event ID 1001) and exits with code 1 so Task Scheduler
|
||||
marks the task as failed (visible in Task Scheduler history).
|
||||
|
||||
If WebhookUrl is provided, also POSTs a JSON payload to that URL
|
||||
(compatible with Discord and Gitea webhooks — content field only).
|
||||
|
||||
A full drive silently causes vmrun clone to fail with cryptic errors
|
||||
(not enough disk space for linked clone delta files). Alert early.
|
||||
|
||||
.PARAMETER MinFreeGB
|
||||
Alert threshold in GB. Default: 50
|
||||
|
||||
.PARAMETER DriveLetter
|
||||
Single letter (no colon) of the drive to monitor. Default: F
|
||||
|
||||
.PARAMETER WebhookUrl
|
||||
Optional Discord/Gitea webhook URL. If empty, only Event Log is written.
|
||||
|
||||
.EXAMPLE
|
||||
# Manual check
|
||||
.\Watch-DiskSpace.ps1 -MinFreeGB 50
|
||||
|
||||
# With Discord webhook
|
||||
.\Watch-DiskSpace.ps1 -WebhookUrl "https://discord.com/api/webhooks/..."
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[ValidateRange(1, 2000)]
|
||||
[int] $MinFreeGB = 50,
|
||||
|
||||
[ValidatePattern('^[A-Za-z]$')]
|
||||
[string] $DriveLetter = 'F',
|
||||
|
||||
[string] $WebhookUrl = ''
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Continue'
|
||||
|
||||
$drive = Get-PSDrive -Name $DriveLetter -ErrorAction SilentlyContinue
|
||||
if (-not $drive) {
|
||||
Write-Warning "[DiskSpace] Drive ${DriveLetter}: not found."
|
||||
exit 2
|
||||
}
|
||||
|
||||
$freeGB = [math]::Round($drive.Free / 1GB, 1)
|
||||
$totalGB = [math]::Round(($drive.Free + $drive.Used) / 1GB, 1)
|
||||
$pctFree = [math]::Round($freeGB / $totalGB * 100, 0)
|
||||
|
||||
if ($freeGB -ge $MinFreeGB) {
|
||||
Write-Host "[DiskSpace] Drive ${DriveLetter}: ${freeGB} GB free / ${totalGB} GB (${pctFree}%) — OK (threshold: ${MinFreeGB} GB)"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ── Alert ─────────────────────────────────────────────────────────────────────
|
||||
$msg = "CI host drive ${DriveLetter}: free space ${freeGB} GB (${pctFree}%) is below " +
|
||||
"the ${MinFreeGB} GB threshold. vmrun linked clones may fail silently. " +
|
||||
"Run Invoke-RetentionPolicy.ps1 or free disk space manually."
|
||||
|
||||
Write-Warning "[DiskSpace] $msg"
|
||||
|
||||
# Windows Event Log
|
||||
$logSource = 'CI-DiskAlert'
|
||||
try {
|
||||
if (-not [System.Diagnostics.EventLog]::SourceExists($logSource)) {
|
||||
New-EventLog -LogName Application -Source $logSource -ErrorAction Stop
|
||||
}
|
||||
Write-EventLog -LogName Application -Source $logSource `
|
||||
-EventId 1001 -EntryType Warning -Message $msg
|
||||
Write-Host "[DiskSpace] Event Log entry written (Application/$logSource, EventId 1001)."
|
||||
} catch {
|
||||
Write-Warning "[DiskSpace] Could not write Event Log: $_"
|
||||
}
|
||||
|
||||
# Optional webhook (Discord / Gitea)
|
||||
if ($WebhookUrl) {
|
||||
$body = @{ content = ":warning: **CI Disk Alert** — $msg" } | ConvertTo-Json -Compress
|
||||
try {
|
||||
$null = Invoke-RestMethod -Uri $WebhookUrl -Method Post `
|
||||
-Body $body -ContentType 'application/json' -TimeoutSec 10
|
||||
Write-Host "[DiskSpace] Webhook notified: $WebhookUrl"
|
||||
} catch {
|
||||
Write-Warning "[DiskSpace] Webhook POST failed: $_"
|
||||
}
|
||||
}
|
||||
|
||||
exit 1 # non-zero → Task Scheduler marks run as failed (visible in history/Event Viewer)
|
||||
Reference in New Issue
Block a user