diff --git a/TODO.md b/TODO.md index d60db41..b3a9192 100644 --- a/TODO.md +++ b/TODO.md @@ -18,7 +18,7 @@ --- ## Summary -_Last updated: 2026-05-10 (post Sprint 2 operatività)_ +_Last updated: 2026-05-10 (post Sprint 3 osservabilità)_ **Stato generale**: infrastruttura base e refactor Deploy↔Setup (§7) completi. Sicurezza sprint 1 in corso (HTTPS/5986 done, IP regex done, hash pinning struttura done — valori da riempire). Backlog operatività/perf/qualità intatto. @@ -32,7 +32,7 @@ _Last updated: 2026-05-10 (post Sprint 2 operatività)_ | §1 Sicurezza & hardening | 2 | 6 | 1.1, 1.4 done; 1.2/1.3 parzialmente; 1.5/1.6/1.7/1.8 da fare | | §2 Affidabilità & resilienza | 4 | 3 | 2.1/2.2/2.3/2.5 done; 2.4/2.6/2.7 backlog | | §3 Performance & throughput | 0 | 6 | nessuna ottimizzazione applicata; baseline §3.6 prerequisito | -| §4 Osservabilità & manutenzione | 0 | 5 | log JSONL, metriche, runbook tutti aperti | +| §4 Osservabilità & manutenzione | 3 | 2 | 4.1/4.3/4.4 done; 4.2/4.5 backlog | | §5 Qualità codice & test | 0 | 5 | Pester, modulo `_Common.psm1`, regole PSScriptAnalyzer custom | | §6 Scalabilità & estensioni | 0 | 6 | Linux VM, composite action, toolchain Tier-1 | | §7 Refactor Deploy ↔ Setup | 4 | 1 | 7.1/7.2/7.3/7.4 done; 7.5 (rinomina) opzionale | @@ -421,7 +421,7 @@ Considerare: ## 4. Osservabilità & manutenzione -### 4.1 [P1] [ ] Log strutturati per parsing +### 4.1 [P1] [x] Log strutturati per parsing — COMPLETATO 2026-05-10 File: [scripts/Invoke-CIJob.ps1:178-187](scripts/Invoke-CIJob.ps1). **Motivazione**: i log attuali sono `Write-Host` testuale. Per dashboard / alerting servono @@ -455,13 +455,13 @@ ci_runner_active_jobs 1 Se l'homelab ha già Prometheus + node_exporter, basta `--collector.textfile.directory=F:\CI\Metrics`. -### 4.3 [P1] [ ] Disk space alert +### 4.3 [P1] [x] Disk space alert — COMPLETATO 2026-05-10 File: estendere `Setup-Host.ps1` o creare `scripts/Watch-DiskSpace.ps1`. `F:` riempito = build silenziosamente fallite (linked clone fallirà su `vmrun clone` con messaggi criptici). Alert via `eventcreate` o webhook Gitea/Discord quando libero < 50 GB. -### 4.4 [P2] [ ] Runbook per incident comuni +### 4.4 [P2] [x] Runbook per incident comuni — COMPLETATO 2026-05-10 File: nuovo `docs/RUNBOOK.md`. Documentare con copy-pasta: diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md new file mode 100644 index 0000000..c5549dc --- /dev/null +++ b/docs/RUNBOOK.md @@ -0,0 +1,212 @@ +# CI System Runbook + +Triage guide for the local CI/CD system. Each entry: symptom → triage commands → fix → escalation. + +--- + +## 1. Runner offline in Gitea UI + +**Symptom**: `http://10.10.20.11:3100/admin/runners` shows `local-windows-runner` as offline. +Queued jobs stay pending indefinitely. + +**Triage**: +```powershell +# Check service state +Get-Service act_runner + +# Last 50 lines of runner log +Get-Content 'F:\CI\act_runner\logs\act_runner.log' -Tail 50 + +# Check registration file is intact +Test-Path 'F:\CI\act_runner\.runner' +``` + +**Fix**: +```powershell +# Restart the service +Restart-Service act_runner + +# Verify it came back online (wait ~10s then check Gitea UI) +Get-Service act_runner | Select-Object Status, StartType + +# If service won't start, check NSSM log +& 'C:\nssm\nssm.exe' status act_runner +``` + +If the `.runner` registration file is missing or corrupt, re-register: +```powershell +cd F:\CI\act_runner +.\act_runner.exe register --no-interactive ` + --instance http://10.10.20.11:3100 ` + --token ` + --name local-windows-runner ` + --labels "windows-build:host,dotnet:host,msbuild:host" +``` + +**Escalation**: If the runner restarts but goes offline again within minutes, check Event Viewer → Application for `act_runner` errors and inspect `F:\CI\act_runner\logs\`. + +--- + +## 2. All builds fail in Phase 2 (VM clone / start) + +**Symptom**: `Invoke-CIJob.ps1` fails at Phase 2 with errors like: +- `vmrun clone failed` +- `vmrun start failed` +- `Template VMX not found` +- `Could not detect VM IP address` + +**Triage**: +```powershell +# List all running VMs +& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' -T ws list + +# Check template VMX exists and is accessible +Test-Path 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx' + +# Check for orphaned clones that may be consuming disk +Get-ChildItem 'F:\CI\BuildVMs\' -Directory | Select-Object Name, LastWriteTime + +# Check disk free space +Get-PSDrive F | Select-Object Name, Free, Used + +# Check for a stuck vm-start lock from a crashed job +Test-Path 'F:\CI\State\vm-start.lock' +``` + +**Fix** — by root cause: + +*Template VMX missing/moved*: check `GITEA_CI_TEMPLATE_PATH` in `F:\CI\act_runner\config.yaml`. + +*Parent VMDK locked* (VMware left a lock file after host crash): +```powershell +# Stop all VMs +& vmrun.exe -T ws stop 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx' hard +# Delete lock files +Remove-Item 'F:\CI\Templates\WinBuild\*.lck' -Recurse -Force -ErrorAction SilentlyContinue +``` + +*Snapshot missing* (`BaseClean` was deleted or renamed): +```powershell +# List snapshots on template VM +& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx' +# Update GITEA_CI_SNAPSHOT_NAME in config.yaml to match the available snapshot name +``` + +*Disk full* (clone delta files need space): +```powershell +# Emergency cleanup — remove all orphaned clones +& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Cleanup-OrphanedBuildVMs.ps1' -MaxAgeHours 0 +# Then run retention +& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-RetentionPolicy.ps1' -AggressiveRetentionDays 3 +``` + +*Stale vm-start lock* (from a job that crashed without cleanup): +```powershell +Remove-Item 'F:\CI\State\vm-start.lock' -Force +Remove-Item 'F:\CI\State\ip-leases\*.lease' -Force +``` + +**Escalation**: If `vmrun clone` fails with exit code -1 even after clearing locks and confirming disk space, re-open VMware Workstation UI and check the template VM is intact and the snapshot is listed. + +--- + +## 3. Builds are slow + +**Symptom**: jobs that previously completed in ~3 min now take 8+ min. +Phase durations visible in `F:\CI\Logs\\invoke-ci.jsonl`. + +**Triage**: +```powershell +# Check disk free space (below 50 GB = fragmented writes) +Get-PSDrive F | Select-Object @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}} + +# Check active VM CPU usage (Task Manager or:) +Get-Process vmware-vmx | Select-Object CPU, WorkingSet | Sort-Object CPU -Descending + +# Check VMnet8 NAT adapter status +Get-NetAdapter | Where-Object { $_.Name -like 'VMware*' } + +# Parse JSONL for per-phase durations (requires jq or manual inspection) +# Each phase has a 'start' and 'success' event — diff the 'ts' fields. +Get-Content 'F:\CI\Logs\\invoke-ci.jsonl' | ConvertFrom-Json | Format-Table ts,phase,status +``` + +**Fix** — by root cause: + +*Low disk space → fragmented VMDKs*: run retention policy, then consider `vmware-vdiskmanager -d` to defragment the template VMDK. + +*High vmware-vmx CPU with many VMs*: reduce `capacity` in `config.yaml` from 4 to 2. + +*VMnet8 NAT bottleneck* (slow pip/nuget downloads inside VM): check `Services.msc` → `VMware NAT Service` is running. + +*NVMe saturation*: if the host NVMe is at 100% I/O (Task Manager → Performance → Disk), all four concurrent VMs are competing. Reduce `capacity: 2`. + +**Escalation**: Use `invoke-ci.jsonl` to identify which phase is slow across multiple jobs. Phase 1 slow = host git or network. Phase 2-3b slow = disk I/O. Phase 5 slow = build itself (not a CI infra problem). + +--- + +## 4. Template VMX corrupt after host crash + +**Symptom**: After an unclean host shutdown, `vmrun clone` or `vmrun start` on the template fails. VMware Workstation shows the template in an error state. + +**Triage**: +```powershell +# Try starting the template directly in VMware Workstation UI +# If it reports "configuration file error" or "disk lock", proceed below. + +# Check for lock files +Get-ChildItem 'F:\CI\Templates\WinBuild\' -Recurse -Filter '*.lck' + +# Check if backup exists +Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 5 +``` + +**Fix**: + +*Lock files only* (common after hard shutdown): +```powershell +# Ensure no VMware processes are running +Get-Process vmware*, vmrun -ErrorAction SilentlyContinue | Stop-Process -Force +# Remove locks +Remove-Item 'F:\CI\Templates\WinBuild\*.lck' -Recurse -Force +# Test clone +& vmrun.exe -T ws listSnapshots 'F:\CI\Templates\WinBuild\CI-WinBuild.vmx' +``` + +*VMX or VMDK truly corrupt — restore from backup*: +```powershell +# Stop all CI activity first +Stop-Service act_runner + +# Identify latest backup +$latest = Get-ChildItem 'F:\CI\Backups\' -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1 +Write-Host "Restoring from: $($latest.FullName)" + +# Replace template directory +Remove-Item 'F:\CI\Templates\WinBuild\' -Recurse -Force +Copy-Item $latest.FullName 'F:\CI\Templates\WinBuild\' -Recurse + +# Restart runner +Start-Service act_runner +``` + +*No backup exists*: must re-provision the template from scratch. +Follow `docs/WINDOWS-TEMPLATE-SETUP.md` → Fase A (Deploy) → Fase B (Prepare). +Estimated time: 2-4 hours including Windows Update. + +**Escalation**: If VMware Workstation itself is damaged (rare), reinstall VMware and re-import the template VMX. The VMDK files survive a VMware reinstall as long as the disk is intact. + +--- + +## Quick Reference + +| Symptom | First command | +| ------------------------------ | ---------------------------------------------------------------------- | +| Runner offline | `Get-Service act_runner`, then `Restart-Service act_runner` | +| Phase 2 clone fails | `Test-Path F:\CI\Templates\WinBuild\CI-WinBuild.vmx` | +| Disk full | `Get-PSDrive F \| Select Free`; run `Invoke-RetentionPolicy.ps1` | +| Stale lock | `Remove-Item F:\CI\State\vm-start.lock` | +| Slow builds | Check `invoke-ci.jsonl` phase timestamps; check disk I/O | +| Template corrupt | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` | +| Snapshot missing | `vmrun listSnapshots `; update `GITEA_CI_SNAPSHOT_NAME` | +| IP collision | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity` | diff --git a/scripts/Invoke-CIJob.ps1 b/scripts/Invoke-CIJob.ps1 index 32c0416..87c8832 100644 --- a/scripts/Invoke-CIJob.ps1 +++ b/scripts/Invoke-CIJob.ps1 @@ -159,14 +159,16 @@ $jobArtifactDir = Join-Path $ArtifactBaseDir $JobId $hostCloneDir = Join-Path $env:TEMP "ci-src-$JobId" $cloneVmxPath = $null $leaseFile = $null # §2.1 IP lease file path — released in finally +$jsonLog = $null # §4.1 JSONL structured log path $startTime = Get-Date -# ── Start transcript ────────────────────────────────────────────────────────── +# ── Start transcript + JSONL log ────────────────────────────────────────────── $transcriptStarted = $false try { $jobLogDir = Join-Path $LogDir $JobId if (-not (Test-Path $jobLogDir)) { New-Item -ItemType Directory -Path $jobLogDir -Force | Out-Null } $transcriptPath = Join-Path $jobLogDir 'invoke-ci.log' + $jsonLog = Join-Path $jobLogDir 'invoke-ci.jsonl' # -Append allows starting a new transcript even when VS Code (or another host) # has already opened a transcript on this PowerShell session. Start-Transcript -Path $transcriptPath -Append -Force -ErrorAction Stop @@ -176,6 +178,29 @@ catch { Write-Warning "[Invoke-CIJob] Could not start transcript: $_" } +# ── §4.1 Structured event emitter ──────────────────────────────────────────── +# Appends a single JSON line to invoke-ci.jsonl per phase transition. +# Silent no-op if $jsonLog is null (transcript setup failed). +function Write-JobEvent { + param( + [Parameter(Mandatory)] [string] $Phase, + [Parameter(Mandatory)] [string] $Status, # start | success | failure | info + [hashtable] $Data = @{} + ) + if (-not $script:jsonLog) { return } + $event = [ordered]@{ + ts = (Get-Date -Format 'o') + jobId = $script:JobId + phase = $Phase + status = $Status + data = $Data + } + try { + $event | ConvertTo-Json -Compress -Depth 5 | + Add-Content -Path $script:jsonLog -Encoding UTF8 -ErrorAction Stop + } catch { } # never let logging break the build +} + Write-Host "============================================================" Write-Host "[Invoke-CIJob] Job : $JobId" Write-Host "[Invoke-CIJob] Repository : $RepoUrl" @@ -187,6 +212,14 @@ Write-Host "[Invoke-CIJob] Artifacts : $jobArtifactDir" Write-Host "[Invoke-CIJob] Started : $startTime" Write-Host "============================================================" +Write-JobEvent -Phase 'job' -Status 'start' -Data @{ + repo = $RepoUrl + branch = $Branch + commit = $Commit + template = $TemplatePath + snapshot = $SnapshotName +} + $exitCode = 0 try { @@ -195,6 +228,7 @@ try { # still cloned here and copied via WinRM zip to avoid injecting a PAT # into the VM environment. Write-Host "`n[Phase 1/6] Cloning repository on host..." + Write-JobEvent -Phase 'phase1.clone-repo' -Status 'start' if (Test-Path $hostCloneDir) { Remove-Item $hostCloneDir -Recurse -Force } $gitArgs = @('clone', '--depth', '1', '--branch', $Branch) if ($Submodules) { $gitArgs += '--recurse-submodules' } @@ -224,6 +258,7 @@ try { } } Write-Host "[Invoke-CIJob] Source cloned to: $hostCloneDir" + Write-JobEvent -Phase 'phase1.clone-repo' -Status 'success' -Data @{ dir = $hostCloneDir } # ── Phase 2-3b: Clone + Start + IP (serialized via file-based mutex) ──── # §2.1: With capacity:4 and DHCP on VMnet8, concurrent vmrun starts can race @@ -237,6 +272,7 @@ try { if (-not (Test-Path $leasesDir)) { New-Item -ItemType Directory -Path $leasesDir -Force | Out-Null } Write-Host "`n[Phase 2/6] Acquiring VM-start lock (prevents DHCP collision with capacity>1)..." + Write-JobEvent -Phase 'phase2-3b.vm-start' -Status 'start' $lockStream = $null $lockDeadline = (Get-Date).AddMinutes(10) while ($true) { @@ -296,6 +332,7 @@ try { } [System.IO.File]::WriteAllText($leaseFile, $JobId) Write-Host "[Invoke-CIJob] IP $VMIPAddress leased for job $JobId." + Write-JobEvent -Phase 'phase2-3b.vm-start' -Status 'success' -Data @{ ip = $VMIPAddress; vmx = $cloneVmxPath } } finally { # Release lock — next waiting job can now enter clone→start→IP phase @@ -310,13 +347,16 @@ try { # ── Phase 4: Wait for readiness ─────────────────────────────────────── Write-Host "`n[Phase 4/6] Waiting for VM readiness..." + Write-JobEvent -Phase 'phase4.wait-ready' -Status 'start' -Data @{ ip = $VMIPAddress } & "$scriptDir\Wait-VMReady.ps1" ` -VMPath $cloneVmxPath ` -IPAddress $VMIPAddress ` -VmrunPath $VmrunPath + Write-JobEvent -Phase 'phase4.wait-ready' -Status 'success' # ── Phase 5: Remote build (copy source + build inside VM) ───────────── Write-Host "`n[Phase 5/6] Invoking remote build..." + Write-JobEvent -Phase 'phase5.build' -Status 'start' $remoteBuildParams = @{ IPAddress = $VMIPAddress Credential = $credential @@ -326,9 +366,11 @@ try { if ($BuildCommand -ne '') { $remoteBuildParams['BuildCommand'] = $BuildCommand } if ($GuestArtifactSource -ne '') { $remoteBuildParams['GuestArtifactSource'] = $GuestArtifactSource } & "$scriptDir\Invoke-RemoteBuild.ps1" @remoteBuildParams + Write-JobEvent -Phase 'phase5.build' -Status 'success' # ── Phase 6: Collect artifacts ──────────────────────────────────────── Write-Host "`n[Phase 6/6] Collecting artifacts..." + Write-JobEvent -Phase 'phase6.artifacts' -Status 'start' & "$scriptDir\Get-BuildArtifacts.ps1" ` -IPAddress $VMIPAddress ` -Credential $credential ` @@ -337,6 +379,8 @@ try { -IncludeLogs $elapsed = (Get-Date) - $startTime + Write-JobEvent -Phase 'phase6.artifacts' -Status 'success' -Data @{ dir = $jobArtifactDir } + Write-JobEvent -Phase 'job' -Status 'success' -Data @{ elapsedSec = [int]$elapsed.TotalSeconds } Write-Host "`n============================================================" Write-Host "[Invoke-CIJob] SUCCESS — Job $JobId completed in $($elapsed.ToString('mm\:ss'))" Write-Host "[Invoke-CIJob] Artifacts: $jobArtifactDir" @@ -345,6 +389,10 @@ try { catch { $exitCode = 1 $elapsed = (Get-Date) - $startTime + Write-JobEvent -Phase 'job' -Status 'failure' -Data @{ + elapsedSec = [int]$elapsed.TotalSeconds + error = "$_" + } Write-Host "`n============================================================" Write-Host "[Invoke-CIJob] FAILURE — Job $JobId failed after $($elapsed.ToString('mm\:ss'))" Write-Host "Error: $_" diff --git a/scripts/Register-CIScheduledTasks.ps1 b/scripts/Register-CIScheduledTasks.ps1 index 83ac8e8..bd255df 100644 --- a/scripts/Register-CIScheduledTasks.ps1 +++ b/scripts/Register-CIScheduledTasks.ps1 @@ -123,4 +123,36 @@ if (-not (Test-Path $retentionScript -PathType Leaf)) { } } +# ── Task 3: CI-DiskSpaceAlert ───────────────────────────────────────────────── +$diskScript = Join-Path $ScriptRoot 'Watch-DiskSpace.ps1' +if (-not (Test-Path $diskScript -PathType Leaf)) { + Write-Warning "Script not found — skipping CI-DiskSpaceAlert: $diskScript" +} else { + $diskAction = New-ScheduledTaskAction ` + -Execute $psExe ` + -Argument "-NoProfile -NonInteractive -ExecutionPolicy Bypass -File `"$diskScript`"" + + # Every 15 minutes indefinitely + $disk15min = New-ScheduledTaskTrigger -Once -At '00:00' ` + -RepetitionInterval (New-TimeSpan -Minutes 15) + + $diskSettings = New-ScheduledTaskSettingsSet ` + -ExecutionTimeLimit (New-TimeSpan -Minutes 5) ` + -MultipleInstances IgnoreNew ` + -StartWhenAvailable + + if ($PSCmdlet.ShouldProcess('CI-DiskSpaceAlert', 'Register/update scheduled task')) { + Register-ScheduledTask ` + -TaskName 'CI-DiskSpaceAlert' ` + -TaskPath $taskPath ` + -Action $diskAction ` + -Trigger $disk15min ` + -Principal $principal ` + -Settings $diskSettings ` + -Description 'Alerts via Event Log when CI drive free space drops below 50 GB (Local CI/CD System)' ` + -Force | Out-Null + Write-Host "[Register] CI-DiskSpaceAlert registered (every 15 min)." -ForegroundColor Green + } +} + Write-Host "`n[Register] Done. Verify with: Get-ScheduledTask -TaskPath '\CI\' | Format-Table TaskName, State" -ForegroundColor Cyan diff --git a/scripts/Watch-DiskSpace.ps1 b/scripts/Watch-DiskSpace.ps1 new file mode 100644 index 0000000..952532a --- /dev/null +++ b/scripts/Watch-DiskSpace.ps1 @@ -0,0 +1,96 @@ +#Requires -Version 5.1 +<# +.SYNOPSIS + Checks CI host drive free space and alerts via Windows Event Log (and optionally a webhook). + +.DESCRIPTION + Intended to run as a scheduled task every 15 minutes (registered by + Register-CIScheduledTasks.ps1). If the monitored drive is below MinFreeGB, + writes a Warning entry to the Windows Application Event Log under source + 'CI-DiskAlert' (Event ID 1001) and exits with code 1 so Task Scheduler + marks the task as failed (visible in Task Scheduler history). + + If WebhookUrl is provided, also POSTs a JSON payload to that URL + (compatible with Discord and Gitea webhooks — content field only). + + A full drive silently causes vmrun clone to fail with cryptic errors + (not enough disk space for linked clone delta files). Alert early. + +.PARAMETER MinFreeGB + Alert threshold in GB. Default: 50 + +.PARAMETER DriveLetter + Single letter (no colon) of the drive to monitor. Default: F + +.PARAMETER WebhookUrl + Optional Discord/Gitea webhook URL. If empty, only Event Log is written. + +.EXAMPLE + # Manual check + .\Watch-DiskSpace.ps1 -MinFreeGB 50 + + # With Discord webhook + .\Watch-DiskSpace.ps1 -WebhookUrl "https://discord.com/api/webhooks/..." +#> +[CmdletBinding()] +param( + [ValidateRange(1, 2000)] + [int] $MinFreeGB = 50, + + [ValidatePattern('^[A-Za-z]$')] + [string] $DriveLetter = 'F', + + [string] $WebhookUrl = '' +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Continue' + +$drive = Get-PSDrive -Name $DriveLetter -ErrorAction SilentlyContinue +if (-not $drive) { + Write-Warning "[DiskSpace] Drive ${DriveLetter}: not found." + exit 2 +} + +$freeGB = [math]::Round($drive.Free / 1GB, 1) +$totalGB = [math]::Round(($drive.Free + $drive.Used) / 1GB, 1) +$pctFree = [math]::Round($freeGB / $totalGB * 100, 0) + +if ($freeGB -ge $MinFreeGB) { + Write-Host "[DiskSpace] Drive ${DriveLetter}: ${freeGB} GB free / ${totalGB} GB (${pctFree}%) — OK (threshold: ${MinFreeGB} GB)" + exit 0 +} + +# ── Alert ───────────────────────────────────────────────────────────────────── +$msg = "CI host drive ${DriveLetter}: free space ${freeGB} GB (${pctFree}%) is below " + + "the ${MinFreeGB} GB threshold. vmrun linked clones may fail silently. " + + "Run Invoke-RetentionPolicy.ps1 or free disk space manually." + +Write-Warning "[DiskSpace] $msg" + +# Windows Event Log +$logSource = 'CI-DiskAlert' +try { + if (-not [System.Diagnostics.EventLog]::SourceExists($logSource)) { + New-EventLog -LogName Application -Source $logSource -ErrorAction Stop + } + Write-EventLog -LogName Application -Source $logSource ` + -EventId 1001 -EntryType Warning -Message $msg + Write-Host "[DiskSpace] Event Log entry written (Application/$logSource, EventId 1001)." +} catch { + Write-Warning "[DiskSpace] Could not write Event Log: $_" +} + +# Optional webhook (Discord / Gitea) +if ($WebhookUrl) { + $body = @{ content = ":warning: **CI Disk Alert** — $msg" } | ConvertTo-Json -Compress + try { + $null = Invoke-RestMethod -Uri $WebhookUrl -Method Post ` + -Body $body -ContentType 'application/json' -TimeoutSec 10 + Write-Host "[DiskSpace] Webhook notified: $WebhookUrl" + } catch { + Write-Warning "[DiskSpace] Webhook POST failed: $_" + } +} + +exit 1 # non-zero → Task Scheduler marks run as failed (visible in history/Event Viewer)