feat(sprint3): §4.1/4.3/4.4 — JSONL logging, disk alert, runbook

§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
  Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
  Write-JobEvent appends one JSON line per phase transition (ts, jobId,
  phase, status, data). Events emitted at: job.start, phase1.clone-repo
  start/success, phase2-3b.vm-start start/success, phase4.wait-ready
  start/success, phase5.build start/success, phase6.artifacts start/success,
  job.success/failure (with elapsedSec and error string).
  Silent no-op if $jsonLog is null (log dir setup failed). Errors in
  Write-JobEvent are swallowed — logging never breaks the build.
  Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.

§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
  Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
  Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
  (source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
  Optional -WebhookUrl for Discord/Gitea webhook notification.
  Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).

§4.4 — Incident runbook (docs/RUNBOOK.md)
  Four scenarios with symptom / triage commands / fix / escalation:
    1. Runner offline in Gitea UI
    2. All builds fail in Phase 2 (clone/start/IP)
    3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
    4. Template VMX corrupt after host crash (lock removal + backup restore)
  Quick-reference table at the end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Simone
2026-05-10 19:07:48 +02:00
parent bc894a8058
commit 1962c977ba
5 changed files with 394 additions and 6 deletions
+5 -5
View File
@@ -18,7 +18,7 @@
---
## Summary
_Last updated: 2026-05-10 (post Sprint 2 operatività)_
_Last updated: 2026-05-10 (post Sprint 3 osservabilità)_
**Stato generale**: infrastruttura base e refactor Deploy↔Setup (§7) completi. Sicurezza sprint 1 in corso (HTTPS/5986 done, IP regex done, hash pinning struttura done — valori da riempire). Backlog operatività/perf/qualità intatto.
@@ -32,7 +32,7 @@ _Last updated: 2026-05-10 (post Sprint 2 operatività)_
| §1 Sicurezza & hardening | 2 | 6 | 1.1, 1.4 done; 1.2/1.3 parzialmente; 1.5/1.6/1.7/1.8 da fare |
| §2 Affidabilità & resilienza | 4 | 3 | 2.1/2.2/2.3/2.5 done; 2.4/2.6/2.7 backlog |
| §3 Performance & throughput | 0 | 6 | nessuna ottimizzazione applicata; baseline §3.6 prerequisito |
| §4 Osservabilità & manutenzione | 0 | 5 | log JSONL, metriche, runbook tutti aperti |
| §4 Osservabilità & manutenzione | 3 | 2 | 4.1/4.3/4.4 done; 4.2/4.5 backlog |
| §5 Qualità codice & test | 0 | 5 | Pester, modulo `_Common.psm1`, regole PSScriptAnalyzer custom |
| §6 Scalabilità & estensioni | 0 | 6 | Linux VM, composite action, toolchain Tier-1 |
| §7 Refactor Deploy ↔ Setup | 4 | 1 | 7.1/7.2/7.3/7.4 done; 7.5 (rinomina) opzionale |
@@ -421,7 +421,7 @@ Considerare:
## 4. Osservabilità & manutenzione
### 4.1 [P1] [ ] Log strutturati per parsing
### 4.1 [P1] [x] Log strutturati per parsing — COMPLETATO 2026-05-10
File: [scripts/Invoke-CIJob.ps1:178-187](scripts/Invoke-CIJob.ps1).
**Motivazione**: i log attuali sono `Write-Host` testuale. Per dashboard / alerting servono
@@ -455,13 +455,13 @@ ci_runner_active_jobs 1
Se l'homelab ha già Prometheus + node_exporter, basta
`--collector.textfile.directory=F:\CI\Metrics`.
### 4.3 [P1] [ ] Disk space alert
### 4.3 [P1] [x] Disk space alert — COMPLETATO 2026-05-10
File: estendere `Setup-Host.ps1` o creare `scripts/Watch-DiskSpace.ps1`.
`F:` riempito = build silenziosamente fallite (linked clone fallirà su `vmrun clone` con
messaggi criptici). Alert via `eventcreate` o webhook Gitea/Discord quando libero < 50 GB.
### 4.4 [P2] [ ] Runbook per incident comuni
### 4.4 [P2] [x] Runbook per incident comuni — COMPLETATO 2026-05-10
File: nuovo `docs/RUNBOOK.md`.
Documentare con copy-pasta: