§4.1 — Structured JSONL log (Invoke-CIJob.ps1)
Each job now emits a parallel invoke-ci.jsonl alongside the transcript.
Write-JobEvent appends one JSON line per phase transition (ts, jobId,
phase, status, data). Events emitted at: job.start, phase1.clone-repo
start/success, phase2-3b.vm-start start/success, phase4.wait-ready
start/success, phase5.build start/success, phase6.artifacts start/success,
job.success/failure (with elapsedSec and error string).
Silent no-op if $jsonLog is null (log dir setup failed). Errors in
Write-JobEvent are swallowed — logging never breaks the build.
Enables post-hoc per-phase duration analysis with jq or ConvertFrom-Json.
§4.3 — Disk space alert (scripts/Watch-DiskSpace.ps1)
Checks drive free space every 15 min (scheduled by Register-CIScheduledTasks).
Below MinFreeGB (default 50 GB): writes Warning to Windows Application Event Log
(source CI-DiskAlert, EventId 1001) and exits 1 so Task Scheduler flags the run.
Optional -WebhookUrl for Discord/Gitea webhook notification.
Register-CIScheduledTasks.ps1 updated with Task 3: CI-DiskSpaceAlert (every 15 min).
§4.4 — Incident runbook (docs/RUNBOOK.md)
Four scenarios with symptom / triage commands / fix / escalation:
1. Runner offline in Gitea UI
2. All builds fail in Phase 2 (clone/start/IP)
3. Builds are slow (per-phase JSONL analysis, disk/CPU/NAT checks)
4. Template VMX corrupt after host crash (lock removal + backup restore)
Quick-reference table at the end.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
§2.1 — IP race with capacity>1 (Invoke-CIJob.ps1)
Phases 2-3b (clone+start+IP) now run inside a file-based exclusive mutex
(F:\CI\State\vm-start.lock via FileShare.None). Only one job at a time is
in the VM-start phase, preventing concurrent vmrun starts from racing for
the same DHCP lease. After IP is confirmed unique it is written to an IP
lease file (F:\CI\State\ip-leases\<IP>.lease). Lock is released immediately
so build phases run fully in parallel. Lease released in finally block.
Collision detection: if a lease file already exists for the detected IP,
the job fails fast with a clear error rather than silently sharing a WinRM target.
§2.5 — Versioned snapshot (Invoke-CIJob.ps1 + runner/config.yaml)
$SnapshotName now defaults to GITEA_CI_SNAPSHOT_NAME env var, falling back
to 'BaseClean'. config.yaml documents the commented-out variable for snapshot
refresh workflow (BaseClean_<yyyyMMdd> naming convention).
§2.3 — Artifact/log retention (scripts/Invoke-RetentionPolicy.ps1)
Purges per-job subdirs under F:\CI\Artifacts and F:\CI\Logs older than
RetentionDays (default 30). Switches to AggressiveRetentionDays (default 7)
when F: free space drops below MinFreeGB (default 50 GB). Also removes stale
IP lease files older than 12 hours (defensive cleanup for crashed jobs).
SupportsShouldProcess (-WhatIf) for dry-run.
§2.2 — Scheduled task registration (scripts/Register-CIScheduledTasks.ps1)
Registers two tasks under \CI\ task folder as SYSTEM/HighestPrivilege:
CI-CleanupOrphans — every 6h + AtStartup, -MaxAgeHours 4
CI-RetentionPolicy — daily 03:00 (+30min random) + AtStartup (+15min random)
Idempotent (-Force), SupportsShouldProcess (-WhatIf), validates script paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Simone