Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sprint2): §2.1/2.2/2.3/2.5 — IP lock/lease, scheduled tasks, retention, versioned snapshot

Browse Source 
§2.1 — IP race with capacity>1 (Invoke-CIJob.ps1)
  Phases 2-3b (clone+start+IP) now run inside a file-based exclusive mutex
  (F:\CI\State\vm-start.lock via FileShare.None). Only one job at a time is
  in the VM-start phase, preventing concurrent vmrun starts from racing for
  the same DHCP lease. After IP is confirmed unique it is written to an IP
  lease file (F:\CI\State\ip-leases\<IP>.lease). Lock is released immediately
  so build phases run fully in parallel. Lease released in finally block.
  Collision detection: if a lease file already exists for the detected IP,
  the job fails fast with a clear error rather than silently sharing a WinRM target.

§2.5 — Versioned snapshot (Invoke-CIJob.ps1 + runner/config.yaml)
  $SnapshotName now defaults to GITEA_CI_SNAPSHOT_NAME env var, falling back
  to 'BaseClean'. config.yaml documents the commented-out variable for snapshot
  refresh workflow (BaseClean_<yyyyMMdd> naming convention).

§2.3 — Artifact/log retention (scripts/Invoke-RetentionPolicy.ps1)
  Purges per-job subdirs under F:\CI\Artifacts and F:\CI\Logs older than
  RetentionDays (default 30). Switches to AggressiveRetentionDays (default 7)
  when F: free space drops below MinFreeGB (default 50 GB). Also removes stale
  IP lease files older than 12 hours (defensive cleanup for crashed jobs).
  SupportsShouldProcess (-WhatIf) for dry-run.

§2.2 — Scheduled task registration (scripts/Register-CIScheduledTasks.ps1)
  Registers two tasks under \CI\ task folder as SYSTEM/HighestPrivilege:
    CI-CleanupOrphans  — every 6h + AtStartup, -MaxAgeHours 4
    CI-RetentionPolicy — daily 03:00 (+30min random) + AtStartup (+15min random)
  Idempotent (-Force), SupportsShouldProcess (-WhatIf), validates script paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Simone
2026-05-10 18:36:38 +02:00
parent d4e619ddc2
commit bc894a8058
5 changed files with 353 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/Invoke-CIJob.ps1
+87 -25
View File
@@ -97,7 +97,7 @@ param(
[string] $TemplatePath = $env:GITEA_CI_TEMPLATE_PATH,
[string] $SnapshotName = 'BaseClean',
[string] $SnapshotName = $(if ($env:GITEA_CI_SNAPSHOT_NAME) { $env:GITEA_CI_SNAPSHOT_NAME } else { 'BaseClean' }),
[string] $CloneBaseDir = $(if ($env:GITEA_CI_CLONE_BASE_DIR) { $env:GITEA_CI_CLONE_BASE_DIR } else { 'F:\CI\BuildVMs' }),
@@ -158,6 +158,7 @@ catch {
$jobArtifactDir = Join-Path $ArtifactBaseDir $JobId
$hostCloneDir = Join-Path $env:TEMP "ci-src-$JobId"
$cloneVmxPath = $null
$leaseFile = $null # §2.1 IP lease file path — released in finally
$startTime = Get-Date
# ── Start transcript ──────────────────────────────────────────────────────────
@@ -224,32 +225,87 @@ try {
}
Write-Host "[Invoke-CIJob] Source cloned to: $hostCloneDir"
# ── Phase 2: Clone VM ─────────────────────────────────────────────────
Write-Host "`n[Phase 2/6] Cloning VM..."
$cloneVmxPath = & "$scriptDir\New-BuildVM.ps1" `
-TemplatePath $TemplatePath `
-SnapshotName $SnapshotName `
-CloneBaseDir $CloneBaseDir `
-JobId $JobId `
-VmrunPath $VmrunPath
# ── Phase 2-3b: Clone + Start + IP (serialized via file-based mutex) ────
# §2.1: With capacity:4 and DHCP on VMnet8, concurrent vmrun starts can race
# and produce two VMs with the same DHCP lease. A file-based mutex ensures
# only one job is in the clone→start→IP phase at a time. After the IP is
# acquired and written to a lease file the lock is released, so build phases
# (the slow part) run fully in parallel. Lease is released in finally.
$stateDir = 'F:\CI\State'
$leasesDir = Join-Path $stateDir 'ip-leases'
$lockPath = Join-Path $stateDir 'vm-start.lock'
if (-not (Test-Path $leasesDir)) { New-Item -ItemType Directory -Path $leasesDir -Force | Out-Null }
# ── Phase 3: Start VM ─────────────────────────────────────────────────
Write-Host "`n[Phase 3/6] Starting VM..."
$startOutput = & $VmrunPath -T ws start $cloneVmxPath nogui 2>&1
if ($LASTEXITCODE -ne 0) {
throw "vmrun start failed (exit $LASTEXITCODE): $startOutput"
}
Write-Host "[Invoke-CIJob] VM started."
# ── Phase 3b: Resolve VM IP if not provided ─────────────────────────
if ([string]::IsNullOrWhiteSpace($VMIPAddress)) {
Write-Host "[Invoke-CIJob] Auto-detecting VM IP via vmrun getGuestIPAddress..."
$detectedIP = & $VmrunPath getGuestIPAddress $cloneVmxPath -wait 2>&1
if ($LASTEXITCODE -ne 0 -or $detectedIP -notmatch '^(\d{1,3}\.){3}\d{1,3}$') {
throw "Could not detect VM IP address: $detectedIP"
Write-Host "`n[Phase 2/6] Acquiring VM-start lock (prevents DHCP collision with capacity>1)..."
$lockStream = $null
$lockDeadline = (Get-Date).AddMinutes(10)
while ($true) {
try {
$lockStream = [System.IO.File]::Open(
$lockPath,
[System.IO.FileMode]::OpenOrCreate,
[System.IO.FileAccess]::ReadWrite,
[System.IO.FileShare]::None)
break
} catch [System.IO.IOException] {
if ((Get-Date) -ge $lockDeadline) {
throw "[Phase 2] VM-start lock timeout after 10 min — a concurrent job may be stuck."
}
Write-Host "[Invoke-CIJob] VM-start lock busy, retrying in 5s..."
Start-Sleep -Seconds 5
}
$VMIPAddress = $detectedIP.Trim()
Write-Host "[Invoke-CIJob] Detected VM IP: $VMIPAddress"
}
Write-Host "[Invoke-CIJob] VM-start lock acquired."
try {
# ── Phase 2: Clone VM ─────────────────────────────────────────────
Write-Host "[Phase 2/6] Cloning VM..."
$cloneVmxPath = & "$scriptDir\New-BuildVM.ps1" `
-TemplatePath $TemplatePath `
-SnapshotName $SnapshotName `
-CloneBaseDir $CloneBaseDir `
-JobId $JobId `
-VmrunPath $VmrunPath
# ── Phase 3: Start VM ─────────────────────────────────────────────
Write-Host "`n[Phase 3/6] Starting VM..."
$startOutput = & $VmrunPath -T ws start $cloneVmxPath nogui 2>&1
if ($LASTEXITCODE -ne 0) {
throw "vmrun start failed (exit $LASTEXITCODE): $startOutput"
}
Write-Host "[Invoke-CIJob] VM started."
# ── Phase 3b: Resolve VM IP ───────────────────────────────────────
if ([string]::IsNullOrWhiteSpace($VMIPAddress)) {
Write-Host "[Invoke-CIJob] Auto-detecting VM IP via vmrun getGuestIPAddress -wait..."
$detectedIP = & $VmrunPath getGuestIPAddress $cloneVmxPath -wait 2>&1
if ($LASTEXITCODE -ne 0 -or $detectedIP -notmatch '^(\d{1,3}\.){3}\d{1,3}$') {
throw "Could not detect VM IP address: $detectedIP"
}
$VMIPAddress = $detectedIP.Trim()
Write-Host "[Invoke-CIJob] Detected VM IP: $VMIPAddress"
}
# ── Lease IP ──────────────────────────────────────────────────────
# If two VMs somehow got the same IP (DHCP edge case), fail fast here
# rather than letting both jobs write to the same WinRM target.
$leaseFile = Join-Path $leasesDir "$($VMIPAddress.Replace('.', '-')).lease"
if (Test-Path $leaseFile) {
$holder = (Get-Content $leaseFile -Raw -ErrorAction SilentlyContinue).Trim()
throw "IP collision: $VMIPAddress already leased by job '$holder'. Retry or reduce runner capacity."
}
[System.IO.File]::WriteAllText($leaseFile, $JobId)
Write-Host "[Invoke-CIJob] IP $VMIPAddress leased for job $JobId."
} finally {
# Release lock — next waiting job can now enter clone→start→IP phase
if ($lockStream) {
$lockStream.Close()
$lockStream.Dispose()
$lockStream = $null
}
Remove-Item $lockPath -Force -ErrorAction SilentlyContinue
Write-Host "[Invoke-CIJob] VM-start lock released."
}
# ── Phase 4: Wait for readiness ───────────────────────────────────────
@@ -295,6 +351,12 @@ catch {
Write-Host "============================================================"
}
finally {
# ── Release IP lease (§2.1) ───────────────────────────────────────────
if ($leaseFile -and (Test-Path $leaseFile)) {
Remove-Item $leaseFile -Force -ErrorAction SilentlyContinue
Write-Host "[Cleanup] IP lease released: $VMIPAddress"
}
# ── Always destroy the VM ─────────────────────────────────────────────
if ($null -ne $cloneVmxPath -and (Test-Path $cloneVmxPath -PathType Leaf)) {
Write-Host "`n[Cleanup] Destroying ephemeral VM..."