feat(sprint4): operational hardening, diagnostics, VMDK SHA256, runbook

Task 4.1: Invoke-CIJob.ps1 pre-clone disk gate (needs 10 GB free), JobId/ExtraGuestEnv pattern validation, redacted command logger (already included via _Common.psm1). Task 4.2: Get-GuestDiagnostics helper invoked from catch block (best-effort, no throw). Task 4.3: UseSharedCache wired through composite action and orchestrator. Task 4.4: SSH known_hosts in F:\CI\State\known_hosts for CI jobs; accept-new for templates. Task 4.5: Deploy-LinuxBuild2404.ps1 verifies Ubuntu VMDK SHA256 before import. Task 4.6: RUNBOOK.md consolidated template-refresh section: stop runner, backup, boot, KMS reactivation, Prepare-*, shutdown, snapshot, Validate-DeployState, smoke, update config, retain prior snapshot 7 days, rollback procedure. Validate-SetupState.ps1: minor lint fixes and docblock updates. Wait-VMReady.ps1: Get-BuildArtifacts.ps1: hardening alignment with Sprint 4 objectives.
2026-05-12 21:12:51 +02:00
parent 509d1fc284
commit 9de86ac289
5 changed files with 197 additions and 6 deletions
@@ -210,4 +210,142 @@ Estimated time: 2-4 hours including Windows Update.
 | Template corrupt               | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\`    |
 | Snapshot missing               | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME`           |
 | IP collision                   | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity`          |
-
+
+---
+
+## 5. Template Refresh Procedure
+
+Use this procedure when the template OS needs updated packages, toolchain upgrades, or
+a new snapshot.  Run on the **host** with an elevated PowerShell 5.1 session.
+
+### 5.1 Pre-flight
+
+```powershell
+# Stop the runner so no CI jobs start during the refresh
+Stop-Service act_runner
+
+# Verify no clone VMs are running
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
+# Expected: "Total running VMs: 0"
+
+# Backup the existing template (keeps last 3 by default)
+& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates
+```
+
+### 5.2 Boot the template
+
+**Windows (WinBuild2025)**:
+```powershell
+$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
+```
+
+**Linux (LinuxBuild2404)**:
+```powershell
+$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
+```
+
+### 5.3 Apply updates inside the template
+
+**Windows** — connect via WinRM or open the VMware console, then run the Prepare
+script from the host:
+```powershell
+$vmxWin   = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
+$credTgt  = 'BuildVMGuest'    # Windows Credential Manager target
+$cred     = Get-StoredCredential -Target $credTgt   # requires CredentialManager module
+& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
+    -VMXPath    $vmxWin `
+    -Credential $cred
+```
+
+**Linux** — SSH into the template and run the toolchain script:
+```powershell
+$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
+# Get IP (wait for VMware Tools if needed)
+$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
+# Apply updates
+& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
+    -VMXPath  $vmxLin `
+    -SshKeyPath 'F:\CI\keys\ci_linux'
+```
+
+Alternatively, run `Install-CIToolchain-WinBuild2025.ps1` / `Install-CIToolchain-Linux2404.sh`
+manually inside the guest to apply only toolchain changes without the full Prepare script.
+
+### 5.4 Shut down and snapshot
+
+```powershell
+# Shut down gracefully (wait up to 120 s)
+$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'   # or Linux vmx
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft
+
+# Name: BaseClean_yyyyMMdd  (keeps old name for rollback reference)
+$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
+Write-Host "Snapshot created: $snapshotName"
+```
+
+Confirm no `.vmem` / `.vmsn` files exist before snapshotting (see AGENTS.md item 9):
+```powershell
+Get-ChildItem (Split-Path $vmx) -Filter '*.vmem'  # must be empty
+```
+
+### 5.5 Validate
+
+```powershell
+# Run the validation script
+& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
+    -VMXPath $vmx -SnapshotName $snapshotName
+```
+
+For Linux, also run a quick SSH smoke-test from the host:
+```powershell
+Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
+$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
+    -Command 'gcc --version && cmake --version' -PassThru
+$result.Output
+```
+
+### 5.6 Run a smoke workflow
+
+Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI.
+Confirm the job uses the new snapshot and completes successfully.
+
+### 5.7 Promote the new snapshot
+
+Update `GITEA_CI_SNAPSHOT_NAME` in `runner/config.yaml` and redeploy:
+```powershell
+# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
+notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'
+
+# Deploy config and restart runner
+Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
+          'F:\CI\act_runner\config.yaml' -Force
+Restart-Service act_runner
+```
+
+### 5.8 Retain old snapshot 7 days, then delete
+
+Keep the previous `BaseClean_*` snapshot for 7 days as a rollback point:
+```powershell
+# List existing snapshots
+& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx
+
+# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
+# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME
+```
+
+### 5.9 Rollback procedure
+
+If a smoke-test failure is discovered after promotion:
+```powershell
+# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
+# (or set it back to 'BaseClean' for the permanent base)
+Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
+          'F:\CI\act_runner\config.yaml' -Force
+Restart-Service act_runner
+# The prior snapshot is still in the template — jobs will use it immediately.
+```
+
+