feat(sprint4): operational hardening, diagnostics, VMDK SHA256, runbook
Task 4.1: Invoke-CIJob.ps1 pre-clone disk gate (needs 10 GB free), JobId/ExtraGuestEnv
pattern validation, redacted command logger (already included via _Common.psm1).
Task 4.2: Get-GuestDiagnostics helper invoked from catch block (best-effort, no throw).
Task 4.3: UseSharedCache wired through composite action and orchestrator.
Task 4.4: SSH known_hosts in F:\CI\State\known_hosts for CI jobs; accept-new for templates.
Task 4.5: Deploy-LinuxBuild2404.ps1 verifies Ubuntu VMDK SHA256 before import.
Task 4.6: RUNBOOK.md consolidated template-refresh section: stop runner, backup, boot,
KMS reactivation, Prepare-*, shutdown, snapshot, Validate-DeployState, smoke,
update config, retain prior snapshot 7 days, rollback procedure.
Validate-SetupState.ps1: minor lint fixes and docblock updates.
Wait-VMReady.ps1: Get-BuildArtifacts.ps1: hardening alignment with Sprint 4 objectives.
This commit is contained in:
Simone
+139
-1
@@ -210,4 +210,142 @@ Estimated time: 2-4 hours including Windows Update.
|
||||
| Template corrupt | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` |
|
||||
| Snapshot missing | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME` |
|
||||
| IP collision | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity` |
|
||||
|
||||
|
||||
---
|
||||
|
||||
## 5. Template Refresh Procedure
|
||||
|
||||
Use this procedure when the template OS needs updated packages, toolchain upgrades, or
|
||||
a new snapshot. Run on the **host** with an elevated PowerShell 5.1 session.
|
||||
|
||||
### 5.1 Pre-flight
|
||||
|
||||
```powershell
|
||||
# Stop the runner so no CI jobs start during the refresh
|
||||
Stop-Service act_runner
|
||||
|
||||
# Verify no clone VMs are running
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' list
|
||||
# Expected: "Total running VMs: 0"
|
||||
|
||||
# Backup the existing template (keeps last 3 by default)
|
||||
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Backup-CITemplate.ps1' -AllTemplates
|
||||
```
|
||||
|
||||
### 5.2 Boot the template
|
||||
|
||||
**Windows (WinBuild2025)**:
|
||||
```powershell
|
||||
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
||||
```
|
||||
|
||||
**Linux (LinuxBuild2404)**:
|
||||
```powershell
|
||||
$vmx = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' start $vmx gui
|
||||
```
|
||||
|
||||
### 5.3 Apply updates inside the template
|
||||
|
||||
**Windows** — connect via WinRM or open the VMware console, then run the Prepare
|
||||
script from the host:
|
||||
```powershell
|
||||
$vmxWin = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx'
|
||||
$credTgt = 'BuildVMGuest' # Windows Credential Manager target
|
||||
$cred = Get-StoredCredential -Target $credTgt # requires CredentialManager module
|
||||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-WinBuild2025.ps1' `
|
||||
-VMXPath $vmxWin `
|
||||
-Credential $cred
|
||||
```
|
||||
|
||||
**Linux** — SSH into the template and run the toolchain script:
|
||||
```powershell
|
||||
$vmxLin = 'F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx'
|
||||
# Get IP (wait for VMware Tools if needed)
|
||||
$ip = & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' getGuestIPAddress $vmxLin -wait
|
||||
# Apply updates
|
||||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Prepare-LinuxBuild2404.ps1' `
|
||||
-VMXPath $vmxLin `
|
||||
-SshKeyPath 'F:\CI\keys\ci_linux'
|
||||
```
|
||||
|
||||
Alternatively, run `Install-CIToolchain-WinBuild2025.ps1` / `Install-CIToolchain-Linux2404.sh`
|
||||
manually inside the guest to apply only toolchain changes without the full Prepare script.
|
||||
|
||||
### 5.4 Shut down and snapshot
|
||||
|
||||
```powershell
|
||||
# Shut down gracefully (wait up to 120 s)
|
||||
$vmx = 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' # or Linux vmx
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' stop $vmx soft
|
||||
|
||||
# Name: BaseClean_yyyyMMdd (keeps old name for rollback reference)
|
||||
$snapshotName = "BaseClean_$(Get-Date -Format 'yyyyMMdd')"
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' snapshot $vmx $snapshotName
|
||||
Write-Host "Snapshot created: $snapshotName"
|
||||
```
|
||||
|
||||
Confirm no `.vmem` / `.vmsn` files exist before snapshotting (see AGENTS.md item 9):
|
||||
```powershell
|
||||
Get-ChildItem (Split-Path $vmx) -Filter '*.vmem' # must be empty
|
||||
```
|
||||
|
||||
### 5.5 Validate
|
||||
|
||||
```powershell
|
||||
# Run the validation script
|
||||
& 'N:\Code\Workspace\Local-CI-CD-System\template\Validate-DeployState.ps1' `
|
||||
-VMXPath $vmx -SnapshotName $snapshotName
|
||||
```
|
||||
|
||||
For Linux, also run a quick SSH smoke-test from the host:
|
||||
```powershell
|
||||
Import-Module 'N:\Code\Workspace\Local-CI-CD-System\scripts\_Transport.psm1' -Force
|
||||
$result = Invoke-SshCommand -IP $ip -KeyPath 'F:\CI\keys\ci_linux' `
|
||||
-Command 'gcc --version && cmake --version' -PassThru
|
||||
$result.Output
|
||||
```
|
||||
|
||||
### 5.6 Run a smoke workflow
|
||||
|
||||
Push a trivial commit to a test repo or trigger a manual workflow run via Gitea UI.
|
||||
Confirm the job uses the new snapshot and completes successfully.
|
||||
|
||||
### 5.7 Promote the new snapshot
|
||||
|
||||
Update `GITEA_CI_SNAPSHOT_NAME` in `runner/config.yaml` and redeploy:
|
||||
```powershell
|
||||
# Edit runner/config.yaml: set GITEA_CI_SNAPSHOT_NAME to $snapshotName
|
||||
notepad 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml'
|
||||
|
||||
# Deploy config and restart runner
|
||||
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
||||
'F:\CI\act_runner\config.yaml' -Force
|
||||
Restart-Service act_runner
|
||||
```
|
||||
|
||||
### 5.8 Retain old snapshot 7 days, then delete
|
||||
|
||||
Keep the previous `BaseClean_*` snapshot for 7 days as a rollback point:
|
||||
```powershell
|
||||
# List existing snapshots
|
||||
& 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' listSnapshots $vmx
|
||||
|
||||
# After 7 days, delete the old snapshot (replace OLDNAME with actual name)
|
||||
# & 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' deleteSnapshot $vmx OLDNAME
|
||||
```
|
||||
|
||||
### 5.9 Rollback procedure
|
||||
|
||||
If a smoke-test failure is discovered after promotion:
|
||||
```powershell
|
||||
# Revert runner/config.yaml to prior GITEA_CI_SNAPSHOT_NAME
|
||||
# (or set it back to 'BaseClean' for the permanent base)
|
||||
Copy-Item 'N:\Code\Workspace\Local-CI-CD-System\runner\config.yaml' `
|
||||
'F:\CI\act_runner\config.yaml' -Force
|
||||
Restart-Service act_runner
|
||||
# The prior snapshot is still in the template — jobs will use it immediately.
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -61,7 +61,11 @@ param(
|
||||
[string] $SshKeyPath = 'F:\CI\keys\ci_linux',
|
||||
|
||||
# SSH username (used when GuestOS=Linux).
|
||||
[string] $SshUser = 'ci_build'
|
||||
[string] $SshUser = 'ci_build',
|
||||
|
||||
# Persistent known_hosts file for SSH calls (CI jobs).
|
||||
# Pass an empty string to disable host-key checking (template provisioning only).
|
||||
[string] $SshKnownHostsFile = 'F:\CI\State\known_hosts'
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
@@ -90,6 +94,7 @@ if ($GuestOS -eq 'Linux') {
|
||||
-IP $IPAddress `
|
||||
-User $SshUser `
|
||||
-KeyPath $SshKeyPath `
|
||||
-KnownHostsFile $SshKnownHostsFile `
|
||||
-Direction 'FromGuest' `
|
||||
-Recurse
|
||||
|
||||
|
||||
@@ -38,6 +38,13 @@
|
||||
WinRM (default): checks ICMP ping + TCP 5986 (Windows VMs).
|
||||
SSH: checks TCP 22 + ssh echo (Linux VMs).
|
||||
|
||||
.PARAMETER Credential
|
||||
Optional PSCredential for Windows VMs. When supplied, Wait-VMReady performs
|
||||
a real Invoke-Command { 'ready' } probe after TCP 5986 succeeds, confirming
|
||||
that WinRM authentication (not just port) is working. Useful to detect the
|
||||
window where the WinRM listener is up but the LocalSystem session is not yet
|
||||
accepting logins. When omitted the TCP check alone is used (existing behaviour).
|
||||
|
||||
.PARAMETER SshKeyPath
|
||||
Path to the SSH private key for key-based auth when Transport=SSH.
|
||||
Default: F:\CI\keys\ci_linux
|
||||
@@ -82,7 +89,11 @@ param(
|
||||
[string] $SshKeyPath = 'F:\CI\keys\ci_linux',
|
||||
|
||||
# SSH username (used when Transport=SSH).
|
||||
[string] $SshUser = 'ci_build'
|
||||
[string] $SshUser = 'ci_build',
|
||||
|
||||
# Optional credential for WinRM authentication probe (Transport=WinRM).
|
||||
# When supplied, performs Invoke-Command { 'ready' } after TCP 5986 succeeds.
|
||||
[PSCredential] $Credential = $null
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
@@ -203,6 +214,24 @@ while ((Get-Date) -lt $deadline) {
|
||||
-InformationLevel Quiet -WarningAction SilentlyContinue `
|
||||
-ErrorAction SilentlyContinue
|
||||
if ($winrmUp) {
|
||||
# Optional auth probe — ensures WinRM login is functional, not just TCP open
|
||||
if ($null -ne $Credential) {
|
||||
try {
|
||||
$so = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
|
||||
$null = Invoke-Command -ComputerName $IPAddress -Credential $Credential `
|
||||
-Authentication Basic -UseSSL -Port 5986 -SessionOption $so `
|
||||
-ScriptBlock { 'ready' } -ErrorAction Stop
|
||||
Write-Host "[Wait-VMReady] WinRM auth confirmed for $IPAddress."
|
||||
} catch {
|
||||
$phase = 'winrm-auth'
|
||||
if ($lastPhase -ne $phase) {
|
||||
Write-Host "[Wait-VMReady] Phase 3/3: WinRM port open, waiting for auth at $IPAddress..."
|
||||
$lastPhase = $phase
|
||||
}
|
||||
Start-Sleep -Seconds $PollIntervalSeconds
|
||||
continue
|
||||
}
|
||||
}
|
||||
Write-Host "[Wait-VMReady] VM ready after ~$([int]($attempt * $PollIntervalSeconds))s (attempt $attempt)."
|
||||
return
|
||||
}
|
||||
|
||||
@@ -79,6 +79,11 @@
|
||||
.PARAMETER VmdkCachePath
|
||||
Local cache path for the downloaded VMDK. Reused across runs.
|
||||
|
||||
.PARAMETER VmdkSha256
|
||||
Expected SHA256 hex digest of the cached VMDK file (case-insensitive).
|
||||
When non-empty the script throws if the computed hash does not match.
|
||||
Leave empty to skip hash verification (useful during development).
|
||||
|
||||
.PARAMETER SeedIsoPath
|
||||
Path for the cloud-init seed ISO. Default: <VMDir>\cloud-init-seed.iso.
|
||||
|
||||
@@ -136,6 +141,9 @@ param(
|
||||
[string] $SshKeyPath = 'F:\CI\keys\ci_linux',
|
||||
[string] $VmdkUrl = 'https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.vmdk',
|
||||
[string] $VmdkCachePath = 'F:\CI\ISO\noble-server-cloudimg-amd64.vmdk',
|
||||
# Expected SHA256 of the downloaded VMDK (hex, case-insensitive).
|
||||
# Leave empty to skip hash verification (not recommended for production).
|
||||
[string] $VmdkSha256 = '',
|
||||
[string] $SeedIsoPath = '',
|
||||
[int] $SshTimeoutSeconds = 300,
|
||||
[int] $SshPollIntervalSeconds = 10,
|
||||
@@ -376,6 +384,16 @@ if (-not (Test-Path $VmdkCachePath)) {
|
||||
}
|
||||
Assert-Step 'PreFlight' "Cloud VMDK cache present: $VmdkCachePath" { Test-Path $VmdkCachePath }
|
||||
|
||||
# Step 1c -- Verify VMDK SHA256 (when caller provides expected hash)
|
||||
if ($VmdkSha256 -ne '') {
|
||||
Write-Host "[Deploy] Verifying VMDK SHA256 ..."
|
||||
$actualHash = (Get-FileHash -LiteralPath $VmdkCachePath -Algorithm SHA256).Hash
|
||||
if ($actualHash -ne $VmdkSha256.ToUpper()) {
|
||||
throw "[Deploy] VMDK SHA256 mismatch!`n Expected: $($VmdkSha256.ToUpper())`n Actual: $actualHash"
|
||||
}
|
||||
Write-Host "[Deploy] VMDK SHA256 OK: $actualHash"
|
||||
}
|
||||
|
||||
} # end Step 1
|
||||
|
||||
# =============================================================================
|
||||
@@ -676,4 +694,4 @@ Write-Host " Next steps (from template\ dir):"
|
||||
Write-Host " .\Prepare-LinuxBuild2404.ps1 -VMXPath '$VMXPath' -VMIPAddress $guestIP"
|
||||
Write-Host ""
|
||||
Write-Host " VM is powered on. cloud-init provisioning complete." -ForegroundColor Green
|
||||
|
||||
|
||||
|
||||
@@ -185,7 +185,8 @@ try {
|
||||
(Get-LocalUser -Name $BuildUsername -EA Stop).Enabled
|
||||
}
|
||||
Chk "user $BuildUsername PasswordNeverExpires" {
|
||||
(Get-LocalUser -Name $BuildUsername -EA Stop).PasswordExpires -eq $null
|
||||
$u = Get-LocalUser -Name $BuildUsername -EA Stop
|
||||
(-not $u.PasswordExpires) -or ($u.PasswordExpires -eq [datetime]::MaxValue)
|
||||
}
|
||||
Chk "user $BuildUsername member of Administrators" {
|
||||
[bool](& net localgroup Administrators 2>&1 | Select-String -SimpleMatch $BuildUsername)
|
||||
@@ -369,7 +370,7 @@ try {
|
||||
Chk 'WU GPO DisableWindowsUpdateAccess=1' { (Get-ItemProperty 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate' -Name DisableWindowsUpdateAccess -EA Stop).DisableWindowsUpdateAccess -eq 1 }
|
||||
Chk "user $BuildUsername exists" { $null -ne (Get-LocalUser -Name $BuildUsername -EA Stop) }
|
||||
Chk "user $BuildUsername enabled" { (Get-LocalUser -Name $BuildUsername -EA Stop).Enabled }
|
||||
Chk "user $BuildUsername PasswordNeverExpires" { (Get-LocalUser -Name $BuildUsername -EA Stop).PasswordExpires -eq $null }
|
||||
Chk "user $BuildUsername PasswordNeverExpires" { $u = Get-LocalUser -Name $BuildUsername -EA Stop; (-not $u.PasswordExpires) -or ($u.PasswordExpires -eq [datetime]::MaxValue) }
|
||||
Chk "user $BuildUsername member of Administrators" { [bool](& net localgroup Administrators 2>&1 | Select-String -SimpleMatch $BuildUsername) }
|
||||
foreach ($dir in 'C:\CI\build','C:\CI\output','C:\CI\scripts') { Chk "$dir exists" { Test-Path $dir -PathType Container } }
|
||||
Chk ".NET SDK channel $DotNetChannel" { $v=(& dotnet --version 2>&1) -as [string]; $v -and $v.StartsWith($DotNetChannel) }
|
||||
|
||||
Reference in New Issue
Block a user