diff --git a/.gitea/workflows/lint.yml b/.gitea/workflows/lint.yml index 6b43a48..955e39c 100644 --- a/.gitea/workflows/lint.yml +++ b/.gitea/workflows/lint.yml @@ -1,8 +1,13 @@ -# PSScriptAnalyzer + Python lint — runs on every push/PR that touches code files. -# Requires PSScriptAnalyzer installed on the runner host: -# Install-Module PSScriptAnalyzer -Scope AllUsers -Force +# Python lint runs on every push/PR (linux-build runner) and blocks the PR. # -# Failures block the PR. Fix warnings with: +# PSScriptAnalyzer (pssa) runs on the Windows runner (windows-build) on-demand +# only — triggered via workflow_dispatch — because the .ps1/.psm1 targets are +# Windows-native (PS 5.1). It is intentionally NOT queued on push/PR so it does +# not block the pipeline when no Windows runner is registered (dual-boot Passo 9 +# may be down). This keeps the Linux host/runner free of pwsh (Phase C goal). +# Requires PSScriptAnalyzer installed on the Windows runner host: +# Install-Module PSScriptAnalyzer -Scope AllUsers -Force +# Fix warnings with: # Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Fix @@ -27,7 +32,10 @@ on: jobs: pssa: - runs-on: linux-build + # On-demand only: runs on the Windows runner (PS 5.1), never on push/PR, + # so it cannot block the pipeline when no Windows runner is registered. + if: ${{ github.event_name == 'workflow_dispatch' }} + runs-on: windows-build timeout-minutes: 10 steps: @@ -37,7 +45,7 @@ jobs: fetch-depth: 1 - name: Run PSScriptAnalyzer - shell: pwsh + shell: powershell run: | $ErrorActionPreference = 'Stop' diff --git a/CLAUDE.md b/CLAUDE.md index 06674e4..740494b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,6 +10,8 @@ Self-hosted CI/CD system running on a Windows 11 host with VMware Workstation. E **Current state**: The Python orchestrator (`src/ci_orchestrator/`) is the production entry point. The PowerShell scripts in `scripts/` are 3-line shims that forward to the Python CLI. All new logic goes in Python. +**Phase C (PowerShell-free Linux host)**: the manual burn-in / benchmark / smoke / host-validation / credential-setup tooling that used to run on the Linux host via `pwsh` is now native Python: `bench run`, `bench measure`, `smoke run`, `validate host`, `validate guest`, `creds set`. The Linux host no longer needs `pwsh` for any operation — it may be uninstalled (`setup-host-linux.sh` no longer installs it by default; opt in with `--with-pwsh`). PSScriptAnalyzer lint runs on a Windows runner on-demand, not on the Linux runner. + --- ## Development Commands @@ -115,6 +117,8 @@ The config file is auto-discovered at `$CI_ROOT/config.toml` (Linux default: `/v ## PowerShell 5.1 Constraints +These constraints apply to the `.ps1` / `.psm1` scripts that still run **inside Windows guests** and on the optional Windows runner. The **Linux host itself no longer runs any `.ps1`** (Phase C) — do not add PowerShell to the Linux host orchestration path. + All `.ps1` / `.psm1` scripts run on **Windows PowerShell 5.1** (not Core/7). Every script must start with: ```powershell diff --git a/TODO.md b/TODO.md index 8911adb..c8dcc82 100644 --- a/TODO.md +++ b/TODO.md @@ -849,17 +849,18 @@ liberi su 63.7). **Mitigazioni**: ridurre vCPU/VM (lega a §3.5), `-Parallelism alzare connect-timeout pypsrp + retry a livello job in `_windows_build`. **Rifiori quando**: si vuole concorrenza 4× Windows-guest affidabile su host Windows. -### 3.7 [P3] [ ] Shim Invoke-CIJob.ps1 — `$IsWindows` non guardato (PS 5.1 + StrictMode) +### 3.7 [P3] [x] Shim Invoke-CIJob.ps1 — `$IsWindows` non guardato (PS 5.1 + StrictMode) — RISOLTO 2026-06-07 (Fase C) File: [scripts/Invoke-CIJob.ps1:37](scripts/Invoke-CIJob.ps1). `if ($null -ne $IsWindows -and $IsWindows -eq $false)` dereferenzia `$IsWindows`, inesistente su Windows PowerShell 5.1 → con `Set-StrictMode -Version Latest` -lancia `VariableIsUndefined` e il job fallisce subito. Si manifesta solo quando lo -shim gira sotto PS 5.1 **senza** `CI_VENV_PYTHON` settato (il ramo auto-detect). -Produzione setta `CI_VENV_PYTHON` via runner, quindi non si vede; colpisce invocazioni -manuali/burn-in. **Fix**: guardare con `if ((Test-Path variable:IsWindows) -and -not $IsWindows)` -o `Get-Variable IsWindows -EA SilentlyContinue`. -**Workaround attuale**: esportare `CI_VENV_PYTHON` prima di lanciare lo shim. +lancia `VariableIsUndefined` e il job fallisce subito. Si manifestava solo quando lo +shim girava sotto PS 5.1 **senza** `CI_VENV_PYTHON` settato (il ramo auto-detect), +cioè nelle invocazioni manuali/burn-in. +**Risoluzione**: il percorso burn-in manuale non passa più dallo shim — `bench run` +(Fase C) lancia i job concorrenti in-process via `concurrent.futures`, senza +`Start-Job`/`pwsh`/`$IsWindows`. Lo shim resta solo per il runner Windows, dove +`CI_VENV_PYTHON` è sempre settato. Il bug non è più raggiungibile sull'host Linux. ### 8.1 [P3] [ ] Mirror GitHub Actions su Gitea locale (rimuovi dipendenza github.com) File: [.gitea/workflows/lint.yml](.gitea/workflows/lint.yml), Gitea (org `actions`). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 76c1215..5bed577 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -738,8 +738,28 @@ running §11–§13; required for any Windows-host orchestration: `Invoke-CIJob.ps1` interactively. The shim's auto-detect branch dereferences `$IsWindows`, which is undefined under Windows PowerShell 5.1 + `StrictMode` and throws `VariableIsUndefined`; setting `CI_VENV_PYTHON` skips that branch. - Production sets it via the runner (`CI_PYTHON_LAUNCHER`). A code fix to guard - the `$IsWindows` reference is tracked in `TODO.md`. + Production sets it via the runner (`CI_PYTHON_LAUNCHER`). *Note (Phase C): on + the Linux host this is moot — burn-in now runs via `bench run` (below), which + never touches the shim.* + +### Phase C — Python equivalents (run these on the Linux host; no `pwsh`) + +The manual `.ps1` procedures above are superseded by native CLI commands. Invoke +as the service user (`sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator ...`): + +| Old PowerShell (Linux host) | Python command | +| --------------------------------------------------- | ------------------------------------------------ | +| `Test-CapacityBurnIn.ps1 -Parallelism 4 -Rounds 10` | `bench run --concurrency 4 --rounds 10 --guest-os linux` | +| `Measure-CIBenchmark.ps1 -Iterations N` | `bench measure --iterations N --guest-os linux` | +| `Test-Smoke.ps1 -GuestOS Linux` | `smoke run --guest-os linux` | +| `Validate-HostState.ps1` (host checks) | `validate host` | +| `Test-CIGuestWinRM.ps1` (transport diag) | `validate guest --host [--winrm\|--ssh]` | +| `Set-CIGuestCredential.ps1` | `creds set --target BuildVMGuest --user --password-stdin` | + +`bench run` writes a JSON report to `CI_ARTIFACTS/bench/`; `bench measure` appends +to `benchmark.jsonl` (same fields as before, so the §6–§13 trend lines stay +comparable). The benchmark records in §6–§13 above are kept as historical +provenance — they were captured with the `.ps1` tools before the cutover. --- diff --git a/plans/idea-3-implementation-plan.md b/plans/idea-3-implementation-plan.md new file mode 100644 index 0000000..d5250ed --- /dev/null +++ b/plans/idea-3-implementation-plan.md @@ -0,0 +1,185 @@ +# Fase C — Piano di applicazione (eliminazione `pwsh` dall'host Linux) + +> Piano operativo per [idea-3-powershell-removal.md](idea-3-powershell-removal.md). +> Generato il 2026-06-07. Eseguire i passi **in ordine**: ogni passo è +> autonomo, testabile e con rollback banale (i `.ps1` non vengono rimossi +> finché C9 non chiude il burn-in di validazione). + +--- + +## Summary — checklist master + +Stato passi (spunta man mano): + +- [x] **C1** — Scaffolding gruppi CLI `bench` / `smoke` / `validate` / `creds` + test skeleton +- [x] **C2** — `bench run`: porting burn-in concorrenza (`Test-CapacityBurnIn` + `Start-BurnInTest*`) +- [x] **C3** — `bench measure`: porting timing fasi (`Measure-CIBenchmark`) + retire `_Common.psm1`/`_Transport.psm1`/`_Common.Tests.ps1` +- [x] **C4** — `smoke run`: porting smoke + build E2E (`Test-Smoke`, `Test-*Build-Linux`) +- [x] **C5** — `validate host`: equivalente Linux di `Validate-HostState` +- [x] **C6** — `creds set` (rimpiazza `Set-CIGuestCredential`) + `validate guest` (diag transport) +- [x] **C7** — Lint CI: risolvere dipendenza `pwsh` di `lint.yml` (decisione §4 dell'analisi) +- [x] **C8** — `setup-host-linux.sh`: `pwsh` non più installato di default (`--with-pwsh` opt-in) +- [~] **C9** — Docs ✅ + cutover/burn-in ≥1 settimana ⏳ + disinstallazione `pwsh` ⏳ (operativi, vedi sotto) + +**Definition of done globale**: tutti i criteri del §7 dell'analisi +verificati; `pwsh` rimosso dall'host Linux; pytest verde ≥90%. + +--- + +## Convenzioni trasversali (valgono per ogni passo) + +- Codice nuovo solo in `src/ci_orchestrator/`. `click.echo`, no `print()`, + no `subprocess(..., shell=True)`, type hints completi, `mypy --strict`. +- Importare **solo** `backends.load_backend` + transport esistenti + (`transport/ssh.py`, `transport/winrm.py`). Mai `WorkstationVmrunBackend`. +- Ogni nuovo comando ha test in `tests/python/test_commands_.py` + con backend e transport mockati (no VM reali in unit test). +- Gate coverage 90% per ogni PR. `ruff` + `mypy --strict` verdi. +- Un passo = una PR. Non toccare il branch WinRM e SSH nello stesso commit + (gotcha CLAUDE.md): se un comando ha entrambi i rami, separarli o + isolare dietro `if guest_os == "linux"`. + +--- + +## C1 — Scaffolding + +**Obiettivo**: registrare i gruppi vuoti e i file, senza logica reale, +per fissare l'interfaccia CLI e i test. + +- [ ] Creare `commands/bench.py` con `@click.group("bench")` e i comandi + stub `run` / `measure` (firma completa, corpo `raise NotImplementedError` + dietro `--dry-run` per i test). +- [ ] Creare `commands/smoke.py` (`@click.group("smoke")` → `run`). +- [ ] Creare `commands/validate.py` (`@click.group("validate")` → `host`, `guest`). +- [ ] Creare `commands/creds.py` (`@click.group("creds")` → `set`). +- [ ] Registrarli in `__main__.py` (`cli.add_command(...)`). +- [ ] `tests/python/test_commands_bench.py` ecc.: verificare `--help` e + registrazione comandi (smoke test CLI). +- **Done**: `python -m ci_orchestrator bench --help` ecc. funzionano; + pytest verde. + +## C2 — `bench run` (burn-in concorrenza) + +**Sostituisce**: `Test-CapacityBurnIn.ps1`, `Start-BurnInTest.ps1`, +`Start-BurnInTest-Linux.ps1`. + +- [ ] Implementare il lancio di N job concorrenti **in-process** + (`concurrent.futures` su `commands/job.py`) o come N subprocess del + CLI Python — **niente `Start-Job`/`pwsh`/`$IsWindows`** (chiude il bug + shim f22b142). +- [ ] Per ogni round, asserire: tutti i job exit 0; nessuna clone-dir + orfana in `CI_BUILD_VMS`; nessun `vm-start.lock` stale (>5 min). +- [ ] Report JSON in `CI_ARTIFACTS/bench/` + tabella riepilogo a video. +- [ ] Default lab: `--concurrency 4 --rounds 10`, build-command no-op + (`ping`/`sleep ~30s`) come negli wrapper. +- [ ] Test: round con job mockati (PASS/FAIL), rilevamento orfani e lock. +- **Done**: output equivalente a `Test-CapacityBurnIn.ps1` su Win e Linux. + +## C3 — `bench measure` (timing fasi) + retire moduli PS + +**Sostituisce**: `Measure-CIBenchmark.ps1` (+ `_Common.psm1`, +`_Transport.psm1`, `tests/_Common.Tests.ps1`). + +- [ ] Misurare fasi clone / start / IP acquire / transport ready / destroy + via `load_backend` + transport, su 1+ VM effimere. +- [ ] Append a `benchmark.jsonl` (stesso formato, per continuità trend). +- [ ] Test con backend/transport mockati che simulano i tempi di fase. +- [ ] **Rimuovere** `scripts/_Common.psm1`, `scripts/_Transport.psm1`, + `tests/_Common.Tests.ps1` (nessun altro consumatore — verificato). +- [ ] Aggiornare `lint.yml` se elencava esplicitamente quei path. +- **Done**: `bench measure` riproduce le metriche; moduli PS rimossi; + pytest verde. + +## C4 — `smoke run` + +**Sostituisce**: `Test-Smoke.ps1`, `Test-Ns7zipBuild-Linux.ps1`, +`Test-NsinnounpBuild-Linux.ps1` (e i corrispettivi Win restano per fallback). + +- [ ] `smoke run` esegue un job no-op (marker file) end-to-end e verifica: + exit 0, dir artifact creata, evento `job-success` nel JSONL. +- [ ] `--guest-os windows|linux`; preset build-command per le build reali + ns7zip / nsinnounp (opzionali, dietro flag o `--build-command`). +- [ ] Test: pipeline mockata, asserzioni su artifact dir + evento. +- **Done**: `smoke run --guest-os linux` e `--guest-os windows` passano. + +## C5 — `validate host` + +**Nuovo** equivalente Linux (non porting 1:1 di `Validate-HostState.ps1`). + +- [ ] Verificare: `vmrun` presente/eseguibile; snapshot template presenti + (`BaseClean` / `BaseClean-Linux`); keyring file-based leggibile + (`keyring.get_credential`); permessi `/var/lib/ci/...`; unit + `act-runner` / `ci-*` attive (`systemctl is-active`). +- [ ] Exit 0 = tutto ok; non-zero + report per check fallito. +- [ ] Test: ogni check mockato (pass/fail) → exit code corretto. +- **Done**: `validate host` esce 0 su host configurato. + +## C6 — `creds set` + `validate guest` + +**Sostituisce**: `Set-CIGuestCredential.ps1` (host Linux) e +`Test-CIGuestWinRM.ps1` (parte diagnostica host-side). + +- [ ] `creds set --target ... --user ... --password-stdin`: scrive nel + backend `keyring` letto da `ci_orchestrator` + (`keyrings.alt.file.PlaintextKeyring`). Password mai su command line. +- [ ] `validate guest --host IP [--winrm|--ssh]`: tenta connessione + + comando banale, riporta causa esplicita su fallimento (la sonda + `is_ready` ingoia gli errori — vedi gotcha). +- [ ] Test: scrittura/lettura credenziale (keyring mockato); diag con + transport mockato (connect ok / auth fail / connect fail). +- **Done**: credenziali settabili e transport diagnosticabile senza `pwsh`. + +## C7 — Lint CI (decisione `pwsh`) + +**Risolve dipendenza #5**. Applicare l'opzione scelta nel §4 dell'analisi. + +- [ ] Decidere: (1) spostare job `pssa` su runner Windows / on-demand + [consigliata]; (2) isolare `pwsh` nel solo runner di lint; (3) + rimuovere PSSA da CI. +- [ ] Aggiornare `.gitea/workflows/lint.yml` di conseguenza. +- [ ] Verificare che nessun job che gira su `runs-on: linux-build` invochi + più `shell: pwsh`. +- **Done**: pipeline lint verde senza `pwsh` su runner Linux. + +## C8 — `setup-host-linux.sh` + +- [ ] Invertire lo step 6: non installare `pwsh` di default; aggiungere + flag `--with-pwsh` per opt-in (mantenere `--skip-pwsh` come alias + no-op deprecato o rimuoverlo). +- [ ] Aggiornare l'help e gli esempi d'uso nello script. +- **Done**: `setup-host-linux.sh` su host pulito non installa `pwsh`. + +## C9 — Docs, cutover, burn-in, disinstallazione + +Parte **docs** completata in codice (2026-06-07). Parte **operativa** (cutover ++ burn-in di validazione + disinstallazione) resta da eseguire sull'host reale. + +- [x] `docs/RUNBOOK.md`: aggiunta tabella «Phase C — Python equivalents» + nota + che i record §6–§13 restano provenance storica; op-note §13 punto 3 aggiornato. +- [x] `CLAUDE.md`: nota "host Linux non usa più `pwsh`" (constraints PS 5.1 + restano per script guest Windows + runner Windows). +- [x] `TODO.md`: chiusa voce bug shim `$IsWindows` (§3.7, risolto-per-rimpiazzo). +- [ ] **Cutover** (operativo): usare esclusivamente i comandi Python per ≥1 + settimana (burn-in `bench run` periodico + smoke), `pwsh` ancora installato. +- [ ] **Burn-in verde ≥1 settimana** → `sudo apt-get remove powershell` sull'host + Linux. +- [ ] Riconfermare i criteri "C done" del §7 dell'analisi sull'host reale. +- **Done (codice/docs)**: porting C1–C8 mergeabile, suite verde ≥90%. +- **Done (operativo, pendente)**: `pwsh` assente dall'host dopo burn-in verde. + +--- + +## Ordine consigliato e parallelizzabilità + +`C1` prima di tutto. `C2`–`C6` indipendenti tra loro (parallelizzabili in +PR separate, ognuna sopra C1). `C3` deve precedere la rimozione dei moduli +`.psm1`. `C7` e `C8` indipendenti dai porting. `C9` ultimo, dopo che +C2–C8 sono merge-ati e il burn-in di validazione è verde. + +## Rischi + +- **WinRM 4× concorrenza instabile** (TODO noto): può far fallire `bench run` + su Windows con concorrenza alta — non è regressione del porting. Tarare + `--concurrency` e annotare. +- **Equivalenza output**: validare A/B (`.ps1` vs Python) finché `pwsh` è + ancora installato (prerequisito §6) prima della disinstallazione in C9. diff --git a/plans/idea-3-powershell-removal.md b/plans/idea-3-powershell-removal.md index d142b8d..bbc8e5b 100644 --- a/plans/idea-3-powershell-removal.md +++ b/plans/idea-3-powershell-removal.md @@ -1,103 +1,198 @@ # Fase C — Eliminazione dipendenza PowerShell dall'host Linux -> **Stato**: **pianificata**, da avviare dopo che Fase B è stabile in -> produzione (≥1 settimana di burn-in, Passo 8 completato). -> Vedi [overview](ideas-overview.md). +> **Stato**: **codice completato** (C1–C8 implementati + docs C9, 2026-06-07); +> resta solo la parte **operativa** di C9 (cutover ≥1 settimana + `apt-get +> remove powershell` sull'host reale). +> Vedi [overview](ideas-overview.md) e il +> [piano di applicazione](idea-3-implementation-plan.md) (checklist operativa). +> +> **Analisi rifatta il 2026-06-07** sullo stato reale del repo (la versione +> precedente proponeva gruppi CLI inesistenti e ometteva metà della +> superficie `pwsh`). --- ## 1. Obiettivo Rendere l'host Linux completamente indipendente da PowerShell Core (`pwsh`). -Dopo la Fase B l'orchestrazione ordinaria (job, timer, monitoring, backup) -è già Python puro. Rimangono script PS usati per test manuali, burn-in e -validazione host che richiedono `pwsh` installato. +Dopo la Fase B l'orchestrazione ordinaria (job, timer, monitoring, backup, +retention, deploy template) è già Python puro. Restano script `.ps1` +eseguiti **manualmente** sull'host Linux con `pwsh` per: burn-in di +capacità, benchmark di timing, smoke/E2E test, validazione stato host e +setup credenziali. Inoltre il job di lint CI usa `shell: pwsh`. **C done**: nessuno script in `scripts/*.ps1` viene più eseguito -direttamente sull'host Linux; `pwsh` può essere disinstallato senza -impatti operativi. I `.ps1` rimangono nel repo come riferimento storico -e per il runner Windows (fallback dual-boot). +direttamente sull'host Linux, e nessun job CI che gira su un runner Linux +richiede `pwsh`; `pwsh` può essere disinstallato dall'host Linux senza +impatti operativi. I `.ps1` restano nel repo come riferimento storico e +per l'eventuale runner Windows (fallback dual-boot Passo 9). --- -## 2. Script da portare +## 2. Superficie `pwsh` reale sull'host Linux -| Script PS | Uso attuale (Linux host) | Target Python | -| ---------------------------- | -------------------------------- | -------------------------------------------------------- | -| `Test-CapacityBurnIn.ps1` | burn-in B7: 4 job × 10 round | `python -m ci_orchestrator bench run` | -| `Start-BurnInTest.ps1` | burn-in Windows single-shot | `python -m ci_orchestrator bench run --guest-os windows` | -| `Start-BurnInTest-Linux.ps1` | burn-in Linux single-shot | `python -m ci_orchestrator bench run --guest-os linux` | -| `Validate-HostState.ps1` | verifica stato host pre-cutover | `python -m ci_orchestrator validate host` | -| `Measure-CIBenchmark.ps1` | raccolta metriche timing | integrato in `bench run --collect-metrics` | -| `Test-Smoke.ps1` | smoke end-to-end manuale | `python -m ci_orchestrator smoke run` | -| `Test-CapacityBurnIn.ps1` | già referenziato in B7 checklist | vedi sopra | +Lo stato CLI attuale (`src/ci_orchestrator/__main__.py`) espone i gruppi: +`wait-ready`, `vm`, `build`, `artifacts`, `monitor`, `report`, +`retention`, `template`, `job`. **Non esistono** gruppi `bench`, +`smoke`, `validate`, `creds` — vanno creati. -Script **non da portare** (girano solo su Windows o sono già shim): -- `Register-CIScheduledTasks.ps1` — Windows Scheduler, non serve su Linux -- `Invoke-RetentionPolicy.ps1` / `Backup-CITemplate.ps1` — già portati in B5 -- Tutti gli script shim in `scripts/` — 3 righe che chiamano Python, restano -- Script guest (`Install-CIToolchain-*.ps1`, `Prepare-*.ps1`, ecc.) — girano - dentro le VM Windows, non sull'host +### 2.1 Cosa blocca davvero la disinstallazione di `pwsh` + +| # | Dipendenza | Dove | Tipo | +| - | ---------- | ---- | ---- | +| 1 | Burn-in di capacità (4 job × N round) | `Test-CapacityBurnIn.ps1` + wrapper `Start-BurnInTest-Linux.ps1` (lanciato via `sudo -u ci-runner -H pwsh …`, vedi PhaseB-user-checklist §B7) | host Linux, manuale | +| 2 | Benchmark timing fasi (clone/start/IP/transport/destroy) | `Measure-CIBenchmark.ps1` — **unico consumatore** di `scripts/_Common.psm1` e `scripts/_Transport.psm1` | host Linux, manuale | +| 3 | Smoke end-to-end | `Test-Smoke.ps1 -GuestOS Linux` | host Linux, manuale | +| 4 | Build E2E reali (validazione pipeline) | `Test-Ns7zipBuild-Linux.ps1`, `Test-NsinnounpBuild-Linux.ps1` | host Linux, manuale | +| 5 | Lint CI (PSScriptAnalyzer) | `.gitea/workflows/lint.yml` → `shell: pwsh`, `runs-on: linux-build` | runner Linux, automatico | +| 6 | Installazione `pwsh` di default | `setup-host-linux.sh` step 6/6 (opt-out `--skip-pwsh`) | provisioning host | +| 7 | Pester (test dei moduli PS) | `tests/_Common.Tests.ps1` | host/CI, manuale | +| 8 | Setup credenziali guest | `Set-CIGuestCredential.ps1` (equivalente Linux mancante in CLI) | host Linux, manuale | + +### 2.2 Script Windows-only (NON bloccano `pwsh` su Linux) + +Hardcodano `F:\`, richiedono `#Requires -RunAsAdministrator` o usano il +Windows Credential Manager: **non girano sull'host Linux**, quindi non +vanno portati per l'obiettivo di Fase C. Restano per il runner Windows. + +- `Validate-HostState.ps1` — logica Credential Manager/ACL NTFS, path `F:\`. + Serve però un **equivalente Linux** (`validate host`), non un porting 1:1. +- `Set-CIGuestCredential.ps1` — scrive nel vault SYSTEM via scheduled task. + Su Linux le credenziali stanno in `keyrings.alt.file.PlaintextKeyring` + (B3, `secret-tool` scartato). Serve un `creds set` Linux-nativo. +- `Test-CIGuestWinRM.ps1` — diagnostica WinRM, `RunAsAdministrator`. +- `Start-BurnInTest.ps1`, `Test-NsinnounpBuild.ps1`, `Test-E2E-Section3.3.ps1` + — wrapper Windows (path `F:\`, template WinBuild). + +### 2.3 Già fuori scope (confermato) + +- `Register-CIScheduledTasks.ps1` — Windows Scheduler. +- `Invoke-RetentionPolicy.ps1` / `Backup-CITemplate.ps1` — portati in B5 + (`retention run` / `template backup`). +- Shim in `scripts/` (3 righe → Python) — restano. +- Script guest (`Install-CIToolchain-*`, `Prepare-*`, `Deploy-*`, + `template/guest-setup/windows/*`) — girano dentro le VM, non sull'host. --- -## 3. Architettura prevista - -Nuovi sub-comandi CLI sotto due nuovi gruppi: +## 3. Architettura prevista (nuovi gruppi CLI) ``` -python -m ci_orchestrator bench run --guest-os windows|linux - --template - --concurrency N - --rounds N - --collect-metrics +python -m ci_orchestrator bench run --guest-os windows|linux + --concurrency N # default 4 + --rounds N # default 10 + --build-command "..." + --json-out PATH +python -m ci_orchestrator bench measure --guest-os windows|linux + --iterations N + --json-out PATH # append a benchmark.jsonl -python -m ci_orchestrator validate host # verifica stato host -python -m ci_orchestrator smoke run # end-to-end smoke test +python -m ci_orchestrator smoke run --guest-os windows|linux + [--build-command "..."] # default: marker no-op + +python -m ci_orchestrator validate host # vmrun, snapshot template, keyring, + # permessi dir, act-runner systemd attivo +python -m ci_orchestrator validate guest --host IP [--winrm|--ssh] # diag transport + +python -m ci_orchestrator creds set --target BuildVMGuest --user ... [--password-stdin] ``` -Il gruppo `bench` sostituisce i tre script `*BurnIn*.ps1` con logica -unificata: avvia N clone in parallelo, misura tempo per job, confronta -con baseline, emette un report JSON in `CI_ARTIFACTS/bench/`. +- **`bench run`** unifica `Test-CapacityBurnIn.ps1` + `Start-BurnInTest*.ps1`: + lancia N job concorrenti (riusando `commands/job.py` in-process o come + subprocess del CLI Python — niente `Start-Job`/`pwsh`), per R round, + e asserisce dopo ogni round: tutti exit 0, nessuna clone-dir orfana in + `CI_BUILD_VMS`, nessun `vm-start.lock` stale. Emette report JSON. +- **`bench measure`** porta `Measure-CIBenchmark.ps1` (fasi + clone/start/IP/transport/destroy) → consente di **retirare** + `_Common.psm1` e `_Transport.psm1` (nessun altro consumatore). +- **`smoke run`** porta `Test-Smoke.ps1` + i `Test-*Build-Linux.ps1` + (build E2E reali come preset opzionali del build-command). +- **`validate host`** è un nuovo equivalente Linux (non un porting di + `Validate-HostState.ps1`): verifica `vmrun`, presenza snapshot template, + leggibilità keyring file-based, permessi `/var/lib/ci/...`, unit + `act-runner`/`ci-*` attive. +- **`creds set`** sostituisce `Set-CIGuestCredential.ps1` su Linux usando + direttamente il backend `keyring` letto da `ci_orchestrator`. -Il gruppo `validate` porta `Validate-HostState.ps1`: verifica vmrun, -template snapshot, credenziali, permessi directory, act-runner attivo. +**Vincolo di design**: i nuovi comandi devono importare +`backends.load_backend` e i transport esistenti, **mai** +`WorkstationVmrunBackend` diretto (rotta ESXi Fase D aperta). --- -## 4. Aggiornamenti collaterali +## 4. Nodo aperto — lint PSScriptAnalyzer (dipendenza #5) +Gli script `.ps1` restano nel repo (guest + fallback Windows), quindi il +lint PSSA ha ancora valore. Ma `lint.yml` lo esegue con `shell: pwsh` su +`runs-on: linux-build`, il che reintroduce `pwsh` sul runner Linux. +**Decisione applicata (C7, 2026-06-07): opzione 1.** Il job `pssa` gira ora su +`runs-on: windows-build`, `shell: powershell` (PS 5.1), con guardia +`if: github.event_name == 'workflow_dispatch'` così non si accoda su push/PR se +il runner Windows non è registrato. Il job `python` resta su `linux-build`. +Nessun job su runner Linux usa più `pwsh`. + +Tre opzioni considerate: + +1. **(Consigliata)** Spostare il job `pssa` su un runner Windows + (`runs-on: windows-build`, `shell: powershell` PS 5.1) — coerente col + fatto che i `.ps1` target sono Windows-native. Richiede un runner + Windows registrato (dual-boot Passo 9) o eseguirlo solo on-demand. +2. Mantenere `pwsh` **solo nel container/runner di lint**, non sull'host + orchestratore: l'obiettivo "host Linux senza pwsh" resta soddisfatto se + il job gira in un runner isolato. Pragmatica ma ambigua sul "C done". +3. Rimuovere il job PSSA da CI e affidare il lint PS a un hook locale + manuale. Riduce copertura. + +--- + +## 5. Aggiornamenti collaterali + +- `setup-host-linux.sh`: invertire il default dello step 6 (no install + `pwsh`; flag `--with-pwsh` per opt-in) o rimuovere lo step. +- `.gitea/workflows/lint.yml`: applicare la decisione del §4. +- `tests/_Common.Tests.ps1`: rimuovere insieme ai moduli `.psm1` quando + `bench measure` li retira (Passo C3); la copertura passa a pytest. +- `CLAUDE.md`: la sezione "PowerShell 5.1 Constraints" resta valida per gli + script guest Windows; aggiungere nota che l'host Linux non usa più `pwsh`. +- `docs/RUNBOOK.md`: aggiornare procedure burn-in / smoke / validazione ai + nuovi comandi Python. - `plans/archived/2026-06-07/PhaseB-user-checklist.md` §B7 (archiviato): - sostituire il comando `pwsh Test-CapacityBurnIn.ps1` con - `python -m ci_orchestrator bench run`. -- `deploy/systemd/README.md`: nessun riferimento a `pwsh`. -- `CLAUDE.md`: rimuovere nota "PowerShell 5.1 Constraints" se non più - rilevante per l'host (rimane valida per gli script guest Windows). -- `docs/RUNBOOK.md`: aggiornare procedure di burn-in e smoke test. + nota che il comando `pwsh Test-CapacityBurnIn.ps1` è superato da + `bench run`. +- `TODO.md`: chiudere la voce shim `$IsWindows` bug (f22b142) — sparisce + con la rimozione di `Start-Job`/`pwsh` nel burn-in. --- -## 5. Prerequisiti +## 6. Prerequisiti - Fase B completata e stabile (≥1 settimana, Passo 8 ✅). +- `pwsh` ancora installato durante il porting (per confronto output A/B). - Nessuna dipendenza da `pwsh` nei timer systemd (già soddisfatta da B5). --- -## 6. Criteri di "C done" +## 7. Criteri di "C done" -- `python -m ci_orchestrator bench run --concurrency 4 --rounds 10` esegue - burn-in completo (Win + Linux) con output equivalente a `Test-CapacityBurnIn.ps1`. -- `python -m ci_orchestrator validate host` esce con 0 su host configurato - correttamente. -- `pwsh --version` può essere assente sull'host Linux senza che nessun - test, timer o script CI fallisca. +- `bench run --concurrency 4 --rounds 10` (Win + Linux) produce esito + equivalente a `Test-CapacityBurnIn.ps1` (stesse asserzioni di cleanup). +- `bench measure` riproduce le fasi di `Measure-CIBenchmark.ps1`; i moduli + `_Common.psm1`/`_Transport.psm1` e `tests/_Common.Tests.ps1` rimossi. +- `smoke run --guest-os linux` e `--guest-os windows` passano. +- `validate host` esce 0 su host configurato correttamente. +- `creds set` sostituisce `Set-CIGuestCredential.ps1` sull'host Linux. +- Decisione §4 applicata: nessun job CI su runner Linux richiede `pwsh`. +- `setup-host-linux.sh` non installa più `pwsh` di default. +- `pwsh --version` può essere assente sull'host Linux senza che alcun + test, timer, job CI o procedura di RUNBOOK fallisca. - Suite pytest ≥90% coverage, verde. --- -## 7. Rollback +## 8. Rollback -I `.ps1` originali rimangono nel repo e funzionano su host Windows o su -Linux con `pwsh` installato. Il rollback è semplicemente reinstallare -`pwsh` e usare gli script esistenti. +I `.ps1` originali restano nel repo e funzionano su host Windows o su +Linux con `pwsh` reinstallato (`setup-host-linux.sh --with-pwsh`). Il +rollback è reinstallare `pwsh` e usare gli script esistenti. Nessuna +distruzione di dati: il porting aggiunge comandi, non rimuove i `.ps1` +finché il burn-in di validazione non è verde. diff --git a/scripts/Measure-CIBenchmark.ps1 b/scripts/Measure-CIBenchmark.ps1 deleted file mode 100644 index 50e95be..0000000 --- a/scripts/Measure-CIBenchmark.ps1 +++ /dev/null @@ -1,335 +0,0 @@ -#Requires -Version 5.1 -<# -.SYNOPSIS - Measures CI infrastructure phase timings: clone, start, IP acquire, transport ready, destroy. - -.DESCRIPTION - Creates one or more ephemeral VMs from the template, times each phase, destroys - them, then prints a summary table. Results are appended to benchmark.jsonl - for long-term trend tracking. - - Phases measured: - clone — vmrun clone (linked clone from snapshot) - start — vmrun start (until command returns) - ip — vmrun getGuestIPAddress (polling until IP appears) - ready — TCP port reachable: 5986 (WinRM, Windows) or 22 (SSH, Linux) - destroy — stop + deleteVM + dir removal - - No actual build is run. - -.PARAMETER TemplatePath - Full path to the template VM's .vmx file. - -.PARAMETER SnapshotName - Snapshot name to clone from. Default: BaseClean - -.PARAMETER CloneBaseDir - Directory for ephemeral clone folders. - -.PARAMETER VmrunPath - Path to vmrun executable. - -.PARAMETER GuestOS - 'Windows', 'Linux', or 'Auto' (default). Auto reads guestOS from the VMX: - ubuntu-* → Linux (probe SSH/22); anything else → Windows (probe WinRM/5986). - -.PARAMETER Iterations - Number of clone→ready→destroy cycles to run. Default: 1 - -.PARAMETER TimeoutSeconds - Maximum seconds to wait for IP and transport per iteration. Default: 300 - -.PARAMETER OutputDir - Directory where benchmark.jsonl is appended. - -.EXAMPLE - # Single baseline run (Linux host, Windows template) - .\Measure-CIBenchmark.ps1 - - # 4 iterations, Linux guest template - .\Measure-CIBenchmark.ps1 -TemplatePath /var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx ` - -SnapshotName BaseClean-Linux -Iterations 4 - - # Windows host - .\Measure-CIBenchmark.ps1 -TemplatePath 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' ` - -Iterations 3 -#> -[CmdletBinding()] -param( - [string] $TemplatePath = $(if ($null -ne $IsWindows -and $IsWindows -eq $false) { - '/var/lib/ci/templates/WinBuild2025/WinBuild2025.vmx' - } else { - 'F:\CI\Templates\WinBuild2025\WinBuild2025.vmx' - }), - - [string] $SnapshotName = 'BaseClean', - - [string] $CloneBaseDir = $(if ($null -ne $IsWindows -and $IsWindows -eq $false) { - '/var/lib/ci/build-vms/' - } else { - 'F:\CI\BuildVMs' - }), - - [string] $VmrunPath = $(if ($null -ne $IsWindows -and $IsWindows -eq $false) { - '/usr/bin/vmrun' - } else { - 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' - }), - - [ValidateSet('Auto', 'Windows', 'Linux')] - [string] $GuestOS = 'Auto', - - [ValidateRange(1, 10)] - [int] $Iterations = 1, - - [ValidateRange(60, 600)] - [int] $TimeoutSeconds = 300, - - [string] $OutputDir = $(if ($null -ne $IsWindows -and $IsWindows -eq $false) { - '/var/lib/ci/logs/' - } else { - 'F:\CI\Logs' - }), - - # Optional static IP injection (mirrors ip_pool in job.py). - # If supplied, guestinfo.ip-assignment/netmask/gateway are written into the - # cloned VMX before start so ci-static-ip.ps1 applies the IP at boot. - [string] $StaticIP = '', - [string] $Netmask = '255.255.255.0', - [string] $Gateway = '' -) - -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' -$ProgressPreference = 'SilentlyContinue' - -Import-Module (Join-Path $PSScriptRoot '_Common.psm1') -Force -Resolve-VmrunPath -VmrunPath $VmrunPath | Out-Null - -# Cross-platform TCP port probe (replaces Test-NetConnection which is Windows-only). -function Test-TcpPort { - param([string]$ComputerName, [int]$Port, [int]$TimeoutMs = 3000) - try { - $tcp = [System.Net.Sockets.TcpClient]::new() - $ar = $tcp.BeginConnect($ComputerName, $Port, $null, $null) - $ok = $ar.AsyncWaitHandle.WaitOne($TimeoutMs, $false) - $connected = $ok -and $tcp.Connected - $tcp.Close() - return $connected - } catch { - return $false - } -} - -# Resolve effective GuestOS from VMX when Auto. -$resolvedGuestOS = $GuestOS -if ($resolvedGuestOS -eq 'Auto') { - $vmxRaw = Get-Content -LiteralPath $TemplatePath -Raw -ErrorAction SilentlyContinue - if ($vmxRaw -match 'guestOS\s*=\s*"([^"]+)"' -and $Matches[1] -like 'ubuntu*') { - $resolvedGuestOS = 'Linux' - } else { - $resolvedGuestOS = 'Windows' - } - Write-Host "[bench] GuestOS auto-detected: $resolvedGuestOS" -} -$transportPort = if ($resolvedGuestOS -eq 'Linux') { 22 } else { 5986 } -$transportLabel = if ($resolvedGuestOS -eq 'Linux') { 'SSH/22' } else { 'WinRM/5986' } - -if (-not (Test-Path $TemplatePath -PathType Leaf)) { - throw "Template VMX not found: $TemplatePath" -} -if (-not (Test-Path $CloneBaseDir)) { - New-Item -ItemType Directory -Path $CloneBaseDir -Force | Out-Null -} -if (-not (Test-Path $OutputDir)) { - New-Item -ItemType Directory -Path $OutputDir -Force | Out-Null -} - -function Set-GuestInfoIP { - [CmdletBinding(SupportsShouldProcess)] - param([string]$VmxPath, [string]$IP, [string]$Mask, [string]$Gw) - $keys = @('guestinfo.ip-assignment', 'guestinfo.netmask', 'guestinfo.gateway') - $lines = (Get-Content -LiteralPath $VmxPath -Encoding UTF8) | - Where-Object { ($_ -split '=')[0].Trim().ToLower() -notin $keys } - $lines += "guestinfo.ip-assignment = `"$IP`"" - if ($Mask) { $lines += "guestinfo.netmask = `"$Mask`"" } - if ($Gw) { $lines += "guestinfo.gateway = `"$Gw`"" } - Set-Content -LiteralPath $VmxPath -Value $lines -Encoding UTF8 -} - -$benchmarkLog = Join-Path $OutputDir 'benchmark.jsonl' -$results = [System.Collections.Generic.List[pscustomobject]]::new() - -$useStaticIP = $StaticIP -match '^\d{1,3}(\.\d{1,3}){3}$' -if ($useStaticIP) { - Write-Host "[bench] Static IP mode: $StaticIP / $Netmask gw=$Gateway" -} else { - Write-Host "[bench] DHCP mode (no -StaticIP supplied)" -} - -Write-Host "" -Write-Host "=== CI Benchmark template: $(Split-Path $TemplatePath -Leaf) snapshot: $SnapshotName iterations: $Iterations ===" -Write-Host "" - -for ($i = 1; $i -le $Iterations; $i++) { - Write-Host "--- Iteration $i / $Iterations ---" - $runId = "bench-$(Get-Date -Format 'yyyyMMdd-HHmmss')-$i" - $cloneVmx = $null - $cloneDir = $null - - $t = [ordered]@{ - clone = $null - start = $null - ip = $null - ready = $null - destroy = $null - deltaKB = $null - vmIP = $null - error = $null - } - - try { - # ── Phase: clone ────────────────────────────────────────────────────── - $cloneName = "Clone_${runId}" - $cloneDir = Join-Path $CloneBaseDir $cloneName - $cloneVmx = Join-Path $cloneDir "$cloneName.vmx" - - Write-Host "[bench] Cloning..." - $sw = [System.Diagnostics.Stopwatch]::StartNew() - $null = Invoke-Vmrun -VmrunPath $VmrunPath -Operation clone ` - -Arguments @($TemplatePath, $cloneVmx, 'linked', '-snapshot', $SnapshotName) ` - -ThrowOnError - $t.clone = [math]::Round($sw.Elapsed.TotalSeconds, 2) - Write-Host "[bench] clone: $($t.clone)s" - - # Measure linked-clone disk footprint - try { - $cloneSizeBytes = (Get-ChildItem -Path $cloneDir -Recurse -File -ErrorAction Stop | - Measure-Object -Property Length -Sum).Sum - $t.deltaKB = [math]::Round($cloneSizeBytes / 1KB, 0) - Write-Host "[bench] clone size: $($t.deltaKB) KB" - } catch { - Write-Warning "[bench] Could not measure clone size: $_" - } - - # ── Static IP injection (before start) ─────────────────────────────── - if ($useStaticIP) { - Set-GuestInfoIP -VmxPath $cloneVmx -IP $StaticIP -Mask $Netmask -Gw $Gateway - Write-Host "[bench] guestinfo injected: $StaticIP" - } - - # ── Phase: start ────────────────────────────────────────────────────── - Write-Host "[bench] Starting VM..." - $sw.Restart() - Invoke-Vmrun -VmrunPath $VmrunPath -Operation start ` - -Arguments @($cloneVmx, 'nogui') -ThrowOnError | Out-Null - $t.start = [math]::Round($sw.Elapsed.TotalSeconds, 2) - Write-Host "[bench] start: $($t.start)s" - - # ── Phase: IP acquire (primary: guestinfo.ci-ip, fallback: getGuestIPAddress) ── - Write-Host "[bench] Waiting for guest IP..." - $sw.Restart() - $vmIP = Get-GuestIPAddress -VmrunPath $VmrunPath -VmxPath $cloneVmx ` - -TimeoutSeconds $TimeoutSeconds -GuestInfoOnly:$useStaticIP - if (-not $vmIP) { throw "Timed out waiting for guest IP after $TimeoutSeconds s" } - $t.ip = [math]::Round($sw.Elapsed.TotalSeconds, 2) - $t.vmIP = $vmIP - Write-Host "[bench] ip ($vmIP): $($t.ip)s" - - # ── Phase: transport ready (SSH/22 for Linux, WinRM/5986 for Windows) ── - Write-Host "[bench] Waiting for $transportLabel..." - $sw.Restart() - $deadline = (Get-Date).AddSeconds($TimeoutSeconds) - $transportUp = $false - while ((Get-Date) -lt $deadline) { - $transportUp = Test-TcpPort -ComputerName $vmIP -Port $transportPort - if ($transportUp) { break } - Start-Sleep -Seconds 3 - } - if (-not $transportUp) { - throw "Timed out waiting for $transportLabel on $vmIP after $TimeoutSeconds s" - } - $t.ready = [math]::Round($sw.Elapsed.TotalSeconds, 2) - Write-Host "[bench] ready ($transportLabel): $($t.ready)s" - } - catch { - $t.error = "$_" - Write-Warning "[bench] Iteration $i failed: $_" - } - finally { - # ── Phase: destroy ──────────────────────────────────────────────────── - if ($cloneVmx -and (Test-Path $cloneVmx -PathType Leaf)) { - Write-Host "[bench] Destroying clone..." - $sw = [System.Diagnostics.Stopwatch]::StartNew() - & (Join-Path $PSScriptRoot 'Remove-BuildVM.ps1') ` - -VMPath $cloneVmx -VmrunPath $VmrunPath -GracefulTimeoutSeconds 10 ` - -ErrorAction SilentlyContinue - $t.destroy = [math]::Round($sw.Elapsed.TotalSeconds, 2) - Write-Host "[bench] destroy: $($t.destroy)s" - } elseif ($cloneDir -and (Test-Path $cloneDir)) { - Remove-Item $cloneDir -Recurse -Force -ErrorAction SilentlyContinue - } - } - - $totalBootSec = if ($null -ne $t.ip -and $null -ne $t.ready) { - [math]::Round($t.clone + $t.start + $t.ip + $t.ready, 2) - } else { $null } - - $result = [pscustomobject]@{ - ts = (Get-Date -Format 'o') - runId = $runId - iteration = $i - template = Split-Path $TemplatePath -Leaf - snapshot = $SnapshotName - guestOS = $resolvedGuestOS - staticIP = if ($useStaticIP) { $StaticIP } else { $null } - cloneSec = $t.clone - startSec = $t.start - ipSec = $t.ip - readySec = $t.ready - destroySec = $t.destroy - totalBootSec = $totalBootSec - deltaKB = $t.deltaKB - vmIP = $t.vmIP - error = $t.error - } - $results.Add($result) - - # Append to JSONL log - try { - $result | ConvertTo-Json -Compress -Depth 3 | - Add-Content -Path $benchmarkLog -Encoding UTF8 -ErrorAction Stop - } catch { - Write-Warning "[bench] Could not write benchmark.jsonl: $_" - } - - Write-Host "" -} - -# ── Summary table ───────────────────────────────────────────────────────────── -Write-Host "=== Results ===" -$results | Format-Table @( - @{ L = 'Iter'; E = { $_.iteration } } - @{ L = 'Clone(s)'; E = { $_.cloneSec } } - @{ L = 'Start(s)'; E = { $_.startSec } } - @{ L = 'IP(s)'; E = { $_.ipSec } } - @{ L = 'Ready(s)'; E = { $_.readySec } } - @{ L = 'Destroy(s)'; E = { $_.destroySec } } - @{ L = 'Boot total'; E = { $_.totalBootSec } } - @{ L = 'Delta(KB)'; E = { $_.deltaKB } } - @{ L = 'IP'; E = { $_.vmIP } } - @{ L = 'Error'; E = { $_.error } } -) -AutoSize - -if ($Iterations -gt 1) { - $ok = @($results | Where-Object { -not $_.error }) - if ($ok.Count -gt 0) { - $avgBoot = [math]::Round(($ok | ForEach-Object { $_.totalBootSec } | - Measure-Object -Average).Average, 2) - Write-Host "Average boot-to-ready ($($ok.Count) successful runs): $avgBoot s" - } -} - -Write-Host "" -Write-Host "Results appended to: $benchmarkLog" - diff --git a/scripts/_Common.psm1 b/scripts/_Common.psm1 deleted file mode 100644 index 20d9bdf..0000000 --- a/scripts/_Common.psm1 +++ /dev/null @@ -1,435 +0,0 @@ -#Requires -Version 5.1 -<# -.SYNOPSIS - Shared helper functions for the Local CI/CD System scripts. - -.DESCRIPTION - Import this module at the top of any CI script that needs WinRM session - options or vmrun wrappers: - - Import-Module (Join-Path $PSScriptRoot '_Common.psm1') -Force - - Exported functions: - New-CISessionOption — PSSessionOption for WinRM over self-signed TLS - Resolve-VmrunPath — Validates vmrun.exe path, throws if missing - Invoke-Vmrun — Uniform vmrun -T ws wrapper; returns ExitCode + Output -#> -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' - -function New-CISessionOption { -<# -.SYNOPSIS - Returns a PSSessionOption for WinRM HTTPS with self-signed certificate. -.DESCRIPTION - All CI build VMs use a self-signed TLS cert on WinRM port 5986. - This option set skips CA, CN, and revocation checks — appropriate for - an isolated lab network where PKI is not present. -.OUTPUTS - [System.Management.Automation.Remoting.PSSessionOption] -#> - [OutputType([System.Management.Automation.Remoting.PSSessionOption])] - [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseShouldProcessForStateChangingFunctions', '', - Justification = 'New-CISessionOption creates an in-memory PSSessionOption object; no system state is changed.')] - [CmdletBinding()] - param() - New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck -} - -function Resolve-VmrunPath { -<# -.SYNOPSIS - Validates that vmrun.exe exists at the specified path and returns it. -.DESCRIPTION - Throws a descriptive error if the file is not found, rather than letting - a later & call fail with a less useful message. -.PARAMETER VmrunPath - Full path to vmrun.exe. - Default: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe -.OUTPUTS - [string] The validated VmrunPath. -#> - [OutputType([string])] - [CmdletBinding()] - param( - [string] $VmrunPath = 'C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe' - ) - if (-not (Test-Path $VmrunPath -PathType Leaf)) { - throw "vmrun.exe not found at: $VmrunPath`nInstall VMware Workstation Pro and update -VmrunPath." - } - return $VmrunPath -} - -function Invoke-Vmrun { -<# -.SYNOPSIS - Runs a vmrun -T ws command and returns output and exit code. -.DESCRIPTION - Uniform wrapper so all vmrun calls share the same -T ws flag and output - capture pattern. Use -ThrowOnError for operations that must succeed - (clone, start); omit it for best-effort operations (stop, getState). -.PARAMETER VmrunPath - Full path to vmrun.exe. -.PARAMETER Operation - vmrun sub-command: clone, start, stop, deleteVM, list, getState, - listSnapshots, getGuestIPAddress, etc. -.PARAMETER Arguments - Additional positional arguments after the operation name. -.PARAMETER ThrowOnError - When set, throws if vmrun exits non-zero. -.OUTPUTS - [pscustomobject] with: - ExitCode [int] — vmrun exit code - Output [string[]] — combined stdout/stderr lines -#> - [OutputType([pscustomobject])] - [CmdletBinding()] - param( - [Parameter(Mandatory)] [string] $VmrunPath, - [Parameter(Mandatory)] [string] $Operation, - [string[]] $Arguments = @(), - [switch] $ThrowOnError - ) - $cmdArgs = @('-T', 'ws', $Operation) + $Arguments - $output = & $VmrunPath @cmdArgs 2>&1 - $exit = $LASTEXITCODE - - if ($ThrowOnError -and $exit -ne 0) { - throw "vmrun $Operation failed (exit $exit): $($output -join '; ')" - } - - return [pscustomobject]@{ - ExitCode = $exit - Output = $output - } -} - -function Invoke-VmrunBounded { -<# -.SYNOPSIS - Runs a vmrun -T ws command with a hard wall-clock timeout. -.DESCRIPTION - Uses [System.Diagnostics.Process] directly (not Start-Process -PassThru) because - Start-Process in PowerShell 5.1 does not reliably populate Process.ExitCode after - WaitForExit(ms) when output is redirected. ReadToEndAsync() is used for stdout and - stderr to prevent deadlock if output buffers fill up before the process exits. - - Use for lifecycle operations: start (180s), stop (60s), deleteVM (90s). - Keep Invoke-Vmrun for fast queries: list, readVariable, getGuestIPAddress, getState. -.PARAMETER VmrunPath - Full path to vmrun.exe. -.PARAMETER Operation - vmrun sub-command (start, stop, deleteVM, clone, …). -.PARAMETER Arguments - Additional positional arguments after the operation name. -.PARAMETER TimeoutSeconds - Hard timeout in seconds. Process is killed if not finished within this time. - Default: 120. -.PARAMETER ThrowOnTimeout - When set, throws if the timeout is reached before vmrun exits. -.PARAMETER ThrowOnError - When set, throws if vmrun exits with a non-zero code. -.OUTPUTS - [pscustomobject] with: - ExitCode [int] — vmrun exit code (-1 if killed due to timeout) - Output [string] — combined stdout/stderr text - TimedOut [bool] — true if the process was killed for exceeding TimeoutSeconds - ElapsedSeconds [double] — wall-clock seconds from start to exit/kill -#> - [OutputType([pscustomobject])] - [CmdletBinding()] - param( - [Parameter(Mandatory)] [string] $VmrunPath, - [Parameter(Mandatory)] [string] $Operation, - [string[]] $Arguments = @(), - [int] $TimeoutSeconds = 120, - [switch] $ThrowOnTimeout, - [switch] $ThrowOnError - ) - - $cmdArgs = @('-T', 'ws', $Operation) + $Arguments - - # Quote tokens that contain spaces so the argument string is correct when - # passed to ProcessStartInfo.Arguments as a single string. - $quotedArgs = $cmdArgs | ForEach-Object { - if ($_ -match '\s') { '"' + ($_ -replace '"', '""') + '"' } else { $_ } - } - $argStr = $quotedArgs -join ' ' - - $psi = New-Object System.Diagnostics.ProcessStartInfo - $psi.FileName = $VmrunPath - $psi.Arguments = $argStr - $psi.UseShellExecute = $false - $psi.CreateNoWindow = $true - $psi.RedirectStandardOutput = $true - $psi.RedirectStandardError = $true - - $proc = New-Object System.Diagnostics.Process - $proc.StartInfo = $psi - $sw = [System.Diagnostics.Stopwatch]::StartNew() - $timedOut = $false - - [void]$proc.Start() - - # Begin async reads before WaitForExit to avoid deadlock when output buffers fill. - $stdoutTask = $proc.StandardOutput.ReadToEndAsync() - $stderrTask = $proc.StandardError.ReadToEndAsync() - - $exited = $proc.WaitForExit($TimeoutSeconds * 1000) - if (-not $exited) { - $timedOut = $true - # Kill the entire process tree (taskkill /T) to release any inherited pipe - # handles held by child processes (e.g. cmd.exe spawning ping.exe in tests, - # or vmrun spawning helper subprocesses). Plain proc.Kill() only kills the - # direct process; child processes keep the write end of stdout open. - if ($null -ne $IsWindows -and $IsWindows -eq $false) { - # .NET 5+ Kill(entireProcessTree) — works on Linux/macOS. - $proc.Kill($true) - } else { - & taskkill /F /T /PID $proc.Id 2>&1 | Out-Null - } - } - $sw.Stop() - - $outputText = '' - if (-not $timedOut) { - # Process exited normally: drain stdout/stderr (up to 5s safety net). - [void][System.Threading.Tasks.Task]::WhenAll($stdoutTask, $stderrTask).Wait(5000) - $stdoutText = '' - $stderrText = '' - if ($stdoutTask.Status -eq [System.Threading.Tasks.TaskStatus]::RanToCompletion) { - $stdoutText = $stdoutTask.Result - } - if ($stderrTask.Status -eq [System.Threading.Tasks.TaskStatus]::RanToCompletion) { - $stderrText = $stderrTask.Result - } - $parts = @($stdoutText, $stderrText) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } - $outputText = ($parts | ForEach-Object { $_.TrimEnd() }) -join "`n" - } - # When timed-out: process was killed; output is irrelevant and potentially - # incomplete. Skip drain entirely — no point blocking for a killed process. - - $exitCode = if ($timedOut) { -1 } else { $proc.ExitCode } - - $result = [pscustomobject]@{ - ExitCode = $exitCode - Output = $outputText - TimedOut = $timedOut - ElapsedSeconds = [math]::Round($sw.Elapsed.TotalSeconds, 1) - } - - if ($timedOut -and $ThrowOnTimeout) { - throw "vmrun $Operation timed out after ${TimeoutSeconds}s" - } - if (-not $timedOut -and $exitCode -ne 0 -and $ThrowOnError) { - throw "vmrun $Operation failed (exit $exitCode): $outputText" - } - - return $result -} - -function Get-GuestIPAddress { -<# -.SYNOPSIS - Polls a running VMware guest VM until its IP address is available, then returns it. -.DESCRIPTION - Primary method: reads the 'ci-ip' guestVar written by ci-report-ip.service (Linux) - or the equivalent guest-side script via vmware-rpctool. This uses the VMware VMCI - channel and does not require TCP/IP connectivity from the host. - - Fallback: vmrun getGuestIPAddress (reliable on Windows guests; may be slow on Linux). - - Returns the detected IP as a plain string, or an empty string if the timeout expires. -.PARAMETER VmrunPath - Full path to vmrun.exe. -.PARAMETER VmxPath - Full path to the clone VM's .vmx file. -.PARAMETER TimeoutSeconds - Maximum seconds to poll before giving up. Default: 120. -.OUTPUTS - [string] Detected IPv4 address, or empty string on timeout. -#> - [OutputType([string])] - [CmdletBinding()] - param( - [Parameter(Mandatory)] [string] $VmrunPath, - [Parameter(Mandatory)] [string] $VmxPath, - [int] $TimeoutSeconds = 120, - # When set, skip vmrun getGuestIPAddress fallback and poll only - # guestinfo.ci-ip. Use when guestinfo.ip-assignment was injected into - # the VMX — avoids returning a transient DHCP address before - # ci-static-ip.ps1 has finished reconfiguring the NIC. - [switch] $GuestInfoOnly - ) - - $modeLabel = if ($GuestInfoOnly) { 'guestinfo only' } else { 'guestinfo + getGuestIPAddress' } - Write-Host "[Get-GuestIPAddress] Polling for VM IP (max ${TimeoutSeconds}s, mode: $modeLabel)..." - $deadline = (Get-Date).AddSeconds($TimeoutSeconds) - $attempt = 0 - - while ((Get-Date) -lt $deadline) { - $attempt++ - - # Primary: guestinfo.ci-ip written by ci-report-ip.service via vmware-rpctool. - # NOTE: vmrun readVariable guestVar reads WITHOUT the 'guestinfo.' prefix. - $savedEap = $ErrorActionPreference; $ErrorActionPreference = 'Continue' - $ipOut = (& $VmrunPath -T ws readVariable $VmxPath guestVar 'ci-ip' 2>&1).Trim() - $ipRc = $LASTEXITCODE - $ErrorActionPreference = $savedEap - - if ($attempt -le 3 -or ($attempt % 10 -eq 0)) { - Write-Host "[Get-GuestIPAddress] [attempt $attempt] guestVar ci-ip: rc=$ipRc out='$ipOut'" - } - if ($ipRc -eq 0 -and $ipOut -match '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$') { - Write-Host "[Get-GuestIPAddress] IP via guestinfo: $ipOut" - return $ipOut - } - - # Fallback: vmrun getGuestIPAddress — skipped when GuestInfoOnly is set - # to avoid returning a transient DHCP address during static-IP reconfiguration. - if (-not $GuestInfoOnly) { - $savedEap = $ErrorActionPreference; $ErrorActionPreference = 'Continue' - $ipOut2 = (& $VmrunPath -T ws getGuestIPAddress $VmxPath 2>&1).Trim() - $ipRc2 = $LASTEXITCODE - $ErrorActionPreference = $savedEap - - if ($ipRc2 -eq 0 -and $ipOut2 -match '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$') { - Write-Host "[Get-GuestIPAddress] IP via getGuestIPAddress: $ipOut2" - return $ipOut2 - } - } - - Start-Sleep -Seconds 2 - } - - return '' -} - -function Get-GuestDiagnostics { -<# -.SYNOPSIS - Best-effort collection of guest diagnostics after a build failure. - Never throws — always safe to call from a catch/finally block. - -.PARAMETER IPAddress - IP address of the guest VM. - -.PARAMETER GuestOS - 'Windows' or 'Linux'. - -.PARAMETER Credential - PSCredential for WinRM (Windows guests). Ignored for Linux. - -.PARAMETER SshKeyPath - Path to SSH private key (Linux guests). Ignored for Windows. - -.PARAMETER SshUser - SSH username (Linux guests). Default: ci_build. - -.PARAMETER DestinationDir - Host directory where diagnostics files are written. -#> - [CmdletBinding()] - param( - [Parameter(Mandatory)] [string] $IPAddress, - [Parameter(Mandatory)] [ValidateSet('Windows','Linux')] - [string] $GuestOS, - [pscredential] $Credential, - [string] $SshKeyPath, - [string] $SshUser = 'ci_build', - [Parameter(Mandatory)] [string] $DestinationDir - ) - - Write-Host "[Diagnostics] Collecting guest diagnostics from $IPAddress ($GuestOS) -> $DestinationDir" - try { - if (-not (Test-Path $DestinationDir)) { - New-Item -ItemType Directory -Path $DestinationDir -Force | Out-Null - } - } catch { - Write-Warning "[Diagnostics] Could not create destination dir: $_" - return - } - - if ($GuestOS -eq 'Windows') { - if (-not $Credential) { - Write-Warning "[Diagnostics] No Credential supplied for Windows guest — skipping." - return - } - try { - $sessionOption = New-CISessionOption - $session = New-PSSession -ComputerName $IPAddress ` - -Credential $Credential ` - -SessionOption $sessionOption ` - -ErrorAction Stop - try { - # Collect: event log errors (last 50), running processes, disk usage - $diagScript = { - $out = [System.Text.StringBuilder]::new() - [void]$out.AppendLine("=== System Event Log (last 50 errors) ===") - Get-EventLog -LogName System -EntryType Error -Newest 50 -ErrorAction SilentlyContinue | - ForEach-Object { [void]$out.AppendLine("$($_.TimeGenerated) $($_.Source): $($_.Message)") } - [void]$out.AppendLine("`n=== Running Processes ===") - Get-Process -ErrorAction SilentlyContinue | - Sort-Object CPU -Descending | - Select-Object -First 30 | - ForEach-Object { [void]$out.AppendLine("$($_.Name) PID=$($_.Id) CPU=$($_.CPU)") } - [void]$out.AppendLine("`n=== Disk Free Space ===") - Get-PSDrive -PSProvider FileSystem -ErrorAction SilentlyContinue | - ForEach-Object { [void]$out.AppendLine("$($_.Name): Free=$([math]::Round($_.Free/1GB,1))GB Used=$([math]::Round($_.Used/1GB,1))GB") } - $out.ToString() - } - $diagText = Invoke-Command -Session $session -ScriptBlock $diagScript -ErrorAction Stop - $diagText | Set-Content -Path (Join-Path $DestinationDir 'guest-diag.txt') -Encoding UTF8 - Write-Host "[Diagnostics] Windows diagnostics written." - - # Best-effort: copy CI build log - $buildLog = 'C:\CI\build.log' - try { - Copy-Item -FromSession $session -Path $buildLog ` - -Destination (Join-Path $DestinationDir 'build.log') ` - -ErrorAction Stop - Write-Host "[Diagnostics] Build log copied." - } catch { - Write-Host "[Diagnostics] Build log not found in guest (skipping)." - } - } finally { - Remove-PSSession $session -ErrorAction SilentlyContinue - } - } catch { - Write-Warning "[Diagnostics] Windows guest diagnostics failed (non-fatal): $_" - } - } - else { - # Linux - if (-not $SshKeyPath) { - Write-Warning "[Diagnostics] No SshKeyPath supplied for Linux guest — skipping." - return - } - $sshOpts = @('-i', $SshKeyPath, '-o', 'StrictHostKeyChecking=no', - '-o', 'UserKnownHostsFile=NUL', - '-o', 'ConnectTimeout=10', '-o', 'BatchMode=yes') - try { - # Collect: journalctl errors, ps, df - $diagCmd = "journalctl -p err -n 50 --no-pager 2>/dev/null; echo '=== ps ==='; ps aux --sort=-%cpu | head -30; echo '=== df ==='; df -h" - $savedEap = $ErrorActionPreference - $ErrorActionPreference = 'Continue' - $diagOutput = & ssh.exe @sshOpts "${SshUser}@${IPAddress}" $diagCmd 2>&1 - $ErrorActionPreference = $savedEap - $diagOutput | Set-Content -Path (Join-Path $DestinationDir 'guest-diag.txt') -Encoding UTF8 - Write-Host "[Diagnostics] Linux diagnostics written." - - # Best-effort: copy CI build log - $scpSrc = "${SshUser}@${IPAddress}:/opt/ci/build/ci-build.log" - $scpDst = Join-Path $DestinationDir 'build.log' - $ErrorActionPreference = 'Continue' - & scp.exe @sshOpts $scpSrc $scpDst 2>&1 | Out-Null - $ErrorActionPreference = $savedEap - if (Test-Path $scpDst) { Write-Host "[Diagnostics] Build log copied." } - else { Write-Host "[Diagnostics] Build log not found in guest (skipping)." } - } catch { - Write-Warning "[Diagnostics] Linux guest diagnostics failed (non-fatal): $_" - } - } -} - -Export-ModuleMember -Function New-CISessionOption, Resolve-VmrunPath, Invoke-Vmrun, Invoke-VmrunBounded, Get-GuestIPAddress, Get-GuestDiagnostics diff --git a/scripts/_Transport.psm1 b/scripts/_Transport.psm1 deleted file mode 100644 index 08558b0..0000000 --- a/scripts/_Transport.psm1 +++ /dev/null @@ -1,201 +0,0 @@ -#Requires -Version 5.1 -<# -.SYNOPSIS - SSH/SCP transport helpers for Linux build VMs. - -.DESCRIPTION - SSH-only transport layer for the Linux branch of the CI/CD system. - The existing WinRM transport (used by Windows VMs) is NOT modified. - - Exported functions: - Invoke-SshCommand — run a command in a Linux guest via ssh.exe - Copy-SshItem — copy files to/from a Linux guest via scp.exe - Test-SshReady — poll until SSH is ready (port 22 open + login succeeds) - - Requirements: - - ssh.exe and scp.exe in PATH (default on Windows 11 / Server 2022) - - SSH private key at the path provided - - Guest user must be set up for key-based auth (no passphrase) -#> -Set-StrictMode -Version Latest -$ErrorActionPreference = 'Stop' -$ProgressPreference = 'SilentlyContinue' # suppress Write-Progress (Test-NetConnection, IWR, etc.) - -function Invoke-SshCommand { - <# - .SYNOPSIS - Run a command in a Linux guest via ssh.exe - #> - [CmdletBinding()] - param( - [Parameter(Mandatory)] - [string] $IP, - - [string] $User = 'ci_build', - - [string] $KeyPath = 'F:\CI\keys\ci_linux', - - [Parameter(Mandatory)] - [string] $Command, - - [int] $ConnectTimeout = 10, - - # Persistent known_hosts file for CI jobs. - # When non-empty: StrictHostKeyChecking=accept-new + this file. - # When empty (default): permissive mode (StrictHostKeyChecking=no, UserKnownHostsFile=NUL) - # — used during template provisioning when host key is not yet known. - [string] $KnownHostsFile = '', - - # Capture + return stdout/stderr instead of printing to console. - [switch] $PassThru, - - # Don't throw on non-zero exit — just return the output. - [switch] $AllowFail - ) - - if ($KnownHostsFile -ne '') { - $sshHostKeyOpts = @('-o', 'StrictHostKeyChecking=accept-new', '-o', "UserKnownHostsFile=$KnownHostsFile") - } else { - $sshHostKeyOpts = @('-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=NUL') - } - $sshArgs = @( - '-i', $KeyPath - ) + $sshHostKeyOpts + @( - '-o', "ConnectTimeout=$ConnectTimeout", - '-o', 'BatchMode=yes', - "$User@$IP", - $Command - ) - - if ($PassThru) { - $output = & ssh @sshArgs 2>&1 - $exit = $LASTEXITCODE - } - else { - & ssh @sshArgs - $exit = $LASTEXITCODE - $output = @() - } - - if ($exit -ne 0 -and -not $AllowFail) { - throw "[SSH] Command failed (exit $exit): $Command" - } - - return [pscustomobject]@{ - ExitCode = [int]$exit - Output = [string[]]$output - } -} - -function Copy-SshItem { - <# - .SYNOPSIS - Copy files to/from Linux guest via scp.exe - #> - [CmdletBinding()] - param( - [Parameter(Mandatory)] - [string] $Source, - - [Parameter(Mandatory)] - [string] $Destination, - - # Guest IP — used to build user@host prefix when Direction is set. - [string] $IP = '', - - [string] $User = 'ci_build', - - [Parameter(Mandatory)] - [string] $KeyPath, - - [ValidateSet('ToGuest', 'FromGuest', 'Direct')] - [string] $Direction = 'Direct', - - # Pass -r to scp (recursive copy). - [switch] $Recurse, - - [int] $ConnectTimeout = 10, - - # Persistent known_hosts file for CI jobs. - # When non-empty: StrictHostKeyChecking=accept-new + this file. - # When empty (default): permissive mode — used during template provisioning. - [string] $KnownHostsFile = '' - ) - - if ($Direction -eq 'ToGuest') { - $Destination = "$User@${IP}:$Destination" - } - elseif ($Direction -eq 'FromGuest') { - $Source = "$User@${IP}:$Source" - } - # Direction = 'Direct': use Source + Destination as-is - - if ($KnownHostsFile -ne '') { - $scpHostKeyOpts = @('-o', 'StrictHostKeyChecking=accept-new', '-o', "UserKnownHostsFile=$KnownHostsFile") - } else { - $scpHostKeyOpts = @('-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=NUL') - } - $scpArgs = @( - '-i', $KeyPath - ) + $scpHostKeyOpts + @( - '-o', "ConnectTimeout=$ConnectTimeout" - ) - if ($Recurse) { $scpArgs += '-r' } - $scpArgs += @($Source, $Destination) - - & scp @scpArgs - if ($LASTEXITCODE -ne 0) { - throw "[SCP] Copy failed (exit $LASTEXITCODE): $Source -> $Destination" - } -} - -function Test-SshReady { - <# - .SYNOPSIS - Poll until SSH port 22 is open and a login command succeeds. - #> - [CmdletBinding()] - param( - [Parameter(Mandatory)] - [string] $IP, - - [string] $User = 'ci_build', - - [string] $KeyPath = 'F:\CI\keys\ci_linux', - - [int] $TimeoutSec = 300, - - [int] $PollSec = 10 - ) - - $deadline = (Get-Date).AddSeconds($TimeoutSec) - - # Phase 1: wait for port 22 to be open - $portOpen = $false - while (-not $portOpen -and (Get-Date) -lt $deadline) { - $portOpen = (Test-NetConnection -ComputerName $IP -Port 22 ` - -InformationLevel Quiet -WarningAction SilentlyContinue ` - -ErrorAction SilentlyContinue) - if (-not $portOpen) { - Start-Sleep -Seconds $PollSec - } - } - - if (-not $portOpen) { - throw "[Test-SshReady] Timed out after ${TimeoutSec}s waiting for SSH at $IP" - } - - # Phase 2: try ssh echo — success when output contains 'ready' - while ((Get-Date) -lt $deadline) { - $result = Invoke-SshCommand -IP $IP -User $User -KeyPath $KeyPath ` - -Command 'echo ready' -AllowFail -PassThru - if ($result.ExitCode -eq 0 -and ($result.Output -join '') -like '*ready*') { - return $true - } - Start-Sleep -Seconds $PollSec - } - - throw "[Test-SshReady] Timed out after ${TimeoutSec}s waiting for SSH at $IP" -} - -Export-ModuleMember -Function Invoke-SshCommand, Copy-SshItem, Test-SshReady diff --git a/setup-host-linux.sh b/setup-host-linux.sh index d4d573a..31bb6db 100755 --- a/setup-host-linux.sh +++ b/setup-host-linux.sh @@ -11,8 +11,10 @@ # portando i contenuti alla root della partizione (nessun dato perso) # 3. Crea il layout directory e permessi # 4. Installa Python 3.11+ e crea venv produzione in /opt/ci/venv -# 5. (Opzionale) Installa PowerShell Core -# 6. (Opzionale) Clona il repo e installa il package nel venv +# 5. (Opzionale) Clona il repo e installa il package nel venv +# 6. (Opt-in, --with-pwsh) Installa PowerShell Core +# L'host Linux NON ha più bisogno di pwsh per il normale funzionamento +# (orchestratore Python). Installalo solo se ti serve per script legacy. set -euo pipefail @@ -24,7 +26,7 @@ OPT_CI="/opt/ci" REPO_URL="https://gitea.emulab.it/Simone/local-ci-cd-system.git" SKIP_PYTHON=false -SKIP_PWSH=false +WITH_PWSH=false SKIP_CLONE=false # ── Colori ─────────────────────────────────────────────────────────────────── @@ -45,12 +47,16 @@ Opzionali: -r CI root (default: /var/lib/ci) -u Utente di servizio (default: ci-runner) --skip-python Salta installazione Python/venv - --skip-pwsh Salta installazione PowerShell Core --skip-clone Salta clone repo + pip install + --with-pwsh Installa PowerShell Core (opt-in; NON installato di default). + L'host Linux non ha più bisogno di pwsh per il normale + funzionamento — usa questo flag solo per script legacy. + --skip-pwsh DEPRECATO: no-op. pwsh non è più installato di default. Esempi: sudo bash setup-host-linux.sh -d /dev/sdf1 - sudo bash setup-host-linux.sh -d /dev/sdf1 --skip-pwsh --skip-clone + sudo bash setup-host-linux.sh -d /dev/sdf1 --with-pwsh + sudo bash setup-host-linux.sh -d /dev/sdf1 --skip-clone EOF exit 1 } @@ -63,7 +69,10 @@ while [[ $# -gt 0 ]]; do -r) CI_ROOT="$2"; shift 2 ;; -u) CI_USER="$2"; shift 2 ;; --skip-python) SKIP_PYTHON=true; shift ;; - --skip-pwsh) SKIP_PWSH=true; shift ;; + --with-pwsh) WITH_PWSH=true; shift ;; + --skip-pwsh) + echo "WARN: --skip-pwsh è deprecato e non ha più effetto: pwsh non è installato di default (usa --with-pwsh per installarlo)." >&2 + shift ;; --skip-clone) SKIP_CLONE=true; shift ;; -h|--help) usage ;; *) die "Parametro sconosciuto: $1"; ;; @@ -230,8 +239,8 @@ else info "[5/6] Clone repo — SKIPPED" fi -# ── Step 6: PowerShell Core ─────────────────────────────────────────────────── -if [[ "$SKIP_PWSH" == false ]]; then +# ── Step 6: PowerShell Core (opt-in via --with-pwsh) ────────────────────────── +if [[ "$WITH_PWSH" == true ]]; then info "[6/6] PowerShell Core" if command -v pwsh &>/dev/null; then warn "pwsh già installato: $(pwsh --version) — skip" @@ -254,7 +263,7 @@ if [[ "$SKIP_PWSH" == false ]]; then echo " Installato: $(pwsh --version)" fi else - info "[6/6] PowerShell Core — SKIPPED" + info "[6/6] PowerShell Core — SKIPPED (opt-in con --with-pwsh)" fi # ── Riepilogo ───────────────────────────────────────────────────────────────── diff --git a/src/ci_orchestrator/__main__.py b/src/ci_orchestrator/__main__.py index f06cabe..3449c11 100644 --- a/src/ci_orchestrator/__main__.py +++ b/src/ci_orchestrator/__main__.py @@ -13,12 +13,16 @@ import click from ci_orchestrator import __version__ from ci_orchestrator.commands.artifacts import artifacts +from ci_orchestrator.commands.bench import bench from ci_orchestrator.commands.build import build +from ci_orchestrator.commands.creds import creds from ci_orchestrator.commands.job import job from ci_orchestrator.commands.monitor import monitor from ci_orchestrator.commands.report import report from ci_orchestrator.commands.retention import retention +from ci_orchestrator.commands.smoke import smoke from ci_orchestrator.commands.template import template +from ci_orchestrator.commands.validate import validate from ci_orchestrator.commands.vm import vm # Re-export so legacy A1 tests that patched ``cli_module.`` keep working. @@ -46,6 +50,10 @@ cli.add_command(report) cli.add_command(retention) cli.add_command(template) cli.add_command(job) +cli.add_command(bench) +cli.add_command(smoke) +cli.add_command(validate) +cli.add_command(creds) if __name__ == "__main__": # pragma: no cover diff --git a/src/ci_orchestrator/commands/bench.py b/src/ci_orchestrator/commands/bench.py new file mode 100644 index 0000000..a140ae7 --- /dev/null +++ b/src/ci_orchestrator/commands/bench.py @@ -0,0 +1,704 @@ +"""``bench`` sub-commands. + +Phase C ports the manual PowerShell benchmark/burn-in tooling to Python so +the Linux host no longer needs ``pwsh``: + +* ``bench run`` — concurrency burn-in (replaces ``Test-CapacityBurnIn.ps1`` + + ``Start-BurnInTest*.ps1``). +* ``bench measure`` — phase timing (replaces ``Measure-CIBenchmark.ps1``). + +Design rule: import ``backends.load_backend`` and the existing transports +only — never ``WorkstationVmrunBackend`` directly (keeps the Phase D ESXi +path open). No ``Start-Job``/``pwsh``/``$IsWindows``. +""" + +from __future__ import annotations + +import contextlib +import json +import shutil +import socket +import time +from concurrent.futures import ThreadPoolExecutor, as_completed +from dataclasses import asdict, dataclass, field +from datetime import UTC, datetime +from pathlib import Path +from typing import TYPE_CHECKING + +import click + +from ci_orchestrator.backends import load_backend +from ci_orchestrator.backends.errors import BackendError, BackendNotAvailable +from ci_orchestrator.backends.protocol import VmHandle +from ci_orchestrator.config import Config, load_config + +if TYPE_CHECKING: # pragma: no cover + from ci_orchestrator.backends.protocol import VmBackend + + +# Clone directories left behind by a job are named ``Clone__...`` +# (see commands/job.py:_make_clone_name and commands/vm.py:vm_new). +_CLONE_PREFIX = "Clone_" +# A vm-start.lock older than this (seconds) is considered stale leftover. +_STALE_LOCK_SECONDS = 5 * 60 +# Linux guests probe SSH/22; everything else probes WinRM/5986. +_SSH_PORT = 22 +_WINRM_PORT = 5986 + + +def _now() -> float: + return time.monotonic() + + +def _timestamp() -> str: + return datetime.now(tz=UTC).strftime("%Y%m%d-%H%M%S") + + +# ─────────────────────────────────────────────────────────────────── group + + +@click.group("bench") +def bench() -> None: + """Capacity burn-in and phase-timing benchmarks.""" + + +# ═══════════════════════════════════════════════════════════════ bench run + + +@dataclass +class JobResult: + """Outcome of a single burn-in job slot.""" + + job_id: str + slot: int + exit_code: int + status: str + elapsed_sec: float + error: str = "" + + +@dataclass +class RoundResult: + """Outcome of one burn-in round (all slots + cleanup assertions).""" + + round: int + passed: int + failed: int + status: str + elapsed_sec: float + orphans: list[str] = field(default_factory=list) + stale_lock: bool = False + jobs: list[JobResult] = field(default_factory=list) + + +def _run_one_job( + *, + job_id: str, + guest_os: str, + build_command: str, + clone_base_dir: Path, +) -> tuple[int, str]: + """Run a single job end to end, returning ``(exit_code, error_message)``. + + Invokes the in-process ``commands.job.job`` command (the same code path + production uses) rather than re-spawning ``python -m ci_orchestrator`` or + a PowerShell child — closes the ``Start-Job``/``$IsWindows`` shim bug. + + Isolated in its own module-level function so the burn-in harness can be + unit tested by monkeypatching it (no real VMs needed). + """ + from click.testing import CliRunner # local import: drives the job command + + from ci_orchestrator.commands.job import job + + args = [ + "--job-id", + job_id, + "--repo-url", + "", + "--branch", + "main", + "--guest-os", + guest_os, + "--clone-base-dir", + str(clone_base_dir), + "--skip-artifact", + ] + if build_command: + args += ["--build-command", build_command] + result = CliRunner().invoke(job, args, catch_exceptions=True) + if result.exit_code == 0: + return 0, "" + tail = result.output.strip().splitlines() + return result.exit_code, tail[-1] if tail else "" + + +def _find_orphans(clone_base: Path, job_ids: list[str]) -> list[Path]: + """Return clone dirs in ``clone_base`` belonging to the given jobs.""" + if not clone_base.is_dir(): + return [] + prefixes = tuple(f"{_CLONE_PREFIX}{jid}_" for jid in job_ids) + return [ + entry + for entry in clone_base.iterdir() + if entry.is_dir() and entry.name.startswith(prefixes) + ] + + +def _stale_lock(lock_file: Path, max_age_seconds: int = _STALE_LOCK_SECONDS) -> bool: + """True if ``lock_file`` exists and is older than ``max_age_seconds``.""" + if not lock_file.is_file(): + return False + try: + mtime = lock_file.stat().st_mtime + except OSError: + return False + return (time.time() - mtime) > max_age_seconds + + +def _run_round( + *, + round_no: int, + concurrency: int, + guest_os: str, + build_command: str, + clone_base_dir: Path, + lock_file: Path, +) -> RoundResult: + """Launch ``concurrency`` concurrent jobs and assert cleanup invariants.""" + click.echo(f"\n[bench run] Round {round_no}: launching {concurrency} job(s)...") + started = _now() + ts = _timestamp() + slots = { + slot: f"burnin-r{round_no}-j{slot}-{ts}" + for slot in range(1, concurrency + 1) + } + + jobs: list[JobResult] = [] + with ThreadPoolExecutor(max_workers=concurrency) as pool: + futures = { + pool.submit( + _run_one_job, + job_id=job_id, + guest_os=guest_os, + build_command=build_command, + clone_base_dir=clone_base_dir, + ): (slot, job_id) + for slot, job_id in slots.items() + } + for fut in as_completed(futures): + slot, job_id = futures[fut] + try: + exit_code, error = fut.result() + except Exception as exc: # pragma: no cover - defensive + exit_code, error = 1, str(exc) + status = "PASS" if exit_code == 0 else "FAIL" + jobs.append( + JobResult( + job_id=job_id, + slot=slot, + exit_code=exit_code, + status=status, + elapsed_sec=round(_now() - started, 2), + error=error, + ) + ) + + jobs.sort(key=lambda j: j.slot) + passed = sum(1 for j in jobs if j.status == "PASS") + failed = len(jobs) - passed + + # ── Assertion: no orphaned clone directories ───────────────────────── + orphans = _find_orphans(clone_base_dir, list(slots.values())) + if orphans: + failed += 1 + + # ── Assertion: no stale vm-start.lock ──────────────────────────────── + stale = _stale_lock(lock_file) + if stale: + failed += 1 + + status = "PASS" if failed == 0 else "FAIL" + result = RoundResult( + round=round_no, + passed=passed, + failed=failed, + status=status, + elapsed_sec=round(_now() - started, 2), + orphans=[str(p) for p in orphans], + stale_lock=stale, + jobs=jobs, + ) + + for j in jobs: + line = f" slot {j.slot} {j.job_id:<32} {j.status:<5} exit={j.exit_code}" + if j.error: + line += f" ({j.error})" + click.echo(line) + click.echo( + f" clone cleanup : {'OK' if not orphans else f'FAIL ({len(orphans)} orphan)'}" + ) + for p in result.orphans: + click.echo(f" orphan: {p}") + click.echo(f" lock cleanup : {'OK' if not stale else 'FAIL (stale lock)'}") + click.echo( + f" [Round {round_no}] {status} " + f"({passed} PASS, {failed} FAIL, {result.elapsed_sec}s)" + ) + return result + + +def _write_run_report(report: dict[str, object], json_out: Path) -> None: + json_out.parent.mkdir(parents=True, exist_ok=True) + json_out.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +@bench.command("run") +@click.option( + "--guest-os", + type=click.Choice(["windows", "linux"], case_sensitive=False), + default="linux", + show_default=True, + help="Guest OS family of the build VMs under test.", +) +@click.option( + "--concurrency", + type=click.IntRange(1, 32), + default=4, + show_default=True, + help="Number of concurrent jobs per round.", +) +@click.option( + "--rounds", + type=click.IntRange(1, 1000), + default=10, + show_default=True, + help="Number of sequential rounds to run.", +) +@click.option( + "--build-command", + "build_command", + default="", + help="Build command each job runs (default: a no-op sleep/ping).", +) +@click.option( + "--clone-base-dir", + "clone_base_dir", + default=None, + help="Override clone base dir (defaults to config.paths.build_vms).", +) +@click.option( + "--json-out", + "json_out", + default=None, + help="Path for the JSON report (default: CI_ARTIFACTS/bench/.json).", +) +def bench_run( + guest_os: str, + concurrency: int, + rounds: int, + build_command: str, + clone_base_dir: str | None, + json_out: str | None, +) -> None: + """Run a concurrency burn-in and assert cleanup invariants per round. + + Replaces ``Test-CapacityBurnIn.ps1`` + ``Start-BurnInTest*.ps1``. For + each of ``--rounds`` rounds it launches ``--concurrency`` concurrent + jobs (in-process, via :func:`concurrent.futures`), then asserts: every + job exited 0, no orphaned clone dir remains in the clone base dir, and + no ``vm-start.lock`` older than 5 minutes is left behind. + """ + guest_os = guest_os.lower() + config = load_config() + base = Path(clone_base_dir) if clone_base_dir else config.paths.build_vms + lock_file = base / "vm-start.lock" + + started_at = datetime.now(tz=UTC) + out_path = ( + Path(json_out) + if json_out + else config.paths.artifacts / "bench" / f"{_timestamp()}.json" + ) + + click.echo("=" * 60) + click.echo("[bench run] capacity burn-in") + click.echo(f" guest-os : {guest_os}") + click.echo(f" concurrency : {concurrency}") + click.echo(f" rounds : {rounds}") + click.echo(f" clone base : {base}") + click.echo(f" build cmd : {build_command or '(default no-op)'}") + click.echo("=" * 60) + + round_results: list[RoundResult] = [] + for r in range(1, rounds + 1): + round_results.append( + _run_round( + round_no=r, + concurrency=concurrency, + guest_os=guest_os, + build_command=build_command, + clone_base_dir=base, + lock_file=lock_file, + ) + ) + + total_pass = sum(rr.passed for rr in round_results) + total_fail = sum(rr.failed for rr in round_results) + rounds_failed = sum(1 for rr in round_results if rr.status == "FAIL") + overall = "PASS" if total_fail == 0 else "FAIL" + + # ── Summary table ──────────────────────────────────────────────────── + click.echo("\n" + "=" * 60) + click.echo("[bench run] SUMMARY") + click.echo(f" {'Round':<7}{'Pass':<6}{'Fail':<6}{'Elapsed':<10}Status") + click.echo(f" {'-' * 5:<7}{'-' * 4:<6}{'-' * 4:<6}{'-' * 8:<10}{'-' * 6}") + for rr in round_results: + click.echo( + f" {rr.round:<7}{rr.passed:<6}{rr.failed:<6}" + f"{str(rr.elapsed_sec) + 's':<10}{rr.status}" + ) + click.echo("") + click.echo(f" Total jobs : {rounds * concurrency}") + click.echo(f" Passed : {total_pass}") + click.echo(f" Failed : {total_fail}") + click.echo(f" Rounds OK : {rounds - rounds_failed} / {rounds}") + click.echo(f"\n OVERALL: {overall}") + click.echo("=" * 60) + + report: dict[str, object] = { + "ts": started_at.isoformat(), + "guestOS": guest_os, + "concurrency": concurrency, + "rounds": rounds, + "buildCommand": build_command, + "totalJobs": rounds * concurrency, + "totalPass": total_pass, + "totalFail": total_fail, + "roundsFailed": rounds_failed, + "overall": overall, + "results": [asdict(rr) for rr in round_results], + } + try: + _write_run_report(report, out_path) + click.echo(f"[bench run] report written: {out_path}") + except OSError as exc: + click.echo(f"[bench run] could not write report: {exc}", err=True) + + if total_fail > 0: + raise SystemExit(1) + + +# ═══════════════════════════════════════════════════════════ bench measure + + +@dataclass +class PhaseTiming: + """Per-iteration phase timings, mirroring ``Measure-CIBenchmark.ps1``. + + Field names of the serialised record (see :meth:`to_record`) are kept + identical to the PowerShell version for trend-log continuity. + """ + + iteration: int + run_id: str + template: str + snapshot: str + guest_os: str + clone_sec: float | None = None + start_sec: float | None = None + ip_sec: float | None = None + ready_sec: float | None = None + destroy_sec: float | None = None + delta_kb: int | None = None + vm_ip: str | None = None + error: str | None = None + + @property + def total_boot_sec(self) -> float | None: + if self.ip_sec is None or self.ready_sec is None: + return None + return round( + (self.clone_sec or 0.0) + + (self.start_sec or 0.0) + + self.ip_sec + + self.ready_sec, + 2, + ) + + def to_record(self, ts: str) -> dict[str, object]: + """Build a JSON record with the legacy ``Measure-CIBenchmark`` keys.""" + return { + "ts": ts, + "runId": self.run_id, + "iteration": self.iteration, + "template": self.template, + "snapshot": self.snapshot, + "guestOS": self.guest_os, + "cloneSec": self.clone_sec, + "startSec": self.start_sec, + "ipSec": self.ip_sec, + "readySec": self.ready_sec, + "destroySec": self.destroy_sec, + "totalBootSec": self.total_boot_sec, + "deltaKB": self.delta_kb, + "vmIP": self.vm_ip, + "error": self.error, + } + + +def _dir_size_kb(path: Path) -> int | None: + """Total size of files under ``path`` in KiB, or None on error.""" + try: + total = sum(p.stat().st_size for p in path.rglob("*") if p.is_file()) + except OSError: + return None + return round(total / 1024) + + +def _tcp_open(host: str, port: int, timeout: float = 3.0) -> bool: + """Return True if a TCP connection to ``host:port`` succeeds.""" + try: + with socket.create_connection((host, port), timeout=timeout): + return True + except OSError: + return False + + +def _wait_port(host: str, port: int, deadline: float, poll: float = 3.0) -> bool: + while _now() < deadline: + if _tcp_open(host, port): + return True + time.sleep(poll) + return False + + +def _measure_iteration( + *, + backend: VmBackend, + iteration: int, + template: Path, + snapshot: str, + guest_os: str, + clone_base_dir: Path, + transport_port: int, + timeout_seconds: float, +) -> PhaseTiming: + """Clone → start → IP → transport-ready → destroy, timing each phase.""" + run_id = f"bench-{_timestamp()}-{iteration}" + clone_name = f"{_CLONE_PREFIX}{run_id}" + clone_dir = clone_base_dir / clone_name + clone_vmx = clone_dir / f"{clone_name}.vmx" + timing = PhaseTiming( + iteration=iteration, + run_id=run_id, + template=template.name, + snapshot=snapshot, + guest_os=guest_os, + ) + handle: VmHandle | None = None + try: + # ── clone ──────────────────────────────────────────────────────── + click.echo(f"[bench measure] iteration {iteration}: cloning...") + t0 = _now() + handle = backend.clone_linked( + template=str(template), + snapshot=snapshot, + name=clone_name, + destination=str(clone_vmx), + ) + timing.clone_sec = round(_now() - t0, 2) + timing.delta_kb = _dir_size_kb(clone_dir) + + # ── start ──────────────────────────────────────────────────────── + click.echo("[bench measure] starting...") + t0 = _now() + backend.start(handle, headless=True) + timing.start_sec = round(_now() - t0, 2) + + # ── IP acquire ──────────────────────────────────────────────────── + click.echo("[bench measure] waiting for guest IP...") + t0 = _now() + deadline = t0 + timeout_seconds + vm_ip: str | None = None + while _now() < deadline: + vm_ip = backend.get_ip(handle, timeout=min(5.0, deadline - _now())) + if vm_ip: + break + if not vm_ip: + raise TimeoutError(f"no guest IP after {timeout_seconds}s") + timing.ip_sec = round(_now() - t0, 2) + timing.vm_ip = vm_ip + + # ── transport ready ─────────────────────────────────────────────── + click.echo(f"[bench measure] waiting for port {transport_port}...") + t0 = _now() + if not _wait_port(vm_ip, transport_port, _now() + timeout_seconds): + raise TimeoutError( + f"port {transport_port} on {vm_ip} not open after {timeout_seconds}s" + ) + timing.ready_sec = round(_now() - t0, 2) + except (BackendError, TimeoutError, OSError) as exc: + timing.error = str(exc) + click.echo(f"[bench measure] iteration {iteration} failed: {exc}", err=True) + finally: + # ── destroy ─────────────────────────────────────────────────────── + t0 = _now() + if handle is not None: + with contextlib.suppress(BackendError): + backend.stop(handle, hard=True) + with contextlib.suppress(BackendError): + backend.delete(handle) + if clone_dir.exists(): + shutil.rmtree(clone_dir, ignore_errors=True) + timing.destroy_sec = round(_now() - t0, 2) + return timing + + +def _resolve_template(config: Config, guest_os: str) -> tuple[Path, str]: + """Pick a default template VMX + snapshot for ``guest_os``.""" + if guest_os == "linux": + return ( + config.paths.templates / "LinuxBuild2404" / "LinuxBuild2404.vmx", + "BaseClean-Linux", + ) + return ( + config.paths.templates / "WinBuild2025" / "WinBuild2025.vmx", + "BaseClean", + ) + + +@bench.command("measure") +@click.option( + "--guest-os", + type=click.Choice(["windows", "linux"], case_sensitive=False), + default="linux", + show_default=True, + help="Guest OS family of the VM to benchmark.", +) +@click.option( + "--iterations", + type=click.IntRange(1, 100), + default=1, + show_default=True, + help="Number of ephemeral VMs to time.", +) +@click.option( + "--template-path", + "template_path", + default=None, + help="Template VMX path (defaults to the per-guest-os template).", +) +@click.option( + "--snapshot-name", + "snapshot_name", + default=None, + help="Snapshot to clone from (defaults to BaseClean / BaseClean-Linux).", +) +@click.option( + "--clone-base-dir", + "clone_base_dir", + default=None, + help="Override clone base dir (defaults to config.paths.build_vms).", +) +@click.option( + "--timeout-seconds", + "timeout_seconds", + type=click.FloatRange(30.0, 1200.0), + default=300.0, + show_default=True, + help="Per-iteration timeout for IP and transport readiness.", +) +@click.option( + "--json-out", + "json_out", + default=None, + help="Path to append JSONL records (default: CI_ARTIFACTS/benchmark.jsonl).", +) +def bench_measure( + guest_os: str, + iterations: int, + template_path: str | None, + snapshot_name: str | None, + clone_base_dir: str | None, + timeout_seconds: float, + json_out: str | None, +) -> None: + """Measure clone/start/IP/transport/destroy phase timings. + + Replaces ``Measure-CIBenchmark.ps1``. Records are appended to + ``benchmark.jsonl`` with the same field names so historical trend data + stays comparable. + """ + guest_os = guest_os.lower() + config = load_config() + + default_tpl, default_snap = _resolve_template(config, guest_os) + template = Path(template_path) if template_path else default_tpl + snapshot = snapshot_name or default_snap + base = Path(clone_base_dir) if clone_base_dir else config.paths.build_vms + transport_port = _SSH_PORT if guest_os == "linux" else _WINRM_PORT + out_path = Path(json_out) if json_out else config.paths.artifacts / "benchmark.jsonl" + + if not template.is_file(): + raise click.ClickException(f"Template VMX not found: {template}") + + try: + backend = load_backend(config) + except BackendNotAvailable as exc: + raise click.ClickException(f"backend unavailable: {exc}") from exc + + base.mkdir(parents=True, exist_ok=True) + out_path.parent.mkdir(parents=True, exist_ok=True) + + click.echo("=" * 60) + click.echo("[bench measure] phase timing") + click.echo(f" template : {template.name}") + click.echo(f" snapshot : {snapshot}") + click.echo(f" guest-os : {guest_os} (port {transport_port})") + click.echo(f" iterations : {iterations}") + click.echo("=" * 60) + + timings: list[PhaseTiming] = [] + for i in range(1, iterations + 1): + timing = _measure_iteration( + backend=backend, + iteration=i, + template=template, + snapshot=snapshot, + guest_os=guest_os, + clone_base_dir=base, + transport_port=transport_port, + timeout_seconds=timeout_seconds, + ) + timings.append(timing) + record = timing.to_record(datetime.now(tz=UTC).isoformat()) + try: + with out_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(record) + "\n") + except OSError as exc: + click.echo(f"[bench measure] could not append jsonl: {exc}", err=True) + + # ── Summary table ──────────────────────────────────────────────────── + click.echo("\n[bench measure] Results") + header = f" {'Iter':<5}{'Clone':<8}{'Start':<8}{'IP':<8}{'Ready':<8}{'Destroy':<9}{'Boot':<8}IP" + click.echo(header) + for t in timings: + click.echo( + f" {t.iteration:<5}" + f"{t.clone_sec!s:<8}{t.start_sec!s:<8}{t.ip_sec!s:<8}" + f"{t.ready_sec!s:<8}{t.destroy_sec!s:<9}" + f"{t.total_boot_sec!s:<8}{t.vm_ip or '-'}" + ) + if t.error: + click.echo(f" error: {t.error}") + + successful = [t for t in timings if t.error is None and t.total_boot_sec is not None] + if len(successful) > 1: + boots = [t.total_boot_sec for t in successful if t.total_boot_sec is not None] + avg = round(sum(boots) / len(boots), 2) + click.echo(f"\n Average boot-to-ready ({len(successful)} ok): {avg}s") + click.echo(f"\n[bench measure] records appended to: {out_path}") + + +__all__ = ["bench"] diff --git a/src/ci_orchestrator/commands/creds.py b/src/ci_orchestrator/commands/creds.py new file mode 100644 index 0000000..c0ea44d --- /dev/null +++ b/src/ci_orchestrator/commands/creds.py @@ -0,0 +1,87 @@ +"""``creds`` sub-commands. + +Phase C replaces ``Set-CIGuestCredential.ps1`` on the Linux host with a +keyring-native writer (``keyrings.alt.file.PlaintextKeyring``), matching the +read path in :mod:`ci_orchestrator.credentials`. + +* ``creds set`` — store a guest username/password under a target. Passwords + are never passed on the command line. + +Write scheme (must mirror :meth:`KeyringCredentialStore.get`): + +* ``keyring.set_password("{target}:meta", "username", )`` — so the + reader can recover the username for file backends where + ``get_credential(target, None)`` is a no-op. +* ``keyring.set_password(target, , )`` — the password, + keyed by the username under the target service. +""" + +from __future__ import annotations + +import sys + +import click + + +def _store_credential(target: str, user: str, password: str) -> None: + """Write the two keyring entries the credential reader expects. + + Raises :class:`RuntimeError` if ``keyring`` is not installed. + """ + try: + import keyring + except ImportError as exc: # pragma: no cover - dep declared in pyproject + raise RuntimeError( + "keyring is not installed; run `pip install keyring`." + ) from exc + + # 1. Username under the "{target}:meta" service, key "username". + keyring.set_password(f"{target}:meta", "username", user) + # 2. Password under the target service, keyed by the username. + keyring.set_password(target, user, password) + + +@click.group("creds") +def creds() -> None: + """Manage guest credentials in the keyring.""" + + +@creds.command("set") +@click.option( + "--target", + default="BuildVMGuest", + show_default=True, + help="Credential target name (matches config.guest_cred_target).", +) +@click.option("--user", "user", required=True, help="Guest username to store.") +@click.option( + "--password-stdin", + "password_stdin", + is_flag=True, + default=False, + help="Read the password from stdin instead of prompting.", +) +def creds_set(target: str, user: str, password_stdin: bool) -> None: + """Store guest credentials so ci_orchestrator can read them back.""" + if password_stdin: + # Read the whole of stdin and strip a single trailing newline only, + # so passwords containing internal whitespace survive intact. + password = sys.stdin.readline().rstrip("\n").rstrip("\r") + else: + password = click.prompt( + "Guest password", + hide_input=True, + confirmation_prompt=True, + ) + if not password: + raise click.UsageError("password must not be empty.") + + try: + _store_credential(target, user, password) + except RuntimeError as exc: + raise click.ClickException(str(exc)) from exc + + click.echo(f"Stored credential for target {target!r} (user={user}).") + + +__all__ = ["creds"] diff --git a/src/ci_orchestrator/commands/smoke.py b/src/ci_orchestrator/commands/smoke.py new file mode 100644 index 0000000..9b544a0 --- /dev/null +++ b/src/ci_orchestrator/commands/smoke.py @@ -0,0 +1,351 @@ +"""``smoke`` sub-commands. + +Phase C4 ports the manual smoke / E2E build tooling to Python so the Linux +host no longer needs ``pwsh``: + +* ``smoke run`` — end-to-end no-op (or real build) job, replacing + ``Test-Smoke.ps1`` and ``Test-*Build-Linux.ps1``. + +The command runs **one** full end-to-end job through the in-process +``job`` orchestrator (reusing ``commands/job.py`` — no re-spawn of +``python -m ci_orchestrator``) and then asserts the three smoke +invariants of ``Test-Smoke.ps1``: + +1. the pipeline exited 0, +2. the per-job artifact directory was created, and +3. a ``job``/``success`` event is present in the ``invoke-ci.jsonl`` + event log (the same JSONL format ``report job`` consumes). + +Because the in-process ``job`` command does not itself emit the JSONL +event log (that was the job of the retired ``Invoke-CIJob.ps1``), this +command writes the ``job``/``start`` and ``job``/``success`` events +around the orchestration, exactly as the PowerShell wrapper did. + +Design rule: import ``backends.load_backend`` and the existing transports +only — never ``WorkstationVmrunBackend`` directly (keeps the Phase D ESXi +path open). +""" + +from __future__ import annotations + +import json +import time +from datetime import UTC, datetime +from pathlib import Path + +import click + +from ci_orchestrator.commands.job import job as job_command +from ci_orchestrator.config import Config, load_config + +# ─────────────────────────────────────────────────────── OS presets + +# Per-guest defaults mirroring ``Test-Smoke.ps1``: a trivial build that +# writes a marker file into the standard output directory, plus the +# matching template / snapshot / artifact-source for the OS family. +_LINUX_TEMPLATE = "/var/lib/ci/templates/LinuxBuild2404/LinuxBuild2404.vmx" +_WINDOWS_TEMPLATE = r"F:\CI\Templates\WinBuild2025\WinBuild2025.vmx" + +_NOOP_BUILD_LINUX = "mkdir -p /opt/ci/output && echo smoke-ok > /opt/ci/output/smoke.txt" +_NOOP_BUILD_WINDOWS = ( + "New-Item -ItemType Directory -Path C:\\CI\\output -Force | Out-Null; " + 'Set-Content C:\\CI\\output\\smoke.txt "smoke-ok"' +) + +# Real Linux build E2E presets (ports of ``Test-*Build-Linux.ps1``). +# Exposed via ``--preset`` so the no-op default is never overridden by a +# hard-coded ``F:\`` Windows path. Each preset supplies a build command +# and the in-guest artifact source directory. +_PRESETS: dict[str, tuple[str, str]] = { + "ns7zip": ( + "python3 build_plugin.py --host linux --7zip-version 26.01 " + "--configs x64-unicode --verbose", + "plugins", + ), + "nsinnounp": ( + "python3 build_plugin.py --final --dist-dir dist", + "dist", + ), +} + + +def _default_template(guest_os: str) -> str: + return _LINUX_TEMPLATE if guest_os == "linux" else _WINDOWS_TEMPLATE + + +def _default_snapshot(guest_os: str) -> str: + return "BaseClean-Linux" if guest_os == "linux" else "BaseClean" + + +def _noop_build_command(guest_os: str) -> tuple[str, str]: + """Return ``(build_command, guest_artifact_source)`` for the marker job.""" + if guest_os == "linux": + return _NOOP_BUILD_LINUX, "/opt/ci/output" + return _NOOP_BUILD_WINDOWS, "C:\\CI\\output" + + +def _utcnow_iso() -> str: + return datetime.now(tz=UTC).isoformat() + + +def _write_event( + jsonl_path: Path, + *, + job_id: str, + phase: str, + status: str, + elapsed_sec: float | None = None, + error: str | None = None, +) -> None: + """Append one event to ``invoke-ci.jsonl`` in ``report job`` format.""" + data: dict[str, object] = {} + if elapsed_sec is not None: + data["elapsedSec"] = int(elapsed_sec) + if error is not None: + data["error"] = error + record: dict[str, object] = { + "jobId": job_id, + "phase": phase, + "status": status, + "ts": _utcnow_iso(), + "data": data, + } + jsonl_path.parent.mkdir(parents=True, exist_ok=True) + with jsonl_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(record) + "\n") + + +def _has_job_success(jsonl_path: Path) -> bool: + """True iff the JSONL log contains a ``job``/``success`` event.""" + try: + text = jsonl_path.read_text(encoding="utf-8") + except OSError: + return False + for line in text.splitlines(): + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(obj, dict): + continue + if obj.get("phase") == "job" and obj.get("status") == "success": + return True + return False + + +# ────────────────────────────────────────────────────────── command + + +@click.group("smoke") +def smoke() -> None: + """End-to-end smoke and build validation.""" + + +@smoke.command("run") +@click.option( + "--guest-os", + type=click.Choice(["windows", "linux"], case_sensitive=False), + default="linux", + show_default=True, + help="Guest OS family to smoke-test.", +) +@click.option( + "--build-command", + "build_command", + default="", + help="Build command to run (default: a marker no-op job).", +) +@click.option( + "--preset", + type=click.Choice(sorted(_PRESETS), case_sensitive=False), + default=None, + help="Real Linux build E2E preset (ns7zip / nsinnounp). Linux only.", +) +@click.option( + "--repo-url", + "repo_url", + default="https://example.invalid/smoke-test.git", + show_default=True, + help="Repository to build (default: a placeholder; the no-op marker " + "job never depends on the clone).", +) +@click.option("--branch", default="main", show_default=True) +@click.option( + "--job-id", + "job_id", + default=None, + help="Job id (default: smoke-).", +) +@click.option( + "--template-path", + "template_path", + default=None, + help="Template VMX path (default: per-guest-os standard template).", +) +@click.option( + "--snapshot-name", + "snapshot_name", + default=None, + help="Snapshot to clone from (default: per-guest-os BaseClean[-Linux]).", +) +@click.option( + "--ssh-key-path", + "ssh_key_path", + default=None, + help="SSH private key for the guest (Linux).", +) +@click.option( + "--log-dir", + "log_dir", + default=None, + help="Base directory for invoke-ci.jsonl logs (default: CI_ARTIFACTS).", +) +def smoke_run( + guest_os: str, + build_command: str, + preset: str | None, + repo_url: str, + branch: str, + job_id: str | None, + template_path: str | None, + snapshot_name: str | None, + ssh_key_path: str | None, + log_dir: str | None, +) -> None: + """Run one end-to-end job and assert success + artifact + event.""" + guest_os = guest_os.lower() + + if preset is not None: + preset = preset.lower() + if guest_os != "linux": + raise click.UsageError("--preset is only supported with --guest-os linux.") + if build_command: + raise click.UsageError("--preset and --build-command are mutually exclusive.") + preset_cmd, artifact_source = _PRESETS[preset] + effective_build_command = preset_cmd + elif build_command: + effective_build_command = build_command + # A custom command writes to the standard output dir for the OS. + _, artifact_source = _noop_build_command(guest_os) + else: + effective_build_command, artifact_source = _noop_build_command(guest_os) + + config: Config = load_config() + resolved_job_id = job_id or f"smoke-{datetime.now(tz=UTC).strftime('%Y%m%d-%H%M%S')}" + resolved_template = template_path or _default_template(guest_os) + resolved_snapshot = snapshot_name or _default_snapshot(guest_os) + + artifact_base: Path = config.paths.artifacts + # Logs live alongside (not inside) the artifacts tree so that writing the + # event log never accidentally creates the per-job artifact directory and + # masks a build that produced no artifacts (mirrors F:\CI\Logs vs + # F:\CI\Artifacts in Test-Smoke.ps1). + log_base = Path(log_dir) if log_dir else config.paths.root / "logs" + job_artifact_dir = artifact_base / resolved_job_id + jsonl_path = log_base / resolved_job_id / "invoke-ci.jsonl" + + click.echo("=" * 60) + click.echo(f"[smoke] Guest OS : {guest_os}") + click.echo(f"[smoke] Template : {resolved_template}") + click.echo(f"[smoke] Snapshot : {resolved_snapshot}") + click.echo(f"[smoke] Job id : {resolved_job_id}") + if preset is not None: + click.echo(f"[smoke] Preset : {preset}") + click.echo(f"[smoke] Artifacts : {job_artifact_dir}") + click.echo("=" * 60) + + _write_event( + jsonl_path, job_id=resolved_job_id, phase="job", status="start" + ) + + start = time.monotonic() + failed = False + try: + job_command.callback( # type: ignore[misc] + job_id=resolved_job_id, + repo_url=repo_url, + branch=branch, + commit="", + configuration="Release", + build_command=effective_build_command, + guest_artifact_source=artifact_source, + submodules=True, + use_git_clone=True, + use_shared_cache=False, + skip_artifact=False, + use_xvfb=False, + gitea_credential_target="GiteaPAT", + template_path=resolved_template, + snapshot_name=resolved_snapshot, + clone_base_dir=None, + artifact_base_dir=str(artifact_base), + guest_os=guest_os, + guest_credential_target=None, + ssh_key_path=ssh_key_path, + ssh_user="ci_build", + ssh_known_hosts_file=None, + extra_env_json="", + guest_cpu=0, + guest_memory_mb=0, + ready_timeout=600.0, + poll_interval=5.0, + vmrun_path=None, + ) + except click.ClickException as exc: + failed = True + _write_event( + jsonl_path, + job_id=resolved_job_id, + phase="job", + status="failure", + elapsed_sec=time.monotonic() - start, + error=exc.format_message(), + ) + click.echo(f"[smoke] FAIL: pipeline raised: {exc.format_message()}", err=True) + + if not failed: + _write_event( + jsonl_path, + job_id=resolved_job_id, + phase="job", + status="success", + elapsed_sec=time.monotonic() - start, + ) + + # ── Assert the three smoke invariants ─────────────────────────────── + click.echo("\n[smoke] Verifying results...") + checks_ok = True + + if failed: + checks_ok = False + else: + click.echo("[smoke] OK: pipeline exit 0") + + if job_artifact_dir.is_dir(): + click.echo(f"[smoke] OK: artifact dir exists: {job_artifact_dir}") + else: + click.echo( + f"[smoke] FAIL: artifact dir not found: {job_artifact_dir}", err=True + ) + checks_ok = False + + if _has_job_success(jsonl_path): + click.echo("[smoke] OK: JSONL contains job/success event") + else: + click.echo( + f"[smoke] FAIL: JSONL missing job/success event in: {jsonl_path}", + err=True, + ) + checks_ok = False + + click.echo("") + if checks_ok: + click.echo(f"[smoke] PASSED — {guest_os} smoke test completed successfully.") + return + raise click.ClickException("smoke test FAILED — one or more checks did not pass.") + + +__all__ = ["smoke"] diff --git a/src/ci_orchestrator/commands/validate.py b/src/ci_orchestrator/commands/validate.py new file mode 100644 index 0000000..193295d --- /dev/null +++ b/src/ci_orchestrator/commands/validate.py @@ -0,0 +1,285 @@ +"""``validate`` sub-commands. + +Phase C adds Linux-native host/guest validation, replacing host-side use of +``Validate-HostState.ps1`` and the diagnostic part of ``Test-CIGuestWinRM.ps1``: + +* ``validate host`` — vmrun present, template snapshots present, keyring + readable, ``/var/lib/ci`` permissions, systemd units active. +* ``validate guest`` — connect to a guest over WinRM/SSH and run a trivial + command, reporting the explicit failure cause. + +The host checks are a Linux-native equivalent of ``Validate-HostState.ps1`` +(not a 1:1 port): no Windows drive paths, no Windows Credential Manager, no +NTFS ACLs. The guest check is the diagnostic core of ``Test-CIGuestWinRM.ps1``: +unlike the ``is_ready`` probe (which swallows errors and returns a bool), +this reports the explicit failure cause. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from dataclasses import dataclass + +import click + +from ci_orchestrator.backends import load_backend +from ci_orchestrator.backends.errors import BackendError, BackendNotAvailable +from ci_orchestrator.backends.protocol import VmHandle +from ci_orchestrator.config import Config, load_config +from ci_orchestrator.credentials import KeyringCredentialStore +from ci_orchestrator.transport.errors import TransportCommandError, TransportConnectError +from ci_orchestrator.transport.ssh import SshTransport +from ci_orchestrator.transport.winrm import WinRmTransport + +# Each template directory and the snapshot a clone is taken from. Mirrors the +# "VM templates and snapshots" table in CLAUDE.md. +_TEMPLATE_SNAPSHOTS: tuple[tuple[str, str], ...] = ( + ("WinBuild2025", "BaseClean"), + ("WinBuild2022", "BaseClean"), + ("LinuxBuild2404", "BaseClean-Linux"), +) + +# systemd units expected to be active on a configured Linux host. +_EXPECTED_UNITS: tuple[str, ...] = ("act-runner.service",) + + +@dataclass(frozen=True) +class CheckResult: + """Outcome of a single host check.""" + + name: str + ok: bool + detail: str + + +@click.group("validate") +def validate() -> None: + """Host and guest readiness checks.""" + + +# ──────────────────────────────────────────────────────── validate host + +def _check_vmrun(config: Config) -> CheckResult: + """vmrun is present and executable (via the backend factory).""" + try: + backend = load_backend(config) + except BackendNotAvailable as exc: + return CheckResult("vmrun", False, str(exc)) + except BackendError as exc: # pragma: no cover - defensive + return CheckResult("vmrun", False, str(exc)) + vmrun_path = getattr(backend, "vmrun_path", None) + if vmrun_path: + return CheckResult("vmrun", True, f"found at {vmrun_path}") + return CheckResult("vmrun", True, "backend loaded") + + +def _check_snapshots(config: Config) -> CheckResult: + """Each known template VMX exists and exposes its expected snapshot.""" + try: + backend = load_backend(config) + except BackendError as exc: + return CheckResult("snapshots", False, f"backend unavailable: {exc}") + + templates_dir = config.paths.templates + problems: list[str] = [] + found: list[str] = [] + for template_name, snapshot in _TEMPLATE_SNAPSHOTS: + vmx = templates_dir / template_name / f"{template_name}.vmx" + if not vmx.is_file(): + # A host may run only a subset of templates; skip absent ones + # rather than failing on them. + continue + try: + snapshots = backend.list_snapshots(VmHandle(identifier=str(vmx))) + except BackendError as exc: + problems.append(f"{template_name}: listSnapshots failed ({exc})") + continue + if snapshot in snapshots: + found.append(f"{template_name}->{snapshot}") + else: + problems.append( + f"{template_name}: snapshot {snapshot!r} absent " + f"(have: {', '.join(snapshots) or 'none'})" + ) + + if problems: + return CheckResult("snapshots", False, "; ".join(problems)) + if not found: + return CheckResult( + "snapshots", False, f"no known template VMX found under {templates_dir}" + ) + return CheckResult("snapshots", True, ", ".join(found)) + + +def _check_keyring(config: Config) -> CheckResult: + """The file-based keyring vault is readable for the guest cred target.""" + target = config.guest_cred_target + try: + cred = KeyringCredentialStore().get(target) + except KeyError: + return CheckResult( + "keyring", False, f"credential {target!r} not found in keyring" + ) + except RuntimeError as exc: + return CheckResult("keyring", False, str(exc)) + return CheckResult("keyring", True, f"{target!r} readable (user={cred.username})") + + +def _check_permissions(config: Config) -> CheckResult: + """``$CI_ROOT`` and key sub-dirs exist and are readable+writable.""" + root = config.paths.root + to_check = [root, config.paths.build_vms, config.paths.artifacts] + problems: list[str] = [] + for path in to_check: + if not path.exists(): + problems.append(f"{path}: missing") + continue + if not os.access(path, os.R_OK | os.W_OK): + problems.append(f"{path}: not read/write for current user") + if problems: + return CheckResult("permissions", False, "; ".join(problems)) + return CheckResult("permissions", True, f"{root} read/write") + + +def _systemctl_is_active(unit: str) -> str | None: + """Return ``systemctl is-active`` state, or ``None`` if unavailable.""" + systemctl = shutil.which("systemctl") + if systemctl is None: + return None + try: + result = subprocess.run( + [systemctl, "is-active", unit], + capture_output=True, + text=True, + check=False, + timeout=10.0, + ) + except (subprocess.TimeoutExpired, OSError): + return None + return result.stdout.strip().lower() or None + + +def _check_units() -> CheckResult: + """Expected systemd units report ``active``.""" + systemctl = shutil.which("systemctl") + if systemctl is None: + return CheckResult( + "systemd", False, "systemctl not found (not a systemd host?)" + ) + inactive: list[str] = [] + active: list[str] = [] + for unit in _EXPECTED_UNITS: + state = _systemctl_is_active(unit) + if state == "active": + active.append(unit) + else: + inactive.append(f"{unit}={state or 'unknown'}") + if inactive: + return CheckResult("systemd", False, "; ".join(inactive)) + return CheckResult("systemd", True, ", ".join(active)) + + +def run_host_checks(config: Config) -> list[CheckResult]: + """Run every host check and return the per-check results.""" + return [ + _check_vmrun(config), + _check_snapshots(config), + _check_keyring(config), + _check_permissions(config), + _check_units(), + ] + + +@validate.command("host") +def validate_host() -> None: + """Check host prerequisites; exit non-zero on any failure.""" + config = load_config() + results = run_host_checks(config) + failed = [r for r in results if not r.ok] + for result in results: + marker = "OK " if result.ok else "FAIL" + click.echo(f"[{marker}] {result.name}: {result.detail}") + if failed: + click.echo(f"\n{len(failed)} check(s) failed.") + raise SystemExit(1) + click.echo("\nAll host checks passed.") + + +# ──────────────────────────────────────────────────────── validate guest + +def _resolve_use_winrm(use_winrm: bool | None, guest_os: str | None) -> bool: + """Decide the transport: explicit flag wins, else infer from guest OS.""" + if use_winrm is not None: + return use_winrm + if guest_os is not None: + return guest_os.lower() != "linux" + raise click.UsageError( + "specify the transport: pass --winrm/--ssh or --guest-os." + ) + + +def _probe_winrm(host: str, config: Config) -> CheckResult: + """Connect over WinRM and run a trivial command, reporting the cause.""" + try: + cred = KeyringCredentialStore().get(config.guest_cred_target) + except (KeyError, RuntimeError) as exc: + return CheckResult("winrm", False, f"credential lookup failed: {exc}") + transport = WinRmTransport(host, cred.username, cred.password) + try: + result = transport.run("$true | Out-Null", check=True) + except TransportConnectError as exc: + return CheckResult("winrm", False, f"connect/auth failed: {exc}") + except TransportCommandError as exc: + return CheckResult("winrm", False, f"command failed: {exc}") + finally: + transport.close() + return CheckResult("winrm", True, f"connected; exit={result.returncode}") + + +def _probe_ssh(host: str, config: Config) -> CheckResult: + """Connect over SSH and run a trivial command, reporting the cause.""" + key_path = str(config.ssh_key_path) if config.ssh_key_path else None + transport = SshTransport(host, key_path=key_path) + try: + result = transport.run("true", check=True, timeout=10.0) + except TransportConnectError as exc: + return CheckResult("ssh", False, f"connect/auth failed: {exc}") + except TransportCommandError as exc: + return CheckResult("ssh", False, f"command failed: {exc}") + finally: + transport.close() + return CheckResult("ssh", True, f"connected; exit={result.returncode}") + + +@validate.command("guest") +@click.option("--host", "host", required=True, help="Guest IP or hostname.") +@click.option( + "--winrm/--ssh", + "use_winrm", + default=None, + help="Force transport (default: inferred from --guest-os).", +) +@click.option( + "--guest-os", + type=click.Choice(["windows", "linux"], case_sensitive=False), + default=None, + help="Guest OS family (used to pick transport when --winrm/--ssh omitted).", +) +def validate_guest( + host: str, + use_winrm: bool | None, + guest_os: str | None, +) -> None: + """Probe a guest's transport and report the explicit failure cause.""" + config = load_config() + winrm = _resolve_use_winrm(use_winrm, guest_os) + result = _probe_winrm(host, config) if winrm else _probe_ssh(host, config) + marker = "OK " if result.ok else "FAIL" + click.echo(f"[{marker}] {result.name} {host}: {result.detail}") + if not result.ok: + raise SystemExit(1) + + +__all__ = ["validate"] diff --git a/tests/_Common.Tests.ps1 b/tests/_Common.Tests.ps1 deleted file mode 100644 index 857bfdc..0000000 --- a/tests/_Common.Tests.ps1 +++ /dev/null @@ -1,181 +0,0 @@ -#Requires -Version 5.1 -<# -.SYNOPSIS - Pester v5 tests for scripts/_Common.psm1 -#> - -BeforeAll { - Import-Module (Join-Path $PSScriptRoot '..\scripts\_Common.psm1') -Force - - # Fake vmrun: accepts any args, exits with %FAKE_VMRUN_EXIT% (default 0) - $script:FakeVmrun = Join-Path $env:TEMP "fake-vmrun-$PID.cmd" - Set-Content $script:FakeVmrun -Value @' -@echo off -echo fake-vmrun-output -if "%FAKE_VMRUN_EXIT%"=="" ( exit /b 0 ) else ( exit /b %FAKE_VMRUN_EXIT% ) -'@ -} - -AfterAll { - Remove-Item $script:FakeVmrun -Force -ErrorAction SilentlyContinue -} - -# ───────────────────────────────────────────────────────────────────────────── -Describe 'New-CISessionOption' { - It 'returns a PSSessionOption' { - $opt = New-CISessionOption - $opt | Should -BeOfType [System.Management.Automation.Remoting.PSSessionOption] - } - - It 'has SkipCACheck = true' { - (New-CISessionOption).SkipCACheck | Should -Be $true - } - - It 'has SkipCNCheck = true' { - (New-CISessionOption).SkipCNCheck | Should -Be $true - } - - It 'has SkipRevocationCheck = true' { - (New-CISessionOption).SkipRevocationCheck | Should -Be $true - } -} - -# ───────────────────────────────────────────────────────────────────────────── -Describe 'Resolve-VmrunPath' { - It 'returns the path when the file exists' { - $tmp = [System.IO.Path]::GetTempFileName() - try { - Resolve-VmrunPath -VmrunPath $tmp | Should -Be $tmp - } finally { - Remove-Item $tmp -Force -ErrorAction SilentlyContinue - } - } - - It 'throws with a descriptive message when file is missing' { - { Resolve-VmrunPath -VmrunPath 'C:\DoesNotExist\vmrun.exe' } | - Should -Throw -ExpectedMessage '*vmrun.exe not found*' - } -} - -# ───────────────────────────────────────────────────────────────────────────── -Describe 'Invoke-Vmrun' { - It 'returns ExitCode 0 and Output when command succeeds' { - $env:FAKE_VMRUN_EXIT = '0' - $r = Invoke-Vmrun -VmrunPath $script:FakeVmrun -Operation 'list' - $r.ExitCode | Should -Be 0 - "$($r.Output)" | Should -Match 'fake-vmrun-output' - } - - It 'returns non-zero ExitCode without throwing by default' { - $env:FAKE_VMRUN_EXIT = '1' - $r = Invoke-Vmrun -VmrunPath $script:FakeVmrun -Operation 'clone' - $r.ExitCode | Should -Be 1 - } - - It 'throws when -ThrowOnError and ExitCode is non-zero' { - $env:FAKE_VMRUN_EXIT = '1' - { Invoke-Vmrun -VmrunPath $script:FakeVmrun -Operation 'clone' -ThrowOnError } | - Should -Throw -ExpectedMessage '*clone failed*' - } - - It 'does not throw when -ThrowOnError and ExitCode is 0' { - $env:FAKE_VMRUN_EXIT = '0' - { Invoke-Vmrun -VmrunPath $script:FakeVmrun -Operation 'start' -ThrowOnError } | - Should -Not -Throw - } - - AfterEach { - $env:FAKE_VMRUN_EXIT = $null - } -} - -# ───────────────────────────────────────────────────────────────────────────── -Describe 'Invoke-VmrunBounded' { - BeforeAll { - # Fake vmrun that sleeps for $FAKE_VMRUN_SLEEP seconds then exits with - # $FAKE_VMRUN_EXIT. Both default to 0 if unset. - $script:SlowFakeVmrun = Join-Path $env:TEMP "fake-vmrun-slow-$PID.cmd" - Set-Content $script:SlowFakeVmrun -Value @' -@echo off -if "%FAKE_VMRUN_SLEEP%"=="" goto :nosleep -ping -n %FAKE_VMRUN_SLEEP% 127.0.0.1 >nul -:nosleep -echo fake-vmrun-bounded-output -if "%FAKE_VMRUN_EXIT%"=="" ( exit /b 0 ) else ( exit /b %FAKE_VMRUN_EXIT% ) -'@ - } - - AfterAll { - Remove-Item $script:SlowFakeVmrun -Force -ErrorAction SilentlyContinue - } - - AfterEach { - $env:FAKE_VMRUN_EXIT = $null - $env:FAKE_VMRUN_SLEEP = $null - } - - It 'returns ExitCode 0, TimedOut false and Output when command succeeds quickly' { - $env:FAKE_VMRUN_EXIT = '0' - $env:FAKE_VMRUN_SLEEP = '0' - $r = Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'start' -Arguments @('some.vmx', 'nogui') -TimeoutSeconds 30 - $r.ExitCode | Should -Be 0 - $r.TimedOut | Should -Be $false - "$($r.Output)" | Should -Match 'fake-vmrun-bounded-output' - $r.ElapsedSeconds | Should -BeGreaterOrEqual 0 - } - - It 'returns non-zero ExitCode without throwing by default' { - $env:FAKE_VMRUN_EXIT = '3' - $env:FAKE_VMRUN_SLEEP = '0' - $r = Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'deleteVM' -TimeoutSeconds 30 - $r.ExitCode | Should -Be 3 - $r.TimedOut | Should -Be $false - } - - It 'throws when -ThrowOnError and ExitCode is non-zero' { - $env:FAKE_VMRUN_EXIT = '2' - $env:FAKE_VMRUN_SLEEP = '0' - { Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'stop' -TimeoutSeconds 30 -ThrowOnError } | - Should -Throw -ExpectedMessage '*stop failed*' - } - - It 'does not throw when -ThrowOnError and ExitCode is 0' { - $env:FAKE_VMRUN_EXIT = '0' - $env:FAKE_VMRUN_SLEEP = '0' - { Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'start' -TimeoutSeconds 30 -ThrowOnError } | - Should -Not -Throw - } - - It 'times out and returns TimedOut = true when process exceeds TimeoutSeconds' { - # FAKE_VMRUN_SLEEP uses ping -n N which waits roughly (N-1) seconds. - # Set sleep >> timeout so the process is guaranteed to still be running. - $env:FAKE_VMRUN_SLEEP = '60' - $env:FAKE_VMRUN_EXIT = '0' - $r = Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'start' -TimeoutSeconds 2 - $r.TimedOut | Should -Be $true - $r.ExitCode | Should -Be -1 - $r.ElapsedSeconds | Should -BeLessOrEqual 10 # killed well before 60s - } - - It 'throws when -ThrowOnTimeout and process times out' { - $env:FAKE_VMRUN_SLEEP = '60' - $env:FAKE_VMRUN_EXIT = '0' - { Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'deleteVM' -TimeoutSeconds 2 -ThrowOnTimeout } | - Should -Throw -ExpectedMessage '*timed out*' - } - - It 'does not throw on timeout when neither flag is set' { - $env:FAKE_VMRUN_SLEEP = '60' - $env:FAKE_VMRUN_EXIT = '0' - { Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'start' -TimeoutSeconds 2 } | - Should -Not -Throw - } - - It 'ElapsedSeconds is populated and plausible' { - $env:FAKE_VMRUN_EXIT = '0' - $env:FAKE_VMRUN_SLEEP = '0' - $r = Invoke-VmrunBounded -VmrunPath $script:SlowFakeVmrun -Operation 'list' -TimeoutSeconds 30 - $r.ElapsedSeconds | Should -BeGreaterOrEqual 0 - $r.ElapsedSeconds | Should -BeLessThan 30 - } -} diff --git a/tests/python/test_commands_bench.py b/tests/python/test_commands_bench.py new file mode 100644 index 0000000..b85c87a --- /dev/null +++ b/tests/python/test_commands_bench.py @@ -0,0 +1,487 @@ +"""Tests for the ``bench`` sub-commands (Phase C2 burn-in + C3 measure). + +The C1 ``--help`` smoke tests are kept; behavioural tests mock the backend +and the per-job runner so no real VMs are spun up. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest +from click.testing import CliRunner + +import ci_orchestrator.commands.bench as bench_module +from ci_orchestrator.__main__ import cli +from ci_orchestrator.backends.errors import BackendNotAvailable, BackendOperationFailed +from ci_orchestrator.backends.protocol import VmHandle +from ci_orchestrator.commands.bench import ( + JobResult, + PhaseTiming, + _find_orphans, + _stale_lock, + bench, +) + +# ── C1 smoke tests (kept) ─────────────────────────────────────────────────── + + +def test_bench_help_lists_subcommands() -> None: + result = CliRunner().invoke(cli, ["bench", "--help"]) + assert result.exit_code == 0, result.output + assert "run" in result.output + assert "measure" in result.output + + +def test_bench_run_help() -> None: + result = CliRunner().invoke(cli, ["bench", "run", "--help"]) + assert result.exit_code == 0, result.output + assert "--concurrency" in result.output + assert "--rounds" in result.output + + +def test_bench_measure_help() -> None: + result = CliRunner().invoke(cli, ["bench", "measure", "--help"]) + assert result.exit_code == 0, result.output + assert "--iterations" in result.output + + +# ── helpers ───────────────────────────────────────────────────────────────── + + +def test_find_orphans_matches_job_prefix(tmp_path: Path) -> None: + (tmp_path / "Clone_burnin-r1-j1-x_20260101_aaa").mkdir() + (tmp_path / "Clone_other-job_20260101_bbb").mkdir() + (tmp_path / "not-a-clone").mkdir() + orphans = _find_orphans(tmp_path, ["burnin-r1-j1-x"]) + assert len(orphans) == 1 + assert orphans[0].name.startswith("Clone_burnin-r1-j1-x_") + + +def test_find_orphans_missing_base_dir(tmp_path: Path) -> None: + assert _find_orphans(tmp_path / "ghost", ["j1"]) == [] + + +def test_stale_lock_detects_old_file(tmp_path: Path) -> None: + import os + + lock = tmp_path / "vm-start.lock" + lock.write_text("x") + old = lock.stat().st_mtime - (10 * 60) + os.utime(lock, (old, old)) + assert _stale_lock(lock) is True + + +def test_stale_lock_fresh_file_is_ok(tmp_path: Path) -> None: + lock = tmp_path / "vm-start.lock" + lock.write_text("x") + assert _stale_lock(lock) is False + + +def test_stale_lock_missing_file(tmp_path: Path) -> None: + assert _stale_lock(tmp_path / "absent.lock") is False + + +# ── bench run ─────────────────────────────────────────────────────────────── + + +@pytest.fixture +def _ci_paths(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Path: + """Point load_config at temp build-vms/artifacts dirs.""" + env = { + "CI_BUILD_VMS": str(tmp_path / "build-vms"), + "CI_ARTIFACTS": str(tmp_path / "artifacts"), + } + (tmp_path / "build-vms").mkdir() + (tmp_path / "artifacts").mkdir() + real = bench_module.load_config + monkeypatch.setattr(bench_module, "load_config", lambda: real(env=env)) + return tmp_path + + +def test_bench_run_all_pass(monkeypatch: pytest.MonkeyPatch, _ci_paths: Path) -> None: + calls: list[str] = [] + + def _fake_job(*, job_id: str, **_kw: Any) -> tuple[int, str]: + calls.append(job_id) + return 0, "" + + monkeypatch.setattr(bench_module, "_run_one_job", _fake_job) + result = CliRunner().invoke( + cli, ["bench", "run", "--concurrency", "3", "--rounds", "2"] + ) + assert result.exit_code == 0, result.output + assert "OVERALL: PASS" in result.output + assert len(calls) == 6 # 3 jobs * 2 rounds + # Report file written under artifacts/bench. + reports = list((_ci_paths / "artifacts" / "bench").glob("*.json")) + assert len(reports) == 1 + data = json.loads(reports[0].read_text()) + assert data["overall"] == "PASS" + assert data["totalJobs"] == 6 + assert len(data["results"]) == 2 + + +def test_bench_run_failing_job_marks_round_fail( + monkeypatch: pytest.MonkeyPatch, _ci_paths: Path +) -> None: + def _fake_job(*, job_id: str, slot: int = 0, **_kw: Any) -> tuple[int, str]: + return 1, "clone failed: boom" + + monkeypatch.setattr(bench_module, "_run_one_job", _fake_job) + result = CliRunner().invoke( + cli, ["bench", "run", "--concurrency", "1", "--rounds", "1"] + ) + assert result.exit_code == 1 + assert "OVERALL: FAIL" in result.output + assert "clone failed: boom" in result.output + + +def test_bench_run_orphan_dir_fails_round( + monkeypatch: pytest.MonkeyPatch, _ci_paths: Path +) -> None: + base = _ci_paths / "build-vms" + + def _fake_job(*, job_id: str, **_kw: Any) -> tuple[int, str]: + # Leave an orphan clone dir behind for this job. + (base / f"Clone_{job_id}_20260101_orphan").mkdir() + return 0, "" + + monkeypatch.setattr(bench_module, "_run_one_job", _fake_job) + result = CliRunner().invoke( + cli, ["bench", "run", "--concurrency", "1", "--rounds", "1"] + ) + assert result.exit_code == 1 + assert "orphan" in result.output + assert "OVERALL: FAIL" in result.output + + +def test_bench_run_stale_lock_fails_round( + monkeypatch: pytest.MonkeyPatch, _ci_paths: Path +) -> None: + base = _ci_paths / "build-vms" + monkeypatch.setattr(bench_module, "_run_one_job", lambda **_kw: (0, "")) + # _stale_lock reports a stale lock regardless of file age. + monkeypatch.setattr(bench_module, "_stale_lock", lambda _p: True) + result = CliRunner().invoke( + cli, + ["bench", "run", "--concurrency", "1", "--rounds", "1", "--clone-base-dir", str(base)], + ) + assert result.exit_code == 1 + assert "stale lock" in result.output + + +def test_bench_run_custom_json_out( + monkeypatch: pytest.MonkeyPatch, _ci_paths: Path +) -> None: + out = _ci_paths / "custom" / "report.json" + monkeypatch.setattr(bench_module, "_run_one_job", lambda **_kw: (0, "")) + result = CliRunner().invoke( + cli, + ["bench", "run", "--concurrency", "1", "--rounds", "1", "--json-out", str(out)], + ) + assert result.exit_code == 0, result.output + assert out.is_file() + + +def test_run_one_job_invokes_job_command(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + """_run_one_job drives the real job command and maps exit codes.""" + import ci_orchestrator.commands.job as job_module + + captured: dict[str, Any] = {} + + class _FakeResult: + exit_code = 0 + output = "ok\n" + + class _FakeRunner: + def invoke(self, _cmd: Any, args: list[str], **_kw: Any) -> _FakeResult: + captured["args"] = args + return _FakeResult() + + monkeypatch.setattr("click.testing.CliRunner", lambda *a, **k: _FakeRunner()) + del job_module # imported for symmetry; not otherwise needed + code, err = bench_module._run_one_job( + job_id="j1", + guest_os="linux", + build_command="echo hi", + clone_base_dir=tmp_path, + ) + assert code == 0 + assert err == "" + assert "--job-id" in captured["args"] + assert "--build-command" in captured["args"] + + +def test_run_one_job_failure_returns_tail(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + class _FakeResult: + exit_code = 1 + output = "line one\nError: clone failed\n" + + class _FakeRunner: + def invoke(self, *_a: Any, **_k: Any) -> _FakeResult: + return _FakeResult() + + monkeypatch.setattr("click.testing.CliRunner", lambda *a, **k: _FakeRunner()) + code, err = bench_module._run_one_job( + job_id="j1", guest_os="linux", build_command="", clone_base_dir=tmp_path + ) + assert code == 1 + assert err == "Error: clone failed" + + +# ── bench measure ─────────────────────────────────────────────────────────── + + +class _FakeBackend: + def __init__(self, *, ip: str | None = "10.0.0.5", fail: str = "") -> None: + self._ip = ip + self._fail = fail + self.calls: list[str] = [] + + def clone_linked( + self, template: str, snapshot: str, name: str, destination: str | None = None + ) -> VmHandle: + self.calls.append("clone") + if destination: + d = Path(destination) + d.parent.mkdir(parents=True, exist_ok=True) + d.write_text('guestOS = "ubuntu-64"', encoding="utf-8") + if self._fail == "clone": + raise BackendOperationFailed("clone", 1, "no") + return VmHandle(identifier=destination or name, name=name) + + def start(self, handle: VmHandle, headless: bool = True) -> None: + self.calls.append("start") + + def get_ip(self, handle: VmHandle, timeout: float = 0.0) -> str | None: + self.calls.append("get_ip") + return self._ip + + def stop(self, handle: VmHandle, hard: bool = False) -> None: + self.calls.append("stop") + + def delete(self, handle: VmHandle) -> None: + self.calls.append("delete") + + +def _make_template(tmp_path: Path) -> Path: + tpl = tmp_path / "tpl" / "LinuxBuild2404.vmx" + tpl.parent.mkdir(parents=True) + tpl.write_text('guestOS = "ubuntu-64"', encoding="utf-8") + return tpl + + +@pytest.fixture +def _measure_paths(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Path: + env = { + "CI_BUILD_VMS": str(tmp_path / "build-vms"), + "CI_ARTIFACTS": str(tmp_path / "artifacts"), + } + real = bench_module.load_config + monkeypatch.setattr(bench_module, "load_config", lambda: real(env=env)) + return tmp_path + + +def test_bench_measure_happy_path( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + backend = _FakeBackend(ip="10.0.0.5") + monkeypatch.setattr(bench_module, "load_backend", lambda _c: backend) + # transport ready immediately + monkeypatch.setattr(bench_module, "_tcp_open", lambda *a, **k: True) + + result = CliRunner().invoke( + cli, + [ + "bench", + "measure", + "--guest-os", + "linux", + "--iterations", + "2", + "--template-path", + str(tpl), + "--timeout-seconds", + "30", + ], + ) + assert result.exit_code == 0, result.output + assert "clone" in backend.calls + assert "delete" in backend.calls + jsonl = _measure_paths / "artifacts" / "benchmark.jsonl" + assert jsonl.is_file() + records = [json.loads(line) for line in jsonl.read_text().splitlines()] + assert len(records) == 2 + # Legacy field names preserved. + assert set(records[0]) >= { + "ts", "runId", "iteration", "template", "snapshot", "guestOS", + "cloneSec", "startSec", "ipSec", "readySec", "destroySec", + "totalBootSec", "deltaKB", "vmIP", "error", + } + assert records[0]["vmIP"] == "10.0.0.5" + assert records[0]["error"] is None + assert "Average boot-to-ready" in result.output + + +def test_bench_measure_ip_timeout_records_error( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + backend = _FakeBackend(ip=None) # never returns an IP + monkeypatch.setattr(bench_module, "load_backend", lambda _c: backend) + # Make the IP-wait loop terminate fast: a monotonic clock that jumps + # 100s per call, so any deadline is exceeded within a couple of polls. + counter = {"t": 0.0} + + def _fast_now() -> float: + counter["t"] += 100.0 + return counter["t"] + + monkeypatch.setattr(bench_module, "_now", _fast_now) + + result = CliRunner().invoke( + cli, + ["bench", "measure", "--template-path", str(tpl), "--timeout-seconds", "30"], + ) + assert result.exit_code == 0, result.output + jsonl = _measure_paths / "artifacts" / "benchmark.jsonl" + rec = json.loads(jsonl.read_text().splitlines()[0]) + assert rec["error"] is not None + assert rec["readySec"] is None + # destroy still ran (finally block). + assert "delete" in backend.calls + + +def test_bench_measure_template_missing( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + result = CliRunner().invoke( + cli, ["bench", "measure", "--template-path", str(_measure_paths / "nope.vmx")] + ) + assert result.exit_code != 0 + assert "Template VMX not found" in result.output + + +def test_bench_measure_backend_unavailable( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + + def _raise(_c: Any) -> Any: + raise BackendNotAvailable("no vmrun") + + monkeypatch.setattr(bench_module, "load_backend", _raise) + result = CliRunner().invoke( + cli, ["bench", "measure", "--template-path", str(tpl)] + ) + assert result.exit_code != 0 + assert "backend unavailable" in result.output + + +def test_bench_measure_clone_failure_records_error( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + backend = _FakeBackend(fail="clone") + monkeypatch.setattr(bench_module, "load_backend", lambda _c: backend) + result = CliRunner().invoke( + cli, ["bench", "measure", "--template-path", str(tpl), "--timeout-seconds", "30"] + ) + assert result.exit_code == 0, result.output + jsonl = _measure_paths / "artifacts" / "benchmark.jsonl" + rec = json.loads(jsonl.read_text().splitlines()[0]) + assert rec["error"] is not None + assert rec["cloneSec"] is None + + +def test_bench_measure_transport_port_timeout( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + backend = _FakeBackend(ip="10.0.0.9") + monkeypatch.setattr(bench_module, "load_backend", lambda _c: backend) + # IP is acquired immediately, but the transport port never opens. + monkeypatch.setattr(bench_module, "_wait_port", lambda *a, **k: False) + result = CliRunner().invoke( + cli, ["bench", "measure", "--template-path", str(tpl), "--timeout-seconds", "30"] + ) + assert result.exit_code == 0, result.output + jsonl = _measure_paths / "artifacts" / "benchmark.jsonl" + rec = json.loads(jsonl.read_text().splitlines()[0]) + assert rec["readySec"] is None + assert "port" in (rec["error"] or "") + + +def test_bench_measure_destroy_suppresses_backend_errors( + monkeypatch: pytest.MonkeyPatch, _measure_paths: Path +) -> None: + tpl = _make_template(_measure_paths) + + class _FlakyBackend(_FakeBackend): + def stop(self, handle: VmHandle, hard: bool = False) -> None: + self.calls.append("stop") + raise BackendOperationFailed("stop", 1, "no") + + def delete(self, handle: VmHandle) -> None: + self.calls.append("delete") + raise BackendOperationFailed("delete", 1, "no") + + backend = _FlakyBackend(ip="10.0.0.5") + monkeypatch.setattr(bench_module, "load_backend", lambda _c: backend) + monkeypatch.setattr(bench_module, "_tcp_open", lambda *a, **k: True) + result = CliRunner().invoke( + cli, ["bench", "measure", "--template-path", str(tpl), "--timeout-seconds", "30"] + ) + # stop/delete raising must not crash the command. + assert result.exit_code == 0, result.output + assert "stop" in backend.calls + assert "delete" in backend.calls + + +def test_bench_run_report_write_failure_is_tolerated( + monkeypatch: pytest.MonkeyPatch, _ci_paths: Path +) -> None: + monkeypatch.setattr(bench_module, "_run_one_job", lambda **_kw: (0, "")) + + def _boom(_report: Any, _path: Any) -> None: + raise OSError("disk full") + + monkeypatch.setattr(bench_module, "_write_run_report", _boom) + result = CliRunner().invoke( + cli, ["bench", "run", "--concurrency", "1", "--rounds", "1"] + ) + assert result.exit_code == 0, result.output + assert "could not write report" in result.output + + +def test_bench_measure_default_template_windows(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + """--guest-os windows resolves the WinBuild default template + snapshot.""" + from ci_orchestrator.config import load_config as real_load + + cfg = real_load(env={"CI_ROOT": str(tmp_path)}) + tpl, snap = bench_module._resolve_template(cfg, "windows") + assert tpl.name == "WinBuild2025.vmx" + assert snap == "BaseClean" + tpl_l, snap_l = bench_module._resolve_template(cfg, "linux") + assert tpl_l.name == "LinuxBuild2404.vmx" + assert snap_l == "BaseClean-Linux" + + +# ── dataclass behaviour ───────────────────────────────────────────────────── + + +def test_phase_timing_total_boot_none_when_incomplete() -> None: + t = PhaseTiming(iteration=1, run_id="r", template="t", snapshot="s", guest_os="linux") + assert t.total_boot_sec is None + t.clone_sec, t.start_sec, t.ip_sec, t.ready_sec = 1.0, 2.0, 3.0, 4.0 + assert t.total_boot_sec == 10.0 + + +def test_job_result_dataclass_defaults() -> None: + jr = JobResult(job_id="j", slot=1, exit_code=0, status="PASS", elapsed_sec=1.0) + assert jr.error == "" + assert bench is not None diff --git a/tests/python/test_commands_creds.py b/tests/python/test_commands_creds.py new file mode 100644 index 0000000..dda562f --- /dev/null +++ b/tests/python/test_commands_creds.py @@ -0,0 +1,184 @@ +"""Tests for ``creds set`` (C6). + +CLI registration smoke tests (``--help``) are kept; behavioural tests mock +the ``keyring`` module and include a round-trip via ``KeyringCredentialStore``. +""" + +from __future__ import annotations + +import sys +import types +from typing import Any + +import pytest +from click.testing import CliRunner + +from ci_orchestrator.__main__ import cli +from ci_orchestrator.credentials import KeyringCredentialStore + +# ── --help smoke tests (kept) ─────────────────────────────────────────────── + + +def test_creds_help_lists_set() -> None: + result = CliRunner().invoke(cli, ["creds", "--help"]) + assert result.exit_code == 0, result.output + assert "set" in result.output + + +def test_creds_set_help() -> None: + result = CliRunner().invoke(cli, ["creds", "set", "--help"]) + assert result.exit_code == 0, result.output + assert "--target" in result.output + assert "--user" in result.output + + +# ── fake keyring ──────────────────────────────────────────────────────────── + + +def _install_fake_keyring(monkeypatch: pytest.MonkeyPatch) -> dict[tuple[str, str], str]: + """Install a fake ``keyring`` module with a two-key password store. + + Returns the backing dict so tests can inspect what was written. The fake + implements both ``set_password`` (writer) and ``get_password`` / + ``get_credential`` (reader), so round-trips work end to end. + """ + store: dict[tuple[str, str], str] = {} + fake = types.ModuleType("keyring") + + def set_password(service: str, key: str, password: str) -> None: + store[(service, key)] = password + + def get_password(service: str, key: str) -> str | None: + return store.get((service, key)) + + def get_credential(service: str, _username: str | None) -> Any: + # File backends are a no-op here (mirrors PlaintextKeyring); the reader + # falls through to the two-entry scheme. + return None + + fake.set_password = set_password # type: ignore[attr-defined] + fake.get_password = get_password # type: ignore[attr-defined] + fake.get_credential = get_credential # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "keyring", fake) + return store + + +# ── write scheme ──────────────────────────────────────────────────────────── + + +def test_creds_set_writes_two_entries(monkeypatch: pytest.MonkeyPatch) -> None: + store = _install_fake_keyring(monkeypatch) + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "BuildVMGuest", "--user", "ci_build", + "--password-stdin"], + input="s3cret\n", + ) + assert result.exit_code == 0, result.output + # username under "{target}:meta" + assert store[("BuildVMGuest:meta", "username")] == "ci_build" + # password under target keyed by username + assert store[("BuildVMGuest", "ci_build")] == "s3cret" + + +def test_creds_set_round_trips_via_store(monkeypatch: pytest.MonkeyPatch) -> None: + """Round-trip: write with `creds set`, read back with the reader.""" + _install_fake_keyring(monkeypatch) + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "BuildVMGuest", "--user", "WINBUILD\\ci", + "--password-stdin"], + input="p@ss word!\n", + ) + assert result.exit_code == 0, result.output + cred = KeyringCredentialStore().get("BuildVMGuest") + assert cred.username == "WINBUILD\\ci" + assert cred.password == "p@ss word!" + + +def test_creds_set_default_target(monkeypatch: pytest.MonkeyPatch) -> None: + store = _install_fake_keyring(monkeypatch) + result = CliRunner().invoke( + cli, + ["creds", "set", "--user", "ci_build", "--password-stdin"], + input="abc\n", + ) + assert result.exit_code == 0, result.output + assert store[("BuildVMGuest:meta", "username")] == "ci_build" + + +# ── password sourcing ─────────────────────────────────────────────────────── + + +def test_creds_set_prompts_when_no_stdin_flag( + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = _install_fake_keyring(monkeypatch) + # click.prompt with confirmation_prompt reads the value twice. + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "T", "--user", "u"], + input="hunter2\nhunter2\n", + ) + assert result.exit_code == 0, result.output + assert store[("T", "u")] == "hunter2" + + +def test_creds_set_empty_password_rejected( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_fake_keyring(monkeypatch) + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "T", "--user", "u", "--password-stdin"], + input="\n", + ) + assert result.exit_code != 0 + assert "empty" in result.output.lower() + + +def test_creds_set_strips_trailing_newline( + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = _install_fake_keyring(monkeypatch) + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "T", "--user", "u", "--password-stdin"], + input="secret\r\n", + ) + assert result.exit_code == 0, result.output + assert store[("T", "u")] == "secret" + + +def test_creds_set_never_accepts_password_option() -> None: + """The CLI must NOT expose a --password option.""" + result = CliRunner().invoke(cli, ["creds", "set", "--help"]) + assert "--password " not in result.output + assert "--password=" not in result.output + + +def test_creds_set_keyring_missing(monkeypatch: pytest.MonkeyPatch) -> None: + """When keyring import fails, surface a clean ClickException.""" + import ci_orchestrator.commands.creds as creds_module + + def _raise(_t: str, _u: str, _p: str) -> None: + raise RuntimeError("keyring is not installed; run `pip install keyring`.") + + monkeypatch.setattr(creds_module, "_store_credential", _raise) + result = CliRunner().invoke( + cli, + ["creds", "set", "--target", "T", "--user", "u", "--password-stdin"], + input="x\n", + ) + assert result.exit_code != 0 + assert "keyring is not installed" in result.output + + +def test_store_credential_invokes_keyring(monkeypatch: pytest.MonkeyPatch) -> None: + """Unit-level: _store_credential writes both entries to the module.""" + import ci_orchestrator.commands.creds as creds_module + + store = _install_fake_keyring(monkeypatch) + creds_module._store_credential("Tgt", "user1", "pw1") + assert store[("Tgt:meta", "username")] == "user1" + assert store[("Tgt", "user1")] == "pw1" diff --git a/tests/python/test_commands_smoke.py b/tests/python/test_commands_smoke.py new file mode 100644 index 0000000..b16ca62 --- /dev/null +++ b/tests/python/test_commands_smoke.py @@ -0,0 +1,286 @@ +"""Tests for ``ci_orchestrator smoke`` (Phase C1 scaffolding + C4 behaviour).""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +import pytest +from click.testing import CliRunner + +import ci_orchestrator.commands.smoke as smoke_module +from ci_orchestrator.__main__ import cli +from ci_orchestrator.config import BackendConfig, Config, Paths + +# ──────────────────────────────────────────────────────── C1 --help smoke + + +def test_smoke_help_lists_run() -> None: + result = CliRunner().invoke(cli, ["smoke", "--help"]) + assert result.exit_code == 0, result.output + assert "run" in result.output + + +def test_smoke_run_help() -> None: + result = CliRunner().invoke(cli, ["smoke", "run", "--help"]) + assert result.exit_code == 0, result.output + assert "--guest-os" in result.output + + +# ──────────────────────────────────────────────────────────── helpers + + +def _patch_config(monkeypatch: pytest.MonkeyPatch, artifacts: Path) -> None: + paths = Paths( + root=artifacts.parent, + templates=artifacts.parent / "templates", + build_vms=artifacts.parent / "build-vms", + artifacts=artifacts, + keys=artifacts.parent / "keys", + ) + cfg = Config(paths=paths, backend=BackendConfig(), ip_pool=None) + monkeypatch.setattr(smoke_module, "load_config", lambda: cfg) + + +def _patch_job( + monkeypatch: pytest.MonkeyPatch, + *, + artifacts: Path, + make_artifact: bool = True, + raises: bool = False, +) -> dict[str, list[Any]]: + """Replace the in-process ``job`` command with a recording fake.""" + calls: dict[str, list[Any]] = {"job": []} + + def _fake_callback(**kwargs: Any) -> None: + calls["job"].append(kwargs) + if make_artifact: + job_dir = artifacts / kwargs["job_id"] + job_dir.mkdir(parents=True, exist_ok=True) + (job_dir / "smoke.txt").write_text("smoke-ok", encoding="utf-8") + if raises: + import click + + raise click.ClickException("clone failed: boom") + + # ``job_command`` is a click.Command; patch its callback. + monkeypatch.setattr(smoke_module.job_command, "callback", _fake_callback) + return calls + + +def _read_events(jsonl_path: Path) -> list[dict[str, Any]]: + out: list[dict[str, Any]] = [] + for line in jsonl_path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if line: + out.append(json.loads(line)) + return out + + +# ──────────────────────────────────────────────────────── C4 behaviour + + +def test_smoke_run_linux_happy_path( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--guest-os", "linux", "--job-id", "smoke-1"] + ) + assert result.exit_code == 0, result.output + assert "PASSED" in result.output + + # The job ran once with the Linux no-op marker preset. + assert len(calls["job"]) == 1 + kw = calls["job"][0] + assert kw["guest_os"] == "linux" + assert kw["job_id"] == "smoke-1" + assert "smoke.txt" in kw["build_command"] + assert kw["guest_artifact_source"] == "/opt/ci/output" + + # Artifact dir was created and the success event is present. + assert (artifacts / "smoke-1").is_dir() + events = _read_events(tmp_path / "logs" / "smoke-1" / "invoke-ci.jsonl") + assert any(e["phase"] == "job" and e["status"] == "start" for e in events) + assert any(e["phase"] == "job" and e["status"] == "success" for e in events) + + +def test_smoke_run_windows_uses_windows_preset( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--guest-os", "windows", "--job-id", "smoke-win"] + ) + assert result.exit_code == 0, result.output + kw = calls["job"][0] + assert kw["guest_os"] == "windows" + assert kw["guest_artifact_source"] == "C:\\CI\\output" + assert "C:\\CI\\output" in kw["build_command"] + + +def test_smoke_run_failure_when_job_raises( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + _patch_job(monkeypatch, artifacts=artifacts, make_artifact=False, raises=True) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--guest-os", "linux", "--job-id", "smoke-fail"] + ) + assert result.exit_code != 0 + assert "FAILED" in result.output + # A failure event was logged and there is no success event. + events = _read_events(tmp_path / "logs" / "smoke-fail" / "invoke-ci.jsonl") + assert any(e["status"] == "failure" for e in events) + assert not any(e["status"] == "success" for e in events) + + +def test_smoke_run_fails_when_artifact_missing( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + # Job "succeeds" but produces no artifact dir → smoke must still fail. + _patch_job(monkeypatch, artifacts=artifacts, make_artifact=False) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--guest-os", "linux", "--job-id", "smoke-noart"] + ) + assert result.exit_code != 0 + assert "artifact dir not found" in result.output + + +def test_smoke_run_preset_ns7zip( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, + ["smoke", "run", "--guest-os", "linux", "--preset", "ns7zip", "--job-id", "p1"], + ) + assert result.exit_code == 0, result.output + kw = calls["job"][0] + assert "build_plugin.py" in kw["build_command"] + assert "7zip-version" in kw["build_command"] + assert kw["guest_artifact_source"] == "plugins" + + +def test_smoke_run_preset_nsinnounp( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, + ["smoke", "run", "--preset", "nsinnounp", "--job-id", "p2"], + ) + assert result.exit_code == 0, result.output + kw = calls["job"][0] + assert kw["build_command"] == "python3 build_plugin.py --final --dist-dir dist" + assert kw["guest_artifact_source"] == "dist" + + +def test_smoke_run_preset_rejected_on_windows( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--guest-os", "windows", "--preset", "ns7zip"] + ) + assert result.exit_code != 0 + assert "only supported with --guest-os linux" in result.output + + +def test_smoke_run_preset_and_build_command_mutually_exclusive( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, + [ + "smoke", "run", "--preset", "ns7zip", + "--build-command", "echo hi", + ], + ) + assert result.exit_code != 0 + assert "mutually exclusive" in result.output + + +def test_smoke_run_custom_build_command( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, + [ + "smoke", "run", "--guest-os", "linux", + "--build-command", "make all", + "--job-id", "custom-1", + ], + ) + assert result.exit_code == 0, result.output + kw = calls["job"][0] + assert kw["build_command"] == "make all" + # Custom command keeps the OS-default artifact source. + assert kw["guest_artifact_source"] == "/opt/ci/output" + + +def test_has_job_success_tolerates_malformed_jsonl(tmp_path: Path) -> None: + """Garbage/missing log lines must not raise and must report no success.""" + missing = tmp_path / "nope.jsonl" + assert smoke_module._has_job_success(missing) is False + + jsonl = tmp_path / "log.jsonl" + jsonl.write_text( + "\n" # blank line + "not json\n" # JSONDecodeError + "[1, 2, 3]\n" # valid JSON but not a dict + '{"phase": "job", "status": "start"}\n', # dict, no success + encoding="utf-8", + ) + assert smoke_module._has_job_success(jsonl) is False + + jsonl.write_text('{"phase": "job", "status": "success"}\n', encoding="utf-8") + assert smoke_module._has_job_success(jsonl) is True + + +def test_smoke_run_default_job_id_and_separate_log_dir( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + artifacts = tmp_path / "artifacts" + logs = tmp_path / "logs" + _patch_config(monkeypatch, artifacts) + calls = _patch_job(monkeypatch, artifacts=artifacts) + + result = CliRunner().invoke( + cli, ["smoke", "run", "--log-dir", str(logs)] + ) + assert result.exit_code == 0, result.output + generated_id = calls["job"][0]["job_id"] + assert generated_id.startswith("smoke-") + # JSONL written under the override log dir, artifacts under config dir. + assert (logs / generated_id / "invoke-ci.jsonl").is_file() + assert (artifacts / generated_id).is_dir() diff --git a/tests/python/test_commands_validate.py b/tests/python/test_commands_validate.py new file mode 100644 index 0000000..7e8ea52 --- /dev/null +++ b/tests/python/test_commands_validate.py @@ -0,0 +1,542 @@ +"""Tests for ``validate host`` (C5) and ``validate guest`` (C6). + +CLI registration smoke tests (``--help``) are kept; behavioural tests mock +the backend, keyring, systemctl, and transports. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest +from click.testing import CliRunner + +import ci_orchestrator.commands.validate as validate_module +from ci_orchestrator.__main__ import cli +from ci_orchestrator.backends.errors import ( + BackendNotAvailable, + BackendOperationFailed, +) +from ci_orchestrator.config import BackendConfig, Config, Paths +from ci_orchestrator.credentials import Credential +from ci_orchestrator.transport.errors import ( + TransportCommandError, + TransportConnectError, +) + +# ── --help smoke tests (kept) ─────────────────────────────────────────────── + + +def test_validate_help_lists_subcommands() -> None: + result = CliRunner().invoke(cli, ["validate", "--help"]) + assert result.exit_code == 0, result.output + assert "host" in result.output + assert "guest" in result.output + + +def test_validate_guest_help() -> None: + result = CliRunner().invoke(cli, ["validate", "guest", "--help"]) + assert result.exit_code == 0, result.output + assert "--host" in result.output + + +# ── helpers ───────────────────────────────────────────────────────────────── + + +def _make_config(tmp_path: Path) -> Config: + root = tmp_path / "ci" + templates = root / "templates" + build_vms = root / "build-vms" + artifacts = root / "artifacts" + keys = root / "keys" + for p in (root, templates, build_vms, artifacts, keys): + p.mkdir(parents=True, exist_ok=True) + return Config( + paths=Paths( + root=root, + templates=templates, + build_vms=build_vms, + artifacts=artifacts, + keys=keys, + ), + backend=BackendConfig(type="workstation"), + guest_cred_target="BuildVMGuest", + ) + + +class _FakeBackend: + def __init__(self, snapshots: dict[str, list[str]] | None = None) -> None: + self.vmrun_path = "/usr/bin/vmrun" + self._snapshots = snapshots or {} + + def list_snapshots(self, handle: Any) -> list[str]: + return self._snapshots.get(handle.identifier, []) + + +def _make_template_vmx(config: Config, name: str) -> str: + vmx_dir = config.paths.templates / name + vmx_dir.mkdir(parents=True, exist_ok=True) + vmx = vmx_dir / f"{name}.vmx" + vmx.write_text('config.version = "8"', encoding="utf-8") + return str(vmx) + + +# ── validate host: individual checks ──────────────────────────────────────── + + +def test_check_vmrun_ok(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + config = _make_config(tmp_path) + monkeypatch.setattr(validate_module, "load_backend", lambda _c: _FakeBackend()) + result = validate_module._check_vmrun(config) + assert result.ok + assert "vmrun" in result.detail + + +def test_check_vmrun_no_path_attr( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + class _NoPath: + pass + + monkeypatch.setattr(validate_module, "load_backend", lambda _c: _NoPath()) + result = validate_module._check_vmrun(config) + assert result.ok + assert "backend loaded" in result.detail + + +def test_check_vmrun_unavailable( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + def _raise(_c: Any) -> Any: + raise BackendNotAvailable("vmrun missing") + + monkeypatch.setattr(validate_module, "load_backend", _raise) + result = validate_module._check_vmrun(config) + assert not result.ok + assert "missing" in result.detail + + +def test_check_snapshots_backend_unavailable( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + def _raise(_c: Any) -> Any: + raise BackendNotAvailable("no vmrun") + + monkeypatch.setattr(validate_module, "load_backend", _raise) + result = validate_module._check_snapshots(config) + assert not result.ok + assert "backend unavailable" in result.detail + + +def test_check_snapshots_list_raises( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + _make_template_vmx(config, "LinuxBuild2404") + + class _Backend: + vmrun_path = "/usr/bin/vmrun" + + def list_snapshots(self, _handle: Any) -> list[str]: + raise BackendOperationFailed("listSnapshots", 1, "boom") + + monkeypatch.setattr(validate_module, "load_backend", lambda _c: _Backend()) + result = validate_module._check_snapshots(config) + assert not result.ok + assert "listSnapshots failed" in result.detail + + +def test_check_keyring_runtime_error( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + class _Store: + def get(self, target: str) -> Credential: + raise RuntimeError("keyring is not installed") + + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + result = validate_module._check_keyring(config) + assert not result.ok + assert "keyring is not installed" in result.detail + + +def test_check_permissions_not_writable( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + def _no_access(_path: Any, _mode: int) -> bool: + return False + + monkeypatch.setattr(validate_module.os, "access", _no_access) + result = validate_module._check_permissions(config) + assert not result.ok + assert "not read/write" in result.detail + + +def test_check_snapshots_ok( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + win_vmx = _make_template_vmx(config, "WinBuild2025") + backend = _FakeBackend({win_vmx: ["BaseClean"]}) + monkeypatch.setattr(validate_module, "load_backend", lambda _c: backend) + result = validate_module._check_snapshots(config) + assert result.ok, result.detail + assert "WinBuild2025->BaseClean" in result.detail + + +def test_check_snapshots_missing_snapshot( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + lin_vmx = _make_template_vmx(config, "LinuxBuild2404") + backend = _FakeBackend({lin_vmx: ["SomethingElse"]}) + monkeypatch.setattr(validate_module, "load_backend", lambda _c: backend) + result = validate_module._check_snapshots(config) + assert not result.ok + assert "BaseClean-Linux" in result.detail + + +def test_check_snapshots_no_templates( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + backend = _FakeBackend({}) + monkeypatch.setattr(validate_module, "load_backend", lambda _c: backend) + result = validate_module._check_snapshots(config) + assert not result.ok + assert "no known template VMX found" in result.detail + + +def test_check_keyring_ok( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + class _Store: + def get(self, target: str) -> Credential: + return Credential(username="ci_build", password="x") + + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + result = validate_module._check_keyring(config) + assert result.ok + assert "ci_build" in result.detail + + +def test_check_keyring_missing( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + class _Store: + def get(self, target: str) -> Credential: + raise KeyError(target) + + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + result = validate_module._check_keyring(config) + assert not result.ok + + +def test_check_permissions_ok(tmp_path: Path) -> None: + config = _make_config(tmp_path) + result = validate_module._check_permissions(config) + assert result.ok, result.detail + + +def test_check_permissions_missing_dir(tmp_path: Path) -> None: + config = _make_config(tmp_path) + # Remove a required dir. + config.paths.build_vms.rmdir() + result = validate_module._check_permissions(config) + assert not result.ok + assert "missing" in result.detail + + +def test_check_units_no_systemctl(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: None) + result = validate_module._check_units() + assert not result.ok + assert "systemctl" in result.detail + + +def test_check_units_active(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: "/bin/systemctl") + monkeypatch.setattr(validate_module, "_systemctl_is_active", lambda _u: "active") + result = validate_module._check_units() + assert result.ok + + +def test_check_units_inactive(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: "/bin/systemctl") + monkeypatch.setattr(validate_module, "_systemctl_is_active", lambda _u: "inactive") + result = validate_module._check_units() + assert not result.ok + assert "inactive" in result.detail + + +def test_systemctl_is_active_invokes_subprocess( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: "/bin/systemctl") + + class _CP: + stdout = "active\n" + + monkeypatch.setattr( + validate_module.subprocess, "run", lambda *a, **k: _CP() + ) + assert validate_module._systemctl_is_active("act-runner.service") == "active" + + +def test_systemctl_is_active_no_binary(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: None) + assert validate_module._systemctl_is_active("x") is None + + +def test_systemctl_is_active_timeout(monkeypatch: pytest.MonkeyPatch) -> None: + import subprocess as _sp + + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: "/bin/systemctl") + + def _boom(*_a: Any, **_k: Any) -> Any: + raise _sp.TimeoutExpired(cmd="systemctl", timeout=10.0) + + monkeypatch.setattr(validate_module.subprocess, "run", _boom) + assert validate_module._systemctl_is_active("x") is None + + +# ── validate host: end-to-end command ─────────────────────────────────────── + + +def test_validate_host_all_pass( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + win_vmx = _make_template_vmx(config, "WinBuild2025") + backend = _FakeBackend({win_vmx: ["BaseClean"]}) + + class _Store: + def get(self, target: str) -> Credential: + return Credential("ci_build", "x") + + monkeypatch.setattr(validate_module, "load_config", lambda: config) + monkeypatch.setattr(validate_module, "load_backend", lambda _c: backend) + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: "/bin/systemctl") + monkeypatch.setattr(validate_module, "_systemctl_is_active", lambda _u: "active") + + result = CliRunner().invoke(cli, ["validate", "host"]) + assert result.exit_code == 0, result.output + assert "All host checks passed" in result.output + + +def test_validate_host_reports_failures_and_exits_nonzero( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + def _raise(_c: Any) -> Any: + raise BackendNotAvailable("vmrun missing") + + class _Store: + def get(self, target: str) -> Credential: + raise KeyError(target) + + monkeypatch.setattr(validate_module, "load_config", lambda: config) + monkeypatch.setattr(validate_module, "load_backend", _raise) + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + monkeypatch.setattr(validate_module.shutil, "which", lambda _x: None) + + result = CliRunner().invoke(cli, ["validate", "host"]) + assert result.exit_code == 1, result.output + assert "FAIL" in result.output + assert "check(s) failed" in result.output + + +# ── validate guest ────────────────────────────────────────────────────────── + + +def test_resolve_use_winrm_explicit() -> None: + assert validate_module._resolve_use_winrm(True, None) is True + assert validate_module._resolve_use_winrm(False, "windows") is False + + +def test_resolve_use_winrm_inferred() -> None: + assert validate_module._resolve_use_winrm(None, "linux") is False + assert validate_module._resolve_use_winrm(None, "windows") is True + + +def test_resolve_use_winrm_requires_hint() -> None: + import click as _click + + with pytest.raises(_click.UsageError): + validate_module._resolve_use_winrm(None, None) + + +class _FakeTransport: + """Records connect/run/close and raises a configured error on run.""" + + def __init__(self, *args: Any, raise_on_run: Exception | None = None, **kw: Any) -> None: + self._raise = raise_on_run + self.closed = False + + def run(self, *_a: Any, **_k: Any) -> Any: + if self._raise is not None: + raise self._raise + + class _R: + returncode = 0 + + return _R() + + def close(self) -> None: + self.closed = True + + +def _install_keyring_store(monkeypatch: pytest.MonkeyPatch) -> None: + class _Store: + def get(self, target: str) -> Credential: + return Credential("Administrator", "pw") + + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + + +def test_probe_winrm_ok(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + config = _make_config(tmp_path) + _install_keyring_store(monkeypatch) + transport = _FakeTransport() + monkeypatch.setattr(validate_module, "WinRmTransport", lambda *a, **k: transport) + result = validate_module._probe_winrm("10.0.0.5", config) + assert result.ok + assert transport.closed + + +def test_probe_winrm_auth_fail( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + _install_keyring_store(monkeypatch) + transport = _FakeTransport(raise_on_run=TransportConnectError("auth denied")) + monkeypatch.setattr(validate_module, "WinRmTransport", lambda *a, **k: transport) + result = validate_module._probe_winrm("10.0.0.5", config) + assert not result.ok + assert "auth denied" in result.detail + + +def test_probe_winrm_command_fail( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + _install_keyring_store(monkeypatch) + transport = _FakeTransport(raise_on_run=TransportCommandError(1, "", "boom")) + monkeypatch.setattr(validate_module, "WinRmTransport", lambda *a, **k: transport) + result = validate_module._probe_winrm("10.0.0.5", config) + assert not result.ok + assert "command failed" in result.detail + + +def test_probe_winrm_credential_missing( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + + class _Store: + def get(self, target: str) -> Credential: + raise KeyError(target) + + monkeypatch.setattr(validate_module, "KeyringCredentialStore", _Store) + result = validate_module._probe_winrm("10.0.0.5", config) + assert not result.ok + assert "credential lookup failed" in result.detail + + +def test_probe_ssh_ok(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + config = _make_config(tmp_path) + transport = _FakeTransport() + monkeypatch.setattr(validate_module, "SshTransport", lambda *a, **k: transport) + result = validate_module._probe_ssh("10.0.0.6", config) + assert result.ok + assert transport.closed + + +def test_probe_ssh_connect_fail( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + transport = _FakeTransport(raise_on_run=TransportConnectError("no route")) + monkeypatch.setattr(validate_module, "SshTransport", lambda *a, **k: transport) + result = validate_module._probe_ssh("10.0.0.6", config) + assert not result.ok + assert "no route" in result.detail + + +def test_probe_ssh_command_fail( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + transport = _FakeTransport(raise_on_run=TransportCommandError(2, "", "nope")) + monkeypatch.setattr(validate_module, "SshTransport", lambda *a, **k: transport) + result = validate_module._probe_ssh("10.0.0.6", config) + assert not result.ok + assert "command failed" in result.detail + + +def test_validate_guest_ssh_ok( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + transport = _FakeTransport() + monkeypatch.setattr(validate_module, "load_config", lambda: config) + monkeypatch.setattr(validate_module, "SshTransport", lambda *a, **k: transport) + result = CliRunner().invoke( + cli, ["validate", "guest", "--host", "10.0.0.6", "--ssh"] + ) + assert result.exit_code == 0, result.output + assert "OK" in result.output + + +def test_validate_guest_winrm_fail_exits_nonzero( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + _install_keyring_store(monkeypatch) + transport = _FakeTransport(raise_on_run=TransportConnectError("bad creds")) + monkeypatch.setattr(validate_module, "load_config", lambda: config) + monkeypatch.setattr(validate_module, "WinRmTransport", lambda *a, **k: transport) + result = CliRunner().invoke( + cli, ["validate", "guest", "--host", "10.0.0.5", "--winrm"] + ) + assert result.exit_code == 1, result.output + assert "FAIL" in result.output + + +def test_validate_guest_requires_transport_hint( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + monkeypatch.setattr(validate_module, "load_config", lambda: config) + result = CliRunner().invoke(cli, ["validate", "guest", "--host", "10.0.0.5"]) + assert result.exit_code != 0 + assert "transport" in result.output.lower() + + +def test_validate_guest_infers_from_guest_os( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + config = _make_config(tmp_path) + transport = _FakeTransport() + monkeypatch.setattr(validate_module, "load_config", lambda: config) + monkeypatch.setattr(validate_module, "SshTransport", lambda *a, **k: transport) + result = CliRunner().invoke( + cli, ["validate", "guest", "--host", "10.0.0.6", "--guest-os", "linux"] + ) + assert result.exit_code == 0, result.output