From 4d9c6d07ba8fc356f098d3b489071c089b3ed9f9 Mon Sep 17 00:00:00 2001 From: Simone Date: Thu, 14 May 2026 17:38:10 +0200 Subject: [PATCH] docs(a5): document Python orchestrator in AGENTS.md, ARCHITECTURE.md, README.md AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates. --- AGENTS.md | 67 +++++++++++++++++++++++++ README.md | 88 ++++++++++++++++++++++++++++----- docs/ARCHITECTURE.md | 113 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 255 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 012647a..4aef687 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,6 +51,73 @@ $ErrorActionPreference = 'Stop' --- +## Python development + +L'orchestratore CI vive in `src/ci_orchestrator/` (package `ci-orchestrator`, +entry point `python -m ci_orchestrator`). Tutta la logica applicativa +nuova va scritta in Python, non in PowerShell. Gli script PS in `scripts/` +sono ridotti a shim a 3 righe verso la CLI Python e sono mantenuti solo +per compatibilita' con Task Scheduler e caller esterni. + +| Parametro | Valore | +| ------------------------ | ------------------------------------------------------------------------------------- | +| Python | **3.11+** richiesto (`requires-python` in `pyproject.toml`) | +| Venv produzione | `F:\CI\python\venv\` (interprete `F:\CI\python\venv\Scripts\python.exe`) | +| Venv dev locale | `.venv\` (NON committare; gitignored) | +| Install editable | `python -m pip install -e .[dev]` | +| Test | `python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80` | +| Lint | `python -m ruff check src tests\python` | +| Type-check | `python -m mypy --strict src` | + +**Convenzioni Python (enforced da ruff + mypy --strict)**: + +- Type hints obbligatori su funzioni pubbliche e helper esportati. Evitare + `Any` se esiste un tipo concreto/Protocol; `Any` solo per fixture pytest + e mock. +- Niente variabili globali. Lo state vive in oggetti passati esplicitamente + (factory `backends.load_backend(config)`, `Config` dataclass). +- Niente `print()`: usare `click.echo` (rispetta il flag `-q` di click e + il redirect `err=True` per stderr). +- Niente `subprocess.run(..., shell=True)`. Niente shell-out per cose che + esistono come libreria Python (paramiko per SSH, pypsrp per WinRM, + keyring per credenziali). +- Encoding stdio sempre UTF-8: i moduli Python lo assumono e + `runner/config.yaml` esporta `PYTHONIOENCODING=utf-8` ad act_runner. + +**Mappatura script PowerShell → sotto-comando Python** (Fase A2/A3/A4): + +| PowerShell legacy (shim) | Python sub-command | +| ----------------------------------- | ------------------------------------------ | +| `Wait-VMReady.ps1` | `python -m ci_orchestrator wait-ready` | +| `New-BuildVM.ps1` | `python -m ci_orchestrator vm new` | +| `Remove-BuildVM.ps1` | `python -m ci_orchestrator vm remove` | +| `Cleanup-OrphanedBuildVMs.ps1` | `python -m ci_orchestrator vm cleanup` | +| `Invoke-RemoteBuild.ps1` | `python -m ci_orchestrator build run` | +| `Get-BuildArtifacts.ps1` | `python -m ci_orchestrator artifacts collect` | +| `Watch-DiskSpace.ps1` | `python -m ci_orchestrator monitor disk` | +| `Watch-RunnerHealth.ps1` | `python -m ci_orchestrator monitor runner` | +| `Get-CIJobSummary.ps1` | `python -m ci_orchestrator report job` | +| `Invoke-CIJob.ps1` | `python -m ci_orchestrator job` | + +Restano in PowerShell (girano sull'host fuori dal flusso CI o nel guest): +`Setup-Host.ps1`, `Register-CIScheduledTasks.ps1`, `Backup-CITemplate.ps1`, +`Invoke-RetentionPolicy.ps1`, `Measure-CIBenchmark.ps1`, +`Test-CapacityBurnIn.ps1`, tutti i `template/Deploy-*.ps1`, +`template/Prepare-*.ps1`, `template/Install-CIToolchain-*.ps1` e i +`tests/Test-*.ps1` di smoke locale. + +**Test gate**: coverage totale ≥80% (alzato in A5). Nuovi moduli devono +includere unit test con mock; gli errori frequenti di questa stessa +sezione sono coperti da `tests/python/test_agents_errors.py` come +regression test — non rimuoverli senza aggiornare la voce in +"Errori frequenti da evitare" e motivare la rimozione. + +**Closeout fasi A**: vedi `plans/A1-closeout.md`, `plans/A2-closeout.md`, +`plans/A3-closeout.md`, `plans/A4-closeout.md`, `plans/A5-closeout.md` +per stato per-fase, decisioni prese e voci pendenti a carico utente. + +--- + ## Struttura directory host (fissa) ``` diff --git a/README.md b/README.md index cef37af..71630e4 100644 --- a/README.md +++ b/README.md @@ -150,34 +150,96 @@ Variante **Windows Server 2022**: `*WinBuild2022.ps1` + VMX `F:\CI\Templates\Win A new Python 3.11+ rewrite of the orchestrator lives under `src/ci_orchestrator/`. Phase A (see [implementation-plan-A-B](plans/implementation-plan-A-B.md)) is on -branch `feature/python-rewrite-phase-a`. +branch `feature/python-rewrite-phase-a`. As of A5 the Python CLI is the +production entry point invoked by `gitea/actions/local-ci-build/action.yml`; +the PowerShell scripts in `scripts/` are 3-line shims that forward to the +Python sub-commands listed below. -Quick setup on the Windows host: +### Setup ```powershell -# Create venv (one-time) +# Production venv on the runner host (one-time) python -m venv F:\CI\python\venv F:\CI\python\venv\Scripts\python.exe -m pip install --upgrade pip F:\CI\python\venv\Scripts\python.exe -m pip install -e ".[dev]" -# Sanity checks -F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator --help -F:\CI\python\venv\Scripts\python.exe -m pytest tests\python -q -F:\CI\python\venv\Scripts\python.exe -m ruff check src tests\python -F:\CI\python\venv\Scripts\python.exe -m mypy --strict src +# Local dev venv (NOT committed; see .gitignore) +python -m venv .venv +.\.venv\Scripts\python.exe -m pip install -e ".[dev]" ``` -Configuration: copy `config.example.toml` to `F:\CI\config.toml` (or set -`$env:CI_CONFIG`). Environment variables (`CI_ROOT`, `CI_TEMPLATES`, …) override -the file. +Configuration: copy [config.example.toml](config.example.toml) to +`F:\CI\config.toml` (or set `$env:CI_CONFIG`). Environment variables +(`CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS`) +override the file. Dependency manifest and tool config (ruff, mypy, +pytest, coverage) live in [pyproject.toml](pyproject.toml). -In Phase A1 only the `wait-ready` PoC is wired: +### CLI sub-commands ```powershell -F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator wait-ready ` +# Discover commands +python -m ci_orchestrator --help + +# Wait for a VM to become reachable (WinRM 5986 or SSH 22 — auto-detected) +python -m ci_orchestrator wait-ready ` --vmx F:\CI\BuildVMs\smoke\smoke.vmx --guest-os windows --timeout 300 + +# Linked-clone lifecycle +python -m ci_orchestrator vm new ` + --template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx ` + --snapshot-name BaseClean ` + --clone-base-dir F:\CI\BuildVMs ` + --job-id smoke-001 +python -m ci_orchestrator vm remove --vmx F:\CI\BuildVMs\Clone_smoke-001_*\Clone_smoke-001_*.vmx +python -m ci_orchestrator vm cleanup --max-age-hours 12 + +# Build + artifact collection +python -m ci_orchestrator build run ` + --vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx ` + --script .\build.ps1 --workdir C:\build --guest-os windows +python -m ci_orchestrator artifacts collect ` + --vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx ` + --remote-path C:\build\dist --local-dir F:\CI\Artifacts\smoke-001 + +# Monitoring (driven by Task Scheduler / systemd timers in B5) +python -m ci_orchestrator monitor disk --min-free-gb 50 --drive-letter F +python -m ci_orchestrator monitor runner --state-dir F:\CI\act_runner + +# Reporting +python -m ci_orchestrator report job --job-id smoke-001 + +# End-to-end orchestrator (replaces Invoke-CIJob.ps1) +python -m ci_orchestrator job ` + --job-id smoke-001 ` + --repo-url ssh://gitea-ci/Simone/nsis-plugin-nsinnounp.git ` + --branch main ` + --template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx ` + --build-command "python build_plugin.py --final --dist-dir dist" ` + --guest-artifact-source dist ``` +The PowerShell wrappers (`scripts\Invoke-CIJob.ps1`, +`scripts\New-BuildVM.ps1`, `scripts\Wait-VMReady.ps1`, +`scripts\Remove-BuildVM.ps1`, `scripts\Cleanup-OrphanedBuildVMs.ps1`, +`scripts\Invoke-RemoteBuild.ps1`, `scripts\Get-BuildArtifacts.ps1`, +`scripts\Watch-DiskSpace.ps1`, `scripts\Watch-RunnerHealth.ps1`, +`scripts\Get-CIJobSummary.ps1`) are kept as 3-line shims for Task +Scheduler and external callers — they translate PascalCase parameters +to the Python kebab-case CLI and propagate the exit code. + +### Local validation gates + +```powershell +.\.venv\Scripts\python.exe -m ruff check src tests\python +.\.venv\Scripts\python.exe -m mypy --strict src +.\.venv\Scripts\python.exe -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80 +``` + +Coverage gate is **80%** as of Phase A5. Regression tests for the +`AGENTS.md` "Errori frequenti da evitare" #9–#12 live in +[tests/python/test_agents_errors.py](tests/python/test_agents_errors.py) +and must stay green. + --- ## Setup rapido diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2b8d7d1..0135439 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -206,3 +206,116 @@ VMs have internet access via NAT — required for pip/nuget/apt during build. - Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time. - With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model. - Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM. + +--- + +## Python orchestrator (Phase A) + +Phase A of [implementation-plan-A-B](../plans/implementation-plan-A-B.md) +replaces the PowerShell orchestration scripts with a Python 3.11+ package +under `src/ci_orchestrator/`. The PS scripts in `scripts/` are now thin +3-line shims that forward to the Python CLI — the diagram above still +applies, except that the boxes labelled `*.ps1` are entry-point names, +not the implementation. + +### Package layout + +``` +src/ci_orchestrator/ +├── __init__.py +├── __main__.py # click entry point: `python -m ci_orchestrator` +├── config.py # Config dataclass + env/TOML loader (OS-aware paths) +├── credentials.py # CredentialStore Protocol + KeyringCredentialStore +│ +├── backends/ +│ ├── __init__.py # load_backend(config) factory (selector for Phase C) +│ ├── protocol.py # VmBackend Protocol + VmHandle dataclass + VmState enum +│ ├── workstation.py # WorkstationVmrunBackend (subprocess + vmrun -T ws) +│ └── errors.py # BackendError / BackendNotAvailable / BackendOperationFailed +│ +├── transport/ +│ ├── __init__.py +│ ├── winrm.py # WinRM transport via pypsrp (replaces New-PSSession) +│ ├── ssh.py # SSH/SFTP transport via paramiko (replaces ssh.exe/scp.exe) +│ └── errors.py # TransportConnectError / TransportCommandError +│ +└── commands/ # One module per `click` sub-command group + ├── wait.py # `wait-ready` + ├── vm.py # `vm new` / `vm remove` / `vm cleanup` + ├── build.py # `build run` + ├── artifacts.py # `artifacts collect` + ├── monitor.py # `monitor disk` / `monitor runner` + ├── report.py # `report job` + └── job.py # `job` — full end-to-end orchestrator +``` + +### `VmBackend` Protocol and the `load_backend(config)` factory + +`backends/protocol.py` defines a hypervisor-agnostic Protocol: + +```python +class VmBackend(Protocol): + def clone_linked(self, template, snapshot, name, destination=None) -> VmHandle: ... + def start(self, handle, headless=True) -> None: ... + def stop(self, handle, hard=False) -> None: ... + def delete(self, handle) -> None: ... + def get_ip(self, handle, timeout=0.0) -> str | None: ... + def list_snapshots(self, handle) -> list[str]: ... + def is_running(self, handle) -> bool: ... +``` + +Names are deliberately neutral (no `vmrun_*` prefixes) and identifiers are +opaque strings (`VmHandle.identifier`): a VMX path on Workstation, a +managed object reference on ESXi. `is_running` MUST NOT use guest tools +(see `AGENTS.md` error #10). + +Backends are constructed via `backends.load_backend(config)`. Application +code (most importantly `commands/job.py`) imports the factory only — never +`WorkstationVmrunBackend` directly. The factory reads +`config.backend.type` from `config.toml` (default: `"workstation"`). + +### Phase C extension point — adding ESXi without forks + +To add an ESXi backend in a future Phase C: + +1. Add a new module `backends/esxi.py` implementing the `VmBackend` + Protocol via `pyVmomi`. `VmHandle.identifier` becomes the managed + object reference; `clone_linked` becomes a `CloneVM_Task` call; + `start` becomes `PowerOnVM_Task`. +2. Register it in `backends/__init__.py` `load_backend(config)` under a + new `config.backend.type == "esxi"` branch. +3. Add `[backend]` keys to `config.example.toml` (host, user, datacenter, + datastore, …). The `Config` dataclass already namespaces backend + options under `config.backend`. +4. Transport layers (`winrm.py`, `ssh.py`) and the `commands/*` modules + need no change — they consume `VmBackend` and `VmHandle` only. + +The single non-trivial constraint is that `commands/vm.py` `vm new` +currently builds the destination VMX path under `--clone-base-dir` (a +host-local concept). For ESXi this becomes a datastore folder; the +parameter rename can be deferred to Phase C as long as the backend +ignores the value and uses `config.backend.datastore` instead. + +### Transport selection + +`commands/job.py` auto-detects the guest OS by parsing `guestOS = "..."` +in the cloned VMX (`ubuntu-*` → Linux/SSH, anything else → Windows/WinRM). +The Linux branch is wired to `transport/ssh.py` (paramiko) and the +Windows branch to `transport/winrm.py` (pypsrp). Both transports raise +typed errors (`TransportConnectError`, `TransportCommandError`) — never +bare exceptions. + +### Validation gates (CI) + +Enforced by `gitea/workflows/lint.yml` job `python`: + +```powershell +python -m ruff check src tests\python +python -m mypy --strict src +python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80 +``` + +Coverage gate is **80%** as of Phase A5. Regression tests for +`AGENTS.md` errors #9–#12 live in +[tests/python/test_agents_errors.py](../tests/python/test_agents_errors.py). +