docs(a5): document Python orchestrator in AGENTS.md, ARCHITECTURE.md, README.md

AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
2026-05-14 17:38:10 +02:00
parent 5459672b42
commit 4d9c6d07ba
3 changed files with 255 additions and 13 deletions
@@ -51,6 +51,73 @@ $ErrorActionPreference = 'Stop'

 ---

+## Python development
+
+L'orchestratore CI vive in `src/ci_orchestrator/` (package `ci-orchestrator`,
+entry point `python -m ci_orchestrator`). Tutta la logica applicativa
+nuova va scritta in Python, non in PowerShell. Gli script PS in `scripts/`
+sono ridotti a shim a 3 righe verso la CLI Python e sono mantenuti solo
+per compatibilita' con Task Scheduler e caller esterni.
+
+| Parametro                | Valore                                                                                |
+| ------------------------ | ------------------------------------------------------------------------------------- |
+| Python                   | **3.11+** richiesto (`requires-python` in `pyproject.toml`)                           |
+| Venv produzione          | `F:\CI\python\venv\` (interprete `F:\CI\python\venv\Scripts\python.exe`)              |
+| Venv dev locale          | `.venv\` (NON committare; gitignored)                                                  |
+| Install editable         | `python -m pip install -e .[dev]`                                                     |
+| Test                     | `python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80`          |
+| Lint                     | `python -m ruff check src tests\python`                                               |
+| Type-check               | `python -m mypy --strict src`                                                         |
+
+**Convenzioni Python (enforced da ruff + mypy --strict)**:
+
+- Type hints obbligatori su funzioni pubbliche e helper esportati. Evitare
+  `Any` se esiste un tipo concreto/Protocol; `Any` solo per fixture pytest
+  e mock.
+- Niente variabili globali. Lo state vive in oggetti passati esplicitamente
+  (factory `backends.load_backend(config)`, `Config` dataclass).
+- Niente `print()`: usare `click.echo` (rispetta il flag `-q` di click e
+  il redirect `err=True` per stderr).
+- Niente `subprocess.run(..., shell=True)`. Niente shell-out per cose che
+  esistono come libreria Python (paramiko per SSH, pypsrp per WinRM,
+  keyring per credenziali).
+- Encoding stdio sempre UTF-8: i moduli Python lo assumono e
+  `runner/config.yaml` esporta `PYTHONIOENCODING=utf-8` ad act_runner.
+
+**Mappatura script PowerShell → sotto-comando Python** (Fase A2/A3/A4):
+
+| PowerShell legacy (shim)            | Python sub-command                         |
+| ----------------------------------- | ------------------------------------------ |
+| `Wait-VMReady.ps1`                  | `python -m ci_orchestrator wait-ready`     |
+| `New-BuildVM.ps1`                   | `python -m ci_orchestrator vm new`         |
+| `Remove-BuildVM.ps1`                | `python -m ci_orchestrator vm remove`      |
+| `Cleanup-OrphanedBuildVMs.ps1`      | `python -m ci_orchestrator vm cleanup`     |
+| `Invoke-RemoteBuild.ps1`            | `python -m ci_orchestrator build run`      |
+| `Get-BuildArtifacts.ps1`            | `python -m ci_orchestrator artifacts collect` |
+| `Watch-DiskSpace.ps1`               | `python -m ci_orchestrator monitor disk`   |
+| `Watch-RunnerHealth.ps1`            | `python -m ci_orchestrator monitor runner` |
+| `Get-CIJobSummary.ps1`              | `python -m ci_orchestrator report job`     |
+| `Invoke-CIJob.ps1`                  | `python -m ci_orchestrator job`            |
+
+Restano in PowerShell (girano sull'host fuori dal flusso CI o nel guest):
+`Setup-Host.ps1`, `Register-CIScheduledTasks.ps1`, `Backup-CITemplate.ps1`,
+`Invoke-RetentionPolicy.ps1`, `Measure-CIBenchmark.ps1`,
+`Test-CapacityBurnIn.ps1`, tutti i `template/Deploy-*.ps1`,
+`template/Prepare-*.ps1`, `template/Install-CIToolchain-*.ps1` e i
+`tests/Test-*.ps1` di smoke locale.
+
+**Test gate**: coverage totale ≥80% (alzato in A5). Nuovi moduli devono
+includere unit test con mock; gli errori frequenti di questa stessa
+sezione sono coperti da `tests/python/test_agents_errors.py` come
+regression test — non rimuoverli senza aggiornare la voce in
+"Errori frequenti da evitare" e motivare la rimozione.
+
+**Closeout fasi A**: vedi `plans/A1-closeout.md`, `plans/A2-closeout.md`,
+`plans/A3-closeout.md`, `plans/A4-closeout.md`, `plans/A5-closeout.md`
+per stato per-fase, decisioni prese e voci pendenti a carico utente.
+
+---
+
 ## Struttura directory host (fissa)

 ```
@@ -150,34 +150,96 @@ Variante **Windows Server 2022**: `*WinBuild2022.ps1` + VMX `F:\CI\Templates\Win

 A new Python 3.11+ rewrite of the orchestrator lives under `src/ci_orchestrator/`.
 Phase A (see [implementation-plan-A-B](plans/implementation-plan-A-B.md)) is on
-branch `feature/python-rewrite-phase-a`.
+branch `feature/python-rewrite-phase-a`. As of A5 the Python CLI is the
+production entry point invoked by `gitea/actions/local-ci-build/action.yml`;
+the PowerShell scripts in `scripts/` are 3-line shims that forward to the
+Python sub-commands listed below.

-Quick setup on the Windows host:
+### Setup

 ```powershell
-# Create venv (one-time)
+# Production venv on the runner host (one-time)
 python -m venv F:\CI\python\venv
 F:\CI\python\venv\Scripts\python.exe -m pip install --upgrade pip
 F:\CI\python\venv\Scripts\python.exe -m pip install -e ".[dev]"

-# Sanity checks
-F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator --help
-F:\CI\python\venv\Scripts\python.exe -m pytest tests\python -q
-F:\CI\python\venv\Scripts\python.exe -m ruff check src tests\python
-F:\CI\python\venv\Scripts\python.exe -m mypy --strict src
+# Local dev venv (NOT committed; see .gitignore)
+python -m venv .venv
+.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
 ```

-Configuration: copy `config.example.toml` to `F:\CI\config.toml` (or set
-`$env:CI_CONFIG`). Environment variables (`CI_ROOT`, `CI_TEMPLATES`, …) override
-the file.
+Configuration: copy [config.example.toml](config.example.toml) to
+`F:\CI\config.toml` (or set `$env:CI_CONFIG`). Environment variables
+(`CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS`)
+override the file. Dependency manifest and tool config (ruff, mypy,
+pytest, coverage) live in [pyproject.toml](pyproject.toml).

-In Phase A1 only the `wait-ready` PoC is wired:
+### CLI sub-commands

 ```powershell
-F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator wait-ready `
+# Discover commands
+python -m ci_orchestrator --help
+
+# Wait for a VM to become reachable (WinRM 5986 or SSH 22 — auto-detected)
+python -m ci_orchestrator wait-ready `
    --vmx F:\CI\BuildVMs\smoke\smoke.vmx --guest-os windows --timeout 300
+
+# Linked-clone lifecycle
+python -m ci_orchestrator vm new `
+    --template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx `
+    --snapshot-name BaseClean `
+    --clone-base-dir F:\CI\BuildVMs `
+    --job-id smoke-001
+python -m ci_orchestrator vm remove --vmx F:\CI\BuildVMs\Clone_smoke-001_*\Clone_smoke-001_*.vmx
+python -m ci_orchestrator vm cleanup --max-age-hours 12
+
+# Build + artifact collection
+python -m ci_orchestrator build run `
+    --vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx `
+    --script .\build.ps1 --workdir C:\build --guest-os windows
+python -m ci_orchestrator artifacts collect `
+    --vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx `
+    --remote-path C:\build\dist --local-dir F:\CI\Artifacts\smoke-001
+
+# Monitoring (driven by Task Scheduler / systemd timers in B5)
+python -m ci_orchestrator monitor disk --min-free-gb 50 --drive-letter F
+python -m ci_orchestrator monitor runner --state-dir F:\CI\act_runner
+
+# Reporting
+python -m ci_orchestrator report job --job-id smoke-001
+
+# End-to-end orchestrator (replaces Invoke-CIJob.ps1)
+python -m ci_orchestrator job `
+    --job-id        smoke-001 `
+    --repo-url      ssh://gitea-ci/Simone/nsis-plugin-nsinnounp.git `
+    --branch        main `
+    --template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx `
+    --build-command "python build_plugin.py --final --dist-dir dist" `
+    --guest-artifact-source dist
 ```

+The PowerShell wrappers (`scripts\Invoke-CIJob.ps1`,
+`scripts\New-BuildVM.ps1`, `scripts\Wait-VMReady.ps1`,
+`scripts\Remove-BuildVM.ps1`, `scripts\Cleanup-OrphanedBuildVMs.ps1`,
+`scripts\Invoke-RemoteBuild.ps1`, `scripts\Get-BuildArtifacts.ps1`,
+`scripts\Watch-DiskSpace.ps1`, `scripts\Watch-RunnerHealth.ps1`,
+`scripts\Get-CIJobSummary.ps1`) are kept as 3-line shims for Task
+Scheduler and external callers — they translate PascalCase parameters
+to the Python kebab-case CLI and propagate the exit code.
+
+### Local validation gates
+
+```powershell
+.\.venv\Scripts\python.exe -m ruff check src tests\python
+.\.venv\Scripts\python.exe -m mypy --strict src
+.\.venv\Scripts\python.exe -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80
+```
+
+Coverage gate is **80%** as of Phase A5. Regression tests for the
+`AGENTS.md` "Errori frequenti da evitare" #9–#12 live in
+[tests/python/test_agents_errors.py](tests/python/test_agents_errors.py)
+and must stay green.
+
 ---

 ## Setup rapido
@@ -206,3 +206,116 @@ VMs have internet access via NAT — required for pip/nuget/apt during build.
 - Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time.
 - With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model.
 - Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM.
+
+---
+
+## Python orchestrator (Phase A)
+
+Phase A of [implementation-plan-A-B](../plans/implementation-plan-A-B.md)
+replaces the PowerShell orchestration scripts with a Python 3.11+ package
+under `src/ci_orchestrator/`. The PS scripts in `scripts/` are now thin
+3-line shims that forward to the Python CLI — the diagram above still
+applies, except that the boxes labelled `*.ps1` are entry-point names,
+not the implementation.
+
+### Package layout
+
+```
+src/ci_orchestrator/
+├── __init__.py
+├── __main__.py                # click entry point: `python -m ci_orchestrator`
+├── config.py                  # Config dataclass + env/TOML loader (OS-aware paths)
+├── credentials.py             # CredentialStore Protocol + KeyringCredentialStore
+│
+├── backends/
+│   ├── __init__.py            # load_backend(config) factory (selector for Phase C)
+│   ├── protocol.py            # VmBackend Protocol + VmHandle dataclass + VmState enum
+│   ├── workstation.py         # WorkstationVmrunBackend (subprocess + vmrun -T ws)
+│   └── errors.py              # BackendError / BackendNotAvailable / BackendOperationFailed
+│
+├── transport/
+│   ├── __init__.py
+│   ├── winrm.py               # WinRM transport via pypsrp (replaces New-PSSession)
+│   ├── ssh.py                 # SSH/SFTP transport via paramiko (replaces ssh.exe/scp.exe)
+│   └── errors.py              # TransportConnectError / TransportCommandError
+│
+└── commands/                  # One module per `click` sub-command group
+    ├── wait.py                # `wait-ready`
+    ├── vm.py                  # `vm new` / `vm remove` / `vm cleanup`
+    ├── build.py               # `build run`
+    ├── artifacts.py           # `artifacts collect`
+    ├── monitor.py             # `monitor disk` / `monitor runner`
+    ├── report.py              # `report job`
+    └── job.py                 # `job` — full end-to-end orchestrator
+```
+
+### `VmBackend` Protocol and the `load_backend(config)` factory
+
+`backends/protocol.py` defines a hypervisor-agnostic Protocol:
+
+```python
+class VmBackend(Protocol):
+    def clone_linked(self, template, snapshot, name, destination=None) -> VmHandle: ...
+    def start(self, handle, headless=True) -> None: ...
+    def stop(self, handle, hard=False) -> None: ...
+    def delete(self, handle) -> None: ...
+    def get_ip(self, handle, timeout=0.0) -> str | None: ...
+    def list_snapshots(self, handle) -> list[str]: ...
+    def is_running(self, handle) -> bool: ...
+```
+
+Names are deliberately neutral (no `vmrun_*` prefixes) and identifiers are
+opaque strings (`VmHandle.identifier`): a VMX path on Workstation, a
+managed object reference on ESXi. `is_running` MUST NOT use guest tools
+(see `AGENTS.md` error #10).
+
+Backends are constructed via `backends.load_backend(config)`. Application
+code (most importantly `commands/job.py`) imports the factory only — never
+`WorkstationVmrunBackend` directly. The factory reads
+`config.backend.type` from `config.toml` (default: `"workstation"`).
+
+### Phase C extension point — adding ESXi without forks
+
+To add an ESXi backend in a future Phase C:
+
+1. Add a new module `backends/esxi.py` implementing the `VmBackend`
+   Protocol via `pyVmomi`. `VmHandle.identifier` becomes the managed
+   object reference; `clone_linked` becomes a `CloneVM_Task` call;
+   `start` becomes `PowerOnVM_Task`.
+2. Register it in `backends/__init__.py` `load_backend(config)` under a
+   new `config.backend.type == "esxi"` branch.
+3. Add `[backend]` keys to `config.example.toml` (host, user, datacenter,
+   datastore, …). The `Config` dataclass already namespaces backend
+   options under `config.backend`.
+4. Transport layers (`winrm.py`, `ssh.py`) and the `commands/*` modules
+   need no change — they consume `VmBackend` and `VmHandle` only.
+
+The single non-trivial constraint is that `commands/vm.py` `vm new`
+currently builds the destination VMX path under `--clone-base-dir` (a
+host-local concept). For ESXi this becomes a datastore folder; the
+parameter rename can be deferred to Phase C as long as the backend
+ignores the value and uses `config.backend.datastore` instead.
+
+### Transport selection
+
+`commands/job.py` auto-detects the guest OS by parsing `guestOS = "..."`
+in the cloned VMX (`ubuntu-*` → Linux/SSH, anything else → Windows/WinRM).
+The Linux branch is wired to `transport/ssh.py` (paramiko) and the
+Windows branch to `transport/winrm.py` (pypsrp). Both transports raise
+typed errors (`TransportConnectError`, `TransportCommandError`) — never
+bare exceptions.
+
+### Validation gates (CI)
+
+Enforced by `gitea/workflows/lint.yml` job `python`:
+
+```powershell
+python -m ruff check src tests\python
+python -m mypy --strict src
+python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80
+```
+
+Coverage gate is **80%** as of Phase A5. Regression tests for
+`AGENTS.md` errors #9–#12 live in
+[tests/python/test_agents_errors.py](../tests/python/test_agents_errors.py).
+