Simone/local-ci-cd-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(a5): document Python orchestrator in AGENTS.md, ARCHITECTURE.md, README.md

Browse Source 
AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
This commit is contained in:
Simone
2026-05-14 17:38:10 +02:00
parent 5459672b42
commit 4d9c6d07ba
3 changed files with 255 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+67
View File
@@ -51,6 +51,73 @@ $ErrorActionPreference = 'Stop'
--- ---
## Python development
L'orchestratore CI vive in `src/ci_orchestrator/` (package `ci-orchestrator`,
entry point `python -m ci_orchestrator`). Tutta la logica applicativa
nuova va scritta in Python, non in PowerShell. Gli script PS in `scripts/`
sono ridotti a shim a 3 righe verso la CLI Python e sono mantenuti solo
per compatibilita' con Task Scheduler e caller esterni.
| Parametro | Valore |
| ------------------------ | ------------------------------------------------------------------------------------- |
| Python | **3.11+** richiesto (`requires-python` in `pyproject.toml`) |
| Venv produzione | `F:\CI\python\venv\` (interprete `F:\CI\python\venv\Scripts\python.exe`) |
| Venv dev locale | `.venv\` (NON committare; gitignored) |
| Install editable | `python -m pip install -e .[dev]` |
| Test | `python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80` |
| Lint | `python -m ruff check src tests\python` |
| Type-check | `python -m mypy --strict src` |
**Convenzioni Python (enforced da ruff + mypy --strict)**:
- Type hints obbligatori su funzioni pubbliche e helper esportati. Evitare
`Any` se esiste un tipo concreto/Protocol; `Any` solo per fixture pytest
e mock.
- Niente variabili globali. Lo state vive in oggetti passati esplicitamente
(factory `backends.load_backend(config)`, `Config` dataclass).
- Niente `print()`: usare `click.echo` (rispetta il flag `-q` di click e
il redirect `err=True` per stderr).
- Niente `subprocess.run(..., shell=True)`. Niente shell-out per cose che
esistono come libreria Python (paramiko per SSH, pypsrp per WinRM,
keyring per credenziali).
- Encoding stdio sempre UTF-8: i moduli Python lo assumono e
`runner/config.yaml` esporta `PYTHONIOENCODING=utf-8` ad act_runner.
**Mappatura script PowerShell → sotto-comando Python** (Fase A2/A3/A4):
| PowerShell legacy (shim) | Python sub-command |
| ----------------------------------- | ------------------------------------------ |
| `Wait-VMReady.ps1` | `python -m ci_orchestrator wait-ready` |
| `New-BuildVM.ps1` | `python -m ci_orchestrator vm new` |
| `Remove-BuildVM.ps1` | `python -m ci_orchestrator vm remove` |
| `Cleanup-OrphanedBuildVMs.ps1` | `python -m ci_orchestrator vm cleanup` |
| `Invoke-RemoteBuild.ps1` | `python -m ci_orchestrator build run` |
| `Get-BuildArtifacts.ps1` | `python -m ci_orchestrator artifacts collect` |
| `Watch-DiskSpace.ps1` | `python -m ci_orchestrator monitor disk` |
| `Watch-RunnerHealth.ps1` | `python -m ci_orchestrator monitor runner` |
| `Get-CIJobSummary.ps1` | `python -m ci_orchestrator report job` |
| `Invoke-CIJob.ps1` | `python -m ci_orchestrator job` |
Restano in PowerShell (girano sull'host fuori dal flusso CI o nel guest):
`Setup-Host.ps1`, `Register-CIScheduledTasks.ps1`, `Backup-CITemplate.ps1`,
`Invoke-RetentionPolicy.ps1`, `Measure-CIBenchmark.ps1`,
`Test-CapacityBurnIn.ps1`, tutti i `template/Deploy-*.ps1`,
`template/Prepare-*.ps1`, `template/Install-CIToolchain-*.ps1` e i
`tests/Test-*.ps1` di smoke locale.
**Test gate**: coverage totale ≥80% (alzato in A5). Nuovi moduli devono
includere unit test con mock; gli errori frequenti di questa stessa
sezione sono coperti da `tests/python/test_agents_errors.py` come
regression test — non rimuoverli senza aggiornare la voce in
"Errori frequenti da evitare" e motivare la rimozione.
**Closeout fasi A**: vedi `plans/A1-closeout.md`, `plans/A2-closeout.md`,
`plans/A3-closeout.md`, `plans/A4-closeout.md`, `plans/A5-closeout.md`
per stato per-fase, decisioni prese e voci pendenti a carico utente.
---
## Struttura directory host (fissa) ## Struttura directory host (fissa)
``` ```
README.md
+75 -13
View File
@@ -150,34 +150,96 @@ Variante **Windows Server 2022**: `*WinBuild2022.ps1` + VMX `F:\CI\Templates\Win
A new Python 3.11+ rewrite of the orchestrator lives under `src/ci_orchestrator/`. A new Python 3.11+ rewrite of the orchestrator lives under `src/ci_orchestrator/`.
Phase A (see [implementation-plan-A-B](plans/implementation-plan-A-B.md)) is on Phase A (see [implementation-plan-A-B](plans/implementation-plan-A-B.md)) is on
branch `feature/python-rewrite-phase-a`. branch `feature/python-rewrite-phase-a`. As of A5 the Python CLI is the
production entry point invoked by `gitea/actions/local-ci-build/action.yml`;
the PowerShell scripts in `scripts/` are 3-line shims that forward to the
Python sub-commands listed below.
Quick setup on the Windows host: ### Setup
```powershell ```powershell
# Create venv (one-time) # Production venv on the runner host (one-time)
python -m venv F:\CI\python\venv python -m venv F:\CI\python\venv
F:\CI\python\venv\Scripts\python.exe -m pip install --upgrade pip F:\CI\python\venv\Scripts\python.exe -m pip install --upgrade pip
F:\CI\python\venv\Scripts\python.exe -m pip install -e ".[dev]" F:\CI\python\venv\Scripts\python.exe -m pip install -e ".[dev]"
# Sanity checks # Local dev venv (NOT committed; see .gitignore)
F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator --help python -m venv .venv
F:\CI\python\venv\Scripts\python.exe -m pytest tests\python -q .\.venv\Scripts\python.exe -m pip install -e ".[dev]"
F:\CI\python\venv\Scripts\python.exe -m ruff check src tests\python
F:\CI\python\venv\Scripts\python.exe -m mypy --strict src
``` ```
Configuration: copy `config.example.toml` to `F:\CI\config.toml` (or set Configuration: copy [config.example.toml](config.example.toml) to
`$env:CI_CONFIG`). Environment variables (`CI_ROOT`, `CI_TEMPLATES`, …) override `F:\CI\config.toml` (or set `$env:CI_CONFIG`). Environment variables
the file. (`CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS`)
override the file. Dependency manifest and tool config (ruff, mypy,
pytest, coverage) live in [pyproject.toml](pyproject.toml).
In Phase A1 only the `wait-ready` PoC is wired: ### CLI sub-commands
```powershell ```powershell
F:\CI\python\venv\Scripts\python.exe -m ci_orchestrator wait-ready ` # Discover commands
python -m ci_orchestrator --help
# Wait for a VM to become reachable (WinRM 5986 or SSH 22 — auto-detected)
python -m ci_orchestrator wait-ready `
--vmx F:\CI\BuildVMs\smoke\smoke.vmx --guest-os windows --timeout 300 --vmx F:\CI\BuildVMs\smoke\smoke.vmx --guest-os windows --timeout 300
# Linked-clone lifecycle
python -m ci_orchestrator vm new `
--template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx `
--snapshot-name BaseClean `
--clone-base-dir F:\CI\BuildVMs `
--job-id smoke-001
python -m ci_orchestrator vm remove --vmx F:\CI\BuildVMs\Clone_smoke-001_*\Clone_smoke-001_*.vmx
python -m ci_orchestrator vm cleanup --max-age-hours 12
# Build + artifact collection
python -m ci_orchestrator build run `
--vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx `
--script .\build.ps1 --workdir C:\build --guest-os windows
python -m ci_orchestrator artifacts collect `
--vmx F:\CI\BuildVMs\Clone_smoke-001_*\*.vmx `
--remote-path C:\build\dist --local-dir F:\CI\Artifacts\smoke-001
# Monitoring (driven by Task Scheduler / systemd timers in B5)
python -m ci_orchestrator monitor disk --min-free-gb 50 --drive-letter F
python -m ci_orchestrator monitor runner --state-dir F:\CI\act_runner
# Reporting
python -m ci_orchestrator report job --job-id smoke-001
# End-to-end orchestrator (replaces Invoke-CIJob.ps1)
python -m ci_orchestrator job `
--job-id smoke-001 `
--repo-url ssh://gitea-ci/Simone/nsis-plugin-nsinnounp.git `
--branch main `
--template-path F:\CI\Templates\WinBuild2025\WinBuild2025.vmx `
--build-command "python build_plugin.py --final --dist-dir dist" `
--guest-artifact-source dist
``` ```
The PowerShell wrappers (`scripts\Invoke-CIJob.ps1`,
`scripts\New-BuildVM.ps1`, `scripts\Wait-VMReady.ps1`,
`scripts\Remove-BuildVM.ps1`, `scripts\Cleanup-OrphanedBuildVMs.ps1`,
`scripts\Invoke-RemoteBuild.ps1`, `scripts\Get-BuildArtifacts.ps1`,
`scripts\Watch-DiskSpace.ps1`, `scripts\Watch-RunnerHealth.ps1`,
`scripts\Get-CIJobSummary.ps1`) are kept as 3-line shims for Task
Scheduler and external callers — they translate PascalCase parameters
to the Python kebab-case CLI and propagate the exit code.
### Local validation gates
```powershell
.\.venv\Scripts\python.exe -m ruff check src tests\python
.\.venv\Scripts\python.exe -m mypy --strict src
.\.venv\Scripts\python.exe -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80
```
Coverage gate is **80%** as of Phase A5. Regression tests for the
`AGENTS.md` "Errori frequenti da evitare" #9–#12 live in
[tests/python/test_agents_errors.py](tests/python/test_agents_errors.py)
and must stay green.
--- ---
## Setup rapido ## Setup rapido
docs/ARCHITECTURE.md
+113
View File
@@ -206,3 +206,116 @@ VMs have internet access via NAT — required for pip/nuget/apt during build.
- Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time. - Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time.
- With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model. - With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model.
- Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM. - Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM.
---
## Python orchestrator (Phase A)
Phase A of [implementation-plan-A-B](../plans/implementation-plan-A-B.md)
replaces the PowerShell orchestration scripts with a Python 3.11+ package
under `src/ci_orchestrator/`. The PS scripts in `scripts/` are now thin
3-line shims that forward to the Python CLI — the diagram above still
applies, except that the boxes labelled `*.ps1` are entry-point names,
not the implementation.
### Package layout
```
src/ci_orchestrator/
├── __init__.py
├── __main__.py # click entry point: `python -m ci_orchestrator`
├── config.py # Config dataclass + env/TOML loader (OS-aware paths)
├── credentials.py # CredentialStore Protocol + KeyringCredentialStore
│
├── backends/
│ ├── __init__.py # load_backend(config) factory (selector for Phase C)
│ ├── protocol.py # VmBackend Protocol + VmHandle dataclass + VmState enum
│ ├── workstation.py # WorkstationVmrunBackend (subprocess + vmrun -T ws)
│ └── errors.py # BackendError / BackendNotAvailable / BackendOperationFailed
│
├── transport/
│ ├── __init__.py
│ ├── winrm.py # WinRM transport via pypsrp (replaces New-PSSession)
│ ├── ssh.py # SSH/SFTP transport via paramiko (replaces ssh.exe/scp.exe)
│ └── errors.py # TransportConnectError / TransportCommandError
│
└── commands/ # One module per `click` sub-command group
├── wait.py # `wait-ready`
├── vm.py # `vm new` / `vm remove` / `vm cleanup`
├── build.py # `build run`
├── artifacts.py # `artifacts collect`
├── monitor.py # `monitor disk` / `monitor runner`
├── report.py # `report job`
└── job.py # `job` — full end-to-end orchestrator
```
### `VmBackend` Protocol and the `load_backend(config)` factory
`backends/protocol.py` defines a hypervisor-agnostic Protocol:
```python
class VmBackend(Protocol):
def clone_linked(self, template, snapshot, name, destination=None) -> VmHandle: ...
def start(self, handle, headless=True) -> None: ...
def stop(self, handle, hard=False) -> None: ...
def delete(self, handle) -> None: ...
def get_ip(self, handle, timeout=0.0) -> str | None: ...
def list_snapshots(self, handle) -> list[str]: ...
def is_running(self, handle) -> bool: ...
```
Names are deliberately neutral (no `vmrun_*` prefixes) and identifiers are
opaque strings (`VmHandle.identifier`): a VMX path on Workstation, a
managed object reference on ESXi. `is_running` MUST NOT use guest tools
(see `AGENTS.md` error #10).
Backends are constructed via `backends.load_backend(config)`. Application
code (most importantly `commands/job.py`) imports the factory only — never
`WorkstationVmrunBackend` directly. The factory reads
`config.backend.type` from `config.toml` (default: `"workstation"`).
### Phase C extension point — adding ESXi without forks
To add an ESXi backend in a future Phase C:
1. Add a new module `backends/esxi.py` implementing the `VmBackend`
Protocol via `pyVmomi`. `VmHandle.identifier` becomes the managed
object reference; `clone_linked` becomes a `CloneVM_Task` call;
`start` becomes `PowerOnVM_Task`.
2. Register it in `backends/__init__.py` `load_backend(config)` under a
new `config.backend.type == "esxi"` branch.
3. Add `[backend]` keys to `config.example.toml` (host, user, datacenter,
datastore, …). The `Config` dataclass already namespaces backend
options under `config.backend`.
4. Transport layers (`winrm.py`, `ssh.py`) and the `commands/*` modules
need no change — they consume `VmBackend` and `VmHandle` only.
The single non-trivial constraint is that `commands/vm.py` `vm new`
currently builds the destination VMX path under `--clone-base-dir` (a
host-local concept). For ESXi this becomes a datastore folder; the
parameter rename can be deferred to Phase C as long as the backend
ignores the value and uses `config.backend.datastore` instead.
### Transport selection
`commands/job.py` auto-detects the guest OS by parsing `guestOS = "..."`
in the cloned VMX (`ubuntu-*` → Linux/SSH, anything else → Windows/WinRM).
The Linux branch is wired to `transport/ssh.py` (paramiko) and the
Windows branch to `transport/winrm.py` (pypsrp). Both transports raise
typed errors (`TransportConnectError`, `TransportCommandError`) — never
bare exceptions.
### Validation gates (CI)
Enforced by `gitea/workflows/lint.yml` job `python`:
```powershell
python -m ruff check src tests\python
python -m mypy --strict src
python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80
```
Coverage gate is **80%** as of Phase A5. Regression tests for
`AGENTS.md` errors #9–#12 live in
[tests/python/test_agents_errors.py](../tests/python/test_agents_errors.py).