Simone
451a61c6a1
Update --cov-fail-under in lint workflow + docs (AGENTS, README, ARCHITECTURE). Note: A5 closeout reported 80.10%% so this run will likely fail until additional tests are added.
322 lines
18 KiB
Markdown
322 lines
18 KiB
Markdown
# Architecture — Local CI/CD System
|
||
|
||
## Overview
|
||
|
||
A fully local, isolated CI/CD pipeline running on a Windows host using:
|
||
- **Gitea** as the self-hosted Git server + CI platform
|
||
- **act_runner** as the job executor (runs on the Windows host)
|
||
- **VMware Workstation** for ephemeral build VMs (linked clones)
|
||
- **WinRM / PowerShell Remoting** for communication with Windows build VMs
|
||
- **SSH / SCP** (`ssh.exe` + `scp.exe`) for communication with Linux build VMs
|
||
- **MSBuild / dotnet CLI** (Windows) and **GCC / Clang / CMake** (Linux) executing inside guest VMs only
|
||
|
||
The runner host **never executes build tools directly**. Its only role is orchestration: creating, driving, and destroying VMs.
|
||
|
||
---
|
||
|
||
## System Diagram
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ DEVELOPER WORKSTATION │
|
||
│ │
|
||
│ git push → Gitea (self-hosted, LAN) │
|
||
└───────────────────────────┬─────────────────────────────────────────────┘
|
||
│ HTTP/HTTPS (Gitea Actions API)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ WINDOWS HOST (i9-10900X · 64 GB RAM · NVMe SSD) │
|
||
│ │
|
||
│ ┌───────────────────┐ polls job queue ┌──────────────────┐ │
|
||
│ │ Gitea Server │◄────────────────────────►│ act_runner │ │
|
||
│ │ (Actions API) │ reports status/logs │ (Windows svc) │ │
|
||
│ └───────────────────┘ └────────┬─────────┘ │
|
||
│ │ │
|
||
│ calls Invoke-CIJob.ps1 │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Orchestrator Scripts (PowerShell) │ │
|
||
│ │ │ │
|
||
│ │ New-BuildVM.ps1 ──► vmrun.exe (linked clone from template) │ │
|
||
│ Wait-VMReady.ps1 ─► WinRM poll (Windows) / SSH poll (Linux) │ │
|
||
│ Invoke-RemoteBuild.ps1 ► PSSession (Windows) / SSH+SCP (Linux) │ │
|
||
│ Get-BuildArtifacts.ps1 ► Copy-Item -FromSession / scp (Linux) │ │
|
||
│ Remove-BuildVM.ps1 ───► vmrun stop + deleteVM │ │
|
||
│ _Transport.psm1 ──────► Invoke-SshCommand / Copy-SshItem │ │
|
||
│ └───────────────────────────────┬───────────────────────────────────┘ │
|
||
│ │ vmrun.exe (VMware CLI) │
|
||
│ ▼ │
|
||
│ ┌───────────────────────────────────────────────────────────────────┐ │
|
||
│ │ VMware Workstation │ │
|
||
│ │ │ │
|
||
│ │ Template VM ──snapshot "BaseClean"──┐ │ │
|
||
│ │ ├── Clone_Job_001.vmx │ │
|
||
│ │ ├── Clone_Job_002.vmx │ │
|
||
│ │ └── Clone_Job_003.vmx │ │
|
||
│ └───────────────────────────────────────────────────────────────────┘ │
|
||
│ │ WinRM :5986 HTTPS (192.168.79.0/24 NAT) │
|
||
└──────────────────────────────────┼──────────────────────────────────────┘
|
||
│
|
||
┌────────────────────────┼──────────────────────────┐
|
||
│ │ │
|
||
▼ ▼ ▼
|
||
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
|
||
│ Build VM #1 │ │ Build VM #2 │ │ Build VM #N │
|
||
│ (ephemeral) │ │ (ephemeral) │ │ (ephemeral) │
|
||
│ │ │ │ │ │
|
||
│ VS Build Tools │ │ VS Build Tools │ │ VS Build Tools │
|
||
│ 2026 (v145) │ │ 2026 (v145) │ │ 2026 (v145) │
|
||
│ .NET SDK 10 │ │ .NET SDK 10 │ │ .NET SDK 10 │
|
||
│ Python 3.13.3 │ │ Python 3.13.3 │ │ Python 3.13.3 │
|
||
│ WinRM enabled │ │ WinRM enabled │ │ WinRM enabled │
|
||
│ │ │ │ │ │
|
||
│ > unzip src │ │ > unzip src │ │ > unzip src │
|
||
│ > python build │ │ > python build │ │ > python build │
|
||
│ > dist/ │ │ > dist/ │ │ > dist/ │
|
||
│ │ │ │ │ │
|
||
│ [DESTROYED] │ │ [DESTROYED] │ │ [DESTROYED] │
|
||
└──────────────────┘ └──────────────────┘ └──────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Component Descriptions
|
||
|
||
### Gitea (Self-hosted Git + CI)
|
||
- Hosts Git repositories
|
||
- Provides Gitea Actions (GitHub Actions-compatible YAML workflow engine)
|
||
- Manages job queue and reports build status to commits/PRs
|
||
- Stores build artifacts uploaded by act_runner
|
||
|
||
### act_runner (Job Executor)
|
||
- Runs as a Windows service on the host
|
||
- Polls Gitea Actions API for pending jobs
|
||
- Executes `.gitea/workflows/*.yml` steps
|
||
- **Configured with `container.enabled: false`** — does NOT use Docker
|
||
- Each step runs as a PowerShell command on the host (orchestration only)
|
||
- Supports `capacity: N` for concurrent job execution (N parallel VMs)
|
||
|
||
### VMware Workstation + vmrun CLI
|
||
- Manages the template VM and all ephemeral clones
|
||
- `vmrun.exe` is the command-line interface for all VM lifecycle operations
|
||
- **Linked clones** are used: each build VM is a delta clone from a frozen template snapshot
|
||
- Clone creation: ~5–15 seconds (vs 60–120 seconds for full clone)
|
||
- Disk footprint: ~2–5 GB per clone (vs 40–80 GB for full clone)
|
||
- Tradeoff: template snapshot must remain intact
|
||
|
||
### WinRM / PowerShell Remoting (Windows VMs)
|
||
- Transport for remote build execution inside **Windows** build VMs
|
||
- Port: **5986** (HTTPS, self-signed cert, `AllowUnencrypted=false`)
|
||
- Authentication: **Basic over HTTPS** (`-SkipCACheck` for lab self-signed cert)
|
||
- Used for:
|
||
- Transferring source: `Compress-Archive` on host → WinRM copy → `Expand-Archive` in guest
|
||
- Running build commands (`Invoke-Command`)
|
||
- Packaging artifacts (`Compress-Archive` in guest) and collecting via `Copy-Item -FromSession`
|
||
|
||
### SSH / SCP Transport (Linux VMs)
|
||
- Transport for remote build execution inside **Linux** build VMs
|
||
- Port: **22** (key-based auth, ed25519 key `F:\CI\keys\ci_linux`, user `ci_build`)
|
||
- No password — `BatchMode=yes`, `StrictHostKeyChecking=accept-new`
|
||
- Implemented in `scripts/_Transport.psm1` (exported: `Invoke-SshCommand`, `Copy-SshItem`, `Test-SshReady`)
|
||
- Used by `Wait-VMReady.ps1 -Transport SSH`, `Invoke-RemoteBuild.ps1 -GuestOS Linux`, `Get-BuildArtifacts.ps1 -GuestOS Linux`
|
||
- Selected automatically when `Invoke-CIJob.ps1` detects `guestOS = ubuntu-64` in the cloned VMX
|
||
|
||
### Build Toolchain (inside VM only)
|
||
- Visual Studio Build Tools **2026** (MSBuild 18.5.4 / toolset v145)
|
||
- .NET SDK **10.0.203**
|
||
- Python **3.13.3** (per build script personalizzati come `build_plugin.py`)
|
||
- NuGet CLI (optional, dotnet restore covers most cases)
|
||
- **Git for Windows 2.54** + **7-Zip 26.01** (Tier-1 toolchain, §6.6 — DONE 2026-05-10): consente l'opt-in `-UseGitClone` di `Invoke-CIJob.ps1` per clonare il repository direttamente in VM (skip host zip transfer, e2e PASS, -25.7% durata pipeline).
|
||
|
||
---
|
||
|
||
## Network Topology
|
||
|
||
```
|
||
Host NIC (LAN) → 192.168.1.x → Gitea server (10.10.20.11:3100), developer machines
|
||
VMware VMnet8 (NAT) → 192.168.79.0/24 → Build VMs (DHCP, internet access via NAT)
|
||
|
||
Build VMs:
|
||
Clone_Job_001 → 192.168.79.131
|
||
Clone_Job_002 → 192.168.79.132
|
||
...
|
||
Clone_Job_N → 192.168.79.1xx
|
||
|
||
WinRM port 5986 (HTTPS, self-signed) on each Windows VM IP.
|
||
SSH port 22 on each Linux VM IP (key-based, ci_build@<ip>, key: F:\CI\keys\ci_linux).
|
||
VMs have internet access via NAT — required for pip/nuget/apt during build.
|
||
```
|
||
|
||
> **Note:** VMnet8 NAT gives VMs internet access. The host detects the VM IP via
|
||
> `vmrun getGuestIPAddress` after boot — no static IP assignment needed.
|
||
|
||
---
|
||
|
||
## VM Lifecycle
|
||
|
||
```
|
||
1. TEMPLATE (static, locked)
|
||
└─ Snapshot: "BaseClean" ───────────────────────┐
|
||
│
|
||
2. CLONE CREATION │
|
||
vmrun clone <template.vmx> <clone.vmx> linked ──►│
|
||
Duration: ~5–15 seconds │
|
||
│
|
||
3. START │
|
||
vmrun start <clone.vmx> nogui │
|
||
Duration: ~20–40 seconds boot │
|
||
│
|
||
4. READINESS CHECK │
|
||
Windows: getState=running → ping → Test-WSMan │
|
||
Linux: getState=running → TCP:22 → ssh echo │
|
||
Duration: ~30–90 seconds total │
|
||
│
|
||
5. BUILD │
|
||
Copy script → Invoke-Command → collect artifacts │
|
||
Duration: varies (seconds to minutes) │
|
||
│
|
||
6. DESTROY │
|
||
vmrun stop hard -> vmrun deleteVM → Remove-Item │
|
||
Duration: ~5–10 seconds │
|
||
Result: zero disk footprint remaining ▼
|
||
```
|
||
|
||
---
|
||
|
||
## Resource Limits & Parallel Capacity
|
||
|
||
| Resource | Host Total | Per VM (est.) | Max VMs |
|
||
| --------------- | ---------------------- | --------------- | ---------------- |
|
||
| RAM | 64 GB | ~6–8 GB | 8–10 |
|
||
| CPU | 20 threads (i9-10900X) | ~4 vCPU | 5 |
|
||
| NVMe IOPS | High | Clone delta I/O | ~6–8 |
|
||
| **Recommended** | — | — | **4 concurrent** |
|
||
|
||
`act_runner` `capacity: 4` is the conservative default. Raise to 6 after benchmarking.
|
||
|
||
---
|
||
|
||
## Security Boundaries
|
||
|
||
- Build VMs are on VMnet8 NAT (192.168.79.0/24) — host reaches VMs, VMs have internet access
|
||
- WinRM credentials are stored in Windows Credential Manager on host (not hardcoded)
|
||
- Each VM is fully destroyed after the job — no state carries between builds
|
||
- Runner host does not expose any ports to the build VM network
|
||
- Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time.
|
||
- With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model.
|
||
- Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM.
|
||
|
||
---
|
||
|
||
## Python orchestrator (Phase A)
|
||
|
||
Phase A of [implementation-plan-A-B](../plans/implementation-plan-A-B.md)
|
||
replaces the PowerShell orchestration scripts with a Python 3.11+ package
|
||
under `src/ci_orchestrator/`. The PS scripts in `scripts/` are now thin
|
||
3-line shims that forward to the Python CLI — the diagram above still
|
||
applies, except that the boxes labelled `*.ps1` are entry-point names,
|
||
not the implementation.
|
||
|
||
### Package layout
|
||
|
||
```
|
||
src/ci_orchestrator/
|
||
├── __init__.py
|
||
├── __main__.py # click entry point: `python -m ci_orchestrator`
|
||
├── config.py # Config dataclass + env/TOML loader (OS-aware paths)
|
||
├── credentials.py # CredentialStore Protocol + KeyringCredentialStore
|
||
│
|
||
├── backends/
|
||
│ ├── __init__.py # load_backend(config) factory (selector for Phase C)
|
||
│ ├── protocol.py # VmBackend Protocol + VmHandle dataclass + VmState enum
|
||
│ ├── workstation.py # WorkstationVmrunBackend (subprocess + vmrun -T ws)
|
||
│ └── errors.py # BackendError / BackendNotAvailable / BackendOperationFailed
|
||
│
|
||
├── transport/
|
||
│ ├── __init__.py
|
||
│ ├── winrm.py # WinRM transport via pypsrp (replaces New-PSSession)
|
||
│ ├── ssh.py # SSH/SFTP transport via paramiko (replaces ssh.exe/scp.exe)
|
||
│ └── errors.py # TransportConnectError / TransportCommandError
|
||
│
|
||
└── commands/ # One module per `click` sub-command group
|
||
├── wait.py # `wait-ready`
|
||
├── vm.py # `vm new` / `vm remove` / `vm cleanup`
|
||
├── build.py # `build run`
|
||
├── artifacts.py # `artifacts collect`
|
||
├── monitor.py # `monitor disk` / `monitor runner`
|
||
├── report.py # `report job`
|
||
└── job.py # `job` — full end-to-end orchestrator
|
||
```
|
||
|
||
### `VmBackend` Protocol and the `load_backend(config)` factory
|
||
|
||
`backends/protocol.py` defines a hypervisor-agnostic Protocol:
|
||
|
||
```python
|
||
class VmBackend(Protocol):
|
||
def clone_linked(self, template, snapshot, name, destination=None) -> VmHandle: ...
|
||
def start(self, handle, headless=True) -> None: ...
|
||
def stop(self, handle, hard=False) -> None: ...
|
||
def delete(self, handle) -> None: ...
|
||
def get_ip(self, handle, timeout=0.0) -> str | None: ...
|
||
def list_snapshots(self, handle) -> list[str]: ...
|
||
def is_running(self, handle) -> bool: ...
|
||
```
|
||
|
||
Names are deliberately neutral (no `vmrun_*` prefixes) and identifiers are
|
||
opaque strings (`VmHandle.identifier`): a VMX path on Workstation, a
|
||
managed object reference on ESXi. `is_running` MUST NOT use guest tools
|
||
(see `AGENTS.md` error #10).
|
||
|
||
Backends are constructed via `backends.load_backend(config)`. Application
|
||
code (most importantly `commands/job.py`) imports the factory only — never
|
||
`WorkstationVmrunBackend` directly. The factory reads
|
||
`config.backend.type` from `config.toml` (default: `"workstation"`).
|
||
|
||
### Phase C extension point — adding ESXi without forks
|
||
|
||
To add an ESXi backend in a future Phase C:
|
||
|
||
1. Add a new module `backends/esxi.py` implementing the `VmBackend`
|
||
Protocol via `pyVmomi`. `VmHandle.identifier` becomes the managed
|
||
object reference; `clone_linked` becomes a `CloneVM_Task` call;
|
||
`start` becomes `PowerOnVM_Task`.
|
||
2. Register it in `backends/__init__.py` `load_backend(config)` under a
|
||
new `config.backend.type == "esxi"` branch.
|
||
3. Add `[backend]` keys to `config.example.toml` (host, user, datacenter,
|
||
datastore, …). The `Config` dataclass already namespaces backend
|
||
options under `config.backend`.
|
||
4. Transport layers (`winrm.py`, `ssh.py`) and the `commands/*` modules
|
||
need no change — they consume `VmBackend` and `VmHandle` only.
|
||
|
||
The single non-trivial constraint is that `commands/vm.py` `vm new`
|
||
currently builds the destination VMX path under `--clone-base-dir` (a
|
||
host-local concept). For ESXi this becomes a datastore folder; the
|
||
parameter rename can be deferred to Phase C as long as the backend
|
||
ignores the value and uses `config.backend.datastore` instead.
|
||
|
||
### Transport selection
|
||
|
||
`commands/job.py` auto-detects the guest OS by parsing `guestOS = "..."`
|
||
in the cloned VMX (`ubuntu-*` → Linux/SSH, anything else → Windows/WinRM).
|
||
The Linux branch is wired to `transport/ssh.py` (paramiko) and the
|
||
Windows branch to `transport/winrm.py` (pypsrp). Both transports raise
|
||
typed errors (`TransportConnectError`, `TransportCommandError`) — never
|
||
bare exceptions.
|
||
|
||
### Validation gates (CI)
|
||
|
||
Enforced by `gitea/workflows/lint.yml` job `python`:
|
||
|
||
```powershell
|
||
python -m ruff check src tests\python
|
||
python -m mypy --strict src
|
||
python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=90
|
||
```
|
||
|
||
Coverage gate is **90%**. Regression tests for
|
||
`AGENTS.md` errors #9–#12 live in
|
||
[tests/python/test_agents_errors.py](../tests/python/test_agents_errors.py).
|
||
|