# Architecture — Local CI/CD System

## Overview

A fully local, isolated CI/CD pipeline running on a Windows host using:
- **Gitea** as the self-hosted Git server + CI platform
- **act_runner** as the job executor (runs on the Windows host)
- **VMware Workstation** for ephemeral build VMs (linked clones)
- **WinRM / PowerShell Remoting** for communication with Windows build VMs
- **SSH / SCP** (`ssh.exe` + `scp.exe`) for communication with Linux build VMs
- **MSBuild / dotnet CLI** (Windows) and **GCC / Clang / CMake** (Linux) executing inside guest VMs only

The runner host **never executes build tools directly**. Its only role is orchestration: creating, driving, and destroying VMs.

---

## System Diagram

```
┌─────────────────────────────────────────────────────────────────────────┐
│  DEVELOPER WORKSTATION                                                  │
│                                                                         │
│  git push → Gitea (self-hosted, LAN)                                    │
└───────────────────────────┬─────────────────────────────────────────────┘
                            │  HTTP/HTTPS (Gitea Actions API)
                            ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  WINDOWS HOST  (i9-10900X · 64 GB RAM · NVMe SSD)                       │
│                                                                         │
│  ┌───────────────────┐     polls job queue      ┌──────────────────┐    │
│  │   Gitea Server    │◄────────────────────────►│   act_runner     │    │
│  │  (Actions API)    │   reports status/logs    │  (Windows svc)   │    │
│  └───────────────────┘                          └────────┬─────────┘    │
│                                                          │              │
│                                           calls Invoke-CIJob.ps1        │
│                                                          │              │
│                                                          ▼              │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │  Orchestrator Scripts (PowerShell)                                │  │
│  │                                                                   │  │
│  │  New-BuildVM.ps1 ──► vmrun.exe (linked clone from template)       │  │
  │  Wait-VMReady.ps1 ─► WinRM poll (Windows) / SSH poll (Linux)      │  │
  │  Invoke-RemoteBuild.ps1 ► PSSession (Windows) / SSH+SCP (Linux)   │  │
  │  Get-BuildArtifacts.ps1 ► Copy-Item -FromSession / scp (Linux)    │  │
  │  Remove-BuildVM.ps1 ───► vmrun stop + deleteVM                    │  │
  │  _Transport.psm1 ──────► Invoke-SshCommand / Copy-SshItem         │  │
│  └───────────────────────────────┬───────────────────────────────────┘  │
│                                  │  vmrun.exe (VMware CLI)              │
│                                  ▼                                      │
│  ┌───────────────────────────────────────────────────────────────────┐  │
│  │  VMware Workstation                                               │  │
│  │                                                                   │  │
│  │  Template VM ──snapshot "BaseClean"──┐                            │  │
│  │                                      ├── Clone_Job_001.vmx        │  │
│  │                                      ├── Clone_Job_002.vmx        │  │
│  │                                      └── Clone_Job_003.vmx        │  │
│  └───────────────────────────────────────────────────────────────────┘  │
│                                  │  WinRM :5986 HTTPS (192.168.79.0/24 NAT) │
└──────────────────────────────────┼──────────────────────────────────────┘
                                   │
          ┌────────────────────────┼──────────────────────────┐
          │                        │                          │
          ▼                        ▼                          ▼
┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Build VM #1     │    │  Build VM #2     │    │  Build VM #N     │
│  (ephemeral)     │    │  (ephemeral)     │    │  (ephemeral)     │
│                  │    │                  │    │                  │
│  VS Build Tools  │    │  VS Build Tools  │    │  VS Build Tools  │
│  2026 (v145)     │    │  2026 (v145)     │    │  2026 (v145)     │
│  .NET SDK 10     │    │  .NET SDK 10     │    │  .NET SDK 10     │
│  Python 3.13.3   │    │  Python 3.13.3   │    │  Python 3.13.3   │
│  WinRM enabled   │    │  WinRM enabled   │    │  WinRM enabled   │
│                  │    │                  │    │                  │
│  > unzip src     │    │  > unzip src     │    │  > unzip src     │
│  > python build  │    │  > python build  │    │  > python build  │
│  > dist/         │    │  > dist/         │    │  > dist/         │
│                  │    │                  │    │                  │
│  [DESTROYED]     │    │  [DESTROYED]     │    │  [DESTROYED]     │
└──────────────────┘    └──────────────────┘    └──────────────────┘
```

---

## Component Descriptions

### Gitea (Self-hosted Git + CI)
- Hosts Git repositories
- Provides Gitea Actions (GitHub Actions-compatible YAML workflow engine)
- Manages job queue and reports build status to commits/PRs
- Stores build artifacts uploaded by act_runner

### act_runner (Job Executor)
- Runs as a Windows service on the host
- Polls Gitea Actions API for pending jobs
- Executes `.gitea/workflows/*.yml` steps
- **Configured with `container.enabled: false`** — does NOT use Docker
- Each step runs as a PowerShell command on the host (orchestration only)
- Supports `capacity: N` for concurrent job execution (N parallel VMs)

### VMware Workstation + vmrun CLI
- Manages the template VM and all ephemeral clones
- `vmrun.exe` is the command-line interface for all VM lifecycle operations
- **Linked clones** are used: each build VM is a delta clone from a frozen template snapshot
  - Clone creation: ~5–15 seconds (vs 60–120 seconds for full clone)
  - Disk footprint: ~2–5 GB per clone (vs 40–80 GB for full clone)
  - Tradeoff: template snapshot must remain intact

### WinRM / PowerShell Remoting (Windows VMs)
- Transport for remote build execution inside **Windows** build VMs
- Port: **5986** (HTTPS, self-signed cert, `AllowUnencrypted=false`)
- Authentication: **Basic over HTTPS** (`-SkipCACheck` for lab self-signed cert)
- Used for:
  - Transferring source: `Compress-Archive` on host → WinRM copy → `Expand-Archive` in guest
  - Running build commands (`Invoke-Command`)
  - Packaging artifacts (`Compress-Archive` in guest) and collecting via `Copy-Item -FromSession`

### SSH / SCP Transport (Linux VMs)
- Transport for remote build execution inside **Linux** build VMs
- Port: **22** (key-based auth, ed25519 key `F:\CI\keys\ci_linux`, user `ci_build`)
- No password — `BatchMode=yes`, `StrictHostKeyChecking=accept-new`
- Implemented in `scripts/_Transport.psm1` (exported: `Invoke-SshCommand`, `Copy-SshItem`, `Test-SshReady`)
- Used by `Wait-VMReady.ps1 -Transport SSH`, `Invoke-RemoteBuild.ps1 -GuestOS Linux`, `Get-BuildArtifacts.ps1 -GuestOS Linux`
- Selected automatically when `Invoke-CIJob.ps1` detects `guestOS = ubuntu-64` in the cloned VMX

### Build Toolchain (inside VM only)
- Visual Studio Build Tools **2026** (MSBuild 18.5.4 / toolset v145)
- .NET SDK **10.0.203**
- Python **3.13.3** (per build script personalizzati come `build_plugin.py`)
- NuGet CLI (optional, dotnet restore covers most cases)
- **Git for Windows 2.54** + **7-Zip 26.01** (Tier-1 toolchain, §6.6 — DONE 2026-05-10): consente l'opt-in `-UseGitClone` di `Invoke-CIJob.ps1` per clonare il repository direttamente in VM (skip host zip transfer, e2e PASS, -25.7% durata pipeline).

---

## Network Topology

```
Host NIC (LAN)       → 192.168.1.x     → Gitea server (10.10.20.11:3100), developer machines
VMware VMnet8 (NAT)  → 192.168.79.0/24 → Build VMs (DHCP, internet access via NAT)

Build VMs:
  Clone_Job_001 → 192.168.79.131
  Clone_Job_002 → 192.168.79.132
  ...
  Clone_Job_N   → 192.168.79.1xx

WinRM port 5986 (HTTPS, self-signed) on each Windows VM IP.
SSH port 22 on each Linux VM IP (key-based, ci_build@<ip>, key: F:\CI\keys\ci_linux).
VMs have internet access via NAT — required for pip/nuget/apt during build.
```

> **Note:** VMnet8 NAT gives VMs internet access. The host detects the VM IP via
> `vmrun getGuestIPAddress` after boot — no static IP assignment needed.

---

## VM Lifecycle

```
1. TEMPLATE (static, locked)
   └─ Snapshot: "BaseClean"  ───────────────────────┐
                                                    │
2. CLONE CREATION                                   │
   vmrun clone <template.vmx> <clone.vmx> linked ──►│
   Duration: ~5–15 seconds                          │
                                                    │
3. START                                            │
   vmrun start <clone.vmx> nogui                    │
   Duration: ~20–40 seconds boot                    │
                                                    │
4. READINESS CHECK                                  │
   Windows: getState=running → ping → Test-WSMan    │
   Linux:   getState=running → TCP:22 → ssh echo    │
   Duration: ~30–90 seconds total                   │
                                                    │
5. BUILD                                            │
   Copy script → Invoke-Command → collect artifacts │
   Duration: varies (seconds to minutes)            │
                                                    │
6. DESTROY                                          │
   vmrun stop hard -> vmrun deleteVM → Remove-Item  │
   Duration: ~5–10 seconds                          │
   Result: zero disk footprint remaining            ▼
```

---

## Resource Limits & Parallel Capacity

| Resource        | Host Total             | Per VM (est.)   | Max VMs          |
| --------------- | ---------------------- | --------------- | ---------------- |
| RAM             | 64 GB                  | ~6–8 GB         | 8–10             |
| CPU             | 20 threads (i9-10900X) | ~4 vCPU         | 5                |
| NVMe IOPS       | High                   | Clone delta I/O | ~6–8             |
| **Recommended** | —                      | —               | **4 concurrent** |

`act_runner` `capacity: 4` is the conservative default. Raise to 6 after benchmarking.

---

## Security Boundaries

- Build VMs are on VMnet8 NAT (192.168.79.0/24) — host reaches VMs, VMs have internet access
- WinRM credentials are stored in Windows Credential Manager on host (not hardcoded)
- Each VM is fully destroyed after the job — no state carries between builds
- Runner host does not expose any ports to the build VM network
- Source code is transferred via zip (Compress-Archive → WinRM copy → Expand-Archive in guest) by default — no PAT is ever present inside the VM; VMs have NAT internet for pip/nuget at build time.
- With `-UseGitClone` (§3.3, opt-in) il clone avviene dentro la VM via SSH alias `gitea-ci` (forwardato dall'host); la chiave privata SSH dell'host non viene esposta al guest — vedi BEST-PRACTICES.md per il threat model.
- Linux build VMs usano la chiave SSH `F:\CI\keys\ci_linux` (ed25519, no passphrase), archiviata solo sull'host. Il guest non conosce credenziali WinRM.

---

## Python orchestrator (Phase A)

Phase A of [implementation-plan-A-B](../plans/implementation-plan-A-B.md)
replaces the PowerShell orchestration scripts with a Python 3.11+ package
under `src/ci_orchestrator/`. The PS scripts in `scripts/` are now thin
3-line shims that forward to the Python CLI — the diagram above still
applies, except that the boxes labelled `*.ps1` are entry-point names,
not the implementation.

### Package layout

```
src/ci_orchestrator/
├── __init__.py
├── __main__.py                # click entry point: `python -m ci_orchestrator`
├── config.py                  # Config dataclass + env/TOML loader (OS-aware paths)
├── credentials.py             # CredentialStore Protocol + KeyringCredentialStore
│
├── backends/
│   ├── __init__.py            # load_backend(config) factory (selector for Phase C)
│   ├── protocol.py            # VmBackend Protocol + VmHandle dataclass + VmState enum
│   ├── workstation.py         # WorkstationVmrunBackend (subprocess + vmrun -T ws)
│   └── errors.py              # BackendError / BackendNotAvailable / BackendOperationFailed
│
├── transport/
│   ├── __init__.py
│   ├── winrm.py               # WinRM transport via pypsrp (replaces New-PSSession)
│   ├── ssh.py                 # SSH/SFTP transport via paramiko (replaces ssh.exe/scp.exe)
│   └── errors.py              # TransportConnectError / TransportCommandError
│
└── commands/                  # One module per `click` sub-command group
    ├── wait.py                # `wait-ready`
    ├── vm.py                  # `vm new` / `vm remove` / `vm cleanup`
    ├── build.py               # `build run`
    ├── artifacts.py           # `artifacts collect`
    ├── monitor.py             # `monitor disk` / `monitor runner`
    ├── report.py              # `report job`
    └── job.py                 # `job` — full end-to-end orchestrator
```

### `VmBackend` Protocol and the `load_backend(config)` factory

`backends/protocol.py` defines a hypervisor-agnostic Protocol:

```python
class VmBackend(Protocol):
    def clone_linked(self, template, snapshot, name, destination=None) -> VmHandle: ...
    def start(self, handle, headless=True) -> None: ...
    def stop(self, handle, hard=False) -> None: ...
    def delete(self, handle) -> None: ...
    def get_ip(self, handle, timeout=0.0) -> str | None: ...
    def list_snapshots(self, handle) -> list[str]: ...
    def is_running(self, handle) -> bool: ...
```

Names are deliberately neutral (no `vmrun_*` prefixes) and identifiers are
opaque strings (`VmHandle.identifier`): a VMX path on Workstation, a
managed object reference on ESXi. `is_running` MUST NOT use guest tools
(see `AGENTS.md` error #10).

Backends are constructed via `backends.load_backend(config)`. Application
code (most importantly `commands/job.py`) imports the factory only — never
`WorkstationVmrunBackend` directly. The factory reads
`config.backend.type` from `config.toml` (default: `"workstation"`).

### Phase C extension point — adding ESXi without forks

To add an ESXi backend in a future Phase C:

1. Add a new module `backends/esxi.py` implementing the `VmBackend`
   Protocol via `pyVmomi`. `VmHandle.identifier` becomes the managed
   object reference; `clone_linked` becomes a `CloneVM_Task` call;
   `start` becomes `PowerOnVM_Task`.
2. Register it in `backends/__init__.py` `load_backend(config)` under a
   new `config.backend.type == "esxi"` branch.
3. Add `[backend]` keys to `config.example.toml` (host, user, datacenter,
   datastore, …). The `Config` dataclass already namespaces backend
   options under `config.backend`.
4. Transport layers (`winrm.py`, `ssh.py`) and the `commands/*` modules
   need no change — they consume `VmBackend` and `VmHandle` only.

The single non-trivial constraint is that `commands/vm.py` `vm new`
currently builds the destination VMX path under `--clone-base-dir` (a
host-local concept). For ESXi this becomes a datastore folder; the
parameter rename can be deferred to Phase C as long as the backend
ignores the value and uses `config.backend.datastore` instead.

### Transport selection

`commands/job.py` auto-detects the guest OS by parsing `guestOS = "..."`
in the cloned VMX (`ubuntu-*` → Linux/SSH, anything else → Windows/WinRM).
The Linux branch is wired to `transport/ssh.py` (paramiko) and the
Windows branch to `transport/winrm.py` (pypsrp). Both transports raise
typed errors (`TransportConnectError`, `TransportCommandError`) — never
bare exceptions.

### Validation gates (CI)

Enforced by `gitea/workflows/lint.yml` job `python`:

```powershell
python -m ruff check src tests\python
python -m mypy --strict src
python -m pytest tests\python -q --cov=ci_orchestrator --cov-fail-under=80
```

Coverage gate is **80%** as of Phase A5. Regression tests for
`AGENTS.md` errors #9–#12 live in
[tests/python/test_agents_errors.py](../tests/python/test_agents_errors.py).