local-ci-cd-system/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with this repository.

---

Project Overview

Self-hosted CI/CD system running on a Windows 11 host with VMware Workstation. Each build job runs inside an ephemeral VM (linked clone from a frozen template snapshot), communicates via WinRM (Windows guests) or SSH (Linux guests), and is destroyed after the job.

Current state: The Python orchestrator (src/ci_orchestrator/) is the production entry point. The PowerShell scripts in scripts/ are 3-line shims that forward to the Python CLI. All new logic goes in Python.

Phase C (PowerShell-free Linux host): the manual burn-in / benchmark / smoke / host-validation / credential-setup tooling that used to run on the Linux host via pwsh is now native Python: bench run, bench measure, smoke run, validate host, validate guest, creds set. The Linux host no longer needs pwsh for any operation — it may be uninstalled (setup-host-linux.sh no longer installs it by default; opt in with --with-pwsh). PSScriptAnalyzer lint runs on a Windows runner on-demand, not on the Linux runner.

---

Development Commands

Python (primary)

The production venv is at /opt/ci/venv (Linux host). To invoke the CLI as the service user:

sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help

# Dev venv setup (one-time)
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"

# Run all tests with coverage (gate: ≥90%)
python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90

# Run a single test file
python -m pytest tests/python/test_commands_job.py -q

# Lint
python -m ruff check src tests/python

# Type-check (strict)
python -m mypy --strict src

# CLI
python -m ci_orchestrator --help

Important: The production venv at F:\CI\python\venv must be installed non-editable (pip install ., not -e), because act_runner runs as LocalSystem and cannot resolve editable .pth files from a transient RunnerWork path. Re-run pip install . into the production venv after every code change. The dev .venv may use -e.

PowerShell linting

Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Settings PSScriptAnalyzerSettings.psd1 -Severity Error,Warning
# Auto-fix formatting:
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Fix

---

Architecture

Pipeline flow

git push → Gitea Actions → act_runner (Windows service) → python -m ci_orchestrator job
    → vm new (linked clone)
    → wait-ready (WinRM poll or SSH poll)
    → build run (WinRM Invoke-Command or SSH+SFTP)
    → artifacts collect (WinRM Copy-Item or SCP)
    → vm remove (vmrun stop + deleteVM)

The host never executes build tools directly — it only orchestrates VM lifecycle.

Python package layout (src/ci_orchestrator/)

Module	Role
__main__.py	click entry point; registers all sub-command groups
config.py	Config dataclass + TOML/env loader; resolution: OS defaults → config.toml → env vars
credentials.py	CredentialStore Protocol + KeyringCredentialStore
backends/protocol.py	VmBackend Protocol + VmHandle + VmState (hypervisor-agnostic)
backends/workstation.py	WorkstationVmrunBackend — wraps vmrun.exe subprocesses
backends/__init__.py	load_backend(config) factory — the only import application code should use
transport/winrm.py	WinRM transport via pypsrp
transport/ssh.py	SSH/SFTP transport via paramiko
transport/errors.py	TransportConnectError, TransportCommandError
commands/job.py	Full end-to-end orchestrator (auto-detects guest OS from VMX guestOS field)
commands/vm.py	vm new / remove / cleanup
commands/build.py	build run
commands/artifacts.py	artifacts collect
commands/wait.py	wait-ready
commands/monitor.py	monitor disk / runner
commands/report.py	report job
operations/vms.py, cache.py, host.py, audit.py, executions.py	Pure logic layer shared by the CLI and the web UI — dataclasses only, zero click/print. commands/* and web/routes.py both call these; never duplicate logic in either caller.
commands/web.py	web serve — ci web serve --host --port launches the dashboard (uvicorn/fastapi imported lazily so the core CLI stays dep-light without the [web] extra)
web/app.py, auth.py, routes.py, security.py	FastAPI dashboard: recent-executions/cache/disk/runner view, Gitea OIDC login (Authlib, fail-closed allowlist), CSRF-protected confirm-gated mutating actions (VM remove/cleanup, cache purge, template backup), JSONL audit trail. In-process only — imports operations/ + backends.load_backend directly, never shells the CLI. See docs/WEB-UI.md.

Key design rule: Application code (especially commands/job.py) must import backends.load_backend only — never WorkstationVmrunBackend directly. This keeps the Phase D ESXi extension path open.

Web management dashboard

ci web serve (package web/, optional [web] extra) is a server-rendered FastAPI dashboard for VM/host management — see docs/WEB-UI.md for install, Gitea OIDC setup, and operations, and plans/web-ui-management-plan.md for the design history (steps W0–W7, all closed). Deployed live on this host as ci-web.service (systemd, ci-runner user), bound to the LAN with Gitea OIDC auth active — never bind 0.0.0.0, and never bind non-loopback without [web.oidc] configured (enforced at startup by commands/web.py).

The dashboard's main table is the Executions card (operations/executions.py, GET /executions): recent Gitea Actions runs, live ones included, with the build VMs grouped under the run that spawned them (job id {run_id}-{run_attempt} ↔ clone name). It reuses the operator's OIDC access token for the Gitea API — held server-side only in app.state.gitea_tokens, never in the session cookie, which is signed but not encrypted and travels over plain HTTP on the LAN. Optional keyring PAT fallback: [web] gitea_token_cred.

Transport selection

commands/job.py reads guestOS = "..." from the cloned VMX. ubuntu-* → Linux/SSH (transport/ssh.py); anything else → Windows/WinRM (transport/winrm.py).

VM templates and snapshots

Template	VMX path	Snapshot	Transport
WinBuild2025	F:\CI\Templates\WinBuild2025\WinBuild2025.vmx	BaseClean	WinRM HTTPS/5986
WinBuild2022	F:\CI\Templates\WinBuild2022\WinBuild2022.vmx	BaseClean	WinRM HTTPS/5986
LinuxBuild2404	F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx	BaseClean-Linux	SSH/22 (ci_build, key F:\CI\keys\ci_linux)
LinuxRust2404	LinuxRust2404/LinuxRust2404.vmx (templates dir)	BaseClean-LinuxRust	SSH/22 (same user/key as LinuxBuild2404)

LinuxRust2404 is a LinuxBuild2404 variant for Rust workspaces (Tauri): ci template deploy-rust / ci template prepare-rust provision it via template/Install-CIToolchain-LinuxRust2404.sh (webview/GTK libs + system-wide rustup under /opt/rust + cargo-audit binary; the audit advisory DB is fetched at runtime, never baked). --rust-channel pins the baked toolchain (default stable, a moving target).

Snapshots must be taken from powered-off state — presence of .vmem in the template dir means the snapshot captured running memory and vmrun clone ... linked will fail.

Configuration

The config file is auto-discovered at $CI_ROOT/config.toml (Linux default: /var/lib/ci/config.toml; Windows: F:\CI\config.toml). Set $CI_CONFIG to override. The file at /var/lib/ci/config.toml is the live config on the Linux host — keep its [paths] section pointing to /var/lib/ci/... (not the old Windows F:/CI/... paths). Environment variables CI_ROOT, CI_TEMPLATES, CI_BUILD_VMS, CI_ARTIFACTS, CI_KEYS override the file. CI_PYTHON_LAUNCHER in runner/config.yaml is the single source of truth for the system Python path used to bootstrap the production venv.

---

PowerShell 5.1 Constraints

These constraints apply to the .ps1 / .psm1 scripts that still run inside Windows guests and on the optional Windows runner. The Linux host itself no longer runs any .ps1 (Phase C) — do not add PowerShell to the Linux host orchestration path.

All .ps1 / .psm1 scripts run on Windows PowerShell 5.1 (not Core/7). Every script must start with:

#Requires -Version 5.1
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'

Forbidden PS 7+ syntax (use the PS 5.1 alternatives):

Forbidden	Use instead
$x ??= 'val'	if ($null -eq $x) { $x = 'val' }
$a ?? $b	if ($null -ne $a) { $a } else { $b }
cmd1 && cmd2	cmd1; if ($LASTEXITCODE -eq 0) { cmd2 }
Ternary $a ? $b : $c	if ($a) { $b } else { $c }
ForEach-Object -Parallel	foreach or Start-Job

Always check $LASTEXITCODE after vmrun, ssh, scp and throw if non-zero.

---

Python Conventions

• click.echo everywhere — no bare print().
• No subprocess.run(..., shell=True). Use paramiko for SSH, pypsrp for WinRM, keyring for credentials.
• No global variables — state lives in Config and objects passed explicitly.
• All public functions and exported helpers must have type hints. Avoid Any except in pytest fixtures/mocks.
• ruff + mypy --strict are enforced in CI. The coverage gate is 90%.

---

Known Gotchas (from AGENTS.md)

• vmrun getGuestIPAddress can return exit code 0 with "Error: The VMware Tools are not running..." on stdout. Filter lines starting with "Error:".
• Linux machine-id: Ubuntu cloud images ship with a fixed /etc/machine-id. VMware DHCP assigns IPs by machine-id — clones get the same IP and collide. The BaseClean-Linux snapshot must be taken after truncating machine-id.
• & nativecmd 2>$null under $ErrorActionPreference='Stop': native stderr is converted to an ErrorRecord before 2>$null discards it, causing a terminating error. Wrap with $ErrorActionPreference = 'SilentlyContinue' or 2>&1 | Out-Null.
• Snapshot with VM running: never take a template snapshot while the VM is powered on.
• WinRM vs SSH separation: do not modify the Windows WinRM branch and the Linux SSH branch in the same commit/PR. Keep if ($GuestOS -eq 'Linux') guards clean.

---

Key Files Reference

File	Purpose
AGENTS.md	Environment reference + error registry — read before writing any script
docs/ARCHITECTURE.md	Full system diagram and component descriptions
docs/CI-FLOW.md	Step-by-step pipeline description
docs/RUNBOOK.md	Common incident triage
TODO.md	Roadmap, audit trail, backlog
plans/	Phase closeout docs (A1–A5, B5) and implementation plans
config.example.toml	Annotated config template
tests/python/test_agents_errors.py	Regression tests for AGENTS.md errors #9–#12 — do not remove