diff --git a/docs/WORKFLOW-AUTHORING.md b/docs/WORKFLOW-AUTHORING.md new file mode 100644 index 0000000..2bcc11b --- /dev/null +++ b/docs/WORKFLOW-AUTHORING.md @@ -0,0 +1,568 @@ +# Workflow Authoring Reference + +> Reference document for AI agents and developers writing Gitea Actions workflows +> for the Local CI/CD System. +> Keep this document in sync with `runner/config.yaml` and `scripts/Invoke-CIJob.ps1`. + +--- + +## Table of Contents + +1. [System Overview](#1-system-overview) +2. [Runner Labels](#2-runner-labels) +3. [Shell Requirement](#3-shell-requirement) +4. [Environment Variables (Injected by Runner)](#4-environment-variables-injected-by-runner) +5. [Invoke-CIJob.ps1 Parameter Reference](#5-invoke-cijobps1-parameter-reference) +6. [URL Conventions](#6-url-conventions) +7. [Artifact Path Convention](#7-artifact-path-convention) +8. [Workflow Patterns](#8-workflow-patterns) + - [8.1 Windows build (host-clone mode)](#81-windows-build-host-clone-mode) + - [8.2 Linux build (host-clone mode)](#82-linux-build-host-clone-mode) + - [8.3 Windows build (UseGitClone mode)](#83-windows-build-usegitclone-mode) + - [8.4 Linux build (UseGitClone mode)](#84-linux-build-usegitclone-mode) + - [8.5 PSScriptAnalyzer lint](#85-psscriptanalyzer-lint) + - [8.6 Matrix build (multiple configs)](#86-matrix-build-multiple-configs) +9. [Checkout Step](#9-checkout-step) +10. [Timeouts](#10-timeouts) +11. [Common Mistakes](#11-common-mistakes) + +--- + +## 1. System Overview + +The CI system runs entirely on a local Windows 11 host. Gitea triggers jobs; act_runner +executes them. Every build step runs inside an ephemeral VMware VM — the runner host +itself never invokes compilers or build tools. The entire VM lifecycle is managed by +`Invoke-CIJob.ps1`. + +``` +git push → Gitea → act_runner (host) → Invoke-CIJob.ps1 → VM lifecycle + ├── New-BuildVM + ├── Wait-VMReady + ├── Invoke-RemoteBuild + ├── Get-BuildArtifacts + └── Remove-BuildVM (always) +``` + +Supported guest OS: Windows Server 2025 / 2022 (WinRM transport), Ubuntu 24.04 (SSH transport). + +--- + +## 2. Runner Labels + +| Label | Configured as | Use for | +| ---------------- | ---------------- | ----------------------------------- | +| `windows-build` | `windows-build:host` | Windows VM builds, Windows-only tools | +| `linux-build` | `linux-build:host` | Linux VM builds (cross-compile, GCC/Clang) | +| `dotnet` | `dotnet:host` | Alias for windows-build (legacy) | +| `msbuild` | `msbuild:host` | Alias for windows-build (legacy) | + +Both `windows-build` and `linux-build` run on the same physical host. Only the +template VM used differs. Use `windows-build` unless you specifically need a Linux +guest (cross-compile, GCC, CMake, Python/Linux toolchains). + +--- + +## 3. Shell Requirement + +**Always use `shell: powershell`** (Windows PowerShell 5.1). + +Never use `shell: pwsh` — that invokes PowerShell Core 7+ which is not the +required runtime and may not be present on the host. + +```yaml +# CORRECT +- name: Build + shell: powershell + run: | + ... + +# WRONG — do not use +- name: Build + shell: pwsh # PowerShell 7+ — not supported + run: | + ... +``` + +--- + +## 4. Environment Variables (Injected by Runner) + +These variables are automatically available in every job step. Do not redeclare them +in the workflow unless you need to override the default. + +| Variable | Value (default) | Description | +| ---------------------------- | ---------------------------------------------------- | --------------------------------------- | +| `GITEA_CI_TEMPLATE_PATH` | `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` | Windows 2025 template VMX path | +| `GITEA_CI_LINUX_TEMPLATE_PATH` | `F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx` | Ubuntu 24.04 template VMX path | +| `GITEA_CI_CLONE_BASE_DIR` | `F:\CI\BuildVMs` | Directory for ephemeral VM clones | +| `GITEA_CI_ARTIFACT_DIR` | `F:\CI\Artifacts` | Root directory for job artifacts | +| `GITEA_CI_GUEST_CRED_TARGET` | `BuildVMGuest` | Credential Manager target for guest VM | +| `GITEA_CI_SSH_KEY_PATH` | `F:\CI\keys\ci_linux` | SSH private key for Linux guest access | + +Invoke-CIJob.ps1 reads `GITEA_CI_TEMPLATE_PATH` and `GITEA_CI_CLONE_BASE_DIR` +automatically if `-TemplatePath` and `-CloneBaseDir` are not passed explicitly. + +--- + +## 5. Invoke-CIJob.ps1 Parameter Reference + +Script location (fixed on host): `N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1` + +### Mandatory parameters + +| Parameter | Type | Description | +| ----------- | ------ | --------------------------------------------------------------- | +| `-JobId` | string | Unique job ID. Use `${{ github.run_id }}` or `run_id-run_attempt`. | +| `-RepoUrl` | string | Git URL of the repository to clone/build. See Section 6. | +| `-Branch` | string | Branch to build. Use `${{ github.ref_name }}`. | + +### Optional parameters + +| Parameter | Type | Default | Description | +| ------------------------ | ------ | --------------------------- | -------------------------------------------------------------- | +| `-Commit` | string | `''` | Specific commit SHA. Use `${{ github.sha }}`. Empty = HEAD. | +| `-TemplatePath` | string | `$env:GITEA_CI_TEMPLATE_PATH` | Full path to template VMX. | +| `-SnapshotName` | string | `BaseClean` | Template snapshot name to clone from. | +| `-GuestOS` | string | `Auto` | `Windows`, `Linux`, or `Auto` (detected from VMX guestOS field). | +| `-BuildCommand` | string | `''` | Shell command to run inside VM. Empty = `dotnet build`. | +| `-GuestArtifactSource` | string | `dist` | Path inside VM guest from which artifacts are collected. | +| `-Submodules` | switch | off | Pass to clone with `--recurse-submodules`. | +| `-UseGitClone` | switch | off | Skip host-side clone; let the guest VM clone directly via HTTPS. | +| `-GiteaCredentialTarget` | string | `GiteaPAT` | Credential Manager target for Gitea PAT (UseGitClone only). | +| `-GuestCredentialTarget` | string | `BuildVMGuest` | Credential Manager target for Windows guest VM login. | +| `-SshKeyPath` | string | `F:\CI\keys\ci_linux` | SSH key for Linux guest (ignored for Windows). | +| `-SshUser` | string | `ci_build` | SSH username for Linux guest (ignored for Windows). | +| `-CloneBaseDir` | string | `$env:GITEA_CI_CLONE_BASE_DIR` | Directory for VM clone files. | +| `-ArtifactBaseDir` | string | `F:\CI\Artifacts` | Root artifact directory on host. | +| `-VMIPAddress` | string | `''` | Override guest IP. Empty = auto-detected via vmrun. | +| `-LogDir` | string | `F:\CI\Logs` | Directory for per-job transcripts and JSONL logs. | +| `-LogRetentionDays` | int | `30` | Days before log directories are purged. `0` = disabled. | + +Exit codes: `0` = success, `1` = failure (VM is always destroyed regardless). + +--- + +## 6. URL Conventions + +The `-RepoUrl` value depends on the build mode. + +### Host-clone mode (default, no `-UseGitClone`) + +The host runner clones the repo using the **SSH alias** defined in `~/.ssh/config`: + +``` +ssh://gitea-ci//.git +``` + +- `gitea-ci` is a Host alias in the runner's SSH config pointing to Gitea's SSH port. +- This URL is reachable **only from the host**, not from inside guest VMs. +- After clone, source is transferred to the guest via WinRM (Windows) or tar+scp (Linux). + +### UseGitClone mode (`-UseGitClone`) + +The guest VM clones the repo directly via **HTTPS**: + +``` +https://gitea.emulab.it//.git +``` + +- This URL must be reachable from inside the VM's NAT network. +- For private repos, the Gitea PAT is injected from Credential Manager (`GiteaPAT` target). +- Do not use the SSH alias URL with `-UseGitClone` — it is not accessible inside VMs. + +### Quick reference + +| Mode | RepoUrl to use | +| ----------------- | ---------------------------------------------------------- | +| Default (Windows) | `ssh://gitea-ci//.git` | +| Default (Linux) | `ssh://gitea-ci//.git` | +| UseGitClone | `https://gitea.emulab.it//.git` | + +--- + +## 7. Artifact Path Convention + +Artifacts are written to: + +``` +F:\CI\Artifacts\\ +``` + +Where `` is the value passed to `-JobId`. + +In `upload-artifact` steps use: + +```yaml +path: F:\CI\Artifacts\${{ github.run_id }}\ +``` + +Or for single-file artifact: + +```yaml +path: F:\CI\Artifacts\${{ github.run_id }}\artifacts.zip +``` + +`Invoke-CIJob.ps1` always produces an `artifacts.zip` at the job artifact root, +plus the raw files extracted from the guest under the same directory. + +--- + +## 8. Workflow Patterns + +Place workflow files in `.gitea/workflows/` in the project repository. + +### 8.1 Windows build (host-clone mode) + +Standard Windows build. Host clones the repo and transfers source to the Windows VM +via WinRM/zip transfer. + +```yaml +name: Build (Windows) + +on: + push: + tags: + - 'v*' + workflow_dispatch: + +jobs: + build: + runs-on: windows-build + timeout-minutes: 90 + + steps: + - name: Run CI Job + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` + -JobId '${{ github.run_id }}' ` + -RepoUrl 'ssh://gitea-ci//.git' ` + -Branch '${{ github.ref_name }}' ` + -Commit '${{ github.sha }}' ` + -Submodules ` + -BuildCommand 'python build_plugin.py --final --dist-dir dist' ` + -GuestArtifactSource 'dist' + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload artifacts + if: success() + uses: actions/upload-artifact@v4 + with: + name: build-${{ github.ref_name }} + path: F:\CI\Artifacts\${{ github.run_id }}\artifacts.zip + if-no-files-found: error + + - name: Upload logs on failure + if: failure() + uses: actions/upload-artifact@v4 + with: + name: build-logs-${{ github.run_id }} + path: F:\CI\Artifacts\${{ github.run_id }}\*.log + if-no-files-found: ignore + retention-days: 3 +``` + +### 8.2 Linux build (host-clone mode) + +Host clones the repo, compresses it with tar, and transfers it to the Ubuntu 24.04 +VM via scp. Build runs under `ci_build` user. + +```yaml +name: Build (Linux) + +on: + push: + tags: + - 'v*' + workflow_dispatch: + +jobs: + build: + runs-on: linux-build + timeout-minutes: 60 + + steps: + - name: Run CI Job (Linux) + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` + -JobId '${{ github.run_id }}' ` + -RepoUrl 'ssh://gitea-ci//.git' ` + -Branch '${{ github.ref_name }}' ` + -Commit '${{ github.sha }}' ` + -Submodules ` + -GuestOS 'Linux' ` + -TemplatePath $env:GITEA_CI_LINUX_TEMPLATE_PATH ` + -SshKeyPath $env:GITEA_CI_SSH_KEY_PATH ` + -BuildCommand 'python3 build_plugin.py --host linux --verbose' ` + -GuestArtifactSource 'dist' + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload artifacts + if: success() + uses: actions/upload-artifact@v4 + with: + name: build-linux-${{ github.ref_name }} + path: F:\CI\Artifacts\${{ github.run_id }}\artifacts.zip + if-no-files-found: error +``` + +Notes: +- `-GuestOS 'Linux'` can be omitted if `Auto` detection is sufficient (reads VMX `guestOS` field). +- `-TemplatePath $env:GITEA_CI_LINUX_TEMPLATE_PATH` selects the Ubuntu template. +- `-SshKeyPath $env:GITEA_CI_SSH_KEY_PATH` uses the injected key path. +- The default snapshot name for Linux templates is `BaseClean-Linux`. If you rely on + `Auto`, add `-SnapshotName 'BaseClean-Linux'` explicitly. + +### 8.3 Windows build (UseGitClone mode) + +The Windows guest VM clones the repository directly via HTTPS. Requires Git for Windows +in the template and the `GiteaPAT` Credential Manager entry on the host (for private repos). + +```yaml +name: Build (Windows, in-VM clone) + +on: + push: + branches: + - main + workflow_dispatch: + +jobs: + build: + runs-on: windows-build + timeout-minutes: 90 + + steps: + - name: Run CI Job + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` + -JobId '${{ github.run_id }}' ` + -RepoUrl 'https://gitea.emulab.it//.git' ` + -Branch '${{ github.ref_name }}' ` + -Commit '${{ github.sha }}' ` + -Submodules ` + -UseGitClone ` + -GiteaCredentialTarget 'GiteaPAT' ` + -BuildCommand 'python build_plugin.py --final --dist-dir dist' ` + -GuestArtifactSource 'dist' + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload artifacts + if: success() + uses: actions/upload-artifact@v4 + with: + name: build-${{ github.ref_name }} + path: F:\CI\Artifacts\${{ github.run_id }}\artifacts.zip + if-no-files-found: error +``` + +### 8.4 Linux build (UseGitClone mode) + +Linux guest clones directly. Useful when the host SSH agent is unavailable or for +testing the full HTTPS clone path. + +```yaml +name: Build (Linux, in-VM clone) + +on: + workflow_dispatch: + +jobs: + build: + runs-on: linux-build + timeout-minutes: 60 + + steps: + - name: Run CI Job (Linux, UseGitClone) + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` + -JobId '${{ github.run_id }}' ` + -RepoUrl 'https://gitea.emulab.it//.git' ` + -Branch '${{ github.ref_name }}' ` + -Commit '${{ github.sha }}' ` + -Submodules ` + -UseGitClone ` + -GiteaCredentialTarget 'GiteaPAT' ` + -GuestOS 'Linux' ` + -TemplatePath $env:GITEA_CI_LINUX_TEMPLATE_PATH ` + -SshKeyPath $env:GITEA_CI_SSH_KEY_PATH ` + -SnapshotName 'BaseClean-Linux' ` + -BuildCommand 'python3 build_plugin.py --host linux --verbose' ` + -GuestArtifactSource 'dist' + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload artifacts + if: success() + uses: actions/upload-artifact@v4 + with: + name: build-linux-${{ github.ref_name }} + path: F:\CI\Artifacts\${{ github.run_id }}\artifacts.zip + if-no-files-found: error +``` + +### 8.5 PSScriptAnalyzer lint + +Runs on every push or PR that touches `.ps1` / `.psm1` files. + +```yaml +name: Lint (PSScriptAnalyzer) + +on: + push: + paths: + - '**.ps1' + - '**.psm1' + pull_request: + paths: + - '**.ps1' + - '**.psm1' + +jobs: + lint: + runs-on: windows-build + timeout-minutes: 10 + + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + fetch-depth: 1 + + - name: Run PSScriptAnalyzer + shell: powershell + run: | + $ErrorActionPreference = 'Stop' + + if (-not (Get-Module -ListAvailable PSScriptAnalyzer)) { + Install-Module PSScriptAnalyzer -Scope CurrentUser -Force -AllowClobber + } + + $paths = @('scripts', 'template', 'runner', 'Setup-Host.ps1') + $results = $paths | ForEach-Object { + if (Test-Path $_) { + Invoke-ScriptAnalyzer -Path $_ -Recurse -Severity Error,Warning + } + } + + if ($results) { + $results | Format-Table -AutoSize + Write-Error "PSScriptAnalyzer found $($results.Count) issue(s)." + exit 1 + } + Write-Host "PSScriptAnalyzer: no issues found." +``` + +Note: The lint job uses `actions/checkout@v4` because it analyzes files inside the +project repo (the checked-out workspace), not the CI scripts repo. + +### 8.6 Matrix build (multiple configs) + +```yaml +name: Build (matrix) + +on: + push: + tags: + - 'v*' + +jobs: + build: + runs-on: windows-build + timeout-minutes: 90 + + strategy: + matrix: + config: [x86-ansi, x86-unicode, x64-unicode] + fail-fast: false + + steps: + - name: Run CI Job + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` + -JobId '${{ github.run_id }}-${{ matrix.config }}' ` + -RepoUrl 'ssh://gitea-ci//.git' ` + -Branch '${{ github.ref_name }}' ` + -Commit '${{ github.sha }}' ` + -Submodules ` + -BuildCommand "python build_plugin.py --config ${{ matrix.config }}" ` + -GuestArtifactSource 'dist' + if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE } + + - name: Upload artifacts + if: success() + uses: actions/upload-artifact@v4 + with: + name: build-${{ matrix.config }}-${{ github.ref_name }} + path: F:\CI\Artifacts\${{ github.run_id }}-${{ matrix.config }}\artifacts.zip + if-no-files-found: error +``` + +Note: With `runner.capacity: 4`, up to 4 matrix jobs run in parallel. Each gets its +own ephemeral VM. Ensure `-JobId` is unique per matrix cell. + +--- + +## 9. Checkout Step + +**Do not add `actions/checkout@v4` to build steps** unless the workflow itself needs +access to repository files on the host (e.g. lint, static analysis). + +For build jobs, `Invoke-CIJob.ps1` handles all source acquisition — either by cloning +on the host and transferring to the VM, or by having the VM clone directly. + +The CI scripts are at a fixed path on the runner host and do not need a checkout: + +```yaml +# CORRECT — direct path to CI scripts, no checkout needed +- name: Run CI Job + shell: powershell + run: | + & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ... + +# WRONG — CI scripts are not in the project repo +- name: Checkout + uses: actions/checkout@v4 # this checks out the PROJECT repo, not CI scripts +- name: Run CI Job + shell: powershell + run: | + & "$env:GITHUB_WORKSPACE\scripts\Invoke-CIJob.ps1" ... # path doesn't exist +``` + +--- + +## 10. Timeouts + +| Build type | Recommended `timeout-minutes` | +| --------------------------- | ----------------------------- | +| Windows build (typical) | 90 | +| Windows build (large/slow) | 120 | +| Linux build (typical) | 60 | +| Lint / static analysis | 10 | +| Template provisioning | 180 | + +The runner-level timeout is `2h` (from `runner/config.yaml`). Keep job timeouts +below that value. If a job timeout fires, the VM may not be cleanly destroyed — +the cleanup task `Cleanup-OrphanedBuildVMs.ps1` handles these cases. + +--- + +## 11. Common Mistakes + +| Mistake | Correct approach | +| ------- | ---------------- | +| `shell: pwsh` | Use `shell: powershell` (PS 5.1, not Core) | +| SSH alias URL with `-UseGitClone` | Use `https://gitea.emulab.it/...` for UseGitClone; SSH alias is host-only | +| HTTPS URL without `-UseGitClone` | SSH alias `ssh://gitea-ci/...` for host-clone mode | +| `$env:GITHUB_WORKSPACE` to find CI scripts | Use hardcoded `N:\Code\Workspace\Local-CI-CD-System\scripts\` | +| `actions/checkout@v4` in build jobs | Omit it; source is handled by Invoke-CIJob.ps1 | +| `runs-on: ubuntu-latest` or `windows-latest` | Use `windows-build` or `linux-build` — cloud runners are not available | +| `&&` in PowerShell run blocks | Not valid in PS 5.1; use `if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }` | +| Hardcoded IP in `-VMIPAddress` | Omit it; auto-detected via `vmrun getGuestIPAddress` | +| `-TemplatePath` for Linux without `GITEA_CI_LINUX_TEMPLATE_PATH` | Pass `$env:GITEA_CI_LINUX_TEMPLATE_PATH` or the full path explicitly | +| Forgetting `-SnapshotName 'BaseClean-Linux'` for Linux | Linux template uses a different snapshot name than Windows (`BaseClean`) | +| Same `-JobId` for matrix jobs | Append matrix key: `${{ github.run_id }}-${{ matrix.config }}` | diff --git a/gitea/workflow-example.yml b/gitea/workflow-example.yml index 55d55bb..d81cec8 100644 --- a/gitea/workflow-example.yml +++ b/gitea/workflow-example.yml @@ -47,17 +47,17 @@ jobs: # This script handles the entire VM lifecycle: # clone → start → wait ready → build → collect artifacts → destroy - name: Build in ephemeral VM - shell: pwsh + shell: powershell run: | $ErrorActionPreference = 'Stop' - # Resolve the scripts directory relative to the checked-out repo. - # If your CI scripts are in a separate repo, adjust the path. - $ciScriptsDir = Join-Path $env:GITHUB_WORKSPACE 'scripts' + # CI scripts are at a fixed location on the runner host. + # Do NOT use $env:GITHUB_WORKSPACE — the CI system is in a separate repo. + $ciScriptsDir = 'N:\Code\Workspace\Local-CI-CD-System\scripts' & "$ciScriptsDir\Invoke-CIJob.ps1" ` -JobId "${{ github.run_id }}-${{ github.run_attempt }}" ` - -RepoUrl "${{ github.server_url }}/${{ github.repository }}.git" ` + -RepoUrl "ssh://gitea-ci/${{ github.repository }}.git" ` -Branch "${{ github.ref_name }}" ` -Commit "${{ github.sha }}" ` -Submodules ` diff --git a/gitea/workflows/build-nsis.yml b/gitea/workflows/build-nsis.yml index 48a8166..f36194c 100644 --- a/gitea/workflows/build-nsis.yml +++ b/gitea/workflows/build-nsis.yml @@ -25,7 +25,7 @@ jobs: steps: - name: Run CI Job - shell: pwsh + shell: powershell run: | & 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' ` -JobId '${{ github.run_id }}' ` diff --git a/gitea/workflows/lint.yml b/gitea/workflows/lint.yml index 4f7f4b2..204f4fb 100644 --- a/gitea/workflows/lint.yml +++ b/gitea/workflows/lint.yml @@ -29,7 +29,7 @@ jobs: fetch-depth: 1 - name: Run PSScriptAnalyzer - shell: pwsh + shell: powershell run: | $ErrorActionPreference = 'Stop'