Simone
6ba8cca70a
- ARCHITECTURE.md: fix WinRM subnet (192.168.1179), VM box shows VS 2026/Python 3.13/unzip+python build instead of .NET SDK 8/git clone/dotnet build - BEST-PRACTICES.md: fix VMnet11/192.168.11 VMnet8 NAT/192.168.79 - CI-FLOW.md: Step 3 workflow YAML updated to actual Invoke-CIJob.ps1 params (Submodules, BuildCommand, GuestArtifactSource); note that -VMIPAddress is auto-detected - OPTIMIZATION.md: BaseClean tier updated to VS BuildTools 2026, .NET SDK 10.0.203, Python 3.13.3 - gitea/workflow-example.yml: remove BUILD_VM_IP (auto-detected), replace -VMIPAddress/-Configuration with -Submodules/-BuildCommand/-GuestArtifactSource
263 lines
9.6 KiB
Markdown
263 lines
9.6 KiB
Markdown
# CI Pipeline Flow — Step by Step
|
||
|
||
## Trigger to Artifact: Complete Pipeline Walk-Through
|
||
|
||
---
|
||
|
||
### Step 0 — Prerequisites (one-time setup)
|
||
|
||
Before any CI job runs, the following must be in place:
|
||
|
||
| Component | State |
|
||
| ------------------ | ----------------------------------------------------------------------- |
|
||
| Gitea server | Running, accessible on LAN |
|
||
| act_runner | Registered, running as Windows service on host |
|
||
| Template VM | Provisioned, snapshot "BaseClean" taken, VM powered off |
|
||
| WinRM | Enabled on template VM (inherits to all clones) |
|
||
| vmrun.exe | Present at `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe` |
|
||
| `F:\CI\BuildVMs\` | Directory exists and writable |
|
||
| `F:\CI\Artifacts\` | Directory exists and writable |
|
||
|
||
---
|
||
|
||
### Step 1 — Developer Pushes Code
|
||
|
||
```
|
||
Developer → git push origin main
|
||
```
|
||
|
||
- Git push lands on Gitea server
|
||
- Gitea evaluates `.gitea/workflows/build.yml` in the repository
|
||
- If the workflow trigger matches (e.g. `on: push` to `main`), Gitea enqueues a new **Actions job**
|
||
- Job is tagged with the label defined in the workflow (`runs-on: windows-build`)
|
||
|
||
---
|
||
|
||
### Step 2 — act_runner Picks Up the Job
|
||
|
||
```
|
||
act_runner (host) ─polling─► Gitea Actions API
|
||
◄─job data─
|
||
```
|
||
|
||
- act_runner polls the Gitea API approximately every 5–10 seconds
|
||
- Detects a pending job matching its labels (`windows-build`)
|
||
- Downloads the workflow YAML and job parameters
|
||
- If `capacity` slots are available, starts job execution immediately
|
||
- If all slots are busy, job waits in Gitea queue until a slot frees
|
||
|
||
---
|
||
|
||
### Step 3 — act_runner Executes Workflow Steps
|
||
|
||
The workflow YAML defines steps that run as PowerShell commands **on the host**.
|
||
The first substantive step calls `Invoke-CIJob.ps1`:
|
||
|
||
```yaml
|
||
- name: Run CI Job
|
||
shell: pwsh
|
||
run: |
|
||
& 'N:\Code\Workspace\Local-CI-CD-System\scripts\Invoke-CIJob.ps1' `
|
||
-JobId '${{ github.run_id }}' `
|
||
-RepoUrl 'ssh://gitea-ci/Simone/<repo>.git' `
|
||
-Branch '${{ github.ref_name }}' `
|
||
-Commit '${{ github.sha }}' `
|
||
-TemplatePath $env:GITEA_CI_TEMPLATE_PATH `
|
||
-Submodules `
|
||
-BuildCommand 'python build_plugin.py --final --dist-dir dist' `
|
||
-GuestArtifactSource 'dist'
|
||
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
|
||
```
|
||
|
||
> Il parametro `-VMIPAddress` **non è più necessario** — l'IP viene rilevato automaticamente
|
||
> tramite `vmrun getGuestIPAddress` dopo l'avvio della VM.
|
||
|
||
---
|
||
|
||
### Step 4 — Invoke-CIJob.ps1: Clone VM
|
||
|
||
```
|
||
New-BuildVM.ps1
|
||
vmrun.exe -T ws clone <template.vmx> <clone.vmx> linked -snapshot "BaseClean"
|
||
```
|
||
|
||
- Generates unique clone name: `Clone_{JobId}_{yyyyMMdd_HHmmss}`
|
||
- Creates clone directory under `F:\CI\BuildVMs\`
|
||
- **Linked clone** creation takes ~5–15 seconds
|
||
- The template snapshot "BaseClean" is **not modified**
|
||
- Returns the full path to the new `.vmx` file
|
||
|
||
---
|
||
|
||
### Step 5 — Start VM
|
||
|
||
```
|
||
vmrun.exe -T ws start <clone.vmx> nogui
|
||
```
|
||
|
||
- Starts the cloned VM in headless (no GUI) mode
|
||
- VM boots from the "BaseClean" state: clean OS, build tools installed, WinRM enabled
|
||
- Boot takes ~20–40 seconds
|
||
|
||
---
|
||
|
||
### Step 6 — Wait for VM Readiness
|
||
|
||
```
|
||
Wait-VMReady.ps1 -VMPath <clone.vmx> -IPAddress <dynamic-NAT-ip> -TimeoutSeconds 300
|
||
```
|
||
|
||
IP is discovered dynamically via `vmrun getGuestIPAddress` (VMnet8 NAT, subnet 192.168.79.0/24).
|
||
No hardcoded IPs — each clone gets a DHCP-assigned address from VMware NAT.
|
||
|
||
Three-phase readiness check (retried every 5 seconds, up to 300s timeout):
|
||
|
||
```
|
||
Phase 1: vmrun getState → must return "running"
|
||
Phase 2: Test-Connection (ICMP ping) → must succeed
|
||
Phase 3: Test-WSMan → WinRM listener must respond
|
||
```
|
||
|
||
- Total wait is typically **30–90 seconds** after `vmrun start`
|
||
- If timeout exceeded, VM is destroyed and job fails with a clear error message
|
||
- On success, IP and credentials are ready for PSSession
|
||
|
||
---
|
||
|
||
### Step 7 — Remote Build Execution
|
||
|
||
```
|
||
Invoke-RemoteBuild.ps1
|
||
-IPAddress <dynamic-NAT-ip> # from vmrun getGuestIPAddress
|
||
-Credential (from Windows Credential Manager)
|
||
-BuildCommand 'python build_plugin.py --final --dist-dir dist'
|
||
-GuestArtifactSource 'dist'
|
||
```
|
||
|
||
Sequence inside the PSSession:
|
||
|
||
```
|
||
1. New-PSSession → open WinRM session to VM
|
||
2. Compress-Archive (host) → copy zip to C:\CI\build.zip (guest) → Expand-Archive to C:\CI\build\
|
||
(source code è già clonato sull'host in Phase 1 — non si fa git clone nella VM)
|
||
3. Invoke-Expression "$BuildCommand" in C:\CI\build\
|
||
Esempio: python build_plugin.py --final --dist-dir dist
|
||
a. Build 4 configurazioni MSBuild in parallelo (x86-unicode, x64-unicode, cli-x86, cli-x64)
|
||
b. Raccoglie artifact in C:\CI\build\dist\
|
||
c. Cleanup build dir (--final)
|
||
4. Compress-Archive C:\CI\build\$GuestArtifactSource C:\CI\output\artifacts.zip
|
||
5. Cattura exit code; se non-zero il job fallisce con errore chiaro
|
||
6. Remove-PSSession
|
||
```
|
||
|
||
- Output standard e stderr dalla build sono streamati all'host e catturati da act_runner per i log Gitea Actions
|
||
- Se la build fallisce (exit code ≠ 0), il controllo passa al blocco `finally`
|
||
|
||
---
|
||
|
||
### Step 8 — Collect Artifacts
|
||
|
||
```
|
||
Get-BuildArtifacts.ps1
|
||
-IPAddress <dynamic-NAT-ip>
|
||
-Credential (same as above)
|
||
-GuestArtifactPath C:\CI\output\artifacts.zip
|
||
-HostArtifactDir F:\CI\Artifacts\{JobId}\
|
||
```
|
||
|
||
- Apre PSSession e copia `C:\CI\output\artifacts.zip` dall'host
|
||
- Valida che il file esista e sia non-vuoto
|
||
- Il zip contiene la struttura `dist/` con tutti gli artifact di build
|
||
|
||
---
|
||
|
||
### Step 9 — Destroy VM
|
||
|
||
```
|
||
Remove-BuildVM.ps1 -VMPath <clone.vmx>
|
||
```
|
||
|
||
Sequence:
|
||
|
||
```
|
||
1. vmrun stop <clone.vmx> soft (graceful shutdown attempt)
|
||
2. Wait 5 seconds
|
||
3. vmrun getState → if still "running":
|
||
vmrun stop <clone.vmx> hard (force off)
|
||
4. vmrun deleteVM <clone.vmx> (removes VMX + VMDK delta files)
|
||
5. Remove-Item <clone dir> -Recurse (removes any leftover files)
|
||
```
|
||
|
||
- VM is gone from disk within ~5–10 seconds
|
||
- No state, no temp files, no credentials remain in the clone
|
||
- Template VM and "BaseClean" snapshot are unaffected
|
||
|
||
> **Note:** This step runs in a `finally` block inside `Invoke-CIJob.ps1`, so it
|
||
> executes even if Steps 6–8 failed. A build VM is **never** left running.
|
||
|
||
---
|
||
|
||
### Step 10 — Report Status & Upload Artifacts
|
||
|
||
```
|
||
act_runner ─► Gitea Actions API (job status: success / failure)
|
||
act_runner ─► Gitea artifact store (uploads F:\CI\Artifacts\{JobId}\*)
|
||
```
|
||
|
||
- act_runner reads the exit code from `Invoke-CIJob.ps1`
|
||
- Reports `success` or `failure` to Gitea
|
||
- Uploads artifacts using `actions/upload-artifact` step in the workflow YAML
|
||
- Build log (stdout from all steps) is stored in Gitea Actions UI
|
||
- Commit/PR status badge is updated
|
||
|
||
---
|
||
|
||
## Parallel Build Flow
|
||
|
||
When multiple pushes happen concurrently (or multiple PRs are open):
|
||
|
||
```
|
||
act_runner (capacity: 4)
|
||
├── Job #1 ──► Clone_Job_001 (192.168.79.x) → build → destroy
|
||
├── Job #2 ──► Clone_Job_002 (192.168.79.x) → build → destroy
|
||
├── Job #3 ──► Clone_Job_003 (192.168.79.x) → build → destroy
|
||
└── Job #4 ──► Clone_Job_004 (192.168.79.x) → build → destroy
|
||
(Job #5 waits in Gitea queue)
|
||
```
|
||
|
||
Note: each clone gets a dynamic IP from VMware NAT DHCP (192.168.79.0/24). IP is discovered
|
||
automatically via `vmrun getGuestIPAddress` — no static assignment required.
|
||
|
||
- Each job creates its own independently named clone
|
||
- Clones share the parent "BaseClean" snapshot (read-only CoW)
|
||
- No filesystem or process overlap between concurrent builds
|
||
- act_runner enforces the `capacity` limit
|
||
|
||
---
|
||
|
||
## Failure Scenarios
|
||
|
||
| Failure Point | Behavior |
|
||
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||
| VM clone fails | `Invoke-CIJob.ps1` throws; no VM to destroy; job fails immediately |
|
||
| VM never reaches ready state (timeout) | `Wait-VMReady.ps1` throws; `finally` destroys the (running) VM |
|
||
| Build fails inside VM | `Invoke-RemoteBuild.ps1` throws; `finally` collects partial artifacts (if any), destroys VM |
|
||
| Artifact collection fails | Job marked failed; VM is still destroyed in `finally` |
|
||
| Host machine loses power mid-job | `act_runner` resumes on reboot, marks in-flight job as failed; orphan VMs must be cleaned up manually (see TODO.md) |
|
||
|
||
---
|
||
|
||
## Timing Summary (typical build)
|
||
|
||
| Phase | Typical Duration |
|
||
| ---------------------------- | -------------------- |
|
||
| Gitea → act_runner poll | 5–10 s |
|
||
| Linked clone creation | 5–15 s |
|
||
| VM boot | 20–40 s |
|
||
| WinRM readiness | 10–30 s (after boot) |
|
||
| git clone (from local Gitea) | 2–10 s |
|
||
| dotnet restore + build | project-dependent |
|
||
| Artifact collection | 2–10 s |
|
||
| VM destruction | 5–10 s |
|
||
| **Overhead total** | **~60–120 s** |
|