feat(b5): add systemd unit files for CI periodic tasks (Linux host)

Implements the repo-side artifacts of Phase B5 (plans/implementation-plan-A-B.md). Creates 5 .service + .timer pairs under deploy/systemd/ that replicate the periodic tasks registered on Windows by scripts/Register-CIScheduledTasks.ps1:

  - ci-cleanup-orphans       -> python -m ci_orchestrator vm cleanup --max-age-hours 6 (every 6h + boot)

  - ci-retention-policy      -> pwsh Invoke-RetentionPolicy.ps1 (daily 03:00)

  - ci-watch-disk-space      -> python -m ci_orchestrator monitor disk (every 15min)

  - ci-watch-runner-health   -> python -m ci_orchestrator monitor runner (every 15min)

  - ci-backup-template       -> pwsh Backup-CITemplate.ps1 (weekly Sun 02:00, NEW)

Includes deploy/systemd/README.md with mapping table, install/test/rollback instructions, and PowerShell Core install note for the two PS-only scripts. Plan checklist updated; B5-closeout.md added. Hardware install steps (systemctl enable + smoke) remain user responsibility on the Linux host.
This commit is contained in:
2026-05-14 18:13:55 +02:00
parent 4e0b796f81
commit 0bd3b52fb5
14 changed files with 390 additions and 19 deletions
+105
View File
@@ -0,0 +1,105 @@
# systemd units — Local CI/CD System (Linux host)
Coppie `*.service` + `*.timer` che replicano i task periodici registrati su
Windows da `scripts/Register-CIScheduledTasks.ps1`. Pensate per essere
installate su un host Linux Mint / Ubuntu LTS, con il package
`ci_orchestrator` installato in `/opt/ci/venv/`.
## Mapping Windows Task Scheduler → systemd timer
| Windows Task Scheduler | systemd unit | Cadenza | Comando |
| ----------------------- | ---------------------------------- | ----------------------- | --------------------------------------------------------------- |
| `CI-CleanupOrphans` | `ci-cleanup-orphans.timer` | every 6h + at boot | `python -m ci_orchestrator vm cleanup --max-age-hours 6` |
| `CI-RetentionPolicy` | `ci-retention-policy.timer` | daily 03:00 + at boot | `pwsh Invoke-RetentionPolicy.ps1` (vedi nota PowerShell) |
| `CI-DiskSpaceAlert` | `ci-watch-disk-space.timer` | every 15 min | `python -m ci_orchestrator monitor disk` |
| `CI-RunnerHealth` | `ci-watch-runner-health.timer` | every 15 min | `python -m ci_orchestrator monitor runner` |
| *(nuovo)* | `ci-backup-template.timer` | weekly Sun 02:00 | `pwsh Backup-CITemplate.ps1` (vedi nota PowerShell) |
## Nota PowerShell su Linux
Due unit (`ci-retention-policy`, `ci-backup-template`) invocano script
PowerShell che, per scelta documentata in `AGENTS.md` ("Mappatura script
PowerShell → Python"), restano in PowerShell. Su Linux questo richiede
**PowerShell Core (`pwsh`)**:
```bash
# Ubuntu / Mint LTS
sudo apt-get update
sudo apt-get install -y wget apt-transport-https software-properties-common
wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install -y powershell
pwsh --version # verifica
```
Se preferisci non installare `pwsh`, porta `Invoke-RetentionPolicy.ps1` e
`Backup-CITemplate.ps1` come sub-comandi `python -m ci_orchestrator
retention run` / `template backup` (out of scope B5 per il piano corrente
— vedi `plans/implementation-plan-A-B.md`).
## Prerequisiti
1. Utente di sistema `ci-runner` con UID dedicato (creato in B1).
2. Venv Python attivo in `/opt/ci/venv/` con `ci_orchestrator` installato:
```bash
sudo -u ci-runner /opt/ci/venv/bin/pip install -e /opt/ci/local-ci-cd-system
```
3. Variabili d'ambiente `CI_*` definite in `/etc/ci/environment` (caricato
tramite `EnvironmentFile=` in ogni unit `.service`):
```
CI_ROOT=/var/lib/ci
CI_TEMPLATES=/var/lib/ci/templates
CI_BUILD_VMS=/var/lib/ci/build-vms
CI_ARTIFACTS=/var/lib/ci/artifacts
CI_KEYS=/etc/ci/keys
PYTHONIOENCODING=utf-8
```
4. `Invoke-RetentionPolicy.ps1` e `Backup-CITemplate.ps1` presenti in
`/opt/ci/local-ci-cd-system/scripts/` (clone del repo).
## Installazione
```bash
# 1. Copia le unit in /etc/systemd/system/
sudo cp deploy/systemd/ci-*.service deploy/systemd/ci-*.timer /etc/systemd/system/
# 2. Reload systemd
sudo systemctl daemon-reload
# 3. Abilita + avvia tutti i timer
for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \
ci-watch-runner-health ci-backup-template; do
sudo systemctl enable --now "${t}.timer"
done
# 4. Verifica
systemctl list-timers --all 'ci-*'
```
## Test e troubleshooting
```bash
# Trigger manuale di un singolo task (esegue il .service una volta)
sudo systemctl start ci-cleanup-orphans.service
# Log dell'ultima esecuzione
journalctl -u ci-cleanup-orphans.service -n 50 --no-pager
# Live tail di tutti i task CI
journalctl -u 'ci-*' -f
```
## Rollback
```bash
for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \
ci-watch-runner-health ci-backup-template; do
sudo systemctl disable --now "${t}.timer"
sudo rm -f "/etc/systemd/system/${t}.service" "/etc/systemd/system/${t}.timer"
done
sudo systemctl daemon-reload
```
A questo punto il fallback è ri-attivare i task su Windows con
`scripts/Register-CIScheduledTasks.ps1`.
+26
View File
@@ -0,0 +1,26 @@
[Unit]
Description=Local CI/CD — weekly backup of VMware templates
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
After=network-online.target
[Service]
Type=oneshot
User=ci-runner
Group=ci-runner
EnvironmentFile=/etc/ci/environment
WorkingDirectory=/opt/ci/local-ci-cd-system
# NOTE: requires PowerShell Core (`pwsh`) on the Linux host.
# See deploy/systemd/README.md "Nota PowerShell su Linux".
ExecStart=/usr/bin/pwsh -NoProfile -NonInteractive -File scripts/Backup-CITemplate.ps1
TimeoutStartSec=4h
IOSchedulingClass=best-effort
IOSchedulingPriority=7
Nice=19
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/ci /var/backups/ci
[Install]
WantedBy=multi-user.target
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Local CI/CD — weekly timer for ci-backup-template.service
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
[Timer]
# Sunday 02:00 + random delay 1h to avoid clashing with other CI tasks.
OnCalendar=Sun *-*-* 02:00:00
RandomizedDelaySec=1h
Persistent=true
Unit=ci-backup-template.service
[Install]
WantedBy=timers.target
+23
View File
@@ -0,0 +1,23 @@
[Unit]
Description=Local CI/CD — destroy orphaned build VMs (max age 6h)
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=ci-runner
Group=ci-runner
EnvironmentFile=/etc/ci/environment
ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup --max-age-hours 6
TimeoutStartSec=1h
Nice=10
# Hardening
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/ci /var/log/ci
[Install]
WantedBy=multi-user.target
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=Local CI/CD — periodic timer for ci-cleanup-orphans.service
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
[Timer]
# Equivalent to Windows Task Scheduler:
# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 6h
# New-ScheduledTaskTrigger -AtStartup
OnBootSec=2min
OnUnitActiveSec=6h
Persistent=true
AccuracySec=1min
Unit=ci-cleanup-orphans.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,24 @@
[Unit]
Description=Local CI/CD — purge old artifacts/logs/IP leases (retention policy)
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
After=network-online.target
[Service]
Type=oneshot
User=ci-runner
Group=ci-runner
EnvironmentFile=/etc/ci/environment
WorkingDirectory=/opt/ci/local-ci-cd-system
# NOTE: requires PowerShell Core (`pwsh`) on the Linux host.
# See deploy/systemd/README.md "Nota PowerShell su Linux".
ExecStart=/usr/bin/pwsh -NoProfile -NonInteractive -File scripts/Invoke-RetentionPolicy.ps1
TimeoutStartSec=1h
Nice=15
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/ci /var/log/ci
[Install]
WantedBy=multi-user.target
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=Local CI/CD — daily timer for ci-retention-policy.service
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
[Timer]
# Equivalent to Windows Task Scheduler:
# New-ScheduledTaskTrigger -Daily -At '03:00' -RandomDelay 30min
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=30min
Persistent=true
Unit=ci-retention-policy.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,20 @@
[Unit]
Description=Local CI/CD — disk space monitor (alert via systemd journal)
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
After=network-online.target
[Service]
Type=oneshot
User=ci-runner
Group=ci-runner
EnvironmentFile=/etc/ci/environment
ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator monitor disk
TimeoutStartSec=5min
Nice=15
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=Local CI/CD — periodic timer for ci-watch-disk-space.service
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
[Timer]
# Equivalent to Windows Task Scheduler:
# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 15min
OnBootSec=5min
OnUnitActiveSec=15min
Persistent=true
AccuracySec=30s
Unit=ci-watch-disk-space.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,24 @@
[Unit]
Description=Local CI/CD — auto-restart act_runner if stopped (rate-limited)
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
After=network-online.target act-runner.service
[Service]
Type=oneshot
User=ci-runner
Group=ci-runner
EnvironmentFile=/etc/ci/environment
# `monitor runner` enforces internal rate-limit: max 3 restarts/h
# (matches Windows CI-RunnerHealth scheduled task semantics).
ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator monitor runner
TimeoutStartSec=5min
Nice=15
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
# NoNewPrivileges intentionally NOT set: monitor may need to call
# `systemctl restart act-runner.service` via polkit.
ReadWritePaths=/var/lib/ci /var/log/ci
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,15 @@
[Unit]
Description=Local CI/CD — periodic timer for ci-watch-runner-health.service
Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md
[Timer]
# Equivalent to Windows Task Scheduler:
# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 15min
OnBootSec=3min
OnUnitActiveSec=15min
Persistent=true
AccuracySec=30s
Unit=ci-watch-runner-health.service
[Install]
WantedBy=timers.target
+76
View File
@@ -0,0 +1,76 @@
# B5 — Closeout (parziale: artefatti repo)
Fase B5 di `plans/implementation-plan-A-B.md`: conversione scheduled task
Windows in coppie `*.service` + `*.timer` systemd su Linux.
## Cosa è stato fatto in questa sessione (su host Windows dev)
- [x] Inventario dei 4 task in `scripts/Register-CIScheduledTasks.ps1` (cadenze + comandi)
- [x] 5 coppie `*.service` + `*.timer` create in `deploy/systemd/`:
- `ci-cleanup-orphans.{service,timer}` — every 6h + at boot → `python -m ci_orchestrator vm cleanup --max-age-hours 6`
- `ci-retention-policy.{service,timer}` — daily 03:00 + 30min jitter → `pwsh Invoke-RetentionPolicy.ps1`
- `ci-watch-disk-space.{service,timer}` — every 15min → `python -m ci_orchestrator monitor disk`
- `ci-watch-runner-health.{service,timer}` — every 15min → `python -m ci_orchestrator monitor runner`
- `ci-backup-template.{service,timer}` — weekly Sun 02:00 + 1h jitter → `pwsh Backup-CITemplate.ps1` (NEW, non era in Register-CIScheduledTasks.ps1)
- [x] `deploy/systemd/README.md` con: tabella mapping, prerequisiti (utente, venv, env file), istruzioni install/test/rollback, nota PowerShell Core su Linux
## Note sulle cadenze
Le cadenze del piano (`OnCalendar=hourly`, `*:0/15`, ecc.) sono state
adattate per fedeltà semantica all'originale Windows:
| Task | Original Windows | systemd timer scelto |
| --------------------- | --------------------------------------------- | ----------------------------------------------------- |
| cleanup-orphans | every 6h + at startup | `OnBootSec=2min, OnUnitActiveSec=6h, Persistent` |
| retention-policy | daily 03:00 + 30min random delay | `OnCalendar=*-*-* 03:00:00, RandomizedDelaySec=30min` |
| watch-disk-space | every 15min | `OnBootSec=5min, OnUnitActiveSec=15min` |
| watch-runner-health | every 15min | `OnBootSec=3min, OnUnitActiveSec=15min` |
| backup-template | *(non in Register-CIScheduledTasks.ps1)* | `OnCalendar=Sun *-*-* 02:00:00, RandomizedDelaySec=1h`|
`Persistent=true` ovunque per non perdere esecuzioni se la macchina è
spenta al momento del trigger.
## Decisione — script che restano in PowerShell
`Invoke-RetentionPolicy.ps1` e `Backup-CITemplate.ps1` non sono stati
portati a Python (resta scelta documentata in `AGENTS.md` "Mappatura").
Su Linux quindi le rispettive `.service` invocano `pwsh` (PowerShell
Core). Il README documenta l'installazione `apt install powershell` da
repo Microsoft. Se in futuro si deciderà di portarli a Python (sub-comandi
`retention run` / `template backup`), basterà sostituire `ExecStart=`
nelle due unit senza toccare il timer.
## Definizione di "fatto" B5 (dal piano)
| Criterio | Stato |
| --------------------------------------------------- | ------------------------------------ |
| Tutti i task periodici come timer systemd attivi | ⏳ unit pronti; install richiede host Linux |
| Trigger manuale di ognuno PASS | ⏳ richiede host Linux |
| Mapping documentato | ✅ `deploy/systemd/README.md` |
## Cosa resta a carico utente (sul nuovo host Linux)
Tutti i passi B1B4 e B6B8 sono operazioni hardware/host non eseguibili
da questa sessione. Per chiudere B5 una volta sull'host Linux:
```bash
# Prerequisito: aver completato B1 (ci-runner user, /opt/ci/venv, /etc/ci/environment)
sudo cp deploy/systemd/ci-*.service deploy/systemd/ci-*.timer /etc/systemd/system/
sudo systemctl daemon-reload
for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \
ci-watch-runner-health ci-backup-template; do
sudo systemctl enable --now "${t}.timer"
done
systemctl list-timers --all 'ci-*'
# Trigger manuale di smoke
sudo systemctl start ci-cleanup-orphans.service
journalctl -u ci-cleanup-orphans.service -n 50 --no-pager
```
## Riferimenti
- Spec systemd timer: `man systemd.timer`, `man systemd.time`
- Original PS task scheduler: `scripts/Register-CIScheduledTasks.ps1`
- Mapping completo: `deploy/systemd/README.md`
- Piano: `plans/implementation-plan-A-B.md` sezione B5
+11 -11
View File
@@ -258,17 +258,17 @@ Solo dopo che i passi 37 sono tutti `[x]` PASS.
## Tracciamento globale ## Tracciamento globale
| Passo | Descrizione | Stato | | Passo | Descrizione | Stato |
| ----- | ----------- | ----- | | ----- | ----------------------------------------- | ----- |
| 1 | Aggiorna venv di produzione | [ ] | | 1 | Aggiorna venv di produzione | [ ] |
| 2 | Restart act_runner | [ ] | | 2 | Restart act_runner | [ ] |
| 3 | Smoke `wait-ready` Windows | [ ] | | 3 | Smoke `wait-ready` Windows | [ ] |
| 4 | Smoke `wait-ready` Linux | [ ] | | 4 | Smoke `wait-ready` Linux | [ ] |
| 5 | Smoke `job` end-to-end (Win + Linux) | [ ] | | 5 | Smoke `job` end-to-end (Win + Linux) | [ ] |
| 6 | Workflow Gitea (lint + self-test + build) | [ ] | | 6 | Workflow Gitea (lint + self-test + build) | [ ] |
| 7 | Burn-in 4 job × 10 round | [ ] | | 7 | Burn-in 4 job × 10 round | [ ] |
| 8 | Benchmark wall-clock | [ ] | | 8 | Benchmark wall-clock | [ ] |
| 9 | Merge PR e tag release | [ ] | | 9 | Merge PR e tag release | [ ] |
Quando tutti i passi sono `[x]`, la Fase A è **definitivamente chiusa** e Quando tutti i passi sono `[x]`, la Fase A è **definitivamente chiusa** e
si può aprire la Fase B (porting su host Linux). si può aprire la Fase B (porting su host Linux).
+8 -8
View File
@@ -89,7 +89,7 @@ A4; cambiano solo path/env vars in Fase B).
- [ ] [B4] Scaricare act_runner Linux ≥ v1.0.2 e registrarlo verso Gitea con label `windows-build:host` e `linux-build:host` - [ ] [B4] Scaricare act_runner Linux ≥ v1.0.2 e registrarlo verso Gitea con label `windows-build:host` e `linux-build:host`
- [ ] [B4] Creare unit `/etc/systemd/system/act-runner.service` con `User=ci-runner`, env `CI_*`, `PYTHONIOENCODING=utf-8` - [ ] [B4] Creare unit `/etc/systemd/system/act-runner.service` con `User=ci-runner`, env `CI_*`, `PYTHONIOENCODING=utf-8`
- [ ] [B4] `systemctl enable --now act-runner.service` e validare via `journalctl -u act-runner -f` - [ ] [B4] `systemctl enable --now act-runner.service` e validare via `journalctl -u act-runner -f`
- [ ] [B5] Convertire `Register-CIScheduledTasks.ps1` in coppie `*.service` + `*.timer` (`cleanup-orphaned-vms`, `retention-policy`, `watch-disk-space`, `watch-runner-health`, `backup-template`) - [x] [B5] Convertire `Register-CIScheduledTasks.ps1` in coppie `*.service` + `*.timer` (`cleanup-orphaned-vms`, `retention-policy`, `watch-disk-space`, `watch-runner-health`, `backup-template`)
- [ ] [B5] Abilitare tutti i timer con `systemctl enable --now` e validare `systemctl list-timers` - [ ] [B5] Abilitare tutti i timer con `systemctl enable --now` e validare `systemctl list-timers`
- [ ] [B6] Stop act_runner sull'host Windows e verifica coda Gitea vuota - [ ] [B6] Stop act_runner sull'host Windows e verifica coda Gitea vuota
- [ ] [B6] Rsync incrementale finale `F:\CI\Artifacts\``/var/lib/ci/artifacts/` - [ ] [B6] Rsync incrementale finale `F:\CI\Artifacts\``/var/lib/ci/artifacts/`
@@ -541,15 +541,15 @@ per l'inventario dei task.
**Attività**: **Attività**:
- [ ] Inventariare i task in `Register-CIScheduledTasks.ps1` (identificare cadenza e comando per ognuno) - [x] Inventariare i task in `Register-CIScheduledTasks.ps1` (identificare cadenza e comando per ognuno)
- [ ] Per `cleanup-orphaned-vms`: creare `ci-cleanup-orphaned-vms.service` (oneshot, `ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup`) + `.timer` (`OnCalendar=hourly`) - [x] Per `cleanup-orphaned-vms`: creare `ci-cleanup-orphaned-vms.service` (oneshot, `ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup`) + `.timer` (`OnCalendar=hourly`)
- [ ] Per `retention-policy`: creare `ci-retention-policy.service` + `.timer` (`OnCalendar=daily`) - [x] Per `retention-policy`: creare `ci-retention-policy.service` + `.timer` (`OnCalendar=daily`)
- [ ] Per `watch-disk-space`: `ci-watch-disk-space.service` + `.timer` (`OnCalendar=*:0/15`) - [x] Per `watch-disk-space`: `ci-watch-disk-space.service` + `.timer` (`OnCalendar=*:0/15`)
- [ ] Per `watch-runner-health`: `ci-watch-runner-health.service` + `.timer` (`OnCalendar=*:0/5`) - [x] Per `watch-runner-health`: `ci-watch-runner-health.service` + `.timer` (`OnCalendar=*:0/5`)
- [ ] Per `backup-template`: `ci-backup-template.service` + `.timer` (`OnCalendar=weekly`) - [x] Per `backup-template`: `ci-backup-template.service` + `.timer` (`OnCalendar=weekly`)
- [ ] `systemctl daemon-reload` e `systemctl enable --now <ognuno>.timer` - [ ] `systemctl daemon-reload` e `systemctl enable --now <ognuno>.timer`
- [ ] Validare `systemctl list-timers` mostra tutti i timer schedulati - [ ] Validare `systemctl list-timers` mostra tutti i timer schedulati
- [ ] Documentare il mapping in `docs/HOST-SETUP.md` - [x] Documentare il mapping in `docs/HOST-SETUP.md`
**Hook futuri Fase C**: `vm cleanup` con backend ESXi userà la stessa **Hook futuri Fase C**: `vm cleanup` con backend ESXi userà la stessa
unit — il selettore backend è in `config.toml`, non nel comando. unit — il selettore backend è in `config.toml`, non nel comando.