Simone
19186981a5
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
134 lines
5.2 KiB
Markdown
134 lines
5.2 KiB
Markdown
# systemd units — Local CI/CD System (Linux host)
|
|
|
|
Coppie `*.service` + `*.timer` che replicano i task periodici registrati su
|
|
Windows da `scripts/Register-CIScheduledTasks.ps1`. Pensate per essere
|
|
installate su un host Linux Mint / Ubuntu LTS, con il package
|
|
`ci_orchestrator` installato in `/opt/ci/venv/`.
|
|
|
|
## Mapping Windows Task Scheduler → systemd timer
|
|
|
|
| Windows Task Scheduler | systemd unit | Cadenza | Comando |
|
|
| ----------------------- | ---------------------------------- | ----------------------- | --------------------------------------------------------------- |
|
|
| `CI-CleanupOrphans` | `ci-cleanup-orphans.timer` | every 6h + at boot | `python -m ci_orchestrator vm cleanup --max-age-hours 6` |
|
|
| `CI-RetentionPolicy` | `ci-retention-policy.timer` | daily 03:00 + at boot | `python -m ci_orchestrator retention run` |
|
|
| `CI-DiskSpaceAlert` | `ci-watch-disk-space.timer` | every 15 min | `python -m ci_orchestrator monitor disk` |
|
|
| `CI-RunnerHealth` | `ci-watch-runner-health.timer` | every 15 min | `python -m ci_orchestrator monitor runner` |
|
|
| *(nuovo)* | `ci-backup-template.timer` | weekly Sun 02:00 | `python -m ci_orchestrator template backup --all-templates` |
|
|
|
|
## Prerequisiti
|
|
|
|
1. Utente di sistema `ci-runner` con UID dedicato (creato in B1).
|
|
2. Venv Python attivo in `/opt/ci/venv/` con `ci_orchestrator` installato:
|
|
```bash
|
|
sudo -u ci-runner /opt/ci/venv/bin/pip install -e /opt/ci/local-ci-cd-system
|
|
```
|
|
3. Variabili d'ambiente `CI_*` definite in `/etc/ci/environment` (caricato
|
|
tramite `EnvironmentFile=` in ogni unit `.service`):
|
|
```
|
|
CI_ROOT=/var/lib/ci
|
|
CI_TEMPLATES=/var/lib/ci/templates
|
|
CI_BUILD_VMS=/var/lib/ci/build-vms
|
|
CI_ARTIFACTS=/var/lib/ci/artifacts
|
|
CI_KEYS=/etc/ci/keys
|
|
PYTHONIOENCODING=utf-8
|
|
```
|
|
4. `ci_orchestrator` installato nel venv (punto 2) — nessuna dipendenza PowerShell
|
|
richiesta; `retention run` e `template backup` sono comandi Python puri.
|
|
|
|
> **Nota**: `ci-backup-template.service` gira come `root` (non `ci-runner`)
|
|
> perché chiama `systemctl stop/start act-runner.service` prima e dopo il
|
|
> backup. Tutti gli altri service girano come `ci-runner`.
|
|
|
|
## Installazione
|
|
|
|
```bash
|
|
# 1. Copia le unit in /etc/systemd/system/
|
|
sudo cp deploy/systemd/ci-*.service deploy/systemd/ci-*.timer /etc/systemd/system/
|
|
|
|
# 2. Reload systemd
|
|
sudo systemctl daemon-reload
|
|
|
|
# 3. Abilita + avvia tutti i timer
|
|
for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \
|
|
ci-watch-runner-health ci-backup-template; do
|
|
sudo systemctl enable --now "${t}.timer"
|
|
done
|
|
|
|
# 4. Verifica
|
|
systemctl list-timers --all 'ci-*'
|
|
```
|
|
|
|
## Test e troubleshooting
|
|
|
|
```bash
|
|
# Trigger manuale di un singolo task (esegue il .service una volta)
|
|
sudo systemctl start ci-cleanup-orphans.service
|
|
|
|
# Log dell'ultima esecuzione
|
|
journalctl -u ci-cleanup-orphans.service -n 50 --no-pager
|
|
|
|
# Live tail di tutti i task CI
|
|
journalctl -u 'ci-*' -f
|
|
```
|
|
|
|
## Rollback
|
|
|
|
```bash
|
|
for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \
|
|
ci-watch-runner-health ci-backup-template; do
|
|
sudo systemctl disable --now "${t}.timer"
|
|
sudo rm -f "/etc/systemd/system/${t}.service" "/etc/systemd/system/${t}.timer"
|
|
done
|
|
sudo systemctl daemon-reload
|
|
```
|
|
|
|
A questo punto il fallback è ri-attivare i task su Windows con
|
|
`scripts/Register-CIScheduledTasks.ps1`.
|
|
|
|
---
|
|
|
|
## Servizi VMware host (prerequisito avvio VM)
|
|
|
|
Distinti dai timer periodici sopra: due unit **native** che portano su lo stack
|
|
host di VMware Workstation a ogni boot. Servono perché VMware su Linux **non**
|
|
fornisce unit systemd native, **non** registra `vmmon`/`vmnet` con DKMS e **non**
|
|
installa hook di post-install del kernel.
|
|
|
|
| unit | ruolo |
|
|
| ---- | ----- |
|
|
| `vmware-modules-rebuild.service` | ricostruisce `vmmon`/`vmnet` via `vmware-modconfig` se il `.ko` per il kernel in esecuzione manca (guard *build-if-missing*). Ordinata `Before=vmware-host.service`. Script: `/usr/local/sbin/vmware-modules-rebuild.sh`. |
|
|
| `vmware-host.service` | esegue `/etc/init.d/vmware start` → carica `vmmon`/`vmci` e avvia la rete `vmnet1`/`vmnet8` (DHCP/NAT). Ordinata `After=vmware-modules-rebuild.service`. |
|
|
|
|
Il sysv `vmware.service` (LSB) viene **mascherato**: si è dimostrato inaffidabile
|
|
da generare al boot (`systemd-sysv-generator: Cannot find unit vmware.service`),
|
|
lasciando `vmmon` non caricato e la rete VM giù → le build VM non partono.
|
|
`vmware-host.service` ne diventa l'unico owner.
|
|
|
|
### Installazione
|
|
|
|
```bash
|
|
sudo ./scripts/install-vmware-host-services.sh # installa + enable + start (idempotente)
|
|
```
|
|
|
|
Installato anche automaticamente da `setup-host-linux.sh` (Step 8/9).
|
|
|
|
### Verifica
|
|
|
|
```bash
|
|
systemctl is-enabled vmware-host.service vmware-modules-rebuild.service
|
|
lsmod | grep -E 'vmmon|vmnet'
|
|
ip -o link show | grep vmnet # attese: vmnet1, vmnet8
|
|
```
|
|
|
|
### Recovery (stack giù, VM non partono)
|
|
|
|
```bash
|
|
sudo systemctl start vmware-host.service
|
|
```
|
|
|
|
### Rollback
|
|
|
|
```bash
|
|
sudo ./scripts/install-vmware-host-services.sh --uninstall
|
|
```
|