feat(runner): dedicated rust-build runner instance for LinuxRust2404

`runs-on: rust-build` needs a second act_runner instance, not another label on
the existing one: act_runner resolves a single runner.envs block per process
and never tells the workflow which label picked the job. A label can select a
template only by selecting a runner whose envs name that template. Registered
as ci-linux-rust (id 3), online, announcing rust-build.

- deploy/runner-rust-config.yaml: label rust-build:host only, envs pointing at
  LinuxRust2404 + BaseClean-LinuxRust, its own cache dir (two processes must
  not share one actcache database), capacity 1 on top of the main runner's 2.
  runner.file is absolute: act_runner resolves a relative `file:` against the
  process CWD, so registering from elsewhere drops the registration state --
  which holds the runner token -- wherever it was invoked, and the daemon then
  fails with "registration file not found".
- scripts/install-rust-runner.sh: idempotent, --dry-run, refuses to touch the
  main runner directory, requires the template to exist first. Takes
  --token-file rather than --token so the secret stays out of the process
  argument list; the dry-run echo redacts it.
- tools/gitea-runner-token.py: mints a registration token from the stored
  GiteaPAT and writes it to a 0600 file. Neither secret is ever printed. Sets
  an explicit User-Agent because Cloudflare rejects urllib's default with
  error 1010.
- tools/gitea-runner-list.py: --label NAME exits non-zero unless an ONLINE
  runner announces it. A runs-on value no runner advertises leaves jobs
  unschedulable with no error anywhere, which is exactly the failure the
  consumer repo warned about.
This commit is contained in:
2026-07-22 14:09:22 +02:00
parent aa7e55d558
commit 498b0e9c05
5 changed files with 535 additions and 0 deletions
+84
View File
@@ -0,0 +1,84 @@
# act_runner configuration — rust-build instance (LinuxRust2404 template).
#
# Installed to /var/lib/ci/runner-rust/config.yaml by
# scripts/install-rust-runner.sh. Keep this file and the live one in sync.
#
# This instance exists only to give `runs-on: rust-build` its own template.
# act_runner resolves one `runner.envs` block per process and never tells the
# workflow which label picked the job, so routing a label to a template means
# routing it to a runner. Everything else (host paths, capacity budget) mirrors
# the main runner at /var/lib/ci/runner/config.yaml.
log:
level: info
runner:
# Absolute on purpose: act_runner resolves a relative `file` against the
# process CWD, so `register` run from anywhere but this directory would drop
# the registration state (which holds the runner token) wherever it happened
# to be invoked, and the daemon would then start with "registration file not
# found".
file: /var/lib/ci/runner-rust/.runner
# Concurrent jobs. This is ON TOP of the main runner's capacity — both
# instances clone VMs onto the same host. Keep main.capacity +
# rust.capacity within the host RAM budget (~8 GB per build VM).
capacity: 1
envs:
CI_PYTHON_LAUNCHER: /opt/ci/venv/bin/python
# The Rust template. `guest-os: 'Linux'` in the workflow resolves to this.
GITEA_CI_LINUX_TEMPLATE_PATH: /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx
# Read by the local-ci-build composite action; without it the action falls
# back to 'BaseClean-Linux', which does not exist on this template.
GITEA_CI_LINUX_SNAPSHOT_NAME: BaseClean-LinuxRust
# Windows template kept out on purpose: this runner announces no Windows
# label, so a Windows job can never be scheduled here.
env_file: .env
timeout: 3h
shutdown_timeout: 0s
insecure: false
fetch_timeout: 5s
fetch_interval: 2s
fetch_interval_max: 5s
workdir_cleanup_age: 24h
idle_cleanup_interval: 10m
log_report_interval: 5s
log_report_max_latency: 3s
log_report_batch_size: 100
state_report_interval: 5s
github_mirror: ''
# Exactly one label. Adding linux-build here would make this instance
# eligible for generic Linux jobs and hand them the Rust template.
labels:
- "rust-build:host"
allocate_pty: false
cache:
# Separate cache dir from the main runner: two processes must not share one
# actcache database.
enabled: true
dir: "/var/lib/ci/runner-rust/cache"
host: ""
port: 0
external_server: ""
external_secret: ""
container:
# All builds run in VMware VMs, never containers.
network: ""
privileged: false
options:
workdir_parent:
valid_volumes: []
docker_host: "-"
force_pull: false
force_rebuild: false
require_docker: false
docker_timeout: 0s
bind_workdir: false
host:
workdir_parent:
metrics:
enabled: false
addr: "127.0.0.1:9102"
+31
View File
@@ -0,0 +1,31 @@
[Unit]
# Second act_runner instance, dedicated to the Rust build template.
#
# Why a second instance instead of one more label on the existing runner:
# act_runner has a single `runner.envs` block per process, and it does not
# expose the job's runs-on label to the workflow (no RUNNER_LABELS variable).
# So a label can only select a *template* if it selects a *runner* whose envs
# name that template. One label per runner instance is the only wiring that
# actually routes.
#
# Provisioned by scripts/install-rust-runner.sh.
Description=Gitea Act Runner (rust-build)
After=network-online.target vmware-host.service
Wants=network-online.target
[Service]
Type=simple
User=ci-runner
WorkingDirectory=/var/lib/ci/runner-rust
Environment="PYTHONIOENCODING=utf-8"
Environment="CI_ROOT=/var/lib/ci"
Environment="CI_TEMPLATES=/var/lib/ci/templates"
Environment="CI_BUILD_VMS=/var/lib/ci/build-vms"
Environment="CI_ARTIFACTS=/var/lib/ci/artifacts"
Environment="CI_KEYS=/etc/ci/keys"
ExecStart=/opt/ci/act_runner/act_runner daemon --config /var/lib/ci/runner-rust/config.yaml
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
+192
View File
@@ -0,0 +1,192 @@
#!/usr/bin/env bash
# install-rust-runner.sh
#
# Provision the second act_runner instance that owns the `rust-build:host`
# label and the LinuxRust2404 template.
#
# Why a second instance: act_runner has one `runner.envs` block per process and
# does not expose the job's runs-on label to the workflow (there is no
# RUNNER_LABELS variable — v1.0.4 provides only RUNNER_OS/ARCH/TEMP/
# TOOL_CACHE/DEBUG/VERSION). A label can therefore only select a template by
# selecting a runner whose envs name that template.
#
# Usage:
# sudo ./scripts/install-rust-runner.sh --token-file <PATH>
# [--instance https://gitea.emulab.it] [--name ci-linux-rust]
# [--capacity 1] [--dry-run]
#
# The registration token is single-use and short-lived. Get one either from
# Gitea (Site Administration -> Actions -> Runners -> Create new Runner) or,
# using the stored GiteaPAT, with:
#
# sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
# --out /var/lib/ci/.rust-runner-regtoken
#
# Prefer --token-file over --token: an argument passed on the command line is
# visible to any user via `ps` for the lifetime of the process.
#
# Idempotent: re-running with an existing .runner file skips registration and
# only refreshes the config and the unit.
set -euo pipefail
INSTANCE="https://gitea.emulab.it"
RUNNER_NAME="ci-linux-rust"
TOKEN=""
TOKEN_FILE=""
CAPACITY=""
DRY_RUN=0
RUNNER_DIR="/var/lib/ci/runner-rust"
RUNNER_USER="ci-runner"
ACT_RUNNER_BIN="/opt/ci/act_runner/act_runner"
UNIT_NAME="act-runner-rust.service"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CONFIG_SRC="${REPO_ROOT}/deploy/runner-rust-config.yaml"
UNIT_SRC="${REPO_ROOT}/deploy/systemd/${UNIT_NAME}"
while [[ $# -gt 0 ]]; do
case "$1" in
--token) TOKEN="${2:?--token requires a value}"; shift ;;
# Preferred over --token: keeps the secret out of the process argument
# list (visible in `ps`) and out of shell history.
--token-file)
TOKEN_FILE="${2:?--token-file requires a value}"
[ -r "$TOKEN_FILE" ] || { echo "Token file not readable: $TOKEN_FILE" >&2; exit 1; }
TOKEN="$(tr -d '[:space:]' < "$TOKEN_FILE")"
[ -n "$TOKEN" ] || { echo "Token file is empty: $TOKEN_FILE" >&2; exit 1; }
shift
;;
--instance) INSTANCE="${2:?--instance requires a value}"; shift ;;
--name) RUNNER_NAME="${2:?--name requires a value}"; shift ;;
--capacity) CAPACITY="${2:?--capacity requires a value}"; shift ;;
--dry-run) DRY_RUN=1 ;;
*) echo "Unknown option: $1" >&2; exit 1 ;;
esac
shift
done
run() {
if [ "$DRY_RUN" = "1" ]; then
echo " [DRY-RUN] $*"
else
"$@"
fi
}
echo "=== install-rust-runner.sh ==="
echo " instance : ${INSTANCE}"
echo " name : ${RUNNER_NAME}"
echo " dir : ${RUNNER_DIR}"
echo " unit : ${UNIT_NAME}"
echo ""
# --- Pre-flight -------------------------------------------------------------
echo "--- Pre-flight ---"
[ "$(id -u)" -eq 0 ] || { echo "Must run as root (sudo)." >&2; exit 1; }
[ -x "$ACT_RUNNER_BIN" ] || { echo "act_runner not found: $ACT_RUNNER_BIN" >&2; exit 1; }
[ -f "$CONFIG_SRC" ] || { echo "Config template not found: $CONFIG_SRC" >&2; exit 1; }
[ -f "$UNIT_SRC" ] || { echo "Unit file not found: $UNIT_SRC" >&2; exit 1; }
id "$RUNNER_USER" >/dev/null 2>&1 || { echo "User $RUNNER_USER does not exist." >&2; exit 1; }
# The template this runner is dedicated to must exist, otherwise the first job
# fails at clone time with a much less obvious error.
RUST_VMX="/var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx"
if [ ! -f "$RUST_VMX" ]; then
echo "Rust template not found: $RUST_VMX" >&2
echo "Run 'ci-orchestrator template deploy-rust --then-prepare' first." >&2
exit 1
fi
echo " [OK] act_runner: $ACT_RUNNER_BIN"
echo " [OK] template: $RUST_VMX"
# A second instance must not reuse the main runner's state directory.
if [ "$RUNNER_DIR" = "/var/lib/ci/runner" ]; then
echo "Refusing to overwrite the main runner directory." >&2
exit 1
fi
# --- Directory + config -----------------------------------------------------
echo ""
echo "--- Directory + config ---"
run install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0750 "$RUNNER_DIR"
run install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0750 "${RUNNER_DIR}/cache"
run install -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0644 "$CONFIG_SRC" "${RUNNER_DIR}/config.yaml"
if [ -n "$CAPACITY" ]; then
run sed -i "s/^ capacity: .*/ capacity: ${CAPACITY}/" "${RUNNER_DIR}/config.yaml"
fi
echo " [OK] ${RUNNER_DIR}/config.yaml"
# --- Registration -----------------------------------------------------------
echo ""
echo "--- Registration ---"
if [ -f "${RUNNER_DIR}/.runner" ]; then
echo " [SKIP] ${RUNNER_DIR}/.runner exists — already registered."
echo " Delete it and re-run to register as a new runner."
elif [ -z "$TOKEN" ]; then
echo " No --token given and no existing .runner." >&2
echo " Get a registration token from Gitea (Actions -> Runners ->" >&2
echo " Create new Runner) and re-run with --token <token>." >&2
exit 1
else
# Never let the token reach stdout: `run` echoes its arguments in dry-run
# mode, so the register call is handled separately with a redacted echo.
if [ "$DRY_RUN" = "1" ]; then
echo " [DRY-RUN] ${ACT_RUNNER_BIN} register --no-interactive" \
"--instance ${INSTANCE} --token <redacted:${#TOKEN} chars>" \
"--name ${RUNNER_NAME} --labels rust-build:host"
else
# cd into the runner dir: act_runner resolves `runner.file` relative to
# the CWD. The shipped config uses an absolute path, but staying here
# keeps it correct even if that is ever edited back to `.runner`.
# --labels is accepted but ignored by `register` when the config file
# declares labels; the config is the single source of truth.
( cd "$RUNNER_DIR" && sudo -u "$RUNNER_USER" env HOME="$RUNNER_DIR" \
"$ACT_RUNNER_BIN" register \
--no-interactive \
--instance "$INSTANCE" \
--token "$TOKEN" \
--name "$RUNNER_NAME" \
--config "${RUNNER_DIR}/config.yaml" )
chown "$RUNNER_USER:$RUNNER_USER" "${RUNNER_DIR}/.runner"
chmod 0600 "${RUNNER_DIR}/.runner"
echo " [OK] registered as ${RUNNER_NAME}"
fi
fi
# --- systemd unit -----------------------------------------------------------
echo ""
echo "--- systemd unit ---"
run install -o root -g root -m 0644 "$UNIT_SRC" "/etc/systemd/system/${UNIT_NAME}"
run systemctl daemon-reload
run systemctl enable --now "$UNIT_NAME"
echo " [OK] ${UNIT_NAME} enabled and started"
# --- Verification -----------------------------------------------------------
echo ""
echo "--- Verification ---"
if [ "$DRY_RUN" = "1" ]; then
echo " [DRY-RUN] skipped"
else
sleep 3
systemctl is-enabled "$UNIT_NAME"
systemctl is-active "$UNIT_NAME"
echo ""
echo " Labels registered:"
sudo -u "$RUNNER_USER" grep -A4 '"labels"' "${RUNNER_DIR}/.runner" || true
echo ""
echo " Recent log:"
journalctl -u "$UNIT_NAME" -n 15 --no-pager || true
fi
echo ""
echo "=== done ==="
echo "Confirm the runner shows Online in Gitea with label 'rust-build', then"
echo "a workflow can use:"
echo ""
echo " runs-on: rust-build"
echo " # guest-os: 'Linux' -> LinuxRust2404 + BaseClean-LinuxRust"
echo ""
echo "Check that the label is actually announced BEFORE relying on it: a"
echo "runs-on value no runner announces leaves the job unschedulable with no"
echo "error anywhere."
+106
View File
@@ -0,0 +1,106 @@
#!/opt/ci/venv/bin/python
"""List the act_runners Gitea knows about, with their status and labels.
Answers the question that matters before writing a `runs-on:` value: is that
label actually announced by an online runner? A runs-on that no runner
advertises leaves the job unschedulable with no error anywhere — the CI just
goes quiet.
Usage:
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py
... --label rust-build # exit 1 unless an online runner announces it
Reads the PAT from the keyring and never prints it.
"""
from __future__ import annotations
import argparse
import json
import sys
import urllib.error
import urllib.request
DEFAULT_INSTANCE = "https://gitea.emulab.it"
CRED_TARGET = "GiteaPAT"
# Cloudflare in front of the instance rejects the default "Python-urllib/x.y"
# agent with error 1010; any explicit agent passes.
USER_AGENT = "ci-orchestrator-gitea-runner-list/1.0"
def _read_pat(target: str) -> str:
try:
import keyring
except ImportError: # pragma: no cover - environment problem, not logic
sys.exit("keyring not installed")
cred = keyring.get_credential(target, None)
if cred and cred.password:
return str(cred.password)
username = keyring.get_password(f"{target}:meta", "username")
if username:
password = keyring.get_password(target, username)
if password:
return str(password)
sys.exit(f"credential {target!r} not found in keyring")
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--instance", default=DEFAULT_INSTANCE)
ap.add_argument("--target", default=CRED_TARGET)
ap.add_argument("--label", default="", help="require this label on an online runner")
args = ap.parse_args()
url = args.instance.rstrip("/") + "/api/v1/admin/actions/runners"
req = urllib.request.Request(
url,
headers={
"Authorization": f"token {_read_pat(args.target)}",
"Accept": "application/json",
"User-Agent": USER_AGENT,
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
payload = json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
print(f"HTTP {exc.code} from {url}", file=sys.stderr)
return 1
except urllib.error.URLError as exc:
print(f"cannot reach {url}: {exc.reason}", file=sys.stderr)
return 1
runners = payload.get("runners", payload) if isinstance(payload, dict) else payload
if not isinstance(runners, list):
print(f"unexpected response shape: {type(runners).__name__}", file=sys.stderr)
return 1
print(f"{'ID':<5} {'NAME':<20} {'STATUS':<10} LABELS")
matched = False
for r in runners:
# Gitea reports labels either as plain strings or as {name: ...} objects.
labels = [
str(x.get("name", x)) if isinstance(x, dict) else str(x)
for x in (r.get("labels") or [])
]
status = str(r.get("status", "?"))
name = str(r.get("name", "?"))
print(f"{r.get('id', '?'):<5} {name:<20} {status:<10} {','.join(labels)}")
if args.label and args.label in labels and status.lower() == "online":
matched = True
if args.label:
if matched:
print(f"\nOK — an online runner announces {args.label!r}.")
return 0
print(
f"\nFAIL — no ONLINE runner announces {args.label!r}. "
"A job with that runs-on would sit unschedulable, silently.",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+122
View File
@@ -0,0 +1,122 @@
#!/opt/ci/venv/bin/python
"""Mint a Gitea act_runner registration token using the stored GiteaPAT.
The PAT never leaves this process and the registration token is written to a
0600 file instead of stdout, so neither secret lands in a shell history, a
process argument list, or a terminal log.
Usage:
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
--out /var/lib/ci/.rust-runner-regtoken
# non-instance scopes
... --scope user
... --scope org --org myorg
... --scope repo --repo Simone/local-ci-cd-system
Instance scope (the default) requires the PAT to belong to an admin user and
produces a runner visible to every repository — which is what a shared build
host wants. Registration tokens are short-lived and single-use.
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
DEFAULT_INSTANCE = "https://gitea.emulab.it"
CRED_TARGET = "GiteaPAT"
USER_AGENT = "ci-orchestrator-gitea-runner-token/1.0"
def _read_pat(target: str) -> str:
"""Read the PAT via the same two-tier scheme as KeyringCredentialStore."""
try:
import keyring
except ImportError: # pragma: no cover - environment problem, not logic
sys.exit("keyring not installed")
cred = keyring.get_credential(target, None)
if cred and cred.password:
return str(cred.password)
username = keyring.get_password(f"{target}:meta", "username")
if username:
password = keyring.get_password(target, username)
if password:
return str(password)
sys.exit(f"credential {target!r} not found in keyring")
def _endpoint(scope: str, org: str, repo: str) -> str:
if scope == "admin":
return "/api/v1/admin/actions/runners/registration-token"
if scope == "user":
return "/api/v1/user/actions/runners/registration-token"
if scope == "org":
if not org:
sys.exit("--scope org requires --org")
return f"/api/v1/orgs/{org}/actions/runners/registration-token"
if not repo or "/" not in repo:
sys.exit("--scope repo requires --repo owner/name")
owner, name = repo.split("/", 1)
return f"/api/v1/repos/{owner}/{name}/actions/runners/registration-token"
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--instance", default=DEFAULT_INSTANCE)
ap.add_argument("--scope", default="admin", choices=("admin", "user", "org", "repo"))
ap.add_argument("--org", default="")
ap.add_argument("--repo", default="")
ap.add_argument("--target", default=CRED_TARGET, help="keyring target holding the PAT")
ap.add_argument("--out", required=True, help="file to write the token to (mode 0600)")
args = ap.parse_args()
pat = _read_pat(args.target)
url = args.instance.rstrip("/") + _endpoint(args.scope, args.org, args.repo)
req = urllib.request.Request(
url,
method="POST",
data=b"",
headers={
"Authorization": f"token {pat}",
"Accept": "application/json",
"Content-Length": "0",
# Cloudflare in front of the instance rejects the default
# "Python-urllib/x.y" agent with error 1010; any explicit agent
# passes. Identify the caller rather than impersonate a browser.
"User-Agent": USER_AGENT,
},
)
try:
with urllib.request.urlopen(req, timeout=30) as resp:
payload = json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", "replace")[:300]
# 403 on the admin scope usually means the PAT owner is not an admin.
print(f"HTTP {exc.code} from {url}\n{body}", file=sys.stderr)
return 1
except urllib.error.URLError as exc:
print(f"cannot reach {url}: {exc.reason}", file=sys.stderr)
return 1
token = payload.get("token") or payload.get("data", {}).get("token")
if not token:
print(f"no token in response: {sorted(payload)}", file=sys.stderr)
return 1
fd = os.open(args.out, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w", encoding="ascii") as fh:
fh.write(token)
print(f"OK — registration token ({len(token)} chars) written to {args.out}")
print("Single-use and short-lived: delete the file once the runner is registered.")
return 0
if __name__ == "__main__":
raise SystemExit(main())