GitLab Runner Connected Preparation
Use GitLab Runner as a replaceable connected execution host, not as the source of application truth.
Canonical pattern:
canonical source repository
-> internal GitLab pull mirror
-> current GitLab Runner
-> connected preparation
-> short-lived cloud identity when required
-> bounded internal artifact publication
-> offline/private runtime
1. Keep one source authority
When GitHub is canonical and GitLab exists for internal CI/CD, use GitHub -> GitLab Pull mirror -> GitLab CI/Runner.
Do not configure push or bidirectional mirroring unless the project explicitly requires a different authority model.
Mirror acceptance must be based on real refs and SHA equality, not a UI status alone:
MIRROR_DIRECTION=Pull
SOURCE_MAIN_SHA=<sha>
TARGET_MAIN_SHA=<same sha>
TARGET_MAIN_EXISTS=true
RESULT=PASS
2. Prove a current Runner first
Historical Runner success is architectural evidence, not current capacity. Before CI work record Runner availability, status, executor, architecture, and tags.
Runner discovery itself is a safety gate:
- paginate every project/group Runner listing available to the caller;
- read candidate Runner details rather than trusting only the list row;
- treat an admin-only global Runner API returning 403 as a visibility limitation, not proof that no project-visible Runner can execute;
- require one actually scheduled job before declaring a Runner tag usable.
A first API page can contain only stale historical entries while later pages contain current pools. Do not make an availability decision from page 1 alone.
A stale, paused, offline Runner, or one backed by deleted compute, is a platform-availability problem. If repository automation has no runner-administration authority, classify it as:
RESULT=BLOCKED
BLOCKER=RUNNER_ADMIN_ACTION_REQUIRED
Stop before application preparation.
3. Use the cheapest preflight first
Advance in this order:
- current Runner;
- architecture and executor;
- outbound Internet to one approved public endpoint;
- required local tools;
- short-lived identity if cloud access is needed;
- bounded storage/registry access;
- application-specific preparation.
A generic health result is:
RUNNER_AVAILABLE=PASS
RUNNER_INTERNET=PASS
REQUIRED_TOOLS=PASS
RESULT=PASS
Stop at the first meaningful blocker.
Evidence-based sizing
Choose Runner size from measured workload behavior rather than names alone.
A useful default progression is:
small -> intake, lint, short discovery
medium -> normal connected preparation
large -> image-heavy or artifact-heavy preparation
xlarge -> only after a measured capacity failure justifies it
Record CPU, memory, scratch disk, queue/start delay, run time, and representative image/artifact behavior before standardizing a size.
Prefer the architecture required by the target workload. For x86_64 container and workflow targets, validate amd64 first; use Arm only when specifically required or beneficial.
4. Keep CI orchestration thin
CI configuration should call repository-owned scripts. Do not duplicate discovery, build, migration, validation, or release logic inside a large .gitlab-ci.yml.
Prefer: GitLab CI -> generic runner-health script -> project-owned preparation command -> compact evidence artifact.
5. Use a dedicated preparation image when needed
Hosted or autoscaled runners often begin with a minimal job image. If tools are missing, use a dedicated preparation image and record its immutable image digest plus tool versions.
Do not assume Docker-in-Docker is available. Unprivileged runners may reject privileged Docker builders. Use an approved rootless/unprivileged builder or a prebuilt preparation image.
Keep the preparation image separate from application/runtime images.
Treat preparation identity as a compact immutable set:
source revision
preparation image digest
exact required tool versions
compact result/evidence
For bootstrap installers that read environment variables, ensure the variable reaches the shell that performs installation and assert the resulting version. A successful download of an unpinned latest version is not an equivalent proof.
For Python CLIs on externally-managed system Python installations, prefer an isolated virtual environment over modifying the system interpreter.
6. Prefer short-lived workload identity
When a Runner must access a cloud account, prefer CI-native OIDC/workload identity over static keys.
Use identity-first sequencing: Runner -> CI ID token -> cloud STS/workload identity -> temporary credentials -> identity readback only.
First prove:
OIDC_TRUST=PASS
CLOUD_IDENTITY=PASS
RAW_TOKEN_RECORDED=NO
TEMPORARY_CREDENTIALS_RECORDED=NO
Do not grant storage, registry, deployment, or administration rights as part of the identity proof.
Run identity-only proof from an already trusted project/ref. Do not widen workload-identity trust merely to make a temporary test branch succeed.
7. Add data-plane access incrementally
After identity succeeds, add only the smallest required scope: one storage prefix, one registry namespace, read-only metadata, or one disposable smoke object/image.
Keep evidence truthful. If an out-of-scope deny test was not run, record NOT_RUN rather than inventing PASS.
8. Separate connected preparation from offline runtime
Connected preparation may fetch source, inspect dependencies, resolve plugins/images, and stage approved assets. Offline/private runtime must consume only staged internal artifacts.
Do not infer offline-runtime proof merely because connected preparation succeeded.
When AWS-local validation or promotion belongs inside the cloud trust boundary, use a clean handoff:
GitLab Runner -> immutable prepared handoff / control call -> CodeBuild -> private AWS path
Do not try to run a GitLab Runner inside CodeBuild. Treat the two execution planes as complementary: Runner for connected preparation/control, CodeBuild for AWS-account-local validation or promotion.
9. Mirror-compatible repository policy
Externally authored history may conflict with project push rules that assume every commit was authored, signed, or mapped to a local GitLab user.
For a pull-mirror repository, evaluate verified-user-only authorship, mandatory local-user mapping, and mandatory commit-signing rules for technical compatibility. Do not weaken group or instance policy globally. Apply only the smallest approved repository-level exception.
Keep secret-detection controls enabled unless an owning security policy says otherwise.
10. Common failure modes
- Wrong mirror direction: verify Pull for source -> GitLab use cases.
- Green mirror status but no refs: inspect branches/commits and compare SHAs.
- Runner stale/offline: paginate first, inspect candidate detail, then classify as external platform/admin recovery only after scheduling cannot find a usable pool.
- Global Runner API returns 403: treat it as an authorization/visibility limit, not a Runner-availability result.
- Tools absent: use a preparation image; keep CI YAML thin.
- Privileged Docker unavailable: use approved unprivileged/rootless build.
- OIDC failure: verify issuer, audience, subject/project/ref constraints, and trust before changing permissions.
- Architecture mismatch: record Runner architecture before selecting images or native binaries.
11. Minimal evidence packet
SOURCE_AUTHORITY=
MIRROR_DIRECTION=
SOURCE_MAIN_SHA=
TARGET_MAIN_SHA=
MIRROR_RESULT=
RUNNER_AVAILABLE=
RUNNER_STATUS=
RUNNER_DISCOVERY_PAGINATED=PASS|BLOCKED
RUNNER_SCHEDULE_PROOF=PASS|BLOCKED|NOT_RUN
RUNNER_EXECUTOR=
RUNNER_ARCHITECTURE=
RUNNER_TAGS=
RUNNER_INTERNET=
REQUIRED_TOOLS=
WORKLOAD_IDENTITY=PASS|NOT_REQUIRED|BLOCKED
DATA_PLANE_ACCESS=PASS|NOT_REQUIRED|NOT_RUN
APPLICATION_PREPARATION=PASS|BLOCKED|NOT_RUN
RESULT=PASS|BLOCKED|FAIL
NEXT=<one action>
Keep evidence compact and non-secret.
12. Promotion boundary
Project repositories own scripts, CI files, permissions, runner tags, and runtime evidence. Agent OS owns reusable policy and sequence guidance. Local host/tool defaults belong in local/dotfiles guidance.
Do not turn Agent OS into a copy of one project implementation.
Citations
- https://github.com/amitkarpe/nextflow-offline/issues/22
- https://github.com/amitkarpe/nextflow-offline/pull/23
- https://github.com/amitkarpe/nextflow-offline/pull/25
- https://github.com/amitkarpe/nextflow-offline/issues/26
- https://github.com/amitkarpe/nextflow-offline/pull/27
- https://github.com/amitkarpe/nextflow-offline/issues/28
- https://github.com/amitkarpe/nextflow-offline/pull/29
- https://github.com/amitkarpe/nextflow-offline/pull/39
- https://github.com/amitkarpe/cloudos-offline/pull/6