Stabilize self-hosted CI workflows and resolve issue #62 #73

Open

darkhelm wants to merge 76 commits from feature/two-stage-runtime-images into main

pull from: feature/two-stage-runtime-images

merge into: DarkHelm.org:main

DarkHelm.org:main

Conversation 0 Commits 76 Files Changed 35 +3430 -2743

darkhelm commented 2026-07-10 17:58:15 -04:00

Owner

Copy Link

Summary

Hardens CI workflows for self-hosted Gitea runners by stabilizing E2E execution and Renovate behavior across internal/external network paths.

Closes #62

What Changed

E2E workflow reliability

• Fixed E2E workspace handoff to ensure expected repository contents are present during test execution.
• Added stricter preflight checks for required frontend files before running E2E.
• Reduced mount/path fragility while preserving runtime image pull and compose flow.

Renovate workflow hardening

• Added internal-first endpoint reachability selection with fallback handling.
• Added token preflight checks for repository access.
• Added explicit host-rule auth handling for API/git paths.
• Added container-level connectivity preflight diagnostics.
• Added git URL override aligned with selected endpoint context.
• Removed incorrect forced Dogar host-IP pinning that broke HTTPS clone routing.

Why

CI behavior was sensitive to runner networking and Renovate clone/auth interactions. These changes make the workflow deterministic in our runner topology and address recurring CI failures.

Scope

• Workflow logic only (cicd.yaml, renovate.yml)
• No app feature or API behavior changes

Validation

• Workflow YAML validation passed during updates.
• Changes were applied and verified iteratively from real failing run diagnostics.

## Summary Hardens CI workflows for self-hosted Gitea runners by stabilizing E2E execution and Renovate behavior across internal/external network paths. Closes #62 ## What Changed ### E2E workflow reliability - Fixed E2E workspace handoff to ensure expected repository contents are present during test execution. - Added stricter preflight checks for required frontend files before running E2E. - Reduced mount/path fragility while preserving runtime image pull and compose flow. ### Renovate workflow hardening - Added internal-first endpoint reachability selection with fallback handling. - Added token preflight checks for repository access. - Added explicit host-rule auth handling for API/git paths. - Added container-level connectivity preflight diagnostics. - Added git URL override aligned with selected endpoint context. - Removed incorrect forced Dogar host-IP pinning that broke HTTPS clone routing. ## Why CI behavior was sensitive to runner networking and Renovate clone/auth interactions. These changes make the workflow deterministic in our runner topology and address recurring CI failures. ## Scope - Workflow logic only (`cicd.yaml`, `renovate.yml`) - No app feature or API behavior changes ## Validation - Workflow YAML validation passed during updates. - Changes were applied and verified iteratively from real failing run diagnostics.

copilotcoder was assigned by darkhelm 2026-07-10 17:58:16 -04:00

darkhelm added 60 commits 2026-07-10 17:58:20 -04:00

Refactor CI and runtime images to two-stage builds 7c06fa3474

Trigger CICD on every push fe02644e1d

Reclaim runner disk before CICD image build 9a43d9090b

Keep integration tests out of backend unit lane 0791f9257e

Align CI and Poe test lanes with runtime compose flows 4af111fcdb

ci: retrigger workflow run attempt 2 2df9dbe7a0

ci: retrigger workflow run attempt 2 4855be1c12

Handle missing frontend .yarn directory in runtime build 6f3d96d6ef

ci: retrigger workflow run attempt 2 6963c3ae2f

Resolve CICD image tag for runtime-based jobs 738ab39736

Run integration and E2E against deployable app images 08d36e958b

ci: retrigger workflow run 71b9dda8a4

Add runner diagnostics and rebalance source lanes 8619e00a17

Discover actual runner containers dynamically d59217912d

Use full runner pool for source test lanes 0ac6108fc7

Retry runtime image pushes on transient registry resets f1651affd0

Harden runtime health probes in integration and e2e jobs b7b66c9e26

Add runtime-images postmortem job for setup-stage failures 1e9e3f0c41

Add source-lane postmortem job for setup-stage failures 9301466f9e

Add tester image lanes and production gates to CICD workflow f0b259e1c3

Separate backend/frontend base-main lanes and add postmortem coverage d4aa16e18a

Harden E2E tester build against transient MCR DNS failures bd174d8186

Adding the workflow spreadsheet ... e246e6d5bc

Signed-off-by: copilotcoder <copilotcoder@darkhelm.org>

Fix hyphenated needs references in CI gate expressions ca99393700

Harden CICD image refs against empty SHA/base hash 6f8913ba70

Use internal Playwright mirror for E2E tester builds 93529b91ec

ci: retrigger workflow run 062f19b4aa

Remove residual runner-2 diagnostics drift handling e655540b48

ci: retrigger workflow run d341005a26

Retry base image registry operations on transient resets 26ea4ce136

Use ubuntu-act for non-build jobs and fix E2E backend probe 2352152f50

Retry backend main minimal checkout network operations 6a4484a2f0

Retry Corepack/Yarn bootstrap in frontend image build e124045f1a

Fix Renovate endpoint to use Gitea API path 4b56d4e13b

Standardize CI retries for checkout and runtime pulls 5d37379aac

Run Renovate via container instead of npm install 548232afe2

Fix Renovate config mount path in container run 3fed66c2a0

Fix E2E frontend readiness probe to use compose service check 231074ae40

Fix E2E frontend probe and remove Renovate config mount dependency 758967e4e6

Retry frontend-base registry login and harden Renovate endpoint auth acb446c30f

Use nginx-side readiness probe for E2E frontend aa1c1b7900

Force Renovate token-only auth for Gitea d3890d5726

Instrument Renovate container auth preflight f24de1dcab

Harden runner diagnostics and warm image caches 9731fee9be

Fix E2E frontend path and Renovate endpoint 916b276c08

Pin registry auth realm host in build_cicd f0c4b72dd6

Add guarded pre-job docker prune in CICD 7a3b6f5781

Fix build_cicd function scope and Renovate DNS fallback c4d1102f6f

Fix missing registry auth helper in build_cicd job 955be4e2ec

Refactor CICD workflow anchors and remove duplicate helper 47214027da

Remove dead legacy runtime image workflow path 383b88e87d

Fix Renovate internal and external Gitea endpoint selection a48c7e6ba4

Relax Renovate token preflight and fix E2E frontend mount d0802f70c9

Avoid Renovate preflight failure on missing read:user scope 85960e3599

Fix Renovate repo parsing and git TLS clone in runner 331fb887d5

Remove invalid Renovate gitNoVerify env override a3e864cfcd

Add explicit git host credentials for Renovate clone 0ac17105b8

Finalize workflow fixes for Renovate and E2E 9a68fa1109

Force Renovate git URL to selected endpoint base 64bd56e36e

Remove incorrect dogar host IP override for Renovate 72e775d2b7

darkhelm changed title from Harden CI workflows for self-hosted Gitea runners (E2E + Renovate) to Stabilize self-hosted CI workflows and resolve issue #62 2026-07-10 18:01:57 -04:00

darkhelm added 1 commit 2026-07-10 18:06:11 -04:00

Rewrite Renovate git clone URLs to internal authenticated host fa0f582759

darkhelm added 1 commit 2026-07-10 18:58:02 -04:00

Configure in-container git rewrite for Renovate clone auth 9bb36d18a4

darkhelm added 1 commit 2026-07-10 23:06:34 -04:00

Prefer local kankali:2222 SSH clone with HTTP fallback 6aadca5df2

darkhelm added 1 commit 2026-07-11 08:36:37 -04:00

Keep CI git/registry traffic on local kankali endpoints 4ebf9ef214

darkhelm added 1 commit 2026-07-11 08:45:50 -04:00

Use local renovate image mirror before GHCR fallback fb0d7dabec

darkhelm added 1 commit 2026-07-11 09:06:55 -04:00

Fix renovate image selector output for docker run 8a32017242

darkhelm added 1 commit 2026-07-11 09:33:48 -04:00

Expand mirror repair script for Renovate and Playwright images 35389ad15b

darkhelm added 1 commit 2026-07-11 09:33:57 -04:00

Add xonsh shebang for repair_runner_mirror script 32b218a97b

darkhelm added 1 commit 2026-07-11 12:40:59 -04:00

Fix xonsh syntax in mirror repair remote snippet 6630805d31

darkhelm added 1 commit 2026-07-11 17:08:12 -04:00

Publish mirrors only from kankali and improve diagnostics 5232db7289

darkhelm added 1 commit 2026-07-11 17:38:02 -04:00

Silence renovate image selector stdout contamination 9bea834999

darkhelm added 1 commit 2026-07-11 23:10:38 -04:00

Stabilize Renovate manager scope and remove OSV fetch efe1a7ee50

darkhelm added 1 commit 2026-07-12 07:17:52 -04:00

Reduce Renovate memory pressure and skip OOM retry 7e56467825

darkhelm added 1 commit 2026-07-12 07:38:55 -04:00

Remove unsupported Renovate vulnerability-alerts flag a0903cf797

darkhelm added 1 commit 2026-07-12 08:48:43 -04:00

Force Renovate to use workspace config ff18402320

darkhelm added 1 commit 2026-07-13 00:43:13 -04:00

Load Renovate config from git HEAD b807dcb115

Some checks failed

Hide all checks

CICD / Build and Publish CICD Base Image (push) Successful in 54s

Details

CICD / Build and Publish CICD Base Image (pull_request) Successful in 52s

Details

CICD / Build and Push CICD Image (pull_request) Successful in 21m35s

Details

CICD / Build CICD Image Failure Postmortem (pull_request) Has been skipped

Details

CICD / Backend Tests (pull_request) Successful in 13m46s

Details

CICD / Frontend Tests (pull_request) Successful in 11m9s

Details

CICD / Pre-commit Checks (pull_request) Failing after 32m42s

Details

CICD / Backend Doctests (pull_request) Successful in 10m50s

Details

CICD / CICD Tests Complete (pull_request) Failing after 11s

Details

CICD / Build Backend Base Image (pull_request) Has been skipped

Details

CICD / Build Frontend Base Image (pull_request) Has been skipped

Details

CICD / Build Integration Tester Image (pull_request) Has been skipped

Details

CICD / Build E2E Tester Image (pull_request) Has been skipped

Details

CICD / Build Backend Main Image (pull_request) Has been skipped

Details

CICD / Build Frontend Main Image (pull_request) Has been skipped

Details

CICD / Source Lanes Failure Postmortem (pull_request) Successful in 19s

Details

CICD / Production Images Complete (pull_request) Failing after 11s

Details

CICD / Production Image Failures Postmortem (pull_request) Has been skipped

Details

CICD / Runtime Black-Box Integration Tests (pull_request) Has been skipped

Details

CICD / End-to-End Tests (pull_request) Has been skipped

Details

CICD / Integration Tests Failure Postmortem (pull_request) Has been skipped

Details

CICD / E2E Tests Failure Postmortem (pull_request) Has been skipped

Details

CICD / Build and Push CICD Image (push) Successful in 1h14m22s

Details

CICD / Build CICD Image Failure Postmortem (push) Has been skipped

Details

CICD / Backend Tests (push) Successful in 1m26s

Details

CICD / Frontend Tests (push) Successful in 2m4s

Details

CICD / Backend Doctests (push) Successful in 46s

Details

CICD / Pre-commit Checks (push) Successful in 12m19s

Details

CICD / CICD Tests Complete (push) Successful in 8s

Details

CICD / Source Lanes Failure Postmortem (push) Has been skipped

Details

CICD / Build Backend Base Image (push) Successful in 3m47s

Details

CICD / Build Integration Tester Image (push) Successful in 1m21s

Details

CICD / Build Frontend Base Image (push) Successful in 14m48s

Details

CICD / Build Backend Main Image (push) Successful in 2m48s

Details

CICD / Build E2E Tester Image (push) Successful in 17m27s

Details

CICD / Build Frontend Main Image (push) Successful in 4m2s

Details

CICD / Production Image Failures Postmortem (push) Has been skipped

Details

CICD / Production Images Complete (push) Successful in 7s

Details

CICD / Runtime Black-Box Integration Tests (push) Successful in 2m12s

Details

CICD / Integration Tests Failure Postmortem (push) Has been skipped

Details

CICD / End-to-End Tests (push) Successful in 40m28s

Details

CICD / E2E Tests Failure Postmortem (push) Has been skipped

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/two-stage-runtime-images:feature/two-stage-runtime-images

git checkout feature/two-stage-runtime-images

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

afk

Autonomous-friendly issue

algorithm

Selection logic

api

API contract

backend

Backend related

database

Database related

devex

Developer experience

docs

Documentation

epic

Top-level planning item

frontend

Frontend related

hitl

Requires human-in-the-loop decision

ingestion

Ingestion/sync

ops

Operations

performance

Performance

realtime

Realtime/eventing

release

Release readiness

security

Security

task

Implementation ticket

testing

Testing

ui

UI/UX

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

copilotcoder darkhelm renovate-bot repository-bot

No Assignees copilotcoder

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: DarkHelm.org/plex-playlist#73