Files
plex-playlist/scripts/gitea-actions/repair_runner_mirror.xsh
Xlorep DarkHelm f3698b095b
All checks were successful
CICD / Build and Publish CICD Base Image (push) Successful in 6m8s
CICD / Build and Push CICD Image (push) Successful in 23m14s
CICD / Build CICD Image Failure Postmortem (push) Has been skipped
CICD / Backend Tests (push) Successful in 7m10s
CICD / Frontend Tests (push) Successful in 45s
CICD / Backend Doctests (push) Successful in 18s
CICD / Pre-commit Checks (push) Successful in 14m57s
CICD / Source Lanes Failure Postmortem (push) Has been skipped
CICD / CICD Tests Complete (push) Successful in 3s
CICD / Build Backend Base Image (push) Successful in 18s
CICD / Build Integration Tester Image (push) Successful in 1m5s
CICD / Build Backend Main Image (push) Successful in 1m52s
CICD / Build Frontend Base Image (push) Successful in 10m42s
CICD / Build Frontend Main Image (push) Successful in 33s
CICD / Build E2E Tester Image (push) Successful in 32m17s
CICD / Production Images Complete (push) Successful in 5s
CICD / Production Image Failures Postmortem (push) Has been skipped
CICD / Runtime Black-Box Integration Tests (push) Successful in 1m13s
CICD / Integration Tests Failure Postmortem (push) Has been skipped
CICD / End-to-End Tests (push) Successful in 11m23s
CICD / E2E Tests Failure Postmortem (push) Has been skipped
Stabilize self-hosted CI workflows and resolve issue #62 (#73)
## Summary

Hardens CI workflows for self-hosted Gitea runners by stabilizing E2E execution and Renovate behavior across internal/external network paths.

Closes #62

## What Changed

### E2E workflow reliability
- Fixed E2E workspace handoff to ensure expected repository contents are present during test execution.
- Added stricter preflight checks for required frontend files before running E2E.
- Reduced mount/path fragility while preserving runtime image pull and compose flow.

### Renovate workflow hardening
- Added internal-first endpoint reachability selection with fallback handling.
- Added token preflight checks for repository access.
- Added explicit host-rule auth handling for API/git paths.
- Added container-level connectivity preflight diagnostics.
- Added git URL override aligned with selected endpoint context.
- Removed incorrect forced Dogar host-IP pinning that broke HTTPS clone routing.

## Why

CI behavior was sensitive to runner networking and Renovate clone/auth interactions. These changes make the workflow deterministic in our runner topology and address recurring CI failures.

## Scope

- Workflow logic only (`cicd.yaml`, `renovate.yml`)
- No app feature or API behavior changes

## Validation

- Workflow YAML validation passed during updates.
- Changes were applied and verified iteratively from real failing run diagnostics.

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #73
2026-07-13 11:16:16 -04:00

99 lines
3.4 KiB
Plaintext
Executable File

#!/usr/bin/env xonsh
hosts = [
"kankali.darkhelm.lan",
"zhokq.darkhelm.lan",
"urtzul.darkhelm.lan",
"pi-desktop.darkhelm.lan",
]
publisher_host = "kankali.darkhelm.lan"
# Pull upstream tags on the registry host, publish mirror tags there, then verify pullability on every host.
remote_code_template = """
mirror_pairs = [
("ACT_UBUNTU", "ghcr.io/catthehacker/ubuntu:act-latest", "kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest"),
("RENOVATE", "ghcr.io/renovatebot/renovate:41", "kankali.darkhelm.lan:3001/darkhelm.org/renovate:41"),
("PLAYWRIGHT", "mcr.microsoft.com/playwright:v1.56.1-jammy", "kankali.darkhelm.lan:3001/darkhelm.org/playwright-browsers:v1.56.1-jammy"),
]
is_publisher = __IS_PUBLISHER__
def classify_failure(prefix, log_path):
mismatch = !(grep -qi "http response to https client" @(log_path))
if mismatch.returncode == 0:
print(f"{prefix}:https-mismatch")
else:
print(f"{prefix}:failed")
tail = !(tail -n 20 @(log_path) 2> /dev/null)
if tail.returncode == 0 and str(tail.out).strip():
print(f"{prefix}_LOG_START")
print(str(tail.out).rstrip())
print(f"{prefix}_LOG_END")
def image_present(image):
result = !(docker image inspect @(image) > /dev/null 2>&1)
return result.returncode == 0
ubuntu_pull_log = "/tmp/ubuntu22-pull.log"
ubuntu_pull = !(docker pull ubuntu:22.04 > @(ubuntu_pull_log) 2>&1)
if ubuntu_pull.returncode == 0:
print("UBUNTU22_PULL:ok")
else:
classify_failure("UBUNTU22_PULL", ubuntu_pull_log)
if image_present("ubuntu:22.04"):
print("UBUNTU22_PRESENT")
else:
print("UBUNTU22_MISSING")
for label, source_image, mirror_image in mirror_pairs:
print(f"{label}_SOURCE={source_image}")
print(f"{label}_MIRROR={mirror_image}")
src_pull_log = f"/tmp/{label.lower()}-source-pull.log"
mirror_push_log = f"/tmp/{label.lower()}-mirror-push.log"
mirror_verify_log = f"/tmp/{label.lower()}-mirror-verify.log"
if is_publisher:
src_pull = !(docker pull @(source_image) > @(src_pull_log) 2>&1)
if src_pull.returncode == 0:
print(f"{label}_SOURCE_PULL:ok")
else:
classify_failure(f"{label}_SOURCE_PULL", src_pull_log)
continue
tag = !(docker tag @(source_image) @(mirror_image) > /dev/null 2>&1)
if tag.returncode != 0:
print(f"{label}_TAG:failed")
continue
push = !(docker push @(mirror_image) > @(mirror_push_log) 2>&1)
if push.returncode == 0:
print(f"{label}_MIRROR_PUSH:ok")
else:
classify_failure(f"{label}_MIRROR_PUSH", mirror_push_log)
continue
else:
print(f"{label}_SOURCE_PULL:skipped")
print(f"{label}_MIRROR_PUSH:skipped")
# Validate mirror pullability with this host's Docker registry config.
rm_image = !(docker image rm @(mirror_image) > /dev/null 2>&1)
verify = !(docker pull @(mirror_image) > @(mirror_verify_log) 2>&1)
if verify.returncode == 0:
print(f"{label}_MIRROR_PULL:ok")
else:
classify_failure(f"{label}_MIRROR_PULL", mirror_verify_log)
if image_present(mirror_image):
print(f"{label}_MIRROR_PRESENT")
else:
print(f"{label}_MIRROR_MISSING")
"""
for host in hosts:
print(f"\n=== {host} ===")
remote_code = remote_code_template.replace("__IS_PUBLISHER__", "True" if host == publisher_host else "False")
ssh @(host) @(remote_code)