Stabilize self-hosted CI workflows and resolve issue #62 (#73)

## Summary

Hardens CI workflows for self-hosted Gitea runners by stabilizing E2E execution and Renovate behavior across internal/external network paths.

Closes #62

## What Changed

### E2E workflow reliability
- Fixed E2E workspace handoff to ensure expected repository contents are present during test execution.
- Added stricter preflight checks for required frontend files before running E2E.
- Reduced mount/path fragility while preserving runtime image pull and compose flow.

### Renovate workflow hardening
- Added internal-first endpoint reachability selection with fallback handling.
- Added token preflight checks for repository access.
- Added explicit host-rule auth handling for API/git paths.
- Added container-level connectivity preflight diagnostics.
- Added git URL override aligned with selected endpoint context.
- Removed incorrect forced Dogar host-IP pinning that broke HTTPS clone routing.

## Why

CI behavior was sensitive to runner networking and Renovate clone/auth interactions. These changes make the workflow deterministic in our runner topology and address recurring CI failures.

## Scope

- Workflow logic only (`cicd.yaml`, `renovate.yml`)
- No app feature or API behavior changes

## Validation

- Workflow YAML validation passed during updates.
- Changes were applied and verified iteratively from real failing run diagnostics.

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #73

docs/CICD_MULTI_STAGE_BUILD.md

@@ -142,14 +142,10 @@ jobs:
 
 ### Responsibility Split
 
-- `.gitea/workflows/cicd-start.yaml` owns startup routing and trace propagation.
-- `.gitea/workflows/cicd-source-checks.yaml` owns early source-level
-  format/lint/type gating before any promotion dispatch.
-- `.gitea/workflows/docker-build-base.yaml` owns base publication and verification.
-- `.gitea/workflows/docker-build-main.yaml` owns complete-image publication.
-- `.gitea/workflows/cicd-checks.yaml` and `.gitea/workflows/cicd-tests.yaml`
-  own post-build CI validation and tests.
-- Main CI never rebuilds the base image locally.
+- `.gitea/workflows/cicd.yaml` owns the full CI pipeline: base image publish,
+  CICD image publish, source checks, unit tests, runtime image publication,
+  integration tests, and E2E tests.
+- The older split workflows remain only as deprecated/manual helper paths.
 
 ### Runtime Boundary Enforcement
 
@@ -173,7 +169,7 @@ and deployable runtime artifacts before publishing the complete CICD image.
 
 Workflow location:
 
-- `.gitea/workflows/docker-build-main.yaml`
+- `.gitea/workflows/cicd.yaml`
   - `Verify deployable runtime boundaries`
   - `Build and verify deployable runtime image purity`
 

docs/DEPLOYABLE_RUNTIME_CONTRACT.md

@@ -30,6 +30,7 @@ Excluded:
 ### Runtime Artifact Definition
 
 - Container build source: `Dockerfile.backend`.
+- Build shape: two-stage build with a dependency stage and a minimal runtime stage.
 - Runtime base image: `python:3.14-slim`.
 - Runtime process: `uvicorn backend.main:app --app-dir /app/src --host 0.0.0.0 --port 8000`.
 - Exposed runtime port: `8000`.
@@ -79,6 +80,7 @@ The lockfile in `backend/uv.lock` is the dependency source of truth.
 ### Frontend Runtime Artifact Definition
 
 - Container build source: `Dockerfile.frontend` (target `production`).
+- Build shape: two-stage build with a dependency/build stage and a minimal nginx runtime stage.
 - Runtime base image: `nginx:alpine`.
 - Runtime process: `nginx -g "daemon off;"`.
 - Exposed runtime port: `80`.
@@ -151,14 +153,14 @@ Current enforcement implemented in CI:
 
 - Dockerfile target boundary checks:
   - Script: `scripts/check-dockerfile-boundaries.sh`
-  - Workflow: `.gitea/workflows/docker-build-main.yaml`
+  - Workflow: `.gitea/workflows/cicd.yaml`
 - Deployable runtime image purity checks:
   - Script: `scripts/verify-deployable-image-purity.sh`
-  - Workflow: `.gitea/workflows/docker-build-main.yaml`
+  - Workflow: `.gitea/workflows/cicd.yaml`
   - Checks include binary presence and profile-specific package metadata probes
     to detect CI/development tooling leakage.
 - Runtime black-box integration checks against deployed backend container:
-  - Workflow: `.gitea/workflows/cicd-tests.yaml` (`integration-tests` job)
+  - Workflow: `.gitea/workflows/cicd.yaml` (`integration-tests` job)
   - Inputs: deployable backend commit tag reference and immutable digest
     reference from main build dispatch.
   - Assertions: digest/tag consistency and live endpoint behavior for `/`,

docs/DEVELOPMENT.md

@@ -36,7 +36,7 @@ Scope boundary:
 
 - Contract definition lives in `DEPLOYABLE_RUNTIME_CONTRACT.md`.
 - CI enforcement now includes Dockerfile boundary checks and deployable image
-  purity checks in `.gitea/workflows/docker-build-main.yaml`.
+  purity checks in `.gitea/workflows/cicd.yaml`.
 - Broader workflow redesign and deployment wiring remain out of scope for this
   repository's runtime contract document and belong to follow-up work under
   epic #66.
@@ -60,8 +60,8 @@ Automated checks:
   and fails if CI/development binaries or package metadata artifacts are
   present.
 
-These checks run in `.gitea/workflows/docker-build-main.yaml` before publishing
-the complete CICD image.
+These checks run in `.gitea/workflows/cicd.yaml` before publishing the
+complete CICD image.
 
 ## Quick Start
 

docs/GITEA_ACTIONS_TROUBLESHOOTING.md

@@ -175,10 +175,20 @@ If you want a failover strategy, keep the cached image tagged in the local regis
 Automation scripts for this workflow live in `scripts/gitea-actions/`:
 
 - `scripts/gitea-actions/repair_runner_mirror.xsh`
-  Repairs the mirror by pulling from GHCR, tagging/pushing to local registry, and verifying tag presence on each runner host.
+  Repairs mirrors by pulling upstream images, tagging/pushing to local registry, and verifying pullability on each runner host.
+  Current mirror targets:
+  - `kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest`
+  - `kankali.darkhelm.lan:3001/darkhelm.org/renovate:41`
+  - `kankali.darkhelm.lan:3001/darkhelm.org/playwright-browsers:v1.56.1-jammy`
 
 - `scripts/gitea-actions/check_runner_images.xsh`
-  Verifies host-by-host image presence for both upstream (`GHCR`) and mirrored (`MIRROR`) tags.
+  Verifies host-by-host image presence and pullability for `ubuntu:22.04`, upstream act image (`GHCR`), mirrored act image (`MIRROR`), and Renovate image.
+
+- `scripts/gitea-actions/runner-prechange-capture.xsh`
+  Captures restart counts, OOM flags, memory state, and runner logs across all hosts.
+
+- `scripts/gitea-actions/diagnose_runner_startup.xsh`
+  Focused startup diagnostics for runner containers and mirror pull behavior.
 
 Run from repository root (xonsh):