ci: consolidate cicd workflow lanes, clean up helpers, and align CI docs (#86)

## Summary
This PR simplifies the CICD workflow by merging related lanes, reducing duplicated script logic, and keeping the same overall pipeline behavior and gates. It also updates CI documentation to match the new job topology.

## What Changed

### Workflow consolidation
- Merged base and complete CICD image publication into one producer job:
  - Build and Push CICD Images
- Merged dependency audits into one informational lane:
  - Dependency Audits (Informational)
- Merged runtime image build lanes into one release producer:
  - Build Release Images
- Merged tester image build lanes into one tester producer:
  - Build Tester Images

### Dependency/gate rewiring
- Updated downstream needs to consume merged producers.
- Kept output contracts for deployable and tester image references.
- Updated production and postmortem gates to the new job IDs.

### Cleanup/simplification
- Removed duplicate helper-function definition(s) in CICD scripts.
- Replaced a manual docker login retry loop with existing retry helper usage.
- Removed redundant shell option declarations where behavior was unchanged.
- Removed one unused E2E environment variable.

### Documentation alignment
- Updated CI architecture documentation to reflect merged workflow lanes.
- Updated troubleshooting guidance to reference current job sequencing.

## Why
- Reduce job startup overhead on self-hosted runners.
- Keep behavior consistent while lowering workflow complexity.
- Improve maintainability by removing duplicated/unused script fragments.
- Keep docs in sync with operational workflow reality.

## Validation
- Workflow file checks passed with pre-commit.
- Documentation checks passed with pre-commit (including markdownlint/prettier).
- No diagnostics/errors reported for updated workflow/docs files.

## Risk and Impact
- Low-to-medium operational risk due to job-ID/needs rewiring.
- Mitigated by preserving output keys consumed by integration and e2e lanes.
- Audit lane remains informational-only (non-blocking), same intent as before.

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #86

docs/CICD_MULTI_STAGE_BUILD.md

@@ -4,6 +4,22 @@
 
 This project uses a two-stage Docker build approach to optimize CI/CD performance by separating stable base dependencies from project-specific code and dependencies.
 
+## Current Workflow Status (2026-07)
+
+The source of truth is `.gitea/workflows/cicd.yaml`.
+
+Current behavior to keep in mind:
+
+- CICD base publication and complete CICD publication now run in one merged producer job.
+- Dependency audits run in one informational lane; frontend and backend audits always execute and are non-blocking.
+- Release and tester images are built in two merged producer jobs.
+- Deployable backend/frontend runtime images are first published as staging artifacts.
+- A dedicated promotion lane publishes release image tags only after integration and E2E lanes succeed, and only for automated `push` runs on `main`.
+- If a qualifying `main` commit is untagged, CI auto-creates the next patch tag starting from a `v0.0.0` baseline.
+- Release promotion publishes release-line tags (`v<major>.<minor>.0`) plus patch/build tags (`v<major>.<minor>.<patch>`, `v<major>.<minor>.<patch>-<7-char-short-sha>`).
+- Registry operations include auth-realm host pinning and bounded retry logic for login/pull/push in critical lanes.
+- Runtime integration checks consume tag and digest outputs and verify they resolve to the same immutable artifact before assertions.
+
 ## Architecture
 
 ### Stage 1: Base Image (`Dockerfile.cicd-base`)
@@ -43,7 +59,10 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 - Pre-commit hook environments (leverages global pre-commit)
 - Project-specific tooling verification
 
-**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd:latest`
+**Registry**:
+
+- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:latest`
+- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:<head_sha>`
 
 **Rebuild Triggers**: Every CI/CD run (contains project-specific code and dependencies)
 
@@ -109,37 +128,64 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 
 ```yaml
 jobs:
-  publish-base:
-    name: Build and Publish CICD Base Image
-    steps:
-      - name: Compute base hash
-        # Uses scripts/compute-cicd-base-hash.sh
-        # Hashes Dockerfile.cicd-base and .dockerignore
+  build_cicd:
+    name: Build and Push CICD Images
+    # computes base hash, checks registry, and publishes both CICD base and complete CICD images
 
-      - name: Build and push base image
-        if: needs_build == 'true'
-        # Only runs when the immutable base tag is missing or force rebuild is requested
-        # Tags with both hash and 'latest'
+  source-checks:
+    name: Source Checks
+    needs: build_cicd
 
-      - name: Verify published base image
-        # Confirms the immutable tag is visible and pullable before success
+  dependency-audits:
+    name: Dependency Audits (Informational)
+    needs: build_cicd
+    # frontend/backend audit steps are continue-on-error and backend uses if: always()
 
-  prepare-base-ref:
-    name: Prepare CICD Base Reference
-    steps:
-      - name: Compute immutable base ref
-        # Uses the same helper as the base workflow
+  build-release-images:
+    needs: [build_cicd, source-checks]
+    # publishes deployable-backend-staging and deployable-frontend-staging refs
 
-  setup:
-    name: Build and Push CICD Complete Image
-    needs: prepare-base-ref
-    steps:
-      - name: Build and push complete CICD image
-        # Always runs, inherits from cicd-base:<hash>
-        # Waits briefly for the immutable base tag to appear, then fails clearly if missing
-        # Contains project code and dependencies
+  build-tester-images:
+    needs: [build_cicd, source-checks]
+
+  production-images-complete:
+    needs: [build-release-images, build-tester-images]
+
+  integration-tests:
+    needs:
+      [
+        build_cicd,
+        production-images-complete,
+        build-release-images,
+        build-tester-images,
+      ]
+
+  e2e-tests:
+    needs:
+      [
+        build_cicd,
+        production-images-complete,
+        build-release-images,
+        build-tester-images,
+      ]
+
+  promote-release-images:
+    needs: [build_cicd, build-release-images, integration-tests, e2e-tests]
+    # retags staging images to deployable-backend/deployable-frontend with:
+    # - release series tag (<major>.<minor>.0)
+    # - exact git semver tag from HEAD (authoritative release tag)
+    # - release build tag (<git-semver>-<7-char-short-sha>)
+    # - latest
 ```
 
+### Release Tagging Model
+
+- Promotion is release-tag driven: if HEAD has no semver git tag (`v<major>.<minor>.<patch>`), promotion is skipped.
+- Release identity is derived from the git tag on HEAD.
+- Release series tag is normalized to `<major>.<minor>.0`.
+- Each promoted release also publishes a build-distinguishing tag: `<git-semver>-<short_sha>`, where `<short_sha>` is the 7-character commit shorthand.
+- Promotion emits generated release notes summarizing commit subjects since the previous `<major>.<minor>.0` release tag.
+
 ### Responsibility Split
 
 - `.gitea/workflows/cicd.yaml` owns the full CI pipeline: base image publish,
@@ -259,6 +305,15 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
 - Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs
 - Design note: main CI intentionally fails instead of rebuilding the base locally
 
+### Registry Auth Realm Timeout
+
+- Symptom: `Client.Timeout exceeded while awaiting headers` when docker push/pull calls token endpoint
+- Cause: registry challenge realm host is not consistently reachable/resolved from runner
+- Current workflow mitigation:
+  - host mapping for registry host
+  - challenge parsing and auth-realm host pinning
+  - bounded login and push/pull retries in image lanes
+
 ### Common Issues
 
 - **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured

docs/CICD_SUCCESS_SUMMARY.md

@@ -69,6 +69,26 @@
 
 ## 🏗️ **Architecture Validation**
 
+### **Current Release Lifecycle (2026-07)**
+
+- Runtime deployable images are published first to staging repositories:
+  - `deployable-backend-staging`
+  - `deployable-frontend-staging`
+- Runtime validation (`Runtime Black-Box Integration Tests` and `End-to-End Tests`) must pass before release tagging.
+- `Promote Release Images` retags validated staging artifacts to release repositories, but only on automated `push` runs to `main`:
+  - `deployable-backend`
+  - `deployable-frontend`
+- Published release tags:
+  - `latest`
+  - `v<major>.<minor>.0`
+  - `v<major>.<minor>.<patch>`
+  - `v<major>.<minor>.<patch>-<7-char-short-sha>`
+- Version selection rule:
+  - If the qualifying `main` commit already has a semver git tag (`vX.Y.Z`), use it.
+  - Otherwise, auto-create and use the next patch tag from the latest semver tag in the repository.
+  - If no prior semver tag exists, bootstrap from `v0.0.0` and create `v0.0.1`.
+- Dependency audits are informational: frontend and backend audit steps both run and do not fail the full workflow.
+
 ### **Working Component Integration**
 
 All major components now work seamlessly together:

docs/CICD_TROUBLESHOOTING_GUIDE.md

@@ -4,6 +4,155 @@
 
 This document captures the specific optimizations, fixes, and troubleshooting approaches developed during November 2025 for the plex-playlist CI/CD pipeline. Each entry includes the problem, root cause analysis, solution implementation, and performance impact.
 
+## Current Workflow Reference (2026-07)
+
+The authoritative workflow is `.gitea/workflows/cicd.yaml`.
+
+When this guide conflicts with older examples, prefer:
+
+- current job names and dependencies in `cicd.yaml`
+- current registry endpoint `kankali.darkhelm.lan:3001`
+- current retry and auth-realm host pinning logic embedded in image build lanes
+
+## High-Value Failure Signatures (Current)
+
+### 0. Dependency audit step fails but workflow stays green
+
+**Symptom**:
+
+```text
+frontend audit reported vulnerabilities
+backend audit reported vulnerabilities
+```
+
+and overall workflow still succeeds.
+
+**Cause**: expected behavior. `Dependency Audits (Informational)` is intentionally non-blocking and runs both frontend and backend audit steps.
+
+**Fast check**:
+
+1. Confirm `Dependency Audits (Informational)` ran.
+2. Confirm both audit step logs are present.
+
+**Fix**:
+
+No CI fix required unless policy changes. Treat findings as remediation backlog items.
+
+### 1. `docker_login_with_retry: command not found`
+
+**Symptom**:
+
+```text
+line <n>: docker_login_with_retry: command not found
+```
+
+**Cause**: shell helper function referenced in a job step but missing in that same step's `run` block.
+
+**Fast check**:
+
+1. Open failing job step in `.gitea/workflows/cicd.yaml`.
+2. Confirm `docker_login_with_retry()` is defined before first call in that block.
+
+**Fix**:
+
+Add the helper definition locally in that step block (functions do not cross step boundaries).
+
+### 2. Registry token timeout while pushing/pulling
+
+**Symptom**:
+
+```text
+Client.Timeout exceeded while awaiting headers
+... /v2/token?...service=container_registry
+```
+
+**Cause**: runner resolves/pins registry host, but token realm host from `WWW-Authenticate` challenge is unresolved/unreachable.
+
+**Fast check**:
+
+1. Verify `Configure registry host resolution` step ran.
+2. Confirm auth realm host pinning logic is present in failing lane.
+3. Check lane-specific login/push retry helpers are active.
+
+**Fix**:
+
+Use `ensure_registry_auth_realm_host` + `docker_login_with_retry` + bounded `retry_registry_op` in the failing lane.
+
+### 3. Empty downstream digest/tag outputs
+
+**Symptom**:
+
+```text
+evaluated to '%!t(string=)'
+```
+
+**Cause**: upstream image lane failed before writing expected outputs (`*_tag_ref`, `*_digest_ref`).
+
+**Fast check**:
+
+1. Inspect upstream image job conclusion (`Build Frontend Main Image`, `Build Integration Tester Image`, etc.).
+2. Confirm output writes (`echo key=value >> $GITHUB_OUTPUT`) execute after push and digest resolution.
+
+**Fix**:
+
+Repair failing upstream lane first; downstream expressions become valid once outputs are emitted.
+
+### 4. Base image publication mismatch
+
+**Symptom**:
+
+```text
+Required immutable base image is not available
+```
+
+**Cause**: expected hash tag not yet published or failed publication lane.
+
+**Fix order**:
+
+1. `Build and Push CICD Images`
+2. remaining source, image, and runtime lanes
+
+### 5. Release tags missing after tests passed
+
+**Symptom**:
+
+```text
+staging images exist but deployable-backend/deployable-frontend tags not updated
+```
+
+**Cause**: promotion lane did not run or failed (`Promote Release Images`).
+
+**Fast check**:
+
+1. Confirm `Runtime Black-Box Integration Tests` and `End-to-End Tests` succeeded.
+2. Confirm the workflow run was an automated `push` on `main`; promotion is skipped for PR validation and other non-main events.
+3. Check `Promote Release Images` logs for registry login, tag creation, pull/tag/push failures.
+4. Verify staging refs (`deployable-backend-staging`, `deployable-frontend-staging`) were emitted by `Build Release Images`.
+
+**Fix**:
+
+Re-run `Promote Release Images` after correcting registry/auth issues.
+
+### 6. Unexpected release version chosen
+
+**Symptom**:
+
+```text
+release_version differs from expected manual guess
+```
+
+**Cause**: promotion versioning follows workflow rules:
+
+1. Promotion only runs for automated `push` events on `main`.
+2. If the `main` commit already has semver tag `vX.Y.Z`, use that exact patch tag.
+3. Otherwise, find the latest semver tag in the repository and auto-create the next patch tag.
+4. If no semver tag exists yet, bootstrap from `v0.0.0` and create `v0.0.1`.
+5. Also publish `vX.Y.0` and `vX.Y.Z-<7-char-short-sha>`.
+
+**Fix**:
+
+If you need an exact patch version, tag the `main` commit with semver before promotion runs; otherwise let CI assign the next patch automatically.
+
 ## Performance Optimizations
 
 ### 1. Dependency-First Build Pattern

docs/DEPLOYABLE_RUNTIME_CONTRACT.md

@@ -11,6 +11,14 @@ automation work under epic #66.
 
 ## Scope
 
+Lifecycle note:
+
+- CI first builds deployable runtime artifacts in staging repositories
+  (`deployable-backend-staging`, `deployable-frontend-staging`).
+- After integration and E2E validation pass, CI promotes those immutable
+  artifacts to release repositories (`deployable-backend`,
+  `deployable-frontend`) via tag promotion.
+
 Included:
 
 - Backend deployable runtime image requirements.

docs/DEVELOPMENT.md

@@ -17,10 +17,11 @@ This document outlines how to set up your development environment and work with
 
 - **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks
 - **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
-- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
+- **[CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md)** - Current CI lane failures and remediation paths
 - **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
 - **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules
 - **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries
+- **[ADR004: Registry Image Resolution and Auth Resilience Policy](adr/ADR004-registry-image-resolution-and-auth-resilience.md)** - Cross-workflow reliability policy for registry/auth/image fallback behavior
 
 ## Deployable Runtime Artifacts
 
@@ -209,7 +210,7 @@ git push
 1. Push your feature branch to the remote repository
 2. Navigate to the Gitea web interface
 3. Create a Pull Request from your feature branch to `main`
-4. Ensure all CI checks pass (100% green required)
+4. Ensure required CI checks pass (informational audit warnings do not block merge)
 5. Request review from team members
 6. Merge only after approval and passing CI
 
@@ -388,58 +389,68 @@ pre-commit run end-of-file-fixer --all-files
 
 ### Pipeline Overview
 
-The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance:
+The canonical CI workflow is `.gitea/workflows/cicd.yaml`.
 
-- **Stage 1**: Source-level fast checks (format, lint, type-check) - **hard promotion gate**
-- **Stage 2**: Build base image (system dependencies, Python, Node.js) - **cached across runs**
-- **Stage 3**: Build complete image (project code and dependencies) - **rebuilt every time**
+Current triggers:
 
-Pipeline triggers:
+- Push on `main` and `develop`
+- Pull requests targeting `main` and `develop`
+- Manual `workflow_dispatch`
 
-- Push to any branch
-- Pull requests to `main` or `develop`
+Current dispatch inputs:
 
-### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL**
+- `head_sha`: commit SHA to process
+- `force_rebuild_base`: force base image publication
 
-**Performance Gains**:
+### Current Job Topology
 
-- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes)
-- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs)
-- **100% success rate** achieved with optimized dependency management
-- Raspberry Pi 4GB workers handle builds efficiently with resource optimization
+The pipeline is intentionally staged so expensive image jobs run only after source checks and required gates pass:
 
-**Architecture**:
+1. Producer lane:
+   `Build and Push CICD Images` computes base hash, checks registry, and publishes both CICD base and complete CICD images.
+2. Source + audit lanes:
+   `Source Checks` and `Dependency Audits (Informational)` run after producer completion. Audit failures are intentionally non-blocking so both frontend and backend audits always report.
+3. Runtime image lanes:
+   `Build Release Images` publishes `deployable-backend-staging` and `deployable-frontend-staging`; `Build Tester Images` publishes integration/e2e tester images; then `Production Images Complete` gates downstream runtime tests.
+4. Runtime validation lanes:
+   `Runtime Black-Box Integration Tests` and `End-to-End Tests` validate staged runtime artifacts.
+5. Promotion lane:
+   `Promote Release Images` runs only for automated `push` events on `main`. It retags validated staging artifacts to release repos (`deployable-backend`, `deployable-frontend`) with `latest`, `v<major>.<minor>.0`, `v<major>.<minor>.<patch>`, and `v<major>.<minor>.<patch>-<7-char-short-sha>` tags. If the `main` commit is untagged, CI creates the next patch tag automatically; if no prior semver tag exists, the bootstrap baseline is `v0.0.0`.
+6. Postmortem lanes:
+   targeted postmortem jobs run when key lanes fail to capture diagnostics even when primary jobs fail early.
 
-- `cicd-base:latest` - System dependencies (Python 3.14, Node.js 24, build tools, pre-installed dev packages)
-- `cicd:latest` - Complete environment (project code + optimized dependency installation)
+### Reliability and Traceability Behavior
 
-**Recent Optimizations** (November 2025):
+Current workflow behavior includes:
 
-- **Dependency-first build pattern** prevents cache invalidation on code changes
-- **Yarn PnP state regeneration** ensures reliable frontend builds
-- **Network-resilient E2E testing** with simplified Docker operations
-- **Memory-optimized frontend installations** with proper swap configuration
+- registry auth realm host pinning from `WWW-Authenticate` challenge when registry tokens are issued from a different host
+- bounded retry logic for docker login/pull/push operations in image lanes
+- digest/tag contract checks for deployable image references before runtime black-box tests
+- release-note summary generation in the promotion lane (commit bullets since the previous release tag, or from the `v0.0.0` bootstrap baseline on first release)
+- context hydration for image-build lanes by copying `/workspace` from the published CICD image
+- runner split between `ubuntu-act` and `ubuntu-act-8gb` based on lane resource requirements
 
-For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
+For details of base/complete image build strategy, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
+For incident handling, see [CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md).
 
-### Pipeline Jobs
+### Operator Quick Reference
 
-All jobs run in parallel after the setup phases:
+Use workflow dispatch when you need deterministic reruns on a specific commit:
 
-1. **Source Fast Gate**:
-   Backend source checks (Ruff format/lint, Pyright), frontend source checks (Prettier, ESLint, TypeScript), and dispatches downstream build only on success.
+```bash
+# Example: rerun CI against an explicit commit
+# input head_sha=<commit>
 
-2. **Setup Base**: Builds and pushes base Docker image (conditional)
-3. **Setup Complete**: Builds and pushes complete CI/CD Docker image
-4. **Code Quality**:
-   Trailing whitespace check, end-of-file formatting, YAML syntax validation, and TOML syntax validation.
+# Example: force base image republish
+# input force_rebuild_base=true
+```
 
-5. **Backend Validation**:
-   Ruff formatting check, Ruff linting, Pyright type checking, Darglint docstring validation, unit tests with coverage, integration tests, and doctests (xdoctest).
+Recommended rerun order during flaky infrastructure incidents:
 
-6. **Frontend Validation**:
-   Prettier formatting check, ESLint linting, TypeScript compilation, and unit tests with coverage.
-   - E2E tests (Playwright)
+1. `Build and Push CICD Images`
+2. failing runtime image lane (`Build Release Images` or `Build Tester Images`)
+3. downstream integration/e2e lanes
+4. `Promote Release Images` (if staging validation succeeded but promotion failed)
 
 ### Local CI/CD Testing
 
@@ -477,7 +488,7 @@ Build and test CI/CD images locally:
 1. Navigate to your pull request in Gitea
 2. Check the "Checks" tab for detailed results
 3. Click on individual job names to see logs
-4. All jobs must pass (100% green) before merging
+4. All required jobs must pass before merging (informational dependency-audit findings are non-blocking)
 
 ## Branch Protection and Merge Requirements
 
@@ -486,7 +497,7 @@ Build and test CI/CD images locally:
 The `main` branch is protected with the following requirements:
 
 1. **No Direct Pushes**: All changes must come through pull requests
-2. **CI Must Pass**: All CI/CD jobs must be 100% green
+2. **CI Must Pass**: All required CI/CD jobs must pass (dependency audits are informational)
 3. **Review Required**: At least one team member approval needed
 4. **Up-to-date Branch**: Feature branch must be current with main
 
@@ -497,7 +508,7 @@ The `main` branch is protected with the following requirements:
 3. Address any CI failures by pushing fixes to the feature branch
 4. Request and receive code review approval
 5. Ensure branch is up-to-date with main
-6. Merge pull request (only available when all requirements met)
+6. Merge pull request (available when required jobs pass; informational audit failures do not block merge)
 
 ### If CI Fails
 

docs/RENOVATE_SETUP_GUIDE.md

@@ -4,6 +4,25 @@
 
 Renovate is an automated dependency update tool that creates pull requests to keep your project dependencies up to date. This guide covers setting up Renovate for the plex-playlist project with optimal configuration.
 
+## Repository Current Mode (2026-07)
+
+This repository runs Renovate through `.gitea/workflows/renovate.yml`.
+
+Current operational behavior:
+
+1. Uses `ubuntu-act-8gb` runner due to npm/registry memory pressure.
+2. Prepares Renovate container image with mirror-first strategy:
+
+- primary: `kankali.darkhelm.lan:3001/darkhelm.org/renovate:41`
+- fallback: `ghcr.io/renovatebot/renovate:41`
+
+3. Uses digest-aware freshness checks before deciding whether local cached image is current.
+4. Selects endpoint dynamically between internal and external candidates based on preflight reachability.
+5. Selects token via preflight checks (repo access required) with fallback order from configured secrets.
+6. Uses constrained memory settings (`RENOVATE_NODE_ARGS`) and disables OSV alerts in this runner profile.
+
+Treat workflow behavior as source of truth; use this doc as operator guidance.
+
 ## Setup Options
 
 ### Option 1: GitHub App (Recommended for GitHub)
@@ -89,7 +108,7 @@ on:
 
 jobs:
   renovate:
-    runs-on: ubuntu-act
+    runs-on: ubuntu-act-8gb
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -101,7 +120,7 @@ jobs:
           token: ${{ secrets.RENOVATE_TOKEN }}
         env:
           RENOVATE_PLATFORM: gitea
-          RENOVATE_ENDPOINT: https://dogar.darkhelm.org/api/v1
+          RENOVATE_ENDPOINT: selected at runtime from internal/external candidates
 ```
 
 ## Configuration Explanation
@@ -204,6 +223,37 @@ docker run --rm \
 4. **Large Updates**: Major version updates may need manual review
 5. **Docker Registry**: Ensure base image updates don't break builds
 
+### Current Workflow-Specific Failures
+
+1. **Renovate image pull failures**
+
+- Symptom: both mirror and GHCR candidates fail.
+- Check: runner DNS/egress and registry auth token validity.
+
+2. **Endpoint preflight failures**
+
+- Symptom: cannot reach both internal and external API endpoints.
+- Check: endpoint host mapping, TLS mode, and runner network route.
+
+3. **Token access failures**
+
+- Symptom: API `/repos/<org>/<repo>` check returns non-200.
+- Check: token scopes and secret ordering.
+
+4. **OOM or abrupt termination**
+
+- Symptom: process exits under memory pressure.
+- Check: `RENOVATE_NODE_ARGS`, PR concurrency limits, and optional feature toggles.
+
+## Operator Notes
+
+For this repository, prefer updating `.gitea/workflows/renovate.yml` over local one-off Renovate service changes. Keep docs and workflow in sync after changing:
+
+- endpoint selection logic
+- token preflight/fallback order
+- image source policy (mirror/fallback)
+- memory and concurrency guardrails
+
 ### Quick Validation
 
 For basic JSON validation without installing Renovate:

docs/adr/ADR004-registry-image-resolution-and-auth-resilience.md

@@ -0,0 +1,69 @@
+# ADR004: Registry Image Resolution and Auth Resilience Policy
+
+- Status: Accepted
+- Date: 2026-07-16
+
+## Context
+
+The CI and Renovate workflows run on self-hosted runners with intermittent DNS and network instability. Recent failures showed that image pull/push reliability depends on more than simple retries:
+
+- registry token realms may resolve to a different host than the registry endpoint
+- mirror images can become stale relative to upstream
+- downstream lanes require immutable references from upstream lanes
+- Renovate must operate across internal/external endpoint paths with token variability
+
+Without an explicit policy, each job implements ad hoc behavior and drift reintroduces flakiness.
+
+## Decision
+
+Adopt a cross-workflow reliability policy for registry/image operations:
+
+1. Prefer mirrored images first, then fallback upstream sources when mirror resolution fails.
+2. Use digest-aware freshness checks when deciding whether local image cache is current.
+3. Pin registry auth realm hosts when `WWW-Authenticate` challenge host differs from registry host.
+4. Use bounded login/pull/push retry wrappers in image publication and consumption lanes.
+5. Propagate and verify immutable digest references for downstream runtime validation lanes.
+6. Keep retry counts/timeouts configurable as operational tuning, not architectural invariants.
+
+## Scope
+
+This ADR applies to:
+
+- `.gitea/workflows/cicd.yaml`
+- `.gitea/workflows/renovate.yml`
+- helper scripts used for mirrored image resolution and lane orchestration
+
+This ADR does not prescribe exact retry constants or runner sizing thresholds.
+
+## Consequences
+
+Positive:
+
+- reduced CI flakiness from token realm host mismatch and transient registry failures
+- stronger traceability via digest-first downstream checks
+- clearer operator expectations for endpoint/token/image fallback behavior
+- consistent reliability approach across CICD and Renovate workflows
+
+Negative:
+
+- increased workflow script complexity and duplicated helper logic inside isolated step shells
+- additional maintenance burden to keep helper patterns consistent across lanes
+
+## Alternatives Considered
+
+1. Keep per-job ad hoc retries only.
+
+- Rejected due to repeated regressions and inconsistent behavior.
+
+2. Depend solely on upstream registries.
+
+- Rejected due to local network and availability constraints.
+
+3. Rebuild missing artifacts in downstream lanes.
+
+- Rejected because it breaks publish-once/consume-many behavior and weakens traceability.
+
+## Related Decisions
+
+- `ADR002-cicd_base_image_tagging.md`
+- `ADR003-deployable_runtime_image_contract.md`