Update CI and Renovate docs to match current workflows

docs/CICD_MULTI_STAGE_BUILD.md

@@ -4,6 +4,17 @@
 
 This project uses a two-stage Docker build approach to optimize CI/CD performance by separating stable base dependencies from project-specific code and dependencies.
 
+## Current Workflow Status (2026-07)
+
+The source of truth is `.gitea/workflows/cicd.yaml`.
+
+Current behavior to keep in mind:
+
+- Base image publication and CICD image publication are separate jobs in the same workflow.
+- Runtime image lanes (backend/frontend main + tester images) run only after source and promotion gates succeed.
+- Registry operations include auth-realm host pinning and bounded retry logic for login/pull/push in critical lanes.
+- Runtime integration checks consume tag and digest outputs and verify they resolve to the same immutable artifact before assertions.
+
 ## Architecture
 
 ### Stage 1: Base Image (`Dockerfile.cicd-base`)
@@ -43,7 +54,10 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 - Pre-commit hook environments (leverages global pre-commit)
 - Project-specific tooling verification
 
-**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd:latest`
+**Registry**:
+
+- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:latest`
+- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:<head_sha>`
 
 **Rebuild Triggers**: Every CI/CD run (contains project-specific code and dependencies)
 
@@ -111,33 +125,58 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 jobs:
   publish-base:
     name: Build and Publish CICD Base Image
-    steps:
-      - name: Compute base hash
-        # Uses scripts/compute-cicd-base-hash.sh
-        # Hashes Dockerfile.cicd-base and .dockerignore
+    # computes base hash, checks registry, and conditionally publishes base image
 
-      - name: Build and push base image
-        if: needs_build == 'true'
-        # Only runs when the immutable base tag is missing or force rebuild is requested
-        # Tags with both hash and 'latest'
+  build_cicd:
+    name: Build and Push CICD Image
+    needs: publish-base
+    # builds shared CICD image using immutable base hash
 
-      - name: Verify published base image
-        # Confirms the immutable tag is visible and pullable before success
+  source-checks:
+    name: Source Checks
+    needs: build_cicd
 
-  prepare-base-ref:
-    name: Prepare CICD Base Reference
-    steps:
-      - name: Compute immutable base ref
-        # Uses the same helper as the base workflow
+  build-backend-main-image:
+    needs: [build_cicd, source-checks, build-backend-base-image]
 
-  setup:
-    name: Build and Push CICD Complete Image
-    needs: prepare-base-ref
-    steps:
-      - name: Build and push complete CICD image
-        # Always runs, inherits from cicd-base:<hash>
-        # Waits briefly for the immutable base tag to appear, then fails clearly if missing
-        # Contains project code and dependencies
+  build-frontend-main-image:
+    needs: [build_cicd, source-checks, build-frontend-base-image]
+
+  build-integration-tester-image:
+    needs: [build_cicd, source-checks]
+
+  build-e2e-tester-image:
+    needs: [build_cicd, source-checks]
+
+  production-images-complete:
+    needs:
+      [
+        build-backend-main-image,
+        build-frontend-main-image,
+        build-integration-tester-image,
+        build-e2e-tester-image,
+      ]
+
+  integration-tests:
+    needs:
+      [
+        build_cicd,
+        production-images-complete,
+        build-backend-main-image,
+        build-frontend-main-image,
+        build-integration-tester-image,
+      ]
+
+  e2e-tests:
+    needs:
+      [
+        build_cicd,
+        production-images-complete,
+        build-backend-main-image,
+        build-frontend-main-image,
+        build-integration-tester-image,
+        build-e2e-tester-image,
+      ]
 ```
 
 ### Responsibility Split
@@ -259,6 +298,15 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
 - Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs
 - Design note: main CI intentionally fails instead of rebuilding the base locally
 
+### Registry Auth Realm Timeout
+
+- Symptom: `Client.Timeout exceeded while awaiting headers` when docker push/pull calls token endpoint
+- Cause: registry challenge realm host is not consistently reachable/resolved from runner
+- Current workflow mitigation:
+  - host mapping for registry host
+  - challenge parsing and auth-realm host pinning
+  - bounded login and push/pull retries in image lanes
+
 ### Common Issues
 
 - **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured

docs/CICD_TROUBLESHOOTING_GUIDE.md

@@ -4,6 +4,93 @@
 
 This document captures the specific optimizations, fixes, and troubleshooting approaches developed during November 2025 for the plex-playlist CI/CD pipeline. Each entry includes the problem, root cause analysis, solution implementation, and performance impact.
 
+## Current Workflow Reference (2026-07)
+
+The authoritative workflow is `.gitea/workflows/cicd.yaml`.
+
+When this guide conflicts with older examples, prefer:
+
+- current job names and dependencies in `cicd.yaml`
+- current registry endpoint `kankali.darkhelm.lan:3001`
+- current retry and auth-realm host pinning logic embedded in image build lanes
+
+## High-Value Failure Signatures (Current)
+
+### 1. `docker_login_with_retry: command not found`
+
+**Symptom**:
+
+```text
+line <n>: docker_login_with_retry: command not found
+```
+
+**Cause**: shell helper function referenced in a job step but missing in that same step's `run` block.
+
+**Fast check**:
+
+1. Open failing job step in `.gitea/workflows/cicd.yaml`.
+2. Confirm `docker_login_with_retry()` is defined before first call in that block.
+
+**Fix**:
+
+Add the helper definition locally in that step block (functions do not cross step boundaries).
+
+### 2. Registry token timeout while pushing/pulling
+
+**Symptom**:
+
+```text
+Client.Timeout exceeded while awaiting headers
+... /v2/token?...service=container_registry
+```
+
+**Cause**: runner resolves/pins registry host, but token realm host from `WWW-Authenticate` challenge is unresolved/unreachable.
+
+**Fast check**:
+
+1. Verify `Configure registry host resolution` step ran.
+2. Confirm auth realm host pinning logic is present in failing lane.
+3. Check lane-specific login/push retry helpers are active.
+
+**Fix**:
+
+Use `ensure_registry_auth_realm_host` + `docker_login_with_retry` + bounded `retry_registry_op` in the failing lane.
+
+### 3. Empty downstream digest/tag outputs
+
+**Symptom**:
+
+```text
+evaluated to '%!t(string=)'
+```
+
+**Cause**: upstream image lane failed before writing expected outputs (`*_tag_ref`, `*_digest_ref`).
+
+**Fast check**:
+
+1. Inspect upstream image job conclusion (`Build Frontend Main Image`, `Build Integration Tester Image`, etc.).
+2. Confirm output writes (`echo key=value >> $GITHUB_OUTPUT`) execute after push and digest resolution.
+
+**Fix**:
+
+Repair failing upstream lane first; downstream expressions become valid once outputs are emitted.
+
+### 4. Base image publication mismatch
+
+**Symptom**:
+
+```text
+Required immutable base image is not available
+```
+
+**Cause**: expected hash tag not yet published or failed publication lane.
+
+**Fix order**:
+
+1. `Build and Publish CICD Base Image`
+2. `Build and Push CICD Image`
+3. remaining image and runtime lanes
+
 ## Performance Optimizations
 
 ### 1. Dependency-First Build Pattern

docs/DEVELOPMENT.md

@@ -17,10 +17,11 @@ This document outlines how to set up your development environment and work with
 
 - **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks
 - **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
-- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
+- **[CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md)** - Current CI lane failures and remediation paths
 - **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
 - **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules
 - **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries
+- **[ADR004: Registry Image Resolution and Auth Resilience Policy](adr/ADR004-registry-image-resolution-and-auth-resilience.md)** - Cross-workflow reliability policy for registry/auth/image fallback behavior
 
 ## Deployable Runtime Artifacts
 
@@ -388,58 +389,67 @@ pre-commit run end-of-file-fixer --all-files
 
 ### Pipeline Overview
 
-The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance:
+The canonical CI workflow is `.gitea/workflows/cicd.yaml`.
 
-- **Stage 1**: Source-level fast checks (format, lint, type-check) - **hard promotion gate**
-- **Stage 2**: Build base image (system dependencies, Python, Node.js) - **cached across runs**
-- **Stage 3**: Build complete image (project code and dependencies) - **rebuilt every time**
+Current triggers:
 
-Pipeline triggers:
+- Push on `main` and `develop`
+- Pull requests targeting `main` and `develop`
+- Manual `workflow_dispatch`
 
-- Push to any branch
-- Pull requests to `main` or `develop`
+Current dispatch inputs:
 
-### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL**
+- `head_sha`: commit SHA to process
+- `force_rebuild_base`: force base image publication
 
-**Performance Gains**:
+### Current Job Topology
 
-- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes)
-- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs)
-- **100% success rate** achieved with optimized dependency management
-- Raspberry Pi 4GB workers handle builds efficiently with resource optimization
+The pipeline is intentionally staged so expensive image jobs run only after source checks and required gates pass:
 
-**Architecture**:
+1. Publish base image lane:
+   `Build and Publish CICD Base Image` computes base hash, checks registry, and conditionally builds/pushes immutable + latest tags.
+2. CICD image lane:
+   `Build and Push CICD Image` consumes base hash and publishes the shared CICD image.
+3. Source lanes:
+   `Source Checks`, `Frontend Dependency Audit`, `Backend Dependency Audit`, then `CICD Tests Complete` gate.
+4. Runtime image lanes:
+   backend base/frontend base, backend main/frontend main, integration tester, e2e tester, then `Production Images Complete` gate.
+5. Runtime validation lanes:
+   `Runtime Black-Box Integration Tests` and `End-to-End Tests`.
+6. Postmortem lanes:
+   targeted postmortem jobs run when key lanes fail to capture diagnostics even when primary jobs fail early.
 
-- `cicd-base:latest` - System dependencies (Python 3.14, Node.js 24, build tools, pre-installed dev packages)
-- `cicd:latest` - Complete environment (project code + optimized dependency installation)
+### Reliability and Traceability Behavior
 
-**Recent Optimizations** (November 2025):
+Current workflow behavior includes:
 
-- **Dependency-first build pattern** prevents cache invalidation on code changes
-- **Yarn PnP state regeneration** ensures reliable frontend builds
-- **Network-resilient E2E testing** with simplified Docker operations
-- **Memory-optimized frontend installations** with proper swap configuration
+- registry auth realm host pinning from `WWW-Authenticate` challenge when registry tokens are issued from a different host
+- bounded retry logic for docker login/pull/push operations in image lanes
+- digest/tag contract checks for deployable image references before runtime black-box tests
+- context hydration for image-build lanes by copying `/workspace` from the published CICD image
+- runner split between `ubuntu-act` and `ubuntu-act-8gb` based on lane resource requirements
 
-For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
+For details of base/complete image build strategy, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
+For incident handling, see [CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md).
 
-### Pipeline Jobs
+### Operator Quick Reference
 
-All jobs run in parallel after the setup phases:
+Use workflow dispatch when you need deterministic reruns on a specific commit:
 
-1. **Source Fast Gate**:
-   Backend source checks (Ruff format/lint, Pyright), frontend source checks (Prettier, ESLint, TypeScript), and dispatches downstream build only on success.
+```bash
+# Example: rerun CI against an explicit commit
+# input head_sha=<commit>
 
-2. **Setup Base**: Builds and pushes base Docker image (conditional)
-3. **Setup Complete**: Builds and pushes complete CI/CD Docker image
-4. **Code Quality**:
-   Trailing whitespace check, end-of-file formatting, YAML syntax validation, and TOML syntax validation.
+# Example: force base image republish
+# input force_rebuild_base=true
+```
 
-5. **Backend Validation**:
-   Ruff formatting check, Ruff linting, Pyright type checking, Darglint docstring validation, unit tests with coverage, integration tests, and doctests (xdoctest).
+Recommended rerun order during flaky infrastructure incidents:
 
-6. **Frontend Validation**:
-   Prettier formatting check, ESLint linting, TypeScript compilation, and unit tests with coverage.
-   - E2E tests (Playwright)
+1. `Build and Publish CICD Base Image`
+2. `Build and Push CICD Image`
+3. failing runtime image lane (`Build Integration Tester Image`, `Build Frontend Main Image`, etc.)
+4. downstream integration/e2e lanes
 
 ### Local CI/CD Testing
 

docs/RENOVATE_SETUP_GUIDE.md

@@ -4,6 +4,25 @@
 
 Renovate is an automated dependency update tool that creates pull requests to keep your project dependencies up to date. This guide covers setting up Renovate for the plex-playlist project with optimal configuration.
 
+## Repository Current Mode (2026-07)
+
+This repository runs Renovate through `.gitea/workflows/renovate.yml`.
+
+Current operational behavior:
+
+1. Uses `ubuntu-act-8gb` runner due to npm/registry memory pressure.
+2. Prepares Renovate container image with mirror-first strategy:
+
+- primary: `kankali.darkhelm.lan:3001/darkhelm.org/renovate:41`
+- fallback: `ghcr.io/renovatebot/renovate:41`
+
+3. Uses digest-aware freshness checks before deciding whether local cached image is current.
+4. Selects endpoint dynamically between internal and external candidates based on preflight reachability.
+5. Selects token via preflight checks (repo access required) with fallback order from configured secrets.
+6. Uses constrained memory settings (`RENOVATE_NODE_ARGS`) and disables OSV alerts in this runner profile.
+
+Treat workflow behavior as source of truth; use this doc as operator guidance.
+
 ## Setup Options
 
 ### Option 1: GitHub App (Recommended for GitHub)
@@ -89,7 +108,7 @@ on:
 
 jobs:
   renovate:
-    runs-on: ubuntu-act
+    runs-on: ubuntu-act-8gb
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -101,7 +120,7 @@ jobs:
           token: ${{ secrets.RENOVATE_TOKEN }}
         env:
           RENOVATE_PLATFORM: gitea
-          RENOVATE_ENDPOINT: https://dogar.darkhelm.org/api/v1
+          RENOVATE_ENDPOINT: selected at runtime from internal/external candidates
 ```
 
 ## Configuration Explanation
@@ -204,6 +223,37 @@ docker run --rm \
 4. **Large Updates**: Major version updates may need manual review
 5. **Docker Registry**: Ensure base image updates don't break builds
 
+### Current Workflow-Specific Failures
+
+1. **Renovate image pull failures**
+
+- Symptom: both mirror and GHCR candidates fail.
+- Check: runner DNS/egress and registry auth token validity.
+
+2. **Endpoint preflight failures**
+
+- Symptom: cannot reach both internal and external API endpoints.
+- Check: endpoint host mapping, TLS mode, and runner network route.
+
+3. **Token access failures**
+
+- Symptom: API `/repos/<org>/<repo>` check returns non-200.
+- Check: token scopes and secret ordering.
+
+4. **OOM or abrupt termination**
+
+- Symptom: process exits under memory pressure.
+- Check: `RENOVATE_NODE_ARGS`, PR concurrency limits, and optional feature toggles.
+
+## Operator Notes
+
+For this repository, prefer updating `.gitea/workflows/renovate.yml` over local one-off Renovate service changes. Keep docs and workflow in sync after changing:
+
+- endpoint selection logic
+- token preflight/fallback order
+- image source policy (mirror/fallback)
+- memory and concurrency guardrails
+
 ### Quick Validation
 
 For basic JSON validation without installing Renovate:

docs/adr/ADR004-registry-image-resolution-and-auth-resilience.md

@@ -0,0 +1,69 @@
+# ADR004: Registry Image Resolution and Auth Resilience Policy
+
+- Status: Accepted
+- Date: 2026-07-16
+
+## Context
+
+The CI and Renovate workflows run on self-hosted runners with intermittent DNS and network instability. Recent failures showed that image pull/push reliability depends on more than simple retries:
+
+- registry token realms may resolve to a different host than the registry endpoint
+- mirror images can become stale relative to upstream
+- downstream lanes require immutable references from upstream lanes
+- Renovate must operate across internal/external endpoint paths with token variability
+
+Without an explicit policy, each job implements ad hoc behavior and drift reintroduces flakiness.
+
+## Decision
+
+Adopt a cross-workflow reliability policy for registry/image operations:
+
+1. Prefer mirrored images first, then fallback upstream sources when mirror resolution fails.
+2. Use digest-aware freshness checks when deciding whether local image cache is current.
+3. Pin registry auth realm hosts when `WWW-Authenticate` challenge host differs from registry host.
+4. Use bounded login/pull/push retry wrappers in image publication and consumption lanes.
+5. Propagate and verify immutable digest references for downstream runtime validation lanes.
+6. Keep retry counts/timeouts configurable as operational tuning, not architectural invariants.
+
+## Scope
+
+This ADR applies to:
+
+- `.gitea/workflows/cicd.yaml`
+- `.gitea/workflows/renovate.yml`
+- helper scripts used for mirrored image resolution and lane orchestration
+
+This ADR does not prescribe exact retry constants or runner sizing thresholds.
+
+## Consequences
+
+Positive:
+
+- reduced CI flakiness from token realm host mismatch and transient registry failures
+- stronger traceability via digest-first downstream checks
+- clearer operator expectations for endpoint/token/image fallback behavior
+- consistent reliability approach across CICD and Renovate workflows
+
+Negative:
+
+- increased workflow script complexity and duplicated helper logic inside isolated step shells
+- additional maintenance burden to keep helper patterns consistent across lanes
+
+## Alternatives Considered
+
+1. Keep per-job ad hoc retries only.
+
+- Rejected due to repeated regressions and inconsistent behavior.
+
+2. Depend solely on upstream registries.
+
+- Rejected due to local network and availability constraints.
+
+3. Rebuild missing artifacts in downstream lanes.
+
+- Rejected because it breaks publish-once/consume-many behavior and weakens traceability.
+
+## Related Decisions
+
+- `ADR002-cicd_base_image_tagging.md`
+- `ADR003-deployable_runtime_image_contract.md`