diff --git a/docs/CICD_MULTI_STAGE_BUILD.md b/docs/CICD_MULTI_STAGE_BUILD.md index 6092969..b1f2503 100644 --- a/docs/CICD_MULTI_STAGE_BUILD.md +++ b/docs/CICD_MULTI_STAGE_BUILD.md @@ -4,6 +4,17 @@ This project uses a two-stage Docker build approach to optimize CI/CD performance by separating stable base dependencies from project-specific code and dependencies. +## Current Workflow Status (2026-07) + +The source of truth is `.gitea/workflows/cicd.yaml`. + +Current behavior to keep in mind: + +- Base image publication and CICD image publication are separate jobs in the same workflow. +- Runtime image lanes (backend/frontend main + tester images) run only after source and promotion gates succeed. +- Registry operations include auth-realm host pinning and bounded retry logic for login/pull/push in critical lanes. +- Runtime integration checks consume tag and digest outputs and verify they resolve to the same immutable artifact before assertions. + ## Architecture ### Stage 1: Base Image (`Dockerfile.cicd-base`) @@ -43,7 +54,10 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc - Pre-commit hook environments (leverages global pre-commit) - Project-specific tooling verification -**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd:latest` +**Registry**: + +- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:latest` +- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:` **Rebuild Triggers**: Every CI/CD run (contains project-specific code and dependencies) @@ -111,33 +125,58 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc jobs: publish-base: name: Build and Publish CICD Base Image - steps: - - name: Compute base hash - # Uses scripts/compute-cicd-base-hash.sh - # Hashes Dockerfile.cicd-base and .dockerignore + # computes base hash, checks registry, and conditionally publishes base image - - name: Build and push base image - if: needs_build == 'true' - # Only runs when the immutable base tag is missing or force rebuild is requested - # Tags with both hash and 'latest' + build_cicd: + name: Build and Push CICD Image + needs: publish-base + # builds shared CICD image using immutable base hash - - name: Verify published base image - # Confirms the immutable tag is visible and pullable before success + source-checks: + name: Source Checks + needs: build_cicd - prepare-base-ref: - name: Prepare CICD Base Reference - steps: - - name: Compute immutable base ref - # Uses the same helper as the base workflow + build-backend-main-image: + needs: [build_cicd, source-checks, build-backend-base-image] - setup: - name: Build and Push CICD Complete Image - needs: prepare-base-ref - steps: - - name: Build and push complete CICD image - # Always runs, inherits from cicd-base: - # Waits briefly for the immutable base tag to appear, then fails clearly if missing - # Contains project code and dependencies + build-frontend-main-image: + needs: [build_cicd, source-checks, build-frontend-base-image] + + build-integration-tester-image: + needs: [build_cicd, source-checks] + + build-e2e-tester-image: + needs: [build_cicd, source-checks] + + production-images-complete: + needs: + [ + build-backend-main-image, + build-frontend-main-image, + build-integration-tester-image, + build-e2e-tester-image, + ] + + integration-tests: + needs: + [ + build_cicd, + production-images-complete, + build-backend-main-image, + build-frontend-main-image, + build-integration-tester-image, + ] + + e2e-tests: + needs: + [ + build_cicd, + production-images-complete, + build-backend-main-image, + build-frontend-main-image, + build-integration-tester-image, + build-e2e-tester-image, + ] ``` ### Responsibility Split @@ -259,6 +298,15 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \ - Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs - Design note: main CI intentionally fails instead of rebuilding the base locally +### Registry Auth Realm Timeout + +- Symptom: `Client.Timeout exceeded while awaiting headers` when docker push/pull calls token endpoint +- Cause: registry challenge realm host is not consistently reachable/resolved from runner +- Current workflow mitigation: + - host mapping for registry host + - challenge parsing and auth-realm host pinning + - bounded login and push/pull retries in image lanes + ### Common Issues - **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured diff --git a/docs/CICD_TROUBLESHOOTING_GUIDE.md b/docs/CICD_TROUBLESHOOTING_GUIDE.md index b78a796..9a60e4d 100644 --- a/docs/CICD_TROUBLESHOOTING_GUIDE.md +++ b/docs/CICD_TROUBLESHOOTING_GUIDE.md @@ -4,6 +4,93 @@ This document captures the specific optimizations, fixes, and troubleshooting approaches developed during November 2025 for the plex-playlist CI/CD pipeline. Each entry includes the problem, root cause analysis, solution implementation, and performance impact. +## Current Workflow Reference (2026-07) + +The authoritative workflow is `.gitea/workflows/cicd.yaml`. + +When this guide conflicts with older examples, prefer: + +- current job names and dependencies in `cicd.yaml` +- current registry endpoint `kankali.darkhelm.lan:3001` +- current retry and auth-realm host pinning logic embedded in image build lanes + +## High-Value Failure Signatures (Current) + +### 1. `docker_login_with_retry: command not found` + +**Symptom**: + +```text +line : docker_login_with_retry: command not found +``` + +**Cause**: shell helper function referenced in a job step but missing in that same step's `run` block. + +**Fast check**: + +1. Open failing job step in `.gitea/workflows/cicd.yaml`. +2. Confirm `docker_login_with_retry()` is defined before first call in that block. + +**Fix**: + +Add the helper definition locally in that step block (functions do not cross step boundaries). + +### 2. Registry token timeout while pushing/pulling + +**Symptom**: + +```text +Client.Timeout exceeded while awaiting headers +... /v2/token?...service=container_registry +``` + +**Cause**: runner resolves/pins registry host, but token realm host from `WWW-Authenticate` challenge is unresolved/unreachable. + +**Fast check**: + +1. Verify `Configure registry host resolution` step ran. +2. Confirm auth realm host pinning logic is present in failing lane. +3. Check lane-specific login/push retry helpers are active. + +**Fix**: + +Use `ensure_registry_auth_realm_host` + `docker_login_with_retry` + bounded `retry_registry_op` in the failing lane. + +### 3. Empty downstream digest/tag outputs + +**Symptom**: + +```text +evaluated to '%!t(string=)' +``` + +**Cause**: upstream image lane failed before writing expected outputs (`*_tag_ref`, `*_digest_ref`). + +**Fast check**: + +1. Inspect upstream image job conclusion (`Build Frontend Main Image`, `Build Integration Tester Image`, etc.). +2. Confirm output writes (`echo key=value >> $GITHUB_OUTPUT`) execute after push and digest resolution. + +**Fix**: + +Repair failing upstream lane first; downstream expressions become valid once outputs are emitted. + +### 4. Base image publication mismatch + +**Symptom**: + +```text +Required immutable base image is not available +``` + +**Cause**: expected hash tag not yet published or failed publication lane. + +**Fix order**: + +1. `Build and Publish CICD Base Image` +2. `Build and Push CICD Image` +3. remaining image and runtime lanes + ## Performance Optimizations ### 1. Dependency-First Build Pattern diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 82a4f75..5427940 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -17,10 +17,11 @@ This document outlines how to set up your development environment and work with - **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks - **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system -- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions +- **[CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md)** - Current CI lane failures and remediation paths - **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices - **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules - **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries +- **[ADR004: Registry Image Resolution and Auth Resilience Policy](adr/ADR004-registry-image-resolution-and-auth-resilience.md)** - Cross-workflow reliability policy for registry/auth/image fallback behavior ## Deployable Runtime Artifacts @@ -388,58 +389,67 @@ pre-commit run end-of-file-fixer --all-files ### Pipeline Overview -The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance: +The canonical CI workflow is `.gitea/workflows/cicd.yaml`. -- **Stage 1**: Source-level fast checks (format, lint, type-check) - **hard promotion gate** -- **Stage 2**: Build base image (system dependencies, Python, Node.js) - **cached across runs** -- **Stage 3**: Build complete image (project code and dependencies) - **rebuilt every time** +Current triggers: -Pipeline triggers: +- Push on `main` and `develop` +- Pull requests targeting `main` and `develop` +- Manual `workflow_dispatch` -- Push to any branch -- Pull requests to `main` or `develop` +Current dispatch inputs: -### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL** +- `head_sha`: commit SHA to process +- `force_rebuild_base`: force base image publication -**Performance Gains**: +### Current Job Topology -- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes) -- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs) -- **100% success rate** achieved with optimized dependency management -- Raspberry Pi 4GB workers handle builds efficiently with resource optimization +The pipeline is intentionally staged so expensive image jobs run only after source checks and required gates pass: -**Architecture**: +1. Publish base image lane: + `Build and Publish CICD Base Image` computes base hash, checks registry, and conditionally builds/pushes immutable + latest tags. +2. CICD image lane: + `Build and Push CICD Image` consumes base hash and publishes the shared CICD image. +3. Source lanes: + `Source Checks`, `Frontend Dependency Audit`, `Backend Dependency Audit`, then `CICD Tests Complete` gate. +4. Runtime image lanes: + backend base/frontend base, backend main/frontend main, integration tester, e2e tester, then `Production Images Complete` gate. +5. Runtime validation lanes: + `Runtime Black-Box Integration Tests` and `End-to-End Tests`. +6. Postmortem lanes: + targeted postmortem jobs run when key lanes fail to capture diagnostics even when primary jobs fail early. -- `cicd-base:latest` - System dependencies (Python 3.14, Node.js 24, build tools, pre-installed dev packages) -- `cicd:latest` - Complete environment (project code + optimized dependency installation) +### Reliability and Traceability Behavior -**Recent Optimizations** (November 2025): +Current workflow behavior includes: -- **Dependency-first build pattern** prevents cache invalidation on code changes -- **Yarn PnP state regeneration** ensures reliable frontend builds -- **Network-resilient E2E testing** with simplified Docker operations -- **Memory-optimized frontend installations** with proper swap configuration +- registry auth realm host pinning from `WWW-Authenticate` challenge when registry tokens are issued from a different host +- bounded retry logic for docker login/pull/push operations in image lanes +- digest/tag contract checks for deployable image references before runtime black-box tests +- context hydration for image-build lanes by copying `/workspace` from the published CICD image +- runner split between `ubuntu-act` and `ubuntu-act-8gb` based on lane resource requirements -For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md). +For details of base/complete image build strategy, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md). +For incident handling, see [CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md). -### Pipeline Jobs +### Operator Quick Reference -All jobs run in parallel after the setup phases: +Use workflow dispatch when you need deterministic reruns on a specific commit: -1. **Source Fast Gate**: - Backend source checks (Ruff format/lint, Pyright), frontend source checks (Prettier, ESLint, TypeScript), and dispatches downstream build only on success. +```bash +# Example: rerun CI against an explicit commit +# input head_sha= -2. **Setup Base**: Builds and pushes base Docker image (conditional) -3. **Setup Complete**: Builds and pushes complete CI/CD Docker image -4. **Code Quality**: - Trailing whitespace check, end-of-file formatting, YAML syntax validation, and TOML syntax validation. +# Example: force base image republish +# input force_rebuild_base=true +``` -5. **Backend Validation**: - Ruff formatting check, Ruff linting, Pyright type checking, Darglint docstring validation, unit tests with coverage, integration tests, and doctests (xdoctest). +Recommended rerun order during flaky infrastructure incidents: -6. **Frontend Validation**: - Prettier formatting check, ESLint linting, TypeScript compilation, and unit tests with coverage. - - E2E tests (Playwright) +1. `Build and Publish CICD Base Image` +2. `Build and Push CICD Image` +3. failing runtime image lane (`Build Integration Tester Image`, `Build Frontend Main Image`, etc.) +4. downstream integration/e2e lanes ### Local CI/CD Testing diff --git a/docs/RENOVATE_SETUP_GUIDE.md b/docs/RENOVATE_SETUP_GUIDE.md index 6915651..ef1b57b 100644 --- a/docs/RENOVATE_SETUP_GUIDE.md +++ b/docs/RENOVATE_SETUP_GUIDE.md @@ -4,6 +4,25 @@ Renovate is an automated dependency update tool that creates pull requests to keep your project dependencies up to date. This guide covers setting up Renovate for the plex-playlist project with optimal configuration. +## Repository Current Mode (2026-07) + +This repository runs Renovate through `.gitea/workflows/renovate.yml`. + +Current operational behavior: + +1. Uses `ubuntu-act-8gb` runner due to npm/registry memory pressure. +2. Prepares Renovate container image with mirror-first strategy: + +- primary: `kankali.darkhelm.lan:3001/darkhelm.org/renovate:41` +- fallback: `ghcr.io/renovatebot/renovate:41` + +3. Uses digest-aware freshness checks before deciding whether local cached image is current. +4. Selects endpoint dynamically between internal and external candidates based on preflight reachability. +5. Selects token via preflight checks (repo access required) with fallback order from configured secrets. +6. Uses constrained memory settings (`RENOVATE_NODE_ARGS`) and disables OSV alerts in this runner profile. + +Treat workflow behavior as source of truth; use this doc as operator guidance. + ## Setup Options ### Option 1: GitHub App (Recommended for GitHub) @@ -89,7 +108,7 @@ on: jobs: renovate: - runs-on: ubuntu-act + runs-on: ubuntu-act-8gb steps: - name: Checkout uses: actions/checkout@v4 @@ -101,7 +120,7 @@ jobs: token: ${{ secrets.RENOVATE_TOKEN }} env: RENOVATE_PLATFORM: gitea - RENOVATE_ENDPOINT: https://dogar.darkhelm.org/api/v1 + RENOVATE_ENDPOINT: selected at runtime from internal/external candidates ``` ## Configuration Explanation @@ -204,6 +223,37 @@ docker run --rm \ 4. **Large Updates**: Major version updates may need manual review 5. **Docker Registry**: Ensure base image updates don't break builds +### Current Workflow-Specific Failures + +1. **Renovate image pull failures** + +- Symptom: both mirror and GHCR candidates fail. +- Check: runner DNS/egress and registry auth token validity. + +2. **Endpoint preflight failures** + +- Symptom: cannot reach both internal and external API endpoints. +- Check: endpoint host mapping, TLS mode, and runner network route. + +3. **Token access failures** + +- Symptom: API `/repos//` check returns non-200. +- Check: token scopes and secret ordering. + +4. **OOM or abrupt termination** + +- Symptom: process exits under memory pressure. +- Check: `RENOVATE_NODE_ARGS`, PR concurrency limits, and optional feature toggles. + +## Operator Notes + +For this repository, prefer updating `.gitea/workflows/renovate.yml` over local one-off Renovate service changes. Keep docs and workflow in sync after changing: + +- endpoint selection logic +- token preflight/fallback order +- image source policy (mirror/fallback) +- memory and concurrency guardrails + ### Quick Validation For basic JSON validation without installing Renovate: diff --git a/docs/adr/ADR004-registry-image-resolution-and-auth-resilience.md b/docs/adr/ADR004-registry-image-resolution-and-auth-resilience.md new file mode 100644 index 0000000..399d4e9 --- /dev/null +++ b/docs/adr/ADR004-registry-image-resolution-and-auth-resilience.md @@ -0,0 +1,69 @@ +# ADR004: Registry Image Resolution and Auth Resilience Policy + +- Status: Accepted +- Date: 2026-07-16 + +## Context + +The CI and Renovate workflows run on self-hosted runners with intermittent DNS and network instability. Recent failures showed that image pull/push reliability depends on more than simple retries: + +- registry token realms may resolve to a different host than the registry endpoint +- mirror images can become stale relative to upstream +- downstream lanes require immutable references from upstream lanes +- Renovate must operate across internal/external endpoint paths with token variability + +Without an explicit policy, each job implements ad hoc behavior and drift reintroduces flakiness. + +## Decision + +Adopt a cross-workflow reliability policy for registry/image operations: + +1. Prefer mirrored images first, then fallback upstream sources when mirror resolution fails. +2. Use digest-aware freshness checks when deciding whether local image cache is current. +3. Pin registry auth realm hosts when `WWW-Authenticate` challenge host differs from registry host. +4. Use bounded login/pull/push retry wrappers in image publication and consumption lanes. +5. Propagate and verify immutable digest references for downstream runtime validation lanes. +6. Keep retry counts/timeouts configurable as operational tuning, not architectural invariants. + +## Scope + +This ADR applies to: + +- `.gitea/workflows/cicd.yaml` +- `.gitea/workflows/renovate.yml` +- helper scripts used for mirrored image resolution and lane orchestration + +This ADR does not prescribe exact retry constants or runner sizing thresholds. + +## Consequences + +Positive: + +- reduced CI flakiness from token realm host mismatch and transient registry failures +- stronger traceability via digest-first downstream checks +- clearer operator expectations for endpoint/token/image fallback behavior +- consistent reliability approach across CICD and Renovate workflows + +Negative: + +- increased workflow script complexity and duplicated helper logic inside isolated step shells +- additional maintenance burden to keep helper patterns consistent across lanes + +## Alternatives Considered + +1. Keep per-job ad hoc retries only. + +- Rejected due to repeated regressions and inconsistent behavior. + +2. Depend solely on upstream registries. + +- Rejected due to local network and availability constraints. + +3. Rebuild missing artifacts in downstream lanes. + +- Rejected because it breaks publish-once/consume-many behavior and weakens traceability. + +## Related Decisions + +- `ADR002-cicd_base_image_tagging.md` +- `ADR003-deployable_runtime_image_contract.md`