ci: consolidate cicd workflow lanes, clean up helpers, and align CI docs #86
File diff suppressed because it is too large
Load Diff
@@ -4,6 +4,22 @@
|
||||
|
||||
This project uses a two-stage Docker build approach to optimize CI/CD performance by separating stable base dependencies from project-specific code and dependencies.
|
||||
|
||||
## Current Workflow Status (2026-07)
|
||||
|
||||
The source of truth is `.gitea/workflows/cicd.yaml`.
|
||||
|
||||
Current behavior to keep in mind:
|
||||
|
||||
- CICD base publication and complete CICD publication now run in one merged producer job.
|
||||
- Dependency audits run in one informational lane; frontend and backend audits always execute and are non-blocking.
|
||||
- Release and tester images are built in two merged producer jobs.
|
||||
- Deployable backend/frontend runtime images are first published as staging artifacts.
|
||||
- A dedicated promotion lane publishes release image tags only after integration and E2E lanes succeed, and only for automated `push` runs on `main`.
|
||||
- If a qualifying `main` commit is untagged, CI auto-creates the next patch tag starting from a `v0.0.0` baseline.
|
||||
- Release promotion publishes release-line tags (`v<major>.<minor>.0`) plus patch/build tags (`v<major>.<minor>.<patch>`, `v<major>.<minor>.<patch>-<7-char-short-sha>`).
|
||||
- Registry operations include auth-realm host pinning and bounded retry logic for login/pull/push in critical lanes.
|
||||
- Runtime integration checks consume tag and digest outputs and verify they resolve to the same immutable artifact before assertions.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Stage 1: Base Image (`Dockerfile.cicd-base`)
|
||||
@@ -43,7 +59,10 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
- Pre-commit hook environments (leverages global pre-commit)
|
||||
- Project-specific tooling verification
|
||||
|
||||
**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd:latest`
|
||||
**Registry**:
|
||||
|
||||
- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:latest`
|
||||
- `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd:<head_sha>`
|
||||
|
||||
**Rebuild Triggers**: Every CI/CD run (contains project-specific code and dependencies)
|
||||
|
||||
@@ -109,37 +128,64 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
publish-base:
|
||||
name: Build and Publish CICD Base Image
|
||||
steps:
|
||||
- name: Compute base hash
|
||||
# Uses scripts/compute-cicd-base-hash.sh
|
||||
# Hashes Dockerfile.cicd-base and .dockerignore
|
||||
build_cicd:
|
||||
name: Build and Push CICD Images
|
||||
# computes base hash, checks registry, and publishes both CICD base and complete CICD images
|
||||
|
||||
- name: Build and push base image
|
||||
if: needs_build == 'true'
|
||||
# Only runs when the immutable base tag is missing or force rebuild is requested
|
||||
# Tags with both hash and 'latest'
|
||||
source-checks:
|
||||
name: Source Checks
|
||||
needs: build_cicd
|
||||
|
||||
- name: Verify published base image
|
||||
# Confirms the immutable tag is visible and pullable before success
|
||||
dependency-audits:
|
||||
name: Dependency Audits (Informational)
|
||||
needs: build_cicd
|
||||
# frontend/backend audit steps are continue-on-error and backend uses if: always()
|
||||
|
||||
prepare-base-ref:
|
||||
name: Prepare CICD Base Reference
|
||||
steps:
|
||||
- name: Compute immutable base ref
|
||||
# Uses the same helper as the base workflow
|
||||
build-release-images:
|
||||
needs: [build_cicd, source-checks]
|
||||
# publishes deployable-backend-staging and deployable-frontend-staging refs
|
||||
|
||||
setup:
|
||||
name: Build and Push CICD Complete Image
|
||||
needs: prepare-base-ref
|
||||
steps:
|
||||
- name: Build and push complete CICD image
|
||||
# Always runs, inherits from cicd-base:<hash>
|
||||
# Waits briefly for the immutable base tag to appear, then fails clearly if missing
|
||||
# Contains project code and dependencies
|
||||
build-tester-images:
|
||||
needs: [build_cicd, source-checks]
|
||||
|
||||
production-images-complete:
|
||||
needs: [build-release-images, build-tester-images]
|
||||
|
||||
integration-tests:
|
||||
needs:
|
||||
[
|
||||
build_cicd,
|
||||
production-images-complete,
|
||||
build-release-images,
|
||||
build-tester-images,
|
||||
]
|
||||
|
||||
e2e-tests:
|
||||
needs:
|
||||
[
|
||||
build_cicd,
|
||||
production-images-complete,
|
||||
build-release-images,
|
||||
build-tester-images,
|
||||
]
|
||||
|
||||
promote-release-images:
|
||||
needs: [build_cicd, build-release-images, integration-tests, e2e-tests]
|
||||
# retags staging images to deployable-backend/deployable-frontend with:
|
||||
# - release series tag (<major>.<minor>.0)
|
||||
# - exact git semver tag from HEAD (authoritative release tag)
|
||||
# - release build tag (<git-semver>-<7-char-short-sha>)
|
||||
# - latest
|
||||
```
|
||||
|
||||
### Release Tagging Model
|
||||
|
||||
- Promotion is release-tag driven: if HEAD has no semver git tag (`v<major>.<minor>.<patch>`), promotion is skipped.
|
||||
- Release identity is derived from the git tag on HEAD.
|
||||
- Release series tag is normalized to `<major>.<minor>.0`.
|
||||
- Each promoted release also publishes a build-distinguishing tag: `<git-semver>-<short_sha>`, where `<short_sha>` is the 7-character commit shorthand.
|
||||
- Promotion emits generated release notes summarizing commit subjects since the previous `<major>.<minor>.0` release tag.
|
||||
|
||||
### Responsibility Split
|
||||
|
||||
- `.gitea/workflows/cicd.yaml` owns the full CI pipeline: base image publish,
|
||||
@@ -259,6 +305,15 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
- Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs
|
||||
- Design note: main CI intentionally fails instead of rebuilding the base locally
|
||||
|
||||
### Registry Auth Realm Timeout
|
||||
|
||||
- Symptom: `Client.Timeout exceeded while awaiting headers` when docker push/pull calls token endpoint
|
||||
- Cause: registry challenge realm host is not consistently reachable/resolved from runner
|
||||
- Current workflow mitigation:
|
||||
- host mapping for registry host
|
||||
- challenge parsing and auth-realm host pinning
|
||||
- bounded login and push/pull retries in image lanes
|
||||
|
||||
### Common Issues
|
||||
|
||||
- **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured
|
||||
|
||||
@@ -69,6 +69,26 @@
|
||||
|
||||
## 🏗️ **Architecture Validation**
|
||||
|
||||
### **Current Release Lifecycle (2026-07)**
|
||||
|
||||
- Runtime deployable images are published first to staging repositories:
|
||||
- `deployable-backend-staging`
|
||||
- `deployable-frontend-staging`
|
||||
- Runtime validation (`Runtime Black-Box Integration Tests` and `End-to-End Tests`) must pass before release tagging.
|
||||
- `Promote Release Images` retags validated staging artifacts to release repositories, but only on automated `push` runs to `main`:
|
||||
- `deployable-backend`
|
||||
- `deployable-frontend`
|
||||
- Published release tags:
|
||||
- `latest`
|
||||
- `v<major>.<minor>.0`
|
||||
- `v<major>.<minor>.<patch>`
|
||||
- `v<major>.<minor>.<patch>-<7-char-short-sha>`
|
||||
- Version selection rule:
|
||||
- If the qualifying `main` commit already has a semver git tag (`vX.Y.Z`), use it.
|
||||
- Otherwise, auto-create and use the next patch tag from the latest semver tag in the repository.
|
||||
- If no prior semver tag exists, bootstrap from `v0.0.0` and create `v0.0.1`.
|
||||
- Dependency audits are informational: frontend and backend audit steps both run and do not fail the full workflow.
|
||||
|
||||
### **Working Component Integration**
|
||||
|
||||
All major components now work seamlessly together:
|
||||
|
||||
@@ -4,6 +4,155 @@
|
||||
|
||||
This document captures the specific optimizations, fixes, and troubleshooting approaches developed during November 2025 for the plex-playlist CI/CD pipeline. Each entry includes the problem, root cause analysis, solution implementation, and performance impact.
|
||||
|
||||
## Current Workflow Reference (2026-07)
|
||||
|
||||
The authoritative workflow is `.gitea/workflows/cicd.yaml`.
|
||||
|
||||
When this guide conflicts with older examples, prefer:
|
||||
|
||||
- current job names and dependencies in `cicd.yaml`
|
||||
- current registry endpoint `kankali.darkhelm.lan:3001`
|
||||
- current retry and auth-realm host pinning logic embedded in image build lanes
|
||||
|
||||
## High-Value Failure Signatures (Current)
|
||||
|
||||
### 0. Dependency audit step fails but workflow stays green
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
frontend audit reported vulnerabilities
|
||||
backend audit reported vulnerabilities
|
||||
```
|
||||
|
||||
and overall workflow still succeeds.
|
||||
|
||||
**Cause**: expected behavior. `Dependency Audits (Informational)` is intentionally non-blocking and runs both frontend and backend audit steps.
|
||||
|
||||
**Fast check**:
|
||||
|
||||
1. Confirm `Dependency Audits (Informational)` ran.
|
||||
2. Confirm both audit step logs are present.
|
||||
|
||||
**Fix**:
|
||||
|
||||
No CI fix required unless policy changes. Treat findings as remediation backlog items.
|
||||
|
||||
### 1. `docker_login_with_retry: command not found`
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
line <n>: docker_login_with_retry: command not found
|
||||
```
|
||||
|
||||
**Cause**: shell helper function referenced in a job step but missing in that same step's `run` block.
|
||||
|
||||
**Fast check**:
|
||||
|
||||
1. Open failing job step in `.gitea/workflows/cicd.yaml`.
|
||||
2. Confirm `docker_login_with_retry()` is defined before first call in that block.
|
||||
|
||||
**Fix**:
|
||||
|
||||
Add the helper definition locally in that step block (functions do not cross step boundaries).
|
||||
|
||||
### 2. Registry token timeout while pushing/pulling
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
Client.Timeout exceeded while awaiting headers
|
||||
... /v2/token?...service=container_registry
|
||||
```
|
||||
|
||||
**Cause**: runner resolves/pins registry host, but token realm host from `WWW-Authenticate` challenge is unresolved/unreachable.
|
||||
|
||||
**Fast check**:
|
||||
|
||||
1. Verify `Configure registry host resolution` step ran.
|
||||
2. Confirm auth realm host pinning logic is present in failing lane.
|
||||
3. Check lane-specific login/push retry helpers are active.
|
||||
|
||||
**Fix**:
|
||||
|
||||
Use `ensure_registry_auth_realm_host` + `docker_login_with_retry` + bounded `retry_registry_op` in the failing lane.
|
||||
|
||||
### 3. Empty downstream digest/tag outputs
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
evaluated to '%!t(string=)'
|
||||
```
|
||||
|
||||
**Cause**: upstream image lane failed before writing expected outputs (`*_tag_ref`, `*_digest_ref`).
|
||||
|
||||
**Fast check**:
|
||||
|
||||
1. Inspect upstream image job conclusion (`Build Frontend Main Image`, `Build Integration Tester Image`, etc.).
|
||||
2. Confirm output writes (`echo key=value >> $GITHUB_OUTPUT`) execute after push and digest resolution.
|
||||
|
||||
**Fix**:
|
||||
|
||||
Repair failing upstream lane first; downstream expressions become valid once outputs are emitted.
|
||||
|
||||
### 4. Base image publication mismatch
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
Required immutable base image is not available
|
||||
```
|
||||
|
||||
**Cause**: expected hash tag not yet published or failed publication lane.
|
||||
|
||||
**Fix order**:
|
||||
|
||||
1. `Build and Push CICD Images`
|
||||
2. remaining source, image, and runtime lanes
|
||||
|
||||
### 5. Release tags missing after tests passed
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
staging images exist but deployable-backend/deployable-frontend tags not updated
|
||||
```
|
||||
|
||||
**Cause**: promotion lane did not run or failed (`Promote Release Images`).
|
||||
|
||||
**Fast check**:
|
||||
|
||||
1. Confirm `Runtime Black-Box Integration Tests` and `End-to-End Tests` succeeded.
|
||||
2. Confirm the workflow run was an automated `push` on `main`; promotion is skipped for PR validation and other non-main events.
|
||||
3. Check `Promote Release Images` logs for registry login, tag creation, pull/tag/push failures.
|
||||
4. Verify staging refs (`deployable-backend-staging`, `deployable-frontend-staging`) were emitted by `Build Release Images`.
|
||||
|
||||
**Fix**:
|
||||
|
||||
Re-run `Promote Release Images` after correcting registry/auth issues.
|
||||
|
||||
### 6. Unexpected release version chosen
|
||||
|
||||
**Symptom**:
|
||||
|
||||
```text
|
||||
release_version differs from expected manual guess
|
||||
```
|
||||
|
||||
**Cause**: promotion versioning follows workflow rules:
|
||||
|
||||
1. Promotion only runs for automated `push` events on `main`.
|
||||
2. If the `main` commit already has semver tag `vX.Y.Z`, use that exact patch tag.
|
||||
3. Otherwise, find the latest semver tag in the repository and auto-create the next patch tag.
|
||||
4. If no semver tag exists yet, bootstrap from `v0.0.0` and create `v0.0.1`.
|
||||
5. Also publish `vX.Y.0` and `vX.Y.Z-<7-char-short-sha>`.
|
||||
|
||||
**Fix**:
|
||||
|
||||
If you need an exact patch version, tag the `main` commit with semver before promotion runs; otherwise let CI assign the next patch automatically.
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
### 1. Dependency-First Build Pattern
|
||||
|
||||
@@ -11,6 +11,14 @@ automation work under epic #66.
|
||||
|
||||
## Scope
|
||||
|
||||
Lifecycle note:
|
||||
|
||||
- CI first builds deployable runtime artifacts in staging repositories
|
||||
(`deployable-backend-staging`, `deployable-frontend-staging`).
|
||||
- After integration and E2E validation pass, CI promotes those immutable
|
||||
artifacts to release repositories (`deployable-backend`,
|
||||
`deployable-frontend`) via tag promotion.
|
||||
|
||||
Included:
|
||||
|
||||
- Backend deployable runtime image requirements.
|
||||
|
||||
@@ -17,10 +17,11 @@ This document outlines how to set up your development environment and work with
|
||||
|
||||
- **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks
|
||||
- **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
|
||||
- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
|
||||
- **[CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md)** - Current CI lane failures and remediation paths
|
||||
- **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
|
||||
- **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules
|
||||
- **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries
|
||||
- **[ADR004: Registry Image Resolution and Auth Resilience Policy](adr/ADR004-registry-image-resolution-and-auth-resilience.md)** - Cross-workflow reliability policy for registry/auth/image fallback behavior
|
||||
|
||||
## Deployable Runtime Artifacts
|
||||
|
||||
@@ -209,7 +210,7 @@ git push
|
||||
1. Push your feature branch to the remote repository
|
||||
2. Navigate to the Gitea web interface
|
||||
3. Create a Pull Request from your feature branch to `main`
|
||||
4. Ensure all CI checks pass (100% green required)
|
||||
4. Ensure required CI checks pass (informational audit warnings do not block merge)
|
||||
5. Request review from team members
|
||||
6. Merge only after approval and passing CI
|
||||
|
||||
@@ -388,58 +389,68 @@ pre-commit run end-of-file-fixer --all-files
|
||||
|
||||
### Pipeline Overview
|
||||
|
||||
The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance:
|
||||
The canonical CI workflow is `.gitea/workflows/cicd.yaml`.
|
||||
|
||||
- **Stage 1**: Source-level fast checks (format, lint, type-check) - **hard promotion gate**
|
||||
- **Stage 2**: Build base image (system dependencies, Python, Node.js) - **cached across runs**
|
||||
- **Stage 3**: Build complete image (project code and dependencies) - **rebuilt every time**
|
||||
Current triggers:
|
||||
|
||||
Pipeline triggers:
|
||||
- Push on `main` and `develop`
|
||||
- Pull requests targeting `main` and `develop`
|
||||
- Manual `workflow_dispatch`
|
||||
|
||||
- Push to any branch
|
||||
- Pull requests to `main` or `develop`
|
||||
Current dispatch inputs:
|
||||
|
||||
### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL**
|
||||
- `head_sha`: commit SHA to process
|
||||
- `force_rebuild_base`: force base image publication
|
||||
|
||||
**Performance Gains**:
|
||||
### Current Job Topology
|
||||
|
||||
- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes)
|
||||
- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs)
|
||||
- **100% success rate** achieved with optimized dependency management
|
||||
- Raspberry Pi 4GB workers handle builds efficiently with resource optimization
|
||||
The pipeline is intentionally staged so expensive image jobs run only after source checks and required gates pass:
|
||||
|
||||
**Architecture**:
|
||||
1. Producer lane:
|
||||
`Build and Push CICD Images` computes base hash, checks registry, and publishes both CICD base and complete CICD images.
|
||||
2. Source + audit lanes:
|
||||
`Source Checks` and `Dependency Audits (Informational)` run after producer completion. Audit failures are intentionally non-blocking so both frontend and backend audits always report.
|
||||
3. Runtime image lanes:
|
||||
`Build Release Images` publishes `deployable-backend-staging` and `deployable-frontend-staging`; `Build Tester Images` publishes integration/e2e tester images; then `Production Images Complete` gates downstream runtime tests.
|
||||
4. Runtime validation lanes:
|
||||
`Runtime Black-Box Integration Tests` and `End-to-End Tests` validate staged runtime artifacts.
|
||||
5. Promotion lane:
|
||||
`Promote Release Images` runs only for automated `push` events on `main`. It retags validated staging artifacts to release repos (`deployable-backend`, `deployable-frontend`) with `latest`, `v<major>.<minor>.0`, `v<major>.<minor>.<patch>`, and `v<major>.<minor>.<patch>-<7-char-short-sha>` tags. If the `main` commit is untagged, CI creates the next patch tag automatically; if no prior semver tag exists, the bootstrap baseline is `v0.0.0`.
|
||||
6. Postmortem lanes:
|
||||
targeted postmortem jobs run when key lanes fail to capture diagnostics even when primary jobs fail early.
|
||||
|
||||
- `cicd-base:latest` - System dependencies (Python 3.14, Node.js 24, build tools, pre-installed dev packages)
|
||||
- `cicd:latest` - Complete environment (project code + optimized dependency installation)
|
||||
### Reliability and Traceability Behavior
|
||||
|
||||
**Recent Optimizations** (November 2025):
|
||||
Current workflow behavior includes:
|
||||
|
||||
- **Dependency-first build pattern** prevents cache invalidation on code changes
|
||||
- **Yarn PnP state regeneration** ensures reliable frontend builds
|
||||
- **Network-resilient E2E testing** with simplified Docker operations
|
||||
- **Memory-optimized frontend installations** with proper swap configuration
|
||||
- registry auth realm host pinning from `WWW-Authenticate` challenge when registry tokens are issued from a different host
|
||||
- bounded retry logic for docker login/pull/push operations in image lanes
|
||||
- digest/tag contract checks for deployable image references before runtime black-box tests
|
||||
- release-note summary generation in the promotion lane (commit bullets since the previous release tag, or from the `v0.0.0` bootstrap baseline on first release)
|
||||
- context hydration for image-build lanes by copying `/workspace` from the published CICD image
|
||||
- runner split between `ubuntu-act` and `ubuntu-act-8gb` based on lane resource requirements
|
||||
|
||||
For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
|
||||
For details of base/complete image build strategy, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
|
||||
For incident handling, see [CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md).
|
||||
|
||||
### Pipeline Jobs
|
||||
### Operator Quick Reference
|
||||
|
||||
All jobs run in parallel after the setup phases:
|
||||
Use workflow dispatch when you need deterministic reruns on a specific commit:
|
||||
|
||||
1. **Source Fast Gate**:
|
||||
Backend source checks (Ruff format/lint, Pyright), frontend source checks (Prettier, ESLint, TypeScript), and dispatches downstream build only on success.
|
||||
```bash
|
||||
# Example: rerun CI against an explicit commit
|
||||
# input head_sha=<commit>
|
||||
|
||||
2. **Setup Base**: Builds and pushes base Docker image (conditional)
|
||||
3. **Setup Complete**: Builds and pushes complete CI/CD Docker image
|
||||
4. **Code Quality**:
|
||||
Trailing whitespace check, end-of-file formatting, YAML syntax validation, and TOML syntax validation.
|
||||
# Example: force base image republish
|
||||
# input force_rebuild_base=true
|
||||
```
|
||||
|
||||
5. **Backend Validation**:
|
||||
Ruff formatting check, Ruff linting, Pyright type checking, Darglint docstring validation, unit tests with coverage, integration tests, and doctests (xdoctest).
|
||||
Recommended rerun order during flaky infrastructure incidents:
|
||||
|
||||
6. **Frontend Validation**:
|
||||
Prettier formatting check, ESLint linting, TypeScript compilation, and unit tests with coverage.
|
||||
- E2E tests (Playwright)
|
||||
1. `Build and Push CICD Images`
|
||||
2. failing runtime image lane (`Build Release Images` or `Build Tester Images`)
|
||||
3. downstream integration/e2e lanes
|
||||
4. `Promote Release Images` (if staging validation succeeded but promotion failed)
|
||||
|
||||
### Local CI/CD Testing
|
||||
|
||||
@@ -477,7 +488,7 @@ Build and test CI/CD images locally:
|
||||
1. Navigate to your pull request in Gitea
|
||||
2. Check the "Checks" tab for detailed results
|
||||
3. Click on individual job names to see logs
|
||||
4. All jobs must pass (100% green) before merging
|
||||
4. All required jobs must pass before merging (informational dependency-audit findings are non-blocking)
|
||||
|
||||
## Branch Protection and Merge Requirements
|
||||
|
||||
@@ -486,7 +497,7 @@ Build and test CI/CD images locally:
|
||||
The `main` branch is protected with the following requirements:
|
||||
|
||||
1. **No Direct Pushes**: All changes must come through pull requests
|
||||
2. **CI Must Pass**: All CI/CD jobs must be 100% green
|
||||
2. **CI Must Pass**: All required CI/CD jobs must pass (dependency audits are informational)
|
||||
3. **Review Required**: At least one team member approval needed
|
||||
4. **Up-to-date Branch**: Feature branch must be current with main
|
||||
|
||||
@@ -497,7 +508,7 @@ The `main` branch is protected with the following requirements:
|
||||
3. Address any CI failures by pushing fixes to the feature branch
|
||||
4. Request and receive code review approval
|
||||
5. Ensure branch is up-to-date with main
|
||||
6. Merge pull request (only available when all requirements met)
|
||||
6. Merge pull request (available when required jobs pass; informational audit failures do not block merge)
|
||||
|
||||
### If CI Fails
|
||||
|
||||
|
||||
@@ -4,6 +4,25 @@
|
||||
|
||||
Renovate is an automated dependency update tool that creates pull requests to keep your project dependencies up to date. This guide covers setting up Renovate for the plex-playlist project with optimal configuration.
|
||||
|
||||
## Repository Current Mode (2026-07)
|
||||
|
||||
This repository runs Renovate through `.gitea/workflows/renovate.yml`.
|
||||
|
||||
Current operational behavior:
|
||||
|
||||
1. Uses `ubuntu-act-8gb` runner due to npm/registry memory pressure.
|
||||
2. Prepares Renovate container image with mirror-first strategy:
|
||||
|
||||
- primary: `kankali.darkhelm.lan:3001/darkhelm.org/renovate:41`
|
||||
- fallback: `ghcr.io/renovatebot/renovate:41`
|
||||
|
||||
3. Uses digest-aware freshness checks before deciding whether local cached image is current.
|
||||
4. Selects endpoint dynamically between internal and external candidates based on preflight reachability.
|
||||
5. Selects token via preflight checks (repo access required) with fallback order from configured secrets.
|
||||
6. Uses constrained memory settings (`RENOVATE_NODE_ARGS`) and disables OSV alerts in this runner profile.
|
||||
|
||||
Treat workflow behavior as source of truth; use this doc as operator guidance.
|
||||
|
||||
## Setup Options
|
||||
|
||||
### Option 1: GitHub App (Recommended for GitHub)
|
||||
@@ -89,7 +108,7 @@ on:
|
||||
|
||||
jobs:
|
||||
renovate:
|
||||
runs-on: ubuntu-act
|
||||
runs-on: ubuntu-act-8gb
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
@@ -101,7 +120,7 @@ jobs:
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
env:
|
||||
RENOVATE_PLATFORM: gitea
|
||||
RENOVATE_ENDPOINT: https://dogar.darkhelm.org/api/v1
|
||||
RENOVATE_ENDPOINT: selected at runtime from internal/external candidates
|
||||
```
|
||||
|
||||
## Configuration Explanation
|
||||
@@ -204,6 +223,37 @@ docker run --rm \
|
||||
4. **Large Updates**: Major version updates may need manual review
|
||||
5. **Docker Registry**: Ensure base image updates don't break builds
|
||||
|
||||
### Current Workflow-Specific Failures
|
||||
|
||||
1. **Renovate image pull failures**
|
||||
|
||||
- Symptom: both mirror and GHCR candidates fail.
|
||||
- Check: runner DNS/egress and registry auth token validity.
|
||||
|
||||
2. **Endpoint preflight failures**
|
||||
|
||||
- Symptom: cannot reach both internal and external API endpoints.
|
||||
- Check: endpoint host mapping, TLS mode, and runner network route.
|
||||
|
||||
3. **Token access failures**
|
||||
|
||||
- Symptom: API `/repos/<org>/<repo>` check returns non-200.
|
||||
- Check: token scopes and secret ordering.
|
||||
|
||||
4. **OOM or abrupt termination**
|
||||
|
||||
- Symptom: process exits under memory pressure.
|
||||
- Check: `RENOVATE_NODE_ARGS`, PR concurrency limits, and optional feature toggles.
|
||||
|
||||
## Operator Notes
|
||||
|
||||
For this repository, prefer updating `.gitea/workflows/renovate.yml` over local one-off Renovate service changes. Keep docs and workflow in sync after changing:
|
||||
|
||||
- endpoint selection logic
|
||||
- token preflight/fallback order
|
||||
- image source policy (mirror/fallback)
|
||||
- memory and concurrency guardrails
|
||||
|
||||
### Quick Validation
|
||||
|
||||
For basic JSON validation without installing Renovate:
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# ADR004: Registry Image Resolution and Auth Resilience Policy
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-07-16
|
||||
|
||||
## Context
|
||||
|
||||
The CI and Renovate workflows run on self-hosted runners with intermittent DNS and network instability. Recent failures showed that image pull/push reliability depends on more than simple retries:
|
||||
|
||||
- registry token realms may resolve to a different host than the registry endpoint
|
||||
- mirror images can become stale relative to upstream
|
||||
- downstream lanes require immutable references from upstream lanes
|
||||
- Renovate must operate across internal/external endpoint paths with token variability
|
||||
|
||||
Without an explicit policy, each job implements ad hoc behavior and drift reintroduces flakiness.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt a cross-workflow reliability policy for registry/image operations:
|
||||
|
||||
1. Prefer mirrored images first, then fallback upstream sources when mirror resolution fails.
|
||||
2. Use digest-aware freshness checks when deciding whether local image cache is current.
|
||||
3. Pin registry auth realm hosts when `WWW-Authenticate` challenge host differs from registry host.
|
||||
4. Use bounded login/pull/push retry wrappers in image publication and consumption lanes.
|
||||
5. Propagate and verify immutable digest references for downstream runtime validation lanes.
|
||||
6. Keep retry counts/timeouts configurable as operational tuning, not architectural invariants.
|
||||
|
||||
## Scope
|
||||
|
||||
This ADR applies to:
|
||||
|
||||
- `.gitea/workflows/cicd.yaml`
|
||||
- `.gitea/workflows/renovate.yml`
|
||||
- helper scripts used for mirrored image resolution and lane orchestration
|
||||
|
||||
This ADR does not prescribe exact retry constants or runner sizing thresholds.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- reduced CI flakiness from token realm host mismatch and transient registry failures
|
||||
- stronger traceability via digest-first downstream checks
|
||||
- clearer operator expectations for endpoint/token/image fallback behavior
|
||||
- consistent reliability approach across CICD and Renovate workflows
|
||||
|
||||
Negative:
|
||||
|
||||
- increased workflow script complexity and duplicated helper logic inside isolated step shells
|
||||
- additional maintenance burden to keep helper patterns consistent across lanes
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
1. Keep per-job ad hoc retries only.
|
||||
|
||||
- Rejected due to repeated regressions and inconsistent behavior.
|
||||
|
||||
2. Depend solely on upstream registries.
|
||||
|
||||
- Rejected due to local network and availability constraints.
|
||||
|
||||
3. Rebuild missing artifacts in downstream lanes.
|
||||
|
||||
- Rejected because it breaks publish-once/consume-many behavior and weakens traceability.
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- `ADR002-cicd_base_image_tagging.md`
|
||||
- `ADR003-deployable_runtime_image_contract.md`
|
||||
Reference in New Issue
Block a user