Handling separation of concerns for the CICD.

Signed-off-by: copilotcoder <copilotcoder@darkhelm.org>

docs/CICD_MULTI_STAGE_BUILD.md

@@ -21,9 +21,11 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 - **Playwright CLI and browsers** (Chromium, Firefox, WebKit) - ~400MB cached
 - SSH helper scripts for git operations
 
-**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd-base:latest`
+**Registry**:
+- Immutable: `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:<hash>`
+- Convenience: `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:latest`
 
-**Rebuild Triggers**: Only when `Dockerfile.cicd-base` changes (detected via SHA256 hash)
+**Rebuild Triggers**: Only when `Dockerfile.cicd-base`, `.dockerignore`, or the shared hash helper changes
 
 ### Stage 2: Complete Image (`Dockerfile.cicd`)
 **Purpose**: Inherits from base and adds project code and dependencies.
@@ -87,36 +89,50 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
 
 ### Caching Strategy
 1. **Docker Layer Caching**: Docker automatically caches unchanged layers
-2. **Registry Caching**: Base image pulled from registry if available
-3. **Hash-Based Invalidation**: Base image tagged with Dockerfile hash
-4. **Conditional Building**: Base only rebuilds when `Dockerfile.cicd-base` changes
+2. **Registry Caching**: Base image is built once and then pulled by all runners
+3. **Hash-Based Invalidation**: Base image tagged with a shared helper-derived hash
+4. **Conditional Building**: Base workflow rebuilds only when base inputs change
 
 ## CI/CD Workflow
 
 ```yaml
 jobs:
-  setup-base:
-    name: Build and Push CICD Base Image
+  publish-base:
+    name: Build and Publish CICD Base Image
     steps:
-      - name: Check if base image needs rebuilding
-        # Calculates SHA256 of Dockerfile.cicd-base
-        # Pulls existing image with hash tag if available
-        # Sets needs_build=false if image exists
+      - name: Compute base hash
+        # Uses scripts/compute-cicd-base-hash.sh
+        # Hashes Dockerfile.cicd-base and .dockerignore
 
       - name: Build and push base image
         if: needs_build == 'true'
-        # Only runs when base Dockerfile changed
+        # Only runs when the immutable base tag is missing or force rebuild is requested
         # Tags with both hash and 'latest'
 
+      - name: Verify published base image
+        # Confirms the immutable tag is visible and pullable before success
+
+  prepare-base-ref:
+    name: Prepare CICD Base Reference
+    steps:
+      - name: Compute immutable base ref
+        # Uses the same helper as the base workflow
+
   setup:
     name: Build and Push CICD Complete Image
-    needs: setup-base
+    needs: prepare-base-ref
     steps:
       - name: Build and push complete CICD image
-        # Always runs, inherits from base:latest
+        # Always runs, inherits from cicd-base:<hash>
+        # Waits briefly for the immutable base tag to appear, then fails clearly if missing
         # Contains project code and dependencies
 ```
 
+### Responsibility Split
+- `.gitea/workflows/cicd-base.yml` owns base publication and verification.
+- `.gitea/workflows/cicd.yml` owns complete-image build, tests, and deployment checks.
+- Main CI never rebuilds the base image locally.
+
 ## Local Development
 
 ### Building Base Image
@@ -150,6 +166,9 @@ rm /tmp/ssh_key
 ./scripts/build-cicd-local.sh
 ```
 
+The local script now uses `scripts/compute-cicd-base-hash.sh`, so the local hash-tagged
+base image matches the immutable tag CI expects.
+
 ## Memory Optimization
 
 ### Raspberry Pi 4GB Constraints
@@ -183,9 +202,15 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
 ### Troubleshooting
 1. **Base Image Issues**: Check `Dockerfile.cicd-base` syntax and system dependencies
 2. **Complete Image Issues**: Usually project dependency or SSH access problems
-3. **Cache Misses**: Verify registry connectivity and hash calculation
+3. **Cache Misses**: Verify registry connectivity and the shared base hash calculation
 4. **Memory Issues**: Check swap setup and Node.js memory limits
 
+### Missing Immutable Base Tag
+- Symptom: main CI fails with `Required immutable base image is not available`
+- Cause: the expected `cicd-base:<hash>` has not been published yet
+- Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs
+- Design note: main CI intentionally fails instead of rebuilding the base locally
+
 ### Common Issues
 - **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured
 - **Registry Authentication**: Verify PACKAGE_ACCESS_TOKEN permissions

docs/CICD_TROUBLESHOOTING_GUIDE.md

@@ -217,6 +217,31 @@ RUN echo "=== Base Image Optimization Status ===" && \
 
 ## Troubleshooting Playbook
 
+### Missing Immutable Base Image
+
+**Symptom**:
+```
+❌ Required immutable base image is not available: kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:<hash>
+Publish the base image via the CICD Base Image workflow before rerunning main CI.
+```
+
+**Meaning**: The main CI workflow computed the expected base hash correctly, but the
+dedicated base-image workflow has not published that immutable tag yet.
+
+**Checks**:
+1. Confirm whether `Dockerfile.cicd-base`, `.dockerignore`, or `scripts/compute-cicd-base-hash.sh` changed in the branch.
+2. Check the `CICD Base Image` workflow for the same commit or PR.
+3. Verify the registry contains `plex-playlist-cicd-base:<hash>`.
+
+**Resolution**:
+1. If the base workflow is still running, rerun main CI after it completes.
+2. If the base workflow did not trigger, run it manually with `force_rebuild=false`.
+3. If the tag should be republished despite already existing, run it manually with `force_rebuild=true`.
+
+**Why CI does not self-heal here**: rebuilding the base inside the main workflow would
+make different runners produce and consume different local artifacts, which defeats the
+publish-once/consume-many design.
+
 ### Docker Build Failures
 
 #### 1. rsync Command Not Found

docs/adr/ADR002-cicd_base_image_tagging.md

@@ -6,33 +6,45 @@
 ## Context
 
 CICD builds depend on a shared base image containing stable system dependencies.
-To optimize build times while preserving correctness, the pipeline uses cache-aware
-image reuse and fallback behavior.
+To optimize build times across a fleet of self-hosted runners while preserving
+correctness, the pipeline now separates base-image publication from application CI.
+The expensive base image should be built once per content change, pushed to the
+registry under an immutable tag, and then pulled by every runner that needs it.
 
 This branch reinforced the workflow by ensuring hash-driven base tags are always
 computed and non-empty before tagging/pushing, preventing invalid image references.
+It also expanded the base hash inputs to include `.dockerignore` so context changes
+that affect the base build invalidate the published tag.
 
 ## Decision
 
-Adopt hash-based tagging for the CICD base image with latest fallback:
+Adopt a hardened split workflow for the CICD base image:
 
-1. Compute a deterministic hash from Dockerfile.cicd-base content.
-2. Publish base image as both:
+1. Compute a deterministic hash from `Dockerfile.cicd-base` and `.dockerignore`.
+2. Publish the base image in a dedicated workflow as both:
    - hash-specific tag: cicd-base:<hash>
    - rolling tag: cicd-base:latest
-3. Prefer pulling hash-specific image; fallback to latest when needed.
-4. Validate/fallback hash propagation in workflow to avoid empty tag references.
+3. Treat the hash-specific tag as the source of truth for all CI consumers.
+4. Keep `latest` only as a convenience tag for humans and manual debugging.
+5. Make the main CI workflow consume only the immutable hash tag and fail clearly
+  if that base image has not been published yet.
+6. Add bounded polling in main CI to tolerate short publish/consume races between
+  the dedicated base workflow and the main workflow.
 
 ## Consequences
 
 Positive:
 - Faster CI via stable base-layer reuse
 - Better traceability from base image to Dockerfile content
+- Single publish, many pulls across the runner fleet
 - Reduced risk of malformed image references in workflow execution
+- Clearer separation of concerns between artifact publication and application CI
 
 Negative:
 - Slightly more workflow complexity
 - Registry stores additional hash-tagged images
+- Main CI now fails fast when the expected base image is missing instead of
+  rebuilding it locally
 
 ## Alternatives Considered
 
@@ -40,3 +52,6 @@ Negative:
   - Rejected due to reduced traceability and cache precision
 - Build base image every run
   - Rejected due to slower pipelines and wasted compute
+- Allow main CI to rebuild missing base images locally
+  - Rejected because it reintroduces split-brain behavior across runners and
+    defeats publish-once/consume-many optimization