ci: consolidate cicd workflow lanes, clean up helpers, and align CI docs (#86)

## Summary
This PR simplifies the CICD workflow by merging related lanes, reducing duplicated script logic, and keeping the same overall pipeline behavior and gates. It also updates CI documentation to match the new job topology.

## What Changed

### Workflow consolidation
- Merged base and complete CICD image publication into one producer job:
  - Build and Push CICD Images
- Merged dependency audits into one informational lane:
  - Dependency Audits (Informational)
- Merged runtime image build lanes into one release producer:
  - Build Release Images
- Merged tester image build lanes into one tester producer:
  - Build Tester Images

### Dependency/gate rewiring
- Updated downstream needs to consume merged producers.
- Kept output contracts for deployable and tester image references.
- Updated production and postmortem gates to the new job IDs.

### Cleanup/simplification
- Removed duplicate helper-function definition(s) in CICD scripts.
- Replaced a manual docker login retry loop with existing retry helper usage.
- Removed redundant shell option declarations where behavior was unchanged.
- Removed one unused E2E environment variable.

### Documentation alignment
- Updated CI architecture documentation to reflect merged workflow lanes.
- Updated troubleshooting guidance to reference current job sequencing.

## Why
- Reduce job startup overhead on self-hosted runners.
- Keep behavior consistent while lowering workflow complexity.
- Improve maintainability by removing duplicated/unused script fragments.
- Keep docs in sync with operational workflow reality.

## Validation
- Workflow file checks passed with pre-commit.
- Documentation checks passed with pre-commit (including markdownlint/prettier).
- No diagnostics/errors reported for updated workflow/docs files.

## Risk and Impact
- Low-to-medium operational risk due to job-ID/needs rewiring.
- Mitigated by preserving output keys consumed by integration and e2e lanes.
- Audit lane remains informational-only (non-blocking), same intent as before.

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #86

docs/CICD_TROUBLESHOOTING_GUIDE.md

@@ -4,6 +4,155 @@
 
 This document captures the specific optimizations, fixes, and troubleshooting approaches developed during November 2025 for the plex-playlist CI/CD pipeline. Each entry includes the problem, root cause analysis, solution implementation, and performance impact.
 
+## Current Workflow Reference (2026-07)
+
+The authoritative workflow is `.gitea/workflows/cicd.yaml`.
+
+When this guide conflicts with older examples, prefer:
+
+- current job names and dependencies in `cicd.yaml`
+- current registry endpoint `kankali.darkhelm.lan:3001`
+- current retry and auth-realm host pinning logic embedded in image build lanes
+
+## High-Value Failure Signatures (Current)
+
+### 0. Dependency audit step fails but workflow stays green
+
+**Symptom**:
+
+```text
+frontend audit reported vulnerabilities
+backend audit reported vulnerabilities
+```
+
+and overall workflow still succeeds.
+
+**Cause**: expected behavior. `Dependency Audits (Informational)` is intentionally non-blocking and runs both frontend and backend audit steps.
+
+**Fast check**:
+
+1. Confirm `Dependency Audits (Informational)` ran.
+2. Confirm both audit step logs are present.
+
+**Fix**:
+
+No CI fix required unless policy changes. Treat findings as remediation backlog items.
+
+### 1. `docker_login_with_retry: command not found`
+
+**Symptom**:
+
+```text
+line <n>: docker_login_with_retry: command not found
+```
+
+**Cause**: shell helper function referenced in a job step but missing in that same step's `run` block.
+
+**Fast check**:
+
+1. Open failing job step in `.gitea/workflows/cicd.yaml`.
+2. Confirm `docker_login_with_retry()` is defined before first call in that block.
+
+**Fix**:
+
+Add the helper definition locally in that step block (functions do not cross step boundaries).
+
+### 2. Registry token timeout while pushing/pulling
+
+**Symptom**:
+
+```text
+Client.Timeout exceeded while awaiting headers
+... /v2/token?...service=container_registry
+```
+
+**Cause**: runner resolves/pins registry host, but token realm host from `WWW-Authenticate` challenge is unresolved/unreachable.
+
+**Fast check**:
+
+1. Verify `Configure registry host resolution` step ran.
+2. Confirm auth realm host pinning logic is present in failing lane.
+3. Check lane-specific login/push retry helpers are active.
+
+**Fix**:
+
+Use `ensure_registry_auth_realm_host` + `docker_login_with_retry` + bounded `retry_registry_op` in the failing lane.
+
+### 3. Empty downstream digest/tag outputs
+
+**Symptom**:
+
+```text
+evaluated to '%!t(string=)'
+```
+
+**Cause**: upstream image lane failed before writing expected outputs (`*_tag_ref`, `*_digest_ref`).
+
+**Fast check**:
+
+1. Inspect upstream image job conclusion (`Build Frontend Main Image`, `Build Integration Tester Image`, etc.).
+2. Confirm output writes (`echo key=value >> $GITHUB_OUTPUT`) execute after push and digest resolution.
+
+**Fix**:
+
+Repair failing upstream lane first; downstream expressions become valid once outputs are emitted.
+
+### 4. Base image publication mismatch
+
+**Symptom**:
+
+```text
+Required immutable base image is not available
+```
+
+**Cause**: expected hash tag not yet published or failed publication lane.
+
+**Fix order**:
+
+1. `Build and Push CICD Images`
+2. remaining source, image, and runtime lanes
+
+### 5. Release tags missing after tests passed
+
+**Symptom**:
+
+```text
+staging images exist but deployable-backend/deployable-frontend tags not updated
+```
+
+**Cause**: promotion lane did not run or failed (`Promote Release Images`).
+
+**Fast check**:
+
+1. Confirm `Runtime Black-Box Integration Tests` and `End-to-End Tests` succeeded.
+2. Confirm the workflow run was an automated `push` on `main`; promotion is skipped for PR validation and other non-main events.
+3. Check `Promote Release Images` logs for registry login, tag creation, pull/tag/push failures.
+4. Verify staging refs (`deployable-backend-staging`, `deployable-frontend-staging`) were emitted by `Build Release Images`.
+
+**Fix**:
+
+Re-run `Promote Release Images` after correcting registry/auth issues.
+
+### 6. Unexpected release version chosen
+
+**Symptom**:
+
+```text
+release_version differs from expected manual guess
+```
+
+**Cause**: promotion versioning follows workflow rules:
+
+1. Promotion only runs for automated `push` events on `main`.
+2. If the `main` commit already has semver tag `vX.Y.Z`, use that exact patch tag.
+3. Otherwise, find the latest semver tag in the repository and auto-create the next patch tag.
+4. If no semver tag exists yet, bootstrap from `v0.0.0` and create `v0.0.1`.
+5. Also publish `vX.Y.0` and `vX.Y.Z-<7-char-short-sha>`.
+
+**Fix**:
+
+If you need an exact patch version, tag the `main` commit with semver before promotion runs; otherwise let CI assign the next patch automatically.
+
 ## Performance Optimizations
 
 ### 1. Dependency-First Build Pattern