Files
plex-playlist/.gitea/workflows/docker-build-main.yaml
T
darkhelmandcopilotcoder 1f6cafa1bc
CICD Start / Sanity and Base Decision (push) Failing after 11m34s
Renovate Dependency Updates / Renovate Dependencies (push) Failing after 1h42m33s
TASK: Replace integration lane with post-build backend runtime black-box tests (#72)
## Summary

Replace the existing source-context integration lane with backend runtime black-box integration checks that run against started deployable containers.

This change wires deployable backend image references (both commit tag and immutable digest) from the build workflow into the tests workflow, then validates runtime behavior over network endpoints.

## Why

Integration confidence should come from testing running service artifacts, not only source-mounted or in-process execution.

## What Changed

- Build workflow now:
  - Publishes deployable backend image tag reference and digest reference
  - Exposes both as job outputs
  - Passes both references into CICD Tests dispatch inputs

- CICD Tests workflow now:
  - Accepts deployable backend tag and digest inputs
  - Propagates these through setup outputs
  - Replaces previous integration lane behavior with runtime black-box execution:
    - Starts isolated Docker network
    - Starts Postgres container
    - Starts backend container from digest-pinned deployable image
    - Enforces tag-to-digest consistency before running checks
    - Runs endpoint checks against live container:
      - GET /
      - GET /compatibility
      - GET /health
    - Captures backend/db logs and container state on failure
    - Cleans up containers and network via trap

- Documentation updated:
  - Runtime contract enforcement section now includes runtime black-box integration checks
  - CI success summary now reflects runtime integration lane behavior

## Scope

Included:
- Backend runtime black-box integration replacement for the existing integration lane
- Digest + tag identity enforcement
- Failure diagnostics for triage

Out of scope:
- Frontend runtime smoke checks
- E2E lane redesign

## Acceptance Criteria Mapping

- Integration tests execute against runtime container endpoints: 
- Integration lane consumes built image references (not source-mounted execution): 
- Failures surface service logs and test logs for triage: 

## Verification

- Workflow files pass local validation checks
- Pre-commit hooks pass on committed changes
- Branch pushed and ready for PR review

## Related

- Issue: #61
- Dependency context: #66

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #72
2026-07-05 22:48:57 -04:00

482 lines
19 KiB
YAML

name: Docker Build Main
on:
workflow_dispatch:
inputs:
head_sha:
description: Commit SHA to process
required: false
base_hash:
description: Immutable base hash to use for CICD base image
required: false
source_workflow:
description: Upstream workflow name
required: false
base_needed:
description: Whether base rebuild was required
required: false
trace_id:
description: Correlation id propagated across CICD dispatch chain
required: false
env:
GITEA_SSH_HOST: kankali.darkhelm.lan
GITEA_SSH_PORT: "2222"
GITEA_REPO_SSH_URL: ssh://git@kankali.darkhelm.lan:2222/DarkHelm.org/plex-playlist.git
GITEA_REGISTRY: kankali.darkhelm.lan:3001
GITEA_REGISTRY_IP: 10.18.75.2
GITEA_REGISTRY_HOST: kankali.darkhelm.lan
defaults:
run:
shell: bash
concurrency:
group: main-build-${{ github.sha }}
cancel-in-progress: true
jobs:
startup-audit:
name: Main Workflow Startup Audit
# Pin startup audit to high-memory worker to avoid setup-stage runner churn.
runs-on: ubuntu-act-8gb
timeout-minutes: 5
steps:
- name: Identify runner
shell: sh
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Emit startup diagnostics
shell: sh
env:
EVENT_NAME: ${{ github.event_name }}
SOURCE_WORKFLOW: ${{ github.event.inputs.source_workflow }}
BASE_NEEDED: ${{ github.event.inputs.base_needed }}
HEAD_SHA_INPUT: ${{ github.event.inputs.head_sha }}
HEAD_SHA_FALLBACK: ${{ github.sha }}
BASE_HASH_INPUT: ${{ github.event.inputs.base_hash }}
REF: ${{ github.ref }}
REF_NAME: ${{ github.ref_name }}
HEAD_REF: ${{ github.head_ref }}
TRACE_ID_INPUT: ${{ github.event.inputs.trace_id }}
TARGET_LABEL: ubuntu-act-8gb
run: |
RESOLVED_HEAD_SHA="${HEAD_SHA_INPUT:-${HEAD_SHA_FALLBACK}}"
TRACE_SUFFIX="$(printf '%s' "${RESOLVED_HEAD_SHA}" | cut -c1-8)"
TRACE_ID="${TRACE_ID_INPUT:-cicd-main-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${TRACE_SUFFIX}}"
echo "=== Main Workflow Startup Audit ==="
echo "event_name=${EVENT_NAME}"
echo "source_workflow=${SOURCE_WORKFLOW}"
echo "base_needed=${BASE_NEEDED}"
echo "head_sha_input=${HEAD_SHA_INPUT}"
echo "head_sha=${RESOLVED_HEAD_SHA}"
echo "base_hash_input=${BASE_HASH_INPUT}"
echo "ref=${REF}"
echo "ref_name=${REF_NAME}"
echo "head_ref=${HEAD_REF}"
echo "trace_id=${TRACE_ID}"
echo "target_runner_label=${TARGET_LABEL}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "startup_audit=ok"
- name: Check control-plane reachability
shell: sh
run: |
SERVER_URL="${GITHUB_SERVER_URL:-}"
if command -v curl >/dev/null 2>&1; then
if [ -n "${SERVER_URL}" ] && curl -fsS --connect-timeout 5 --max-time 10 "${SERVER_URL%/}/api/v1/version" >/tmp/gitea-version.json 2>/dev/null; then
echo "gitea_api_reachable=true"
cat /tmp/gitea-version.json || true
else
echo "gitea_api_reachable=false"
fi
else
echo "curl unavailable; skipping reachability check"
fi
- &failure_diagnostics_step
name: Failure diagnostics
if: failure()
run: |
echo "=== Failure Diagnostics ==="
date -u '+timestamp_utc=%Y-%m-%dT%H:%M:%SZ'
echo "runner_name=${RUNNER_NAME:-unknown}"
echo "runner_hostname=${HOSTNAME:-unknown}"
uname -a || true
cat /etc/os-release 2>/dev/null || true
df -h || true
free -h || true
ps aux --sort=-%mem | head -n 30 || true
if command -v docker >/dev/null 2>&1; then
echo "=== Docker Diagnostics ==="
docker version || true
docker info || true
docker ps -a || true
docker images --digests | head -n 50 || true
else
echo "docker not available on this runner"
fi
echo "=== Kernel Tail ==="
dmesg | tail -n 120 || true
build:
name: Build and Push CICD Complete Image
# Pin main image build to high-memory worker to reduce setup-time failures.
runs-on: ubuntu-act-8gb
needs: startup-audit
timeout-minutes: 60
outputs:
head_sha: ${{ steps.meta.outputs.head_sha }}
deployable_backend_tag_ref: ${{ steps.deployable_backend_ref.outputs.deployable_backend_tag_ref }}
deployable_backend_digest_ref: ${{ steps.deployable_backend_ref.outputs.deployable_backend_digest_ref }}
steps:
- name: Identify runner
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Audit trigger context
env:
EVENT_NAME: ${{ github.event_name }}
SOURCE_WORKFLOW: ${{ github.event.inputs.source_workflow }}
BASE_NEEDED: ${{ github.event.inputs.base_needed }}
HEAD_SHA_INPUT: ${{ github.event.inputs.head_sha }}
HEAD_SHA_FALLBACK: ${{ github.sha }}
BASE_HASH_INPUT: ${{ github.event.inputs.base_hash }}
REF: ${{ github.ref }}
REF_NAME: ${{ github.ref_name }}
HEAD_REF: ${{ github.head_ref }}
TRACE_ID_INPUT: ${{ github.event.inputs.trace_id }}
run: |
RESOLVED_HEAD_SHA="${HEAD_SHA_INPUT:-${HEAD_SHA_FALLBACK}}"
TRACE_ID="${TRACE_ID_INPUT:-cicd-main-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${RESOLVED_HEAD_SHA:0:8}}"
echo "=== Dispatch Audit: CICD Main Build ==="
echo "event_name=${EVENT_NAME}"
echo "source_workflow=${SOURCE_WORKFLOW}"
echo "base_needed=${BASE_NEEDED}"
echo "head_sha_input=${HEAD_SHA_INPUT}"
echo "head_sha=${RESOLVED_HEAD_SHA}"
echo "base_hash_input=${BASE_HASH_INPUT}"
echo "ref=${REF}"
echo "ref_name=${REF_NAME}"
echo "head_ref=${HEAD_REF}"
echo "trace_id=${TRACE_ID}"
- name: Resolve head SHA
id: meta
env:
HEAD_SHA_INPUT: ${{ github.event.inputs.head_sha }}
HEAD_SHA_FALLBACK: ${{ github.sha }}
run: |
RESOLVED_HEAD_SHA="${HEAD_SHA_INPUT:-${HEAD_SHA_FALLBACK}}"
echo "head_sha=${RESOLVED_HEAD_SHA}" >> "$GITHUB_OUTPUT"
- name: Minimal checkout for build and verification inputs
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
run: |
set -e
umask 077
trap 'rm -f ~/.ssh/id_rsa' EXIT
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
mkdir -p ~/.ssh
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -p "${GITEA_SSH_PORT}" "${GITEA_SSH_HOST}" >> ~/.ssh/known_hosts 2>/dev/null
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git clone --depth 1 --no-checkout "${GITEA_REPO_SSH_URL}" .
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git fetch --depth 1 origin "${HEAD_SHA}" >/dev/null 2>&1 || true
git checkout FETCH_HEAD -- \
.dockerignore \
Dockerfile.backend \
Dockerfile.frontend \
Dockerfile.cicd \
Dockerfile.cicd-base \
backend \
frontend \
scripts/compute-cicd-base-hash.sh \
scripts/check-dockerfile-boundaries.sh \
scripts/verify-deployable-image-purity.sh
chmod +x scripts/compute-cicd-base-hash.sh
- name: Verify deployable runtime boundaries
run: |
set -e
bash ./scripts/check-dockerfile-boundaries.sh
- name: Build and verify deployable runtime image purity
env:
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
run: |
set -e
docker build -f Dockerfile.backend \
-t deployable-backend:"${HEAD_SHA}" .
docker build -f Dockerfile.frontend \
--target production \
-t deployable-frontend:"${HEAD_SHA}" .
bash ./scripts/verify-deployable-image-purity.sh \
--image deployable-backend:"${HEAD_SHA}" \
--profile backend
bash ./scripts/verify-deployable-image-purity.sh \
--image deployable-frontend:"${HEAD_SHA}" \
--profile frontend
- name: Push deployable backend runtime image
id: deployable_backend_ref
env:
PACKAGE_ACCESS_TOKEN: ${{ secrets.PACKAGE_ACCESS_TOKEN }}
REGISTRY_USER: ${{ github.actor }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
run: |
set -euo pipefail
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
echo "${PACKAGE_ACCESS_TOKEN}" | docker login "http://${GITEA_REGISTRY}" -u "${REGISTRY_USER}" --password-stdin
DEPLOYABLE_BACKEND_REPO="${GITEA_REGISTRY}/darkhelm.org/deployable-backend"
DEPLOYABLE_BACKEND_TAG_REF="${DEPLOYABLE_BACKEND_REPO}:${HEAD_SHA}"
docker tag "deployable-backend:${HEAD_SHA}" "${DEPLOYABLE_BACKEND_TAG_REF}"
docker push "${DEPLOYABLE_BACKEND_TAG_REF}"
docker pull "${DEPLOYABLE_BACKEND_TAG_REF}" >/dev/null
DEPLOYABLE_BACKEND_DIGEST_REF="$({
docker image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "${DEPLOYABLE_BACKEND_TAG_REF}" \
| grep "^${DEPLOYABLE_BACKEND_REPO}@sha256:" \
| head -n 1
} || true)"
if [ -z "${DEPLOYABLE_BACKEND_DIGEST_REF}" ]; then
echo "❌ Unable to resolve deployable backend digest reference"
exit 1
fi
echo "deployable_backend_tag_ref=${DEPLOYABLE_BACKEND_TAG_REF}" >> "$GITHUB_OUTPUT"
echo "deployable_backend_digest_ref=${DEPLOYABLE_BACKEND_DIGEST_REF}" >> "$GITHUB_OUTPUT"
echo "deployable_backend_tag_ref=${DEPLOYABLE_BACKEND_TAG_REF}"
echo "deployable_backend_digest_ref=${DEPLOYABLE_BACKEND_DIGEST_REF}"
- name: Build and push complete CICD image
env:
PACKAGE_ACCESS_TOKEN: ${{ secrets.PACKAGE_ACCESS_TOKEN }}
REGISTRY_USER: ${{ github.actor }}
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
BASE_HASH_INPUT: ${{ github.event.inputs.base_hash }}
run: |
set -e
umask 077
trap 'rm -f /tmp/ssh_key' EXIT
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
echo "${PACKAGE_ACCESS_TOKEN}" | docker login "http://${GITEA_REGISTRY}" -u "${REGISTRY_USER}" --password-stdin
if [ -n "${BASE_HASH_INPUT}" ]; then
BASE_HASH="${BASE_HASH_INPUT}"
echo "Using provided immutable base hash from upstream workflow dispatch: ${BASE_HASH}"
else
BASE_HASH=$(./scripts/compute-cicd-base-hash.sh)
echo "No base_hash input provided; computed base hash locally: ${BASE_HASH}"
fi
BASE_IMAGE="${GITEA_REGISTRY}/darkhelm.org/plex-playlist-cicd-base:${BASE_HASH}"
verify_base_image() {
# Fast path: base image already present locally.
if docker image inspect "${BASE_IMAGE}" >/dev/null 2>&1; then
return 0
fi
# Manifest inspect unreliable on these runners; use pull as truth.
timeout 1800 docker pull "${BASE_IMAGE}" >/tmp/base-image-pull.log 2>&1
}
max_attempts=2
sleep_seconds=15
for i in $(seq 1 "${max_attempts}"); do
echo "Base availability check ${i}/${max_attempts}..."
if verify_base_image; then
echo "✓ Base image available and hash matched: ${BASE_IMAGE}"
break
else
pull_exit=$?
echo "Base pull attempt ${i} failed with exit code ${pull_exit}"
if [ "${pull_exit}" -eq 124 ]; then
echo "Pull timed out after 1800s while downloading base image"
fi
if [ -s /tmp/base-image-pull.log ]; then
tail -n 40 /tmp/base-image-pull.log || true
fi
if [ "${i}" -eq "${max_attempts}" ]; then
echo "❌ Required immutable base image is not available or mismatched: ${BASE_IMAGE}"
exit 1
fi
sleep "${sleep_seconds}"
fi
done
echo "✓ Base image ready: ${BASE_IMAGE}"
echo "${SSH_PRIVATE_KEY}" > /tmp/ssh_key
chmod 600 /tmp/ssh_key
export DOCKER_BUILDKIT=1
docker build -f Dockerfile.cicd \
--secret id=ssh_private_key,src=/tmp/ssh_key \
--add-host "${GITEA_SSH_HOST}:${GITEA_REGISTRY_IP}" \
--build-arg GITHUB_SHA="${HEAD_SHA}" \
--build-arg CICD_BASE_IMAGE="${BASE_IMAGE}" \
-t cicd:latest .
docker tag cicd:latest "${GITEA_REGISTRY}/darkhelm.org/plex-playlist-cicd:latest"
docker tag cicd:latest "${GITEA_REGISTRY}/darkhelm.org/plex-playlist-cicd:${HEAD_SHA}"
docker push "${GITEA_REGISTRY}/darkhelm.org/plex-playlist-cicd:latest"
docker push "${GITEA_REGISTRY}/darkhelm.org/plex-playlist-cicd:${HEAD_SHA}"
- *failure_diagnostics_step
dispatch-tests:
name: Dispatch CICD Tests
runs-on: ubuntu-act
timeout-minutes: 15
needs: build
if: needs.build.result == 'success'
steps:
- name: Dispatch tests workflow
env:
ACTIONS_TRIGGER_TOKEN: ${{ secrets.ACTIONS_TRIGGER_TOKEN }}
PACKAGE_ACCESS_TOKEN: ${{ secrets.PACKAGE_ACCESS_TOKEN }}
HEAD_SHA: ${{ needs.build.outputs.head_sha }}
DEPLOYABLE_BACKEND_TAG_REF: ${{ needs.build.outputs.deployable_backend_tag_ref }}
DEPLOYABLE_BACKEND_DIGEST_REF: ${{ needs.build.outputs.deployable_backend_digest_ref }}
REPO_FULL: ${{ github.repository }}
HEAD_REF: ${{ github.head_ref }}
REF_NAME: ${{ github.ref_name }}
TRACE_ID_INPUT: ${{ github.event.inputs.trace_id }}
run: |
set -e
DISPATCH_TOKEN="${ACTIONS_TRIGGER_TOKEN:-${PACKAGE_ACCESS_TOKEN:-}}"
if [ -z "${DISPATCH_TOKEN}" ]; then
echo "❌ Missing dispatch token. Set ACTIONS_TRIGGER_TOKEN (repo write scope) or ensure PACKAGE_ACCESS_TOKEN has Actions workflow-dispatch permissions."
exit 1
fi
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
REPO_OWNER="${REPO_FULL%/*}"
REPO_NAME="${REPO_FULL#*/}"
TARGET_REF="${HEAD_REF:-${REF_NAME}}"
TRACE_ID="${TRACE_ID_INPUT:-cicd-main-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${HEAD_SHA:0:8}}"
echo "trace_id=${TRACE_ID}"
echo "target_ref=${TARGET_REF}"
CANDIDATE_API_BASES=()
if [ -n "${GITHUB_SERVER_URL:-}" ]; then
CANDIDATE_API_BASES+=("${GITHUB_SERVER_URL%/}/api/v1")
fi
CANDIDATE_API_BASES+=("http://${GITEA_REGISTRY_IP}:3001/api/v1")
CANDIDATE_API_BASES+=("http://${GITEA_REGISTRY_HOST}:3001/api/v1")
ensure_curl() {
if command -v curl >/dev/null 2>&1; then
return 0
fi
if command -v apt-get >/dev/null 2>&1; then
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq curl ca-certificates
fi
command -v curl >/dev/null 2>&1
}
ensure_curl || { echo "❌ curl unavailable for dispatch"; exit 1; }
HELPER_PATH="/tmp/dispatch-workflow.sh"
fetch_dispatch_helper() {
local helper_ref="$1"
local api_base
for api_base in "${CANDIDATE_API_BASES[@]}"; do
helper_url="${api_base}/repos/${REPO_OWNER}/${REPO_NAME}/raw/scripts/dispatch-workflow.sh?ref=${helper_ref}"
if curl -fsS --connect-timeout 5 --max-time 20 \
-H "Authorization: token ${DISPATCH_TOKEN}" \
-H "User-Agent: plex-playlist-cicd-main" \
-o "${HELPER_PATH}" \
"${helper_url}"; then
chmod +x "${HELPER_PATH}"
return 0
fi
done
return 1
}
if ! fetch_dispatch_helper "${TARGET_REF}" && ! fetch_dispatch_helper "${HEAD_SHA}"; then
echo "❌ Failed to fetch scripts/dispatch-workflow.sh from repository"
exit 1
fi
DISPATCH_ARGS=(
--token "${DISPATCH_TOKEN}"
--repo "${REPO_FULL}"
--workflow "cicd-tests.yaml"
--ref "${TARGET_REF}"
--head-sha "${HEAD_SHA}"
--source-workflow "CICD Main Build"
--trace-id "${TRACE_ID}"
--input "deployable_backend_tag_ref=${DEPLOYABLE_BACKEND_TAG_REF}"
--input "deployable_backend_digest_ref=${DEPLOYABLE_BACKEND_DIGEST_REF}"
)
for API_BASE in "${CANDIDATE_API_BASES[@]}"; do
DISPATCH_ARGS+=(--api-base "${API_BASE}")
done
"${HELPER_PATH}" "${DISPATCH_ARGS[@]}"
- *failure_diagnostics_step