Files
plex-playlist/.gitea/workflows/cicd-source-checks.yaml
T
darkhelmandcopilotcoder 1f6cafa1bc
CICD Start / Sanity and Base Decision (push) Failing after 11m34s
Renovate Dependency Updates / Renovate Dependencies (push) Failing after 1h42m33s
TASK: Replace integration lane with post-build backend runtime black-box tests (#72)
## Summary

Replace the existing source-context integration lane with backend runtime black-box integration checks that run against started deployable containers.

This change wires deployable backend image references (both commit tag and immutable digest) from the build workflow into the tests workflow, then validates runtime behavior over network endpoints.

## Why

Integration confidence should come from testing running service artifacts, not only source-mounted or in-process execution.

## What Changed

- Build workflow now:
  - Publishes deployable backend image tag reference and digest reference
  - Exposes both as job outputs
  - Passes both references into CICD Tests dispatch inputs

- CICD Tests workflow now:
  - Accepts deployable backend tag and digest inputs
  - Propagates these through setup outputs
  - Replaces previous integration lane behavior with runtime black-box execution:
    - Starts isolated Docker network
    - Starts Postgres container
    - Starts backend container from digest-pinned deployable image
    - Enforces tag-to-digest consistency before running checks
    - Runs endpoint checks against live container:
      - GET /
      - GET /compatibility
      - GET /health
    - Captures backend/db logs and container state on failure
    - Cleans up containers and network via trap

- Documentation updated:
  - Runtime contract enforcement section now includes runtime black-box integration checks
  - CI success summary now reflects runtime integration lane behavior

## Scope

Included:
- Backend runtime black-box integration replacement for the existing integration lane
- Digest + tag identity enforcement
- Failure diagnostics for triage

Out of scope:
- Frontend runtime smoke checks
- E2E lane redesign

## Acceptance Criteria Mapping

- Integration tests execute against runtime container endpoints: 
- Integration lane consumes built image references (not source-mounted execution): 
- Failures surface service logs and test logs for triage: 

## Verification

- Workflow files pass local validation checks
- Pre-commit hooks pass on committed changes
- Branch pushed and ready for PR review

## Related

- Issue: #61
- Dependency context: #66

Co-authored-by: copilotcoder <copilotcoder@darkhelm.org>
Reviewed-on: #72
2026-07-05 22:48:57 -04:00

479 lines
18 KiB
YAML

name: CICD Source Checks
on:
workflow_dispatch:
inputs:
head_sha:
description: Commit SHA to process
required: false
source_workflow:
description: Upstream workflow name
required: false
trace_id:
description: Correlation id propagated across CICD dispatch chain
required: false
base_needed:
description: Whether base rebuild is required downstream
required: false
base_hash:
description: Immutable base hash to pass downstream
required: false
env:
GITEA_SSH_HOST: kankali.darkhelm.lan
GITEA_SSH_PORT: "2222"
GITEA_REPO_SSH_URL: ssh://git@kankali.darkhelm.lan:2222/DarkHelm.org/plex-playlist.git
GITEA_REGISTRY: kankali.darkhelm.lan:3001
GITEA_REGISTRY_IP: 10.18.75.2
GITEA_REGISTRY_HOST: kankali.darkhelm.lan
concurrency:
group: source-checks-${{ github.sha }}
cancel-in-progress: true
jobs:
setup:
name: Setup Source Checks Context
runs-on: ubuntu-act
timeout-minutes: 10
outputs:
head_sha: ${{ steps.meta.outputs.head_sha }}
trace_id: ${{ steps.meta.outputs.trace_id }}
base_needed: ${{ steps.meta.outputs.base_needed }}
base_hash: ${{ steps.meta.outputs.base_hash }}
steps:
- name: Identify runner
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Audit trigger context
env:
EVENT_NAME: ${{ github.event_name }}
SOURCE_WORKFLOW: ${{ github.event.inputs.source_workflow }}
HEAD_SHA_INPUT: ${{ github.event.inputs.head_sha }}
HEAD_SHA_FALLBACK: ${{ github.sha }}
BASE_NEEDED_INPUT: ${{ github.event.inputs.base_needed }}
BASE_HASH_INPUT: ${{ github.event.inputs.base_hash }}
REF: ${{ github.ref }}
REF_NAME: ${{ github.ref_name }}
HEAD_REF: ${{ github.head_ref }}
TRACE_ID_INPUT: ${{ github.event.inputs.trace_id }}
run: |
RESOLVED_HEAD_SHA="${HEAD_SHA_INPUT:-${HEAD_SHA_FALLBACK}}"
RESOLVED_BASE_NEEDED="${BASE_NEEDED_INPUT:-true}"
TRACE_ID="${TRACE_ID_INPUT:-cicd-source-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${RESOLVED_HEAD_SHA:0:8}}"
echo "=== Dispatch Audit: CICD Source Checks ==="
echo "event_name=${EVENT_NAME}"
echo "source_workflow=${SOURCE_WORKFLOW}"
echo "head_sha_input=${HEAD_SHA_INPUT}"
echo "head_sha=${RESOLVED_HEAD_SHA}"
echo "base_needed=${RESOLVED_BASE_NEEDED}"
echo "base_hash=${BASE_HASH_INPUT:-deferred}"
echo "ref=${REF}"
echo "ref_name=${REF_NAME}"
echo "head_ref=${HEAD_REF}"
echo "trace_id=${TRACE_ID}"
- name: Resolve source check metadata
id: meta
env:
HEAD_SHA_INPUT: ${{ github.event.inputs.head_sha }}
HEAD_SHA_FALLBACK: ${{ github.sha }}
TRACE_ID_INPUT: ${{ github.event.inputs.trace_id }}
BASE_NEEDED_INPUT: ${{ github.event.inputs.base_needed }}
BASE_HASH_INPUT: ${{ github.event.inputs.base_hash }}
run: |
RESOLVED_HEAD_SHA="${HEAD_SHA_INPUT:-${HEAD_SHA_FALLBACK}}"
RESOLVED_TRACE_ID="${TRACE_ID_INPUT:-cicd-source-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${RESOLVED_HEAD_SHA:0:8}}"
RESOLVED_BASE_NEEDED="${BASE_NEEDED_INPUT:-true}"
RESOLVED_BASE_HASH="${BASE_HASH_INPUT:-deferred}"
echo "head_sha=${RESOLVED_HEAD_SHA}" >> "$GITHUB_OUTPUT"
echo "trace_id=${RESOLVED_TRACE_ID}" >> "$GITHUB_OUTPUT"
echo "base_needed=${RESOLVED_BASE_NEEDED}" >> "$GITHUB_OUTPUT"
echo "base_hash=${RESOLVED_BASE_HASH}" >> "$GITHUB_OUTPUT"
- &failure_diagnostics_step
name: Failure diagnostics
if: failure()
run: |
echo "=== Failure Diagnostics ==="
date -u '+timestamp_utc=%Y-%m-%dT%H:%M:%SZ'
echo "runner_name=${RUNNER_NAME:-unknown}"
echo "runner_hostname=${HOSTNAME:-unknown}"
uname -a || true
cat /etc/os-release 2>/dev/null || true
df -h || true
free -h || true
ps aux --sort=-%mem | head -n 30 || true
if command -v docker >/dev/null 2>&1; then
echo "=== Docker Diagnostics ==="
docker version || true
docker info || true
docker ps -a || true
docker images --digests | head -n 50 || true
else
echo "docker not available on this runner"
fi
echo "=== Kernel Tail ==="
dmesg | tail -n 120 || true
source-precommit-checks-backend:
name: Source Pre-commit Checks (backend)
runs-on: ubuntu-act
timeout-minutes: 40
needs: setup
steps:
- name: Identify runner
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Configure registry host resolution
run: |
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
- name: Checkout source snapshot
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
HEAD_SHA: ${{ needs.setup.outputs.head_sha }}
run: |
set -e
umask 022
trap 'rm -f ~/.ssh/id_rsa' EXIT
mkdir -p ~/.ssh
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -p "${GITEA_SSH_PORT}" "${GITEA_SSH_HOST}" >> ~/.ssh/known_hosts 2>/dev/null
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git clone --depth 1 --no-checkout "${GITEA_REPO_SSH_URL}" .
if GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git fetch --depth 1 origin "${HEAD_SHA}" >/dev/null 2>&1; then
git checkout FETCH_HEAD -- .
echo "Using fetched HEAD_SHA checkout: ${HEAD_SHA}"
else
git checkout HEAD -- .
echo "Falling back to default branch HEAD for source checks checkout"
fi
test -f .pre-commit-config.yaml || {
echo "❌ Missing .pre-commit-config.yaml after checkout"
exit 1
}
test -f backend/pyproject.toml || {
echo "❌ Missing backend/pyproject.toml after checkout"
exit 1
}
test -f frontend/package.json || {
echo "❌ Missing frontend/package.json after checkout"
find . -maxdepth 3 -type f | sort | head -n 80
exit 1
}
- name: Bootstrap backend toolchain
run: |
set -euo pipefail
if ! command -v curl >/dev/null 2>&1; then
apt-get update -qq
apt-get install -y -qq curl ca-certificates
fi
if ! command -v uv >/dev/null 2>&1; then
curl -LsSf https://astral.sh/uv/install.sh | sh
fi
export PATH="$HOME/.local/bin:$PATH"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Install backend dependencies
run: |
set -euo pipefail
export PATH="$HOME/.local/bin:$PATH"
export UV_LINK_MODE=copy
cd backend
uv sync --dev
- name: Run backend pre-commit source checks
env:
CI: "true"
SKIP: "eslint,prettier,typescript-check,tsdoc-lint"
run: |
set -euo pipefail
export PATH="$HOME/.local/bin:$PATH"
cd backend
uv run pre-commit run --all-files --show-diff-on-failure --config ../.pre-commit-config.yaml
- *failure_diagnostics_step
source-precommit-checks-frontend:
name: Source Pre-commit Checks (frontend)
runs-on: ubuntu-act
timeout-minutes: 40
needs: setup
steps:
- name: Identify runner
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Configure registry host resolution
run: |
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
- name: Checkout source snapshot
env:
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
HEAD_SHA: ${{ needs.setup.outputs.head_sha }}
run: |
set -e
umask 022
trap 'rm -f ~/.ssh/id_rsa' EXIT
mkdir -p ~/.ssh
echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan -p "${GITEA_SSH_PORT}" "${GITEA_SSH_HOST}" >> ~/.ssh/known_hosts 2>/dev/null
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git clone --depth 1 --no-checkout "${GITEA_REPO_SSH_URL}" .
if GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa -o IdentitiesOnly=yes -o StrictHostKeyChecking=no" \
git fetch --depth 1 origin "${HEAD_SHA}" >/dev/null 2>&1; then
git checkout FETCH_HEAD -- .
echo "Using fetched HEAD_SHA checkout: ${HEAD_SHA}"
else
git checkout HEAD -- .
echo "Falling back to default branch HEAD for source checks checkout"
fi
test -f .pre-commit-config.yaml || {
echo "❌ Missing .pre-commit-config.yaml after checkout"
exit 1
}
test -f frontend/package.json || {
echo "❌ Missing frontend/package.json after checkout"
find . -maxdepth 3 -type f | sort | head -n 80
exit 1
}
- name: Bootstrap frontend toolchain
run: |
set -euo pipefail
if ! command -v curl >/dev/null 2>&1; then
apt-get update -qq
apt-get install -y -qq curl ca-certificates
fi
if ! command -v uv >/dev/null 2>&1; then
curl -LsSf https://astral.sh/uv/install.sh | sh
fi
export PATH="$HOME/.local/bin:$PATH"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
if ! command -v node >/dev/null 2>&1; then
apt-get update -qq
apt-get install -y -qq nodejs npm
fi
if ! command -v corepack >/dev/null 2>&1; then
npm install -g corepack
fi
corepack enable
- name: Install frontend dependencies
run: |
set -euo pipefail
export NODE_OPTIONS="--max-old-space-size=512"
export YARN_NETWORK_CONCURRENCY=1
cd frontend
if ! yarn install --immutable --mode=skip-build; then
echo "Yarn install failed; retrying with npm fallback for constrained runner memory"
rm -f package-lock.json
npm install --ignore-scripts --no-audit --no-fund --prefer-offline
fi
- name: Run frontend pre-commit source checks
env:
CI: "true"
run: |
set -euo pipefail
cd frontend
corepack yarn eslint . --max-warnings=0
corepack yarn vue-tsc --noEmit
corepack yarn prettier --check .
- *failure_diagnostics_step
dispatch-build:
name: Dispatch Downstream Build
runs-on: ubuntu-act
timeout-minutes: 10
needs: [setup, source-precommit-checks-backend, source-precommit-checks-frontend]
steps:
- name: Identify runner
run: |
echo "=== Runner Identity ==="
echo "runner_name=${RUNNER_NAME:-}"
echo "runner_name_hint=${GITEA_RUNNER_NAME:-${ACT_RUNNER_NAME:-${RUNNER_NAME:-unknown}}}"
echo "runner_hostname_env=${HOSTNAME:-unknown}"
echo "runner_uname_n=$(uname -n 2>/dev/null || echo unknown)"
echo "runner_etc_hostname=$(cat /etc/hostname 2>/dev/null || echo unknown)"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "timestamp_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
- name: Configure registry host resolution
run: |
if ! grep -q "${GITEA_REGISTRY_HOST}" /etc/hosts; then
echo "${GITEA_REGISTRY_IP} ${GITEA_REGISTRY_HOST}" >> /etc/hosts
fi
- name: Dispatch downstream workflow
env:
ACTIONS_TRIGGER_TOKEN: ${{ secrets.ACTIONS_TRIGGER_TOKEN }}
PACKAGE_ACCESS_TOKEN: ${{ secrets.PACKAGE_ACCESS_TOKEN }}
BASE_NEEDED: ${{ needs.setup.outputs.base_needed }}
BASE_HASH: ${{ needs.setup.outputs.base_hash }}
HEAD_SHA: ${{ needs.setup.outputs.head_sha }}
TRACE_ID: ${{ needs.setup.outputs.trace_id }}
REPO_FULL: ${{ github.repository }}
HEAD_REF: ${{ github.head_ref }}
REF_NAME: ${{ github.ref_name }}
run: |
set -e
DISPATCH_TOKEN="${ACTIONS_TRIGGER_TOKEN:-${PACKAGE_ACCESS_TOKEN:-}}"
if [ -z "${DISPATCH_TOKEN}" ]; then
echo "❌ Missing dispatch token. Set ACTIONS_TRIGGER_TOKEN (repo write scope) or ensure PACKAGE_ACCESS_TOKEN has Actions workflow-dispatch permissions."
exit 1
fi
REPO_OWNER="${REPO_FULL%/*}"
REPO_NAME="${REPO_FULL#*/}"
TARGET_REF="${HEAD_REF:-${REF_NAME}}"
if [ "${BASE_NEEDED}" = "true" ]; then
TARGET_WORKFLOW="docker-build-base.yaml"
else
TARGET_WORKFLOW="docker-build-main.yaml"
fi
echo "route_decision workflow=${TARGET_WORKFLOW} head_sha=${HEAD_SHA} base_needed=${BASE_NEEDED} trace_id=${TRACE_ID}"
CANDIDATE_API_BASES=()
if [ -n "${GITHUB_SERVER_URL:-}" ]; then
CANDIDATE_API_BASES+=("${GITHUB_SERVER_URL%/}/api/v1")
fi
if [ -n "${GITEA_SSH_HOST:-}" ]; then
CANDIDATE_API_BASES+=("http://${GITEA_SSH_HOST}:3001/api/v1")
fi
if [ -n "${GITEA_REGISTRY_HOST:-}" ]; then
CANDIDATE_API_BASES+=("http://${GITEA_REGISTRY_HOST}:3001/api/v1")
fi
if [ -n "${GITEA_REGISTRY_IP:-}" ]; then
CANDIDATE_API_BASES+=("http://${GITEA_REGISTRY_IP}:3001/api/v1")
fi
ensure_curl() {
if command -v curl >/dev/null 2>&1; then
return 0
fi
if command -v apt-get >/dev/null 2>&1; then
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq curl ca-certificates
fi
if command -v curl >/dev/null 2>&1; then
return 0
fi
echo "❌ curl is required for dispatch and could not be installed"
return 1
}
ensure_curl
HELPER_PATH="/tmp/dispatch-workflow.sh"
fetch_dispatch_helper() {
local helper_ref="$1"
local api_base
for api_base in "${CANDIDATE_API_BASES[@]}"; do
helper_url="${api_base}/repos/${REPO_OWNER}/${REPO_NAME}/raw/scripts/dispatch-workflow.sh?ref=${helper_ref}"
if curl -fsS --connect-timeout 5 --max-time 20 \
-H "Authorization: token ${DISPATCH_TOKEN}" \
-H "User-Agent: plex-playlist-cicd-source-checks" \
-o "${HELPER_PATH}" \
"${helper_url}"; then
chmod +x "${HELPER_PATH}"
return 0
fi
done
return 1
}
if ! fetch_dispatch_helper "${TARGET_REF}" && ! fetch_dispatch_helper "${HEAD_SHA}"; then
echo "❌ Failed to fetch scripts/dispatch-workflow.sh from repository"
exit 1
fi
DISPATCH_ARGS=(
--token "${DISPATCH_TOKEN}"
--repo "${REPO_FULL}"
--workflow "${TARGET_WORKFLOW}"
--ref "${TARGET_REF}"
--head-sha "${HEAD_SHA}"
--source-workflow "CICD Source Checks"
--trace-id "${TRACE_ID}"
--base-needed "${BASE_NEEDED}"
--base-hash "${BASE_HASH}"
)
for API_BASE in "${CANDIDATE_API_BASES[@]}"; do
DISPATCH_ARGS+=(--api-base "${API_BASE}")
done
echo "✅ Source checks passed; dispatching ${TARGET_WORKFLOW}"
"${HELPER_PATH}" "${DISPATCH_ARGS[@]}"
- *failure_diagnostics_step