Optimizing the build so that CICD doesn't take FOREVER to run.

Signed-off-by: Cliff Hill <xlorep@darkhelm.org>

docs/CICD_MULTI_STAGE_BUILD.md

@@ -0,0 +1,184 @@
+# CI/CD Multi-Stage Build Architecture
+
+## Overview
+
+This project uses a two-stage Docker build approach to optimize CI/CD performance by separating stable base dependencies from project-specific code and dependencies.
+
+## Architecture
+
+### Stage 1: Base Image (`Dockerfile.cicd-base`)
+**Purpose**: Contains all system dependencies and language runtimes that change infrequently.
+
+**Contents**:
+- Ubuntu 22.04 base system
+- Python 3.13 with development tools
+- Node.js 24 with npm/yarn
+- Yarn Berry (v4) configured for CI
+- System build tools (build-essential, git, curl, etc.)
+- UV package manager for Python
+- SSH helper scripts for git operations
+
+**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd-base:latest`
+
+**Rebuild Triggers**: Only when `Dockerfile.cicd-base` changes (detected via SHA256 hash)
+
+### Stage 2: Complete Image (`Dockerfile.cicd`)
+**Purpose**: Inherits from base and adds project code and dependencies.
+
+**Contents**:
+- Project source code (cloned via SSH)
+- Backend Python dependencies (`uv sync --dev`)
+- Frontend Node.js dependencies (`yarn install`)
+- Playwright browsers for E2E testing
+- Pre-commit hook environments
+- All project-specific tooling verification
+
+**Registry**: `dogar.darkhelm.org/darkhelm.org/plex-playlist/cicd:latest`
+
+**Rebuild Triggers**: Every CI/CD run (contains project-specific code and dependencies)
+
+## Performance Benefits
+
+### Before Multi-Stage
+- Single monolithic build: ~15-25 minutes on Raspberry Pi 4GB workers
+- Full system dependency installation every time
+- No caching of expensive operations (Python compilation, Node.js setup)
+
+### After Multi-Stage
+- Base image build: ~10-15 minutes (only when base changes)
+- Complete image build: ~5-10 minutes (reuses cached base)
+- **Typical CI run**: ~5-10 minutes (90% of runs use cached base)
+
+### Caching Strategy
+1. **Docker Layer Caching**: Docker automatically caches unchanged layers
+2. **Registry Caching**: Base image pulled from registry if available
+3. **Hash-Based Invalidation**: Base image tagged with Dockerfile hash
+4. **Conditional Building**: Base only rebuilds when `Dockerfile.cicd-base` changes
+
+## CI/CD Workflow
+
+```yaml
+jobs:
+  setup-base:
+    name: Build and Push CICD Base Image
+    steps:
+      - name: Check if base image needs rebuilding
+        # Calculates SHA256 of Dockerfile.cicd-base
+        # Pulls existing image with hash tag if available
+        # Sets needs_build=false if image exists
+
+      - name: Build and push base image
+        if: needs_build == 'true'
+        # Only runs when base Dockerfile changed
+        # Tags with both hash and 'latest'
+
+  setup:
+    name: Build and Push CICD Complete Image
+    needs: setup-base
+    steps:
+      - name: Build and push complete CICD image
+        # Always runs, inherits from base:latest
+        # Contains project code and dependencies
+```
+
+## Local Development
+
+### Building Base Image
+```bash
+# Build base image locally
+docker build -f Dockerfile.cicd-base -t cicd-base:local .
+
+# Test base image
+docker run -it cicd-base:local bash
+```
+
+### Building Complete Image
+```bash
+# Build complete image (requires SSH access to git repo)
+export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)"
+echo "$SSH_PRIVATE_KEY" > /tmp/ssh_key
+chmod 600 /tmp/ssh_key
+
+docker build -f Dockerfile.cicd \
+  --secret id=ssh_private_key,src=/tmp/ssh_key \
+  --build-arg CICD_BASE_IMAGE="cicd-base:local" \
+  --build-arg GITHUB_SHA="$(git rev-parse HEAD)" \
+  -t cicd:local .
+
+rm /tmp/ssh_key
+```
+
+### Using Local Build Script
+```bash
+# Use the provided build script
+./scripts/build-cicd-local.sh
+```
+
+## Memory Optimization
+
+### Raspberry Pi 4GB Constraints
+- **Swap File**: 1GB temporary swap during yarn install
+- **Node.js Memory**: Limited to 1024MB (`--max-old-space-size=1024`)
+- **UV Workers**: Single-threaded Python package installation
+- **Graceful Degradation**: Frontend dependencies optional in constrained environments
+
+### Frontend Dependency Handling
+```dockerfile
+# Conservative installation with fallback
+RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
+    INSTALL_SUCCESS=false && \
+    for i in 1 2 3; do \
+        yarn install --immutable --mode=skip-build && \
+        { INSTALL_SUCCESS=true; break; } || \
+        (echo "Retrying..." && sleep 60); \
+    done && \
+    if [ "$INSTALL_SUCCESS" = "false" ]; then \
+        touch .frontend-deps-failed; \
+    fi
+```
+
+## Monitoring and Debugging
+
+### Build Time Tracking
+- Base image builds logged with timing information
+- Hash-based cache hit/miss tracking
+- Registry pull vs build decision logging
+
+### Troubleshooting
+1. **Base Image Issues**: Check `Dockerfile.cicd-base` syntax and system dependencies
+2. **Complete Image Issues**: Usually project dependency or SSH access problems
+3. **Cache Misses**: Verify registry connectivity and hash calculation
+4. **Memory Issues**: Check swap setup and Node.js memory limits
+
+### Common Issues
+- **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured
+- **Registry Authentication**: Verify PACKAGE_ACCESS_TOKEN permissions
+- **Memory Constraints**: Monitor swap usage on Raspberry Pi workers
+- **Network Timeouts**: Retry mechanisms handle transient failures
+
+## Migration Path
+
+### From Single-Stage Build
+1. **Phase 1**: Deploy both Dockerfiles, workflow uses old single-stage
+2. **Phase 2**: Switch workflow to use multi-stage (this deployment)
+3. **Phase 3**: Remove old `Dockerfile.cicd.old` after successful runs
+
+### Rollback Strategy
+If issues arise, revert workflow to use single-stage:
+```yaml
+# Emergency rollback: use old Dockerfile directly
+docker build -f Dockerfile.cicd.old -t cicd:latest .
+```
+
+## Future Enhancements
+
+### Potential Optimizations
+1. **Dependency Caching**: Pre-install common Python/Node packages in base
+2. **Multi-Architecture**: ARM64 native builds for Raspberry Pi
+3. **Parallel Builds**: Build base and project dependencies in parallel
+4. **Smart Invalidation**: More granular dependency change detection
+
+### Monitoring Additions
+1. **Build Time Metrics**: Track cache hit rates and build duration
+2. **Registry Usage**: Monitor storage and bandwidth usage
+3. **Worker Performance**: Profile builds across different runner types

docs/DEVELOPMENT.md

@@ -12,6 +12,12 @@ This document outlines how to set up your development environment and work with
 6. [CI/CD Pipeline](#cicd-pipeline)
 7. [Branch Protection and Merge Requirements](#branch-protection-and-merge-requirements)
 
+## Related Documentation
+
+- **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
+- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
+- **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
+
 ## Quick Start
 
 ```bash
@@ -274,22 +280,40 @@ pre-commit run end-of-file-fixer --all-files
 
 ### Pipeline Overview
 
-The CI/CD pipeline runs automatically on:
+The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance:
 
+- **Stage 1**: Build base image (system dependencies, Python, Node.js) - **cached across runs**
+- **Stage 2**: Build complete image (project code and dependencies) - **rebuilt every time**
+
+Pipeline triggers:
 - Push to any branch
 - Pull requests to `main` or `develop`
 
+### Multi-Stage Build Benefits
+
+**Performance Gains**:
+- Base image cached when `Dockerfile.cicd-base` unchanged (~90% of runs)
+- Typical build time reduced from 15-25 minutes to 5-10 minutes
+- Raspberry Pi 4GB workers can efficiently handle builds
+
+**Architecture**:
+- `cicd-base:latest` - System dependencies (Python 3.13, Node.js 24, build tools)
+- `cicd:latest` - Complete environment (project code + dependencies)
+
+For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
+
 ### Pipeline Jobs
 
-All jobs run in parallel after the setup phase:
+All jobs run in parallel after the setup phases:
 
-1. **Setup**: Builds and pushes the CI/CD Docker image
-2. **Code Quality**:
+1. **Setup Base**: Builds and pushes base Docker image (conditional)
+2. **Setup Complete**: Builds and pushes complete CI/CD Docker image
+3. **Code Quality**:
    - Trailing whitespace check
    - End-of-file formatting
    - YAML syntax validation
    - TOML syntax validation
-3. **Backend Validation**:
+4. **Backend Validation**:
    - Ruff formatting check
    - Ruff linting
    - Pyright type checking
@@ -297,18 +321,42 @@ All jobs run in parallel after the setup phase:
    - Unit tests with coverage
    - Integration tests
    - Doctests (xdoctest)
-4. **Frontend Validation**:
+5. **Frontend Validation**:
    - Prettier formatting check
    - ESLint linting
    - TypeScript compilation
    - Unit tests with coverage
    - E2E tests (Playwright)
 
+### Local CI/CD Testing
+
+Build and test CI/CD images locally:
+
+```bash
+# Build both base and complete images
+./scripts/build-cicd-local.sh
+
+# Build only base image
+./scripts/build-cicd-local.sh --base-only
+
+# Build only complete image (requires existing base)
+./scripts/build-cicd-local.sh --complete-only
+
+# Force rebuild with no cache
+./scripts/build-cicd-local.sh --force --no-cache
+
+# Test with custom SSH key
+./scripts/build-cicd-local.sh --ssh-key ~/.ssh/custom_key
+```
+
 ### CI/CD Design Principles
 
+- **Multi-Stage Optimization**: Separate stable dependencies from project code
+- **Intelligent Caching**: Base image cached when unchanged (hash-based detection)
 - **Single Source of Truth**: All CI jobs use the same pre-commit hooks as local development
 - **Parallel Execution**: Maximum efficiency with concurrent job execution
 - **Fast Feedback**: Jobs fail fast on first error
+- **Memory Efficiency**: Optimized for 4GB Raspberry Pi workers
 - **Comprehensive Coverage**: Every aspect of code quality is validated
 
 ### Viewing CI Results