feature/pp-58-runtime-image-contract (#68)
## Summary This PR tightens repository quality enforcement around markdown and documentation. It adds `markdownlint` to the `cicd-checks` workflow, expands pre-commit coverage so markdown files are checked repo-wide, and cleans up the PP-58 documentation set to keep it aligned with the new policy. ## What changed - Added a `Markdownlint Check` entry to `.gitea/workflows/cicd-checks.yaml` - Added `markdownlint` to pre-commit and widened prettier coverage to include markdown files across the repo - Updated `README.md` to satisfy markdownlint line-length rules - Normalized the PP-58 documentation set: - `docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - `docs/adr/ADR003-deployable_runtime_image_contract.md` - `docs/DEVELOPMENT.md` - `docs/CICD_MULTI_STAGE_BUILD.md` - `docs/CICD_TROUBLESHOOTING_GUIDE.md` - `docs/SECURE_DOCKER_CICD.md` ## Validation - `pre-commit run markdownlint --files README.md docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - `pre-commit run prettier --files README.md docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - Workflow YAML validation returned no errors ## Notes This change does not alter application runtime behavior. It only strengthens CI and documentation quality enforcement. Co-authored-by: copilotcoder <copilotcoder@darkhelm.org> Reviewed-on: #68
This commit was merged in pull request #68.
This commit is contained in:
@@ -7,9 +7,11 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
## Architecture
|
||||
|
||||
### Stage 1: Base Image (`Dockerfile.cicd-base`)
|
||||
|
||||
**Purpose**: Contains all system dependencies and language runtimes that change infrequently.
|
||||
|
||||
**Contents**:
|
||||
|
||||
- Ubuntu 22.04 base system
|
||||
- Python 3.14 with development tools
|
||||
- Node.js 24 with npm/yarn
|
||||
@@ -22,15 +24,18 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
- SSH helper scripts for git operations
|
||||
|
||||
**Registry**:
|
||||
|
||||
- Immutable: `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:<hash>`
|
||||
- Convenience: `kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:latest`
|
||||
|
||||
**Rebuild Triggers**: Only when `Dockerfile.cicd-base`, `.dockerignore`, or the shared hash helper changes
|
||||
|
||||
### Stage 2: Complete Image (`Dockerfile.cicd`)
|
||||
|
||||
**Purpose**: Inherits from base and adds project code and dependencies.
|
||||
|
||||
**Contents**:
|
||||
|
||||
- Project source code (cloned via SSH)
|
||||
- **Optimized backend dependencies** (leverages pre-installed dev tools)
|
||||
- **Optimized frontend dependencies** (leverages global TypeScript, ESLint, etc.)
|
||||
@@ -45,6 +50,7 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
## Performance Benefits
|
||||
|
||||
### Before Multi-Stage Optimization
|
||||
|
||||
- Single monolithic build: ~15-25 minutes on Raspberry Pi 4GB workers
|
||||
- Full system dependency installation every time
|
||||
- No caching of expensive operations (Python compilation, Node.js setup)
|
||||
@@ -52,6 +58,7 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
- Common dev tools (ruff, pyright, eslint, typescript) compiled from source each time
|
||||
|
||||
### After Multi-Stage Optimization (✅ **VALIDATED SUCCESSFUL**)
|
||||
|
||||
- **Complete CI/CD pipeline: ~3-5 minutes** (85% improvement!)
|
||||
- Base image cached and reused across builds
|
||||
- Pre-installed development tools eliminate compilation overhead
|
||||
@@ -62,7 +69,9 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
## Advanced Optimizations in Base Image
|
||||
|
||||
### Pre-installed Development Tools
|
||||
|
||||
**Python Tools** (cached in `/opt/python-dev-tools/`):
|
||||
|
||||
- `ruff` - Fast Python linter/formatter
|
||||
- `pyright` - Python type checker
|
||||
- `pytest` + plugins - Testing framework
|
||||
@@ -70,6 +79,7 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
- `yamllint`, `toml-sort` - Configuration file tools
|
||||
|
||||
**Node.js Tools** (installed globally via npm):
|
||||
|
||||
- `@playwright/test` - Playwright testing framework
|
||||
- `typescript` - TypeScript compiler
|
||||
- `eslint` - JavaScript/TypeScript linter
|
||||
@@ -82,12 +92,14 @@ This project uses a two-stage Docker build approach to optimize CI/CD performanc
|
||||
- **Build Reliability**: Stable tool versions cached in base
|
||||
|
||||
### After Multi-Stage (Fully Optimized)
|
||||
|
||||
- Base image build: ~20-25 minutes (only when base changes, includes browsers + dev tools)
|
||||
- Complete image build: ~2-3 minutes (reuses cached base with everything pre-installed)
|
||||
- **Typical CI run**: ~2-3 minutes (98% of runs use fully cached base)
|
||||
- **Major wins**: No browser downloads (~400MB), no dev tool compilation, faster dependency resolution
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
1. **Docker Layer Caching**: Docker automatically caches unchanged layers
|
||||
2. **Registry Caching**: Base image is built once and then pulled by all runners
|
||||
3. **Hash-Based Invalidation**: Base image tagged with a shared helper-derived hash
|
||||
@@ -129,13 +141,17 @@ jobs:
|
||||
```
|
||||
|
||||
### Responsibility Split
|
||||
- `.gitea/workflows/cicd-base.yml` owns base publication and verification.
|
||||
- `.gitea/workflows/cicd.yml` owns complete-image build, tests, and deployment checks.
|
||||
|
||||
- `.gitea/workflows/docker-build-base.yaml` owns base publication and verification.
|
||||
- `.gitea/workflows/docker-build-main.yaml` owns complete-image publication.
|
||||
- `.gitea/workflows/cicd-start.yaml`, `.gitea/workflows/cicd-checks.yaml`, and
|
||||
`.gitea/workflows/cicd-tests.yaml` own CI validation and checks.
|
||||
- Main CI never rebuilds the base image locally.
|
||||
|
||||
## Local Development
|
||||
|
||||
### Building Base Image
|
||||
|
||||
```bash
|
||||
# Build base image locally
|
||||
docker build -f Dockerfile.cicd-base -t cicd-base:local .
|
||||
@@ -145,6 +161,7 @@ docker run -it cicd-base:local bash
|
||||
```
|
||||
|
||||
### Building Complete Image
|
||||
|
||||
```bash
|
||||
# Build complete image (requires SSH access to git repo)
|
||||
export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_rsa)"
|
||||
@@ -161,6 +178,7 @@ rm /tmp/ssh_key
|
||||
```
|
||||
|
||||
### Using Local Build Script
|
||||
|
||||
```bash
|
||||
# Use the provided build script
|
||||
./scripts/build-cicd-local.sh
|
||||
@@ -172,12 +190,14 @@ base image matches the immutable tag CI expects.
|
||||
## Memory Optimization
|
||||
|
||||
### Raspberry Pi 4GB Constraints
|
||||
|
||||
- **Swap File**: 1GB temporary swap during yarn install
|
||||
- **Node.js Memory**: Limited to 1024MB (`--max-old-space-size=1024`)
|
||||
- **UV Workers**: Single-threaded Python package installation
|
||||
- **Graceful Degradation**: Frontend dependencies optional in constrained environments
|
||||
|
||||
### Frontend Dependency Handling
|
||||
|
||||
```dockerfile
|
||||
# Conservative installation with fallback
|
||||
RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
@@ -195,23 +215,27 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
## Monitoring and Debugging
|
||||
|
||||
### Build Time Tracking
|
||||
|
||||
- Base image builds logged with timing information
|
||||
- Hash-based cache hit/miss tracking
|
||||
- Registry pull vs build decision logging
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
1. **Base Image Issues**: Check `Dockerfile.cicd-base` syntax and system dependencies
|
||||
2. **Complete Image Issues**: Usually project dependency or SSH access problems
|
||||
3. **Cache Misses**: Verify registry connectivity and the shared base hash calculation
|
||||
4. **Memory Issues**: Check swap setup and Node.js memory limits
|
||||
|
||||
### Missing Immutable Base Tag
|
||||
|
||||
- Symptom: main CI fails with `Required immutable base image is not available`
|
||||
- Cause: the expected `cicd-base:<hash>` has not been published yet
|
||||
- Fix: run or rerun the `CICD Base Image` workflow, or wait for it to finish when a PR changes base inputs
|
||||
- Design note: main CI intentionally fails instead of rebuilding the base locally
|
||||
|
||||
### Common Issues
|
||||
|
||||
- **SSH Key Problems**: Ensure SSH_PRIVATE_KEY secret is properly configured
|
||||
- **Registry Authentication**: Verify PACKAGE_ACCESS_TOKEN permissions
|
||||
- **Memory Constraints**: Monitor swap usage on Raspberry Pi workers
|
||||
@@ -220,9 +244,11 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
#### Base Image Optimization Issues
|
||||
|
||||
**Missing `/opt/python-dev-tools/` (Oct 2025 Resolution)**:
|
||||
|
||||
- **Symptom**: Build fails with `No virtual environment or system Python installation found for path /opt/python-dev-tools/bin/python`
|
||||
- **Cause**: Base image in registry doesn't contain pre-installed Python dev tools optimization
|
||||
- **Fix Applied**: Made complete image resilient to missing optimization
|
||||
|
||||
```dockerfile
|
||||
# In Dockerfile.cicd - now handles missing pre-installed tools gracefully
|
||||
if [ -f "/opt/python-dev-tools/bin/python" ]; then
|
||||
@@ -231,10 +257,12 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
echo "⚠ Pre-installed Python dev tools not found - fresh installation"
|
||||
fi
|
||||
```
|
||||
|
||||
- **Impact**: Builds continue successfully but without optimization benefits (~20s longer)
|
||||
- **Long-term Solution**: Rebuild base image to restore `/opt/python-dev-tools/` optimization
|
||||
|
||||
**Playwright E2E Test Failures (Oct 2025 Resolution)**:
|
||||
|
||||
- **Symptom**: `error: unknown option '--headed=false'` during E2E test execution
|
||||
- **Cause**: Invalid Playwright CLI flag syntax in workflow and documentation
|
||||
- **Fix Applied**:
|
||||
@@ -244,6 +272,7 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
- **Key Learning**: Use yarn scripts (`yarn test:e2e`) rather than direct Playwright CLI calls
|
||||
|
||||
**Missing Playwright Browser Binaries (Nov 2025 Resolution)**:
|
||||
|
||||
- **Symptom**: `Executable doesn't exist at /root/.cache/ms-playwright/chromium_headless_shell-*/` for all browsers
|
||||
- **Cause**: Base image browsers not properly cached or registry image outdated
|
||||
- **Fix Applied**: Added `yarn playwright install --with-deps` step before running E2E tests in CI
|
||||
@@ -252,6 +281,7 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
- **Long-term Solution**: Rebuild base image to restore Playwright browser caching
|
||||
|
||||
**Firefox/WebKit Browser Compatibility in Docker CI (Nov 2025 Resolution)**:
|
||||
|
||||
- **Symptom**: Firefox sandbox/timeout errors, WebKit content loading failures in Docker environment
|
||||
- **Root Cause**: Firefox requires special sandbox configuration, WebKit has timing issues in headless Docker
|
||||
- **Fix Applied**: CI now runs only Chromium browser (most reliable), all browsers available locally
|
||||
@@ -260,6 +290,7 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
- **Coverage**: Chromium provides excellent coverage as it's most widely used browser engine
|
||||
|
||||
**Network Instability Resilience (Nov 2025 Enhancement)**:
|
||||
|
||||
- **Problem**: CI environment has unstable network causing Docker registry timeouts, image pull failures
|
||||
- **Solutions Applied**:
|
||||
- **Docker Login Retry**: 5 attempts with 15s intervals, 60s timeout per attempt
|
||||
@@ -314,6 +345,7 @@ RUN export NODE_OPTIONS="--max-old-space-size=1024" && \
|
||||
**Decision**: Install dependencies before cloning full source code
|
||||
|
||||
**Rationale**:
|
||||
|
||||
- Dependencies change less frequently than source code (~5% vs 95% of commits)
|
||||
- Docker layer caching works best with stable, early layers
|
||||
- Separation allows independent cache invalidation
|
||||
@@ -332,6 +364,7 @@ RUN git clone full_repo && merge_with_dependencies
|
||||
```
|
||||
|
||||
**Trade-offs**:
|
||||
|
||||
- ✅ 85% faster typical builds (3-5min vs 15-20min)
|
||||
- ✅ Better resource utilization (RPi 4GB workers)
|
||||
- ❌ More complex Dockerfile logic
|
||||
@@ -342,6 +375,7 @@ RUN git clone full_repo && merge_with_dependencies
|
||||
**Decision**: Run E2E tests only with Chromium in CI, all browsers locally
|
||||
|
||||
**Rationale**:
|
||||
|
||||
- Firefox sandbox issues in Docker environment require complex configuration
|
||||
- WebKit has timing/content loading issues in headless Docker
|
||||
- Chromium is most stable and widely-used browser engine
|
||||
@@ -352,11 +386,12 @@ RUN git clone full_repo && merge_with_dependencies
|
||||
```typescript
|
||||
// playwright.config.ts - Conditional browser setup
|
||||
const projects = process.env.CI
|
||||
? [{ name: 'chromium', use: devices['Desktop Chrome'] }]
|
||||
: [chromium, firefox, webkit]; // Full coverage locally
|
||||
? [{ name: "chromium", use: devices["Desktop Chrome"] }]
|
||||
: [chromium, firefox, webkit]; // Full coverage locally
|
||||
```
|
||||
|
||||
**Trade-offs**:
|
||||
|
||||
- ✅ Reliable CI runs (100% success rate vs 60% with multi-browser)
|
||||
- ✅ Faster CI execution (single browser vs three)
|
||||
- ✅ Simpler Docker configuration
|
||||
@@ -367,6 +402,7 @@ const projects = process.env.CI
|
||||
**Decision**: Implement comprehensive retry logic for all network operations
|
||||
|
||||
**Rationale**:
|
||||
|
||||
- Self-hosted CI environment has intermittent network instability
|
||||
- Docker registry operations are critical path failures
|
||||
- Playwright browser downloads are large and failure-prone
|
||||
@@ -382,6 +418,7 @@ done
|
||||
```
|
||||
|
||||
**Coverage**:
|
||||
|
||||
- Docker login/pull operations (5 attempts, 15-60s intervals)
|
||||
- Playwright browser installs (3 attempts, 30s intervals)
|
||||
- E2E navigation (built-in retry with network error filtering)
|
||||
@@ -389,12 +426,15 @@ done
|
||||
## Migration Path
|
||||
|
||||
### From Single-Stage Build
|
||||
|
||||
1. **Phase 1**: Deploy both Dockerfiles, workflow uses old single-stage
|
||||
2. **Phase 2**: Switch workflow to use multi-stage (this deployment)
|
||||
3. **Phase 3**: Remove old `Dockerfile.cicd.old` after successful runs
|
||||
|
||||
### Rollback Strategy
|
||||
|
||||
If issues arise, revert workflow to use single-stage:
|
||||
|
||||
```yaml
|
||||
# Emergency rollback: use old Dockerfile directly
|
||||
docker build -f Dockerfile.cicd.old -t cicd:latest .
|
||||
@@ -403,12 +443,14 @@ docker build -f Dockerfile.cicd.old -t cicd:latest .
|
||||
## Future Enhancements
|
||||
|
||||
### Potential Optimizations
|
||||
|
||||
1. **Dependency Caching**: Pre-install common Python/Node packages in base
|
||||
2. **Multi-Architecture**: ARM64 native builds for Raspberry Pi
|
||||
3. **Parallel Builds**: Build base and project dependencies in parallel
|
||||
4. **Smart Invalidation**: More granular dependency change detection
|
||||
|
||||
### Monitoring Additions
|
||||
|
||||
1. **Build Time Metrics**: Track cache hit rates and build duration
|
||||
2. **Registry Usage**: Monitor storage and bandwidth usage
|
||||
3. **Worker Performance**: Profile builds across different runner types
|
||||
|
||||
@@ -6,13 +6,13 @@
|
||||
|
||||
## 📊 **Performance Metrics - Validated Results**
|
||||
|
||||
| Metric | Before Optimization | After Optimization | Improvement |
|
||||
|--------|-------------------|------------------|------------|
|
||||
| **Total Pipeline Time** | 15-25 minutes | 3-5 minutes | **85% faster** |
|
||||
| **Build Success Rate** | ~70% (various failures) | **100%** | **30% improvement** |
|
||||
| **E2E Test Reliability** | ~60% (browser issues) | **100%** | **40% improvement** |
|
||||
| **Resource Efficiency** | High CPU/memory load | Optimized usage | **Significant** |
|
||||
| **Developer Experience** | Frequent CI failures | Reliable pipeline | **Excellent** |
|
||||
| Metric | Before Optimization | After Optimization | Improvement |
|
||||
| ------------------------ | ----------------------- | ------------------ | ------------------- |
|
||||
| **Total Pipeline Time** | 15-25 minutes | 3-5 minutes | **85% faster** |
|
||||
| **Build Success Rate** | ~70% (various failures) | **100%** | **30% improvement** |
|
||||
| **E2E Test Reliability** | ~60% (browser issues) | **100%** | **40% improvement** |
|
||||
| **Resource Efficiency** | High CPU/memory load | Optimized usage | **Significant** |
|
||||
| **Developer Experience** | Frequent CI failures | Reliable pipeline | **Excellent** |
|
||||
|
||||
## 🔧 **Key Technical Achievements**
|
||||
|
||||
@@ -37,6 +37,7 @@
|
||||
## 🛠️ **Critical Issues Resolved**
|
||||
|
||||
### **Build Phase Issues**
|
||||
|
||||
1. **✅ README.md Dependency Error**
|
||||
- **Problem**: Local package build failed during dependency-only phase
|
||||
- **Solution**: Dummy file creation for minimal package structure
|
||||
@@ -53,6 +54,7 @@
|
||||
- **Impact**: 100% reliable frontend dependency management
|
||||
|
||||
### **Test Phase Issues**
|
||||
|
||||
1. **✅ E2E Docker Pull Complexity**
|
||||
- **Problem**: Over-engineered retry logic for E2E tests only
|
||||
- **Solution**: Use same simple approach as all other successful tests
|
||||
|
||||
@@ -30,6 +30,7 @@ RUN git clone full_repo && merge_preserving_deps # ✅ Source changes don't bus
|
||||
**Technical Challenges & Solutions**:
|
||||
|
||||
1. **Local Package Build Error**: `OSError: Readme file does not exist: ../README.md`
|
||||
|
||||
```dockerfile
|
||||
# Fix: Create minimal structure for package build
|
||||
RUN mkdir -p src/backend && \
|
||||
@@ -39,6 +40,7 @@ RUN git clone full_repo && merge_preserving_deps # ✅ Source changes don't bus
|
||||
```
|
||||
|
||||
2. **Dependency Preservation**: Need to preserve installed packages when copying source
|
||||
|
||||
```dockerfile
|
||||
# Fix: Backup/restore strategy
|
||||
RUN if [ -d "/workspace/backend/.venv" ]; then mv /workspace/backend/.venv /tmp/venv_backup; fi && \
|
||||
@@ -47,13 +49,15 @@ RUN git clone full_repo && merge_preserving_deps # ✅ Source changes don't bus
|
||||
```
|
||||
|
||||
3. **No rsync Available**: Base image doesn't include rsync for selective copying
|
||||
```dockerfile
|
||||
# Fix: Use standard cp with backup strategy instead of rsync
|
||||
# rsync -av --exclude='node_modules' /tmp/fullrepo/ /workspace/ # ❌ Not available
|
||||
# Standard cp with manual exclusions # ✅ Works everywhere
|
||||
```
|
||||
|
||||
```dockerfile
|
||||
# Fix: Use standard cp with backup strategy instead of rsync
|
||||
# rsync -av --exclude='node_modules' /tmp/fullrepo/ /workspace/ # ❌ Not available
|
||||
# Standard cp with manual exclusions # ✅ Works everywhere
|
||||
```
|
||||
|
||||
**Metrics**:
|
||||
|
||||
- Dependency cache hit rate: ~95% (only miss when pyproject.toml/package.json change)
|
||||
- Average build time reduction: 12-17 minutes saved per build
|
||||
- Resource efficiency: Better CPU/memory utilization on Raspberry Pi workers
|
||||
@@ -65,6 +69,7 @@ RUN git clone full_repo && merge_preserving_deps # ✅ Source changes don't bus
|
||||
**Problem**: Firefox and WebKit browsers failing consistently in Docker CI environment.
|
||||
|
||||
**Root Cause Analysis**:
|
||||
|
||||
- **Firefox**: Sandbox restrictions in Docker containers, requires `--no-sandbox` and security compromises
|
||||
- **WebKit**: Content loading timeout issues, navigation reliability problems in headless mode
|
||||
- **Docker Environment**: Limited resources (RPi 4GB) exacerbate browser compatibility issues
|
||||
@@ -77,26 +82,28 @@ const projects = process.env.CI
|
||||
? [
|
||||
// CI: Only Chromium (most reliable in Docker)
|
||||
{
|
||||
name: 'chromium',
|
||||
use: { ...devices['Desktop Chrome'] },
|
||||
}
|
||||
name: "chromium",
|
||||
use: { ...devices["Desktop Chrome"] },
|
||||
},
|
||||
]
|
||||
: [
|
||||
// Local: Full browser coverage
|
||||
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
|
||||
{ name: 'firefox', use: { ...devices['Desktop Firefox'] } },
|
||||
{ name: 'webkit', use: { ...devices['Desktop Safari'] } },
|
||||
{ name: "chromium", use: { ...devices["Desktop Chrome"] } },
|
||||
{ name: "firefox", use: { ...devices["Desktop Firefox"] } },
|
||||
{ name: "webkit", use: { ...devices["Desktop Safari"] } },
|
||||
];
|
||||
```
|
||||
|
||||
**Rationale**:
|
||||
|
||||
- Chromium engine powers 95%+ of web browsers (Chrome, Edge, Opera, Brave)
|
||||
- Excellent Docker compatibility and resource efficiency
|
||||
- Core functionality testing coverage maintained
|
||||
- Full browser testing available for local development
|
||||
|
||||
**Error Examples Resolved**:
|
||||
```
|
||||
|
||||
```text
|
||||
Firefox: error: unknown option '--headed=false'
|
||||
WebKit: Test timeout 30000ms exceeded... waiting for navigation
|
||||
Firefox: browserType.launch: Executable doesn't exist
|
||||
@@ -113,8 +120,9 @@ Firefox: browserType.launch: Executable doesn't exist
|
||||
**Solution**: Multi-level retry logic with exponential backoff:
|
||||
|
||||
#### Docker Registry Operations
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/cicd.yml
|
||||
# .gitea/workflows/cicd-checks.yaml
|
||||
- name: Login to Container Registry (with retry)
|
||||
run: |
|
||||
for attempt in {1..5}; do
|
||||
@@ -135,6 +143,7 @@ Firefox: browserType.launch: Executable doesn't exist
|
||||
```
|
||||
|
||||
#### Playwright Browser Installation
|
||||
|
||||
```yaml
|
||||
- name: Install Playwright Browsers (with retry)
|
||||
run: |
|
||||
@@ -151,14 +160,19 @@ Firefox: browserType.launch: Executable doesn't exist
|
||||
```
|
||||
|
||||
#### E2E Test Navigation Resilience
|
||||
|
||||
```typescript
|
||||
// frontend/tests/e2e/app.spec.ts
|
||||
async function navigateWithRetry(page: Page, url: string, maxRetries: number = 3): Promise<void> {
|
||||
async function navigateWithRetry(
|
||||
page: Page,
|
||||
url: string,
|
||||
maxRetries: number = 3,
|
||||
): Promise<void> {
|
||||
for (let attempt = 1; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
await page.goto(url, {
|
||||
waitUntil: 'networkidle',
|
||||
timeout: 90000 // Extended timeout
|
||||
waitUntil: "networkidle",
|
||||
timeout: 90000, // Extended timeout
|
||||
});
|
||||
return;
|
||||
} catch (error) {
|
||||
@@ -171,6 +185,7 @@ async function navigateWithRetry(page: Page, url: string, maxRetries: number = 3
|
||||
```
|
||||
|
||||
**Configuration Enhancements**:
|
||||
|
||||
```typescript
|
||||
// playwright.config.ts - CI optimizations
|
||||
use: {
|
||||
@@ -182,6 +197,7 @@ use: {
|
||||
```
|
||||
|
||||
**Results**:
|
||||
|
||||
- CI success rate: 40% → 95%
|
||||
- Average retry overhead: +30 seconds per build
|
||||
- Network timeout elimination: 100% of Docker operations now succeed
|
||||
@@ -193,7 +209,8 @@ use: {
|
||||
**Problem**: Production base image missing pre-installed Python dev tools optimization.
|
||||
|
||||
**Symptom**:
|
||||
```
|
||||
|
||||
```text
|
||||
⚠ Pre-installed Python dev tools not found - fresh installation
|
||||
Base image may need rebuild for optimal caching
|
||||
```
|
||||
@@ -201,6 +218,7 @@ Base image may need rebuild for optimal caching
|
||||
**Impact**: +15-20 seconds build time (acceptable degradation vs failure)
|
||||
|
||||
**Solution**: Graceful fallback detection:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile.cicd - Resilient optimization detection
|
||||
RUN echo "=== Base Image Optimization Status ===" && \
|
||||
@@ -220,7 +238,8 @@ RUN echo "=== Base Image Optimization Status ===" && \
|
||||
### Missing Immutable Base Image
|
||||
|
||||
**Symptom**:
|
||||
```
|
||||
|
||||
```text
|
||||
❌ Required immutable base image is not available: kankali.darkhelm.lan:3001/darkhelm.org/plex-playlist-cicd-base:<hash>
|
||||
Publish the base image via the CICD Base Image workflow before rerunning main CI.
|
||||
```
|
||||
@@ -229,11 +248,13 @@ Publish the base image via the CICD Base Image workflow before rerunning main CI
|
||||
dedicated base-image workflow has not published that immutable tag yet.
|
||||
|
||||
**Checks**:
|
||||
|
||||
1. Confirm whether `Dockerfile.cicd-base`, `.dockerignore`, or `scripts/compute-cicd-base-hash.sh` changed in the branch.
|
||||
2. Check the `CICD Base Image` workflow for the same commit or PR.
|
||||
3. Verify the registry contains `plex-playlist-cicd-base:<hash>`.
|
||||
|
||||
**Resolution**:
|
||||
|
||||
1. If the base workflow is still running, rerun main CI after it completes.
|
||||
2. If the base workflow did not trigger, run it manually with `force_rebuild=false`.
|
||||
3. If the tag should be republished despite already existing, run it manually with `force_rebuild=true`.
|
||||
@@ -245,47 +266,60 @@ publish-once/consume-many design.
|
||||
### Docker Build Failures
|
||||
|
||||
#### 1. rsync Command Not Found
|
||||
```
|
||||
|
||||
```text
|
||||
/bin/bash: line 1: rsync: command not found
|
||||
```
|
||||
|
||||
**Fix**: Replace with standard cp commands and backup strategy (implemented)
|
||||
|
||||
#### 2. README.md Not Found During uv sync
|
||||
```
|
||||
|
||||
```text
|
||||
OSError: Readme file does not exist: ../README.md
|
||||
```
|
||||
|
||||
**Fix**: Create dummy README.md during dependency installation phase (implemented)
|
||||
|
||||
#### 3. Dependency Cache Invalidation
|
||||
|
||||
**Symptom**: Dependencies rebuilding on every commit
|
||||
**Fix**: Verify dependency-first build pattern is correctly implemented
|
||||
|
||||
### E2E Test Failures
|
||||
|
||||
#### 1. Browser Not Found
|
||||
```
|
||||
|
||||
```text
|
||||
Executable doesn't exist at /root/.cache/ms-playwright/chromium-*/
|
||||
```
|
||||
|
||||
**Fix**: Ensure `yarn playwright install --with-deps` runs before tests
|
||||
|
||||
#### 2. Navigation Timeouts
|
||||
```
|
||||
|
||||
```text
|
||||
Test timeout 30000ms exceeded
|
||||
```
|
||||
|
||||
**Fix**: Use `navigateWithRetry` helper with extended timeouts
|
||||
|
||||
#### 3. Multi-browser Failures in CI
|
||||
|
||||
**Fix**: Use Chromium-only configuration for CI environments
|
||||
|
||||
### Network-Related Issues
|
||||
|
||||
#### 1. Docker Registry Timeouts
|
||||
|
||||
**Fix**: Retry logic with exponential backoff (5 attempts, 15s intervals)
|
||||
|
||||
#### 2. Package Download Failures
|
||||
|
||||
**Fix**: Increase timeouts and add retry mechanisms
|
||||
|
||||
#### 3. SSL Certificate Issues
|
||||
|
||||
**Fix**: Set `ignoreHTTPSErrors: true` and `NODE_TLS_REJECT_UNAUTHORIZED=0`
|
||||
|
||||
## Performance Monitoring
|
||||
@@ -320,12 +354,13 @@ Test timeout 30000ms exceeded
|
||||
**🎉 MILESTONE ACHIEVED**: First fully successful CI/CD workflow completion with all optimizations working together.
|
||||
|
||||
**Final Performance Metrics**:
|
||||
|
||||
- **Total Pipeline Time**: ~3-5 minutes (down from 15-25 minutes)
|
||||
- **Success Rate**: 100% (all test phases passing)
|
||||
- **Build Optimization**: 85% time reduction achieved
|
||||
- **E2E Test Reliability**: 100% (simplified Docker approach)
|
||||
|
||||
### **Key Issues Resolved in Final Sprint**:
|
||||
### **Key Issues Resolved in Final Sprint**
|
||||
|
||||
1. **✅ README.md Dependency Fix**: Dummy file creation for dependency-only builds
|
||||
2. **✅ Rsync Replacement**: Standard cp commands with backup/restore strategy
|
||||
@@ -333,7 +368,8 @@ Test timeout 30000ms exceeded
|
||||
4. **✅ E2E Test Simplification**: Removed unnecessary complex retry logic
|
||||
5. **✅ Memory Management**: Proper swap configuration and Node.js memory limits
|
||||
|
||||
### **Validated Working Components**:
|
||||
### **Validated Working Components**
|
||||
|
||||
- **Multi-stage Docker builds** with optimal layer caching
|
||||
- **Dependency-first build pattern** preventing cache invalidation
|
||||
- **Network-resilient Playwright setup** with Chromium-only CI testing
|
||||
@@ -341,8 +377,10 @@ Test timeout 30000ms exceeded
|
||||
- **SSH-based secure repository access** with proper key management
|
||||
- **Comprehensive test coverage** (linting, unit tests, integration, E2E)
|
||||
|
||||
### **Architecture Stability**:
|
||||
### **Architecture Stability**
|
||||
|
||||
All components now work cohesively:
|
||||
|
||||
- Base image caching (cicd-base) ↔️ Complete image building (cicd)
|
||||
- Python dependency management (uv) ↔️ Backend source integration
|
||||
- Frontend dependency management (Yarn PnP) ↔️ Source code preservation
|
||||
|
||||
156
docs/DEPLOYABLE_RUNTIME_CONTRACT.md
Normal file
156
docs/DEPLOYABLE_RUNTIME_CONTRACT.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Deployable Runtime Image Contract
|
||||
|
||||
## Purpose
|
||||
|
||||
Define the minimum deployable runtime contract for backend and frontend images.
|
||||
This document is the canonical source for what must be present, what must not
|
||||
be present, and what behavior deployment environments can rely on.
|
||||
|
||||
This contract supports issue PP-58 and establishes a baseline for future
|
||||
automation work under epic #66.
|
||||
|
||||
## Scope
|
||||
|
||||
Included:
|
||||
|
||||
- Backend deployable runtime image requirements.
|
||||
- Frontend deployable runtime image requirements.
|
||||
- Runtime entrypoint, ports, health behavior, startup behavior, and environment
|
||||
contracts.
|
||||
- Explicitly disallowed non-runtime tooling classes in deployable images.
|
||||
|
||||
Excluded:
|
||||
|
||||
- CI workflow rewiring.
|
||||
- Test execution redesign.
|
||||
- New runtime hardening implementations not required to define contract.
|
||||
|
||||
## Backend Runtime Contract
|
||||
|
||||
### Runtime Artifact Definition
|
||||
|
||||
- Container build source: `Dockerfile.backend`.
|
||||
- Runtime base image: `python:3.14-slim`.
|
||||
- Runtime process: `uvicorn main:app --host 0.0.0.0 --port 8000`.
|
||||
- Exposed runtime port: `8000`.
|
||||
|
||||
### Required Runtime Dependencies
|
||||
|
||||
The runtime artifact must include versions compatible with:
|
||||
|
||||
- `fastapi==0.120.2`
|
||||
- `psycopg==3.2.12`
|
||||
- `sqlalchemy==2.0.44`
|
||||
- `uvicorn==0.38.0`
|
||||
|
||||
The lockfile in `backend/uv.lock` is the dependency source of truth.
|
||||
|
||||
### Required Runtime Environment Contract
|
||||
|
||||
- `DATABASE_URL` is required.
|
||||
- Accepted form: `postgresql://...` or SQLAlchemy async form.
|
||||
- Runtime normalization to async psycopg dialect is performed in
|
||||
`backend/src/backend/database.py`.
|
||||
- `BACKEND_REQUIRED_PYTHON` is optional and defaults to `3.14`.
|
||||
|
||||
### Backend Health and Startup Behavior
|
||||
|
||||
- Startup must fail fast if runtime policy checks fail.
|
||||
- Source: `backend/src/backend/main.py` lifecycle (`lifespan`) validation.
|
||||
- Health endpoint contract:
|
||||
- `GET /health` returns `200` with `{"status":"healthy","database":"connected"}`
|
||||
when database probe succeeds.
|
||||
- `GET /health` returns `503` with
|
||||
`{"status":"unhealthy","database":"disconnected"}` when probe fails.
|
||||
- Compatibility diagnostics endpoint:
|
||||
- `GET /compatibility` reports policy and package compatibility status.
|
||||
|
||||
### Backend Runtime Checklist
|
||||
|
||||
- [ ] Runtime image built from `Dockerfile.backend`.
|
||||
- [ ] Runtime process is uvicorn serving `main:app` on `0.0.0.0:8000`.
|
||||
- [ ] `DATABASE_URL` is set in deployment runtime.
|
||||
- [ ] `GET /health` behavior matches contract.
|
||||
- [ ] Startup fails on runtime policy mismatch.
|
||||
- [ ] Deployable artifact excludes CI-only and test-only tooling classes.
|
||||
|
||||
## Frontend Runtime Contract
|
||||
|
||||
### Frontend Runtime Artifact Definition
|
||||
|
||||
- Container build source: `Dockerfile.frontend` (target `production`).
|
||||
- Runtime base image: `nginx:alpine`.
|
||||
- Runtime process: `nginx -g "daemon off;"`.
|
||||
- Exposed runtime port: `80`.
|
||||
- Runtime artifact payload: static assets from `/app/dist` copied to
|
||||
`/usr/share/nginx/html`.
|
||||
|
||||
### Frontend Runtime Entry and Routing Contract
|
||||
|
||||
- Nginx configuration source: `frontend/nginx.conf`.
|
||||
- SPA routing behavior must use fallback to `index.html` for unknown routes.
|
||||
- API requests under `/api/` are proxied to backend service endpoint
|
||||
`http://backend:8000/` in compose deployments.
|
||||
|
||||
### Frontend Health and Startup Behavior
|
||||
|
||||
- Startup expectation: nginx process starts and serves static assets on port 80.
|
||||
- Runtime health expectation for deployment checks:
|
||||
- `GET /` should return `200` and serve frontend entry document.
|
||||
- API proxy readiness is dependent on backend runtime availability.
|
||||
|
||||
### Frontend Runtime Environment Contract
|
||||
|
||||
- No required runtime environment variables are defined for nginx static serving.
|
||||
- Build-time frontend mode is production-oriented and not part of runtime env
|
||||
contract.
|
||||
|
||||
### Frontend Runtime Checklist
|
||||
|
||||
- [ ] Runtime image built from `Dockerfile.frontend` production target.
|
||||
- [ ] Runtime process is nginx serving on port 80.
|
||||
- [ ] SPA route fallback behavior is present.
|
||||
- [ ] `/api/` proxy behavior aligns with backend service wiring.
|
||||
- [ ] Deployable artifact excludes CI-only and test-only tooling classes.
|
||||
|
||||
## Disallowed Tooling Classes in Deployable Runtime Images
|
||||
|
||||
Deployable runtime artifacts must not include tooling classes that are only
|
||||
needed for CI, validation, or local development workflows.
|
||||
|
||||
Disallowed classes:
|
||||
|
||||
- Linters and formatters (example: ruff, eslint, prettier).
|
||||
- Type checking and static analysis tooling (example: pyright, vue-tsc).
|
||||
- Test frameworks and test drivers (example: pytest, vitest, playwright).
|
||||
- Browser test binaries and CI runner helper tools.
|
||||
- Build-only package managers and build toolchains not needed at runtime.
|
||||
|
||||
Note:
|
||||
|
||||
- Multi-stage builds may use these tools in build stages.
|
||||
- These tools must not be required by or present in final deployable runtime
|
||||
image layers.
|
||||
|
||||
## Acceptance Criteria Traceability (PP-58)
|
||||
|
||||
1. Backend runtime requirements are documented and approved.
|
||||
- Covered by: "Backend Runtime Contract" and backend checklist.
|
||||
2. Frontend runtime requirements are documented and approved.
|
||||
- Covered by: "Frontend Runtime Contract" and frontend checklist.
|
||||
3. Runtime contracts include health endpoint expectations and startup behavior.
|
||||
- Covered by: backend health/startup section and frontend health/startup
|
||||
section.
|
||||
4. Non-runtime tool classes are explicitly excluded from deployable image
|
||||
definition.
|
||||
- Covered by: "Disallowed Tooling Classes in Deployable Runtime Images".
|
||||
|
||||
## Future Enforcement Hooks (Out of Scope for PP-58)
|
||||
|
||||
Potential follow-up automation under epic #66:
|
||||
|
||||
- Policy checks validating final image layers do not include disallowed tooling
|
||||
classes.
|
||||
- Contract tests that assert documented health/startup behavior.
|
||||
- CI checks that verify Dockerfile target boundaries remain aligned with this
|
||||
contract.
|
||||
@@ -19,6 +19,24 @@ This document outlines how to set up your development environment and work with
|
||||
- **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
|
||||
- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
|
||||
- **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
|
||||
- **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules
|
||||
- **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries
|
||||
|
||||
## Deployable Runtime Artifacts
|
||||
|
||||
When changing deployment behavior or image composition, treat
|
||||
`DEPLOYABLE_RUNTIME_CONTRACT.md` as the source of truth for:
|
||||
|
||||
- Runtime entrypoint and exposed ports.
|
||||
- Health and startup expectations.
|
||||
- Runtime environment contract.
|
||||
- Disallowed non-runtime tooling classes in final deployable images.
|
||||
|
||||
Scope boundary:
|
||||
|
||||
- This repository separates contract definition from enforcement mechanics.
|
||||
- CI workflow rewiring and test execution redesign are out of scope for PP-58
|
||||
and belong to follow-up work under epic #66.
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -69,10 +87,10 @@ docker compose -f compose.dev.yml up -d --build
|
||||
|
||||
### Service Access
|
||||
|
||||
- **Frontend**: http://localhost:3000
|
||||
- **Backend API**: http://localhost:8000
|
||||
- **API Documentation**: http://localhost:8000/docs
|
||||
- **Database**: localhost:5432 (user: `plex`, password: see `secrets/postgres_password`)
|
||||
- **Frontend**: <http://localhost:3000>
|
||||
- **Backend API**: <http://localhost:8000>
|
||||
- **API Documentation**: <http://localhost:8000/docs>
|
||||
- **Database**: `localhost:5432` (user: `plex`, password: see `secrets/postgres_password`)
|
||||
|
||||
## Poe the Poet Task Runner
|
||||
|
||||
@@ -197,6 +215,7 @@ pre-commit run --all-files
|
||||
### How It Works
|
||||
|
||||
Pre-commit automatically runs on every `git commit` and will:
|
||||
|
||||
- Format code (Prettier, Ruff)
|
||||
- Check syntax (ESLint, Pyright)
|
||||
- Validate files (YAML, TOML, trailing whitespace)
|
||||
@@ -207,6 +226,7 @@ If any hook fails, the commit is blocked until issues are fixed.
|
||||
### Pre-commit is Optional
|
||||
|
||||
While strongly recommended, pre-commit is not required because:
|
||||
|
||||
- **CI Validation**: All the same checks run in CI
|
||||
- **Developer Choice**: Some prefer manual tool usage
|
||||
- **Learning**: Developers can run tools individually to understand them
|
||||
@@ -221,7 +241,8 @@ If you prefer not to use pre-commit, here's how to run each tool manually:
|
||||
|
||||
Navigate to the `backend/` directory for all backend commands.
|
||||
|
||||
#### Code Formatting
|
||||
#### Backend Code Formatting
|
||||
|
||||
```bash
|
||||
# Format code with Ruff
|
||||
uv run ruff format .
|
||||
@@ -230,13 +251,15 @@ uv run ruff format .
|
||||
uv run ruff check . --fix
|
||||
```
|
||||
|
||||
#### Type Checking
|
||||
#### Backend Type Checking
|
||||
|
||||
```bash
|
||||
# Run Pyright type checker
|
||||
uv run pyright
|
||||
```
|
||||
|
||||
#### Testing
|
||||
#### Backend Testing
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
uv run pytest
|
||||
@@ -251,7 +274,8 @@ uv run pytest tests/integration/
|
||||
uv run xdoctest src/
|
||||
```
|
||||
|
||||
#### Documentation
|
||||
#### Backend Documentation
|
||||
|
||||
```bash
|
||||
# Check docstring style
|
||||
uv run pydoclint --config=pyproject.toml src/
|
||||
@@ -261,7 +285,8 @@ uv run pydoclint --config=pyproject.toml src/
|
||||
|
||||
Navigate to the `frontend/` directory for all frontend commands.
|
||||
|
||||
#### Code Formatting
|
||||
#### Frontend Code Formatting
|
||||
|
||||
```bash
|
||||
# Format code with Prettier
|
||||
yarn format
|
||||
@@ -271,6 +296,7 @@ yarn format:check
|
||||
```
|
||||
|
||||
#### Linting
|
||||
|
||||
```bash
|
||||
# Run ESLint
|
||||
yarn lint
|
||||
@@ -282,13 +308,15 @@ yarn lint:fix
|
||||
yarn lint:tsdoc
|
||||
```
|
||||
|
||||
#### Type Checking
|
||||
#### Frontend Type Checking
|
||||
|
||||
```bash
|
||||
# Run Vue TypeScript compiler
|
||||
yarn type-check
|
||||
```
|
||||
|
||||
#### Testing
|
||||
#### Frontend Testing
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
yarn test
|
||||
@@ -341,22 +369,26 @@ The CI/CD pipeline uses a **multi-stage build architecture** for optimal perform
|
||||
- **Stage 2**: Build complete image (project code and dependencies) - **rebuilt every time**
|
||||
|
||||
Pipeline triggers:
|
||||
|
||||
- Push to any branch
|
||||
- Pull requests to `main` or `develop`
|
||||
|
||||
### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL**
|
||||
|
||||
**Performance Gains**:
|
||||
|
||||
- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes)
|
||||
- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs)
|
||||
- **100% success rate** achieved with optimized dependency management
|
||||
- Raspberry Pi 4GB workers handle builds efficiently with resource optimization
|
||||
|
||||
**Architecture**:
|
||||
|
||||
- `cicd-base:latest` - System dependencies (Python 3.14, Node.js 24, build tools, pre-installed dev packages)
|
||||
- `cicd:latest` - Complete environment (project code + optimized dependency installation)
|
||||
|
||||
**Recent Optimizations** (November 2025):
|
||||
|
||||
- **Dependency-first build pattern** prevents cache invalidation on code changes
|
||||
- **Yarn PnP state regeneration** ensures reliable frontend builds
|
||||
- **Network-resilient E2E testing** with simplified Docker operations
|
||||
|
||||
@@ -5,6 +5,7 @@ This document contains solutions to common issues with Gitea Actions CI/CD pipel
|
||||
## Critical Issue: Jobs Stuck in "Waiting" State Forever
|
||||
|
||||
### Symptoms
|
||||
|
||||
- Workflows are created but jobs show "Waiting" indefinitely
|
||||
- Runners are online and healthy
|
||||
- No tasks appear in `action_task` database table
|
||||
@@ -12,9 +13,11 @@ This document contains solutions to common issues with Gitea Actions CI/CD pipel
|
||||
- UI shows "Waiting" but database shows status 5 (cancelled)
|
||||
|
||||
### Root Cause
|
||||
|
||||
**Docker syntax in `runs-on` labels** causes Gitea Actions to immediately cancel jobs.
|
||||
|
||||
### Problem Syntax (BROKEN)
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
setup:
|
||||
@@ -26,6 +29,7 @@ jobs:
|
||||
```
|
||||
|
||||
### Solution Syntax (WORKING)
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
setup:
|
||||
@@ -37,7 +41,9 @@ jobs:
|
||||
```
|
||||
|
||||
### Why This Works
|
||||
|
||||
The runners are configured with Docker images in their labels:
|
||||
|
||||
```bash
|
||||
GITEA_RUNNER_LABELS=ubuntu-latest:docker://ubuntu:22.04,node-latest:docker://node:20-bookworm-slim,python-latest:docker://python:3.14-slim
|
||||
```
|
||||
@@ -47,11 +53,13 @@ So jobs still run in the correct Docker containers, but Gitea can properly parse
|
||||
### Diagnosis Steps
|
||||
|
||||
1. **Check if new runs are created:**
|
||||
|
||||
```sql
|
||||
SELECT id, status, title FROM action_run ORDER BY id DESC LIMIT 3;
|
||||
```
|
||||
|
||||
2. **Check job status and duration:**
|
||||
|
||||
```sql
|
||||
SELECT arj.id, arj.job_id, arj.status, ar.created, ar.updated, (ar.updated - ar.created) as duration_seconds
|
||||
FROM action_run_job arj
|
||||
@@ -60,21 +68,25 @@ WHERE ar.id = (SELECT MAX(id) FROM action_run);
|
||||
```
|
||||
|
||||
3. **Check if tasks are created:**
|
||||
|
||||
```sql
|
||||
SELECT * FROM action_task ORDER BY id DESC LIMIT 5;
|
||||
```
|
||||
|
||||
4. **Verify runners are online:**
|
||||
|
||||
```sql
|
||||
SELECT id, name, last_online, agent_labels FROM action_runner WHERE last_online > (EXTRACT(epoch FROM NOW()) - 300)::bigint;
|
||||
```
|
||||
|
||||
### Key Indicators
|
||||
|
||||
- **Duration = 0 seconds** → Immediate cancellation due to syntax issue
|
||||
- **Empty action_task table** → Jobs never converted to executable tasks
|
||||
- **Status 5 jobs with Status 7 dependents** → Setup job cancelled, others skipped
|
||||
|
||||
### Test Procedure
|
||||
|
||||
Create a minimal test workflow to isolate issues:
|
||||
|
||||
```yaml
|
||||
@@ -95,13 +107,17 @@ If this works but your main workflow doesn't, the issue is likely syntax-related
|
||||
## Other Common Issues
|
||||
|
||||
### Cache/UI Synchronization Problems
|
||||
|
||||
If UI shows different status than database:
|
||||
|
||||
1. Restart Gitea: `docker compose restart server`
|
||||
2. Clear browser cache
|
||||
3. Check database vs UI status discrepancies
|
||||
|
||||
### Stuck Runs from Previous Sessions
|
||||
|
||||
Clean up stuck runs:
|
||||
|
||||
```sql
|
||||
-- Clear stuck pending jobs
|
||||
UPDATE action_run_job SET status = 5 WHERE status IN (1, 2);
|
||||
@@ -109,7 +125,9 @@ UPDATE action_run SET status = 5 WHERE status IN (1, 2);
|
||||
```
|
||||
|
||||
### Runner Registration Issues
|
||||
|
||||
If runners show "unregistered runner" errors:
|
||||
|
||||
1. Delete runner registrations: `DELETE FROM action_runner;`
|
||||
2. Restart all runner containers
|
||||
3. Let them auto-register with fresh state
|
||||
@@ -117,6 +135,7 @@ If runners show "unregistered runner" errors:
|
||||
## Infrastructure Overview
|
||||
|
||||
### Current Setup
|
||||
|
||||
- **Gitea Server**: Docker container with PostgreSQL backend
|
||||
- **Runners**: 8 Raspberry Pi runners across 4 servers
|
||||
- pi-desktop: Pi 400 4GB (2 runners)
|
||||
@@ -125,16 +144,20 @@ If runners show "unregistered runner" errors:
|
||||
- zhokq: Pi 4B 8GB (2 runners)
|
||||
|
||||
### Runner Configuration
|
||||
|
||||
Each runner supports multiple Docker environments:
|
||||
|
||||
- `ubuntu-latest` → `ubuntu:22.04`
|
||||
- `python-latest` → `python:3.14-slim`
|
||||
- `node-latest` → `node:20-bookworm-slim`
|
||||
- `ubuntu-act` → `catthehacker/ubuntu:act-latest`
|
||||
|
||||
### Mirroring the `ubuntu-act` Runner Image
|
||||
|
||||
If GHCR pulls are flaky, mirror the runner image into your local registry and point the label at that mirror instead of the upstream tag.
|
||||
|
||||
Example mirror flow:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/catthehacker/ubuntu:act-latest
|
||||
docker tag ghcr.io/catthehacker/ubuntu:act-latest kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
@@ -142,6 +165,7 @@ docker push kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
```
|
||||
|
||||
Recommended runner label once mirrored:
|
||||
|
||||
```bash
|
||||
GITEA_RUNNER_LABELS=ubuntu-latest:docker://ubuntu:22.04,node-latest:docker://node:20-bookworm-slim,python-latest:docker://python:3.14-slim,ubuntu-act:docker://kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
```
|
||||
@@ -164,7 +188,9 @@ source scripts/gitea-actions/check_runner_images.xsh
|
||||
```
|
||||
|
||||
### Workflow Design
|
||||
|
||||
Multi-stage pipeline with artifact passing:
|
||||
|
||||
1. **Setup**: Checkout code, create artifacts
|
||||
2. **Parallel Setup**: Backend (Python/uv) + Frontend (Node.js/Yarn)
|
||||
3. **Parallel Tests**: Backend tests + Frontend tests
|
||||
@@ -179,5 +205,5 @@ Multi-stage pipeline with artifact passing:
|
||||
|
||||
---
|
||||
|
||||
*Last updated: June 2, 2026*
|
||||
*Issue resolved after extensive database-level debugging and syntax isolation*
|
||||
_Last updated: June 2, 2026_
|
||||
_Issue resolved after extensive database-level debugging and syntax isolation_
|
||||
|
||||
@@ -18,6 +18,7 @@ poe
|
||||
## 📋 Essential Tasks
|
||||
|
||||
### Development Environment
|
||||
|
||||
```bash
|
||||
poe dev # Start development environment (Docker Compose)
|
||||
poe dev-down # Stop development environment
|
||||
@@ -26,6 +27,7 @@ poe dev-restart # Restart development environment
|
||||
```
|
||||
|
||||
### Code Quality (Unified Backend + Frontend)
|
||||
|
||||
```bash
|
||||
poe format # Format all code (Python + TypeScript)
|
||||
poe lint # Lint all code (Python + TypeScript)
|
||||
@@ -33,6 +35,7 @@ poe type-check # Type check all code (Python + TypeScript)
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
poe test-unit # Run all unit tests (backend + frontend)
|
||||
poe test-all # Run all tests including integration
|
||||
@@ -41,6 +44,7 @@ poe test-e2e # Run end-to-end tests only
|
||||
```
|
||||
|
||||
### CI/CD Pipeline
|
||||
|
||||
```bash
|
||||
poe ci-quick # Fast quality checks (format, lint, type-check)
|
||||
poe ci-full # Complete CI pipeline simulation
|
||||
@@ -48,6 +52,7 @@ poe quality-gate # All quality checks (like CI)
|
||||
```
|
||||
|
||||
### Docker Images
|
||||
|
||||
```bash
|
||||
poe build-cicd # Build both base and complete CI/CD images
|
||||
poe build-cicd-base # Build only base image (cached dependencies)
|
||||
@@ -81,6 +86,7 @@ poe reset # Complete reset (clean + reinstall)
|
||||
## 🔧 Individual Component Tasks
|
||||
|
||||
### Backend Only
|
||||
|
||||
```bash
|
||||
poe format-backend # Format Python code only
|
||||
poe lint-backend # Lint Python code only
|
||||
@@ -90,6 +96,7 @@ poe test-backend-cov # Backend tests with coverage
|
||||
```
|
||||
|
||||
### Frontend Only
|
||||
|
||||
```bash
|
||||
poe format-frontend # Format TypeScript code only
|
||||
poe lint-frontend # Lint TypeScript code only
|
||||
@@ -116,6 +123,7 @@ poe pre-commit-update # Update hook versions
|
||||
## 💡 Tips & Tricks
|
||||
|
||||
### Task Discovery
|
||||
|
||||
```bash
|
||||
poe --help # List all tasks with descriptions
|
||||
poe <task> --help # Get help for specific task
|
||||
@@ -123,6 +131,7 @@ poe # Interactive task picker
|
||||
```
|
||||
|
||||
### Chaining Tasks
|
||||
|
||||
```bash
|
||||
# Run multiple tasks in sequence
|
||||
poe format lint type-check test-unit
|
||||
@@ -132,12 +141,14 @@ poe clean deps-install ci-quick
|
||||
```
|
||||
|
||||
### Environment Context
|
||||
|
||||
- All tasks run from project root
|
||||
- Backend tasks automatically use `uv run` in correct environment
|
||||
- Frontend tasks automatically use `yarn` in correct directory
|
||||
- Docker tasks use the optimized multi-stage CI/CD setup
|
||||
|
||||
### Performance Tips
|
||||
|
||||
- Use parallel tasks (`*-parallel`) for faster feedback
|
||||
- Use conditional tasks (`*-if-changed`) to save time
|
||||
- `poe setup` configures everything for new developers
|
||||
@@ -146,6 +157,7 @@ poe clean deps-install ci-quick
|
||||
## 🔄 Migration from Manual Commands
|
||||
|
||||
### Before (Manual)
|
||||
|
||||
```bash
|
||||
cd backend && uv run ruff format .
|
||||
cd frontend && yarn prettier --write src/
|
||||
@@ -155,6 +167,7 @@ cd frontend && yarn test
|
||||
```
|
||||
|
||||
### After (Poe)
|
||||
|
||||
```bash
|
||||
poe format
|
||||
poe test-unit
|
||||
|
||||
@@ -84,7 +84,7 @@ Add Renovate to your existing Gitea Actions workflow:
|
||||
name: Renovate
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 8 * * 1' # Monday 8 AM
|
||||
- cron: "0 8 * * 1" # Monday 8 AM
|
||||
workflow_dispatch: # Manual trigger
|
||||
|
||||
jobs:
|
||||
@@ -163,6 +163,7 @@ Once active, Renovate will:
|
||||
### 3. Integration with CI/CD
|
||||
|
||||
Renovate PRs will trigger your existing CI/CD pipeline:
|
||||
|
||||
- Build and test in Docker containers
|
||||
- Run full quality gates (linting, type checking, tests)
|
||||
- Only merge if all checks pass
|
||||
@@ -172,6 +173,7 @@ Renovate PRs will trigger your existing CI/CD pipeline:
|
||||
### Dashboard
|
||||
|
||||
Renovate creates a "Dependency Dashboard" issue showing:
|
||||
|
||||
- Pending updates
|
||||
- Failed PRs
|
||||
- Ignored dependencies
|
||||
@@ -180,6 +182,7 @@ Renovate creates a "Dependency Dashboard" issue showing:
|
||||
### Logs and Debugging
|
||||
|
||||
For self-hosted setup:
|
||||
|
||||
```bash
|
||||
# Run with debug logging
|
||||
docker run --rm \
|
||||
@@ -204,6 +207,7 @@ docker run --rm \
|
||||
### Quick Validation
|
||||
|
||||
For basic JSON validation without installing Renovate:
|
||||
|
||||
```bash
|
||||
# Quick syntax check (no Renovate installation needed)
|
||||
./scripts/quick-renovate-check.sh
|
||||
@@ -265,6 +269,7 @@ For basic JSON validation without installing Renovate:
|
||||
---
|
||||
|
||||
**Related Documentation**:
|
||||
|
||||
- [Renovate Official Docs](https://docs.renovatebot.com/)
|
||||
- [Configuration Options](https://docs.renovatebot.com/configuration-options/)
|
||||
- [Package Rules](https://docs.renovatebot.com/configuration-options/#packagerules)
|
||||
|
||||
@@ -5,22 +5,26 @@ This document explains how our CI/CD pipeline securely handles SSH keys using Do
|
||||
## 🔒 Security Benefits
|
||||
|
||||
### Before (Insecure)
|
||||
|
||||
```dockerfile
|
||||
ARG SSH_PRIVATE_KEY
|
||||
RUN echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
|
||||
```
|
||||
|
||||
- ❌ SSH key stored in Docker image layers
|
||||
- ❌ Visible in `docker history`
|
||||
- ❌ Can be extracted from images
|
||||
- ❌ Security vulnerability
|
||||
|
||||
### After (Secure)
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=secret,id=ssh_private_key \
|
||||
cp /run/secrets/ssh_private_key ~/.ssh/id_rsa && \
|
||||
# ... use key ... && \
|
||||
rm -rf ~/.ssh
|
||||
```
|
||||
|
||||
- ✅ SSH key never stored in image layers
|
||||
- ✅ Not visible in `docker history`
|
||||
- ✅ Cannot be extracted from final image
|
||||
@@ -29,14 +33,17 @@ RUN --mount=type=secret,id=ssh_private_key \
|
||||
## 🏗️ CI/CD Pipeline Implementation
|
||||
|
||||
### Gitea Actions Workflow
|
||||
The `.gitea/workflows/cicd.yml` file now uses:
|
||||
|
||||
The CI workflow files under `.gitea/workflows/` now use:
|
||||
|
||||
1. **Docker BuildKit Enabled**
|
||||
|
||||
```yaml
|
||||
export DOCKER_BUILDKIT=1
|
||||
```
|
||||
|
||||
2. **Secure Secret Mounting**
|
||||
|
||||
```yaml
|
||||
# Create temporary SSH key file
|
||||
echo "${SSH_PRIVATE_KEY}" > /tmp/ssh_key
|
||||
@@ -52,7 +59,9 @@ The `.gitea/workflows/cicd.yml` file now uses:
|
||||
```
|
||||
|
||||
### Local Development
|
||||
|
||||
Use the secure build script:
|
||||
|
||||
```bash
|
||||
./scripts/build-cicd-secure.sh plex-playlist-cicd:latest
|
||||
```
|
||||
@@ -60,11 +69,14 @@ Use the secure build script:
|
||||
## 🔧 Required Setup
|
||||
|
||||
### 1. Gitea Secrets Configuration
|
||||
|
||||
Ensure these secrets are configured in your Gitea repository:
|
||||
|
||||
- `SSH_PRIVATE_KEY`: Your private SSH key for git operations
|
||||
- `GITEA_TOKEN`: Token for pushing to container registry
|
||||
|
||||
### 2. Docker BuildKit Support
|
||||
|
||||
- **Gitea Actions**: Automatically enabled with `DOCKER_BUILDKIT=1`
|
||||
- **Local builds**: Requires Docker 18.09+ with BuildKit enabled
|
||||
- **CI runners**: Ensure BuildKit support in your runner environment
|
||||
@@ -80,6 +92,7 @@ Ensure these secrets are configured in your Gitea repository:
|
||||
## 🧪 Testing Security
|
||||
|
||||
Verify no secrets in image:
|
||||
|
||||
```bash
|
||||
# Build the image
|
||||
./scripts/build-cicd-secure.sh test-image
|
||||
|
||||
@@ -10,6 +10,7 @@ Historically, floating dependency constraints and non-enforced runtime assumptio
|
||||
introduce drift and hard-to-diagnose failures.
|
||||
|
||||
This branch introduced:
|
||||
|
||||
- Python 3.14 as the required runtime baseline
|
||||
- Exact dependency pinning for backend runtime and development tooling
|
||||
- Startup compatibility checks that fail fast when runtime policy is violated
|
||||
@@ -28,11 +29,13 @@ Adopt a deterministic backend runtime policy:
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- Reduced environment drift across dev/CI/prod
|
||||
- Earlier and clearer failure mode for runtime mismatches
|
||||
- Improved reproducibility and troubleshooting
|
||||
|
||||
Negative:
|
||||
|
||||
- More frequent explicit dependency maintenance updates
|
||||
- Stricter upgrade process for Python/runtime packages
|
||||
|
||||
|
||||
@@ -27,13 +27,14 @@ Adopt a hardened split workflow for the CICD base image:
|
||||
3. Treat the hash-specific tag as the source of truth for all CI consumers.
|
||||
4. Keep `latest` only as a convenience tag for humans and manual debugging.
|
||||
5. Make the main CI workflow consume only the immutable hash tag and fail clearly
|
||||
if that base image has not been published yet.
|
||||
if that base image has not been published yet.
|
||||
6. Add bounded polling in main CI to tolerate short publish/consume races between
|
||||
the dedicated base workflow and the main workflow.
|
||||
the dedicated base workflow and the main workflow.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- Faster CI via stable base-layer reuse
|
||||
- Better traceability from base image to Dockerfile content
|
||||
- Single publish, many pulls across the runner fleet
|
||||
@@ -41,6 +42,7 @@ Positive:
|
||||
- Clearer separation of concerns between artifact publication and application CI
|
||||
|
||||
Negative:
|
||||
|
||||
- Slightly more workflow complexity
|
||||
- Registry stores additional hash-tagged images
|
||||
- Main CI now fails fast when the expected base image is missing instead of
|
||||
|
||||
56
docs/adr/ADR003-deployable_runtime_image_contract.md
Normal file
56
docs/adr/ADR003-deployable_runtime_image_contract.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# ADR003: Deployable Runtime Image Contract Boundaries
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-06-19
|
||||
|
||||
## Context
|
||||
|
||||
The repository has both deployable runtime artifacts and CI/development tooling
|
||||
artifacts. Without an explicit boundary, non-runtime concerns can drift into
|
||||
deployable images, making runtime behavior less predictable and increasing
|
||||
artifact complexity.
|
||||
|
||||
Issue PP-58 requires a clear, approved contract for minimal backend and frontend
|
||||
deployable images, including health and startup behavior and explicit exclusion
|
||||
of non-runtime tooling classes.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt a canonical deployable runtime contract at:
|
||||
|
||||
- `docs/DEPLOYABLE_RUNTIME_CONTRACT.md`
|
||||
|
||||
The contract defines, for backend and frontend deployable images:
|
||||
|
||||
1. Runtime artifact boundaries (final image intent and payload).
|
||||
2. Runtime entrypoint and exposed ports.
|
||||
3. Runtime health and startup behavior expectations.
|
||||
4. Runtime environment variable contract.
|
||||
5. Disallowed non-runtime tooling classes in final deployable images.
|
||||
|
||||
Scope guardrails for PP-58:
|
||||
|
||||
- Documentation and architectural decision codification only.
|
||||
- No CI workflow rewiring in this issue.
|
||||
- No test execution redesign in this issue.
|
||||
- Enforcement automation deferred to follow-up work under epic #66.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive:
|
||||
|
||||
- Deployable image intent is explicit and auditable.
|
||||
- Future tickets can implement automated checks against a stable policy.
|
||||
- Reduced ambiguity between runtime artifacts and CI/development environments.
|
||||
|
||||
Negative:
|
||||
|
||||
- Requires ongoing documentation maintenance when runtime contracts evolve.
|
||||
- Drift can still occur if future changes bypass policy review.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
- Rely on Dockerfiles only as implicit contract.
|
||||
- Rejected: too easy for intent drift and inconsistent interpretation.
|
||||
- Enforce contract immediately in CI without documentation-first baseline.
|
||||
- Rejected: increases implementation risk without agreed policy language.
|
||||
Reference in New Issue
Block a user