plex-playlist/docs/DEVELOPMENT.md

# Development Environment Setup and Workflow

This document outlines how to set up your development environment and work with the plex-playlist project.

## Table of Contents

1. [Quick Start](#quick-start)
2. [Development Environment Setup](#development-environment-setup)
3. [Poe the Poet Task Runner](#poe-the-poet-task-runner)
4. [Git Workflow](#git-workflow)
5. [Pre-commit Hooks](#pre-commit-hooks)
6. [Manual Tool Usage](#manual-tool-usage)
7. [CI/CD Pipeline](#cicd-pipeline)
8. [Branch Protection and Merge Requirements](#branch-protection-and-merge-requirements)

## Related Documentation

- **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks
- **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
- **[CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md)** - Current CI lane failures and remediation paths
- **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices
- **[Deployable Runtime Contract](DEPLOYABLE_RUNTIME_CONTRACT.md)** - Canonical backend/frontend runtime artifact contract and exclusion rules
- **[ADR003: Deployable Runtime Image Contract Boundaries](adr/ADR003-deployable_runtime_image_contract.md)** - Decision record for deployable runtime boundaries
- **[ADR004: Registry Image Resolution and Auth Resilience Policy](adr/ADR004-registry-image-resolution-and-auth-resilience.md)** - Cross-workflow reliability policy for registry/auth/image fallback behavior

## Deployable Runtime Artifacts

When changing deployment behavior or image composition, treat
`DEPLOYABLE_RUNTIME_CONTRACT.md` as the source of truth for:

- Runtime entrypoint and exposed ports.
- Health and startup expectations.
- Runtime environment contract.
- Disallowed non-runtime tooling classes in final deployable images.

Scope boundary:

- Contract definition lives in `DEPLOYABLE_RUNTIME_CONTRACT.md`.
- CI enforcement now includes Dockerfile boundary checks and deployable image
  purity checks in `.gitea/workflows/cicd.yaml`.
- Broader workflow redesign and deployment wiring remain out of scope for this
  repository's runtime contract document and belong to follow-up work under
  epic #66.

## Runtime vs Validation Enforcement

Deployable runtime artifacts and validation environments are intentionally
separated.

- Validation tools (lint/typecheck/test/browser tooling) belong to CICD image
  paths (`Dockerfile.cicd-base`, `Dockerfile.cicd`) and CI validation
  workflows.
- Deployable runtime paths (`Dockerfile.backend`, `Dockerfile.frontend`
  production target) must remain independent of CI-only tooling layers.

Automated checks:

- `scripts/check-dockerfile-boundaries.sh` validates deployable Dockerfiles do
  not depend on CICD images or install disallowed CI-only tools.
- `scripts/verify-deployable-image-purity.sh` inspects built deployable images
  and fails if CI/development binaries or package metadata artifacts are
  present.

These checks run in `.gitea/workflows/cicd.yaml` before publishing the
complete CICD image.

## Quick Start

```bash
# Clone the repository
git clone ssh://git@dogar.darkhelm.org:2222/DarkHelm.org/plex-playlist.git
cd plex-playlist

# Complete setup with Poe the Poet (recommended)
cd backend && uv sync --dev && cd ..
poe setup

# OR manual setup
docker compose -f compose.dev.yml up -d
pip install pre-commit && pre-commit install

# Create a feature branch and start developing
git checkout -b feature/your-feature-name

# Use Poe for all development tasks
poe format lint type-check test-unit
```

## Development Environment Setup

### Local Development with Docker Compose

The project uses `compose.dev.yml` for local development, which provides:

- **Backend**: FastAPI application with hot reload
- **Frontend**: Vite development server with hot module replacement
- **Database**: PostgreSQL with persistent data
- **Development Tools**: Pre-configured with all necessary dependencies

```bash
# Start all services
docker compose -f compose.dev.yml up -d

# View logs
docker compose -f compose.dev.yml logs -f

# Stop services
docker compose -f compose.dev.yml down

# Rebuild after dependency changes
docker compose -f compose.dev.yml up -d --build
```

### Service Access

- **Frontend**: <http://localhost:3000>
- **Backend API**: <http://localhost:8000>
- **API Documentation**: <http://localhost:8000/docs>
- **Database**: `localhost:5432` (user: `plex`, password: see `secrets/postgres_password`)

## Poe the Poet Task Runner

This project uses **Poe the Poet** as a unified task runner to simplify and standardize development workflows. Instead of remembering different commands for backend and frontend, you can use consistent `poe` commands from anywhere in the project.

### Quick Start with Poe

```bash
# Complete development setup (new developers)
poe setup

# Start development environment
poe dev

# Run all quality checks (format, lint, type-check)
poe ci-quick

# Run all tests
poe test-all

# See all available tasks
poe --help
```

### Key Benefits

- **Unified Interface**: Same commands work for backend (Python) and frontend (TypeScript)
- **Parallel Execution**: Tasks run in parallel for better performance
- **Smart Workflows**: Conditional and chained task execution
- **Developer Friendly**: Interactive task selection and helpful descriptions
- **CI Integration**: Local simulation of CI/CD pipeline

### Common Workflows

```bash
# Daily development workflow
poe format lint type-check test-unit

# Pre-commit workflow (faster than full CI)
poe ci-quick

# Complete quality gate (like CI pipeline)
poe quality-gate

# Build and test Docker images locally
poe build-cicd
```

For the complete list of tasks and detailed usage, see **[Poe Task Reference](POE_TASK_REFERENCE.md)**.

## Git Workflow

### Branch Strategy

The project follows a feature branch workflow with strict main branch protection:

1. **Main Branch**: Always deployable, protected from direct commits
2. **Feature Branches**: All work done in feature branches (`feature/feature-name`)
3. **Pull Requests**: Required for all changes to main

### Creating a Feature Branch

```bash
# Ensure you're on main and up to date
git checkout main
git pull origin main

# Create and switch to feature branch
git checkout -b feature/your-feature-name

# Push branch to remote
git push -u origin feature/your-feature-name
```

### Making Changes

```bash
# Make your changes
# ... edit files ...

# Stage and commit
git add .
git commit -m "feat: add new playlist functionality"

# Push changes
git push
```

### Creating a Pull Request

1. Push your feature branch to the remote repository
2. Navigate to the Gitea web interface
3. Create a Pull Request from your feature branch to `main`
4. Ensure required CI checks pass (informational audit warnings do not block merge)
5. Request review from team members
6. Merge only after approval and passing CI

## Pre-commit Hooks

### Why Pre-commit is Recommended

Pre-commit hooks provide immediate feedback and prevent CI failures by running the same validation locally that runs in CI. Benefits:

- **Fast Feedback**: Catch issues before pushing
- **Consistent Quality**: Same tools and configurations as CI
- **Time Saving**: Avoid CI failure cycles
- **Team Standards**: Automatic code formatting and linting

### Installation and Setup

```bash
# Install pre-commit (if not already installed)
pip install pre-commit

# Install hooks for this repository
pre-commit install

# Optionally, run on all files initially
pre-commit run --all-files
```

### How It Works

Pre-commit automatically runs on every `git commit` and will:

- Format code (Prettier, Ruff)
- Check syntax (ESLint, Pyright)
- Validate files (YAML, TOML, trailing whitespace)
- Run quick tests and linting

If any hook fails, the commit is blocked until issues are fixed.

### Pre-commit is Optional

While strongly recommended, pre-commit is not required because:

- **CI Validation**: All the same checks run in CI
- **Developer Choice**: Some prefer manual tool usage
- **Learning**: Developers can run tools individually to understand them

However, without pre-commit, you'll need to manually run tools and may face CI failures.

## Manual Tool Usage

If you prefer not to use pre-commit, here's how to run each tool manually:

### Backend Tools (Python)

Navigate to the `backend/` directory for all backend commands.

#### Backend Code Formatting

```bash
# Format code with Ruff
uv run ruff format .

# Fix auto-fixable linting issues
uv run ruff check . --fix
```

#### Backend Type Checking

```bash
# Run Pyright type checker
uv run pyright
```

#### Backend Testing

```bash
# Run unit tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=src --cov-report=html

# Run integration tests
uv run pytest tests/integration/

# Run doctests
uv run xdoctest src/
```

#### Backend Documentation

```bash
# Check docstring style
uv run pydoclint --config=pyproject.toml src/
```

### Frontend Tools (TypeScript/Vue)

Navigate to the `frontend/` directory for all frontend commands.

#### Frontend Code Formatting

```bash
# Format code with Prettier
yarn format

# Check formatting
yarn format:check
```

#### Linting

```bash
# Run ESLint
yarn lint

# Fix auto-fixable ESLint issues
yarn lint:fix

# Check TypeScript documentation
yarn lint:tsdoc
```

#### Frontend Type Checking

```bash
# Run Vue TypeScript compiler
yarn type-check
```

#### Frontend Testing

```bash
# Run unit tests
yarn test

# Run unit tests with coverage
yarn test:coverage

# Run E2E tests (Playwright)
yarn test:e2e

# Run E2E tests in headless mode (CI-like, default)
yarn playwright test

# Run E2E tests with specific reporter
yarn playwright test --reporter=list

# Run E2E tests for specific browser
yarn playwright test --project=chromium
```

### Project-wide Tools

#### YAML/TOML Validation

```bash
# Check YAML files
pre-commit run check-yaml --all-files

# Check TOML files
pre-commit run check-toml --all-files
```

#### File Quality

```bash
# Fix trailing whitespace
pre-commit run trailing-whitespace --all-files

# Fix end-of-file issues
pre-commit run end-of-file-fixer --all-files
```

## CI/CD Pipeline

### Pipeline Overview

The canonical CI workflow is `.gitea/workflows/cicd.yaml`.

Current triggers:

- Push on `main` and `develop`
- Pull requests targeting `main` and `develop`
- Manual `workflow_dispatch`

Current dispatch inputs:

- `head_sha`: commit SHA to process
- `force_rebuild_base`: force base image publication

### Current Job Topology

The pipeline is intentionally staged so expensive image jobs run only after source checks and required gates pass:

1. Producer lane:
   `Build and Push CICD Images` computes base hash, checks registry, and publishes both CICD base and complete CICD images.
2. Source + audit lanes:
   `Source Checks` and `Dependency Audits (Informational)` run after producer completion. Audit failures are intentionally non-blocking so both frontend and backend audits always report.
3. Runtime image lanes:
   `Build Release Images` publishes `plex-playlist-backend-staging` and `plex-playlist-frontend-staging`; `Build Tester Images` publishes integration/e2e tester images; then `Production Images Complete` gates downstream runtime tests.
4. Runtime validation lanes:
   `Runtime Black-Box Integration Tests` and `End-to-End Tests` validate staged runtime artifacts.
5. Promotion lane:
   `Promote Release Images` runs only for automated `push` events on `main`. It retags validated staging artifacts to release repos (`plex-playlist-backend`, `plex-playlist-frontend`) with `latest`, `v<major>.<minor>.0`, `v<major>.<minor>.<patch>`, and `v<major>.<minor>.<patch>-<7-char-short-sha>` tags. If the `main` commit is untagged, CI creates the next patch tag automatically; if no prior semver tag exists, the bootstrap baseline is `v0.0.0`.
6. Postmortem lanes:
   targeted postmortem jobs run when key lanes fail to capture diagnostics even when primary jobs fail early.

### Reliability and Traceability Behavior

Current workflow behavior includes:

- registry auth realm host pinning from `WWW-Authenticate` challenge when registry tokens are issued from a different host
- bounded retry logic for docker login/pull/push operations in image lanes
- digest/tag contract checks for deployable image references before runtime black-box tests
- release-note summary generation in the promotion lane (commit bullets since the previous release tag, or from the `v0.0.0` bootstrap baseline on first release)
- context hydration for image-build lanes by copying `/workspace` from the published CICD image
- runner split between `ubuntu-act` and `ubuntu-act-8gb` based on lane resource requirements

For details of base/complete image build strategy, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).
For incident handling, see [CI/CD Troubleshooting](CICD_TROUBLESHOOTING_GUIDE.md).

### Renovate Manual Run Behavior

The Renovate workflow at `.gitea/workflows/renovate.yml` now treats manual dispatch as an explicit operator override:

- when triggered via `workflow_dispatch`, it forces out-of-window execution by setting `RENOVATE_IGNORE_SCHEDULE=true`
- it also sets `RENOVATE_SCHEDULE="at any time"` for the run
- it executes one single pass across enabled managers instead of splitting work into multiple manager batches

This keeps scheduled behavior unchanged while making manual runs deterministic and immediate.

### Operator Quick Reference

Use workflow dispatch when you need deterministic reruns on a specific commit:

```bash
# Example: rerun CI against an explicit commit
# input head_sha=<commit>

# Example: force base image republish
# input force_rebuild_base=true
```

Recommended rerun order during flaky infrastructure incidents:

1. `Build and Push CICD Images`
2. failing runtime image lane (`Build Release Images` or `Build Tester Images`)
3. downstream integration/e2e lanes
4. `Promote Release Images` (if staging validation succeeded but promotion failed)

### Local CI/CD Testing

Build and test CI/CD images locally:

```bash
# Build both base and complete images
./scripts/build-cicd-local.sh

# Build only base image
./scripts/build-cicd-local.sh --base-only

# Build only complete image (requires existing base)
./scripts/build-cicd-local.sh --complete-only

# Force rebuild with no cache
./scripts/build-cicd-local.sh --force --no-cache

# Test with custom SSH key
./scripts/build-cicd-local.sh --ssh-key ~/.ssh/custom_key
```

### CI/CD Design Principles

- **Multi-Stage Optimization**: Separate stable dependencies from project code
- **Intelligent Caching**: Base image cached when unchanged (hash-based detection)
- **Single Source of Truth**: All CI jobs use the same pre-commit hooks as local development
- **Parallel Execution**: Maximum efficiency with concurrent job execution
- **Fast Feedback**: Jobs fail fast on first error
- **Memory Efficiency**: Optimized for 4GB Raspberry Pi workers
- **Comprehensive Coverage**: Every aspect of code quality is validated

### Viewing CI Results

1. Navigate to your pull request in Gitea
2. Check the "Checks" tab for detailed results
3. Click on individual job names to see logs
4. All required jobs must pass before merging (informational dependency-audit findings are non-blocking)

## Branch Protection and Merge Requirements

### Main Branch Protection

The `main` branch is protected with the following requirements:

1. **No Direct Pushes**: All changes must come through pull requests
2. **CI Must Pass**: All required CI/CD jobs must pass (dependency audits are informational)
3. **Review Required**: At least one team member approval needed
4. **Up-to-date Branch**: Feature branch must be current with main

### Merge Process

1. Create pull request from feature branch
2. Wait for all CI checks to complete successfully
3. Address any CI failures by pushing fixes to the feature branch
4. Request and receive code review approval
5. Ensure branch is up-to-date with main
6. Merge pull request (available when required jobs pass; informational audit failures do not block merge)

### If CI Fails

When CI fails:

1. **Check Logs**: Review the failed job logs in Gitea
2. **Fix Locally**: Make corrections in your feature branch
3. **Test Locally**: Run the same tools locally or use pre-commit
4. **Push Fix**: Commit and push the fix
5. **Wait for CI**: New CI run will start automatically

### Common CI Failure Resolutions

```bash
# Format code issues
pre-commit run --all-files

# Type errors (backend)
cd backend && uv run pyright

# Type errors (frontend)
cd frontend && yarn type-check

# Test failures (backend)
cd backend && uv run pytest

# Test failures (frontend)
cd frontend && yarn test:coverage

# E2E test failures (frontend)
cd frontend && yarn test:e2e
```

## Best Practices

### Development Workflow

1. **Always use feature branches** - never commit directly to main
2. **Keep branches small** - easier to review and merge
3. **Write descriptive commit messages** - helps with project history
4. **Run tests locally** - catch issues before pushing
5. **Use pre-commit** - saves time and ensures quality

### Code Quality

1. **Follow existing patterns** - maintain consistency
2. **Write tests for new features** - maintain coverage thresholds
3. **Add docstrings** - especially for public APIs
4. **Type annotations** - required for all Python functions
5. **Meaningful variable names** - self-documenting code

### Performance Tips

1. **Use Docker efficiently** - leverage layer caching
2. **Run specific tests** - don't always run full suite during development
3. **Parallel development** - multiple developers can work simultaneously
4. **Resource monitoring** - watch Docker memory usage on constrained systems

---

For questions or issues with the development environment, please create an issue in the project repository or contact the development team.