plex-playlist/docs/DEVELOPMENT.md

# Development Environment Setup and Workflow

This document outlines how to set up your development environment and work with the plex-playlist project.

## Table of Contents

1. [Quick Start](#quick-start)
2. [Development Environment Setup](#development-environment-setup)
3. [Poe the Poet Task Runner](#poe-the-poet-task-runner)
4. [Git Workflow](#git-workflow)
5. [Pre-commit Hooks](#pre-commit-hooks)
6. [Manual Tool Usage](#manual-tool-usage)
7. [CI/CD Pipeline](#cicd-pipeline)
8. [Branch Protection and Merge Requirements](#branch-protection-and-merge-requirements)

## Related Documentation

- **[Poe Task Reference](POE_TASK_REFERENCE.md)** - Complete guide to unified development tasks
- **[CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md)** - Technical details of the optimized build system
- **[CI/CD Troubleshooting](GITEA_ACTIONS_TROUBLESHOOTING.md)** - Common issues and solutions
- **[Secure Docker CI/CD](SECURE_DOCKER_CICD.md)** - Security considerations and practices

## Quick Start

```bash
# Clone the repository
git clone ssh://git@dogar.darkhelm.org:2222/DarkHelm.org/plex-playlist.git
cd plex-playlist

# Complete setup with Poe the Poet (recommended)
cd backend && uv sync --dev && cd ..
poe setup

# OR manual setup
docker compose -f compose.dev.yml up -d
pip install pre-commit && pre-commit install

# Create a feature branch and start developing
git checkout -b feature/your-feature-name

# Use Poe for all development tasks
poe format lint type-check test-unit
```

## Development Environment Setup

### Local Development with Docker Compose

The project uses `compose.dev.yml` for local development, which provides:

- **Backend**: FastAPI application with hot reload
- **Frontend**: Vite development server with hot module replacement
- **Database**: PostgreSQL with persistent data
- **Development Tools**: Pre-configured with all necessary dependencies

```bash
# Start all services
docker compose -f compose.dev.yml up -d

# View logs
docker compose -f compose.dev.yml logs -f

# Stop services
docker compose -f compose.dev.yml down

# Rebuild after dependency changes
docker compose -f compose.dev.yml up -d --build
```

### Service Access

- **Frontend**: http://localhost:3000
- **Backend API**: http://localhost:8000
- **API Documentation**: http://localhost:8000/docs
- **Database**: localhost:5432 (user: `plex`, password: see `secrets/postgres_password`)

## Poe the Poet Task Runner

This project uses **Poe the Poet** as a unified task runner to simplify and standardize development workflows. Instead of remembering different commands for backend and frontend, you can use consistent `poe` commands from anywhere in the project.

### Quick Start with Poe

```bash
# Complete development setup (new developers)
poe setup

# Start development environment
poe dev

# Run all quality checks (format, lint, type-check)
poe ci-quick

# Run all tests
poe test-all

# See all available tasks
poe --help
```

### Key Benefits

- **Unified Interface**: Same commands work for backend (Python) and frontend (TypeScript)
- **Parallel Execution**: Tasks run in parallel for better performance
- **Smart Workflows**: Conditional and chained task execution
- **Developer Friendly**: Interactive task selection and helpful descriptions
- **CI Integration**: Local simulation of CI/CD pipeline

### Common Workflows

```bash
# Daily development workflow
poe format lint type-check test-unit

# Pre-commit workflow (faster than full CI)
poe ci-quick

# Complete quality gate (like CI pipeline)
poe quality-gate

# Build and test Docker images locally
poe build-cicd
```

For the complete list of tasks and detailed usage, see **[Poe Task Reference](POE_TASK_REFERENCE.md)**.

## Git Workflow

### Branch Strategy

The project follows a feature branch workflow with strict main branch protection:

1. **Main Branch**: Always deployable, protected from direct commits
2. **Feature Branches**: All work done in feature branches (`feature/feature-name`)
3. **Pull Requests**: Required for all changes to main

### Creating a Feature Branch

```bash
# Ensure you're on main and up to date
git checkout main
git pull origin main

# Create and switch to feature branch
git checkout -b feature/your-feature-name

# Push branch to remote
git push -u origin feature/your-feature-name
```

### Making Changes

```bash
# Make your changes
# ... edit files ...

# Stage and commit
git add .
git commit -m "feat: add new playlist functionality"

# Push changes
git push
```

### Creating a Pull Request

1. Push your feature branch to the remote repository
2. Navigate to the Gitea web interface
3. Create a Pull Request from your feature branch to `main`
4. Ensure all CI checks pass (100% green required)
5. Request review from team members
6. Merge only after approval and passing CI

## Pre-commit Hooks

### Why Pre-commit is Recommended

Pre-commit hooks provide immediate feedback and prevent CI failures by running the same validation locally that runs in CI. Benefits:

- **Fast Feedback**: Catch issues before pushing
- **Consistent Quality**: Same tools and configurations as CI
- **Time Saving**: Avoid CI failure cycles
- **Team Standards**: Automatic code formatting and linting

### Installation and Setup

```bash
# Install pre-commit (if not already installed)
pip install pre-commit

# Install hooks for this repository
pre-commit install

# Optionally, run on all files initially
pre-commit run --all-files
```

### How It Works

Pre-commit automatically runs on every `git commit` and will:
- Format code (Prettier, Ruff)
- Check syntax (ESLint, Pyright)
- Validate files (YAML, TOML, trailing whitespace)
- Run quick tests and linting

If any hook fails, the commit is blocked until issues are fixed.

### Pre-commit is Optional

While strongly recommended, pre-commit is not required because:
- **CI Validation**: All the same checks run in CI
- **Developer Choice**: Some prefer manual tool usage
- **Learning**: Developers can run tools individually to understand them

However, without pre-commit, you'll need to manually run tools and may face CI failures.

## Manual Tool Usage

If you prefer not to use pre-commit, here's how to run each tool manually:

### Backend Tools (Python)

Navigate to the `backend/` directory for all backend commands.

#### Code Formatting
```bash
# Format code with Ruff
uv run ruff format .

# Fix auto-fixable linting issues
uv run ruff check . --fix
```

#### Type Checking
```bash
# Run Pyright type checker
uv run pyright
```

#### Testing
```bash
# Run unit tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=src --cov-report=html

# Run integration tests
uv run pytest tests/integration/

# Run doctests
uv run xdoctest src/
```

#### Documentation
```bash
# Check docstring style
uv run darglint src/
```

### Frontend Tools (TypeScript/Vue)

Navigate to the `frontend/` directory for all frontend commands.

#### Code Formatting
```bash
# Format code with Prettier
yarn format

# Check formatting
yarn format:check
```

#### Linting
```bash
# Run ESLint
yarn lint

# Fix auto-fixable ESLint issues
yarn lint:fix

# Check TypeScript documentation
yarn lint:tsdoc
```

#### Type Checking
```bash
# Run Vue TypeScript compiler
yarn type-check
```

#### Testing
```bash
# Run unit tests
yarn test

# Run unit tests with coverage
yarn test:coverage

# Run E2E tests (Playwright)
yarn test:e2e

# Run E2E tests in headless mode (CI-like, default)
yarn playwright test

# Run E2E tests with specific reporter
yarn playwright test --reporter=list

# Run E2E tests for specific browser
yarn playwright test --project=chromium
```

### Project-wide Tools

#### YAML/TOML Validation

```bash
# Check YAML files
pre-commit run check-yaml --all-files

# Check TOML files
pre-commit run check-toml --all-files
```

#### File Quality

```bash
# Fix trailing whitespace
pre-commit run trailing-whitespace --all-files

# Fix end-of-file issues
pre-commit run end-of-file-fixer --all-files
```

## CI/CD Pipeline

### Pipeline Overview

The CI/CD pipeline uses a **multi-stage build architecture** for optimal performance:

- **Stage 1**: Build base image (system dependencies, Python, Node.js) - **cached across runs**
- **Stage 2**: Build complete image (project code and dependencies) - **rebuilt every time**

Pipeline triggers:
- Push to any branch
- Pull requests to `main` or `develop`

### Multi-Stage Build Benefits ✅ **VALIDATED SUCCESSFUL**

**Performance Gains**:
- **85% build time improvement**: 3-5 minutes (down from 15-25 minutes)
- Base image cached when `Dockerfile.cicd-base` unchanged (~95% of runs)
- **100% success rate** achieved with optimized dependency management
- Raspberry Pi 4GB workers handle builds efficiently with resource optimization

**Architecture**:
- `cicd-base:latest` - System dependencies (Python 3.13, Node.js 24, build tools, pre-installed dev packages)
- `cicd:latest` - Complete environment (project code + optimized dependency installation)

**Recent Optimizations** (November 2025):
- **Dependency-first build pattern** prevents cache invalidation on code changes
- **Yarn PnP state regeneration** ensures reliable frontend builds
- **Network-resilient E2E testing** with simplified Docker operations
- **Memory-optimized frontend installations** with proper swap configuration

For detailed technical information, see [CI/CD Multi-Stage Build Architecture](CICD_MULTI_STAGE_BUILD.md).

### Pipeline Jobs

All jobs run in parallel after the setup phases:

1. **Setup Base**: Builds and pushes base Docker image (conditional)
2. **Setup Complete**: Builds and pushes complete CI/CD Docker image
3. **Code Quality**:
   - Trailing whitespace check
   - End-of-file formatting
   - YAML syntax validation
   - TOML syntax validation
4. **Backend Validation**:
   - Ruff formatting check
   - Ruff linting
   - Pyright type checking
   - Darglint docstring validation
   - Unit tests with coverage
   - Integration tests
   - Doctests (xdoctest)
5. **Frontend Validation**:
   - Prettier formatting check
   - ESLint linting
   - TypeScript compilation
   - Unit tests with coverage
   - E2E tests (Playwright)

### Local CI/CD Testing

Build and test CI/CD images locally:

```bash
# Build both base and complete images
./scripts/build-cicd-local.sh

# Build only base image
./scripts/build-cicd-local.sh --base-only

# Build only complete image (requires existing base)
./scripts/build-cicd-local.sh --complete-only

# Force rebuild with no cache
./scripts/build-cicd-local.sh --force --no-cache

# Test with custom SSH key
./scripts/build-cicd-local.sh --ssh-key ~/.ssh/custom_key
```

### CI/CD Design Principles

- **Multi-Stage Optimization**: Separate stable dependencies from project code
- **Intelligent Caching**: Base image cached when unchanged (hash-based detection)
- **Single Source of Truth**: All CI jobs use the same pre-commit hooks as local development
- **Parallel Execution**: Maximum efficiency with concurrent job execution
- **Fast Feedback**: Jobs fail fast on first error
- **Memory Efficiency**: Optimized for 4GB Raspberry Pi workers
- **Comprehensive Coverage**: Every aspect of code quality is validated

### Viewing CI Results

1. Navigate to your pull request in Gitea
2. Check the "Checks" tab for detailed results
3. Click on individual job names to see logs
4. All jobs must pass (100% green) before merging

## Branch Protection and Merge Requirements

### Main Branch Protection

The `main` branch is protected with the following requirements:

1. **No Direct Pushes**: All changes must come through pull requests
2. **CI Must Pass**: All CI/CD jobs must be 100% green
3. **Review Required**: At least one team member approval needed
4. **Up-to-date Branch**: Feature branch must be current with main

### Merge Process

1. Create pull request from feature branch
2. Wait for all CI checks to complete successfully
3. Address any CI failures by pushing fixes to the feature branch
4. Request and receive code review approval
5. Ensure branch is up-to-date with main
6. Merge pull request (only available when all requirements met)

### If CI Fails

When CI fails:

1. **Check Logs**: Review the failed job logs in Gitea
2. **Fix Locally**: Make corrections in your feature branch
3. **Test Locally**: Run the same tools locally or use pre-commit
4. **Push Fix**: Commit and push the fix
5. **Wait for CI**: New CI run will start automatically

### Common CI Failure Resolutions

```bash
# Format code issues
pre-commit run --all-files

# Type errors (backend)
cd backend && uv run pyright

# Type errors (frontend)
cd frontend && yarn type-check

# Test failures (backend)
cd backend && uv run pytest

# Test failures (frontend)
cd frontend && yarn test:coverage

# E2E test failures (frontend)
cd frontend && yarn test:e2e
```

## Best Practices

### Development Workflow

1. **Always use feature branches** - never commit directly to main
2. **Keep branches small** - easier to review and merge
3. **Write descriptive commit messages** - helps with project history
4. **Run tests locally** - catch issues before pushing
5. **Use pre-commit** - saves time and ensures quality

### Code Quality

1. **Follow existing patterns** - maintain consistency
2. **Write tests for new features** - maintain coverage thresholds
3. **Add docstrings** - especially for public APIs
4. **Type annotations** - required for all Python functions
5. **Meaningful variable names** - self-documenting code

### Performance Tips

1. **Use Docker efficiently** - leverage layer caching
2. **Run specific tests** - don't always run full suite during development
3. **Parallel development** - multiple developers can work simultaneously
4. **Resource monitoring** - watch Docker memory usage on constrained systems

---

For questions or issues with the development environment, please create an issue in the project repository or contact the development team.