feature/pp-58-runtime-image-contract (#68)
## Summary This PR tightens repository quality enforcement around markdown and documentation. It adds `markdownlint` to the `cicd-checks` workflow, expands pre-commit coverage so markdown files are checked repo-wide, and cleans up the PP-58 documentation set to keep it aligned with the new policy. ## What changed - Added a `Markdownlint Check` entry to `.gitea/workflows/cicd-checks.yaml` - Added `markdownlint` to pre-commit and widened prettier coverage to include markdown files across the repo - Updated `README.md` to satisfy markdownlint line-length rules - Normalized the PP-58 documentation set: - `docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - `docs/adr/ADR003-deployable_runtime_image_contract.md` - `docs/DEVELOPMENT.md` - `docs/CICD_MULTI_STAGE_BUILD.md` - `docs/CICD_TROUBLESHOOTING_GUIDE.md` - `docs/SECURE_DOCKER_CICD.md` ## Validation - `pre-commit run markdownlint --files README.md docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - `pre-commit run prettier --files README.md docs/DEPLOYABLE_RUNTIME_CONTRACT.md` - Workflow YAML validation returned no errors ## Notes This change does not alter application runtime behavior. It only strengthens CI and documentation quality enforcement. Co-authored-by: copilotcoder <copilotcoder@darkhelm.org> Reviewed-on: #68
This commit was merged in pull request #68.
This commit is contained in:
@@ -5,6 +5,7 @@ This document contains solutions to common issues with Gitea Actions CI/CD pipel
|
||||
## Critical Issue: Jobs Stuck in "Waiting" State Forever
|
||||
|
||||
### Symptoms
|
||||
|
||||
- Workflows are created but jobs show "Waiting" indefinitely
|
||||
- Runners are online and healthy
|
||||
- No tasks appear in `action_task` database table
|
||||
@@ -12,9 +13,11 @@ This document contains solutions to common issues with Gitea Actions CI/CD pipel
|
||||
- UI shows "Waiting" but database shows status 5 (cancelled)
|
||||
|
||||
### Root Cause
|
||||
|
||||
**Docker syntax in `runs-on` labels** causes Gitea Actions to immediately cancel jobs.
|
||||
|
||||
### Problem Syntax (BROKEN)
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
setup:
|
||||
@@ -26,6 +29,7 @@ jobs:
|
||||
```
|
||||
|
||||
### Solution Syntax (WORKING)
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
setup:
|
||||
@@ -37,7 +41,9 @@ jobs:
|
||||
```
|
||||
|
||||
### Why This Works
|
||||
|
||||
The runners are configured with Docker images in their labels:
|
||||
|
||||
```bash
|
||||
GITEA_RUNNER_LABELS=ubuntu-latest:docker://ubuntu:22.04,node-latest:docker://node:20-bookworm-slim,python-latest:docker://python:3.14-slim
|
||||
```
|
||||
@@ -47,11 +53,13 @@ So jobs still run in the correct Docker containers, but Gitea can properly parse
|
||||
### Diagnosis Steps
|
||||
|
||||
1. **Check if new runs are created:**
|
||||
|
||||
```sql
|
||||
SELECT id, status, title FROM action_run ORDER BY id DESC LIMIT 3;
|
||||
```
|
||||
|
||||
2. **Check job status and duration:**
|
||||
|
||||
```sql
|
||||
SELECT arj.id, arj.job_id, arj.status, ar.created, ar.updated, (ar.updated - ar.created) as duration_seconds
|
||||
FROM action_run_job arj
|
||||
@@ -60,21 +68,25 @@ WHERE ar.id = (SELECT MAX(id) FROM action_run);
|
||||
```
|
||||
|
||||
3. **Check if tasks are created:**
|
||||
|
||||
```sql
|
||||
SELECT * FROM action_task ORDER BY id DESC LIMIT 5;
|
||||
```
|
||||
|
||||
4. **Verify runners are online:**
|
||||
|
||||
```sql
|
||||
SELECT id, name, last_online, agent_labels FROM action_runner WHERE last_online > (EXTRACT(epoch FROM NOW()) - 300)::bigint;
|
||||
```
|
||||
|
||||
### Key Indicators
|
||||
|
||||
- **Duration = 0 seconds** → Immediate cancellation due to syntax issue
|
||||
- **Empty action_task table** → Jobs never converted to executable tasks
|
||||
- **Status 5 jobs with Status 7 dependents** → Setup job cancelled, others skipped
|
||||
|
||||
### Test Procedure
|
||||
|
||||
Create a minimal test workflow to isolate issues:
|
||||
|
||||
```yaml
|
||||
@@ -95,13 +107,17 @@ If this works but your main workflow doesn't, the issue is likely syntax-related
|
||||
## Other Common Issues
|
||||
|
||||
### Cache/UI Synchronization Problems
|
||||
|
||||
If UI shows different status than database:
|
||||
|
||||
1. Restart Gitea: `docker compose restart server`
|
||||
2. Clear browser cache
|
||||
3. Check database vs UI status discrepancies
|
||||
|
||||
### Stuck Runs from Previous Sessions
|
||||
|
||||
Clean up stuck runs:
|
||||
|
||||
```sql
|
||||
-- Clear stuck pending jobs
|
||||
UPDATE action_run_job SET status = 5 WHERE status IN (1, 2);
|
||||
@@ -109,7 +125,9 @@ UPDATE action_run SET status = 5 WHERE status IN (1, 2);
|
||||
```
|
||||
|
||||
### Runner Registration Issues
|
||||
|
||||
If runners show "unregistered runner" errors:
|
||||
|
||||
1. Delete runner registrations: `DELETE FROM action_runner;`
|
||||
2. Restart all runner containers
|
||||
3. Let them auto-register with fresh state
|
||||
@@ -117,6 +135,7 @@ If runners show "unregistered runner" errors:
|
||||
## Infrastructure Overview
|
||||
|
||||
### Current Setup
|
||||
|
||||
- **Gitea Server**: Docker container with PostgreSQL backend
|
||||
- **Runners**: 8 Raspberry Pi runners across 4 servers
|
||||
- pi-desktop: Pi 400 4GB (2 runners)
|
||||
@@ -125,16 +144,20 @@ If runners show "unregistered runner" errors:
|
||||
- zhokq: Pi 4B 8GB (2 runners)
|
||||
|
||||
### Runner Configuration
|
||||
|
||||
Each runner supports multiple Docker environments:
|
||||
|
||||
- `ubuntu-latest` → `ubuntu:22.04`
|
||||
- `python-latest` → `python:3.14-slim`
|
||||
- `node-latest` → `node:20-bookworm-slim`
|
||||
- `ubuntu-act` → `catthehacker/ubuntu:act-latest`
|
||||
|
||||
### Mirroring the `ubuntu-act` Runner Image
|
||||
|
||||
If GHCR pulls are flaky, mirror the runner image into your local registry and point the label at that mirror instead of the upstream tag.
|
||||
|
||||
Example mirror flow:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/catthehacker/ubuntu:act-latest
|
||||
docker tag ghcr.io/catthehacker/ubuntu:act-latest kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
@@ -142,6 +165,7 @@ docker push kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
```
|
||||
|
||||
Recommended runner label once mirrored:
|
||||
|
||||
```bash
|
||||
GITEA_RUNNER_LABELS=ubuntu-latest:docker://ubuntu:22.04,node-latest:docker://node:20-bookworm-slim,python-latest:docker://python:3.14-slim,ubuntu-act:docker://kankali.darkhelm.lan:3001/darkhelm.org/act-ubuntu:act-latest
|
||||
```
|
||||
@@ -164,7 +188,9 @@ source scripts/gitea-actions/check_runner_images.xsh
|
||||
```
|
||||
|
||||
### Workflow Design
|
||||
|
||||
Multi-stage pipeline with artifact passing:
|
||||
|
||||
1. **Setup**: Checkout code, create artifacts
|
||||
2. **Parallel Setup**: Backend (Python/uv) + Frontend (Node.js/Yarn)
|
||||
3. **Parallel Tests**: Backend tests + Frontend tests
|
||||
@@ -179,5 +205,5 @@ Multi-stage pipeline with artifact passing:
|
||||
|
||||
---
|
||||
|
||||
*Last updated: June 2, 2026*
|
||||
*Issue resolved after extensive database-level debugging and syntax isolation*
|
||||
_Last updated: June 2, 2026_
|
||||
_Issue resolved after extensive database-level debugging and syntax isolation_
|
||||
|
||||
Reference in New Issue
Block a user