HealthArchive Documentation Hub

HealthArchive is a monorepo app project with a separate datasets repo that archives Canadian health government websites for research and accountability. This page helps you navigate the documentation and code boundaries.

🚀 Quick Start by Role

Choose your entry point based on what you want to do:

👤 I'm an Operator

Goal: Deploy, monitor, and maintain the production system

Start Here: 1. Production Runbook - Complete production setup guide 2. Operator Responsibilities - Must-do checklist 3. Deploy & Verify - Safe deployment process 4. Incident Response - Emergency procedures

Key Resources: - Ops Cadence Checklist - Daily/weekly/quarterly tasks - Monitoring Checklist - Set up alerts and checks - All Playbooks - 30+ operational procedures

💻 I'm a Developer

Goal: Contribute code, fix bugs, add features

Start Here: 1. Quick Start Guide - Get running in 5 minutes 2. Your First Contribution - Step-by-step tutorial 3. Dev Environment Setup - Detailed local setup 4. Live Testing Guide - Run the full pipeline locally

Key Resources: - Architecture Walkthrough - Visual guide to how it all works - Architecture Deep Dive - Complete technical reference - Testing Guidelines - How to write and run tests - Contributing Guide - Code standards and workflow

🔧 I'm an API Consumer / Researcher

Goal: Search the archive and retrieve historical snapshots

Start Here: 1. API Consumer Guide - Complete API walkthrough with examples 2. Interactive API Docs - Try the API in your browser 3. API Reference - Full OpenAPI specification

Quick API Test:

curl "https://api.healtharchive.ca/api/search?q=vaccines&source=hc"

Key Resources: - Dataset Downloads: healtharchive-datasets - Bulk metadata exports - Data Handling Policy - Retention and privacy - Live Site: healtharchive.ca - Web interface

📚 I'm a Student / New to the Project

Goal: Learn how HealthArchive works

Recommended Reading Order: 1. Quick Start - High-level overview 2. Architecture Walkthrough - Follow a page from crawl to search 3. Architecture Reference - Deep technical details 4. Documentation Guidelines - How docs stay organized

Tutorials: - Your First Contribution - Hands-on coding tutorial - Debugging a Failed Crawl - Practical troubleshooting - Live Testing - Run it yourself locally

📦 Monorepo + Dataset Architecture

HealthArchive keeps app code in one repository and dataset releases in a separate repository:

🔙 Backend (This Repo Root)

Purpose: API, crawler, database, operations, and all internal infrastructure

Location: github.com/jerdaw/healtharchive

Documentation: docs.healtharchive.ca (you are here)

What Lives Here: - ✅ Crawler (archive_tool) and job orchestration - ✅ Database models and indexing pipeline - ✅ RESTful JSON API (FastAPI) - ✅ Operations runbooks and playbooks - ✅ Deployment guides and systemd units - ✅ Architecture and developer docs - ✅ Decision records and incident notes

Tech Stack: Python, FastAPI, SQLAlchemy, PostgreSQL, Docker, systemd

🌐 Frontend (frontend/)

Purpose: Public-facing website and user interface

Location: frontend/ inside the app monorepo

Live Site: healtharchive.ca

What Lives Here: - ✅ Next.js web application (search UI, snapshot viewer) - ✅ Public content (status page, impact statement, changelog) - ✅ Internationalization (i18n) - English and French - ✅ UI/UX documentation

Tech Stack: Next.js 16, React, TypeScript, Tailwind CSS

Canonical Frontend Docs: frontend/docs/

📊 Datasets

Purpose: Versioned, citable metadata-only dataset releases

Location: github.com/jerdaw/healtharchive-datasets

What Lives Here: - ✅ Snapshot metadata exports (JSON/CSV) - ✅ Checksums and integrity manifests - ✅ Dataset release documentation - ✅ Dataset integrity policies

Why Separate?: Enables versioned, citable releases independent of code changes

Datasets Docs in This Repo: datasets-external/README.md (pointer only)

Canonical Datasets Docs: datasets/README.md

🗺️ Where Things Live (Source of Truth Map)

Principle: Each doc has one canonical source. Other repos link to it.

🔗 Linking Conventions

In GitHub Issues/PRs

Use full GitHub URLs:

See the [production runbook](https://github.com/jerdaw/healtharchive/blob/main/docs/deployment/production-single-vps.md)

In Documentation

For docs users: Use the docs site URLs:

See the [Production Runbook](https://docs.healtharchive.ca/deployment/production-single-vps/)

For frontend paths in the monorepo or cross-repo references: Use full GitHub URLs:

Frontend changelog process: [changelog-process.md](https://github.com/jerdaw/healtharchive/blob/main/frontend/docs/changelog-process.md)

Local Development (Monorepo Workspace)

Recommended local layout:

/home/user/healtharchive/
├── healtharchive/
└── healtharchive-datasets/

Some docs use workspace paths like frontend/... or healtharchive-datasets/... for convenience.

Note: These paths only work in local workspaces, not on GitHub.

🏗️ System Architecture (High Level)

graph TB
    subgraph "HealthArchive Backend"
        CLI[CLI Commands] -->|Create| Jobs[(Database)]
        Worker[Worker Process] -->|Poll| Jobs
        Worker -->|Execute| Crawler[Archive Tool]
        Crawler -->|Docker| Zimit[Zimit Crawler]
        Zimit -->|Write| WARC[WARC Files]
        Worker -->|Index| WARC
        WARC -->|Extract| Snapshots[(Snapshots)]
        API[FastAPI API] -->|Query| Snapshots
    end

    subgraph "HealthArchive Frontend"
        UI[Next.js UI] -->|API Calls| API
        API -->|JSON| UI
        UI -->|Display| Users[End Users]
    end

    subgraph "HealthArchive Datasets"
        Export[Export Script] -->|Read| Snapshots
        Export -->|Write| Datasets[Dataset Files]
        Researchers -->|Download| Datasets
    end

Data Flow: 1. Crawl: CLI creates job → Worker runs crawler → Docker writes WARCs 2. Index: Worker parses WARCs → Extracts text → Stores snapshots in DB 3. Serve: API queries DB → Returns JSON → Frontend displays results 4. Export: Scripts export metadata → Version as datasets → Researchers download

See: Architecture Walkthrough for detailed data flow

📖 Documentation Structure

This documentation follows the Diátaxis framework for clarity:

Navigation: Use the sidebar to explore by category

🆘 Getting Help

By Issue Type

Community

• GitHub Discussions: app monorepo
• Issues: app monorepo | datasets
• Contributor Guide: contributing.md

🔄 Documentation Updates

Found something wrong? Documentation lives in git and accepts pull requests!

1. Backend/docs hub: Edit files in docs/
2. Frontend docs: Edit files in frontend/docs/
3. Datasets docs: Edit datasets README

Guidelines: Documentation Guidelines

📊 Project Status

Latest Incidents: See operations/incidents/

🎯 Next Steps

Based on your role, here's what to do next:

Operators

1. ✅ Review Production Runbook
2. ✅ Complete Monitoring Checklist
3. ✅ Bookmark Incident Response

Developers

1. ✅ Complete Quick Start
2. ✅ Follow Your First Contribution
3. ✅ Read Architecture Walkthrough

Researchers

1. ✅ Read API Consumer Guide
2. ✅ Try Interactive API Docs
3. ✅ Explore Datasets

📚 Essential Documentation Index

Getting Started: - Quick Start - Project Overview (you are here)

For Operators: - Production Runbook - All Playbooks - Ops Cadence

For Developers: - First Contribution - Architecture Guide - Dev Setup

For Researchers: - API Guide - API Reference - Datasets

Reference: - Documentation Guidelines - Decision Records - Roadmaps

💡 About This Documentation

This documentation portal is currently built with MkDocs Material and deployed to docs.healtharchive.ca.

Source: docs/ Build: make docs-build Serve Locally: make docs-serve

Last Updated: Auto-generated on every push to main