Operational Thresholds and Tuning Guide

This document centralizes all operational thresholds used in HealthArchive automation, monitoring, and safeguards.

Last updated: 2026-02-06

Overview

HealthArchive uses conservative thresholds to prevent runaway automation and ensure system stability. All automation is opt-in via sentinel files and includes multiple safety caps.

General principles: - Automation is safe-by-default (dry-run unless explicitly enabled) - Rate limits prevent flapping (cooldowns + per-hour/day caps) - Deploy locks prevent conflicts during maintenance - All thresholds are tunable but have sensible defaults

Disk Management Thresholds

Worker Pre-Crawl Disk Check

Tuning guidance: - Lower to 80% if crawls are consistently small (e.g., test jobs) - Raise to 88% if disk is oversized and buffer is excessive - Don't raise above 90% - leaves too little margin for error

Related alerts: HealthArchiveDiskUsageHigh (warning), HealthArchiveDiskUsageCritical

Alerting Thresholds

Tuning guidance: - Warning duration (30min) gives time to react without false positives - Critical threshold (92%) leaves ~6GB for emergency response - Don't raise critical above 95% - risk of sudden disk-full

See: docs/operations/disk-baseline-and-cleanup.md (current baseline + cleanup posture)

Cleanup Automation

Tuning guidance: - Increase threshold_max_jobs_per_run to 7-10 only if disk pressure is chronic and the cleanup is consistently safe - Decrease min_age_days to 7 if disk pressure is chronic - Increase keep_latest_per_source to 3+ if operators need more history

Implementation notes: - Weekly cleanup: docs/deployment/systemd/healtharchive-cleanup-automation.service + docs/deployment/systemd/healtharchive-cleanup-automation.timer - Disk threshold cleanup: docs/deployment/systemd/healtharchive-disk-threshold-cleanup.service + docs/deployment/systemd/healtharchive-disk-threshold-cleanup.timer (runs every 30 min; no-op when below threshold)

Crawl Recovery Thresholds

Stall Detection

Tuning guidance: - Lower to 1800s (30min) for fast sites (e.g., small test crawls) - Raise to 5400s (90min) or 7200s (120min) for very slow sites or flaky networks - Don't lower below 1800s (30min) — risks false positives during normal slow periods - Automation-first alerting note: HealthArchiveCrawlStalled should be treated as a post-watchdog/manual-review signal only if the crawl auto-recover watchdog is enabled and its metrics are fresh in Prometheus.

Recovery Rate Limits

Tuning guidance: - Increase per-job cap to 5 for known-flaky sources (e.g., sites with frequent timeouts) - Disable soft recovery (--no-soft-recover-when-guarded) only for debugging

Recovery enhancements (auto-applied): - enable_adaptive_restart=True - max_container_restarts floor from source profile (hc=24, phac=30, cihr=20) - See: scripts/vps-crawl-auto-recover.py (_ensure_recovery_tool_options)

Degraded Throughput Detection (observe-first)

Tuning guidance: - Keep --degraded-action observe until the 6–24h long-window reassessment confirms persistent degradation. - If false positives occur, increase consecutive runs before lowering threshold strictness. - If persistent low throughput remains after scope reconciliation, use controlled restart playbooks instead of immediate automatic recoveries.

Crawl Auto-Start (Queue Fill)

When enabled, the crawl auto-recover watchdog can also act as a queue fill mechanism: if there are no stalled jobs, but the annual campaign is running fewer than N jobs, it can auto-start one queued/retryable annual job via systemd-run.

This is designed to avoid the operational failure mode where a stalled job gets marked retryable but never returns to running because the worker is already busy with another crawl.

Auto-Start Thresholds

Implementation notes: - Auto-start only considers jobs with config.campaign_kind="annual" and matching config.campaign_year. - Auto-start runs the job using systemd-run (detached) and applies Docker caps via env vars: - HEALTHARCHIVE_DOCKER_CPU_LIMIT (default: 1.0; configurable via --start-docker-cpu-limit) - HEALTHARCHIVE_DOCKER_MEMORY_LIMIT (default: 3g; configurable via --start-docker-memory-limit) - Alerting integration note: worker-down notifications can be delayed/suppressed based on worker auto-start automation only when healtharchive-worker-auto-start.timer is enabled and healtharchive_worker_auto_start.prom metrics are fresh. - Worker auto-start watchdog now includes stale running-row reconciliation: - --reconcile-running-drift --reconcile-older-than-minutes 10 --reconcile-limit 10 - It only reconciles rows with no active crawl process detected for their output dir.

Storage Hot-Path Recovery Thresholds

Stale Mount Detection

Probed locations: 1. Running job output dirs 2. Next queued/retryable job output dirs (prevents retry storms) 3. Manifest hot paths (tiering bind mounts)

Tuning guidance: - Don't lower min_failure_age - transient failures are common - Don't reduce confirm_runs - single observations may be false positives

Recovery Rate Limits

Tuning guidance: - Increase cooldown to 30min if recovery attempts fail repeatedly - Increase hourly/daily caps cautiously - investigate root cause instead - Don't bypass caps in automation - they prevent pathological loops - Alerting integration note: job-level Errno 107 unreadable/writability symptom alerts can be demoted/suppressed only if the storage hot-path watchdog is enabled and healtharchive_storage_hotpath_auto_recover.prom metrics are fresh.

Persistent Failed-Apply Alert

Tuning guidance: - Keep age threshold at 24h until burn-in confirms clear signal quality. - If alert is too noisy, investigate watchdog state churn first; do not hide failures by increasing to multiple days. - Escalate to critical only after at least one week of clean behavior and verified operator response path.

SSHFS Mount Options

Tuning guidance: - Lower ServerAliveInterval to 10s if mounts go stale frequently - Don't raise ServerAliveCountMax - delays detection of stale connections - reconnect should always be enabled

Known issue: Stale mounts still occur despite hardened options (root cause under investigation).

See: docs/planning/implemented/2026-02-01-operational-resilience-improvements.md

Deploy Lock Protection

Tuning guidance: - Increase max age if deploys routinely take >2 hours (investigate why) - Don't decrease below 1 hour - normal deploys can take 30-45 minutes

Infra Error Cooldown

Tuning guidance: - Increase to 20min if infrastructure is persistently unstable - Decrease to 5min if false positives are common (careful!)

See: docs/planning/implemented/2026-01-24-infra-error-and-storage-hotpath-hardening.md

Annual Output Dir Writability Probe

These checks detect permission drift for queued/retryable annual jobs before a crawl attempt consumes retries.

Triage signals:

• ..._errno == 13: permission drift (output dir not writable for worker user)
• ..._errno == 107: stale sshfs hot path (follow storage hot-path recovery)

Archive Tool (Crawler) Adaptive Thresholds

Annual Per-Source Profiles

Annual jobs are source-tuned (not one-size-fits-all). Canonical values live in src/ha_backend/job_registry.py and are reconciled by scripts/vps-crawl-auto-recover.py during recovery/auto-start flows.

Tuning guidance: - Change source profiles in job_registry first; keep watchdog reconciliation aligned. - For completeness-first posture, prefer targeted, versioned source-profile or scope changes in the repo before repeated manual recoveries. - Only increase tolerance (stall/restart budget) when evidence shows the crawl is still making useful progress and the issue is intermittent rather than continuous thrash. - Only reduce thresholds when repeated evidence shows low false-positive restart risk. - For recurring canada.ca transport errors, prefer a source-profile browser compatibility change (for example Browsertrix --extraChromeArgs) before expanding exclusions or repeatedly recycling the same job config.

One-Time Annual Backfill/Reconciliation

When migrating an existing campaign from shared defaults to per-source tuning, reconcile existing annual jobs in-place:

# Review changes first (dry-run)
healtharchive reconcile-annual-tool-options --year 2026

# Apply changes
healtharchive reconcile-annual-tool-options --year 2026 --apply

What this command does: - Reconciles baseline annual values to source profile values (hc, phac, cihr) - Preserves explicit non-baseline overrides (except restart floor enforcement) - Ensures annual safety defaults (enable_monitoring, enable_adaptive_restart, skip_final_build, docker_shm_size=1g)

See: src/archive_tool/constants.py, scripts/vps-crawl-auto-recover.py

Summary Table: All Thresholds

Tuning Workflow

When adjusting thresholds:

1. Document the change: Update this file with new values and rationale
2. Test in staging (if available): Validate behavior before production
3. Monitor metrics: Watch Prometheus/Grafana for impact
4. Iterate conservatively: Small adjustments, measure, repeat
5. Update automation: Adjust watchdog caps if needed

Anti-patterns: - Disabling safety caps to "fix" underlying issues - Tuning based on single incidents without trend analysis - Raising thresholds indefinitely instead of fixing root cause

Related Documentation

• Disk baseline: docs/operations/disk-baseline-and-cleanup.md
• Alerting strategy: docs/operations/monitoring-and-alerting.md
• Stale mount playbook: docs/operations/playbooks/storage/storagebox-sshfs-stale-mount-recovery.md
• Crawl stall playbook: docs/operations/playbooks/crawl/crawl-stalls.md
• Operational resilience improvements: docs/planning/implemented/2026-02-01-operational-resilience-improvements.md