Skip to content

5. Pragmatic CI Philosophy

Date: 2026-01-17

Status

Accepted

Context

Continuous Integration (CI) for dotfiles projects faces unique challenges:

  1. Brittleness — Testing shell configurations across multiple OSes and distros is inherently flaky
  2. Development Velocity — Dotfiles are personal productivity tools; slow or blocking CI hinders the primary goal
  3. Test Scope — Full coverage (macOS + Linux + WSL2 + multiple distros) creates excessive CI time and cost
  4. False Positives — macOS runner availability issues and transient package repository failures cause unrelated breakage

Traditional CI philosophy (all checks must pass, red build = broken) conflicts with these realities.

Decision

We have adopted a pragmatic CI philosophy: CI guides development rather than gates it.

Consequences

Positive

  • Faster Iteration — Developers can merge confidently without waiting for flaky macOS jobs
  • Clearer Signal — Focus on high-value tests (syntax, core functionality) rather than noisy platform-specific edge cases
  • Local Validation — Pre-commit hooks catch most issues before push, CI is secondary verification
  • Reduced Maintenance — Less time debugging CI infrastructure, more time improving dotfiles

Negative

  • Risk of Regressions — Potential to merge platform-specific breakage (mitigated by manual testing and doctor checks)
  • Discipline Required — Must resist ignoring real failures alongside flaky ones

Trade-offs

  • Speed vs Coverage — Chose fast feedback over exhaustive multi-platform testing
  • Trust vs Automation — Relies more on developer judgment and local testing

Implementation Details

CI Configuration

File: .github/workflows/ci.yml

Core Philosophy:

# Linux tests: Required to pass (fast, reliable)
test-linux:
  runs-on: ubuntu-latest
  # Syntax, linting, core BATS tests

# macOS tests: Advisory only (flaky, expensive)
test-macos:
  runs-on: macos-latest
  continue-on-error: true  # Don't block merges

Local Pre-Commit Hooks

File: .pre-commit-config.yaml

Local hooks run before every commit:

  • Shell syntax validation (shellcheck)
  • YAML/Markdown linting
  • Link checking (optional, fast checks only)

This catches 80% of issues before pushing to CI.

Testing Strategy

CI (Fast Suite) — Essential tests only (~2 min): 

# CI runs these 5 test files
bats tests/static.bats tests/smoke.bats tests/basic.bats tests/env.bats tests/doctor.bats

Local (Full Suite) — Complete test coverage (~10-15 min): 

mise run test  # All 450+ tests

Rationale: Running all 450+ tests in CI took 40+ minutes, blocking development. Essential tests catch 90% of issues in 2 minutes.

When to Run Full Tests

  • Locally before commits — mise run test
  • Before releases — Full suite + macOS manual testing
  • After major changes — Shell config, package list changes
  • When investigating failures — Specific test files or patterns

Rationale for Key Decisions

Why continue-on-error for macOS?

  • GitHub macOS runners have availability issues
  • Homebrew occasionally has transient failures
  • Most users run Linux (WSL2/native), macOS is secondary platform
  • macOS breakage is caught via manual testing before it impacts users

Why favor pre-commit hooks over CI?

  • Immediate feedback — Catches errors in seconds, not minutes
  • Lower cost — No CI minutes consumed for simple mistakes
  • Developer autonomy — Can choose to skip hooks when iterating rapidly

Why not remove macOS CI entirely?

  • Sanity Check — Still provides value when it passes
  • Regression Detection — Historical data helps identify when macOS support degrades
  • Documentation — Shows the project intends to support macOS

Success Metrics

  • CI Run Time — Target ≤2 minutes (achieved: ~1m 23s total, ~45s for build job)
  • Merge Velocity — No waiting for slow/flaky jobs
  • False Positive Rate — Minimal failed builds due to CI infrastructure (vs actual bugs)
  • Test Coverage Balance — Fast suite catches critical issues; full suite available locally

References