Testing & Validation¶

This project uses a pragmatic testing strategy: CI should guide development and catch major errors, not block legitimate work.

Philosophy¶

• Fast feedback: CI runs in < 3 minutes
• Reliable: No flaky external dependencies or platform-specific gates
• Essential only: Lint, link validation, core tests, Windows provisioning
• Coverage is optional: Run locally with mise run test-coverage when needed

Quick Reference¶

1. Unit Testing (BATS)¶

Command: mise run test

Tests are built with BATS:

• Provisioning: provisioning.bats, packages.bats
• Security: secrets.bats, ssh.bats
• System: doctor.bats, backup.bats
• Config: mise.bats, chezmoi.bats

Shell Policy Guardrails¶

Shell safety checks are enforced in tests:

• tests/configs.bats: Bash/Zsh config syntax and shell-policy assertions.
• tests/provisioning.bats: Install behavior for server-mode shell defaults.
• tests/doctor.bats: Doctor output expectations for shell support tiers.

Running Specific Tests¶

bats tests/smoke.bats          # Single file
bats --filter "smoke" tests/   # By pattern
bats --filter-tags linux-only tests/  # By tag

2. Static Analysis¶

Command: mise run lint

• ShellCheck: Script analysis
• Yamllint: YAML validation
• Taplo: TOML validation

Pre-commit hooks run automatically on git commit.

3. CI Pipeline¶

Key Principle: CI runs a fast subset of tests for quick feedback. Full suite runs locally.

CI Jobs¶

Total CI time: ~2 minutes (previously 40+ minutes)

Fast Test Suite (CI)¶

CI runs only essential tests:

# What CI runs (~2 min)
bats tests/static.bats tests/smoke.bats tests/basic.bats tests/env.bats tests/doctor.bats

Full Test Suite (Local)¶

# What you run locally (450+ tests, ~10-15 min)
mise run test

Install-flow regression coverage intentionally avoids assuming passwordless sudo.

• tests/install_validation.bats verifies repeat-run convergence and partial-checkout repair with mocked git / chezmoi / mise.
• tests/server_deployment.bats validates non-interactive install behavior with a rootless mocked environment so local runs and CI are not coupled to host sudo policy.

Weekly Full Suite (CI)¶

A weekly workflow runs the complete test suite across Ubuntu versions:

• Schedule: Monday at 6:00 UTC
• Matrix: Ubuntu 22.04, 24.04
• Manual trigger: Available via workflow_dispatch
• Artifacts: Test results uploaded for 30 days

# Equivalent local command
mise run test

Why Fast Suite for PRs?¶

• 40+ min CI defeats the purpose - Blocks development, discourages commits
• Essential tests catch 90% of issues - Syntax, smoke, environment validation
• Full coverage runs weekly - Catches regressions without blocking PRs

Note: macOS testing is manual due to ARM64 tool compatibility issues.

Free-tier note: Browser E2E/Playwright tests are intentionally not part of local workflows and should remain CI-only when/if introduced, to preserve GitHub free-tier minutes.

4. Coverage¶

Local Coverage¶

mise run test-coverage
# Reports saved to /tmp/dotfiles-coverage/

Coverage is informational, not a gate. There are no enforced thresholds.

CI Coverage Reporting¶

On pushes to main, CI automatically runs tests with coverage and updates a dynamic badge in the README.

How it works:

1. The coverage job in ci.yml runs mise run test-coverage with kcov
2. Coverage percentage is extracted from the kcov HTML report
3. A GitHub Gist is updated via schneegans/dynamic-badges-action
4. The README badge reads from the Gist via shields.io

One-time setup required:

1. Create a GitHub Gist (can be empty, note the Gist ID)
2. Create a Personal Access Token with gist scope
3. Add repository secrets:
4. GIST_TOKEN - Your PAT with gist scope
5. COVERAGE_GIST_ID - The Gist ID from step 1
6. Update the badge URL in README.md with your actual Gist ID

Until secrets are configured, the coverage job will silently skip the badge update (it uses continue-on-error: true).

5. Test Tags System¶

BATS tags allow selective test execution. Use --filter-tags to run specific subsets:

bats --filter-tags linux-only tests/     # Run only Linux-specific tests
bats --filter-tags tag:fast tests/       # Run only fast tests
bats --filter-tags tag:smoke tests/      # Run only smoke tests

Available Tags¶

Tag Usage Examples¶

Development workflow:

# Quick iteration - run fast tests only
bats --filter-tags tag:fast tests/

# Pre-commit check - smoke tests
bats --filter-tags tag:smoke tests/

# Platform-specific testing
bats --filter-tags linux-only tests/    # Linux CI
bats --filter-tags macos-only tests/    # macOS manual testing

CI optimization:

# Fast subset for CI (currently used)
bats tests/static.bats tests/smoke.bats tests/basic.bats

# Alternative: Tag-based CI
bats --filter-tags 'tag:fast,tag:smoke' tests/

Adding Tags to Tests¶

Add tags as comments before test cases:

# bats test_tags=tag:fast,tag:smoke
@test "check shell loads" {
  run zsh -c "echo test"
  assert_success
}

# bats test_tags=linux-only
@test "systemd is available" {
  run systemctl --version
  assert_success
}

Tag Guidelines¶

1. Platform tags (no tag: prefix): linux-only, macos-only, wsl-only
2. Category tags (with tag: prefix): tag:fast, tag:smoke, tag:integration
3. Multiple tags: Separate with commas: tag:fast,tag:smoke
4. Specificity: Tag tests based on actual requirements, not hypothetical needs

For a complete tag reference, see tests/tags.txt

6. macOS Manual Testing¶

Since ARM64 tool compatibility varies, macOS testing is manual:

Pre-Release Checklist¶

Run on an actual macOS machine before releases:

# 1. Fresh install simulation
mise run doctor

# 2. Homebrew integration
brew bundle check --file=Brewfile

# 3. macOS-specific tests
bats --filter-tags macos-only tests/

# 4. Shell functionality
zsh -i -c "echo 'Shell loads'" && echo "✅ Pass"

Known macOS Differences¶

Reporting Issues¶

If tests fail on macOS, file an issue with:

• macOS version (sw_vers)
• Chip type (Intel/M1/M2/M3)
• Error output
• Chip type (Intel/M1/M2/M3)

7. Performance Testing¶

To ensure the shell remains fast, we track startup time regressions.

Command: mise run perf

• Tool: hyperfine
• Metric: Mean execution time of zsh -i -c exit
• Thresholds:
  • Local: Warn if > 300ms (default)
  • CI: Fail if > 500ms

Running Benchmarks¶

# Default run (10 samples)
mise run perf

# Custom regression check
mise run perf --threshold 200 --runs 20

If CI fails on performance, investigate recent changes to .zshrc or plugin initialization.

Related Documentation¶

• CI Troubleshooting - Debugging CI test failures
• Contributing - Test requirements for PRs
• Mise Tasks - Running tests via mise run test