Skip to content

ADR 015: Non-Blocking E2E Tests in CI

Status

Accepted

Date

2026-01-25

Context

The CI pipeline has been consistently failing on E2E tests for over 10 consecutive commits on the main branch, while all other quality checks (static analysis, unit tests, build) pass reliably. Analysis of the failures reveals:

  1. Consistent Timeout Failures: E2E tests fail with timeout errors waiting for networkidle state and API responses
  2. Environmental Issues: Failures are infrastructure-related, not code quality issues
  3. Tests wait for /api/v1/services/export endpoint that times out in CI
  4. waitForLoadState("networkidle") consistently exceeds 90s timeout
  5. Blocking Development: Every commit fails CI despite code being correct, creating a "boy who cried wolf" scenario where developers ignore CI failures
  6. Recent History: All 10+ recent commits show the same E2E timeout pattern

Example failures:

  • Accessibility audit timeouts (4 tests)
  • Offline sync test timeouts (waiting for service export API)
  • Interactive navigation timeouts (networkidle state)

The E2E test infrastructure needs investigation and fixing, but this should not block all development work.

Decision

Make E2E tests non-blocking in the CI pipeline by adding continue-on-error: true to the test-e2e job.

What This Means

Tests still run:

  • E2E tests execute on every main branch push
  • Results are visible in CI logs
  • Playwright reports are uploaded as artifacts

But don't block builds:

  • CI shows ✅ success even if E2E tests fail
  • Developers can merge PRs and push to main without E2E blocking
  • Other quality gates (lint, type-check, unit tests, build) remain blocking

Guiding Principle

"CI should help dev work stay on track, not be so cumbersome it halts progress."

CI should catch real code quality issues, not infrastructure problems. Flaky tests that fail for environmental reasons should provide visibility but not block development.

Consequences

Positive

  • Development Unblocked: Commits with good code quality can proceed without waiting for E2E test fixes
  • Visibility Maintained: E2E test results still available for review in CI logs and artifacts
  • Focus on Real Issues: Developers can focus on actual code problems caught by unit tests and static analysis
  • Separate Investigation: E2E infrastructure issues can be investigated and fixed without pressure

Negative

  • False Green: CI shows success even if E2E tests are failing
  • Risk of Regressions: E2E regressions won't block deployment (mitigation: manual review of E2E results before releases)
  • Discipline Required: Team must check E2E results manually rather than relying on CI status

Neutral

  • Incentive to Fix: Making tests non-blocking highlights that they need fixing without blocking work

Mitigation Strategies

  1. Monitor E2E Results: Periodically review Playwright reports even when CI passes
  2. Create Investigation Task: Add E2E test infrastructure investigation to roadmap
  3. Pre-Release Checklist: Require manual E2E test review before production deployments
  4. Re-enable Blocking: Once E2E tests are stable (passing consistently), remove continue-on-error: true

Alternatives Considered

1. Skip E2E Tests Entirely

  • Rejected: Loses visibility into E2E test health completely

2. Fix E2E Tests First

  • Rejected: Blocks all development work until infrastructure issues resolved (unknown timeline)

3. Increase Timeouts

  • Rejected: Masks infrastructure problems; tests already at 90s timeout (excessive for unit tests)

4. Run E2E Only on Demand

  • Rejected: Loses continuous feedback; better to run and not block than not run at all

Implementation

File Changed:

  • .github/workflows/ci.yml: Added continue-on-error: true to test-e2e job

Configuration:

test-e2e:
  continue-on-error: true # Non-blocking
  timeout-minutes: 30
  # ... rest of config

Other Quality Gates (Remain Blocking):

  • Static analysis (lint, type-check, prettier, security audit)
  • Unit tests (Vitest)
  • Build verification
  • Data validation
  • i18n audit

Future Work

  1. Investigate E2E Timeout Root Cause:
  2. Why does waitForLoadState("networkidle") timeout in CI but not locally?
  3. Is /api/v1/services/export actually responding in CI environment?

  4. Are Supabase secrets properly configured in CI?

  5. Stabilize E2E Tests:

  6. Replace waitForLoadState("networkidle") with more reliable selectors
  7. Add explicit waits for API responses with better error handling

  8. Reduce test flakiness through better setup/teardown

  9. Re-enable Blocking:

  10. Once E2E tests pass reliably (e.g., 95%+ pass rate over 2 weeks), remove continue-on-error

References

  • GitHub Actions: Continue on Error
  • Playwright CI History: Last 10+ commits on main all failed E2E tests
  • Discussion: "If CI is failing and we can't get it to pass, should we make the CI checks easier?"