How to Auto-Quarantine Flaky Cypress Tests

1. Define Quarantine Thresholds & Triggers #

Establish clear failure metrics before automation. A test should enter quarantine after exceeding a configurable threshold (e.g., 3 failures in 10 CI runs) while maintaining a minimum retry success rate. This logic forms the foundation of Building Auto-Quarantine Workflows and ensures only genuinely unstable tests are isolated. Avoid single-failure triggers to prevent false positives.

2. Parse Cypress Results & Generate Quarantine List #

Use a lightweight Node.js post-run script to parse cypress/results.json. Extract test titles, file paths, and failure counts. Map failing tests to a version-controlled quarantine.json array. Ensure the script handles concurrent CI jobs by using atomic file writes or a centralized artifact store.

// scripts/quarantine-parser.js
const fs = require('fs');
const results = JSON.parse(fs.readFileSync('cypress/results.json', 'utf8'));
const quarantine = JSON.parse(fs.readFileSync('cypress/quarantine.json', 'utf8'));

results.runs.forEach(run => {
 run.tests.forEach(test => {
 if (test.failures > 3 && test.passes < 7) {
 const id = `${test.file}::${test.title}`;
 if (!quarantine.includes(id)) quarantine.push(id);
 }
 });
});
fs.writeFileSync('cypress/quarantine.json', JSON.stringify(quarantine, null, 2));

3. Dynamically Skip Quarantined Tests in Cypress #

Modify cypress.config.ts to read the quarantine list at runtime. Use Cypress’s before hook or a custom plugin to programmatically apply .skip() to matching tests. This prevents pipeline failures while preserving test execution logs for debugging.

// cypress.config.ts
import { defineConfig } from 'cypress';
import fs from 'fs';

const quarantineList = JSON.parse(fs.readFileSync('cypress/quarantine.json', 'utf8'));

export default defineConfig({
 e2e: {
 setupNodeEvents(on, config) {
 on('before:run', (details) => {
 // Inject quarantine data into test context or use custom reporter
 });
 },
 specPattern: 'cypress/e2e/**/*.cy.{js,jsx,ts,tsx}'
 }
});

4. Automate PR Checks & Notification Routing #

Integrate the quarantine script into GitHub Actions or GitLab CI. Trigger Slack/Teams alerts when a test enters or exits quarantine. Require a reliability engineer sign-off or automated root-cause analysis before re-enabling quarantined tests. Block merges if quarantine rate exceeds defined SLOs.

Common Pitfalls #

• Quarantining tests based on single CI failures instead of statistical flakiness thresholds
• Failing to version-control the quarantine list, causing CI race conditions across parallel runners
• Skipping tests without logging the quarantine event, leading to silent coverage degradation
• Not implementing an automated unquarantine validation step to verify fixes before re-enabling

FAQ #

How do I safely unquarantine a Cypress test? Run the quarantined test in isolation against a staging environment with 10+ consecutive passes. Update the quarantine script to remove it from the list only after CI validation confirms stability.

Does auto-quarantine affect overall test coverage metrics? Yes, skipped tests reduce execution coverage. Track quarantine rate as a separate reliability metric and require root-cause resolution within a defined SLA to restore full coverage.

Can this workflow integrate with Cypress Cloud? Yes. Use Cypress Cloud’s --record flag to capture flakiness metadata, then cross-reference the cloud API with your quarantine parser for centralized tracking.

Reliability Metrics #

• Quarantine Hit Rate (%)
• Mean Time to Quarantine (MTTQ)
• False Positive Quarantine Rate
• CI Pipeline Pass Rate Delta (Pre/Post Quarantine)
• Test Recovery SLA Compliance