Flaky tests undermine CI/CD pipelines by causing unreliable results due to shared state, race conditions, or environmental variables, requiring strategies like reruns, ownership, and test refactoring to address. Solutions emphasize root-cause fixes through test structure improvements and cautious use of AI, framing flakiness as an opportunity to strengthen system reliability.

Software Testing Unleashed

Software Testing Unleashed - hosted by Richard Seidl. Different guest per episode. The official Show notes contain a comprehensive overview of the episode. Released as audio and video.

• https://www.richard-seidl.com/en/testing-unleashed
• https://www.youtube.com/playlist?list=PL48Mbm-L0hjB1OdwYi9h7jrq9t352-Zk_

Episode Details

• Show Notes: https://www.richard-seidl.com/en/blog/flaky-tests
• Published: 2026-04-23T04:00:00Z
• Duration: 00:24:26
• Author: Richard Seidl | Software Development & Testing Expert

Overview

The podcast explores the challenges posed by flaky tests in software development, defined as tests that fail unpredictably due to a lack of repeatability. These tests undermine confidence in CI/CD pipelines by obscuring whether code changes are safe, while also consuming significant time investigating false failures. Key qualities of reliable tests include self-checking, fast feedback, isolation, and consistent results. The discussion highlights that flakiness often stems from shared state (e.g., database inconsistencies), race conditions, or environmental variables, emphasizing the need to root out systemic issues rather than merely masking symptoms.

Strategies for managing flaky tests include temporarily removing them from CI pipelines, assigning ownership to specific individuals for accountability, and prioritizing fixes for critical tests. Techniques like rerunning tests (up to three times) or implementing a “time-to-live” policy for unresolved flaky tests are suggested. Addressing root causes involves refactoring tests into a test pyramid structure, focusing on smaller, isolated tests, or even removing tests that lack value. The episode also touches on the role of AI in testing, which can generate test ideas or debug code but should not replace human judgment, due to its potential for overconfident or erroneous suggestions. Overall, the focus is on treating flakiness as an opportunity to improve system robustness and code reliability, rather than an unavoidable hurdle.

What If

• What if you implemented a “gatekeeping strategy” for all new tests by running them 100 times in isolation before including them in CI/CD?

  • Concrete move: Set up a pre-commit hook or separate script that runs each new test 100 times in isolation (no parallelism, no shared state) to detect flakiness before merging.
  • Why now: Flaky tests are a root cause of CI/CD instability, and this strategy mirrors Metas approach mentioned in the text, which prioritizes stability upfront.
  • Expected upside: Reduce long-term maintenance costs by catching flaky tests early and improving team trust in CI results.

• What if you applied a “time-to-live (TTL)” rule for unresolved flaky tests, automatically archiving or deleting them after 30 days?

  • Concrete move: Create a script or tool to label flaky tests with a timestamp and schedule a periodic review (e.g., monthly) to flag tests unresolved beyond 30 days for removal.
  • Why now: Flaky tests often become “dead code” with no value, and their persistence increases technical debt. The text explicitly recommends deleting unresolved flaky tests as a strategy.
  • Expected upside: Free up engineering time by eliminating outdated tests and reducing noise in CI pipelines.

• What if you used AI-generated test cases to debug the most critical flaky tests, then manually reviewed and refined them?

  • Concrete move: Run an AI tool (e.g., GitHub Copilot, Codota) to generate test cases for your most critical flaky tests, focusing on edge cases or shared state issues. Review and refactor the output.
  • Why now: The text notes that AI can be a debugging aid for identifying problematic code lines, and solo developers can leverage it for efficiency.
  • Expected upside: Speed up debugging by identifying potential causes (e.g., race conditions, shared state) and reduce time spent on manual trial-and-error.

Takeaway

• Temporarily block flaky tests from CI pipelines using test annotations or blocklists, and link them to bug tracking systems to maintain visibility and accountability.
• Assign explicit ownership to each flaky test to ensure individual responsibility for debugging and resolving root causes instead of relying on team-wide efforts.
• Retry unstable tests up to 3 times in CI (depending on test duration and time constraints) to mitigate flakiness, but avoid excessive retries that delay build pipelines.
• Implement a gatekeeping strategy by rigorously validating test stability (e.g., run tests 100 times overnight) before including them in CI/CD, as demonstrated by Metas approach.
• Set a Time-to-Live (TTL) policy for unresolved flaky tests and delete them if they remain unaddressed beyond a defined period to reduce technical debt and maintenance costs.