How To Test With Valuable feedback, fast - Bas Dijkstra
Categories: Podcasts , How To Test This?
Teams often waste resources automating inefficient manual regression tests and relying on vanity metrics that measure effort rather than product value. To improve, engineers should focus on efficient feedback loops, utilize mutation testing to verify test quality, and critically evaluate AI-generated code to ensure it detects meaningful defects.
How To Test This?
interview episodes where Mamadou N’diaye talks with with software testing experts
- https://podcasters.spotify.com/pod/show/spidey1944
- https://www.youtube.com/@HowToTestThis
- https://www.linkedin.com/in/mamadou-ndiaye-consultant/
Episode Details
- Show Notes: https://podcasters.spotify.com/pod/show/spidey1944/episodes/How-To-Test-With-Valuable-feedback--fast---Bas-Dijkstra-e3s1rlu
- Published: 2026-11-17T17:00:00Z
- Duration: 00:42:42
- Author: Mamadou N’diaye
Overview
The conversation covers Bas Dextra’s nearly 20-year career in test automation, from his entry into the field after a master’s in computer science to his current consulting practice. He discusses common pitfalls teams encounter, including over-reliance on automated end-to-end tests, automating existing regression scripts after the fact, and measuring success through vanity metrics like code coverage and test execution time rather than the actual value delivered.
Bas outlines his “valuable feedback fast” methodology, emphasizing that speed alone is insufficient without ensuring feedback is actionable, trustworthy, and focused on what matters. He advocates for smaller, faster tests at appropriate layers of the test pyramid, mutation testing to verify tests can actually detect defects, and asking whether a team is comfortable shipping a build rather than counting test cases. He stresses communication and iteration over trying to get everything right on the first attempt, and recommends putting products into end users’ hands quickly while maintaining due diligence.
The discussion also touches on AI tools as powerful but dangerous in unskilled hands, the importance of craftsmanship and critical thinking when using AI-generated code, and practical advice for testers seeking independence, including the necessity of sales, marketing, and network maintenance alongside technical skill.
What If
-
What if you replaced your test execution time dashboard with a single question before every release: “Are we comfortable shipping this build, knowing what we know now?”
- Move: Propose removing vanity metrics (code coverage percentage, test count, execution time) from your reporting and substituting a structured go/no-go question tied to mutation testing scores and whether your tests can actually fail.
- Why Now? Bas notes these metrics “say absolutely nothing about the value of your tests” and “tell me how hard you are working,” while mutation testing “actually tells me something that not a lot of other coverage metrics can tell me.”
- Expected Upside: You may shift team focus from producing more tests to producing tests that detect real defects, reducing maintenance burden and increasing release confidence.
-
What if you broke your longest end-to-end test into smaller unit and integration tests, then measured whether you lost any coverage you actually needed?
- Move: Propose selecting your slowest or most flaky end-to-end test, identifying which assertions can move to the unit or integration layer, and running a mutation testing pass on the new smaller tests to verify they still catch the same defects.
- Why Now? Bas explains end-to-end tests are “the most inefficient way” to automate because they focus on “graphical user interfaces” and “an end to end perspective,” and that “often, that information can be gathered with an integration test, maybe even a bunch of unit tests.”
- Expected Upside: You could reduce execution time and maintenance overhead while preserving defect detection, as smaller tests have “a smaller scope” but can still “capture everything you need to know.”
-
What if you treated AI-generated test code the same way you’d treat a junior developer’s first submission - compiling and passing is not the same as being correct?
- Move: Propose adding a mandatory mutation testing step after any AI-generated test code is merged, asking “is it actually capable of finding a meaningful defect in my product?” before accepting the green check mark.
- Why Now? Bas warns AI output “will look really, really good” but “can very, very confidently lead you in the wrong direction, give you the wrong advice, write the wrong code, write insecure code,” and that with LLM-written tests it is “double important” to verify capability.
- Expected Upside: You may catch silently broken or insecure tests before they erode trust in your pipeline, maintaining the “craftsmanship” Bas says is “even more important” with AI.
Takeaway
- Before automating any regression test, ask “what is the feedback we’re looking for and how can we get that information in the most efficient way?” rather than replicating the script as-is.
- Replace code coverage and test execution time dashboards with the question “are we comfortable shipping this build, knowing what we know now?” and track mutation testing scores to verify tests can actually fail.
- When a test suite is large and flaky, avoid creating a dedicated improvement project; instead, assign someone to analyze the suite and make small structural changes - breaking end-to-end tests into unit and integration layers - measured by execution time and mutation score.
- After any AI-generated test code passes compilation, run a mutation testing pass to confirm it “is actually capable of finding a meaningful defect in my product” before accepting it.
- Communicate early and often with developers and stakeholders rather than using tools to replace conversation; iterate and put work into end users’ hands “as quickly as possible without being reckless.”
For a PDF of longer Software Testing Podcast Episode Summaries with Briefing Notes and more detailed summary notes, visit EvilTester Patreon Podcast Summaries.