You Need a Test Automation Strategy Before AI - Michael Pilaeten
Categories: Podcasts , Software Testing Unleashed
Testing is a skilled profession that blends manual and automated methods to verify functionality, validate usefulness, and find defects, with quality shifting from QA to shared team responsibility in agile environments. AI can enhance test automation but requires human expertise to ensure accuracy, avoid flaky tests, and align with business value and customer needs.
Software Testing Unleashed
Software Testing Unleashed - hosted by Richard Seidl. Different guest per episode. The official Show notes contain a comprehensive overview of the episode. Released as audio and video.
- https://www.richard-seidl.com/en/testing-unleashed
- https://www.youtube.com/playlist?list=PL48Mbm-L0hjB1OdwYi9h7jrq9t352-Zk_
Episode Details
- Show Notes: https://www.richard-seidl.com/en/podcast/test-automation-strategy-balance
- Published: 2026-09-24T04:00:00Z
- Duration: 00:22:54
- Author: Richard Seidl | Software Development & Testing Expert
Overview
The podcast discusses the evolving role of testing in software development, emphasizing that testing is a skilled profession focused on verifying functionality, validating usefulness, and finding defects. It highlights the complementary nature of manual and automated testing, noting that while automation excels at repetitive tasks, manual testing involves critical thinking and subjective judgment that is difficult to automate. A key theme is the shift from traditional quality assurance to quality engineering, where quality is a shared team responsibility integrated throughout the development process, especially in agile environments. The discussion stresses that test automation should be risk-driven and aligned with business value, rather than pursued as a checkbox exercise.
AI’s growing role in test automation is examined critically, with concerns about flaky or inaccurate AI-generated tests and the potential for false positives due to flawed self-healing mechanisms. While AI can speed up test creation, it lacks reasoning and may produce low-quality outputs if not guided by strong foundational practices. Over-reliance on AI risks displacing junior roles and creating a skills gap, as experienced professionals are still needed to validate results. The conversation underscores that AI can only enhance testing when built on a stable codebase and sound strategy - otherwise, poor practices lead to worse outcomes. Ultimately, successful testing requires a balanced, strategic approach that combines human expertise, appropriate automation, and thoughtful use of AI - all aligned with business goals and customer needs.
What If
-
What if you audited your AI-generated test suite for “self-healing” failures this week?
- Move: Pick the top 3 flaky end-to-end tests flagged by your CI/CD system, inspect their recent AI-generated updates, and manually verify whether assertions were removed or bypassed during auto-repair.
- Why Now?: AI self-healing features are actively being used in tools like Mabl or Testim; undetected assertion removal creates false confidence right before critical releases.
- Expected Upside: Uncover at least one silent failure masked as a pass, prevent production defect leakage, and refine your team’s review checklist for AI-generated test changes.
-
What if you shifted from chasing 100% code coverage to a risk-based automation strategy starting tomorrow?
- Move: Map your top 5 business-critical user flows, identify untested paths in those areas using SonarCube or similar, then write only those missing unit/integration tests - ignore low-risk legacy modules.
- Why Now?: Management pressure for high coverage metrics is peaking, but wasted effort on low-value code delays real quality improvements; now is the time to redirect energy strategically.
- Expected Upside: Reduce test maintenance load by 30 - 50% while increasing detection of critical bugs, and create a defensible, business-aligned testing narrative for stakeholders.
-
What if you offloaded repetitive test case generation to AI but owned validation as your core daily task?
- Move: Use an AI tool (e.g., GitHub Copilot) to generate input variations for a high-volume contract processing module, then spend 90 minutes manually validating edge cases the AI missed or misrepresented.
- Why Now?: Junior roles are being automated away, so solo developers must differentiate by adding value AI can’t - judgment, context, and risk intuition - starting today.
- Expected Upside: Double your effective output by leveraging AI speed without sacrificing reliability, positioning yourself as a quality enabler rather than just a coder.
Takeaway
- Audit and refine your test automation strategy by evaluating the actual business value and risk coverage of existing tests, eliminating redundant or low-impact ones.
- Implement AI-generated testing only after establishing a stable codebase and clear, risk-based testing priorities to avoid amplifying poor practices.
- Focus on human-reviewed validation of AI-generated tests, especially assertions and edge cases, to prevent false positives and maintain test reliability.
- Shift from reporting raw metrics like code coverage to communicating quality in business terms (e.g., defect escape rate, production stability) when engaging stakeholders.
- Adopt a shared quality engineering mindset by integrating basic test automation maintenance into development workflows, even as a solo operator, to catch regressions early and reduce technical debt.
For a PDF of longer Software Testing Podcast Episode Summaries with Briefing Notes and more detailed summary notes, visit EvilTester Patreon Podcast Summaries.