Testing in AI and automation is shifting toward commoditized, cost-driven solutions, risking undervalued human skills like critical thinking and risk assessment. The discussion calls for a return to risk-based testing, clear communication of uncertainties, and education focused on core testing principles over automation.

Test Guild

Test Guild - hosted by Joe Colantonio has main topic focus on Testing or Automating. Each episode has a different guest. Show notes have comprehensive links and usually a full transcript. Released as audio and video.

• https://testguild.com/
• https://testguild.com/podcasts/automation/
• https://www.youtube.com/playlist?list=PL9AgRtJkydU1jqvx46esyr56BXtm1QEds
• https://www.youtube.com/@JoeColantonio

Episode Details

• Show Notes: https://testtalks.libsyn.com/test-automation-wont-save-your-qa-career-but-these-skills-will-with-keith-klain
• Published: 2026-07-28T21:16:00Z
• Duration: 45:26
• Author: Unknown

Overview

The podcast discusses the evolving role of testing in the context of AI and automation, emphasizing that the mechanical aspects of testing - particularly automated checks - are becoming commoditized and may soon require little to no cost. As AI-driven tools advance, there is growing concern that organizations may adopt them even if they perform slightly below human testers, simply because they are cheaper. This shift exposes a critical issue: many testers have aligned their identity with automation rather than core testing skills like critical thinking, risk assessment, and exploratory analysis, leaving them vulnerable as AI takes over repetitive tasks.

A central theme is the misalignment between how testers perceive their value and how organizations actually use testing outcomes. Testing is often undervalued because it is framed around pass/fail metrics rather than business risk, leading to what’s termed “confidence engineering” - a dangerous tendency to seek reassurance rather than uncover real issues. This approach fosters confirmation bias and ignores uncertainty, which should instead be embraced as a key part of effective testing. The discussion advocates for a return to risk-based testing, clear communication of untested areas and potential impacts, and a renewed focus on education that prioritizes fundamental testing principles over tool proficiency. Ultimately, the future of testing lies in strategic thinking, not in competing with machines on automation.

What If

• What if you stopped selling test automation and started selling risk reports instead?

  • Move: Refactor your solo testing deliverables from pass/fail dashboards to one-page risk summaries highlighting untested areas, assumptions, and business impact scenarios.
  • Why Now?: AI-driven automation is collapsing the cost of test execution - your value isn’t in running checks, but in interpreting what’s missing and risky.
  • Expected Upside: Clients will engage you for decision-critical insights, not commodity automation; allows premium pricing and deeper stakeholder alignment.

• What if you treated AI tools as apprentices, not replacements, and built a personal AI-assisted test design workflow?

  • Move: Use an LLM to generate 50 edge-case test ideas from a user story, then apply your judgment to filter, prioritize, and document only the high-risk 10%.
  • Why Now?: AI testing tools are in the “boys with toys” phase - those who combine AI output with human risk sense will outperform both pure automation and pure manual testers.
  • Expected Upside: 3 - 5x faster test design cycles, demonstrable ROI for clients, and a defensible niche as a sense-maker in an era of AI noise.

• What if you audited your own automation suite for “confidence engineering” and replaced it with uncertainty mapping?

  • Move: Delete all passing automated tests that recheck known-good paths; replace them with dynamic traces showing what remains untested after each release.
  • Why Now?: “Confidence engineering” creates dangerous blind spots - clients pay for false security, but recent AI-driven “slop code” increases hidden risks.
  • Expected Upside: Shift from being a maintenance burden to a strategic early-warning system, reducing client failure costs and increasing retention.

Takeaway

• Refocus your testing efforts on high-risk business areas rather than writing or maintaining automated checks; prioritize exploring what could go wrong over proving features work.
• Replace pass/fail test reporting with clear risk communication - document and share untested areas, assumptions, and potential impacts in language stakeholders understand.
• Treat AI and automation tools as assistants, not replacements - use them to extend your reach (e.g., generating test ideas or simulating environments), but maintain human oversight for judgment and context.
• Stop chasing test automation metrics like “10,000 passing tests”; instead, evaluate tooling based on real-world effectiveness by demanding high-risk, production-grade case studies before adoption.
• Invest time in building core testing skills - risk assessment, critical thinking, and exploratory testing - through targeted reading and practice, especially using resources on cognitive bias, complex systems failure, and model evaluation.