The episode reevaluates Automation in Testing (AIT) principles in the AI era, emphasizing human-centric testing, testability over automatability, and risk-based approaches while addressing challenges like blind trust in automation and the evolving role of observability. It highlights the need to prioritize context, education, and collaboration in testing, alongside redefining quality assurance through AI and intelligent tools.

The Vernon Richard Show

The Vernon Richard Show - hosted by Vernon Richards and Richard Bradshaw. Usually a Testing themed discussion between Vernon and Richard. Official show notes have summary description, timestamped chapter headings, resource links. Show notes on the website sometimes have a full transcript. Released as audio and video.

• https://thevernonrichardshow.com/
• https://www.youtube.com/@TheVernonRichardShow

Episode Details

• Show Notes: https://share.transistor.fm/s/c72de091
• Published: 2026-02-23T08:00:00Z
• Duration: 3797
• Author: Vernon Richards and Richard Bradshaw

Overview

The podcast examines the development and continued importance of Automation in Testing (AIT), a framework that aims to support rather than replace human testing. It discusses how artificial intelligence is transforming test automation, while also presenting challenges such as the creation of less effective UI tests and the misuse of automation. The conversation emphasizes the distinction between automatability and testability, advocating for systems that are designed with testing in mind, not just automation. It also highlights the importance of collaboration across teams to improve overall testability.

The episode underlines the role of human judgment in testing, noting the limitations of AI and large language models in grasping context. It also addresses the growing need for testing expertise over coding expertise and raises concerns about the overemphasis on coverage metrics, suggesting a shift toward a risk-based testing approach. Finally, the discussion considers the future of testing in the AI era, exploring the potential for innovative tools and frameworks to advance quality assurance beyond conventional automation methods.

What If

• What if you re-evaluate your test suite using risk-based criteria instead of arbitrary coverage metrics?
  Move: Prioritize test cases by identifying the most critical risks to your product or business, then focus automation efforts on those areas. Remove or refactor tests that dont address real risks.
  Why now? The text emphasizes the flaw of chasing coverage percentages without considering risk. With AI tools and evolving systems, outdated tests can create false confidence.
  Expected upside: A leaner, more effective test suite that aligns with actual business priorities, reducing time spent on irrelevant automation and improving team focus on high-impact areas.

• What if you leverage an LLM to assist in designing test scenarios instead of writing test scripts manually?
  Move: Use a language model (e.g., ChatGPT, Bard) to brainstorm edge cases, identify gaps in current tests, or generate test plans based on user stories. Validate and refine its output manually.
  Why now? The text highlights the growing role of LLMs in testing, including their ability to surface opportunities for automation and reduce reliance on coding expertise. Tools are now accessible and widely available.
  Expected upside: Faster, more creative test design without requiring deep coding skills, freeing you to focus on judgment and strategic test prioritization instead of script mechanics.

• What if you implement observability tools to monitor system behavior in production, reducing reliance on perfect test coverage?
  Move: Integrate logging, monitoring, and alerting systems (e.g., Datadog, Prometheus) to track runtime issues. Use this data to triage failures and update tests dynamically instead of assuming perfect pre-deployment testing.
  Why now? The text stresses that observability is a pragmatic alternative to exhaustive testing, especially with unpredictable AI and user behavior. Modern tools make it feasible for solo developers to adopt without heavy infrastructure.
  Expected upside: Faster issue detection and resolution in production, along with a reduced need for brittle, end-to-end tests that are hard to maintain and often fail to catch real-world edge cases.

Takeaway

• Prioritize Testability Over Automatability in Design
  Focus on designing systems where testing is feasible (e.g., accessible data, observability) from the start, ensuring collaboration across roles (testers, developers, product owners) and using shared data generation tools to enhance overall testability.

• Adopt Risk-Based Testing Instead of Coverage Metrics
  Shift from aiming for arbitrary code or test coverage percentages to identifying and mitigating high-impact risks. Evaluate tests based on their value to the team and business rather than adhering to outdated metrics.

• Leverage Human Judgment to Review Automation and AI Outputs
  Regularly validate automated test results and AI-generated outputs (e.g., from LLMs) with human expertise to avoid over-reliance on tools. This ensures accuracy, contextual understanding, and alignment with actual quality goals.

• Invest in Testing Principles and Framework Knowledge Over Coding Skills
  Master test design, types of tests, and testing frameworks (e.g., Playwright, Jira agents) to identify automation opportunities effectively. This reduces dependency on coders and enables non-coders to contribute meaningfully.

• Evaluate and Prune Tests Based on Risk, Not Tradition
  Periodically delete tests that no longer provide value or align with current risks. Avoid retaining tests solely due to pressure for coverage or outdated practices, ensuring your test suite remains efficient and actionable.