Keith Klein distinguishes between low-cognitive “checking” (automated verification) and high-cognitive “testing” (critical analysis), warning that AI may automate routine checks but cannot replace human-driven exploratory testing or contextual judgment. The discussion emphasizes the need for testers to prioritize strategic thinking, challenge assumptions, and balance AI efficiency with ethical and systemic risks in evolving software practices.

The Vernon Richard Show

The Vernon Richard Show - hosted by Vernon Richards and Richard Bradshaw. Usually a Testing themed discussion between Vernon and Richard. Official show notes have summary description, timestamped chapter headings, resource links. Show notes on the website sometimes have a full transcript. Released as audio and video.

• https://thevernonrichardshow.com/
• https://www.youtube.com/@TheVernonRichardShow

Episode Details

• Show Notes: https://share.transistor.fm/s/21cdc238
• Published: 2026-06-10T08:00:00Z
• Duration: 01:05:36
• Author: Vernon Richards and Richard Bradshaw

Overview

The podcast delves into Keith Kleins distinction between “checking"mechanistic, rule-based verification tasks like executing predefined test casesand “testing,” which requires critical thinking, analysis, and evaluation of system behavior. Klein argues that checking, due to its low cognitive demand, may become obsolete as AI tools increasingly handle routine validation. The discussion explores how this dichotomy challenges current industry practices, emphasizing the need for testers to move beyond repetitive tasks and demonstrate strategic value through exploratory, systems-level thinking. The conversation also critiques the tendency of testers to accept requirements at face value, neglecting edge cases or deeper analysis, and stresses the importance of questioning assumptions to ensure robust testing.

The episode addresses the broader implications of AI in software testing, acknowledging its potential to automate basic checks but highlighting its limitations in critical thinking, contextual interpretation, and handling probabilistic systems. It warns against over-reliance on AI, which could lead to undervaluing human expertise and increased risks if automated checks fail. The dialogue contrasts theoretical debates about software quality with real-world constraints, such as financial pressures and organizational priorities, while advocating for a balance between cost-saving measures and preserving the irreplaceable role of human judgment. Key themes include the evolving role of testers in AI-driven environments, the necessity of technical and analytical skills, and the need for the testing community to adapt by promoting advanced practices, fostering shared understanding, and effectively communicating value to stakeholders. Ethical concerns about AIs environmental impact, monopolization, and systemic risks are also raised, alongside calls for critical engagement with emerging technologies.

What If

• What if you redefined your testing practice to prioritize critical thinking over automated checks?

  • Move: Shift from executing pre-determined test cases to conducting exploratory testing sessions using AI tools to identify probabilistic edge cases in your software.
  • Why Now?: As AI systems become more prevalent, employers will value testers who can analyze non-deterministic outputs (e.g., AI-generated summaries, natural language responses) and evaluate risk through statistical evaluation, not just deterministic checks.
  • Expected Upside: Youll position yourself as a critical thinker in an industry increasingly reliant on AI, making your role irreplaceable by automated tools and aligning with the demand for “testing” over “checking.”

• What if you built a personal brand around high-value testing work instead of low-skill checklists?

  • Move: Publish case studies or tutorials showcasing how you used critical thinking to uncover hidden risks in AI/ML systems (e.g., bias in output, system degradation over time), avoiding generic happy path testing.
  • Why Now?: The market is saturated with professionals who can write test cases but lack the curiosity to challenge assumptions. Demonstrating your ability to think beyond requirements will differentiate you from AI order takers.
  • Expected Upside: Increased visibility as a thought leader, leading to higher-value contracts or roles that emphasize strategic testing, not just compliance with checklists.

• What if you invested in technical expertise to bridge the gap between AI systems and testability?

  • Move: Learn statistical analysis frameworks (e.g., confidence intervals, hypothesis testing) and how to evaluate AI outputs (e.g., email quality scores, summary accuracy) using pre-trained models from Hugging Face or similar platforms.
  • Why Now?: The text highlights a critical gap in the industry: testers lack technical knowledge to validate AI systems. Mastering these skills will let you assess AI efficacy and justify your role amid automation claims.
  • Expected Upside: Youll become a specialist in AI testing, a niche with high demand as organizations seek experts to evaluate systems that probabilistic models interact with, such as user-facing summaries or prioritization algorithms.

Takeaway

• Prioritize Testing Over Checking: Focus on tasks requiring critical thinking, system analysis, and exploration (e.g., designing edge-case scenarios, evaluating AI output variability) instead of repetitive checklist-based “checking” to avoid being perceived as replaceable by AI tools.
• Leverage AI as a Skill Enhancer: Use AI tools as collaborative partners to automate low-value tasks (e.g., documentation, initial test case drafting) and free up time for higher-value work like analyzing probabilistic AI outputs or designing risk-based tests.
• Document and Communicate Impact: Explicitly document the rationale and impact of your testing efforts (e.g., stories around edge-case discoveries, cost savings from risk mitigation) to demonstrate strategic value rather than just output (e.g., test results).
• Challenge Assumptions in Requirements: Actively question and probe requirements with questions like “What is A?” and “When is A not A?” during development, ensuring robustness by simulating alternative scenarios and exploring non-deterministic system behaviors.
• Invest in Technical Depth for AI Testing: Build expertise in AI-specific testing methods (e.g., statistical evaluation of probabilistic models, technical mapping of AI systems) and stay updated on tools like pre-trained models (e.g., Hugging Face) to stay relevant in evolving testing roles.