Testing AI demands a shift from traditional methods to confidence engineering, requiring expertise in probabilistic behaviors and risk assessment. The urgency to adopt robust AI testing grows amid concerns over safety, ethics, and alignment with human goals.

Test Guild

Test Guild - hosted by Joe Colantonio has main topic focus on Testing or Automating. Each episode has a different guest. Show notes have comprehensive links and usually a full transcript. Released as audio and video.

• https://testguild.com/
• https://testguild.com/podcasts/automation/
• https://www.youtube.com/playlist?list=PL9AgRtJkydU1jqvx46esyr56BXtm1QEds
• https://www.youtube.com/@JoeColantonio

Episode Details

• Show Notes: https://testtalks.libsyn.com/testing-ai-engineering-confidence-in-non-deterministic-systems-with-jason-arbon
• Published: 2026-08-12T03:15:00Z
• Duration: 53:20
• Author: Unknown

Overview

The discussion centers on the evolving challenges of testing AI and non-deterministic systems, emphasizing the shift from traditional software testing to confidence engineering. As AI systems become more complex and capable - potentially surpassing human intelligence - conventional testing methods like input/output validation are no longer sufficient. Testers must now understand probabilistic behaviors, model training, and system alignment, requiring new skills in data analysis, risk assessment, and decision-making. The concept of “confidence engineering” is introduced as a critical framework for ensuring AI reliability, where the focus shifts from executing tests to making informed quality decisions.

A major theme is the urgency and fragmentation in AI testing adoption, with many teams still struggling to implement effective practices despite growing reliance on AI. Concerns are raised about AI safety, containment, and the ethical implications of systems that may act unpredictably or deceptively. The conversation explores philosophical questions around AI consciousness, referencing debates like the Chinese Room argument and insights from figures like David Chalmers and Jeffrey Hinton. With AI rapidly advancing - generating vast amounts of code and enabling autonomous agents - the role of the tester is transforming into one that ensures alignment with human, corporate, and societal goals, positioning quality assurance as a central function in the future of software development.

What If

• What if you treated every AI-generated test as a hypothesis to validate, not just execute?

  • Move: Implement a structured evaluation loop where each AI-generated test case is run 5+ times on the same build, log variance, and classify outcomes (pass/fail/flaky). Document discrepancies and use them to refine prompts or model selection.
  • Why Now?: AI is generating 10x more code and tests, but non-determinism means results vary across runs - this variability is currently ignored, leading to false confidence. The shift from deterministic to probabilistic systems demands empirical validation of AI output.
  • Expected Upside: You’ll surface hidden flakiness, improve test reliability, and position yourself as a confidence engineer who doesn’t just run tests but validates their consistency - differentiating your value in an AI-saturated market.

• What if you pivoted from writing test scripts to designing evaluation frameworks for AI behavior?

  • Move: Replace your next Selenium script with an “eval” - a targeted assessment that measures whether an AI agent (e.g., Claude, Gemini) correctly performs a task like form submission or error handling on a live site. Use pass/fail criteria, edge cases, and expected outputs as your spec.
  • Why Now?: Trillion-dollar companies are already using AI agents to build and test software; traditional scripting is being automated away. The new bottleneck is not execution but defining what good looks like - a skill testers uniquely possess.
  • Expected Upside: You transition from a commoditized executor to a high-leverage decision-maker in quality, capable of guiding AI behavior with precision - making your role essential even as automation eats routine tasks.

• What if you became the first solo operator to benchmark AI vs. human testing on real websites and publish the results?

  • Move: Pick 3 random websites (e.g., e-commerce, SaaS login, API service). Have an AI agent generate and run a test suite, then replicate the same scope manually. Record time-to-execute, defect detection rate, flakiness, and coverage. Publish findings in a public report or blog.
  • Why Now?: Critics of AI testing refuse to run head-to-head comparisons, creating a vacuum of real data. As a solo developer, you can act faster than enterprises and establish authority by producing empirical evidence others won’t.
  • Expected Upside: You build credibility as a practical AI tester, attract clients or collaborators seeking real-world insights, and future-proof your brand by aligning with the industry’s shift toward evidence-based confidence engineering.

Takeaway

• Invest in learning and applying confidence engineering principles to evaluate AI system reliability, focusing on data-driven decision-making rather than traditional test execution.
• Run test suites multiple times (at least five) on the same build to identify flakiness and variability, especially when validating non-deterministic AI behaviors.
• Transition from manual or script-based testing by using AI tools (e.g., Claude, Codex) to generate test plans, scripts, and dashboards, increasing coverage and efficiency.
• Position yourself as a quality decision-maker by leveraging domain context and business alignment - areas where AI lacks inherent understanding - to add unique value in AI-driven development.
• Participate in independent industry surveys (e.g., Guild State of Automation) to contribute to and benefit from unbiased insights on AI testing trends and best practices.