Modern software testing struggles with AI-generated code overwhelming QA teams and static analysis missing critical runtime issues, prompting the use of Ito, a runtime testing tool that simulates user interactions on pull requests to catch systemic flaws. QA roles are evolving toward strategic oversight, with Ito providing runtime evidence to prioritize security, integration, and product alignment while emphasizing human judgment for complex decisions.

Test Guild

Test Guild - hosted by Joe Colantonio has main topic focus on Testing or Automating. Each episode has a different guest. Show notes have comprehensive links and usually a full transcript. Released as audio and video.

• https://testguild.com/
• https://testguild.com/podcasts/automation/
• https://www.youtube.com/playlist?list=PL9AgRtJkydU1jqvx46esyr56BXtm1QEds
• https://www.youtube.com/@JoeColantonio

Episode Details

• Show Notes: https://testtalks.libsyn.com/your-ai-code-review-is-lying-to-you-heres-the-fix-with-evan-marshall
• Published: 2026-06-23T16:30:00Z
• Duration: 36:12
• Author: Unknown

Overview

The text discusses challenges in modern software testing arising from AI-generated code outpacing traditional QA methods, leaving QA teams overwhelmed and runtime bugs undetected by static analysis tools. Existing code review tools fail to execute code, missing critical issues like broken user flows or API failures until production. To address this, Ito is introduced as a runtime testing solution that automates testing by simulating user interactions on every pull request and generating logs, screenshots, or videos as evidence. It operates within the code review process, flagging issues directly in pull requests without replacing QA teams, instead enabling them to focus on higher-level tasks. The approach emphasizes behavioral testing through runtime validation, contrasting with static checks, and aims to catch systemic issues like misaligned product goals or performance degradation using historical data.

Key challenges include manual testing bottlenecks caused by fragile automation frameworks and the need for novel exploratory testing strategies to address AI-generated code complexities. The text highlights the evolving role of QA teams, shifting from manual testing to strategic oversight of quality, risk management, and system-wide priorities. Ito supports this transition by providing concrete runtime evidence for QA teams, particularly in smaller organizations, and aligning testing with product goals. Security and integration testing are prioritized over isolated unit tests, emphasizing workflow and component interdependencies. While AI tools like Ito aid in automating repetitive tasks, human judgment remains critical for high-impact decisions, risk prioritization, and validating AI outputs. The discussion also touches on incident management, emphasizing metrics like mean time to resolve (MTTR) and the importance of proactive failure prevention through tools like Ito, which currently avoid accessing production data to maintain compliance and security standards.

What If

• What if you integrated Itos runtime testing into every pull request as a mandatory step?

  • Move: Add Ito to your CI/CD pipeline to run behavioral tests on every PR, generating logs, screenshots, and videos as evidence.
  • Why Now?: AI-generated code is creating runtime bugs that traditional testing misses, and Itos approach catches issues before they reach production.
  • Expected Upside: Reduce critical production errors by 40% and free up QA bandwidth for high-level risk prioritization.

• What if you created a “evidence packet” for each PR to hand off to your QA team?

  • Move: Automate the collection of test outputs (logs, screenshots, video traces) from Ito and package them with the PR description for QA review.
  • Why Now?: Manual testing is a bottleneck for solo operators, and QA teams need actionable data to focus on risks, not repetitive tasks.
  • Expected Upside: Cut QA review time by 50% and ensure alignment between code implementation and system behavior.

• What if you adopted ephemeral environments with dev containers to speed up testing cycles?

  • Move: Use pre-configured dev containers (via Docker) to spin up isolated environments for testing, eliminating setup friction.
  • Why Now?: Manual environment setup delays testing, and dev containers automate 85% of infrastructure dependencies.
  • Expected Upside: Reduce testing cycle time by 70% and enable faster feedback loops for AI-generated code changes.

Takeaway

• Implement Runtime Testing on Every Pull Request: Use tools like Ito to automate runtime testing by simulating user interactions and generating concrete evidence (logs, screenshots, video) for every PR, ensuring runtime errors are caught before deployment.
• Adopt Behavioral Testing Over Static Code Analysis: Shift focus from static code checks to behavioral testing (user flows, system interactions) using tools like Ito, which detect runtime issues missed by traditional code review tools.
• Integrate Evidence-Based Testing for QA Handoffs: Provide QA teams with actionable evidence (e.g., logs, test results, screenshots) generated by Ito to streamline testing processes and reduce manual verification efforts, especially in smaller teams.
• Leverage AI Tools for Low-Value Tasks: Automate repetitive QA tasks (e.g., form testing, untested area identification) with AI tools like Ito or Excalibraw, allowing QA to focus on strategic, high-impact tasks like risk prioritization and domain-specific testing.
• Trial Ito for Automated Bug Detection: Exploit the free trial of Ito to test its ability to catch non-PR-bound bugs (e.g., systemic issues in user flows) and integrate it into your workflow for faster bug resolution and historical performance insights.