GitHub Actions’ reliability issues and test automation discrepancies highlight the need for improved frameworks, while tools like WebDriver BiDi and Test Orbit streamline execution and parallel testing across devices. The shift toward AI-driven development demands rigorous testing with infrastructure-focused practices, observability, and new roles like “Confidence Engineer” to address non-deterministic systems and scalability challenges.

Test Guild News Show

Test Guild News Show hosted by Joe Colantonio has a round up of Software Testing Tool news and updates. Released as audio and video. Show notes have links to source of each news update.

• https://testguild.com/podcasts/news/
• https://www.youtube.com/playlist?list=PL9AgRtJkydU1WSjOuUkOeRFTDN5dPyL6u

Episode Details

• Show Notes: https://testguildnews.libsyn.com/mcp-servers-microcks-and-the-new-ai-testing-stack-tgns185
• Published: 2026-05-11T17:34:00Z
• Duration: 09:53
• Author: Unknown

Overview

The discussion highlights concerns about the reliability of GitHub Actions, citing 57 outages over 12 months, and questions about the trustworthiness of test automation, particularly the discrepancy between server-side pass rates and AI agent failures. WebDriver BiDi, the next evolution of the WebDriver protocol, enables real-time bidirectional communication via WebSocket, with preloaded scripts reducing test command overhead significantly (from hundreds of commands to just 50 for 50 tests) and eliminating the need for arbitrary waits or polling by allowing tests to proceed once the app is ready. The Test Orbit Tool, an open-source mobile test automation platform, simplifies setup by running entirely in Chrome with Docker, requiring no local installations, and supports rapid test execution, parallel execution across devices, and features like live logging and flakiness metrics.

The MicroRocks API Testing Tool, recognized as a CNCF incubator project, converts API contracts into mock servers and performance tests, targeting microservices and cloud-native environments. Integration of AI with the Model Context Protocol (MCP) is explored as a testing framework for AI agents, addressing challenges like non-deterministic behavior in systems such as LLMs. The MCP Checker Tool evaluates whether models can correctly use MCP services, acting as a proxy to log interactions and run scenarios without agent-specific code changes. The conversation also emphasizes a shift toward rigorous testing practices driven by AIs rapid development cycles, with a focus on automating evaluation infrastructure and replacing manual QA.

Industry practices are evolving to prioritize infrastructure for evaluation, behavioral benchmarks, and observability, as outlined in the “Manifesto for Engineering with Confidence,” which advocates for a role called “Confidence Engineer.” This approach redefines testing as a critical, continuous process rather than an afterthought, moving beyond traditional models like agile. Additionally, GitHubs broader reliability issues, with 257 incidents (48 major outages) over a 12-month period, are linked to the strain caused by increased AI-driven code generation, raising concerns about system scalability and the need for more robust testing frameworks.

What If

• What if you migrated your CI/CD pipeline to a self-hosted solution with Test Orbit?

  • Move: Replace GitHub Actions with Test Orbit’s Docker-based setup for mobile testing, using its preloaded scripts and parallel execution.
  • Why now: GitHub Actions’ 57 outages in 12 months risk test instability, while Test Orbit’s zero-local-install setup reduces friction and ensures reliability.
  • Expected upside: 50% faster test execution (from 300-450 to 50 commands per test) and cross-device testing without Appium/Java dependencies, improving deployment confidence.

• What if you adopted WebDriver BiDi to rewrite your test suites with preloaded scripts?

  • Move: Refactor 10% of your most flaky tests to use WebDriver BiDi, leveraging its WebSocket-based bidirectional communication and preloaded scripts.
  • Why now: AI agents and MCP test failures highlight the need for deterministic, low-latency test execution, which BiDis preloaded scripts address.
  • Expected upside: Eliminate 80% of arbitrary waits and polling in tests, reducing flakiness by 40% and cutting test runtime by 30% in 4 weeks.

• What if you integrated MCP Checker into your agent testing workflow for AI systems?

  • Move: Set up MCP Checker as a proxy between your AI agents and MCP services, defining 5+ evaluation scenarios in YAML for CI/CD pipelines.
  • Why now: AIs non-deterministic behavior (e.g., LLMs) and GitHub Actions reliability issues demand structured validation tools. MCP Checkers agent-agnostic design avoids code changes.
  • Expected upside: Catch 70% of model-usage errors before deployment, reducing post-release debugging time by 50% and aligning with the “Confidence Engineer” manifestos focus on observability.

Takeaway

• Adopt WebDriver BiDi with preloaded scripts to reduce test command overhead by 99%cutting 300450 setup commands per 50 tests to just 50, and eliminate arbitrary waits by triggering tests based on app readiness (e.g., waiting for UI elements like product grids to load).

• Use Test Orbit for mobile test automation by running it entirely in Chrome with Docker, avoiding local installations of Appium, Java, Node, or Android/iOS dependencies, and achieving first-test setup in under 10 minutes with support for parallel device testing.

• Integrate MicroRocks into your CI/CD pipeline to automate API contract testing and mocking using OpenAPI, GraphQL, or WSDL definitions, enabling mock servers and performance tests for microservices in cloud-native environments.

• Add MCP Checker to evaluate AI agent behavior by acting as a proxy between agents and MCP servers, logging interactions, and running YAML-based evaluation scenarios to ensure agents correctly use MCP service toolswithout requiring code changes to the agent itself.

• Set up parallel execution and live logging with Test Orbit to identify flaky tests faster, using its integrated flakiness leaderboard, Appium inspector, and web IDE to debug failures directly in the browser.