Focuses on advanced QA strategies, AI orchestration in testing, and structured frameworks, with insights from Ivan Davidov on transitioning from engineering to QA, emphasizing modular design, context management, and mitigating automation pitfalls. Highlights evolving QA roles through strategic architecture, Playwright integration, audit-driven workflows, and resources for democratizing specialized testing knowledge.

How To Test This?

interview episodes where Mamadou N’diaye talks with with software testing experts

• https://podcasters.spotify.com/pod/show/spidey1944
• https://www.youtube.com/@HowToTestThis
• https://www.linkedin.com/in/mamadou-ndiaye-consultant/

Episode Details

• Show Notes: https://podcasters.spotify.com/pod/show/spidey1944/episodes/How-to-Test-with-AI-Native-Playwright-Frameworks---Ivan-Davidov-e3k8ens
• Published: 2026-06-02T19:17:04Z
• Duration: 00:44:28
• Author: Mamadou N’diaye

Overview

The podcast episode centers on software testing strategies, emphasizing risk management, automation frameworks, and the integration of AI in testing. Guest Ivan Davidov, transitioning from civil engineering to QA, discusses his journey in building robust test architectures, drawing parallels between infrastructure design and testing systems. Key topics include the importance of structured frameworks, challenges in adopting Playwright for automation, and the role of continuous learning and mentorship in mastering test engineering. Davidov highlights the “orchestration pattern” as a strategic approach to AI-native testing, emphasizing structured design over full automation, with levels of AI integration ranging from agent-based task execution to complex system orchestration. The discussion also addresses common pitfalls in QA automation, such as unstandardized practices and “tribal knowledge” gaps, advocating for modular frameworks, uniform code standards, and rigorous validation processes to ensure reliability.

The episode explores evolving QA roles, arguing that testers must shift from manual execution to strategic architecture and oversight of automation systems. Davidov outlines a scaffold combining Playwright and AI orchestration principles, designed to reduce test flakiness and enforce a “single source of truth” through standardized code and documentation. Key practices include audit-driven workflows, context management to prevent AI hallucination, and just-in-time skill loading to avoid information overload. Challenges like context voids in AI agents are mitigated by incremental, task-specific information disclosure. The conversation also stresses the need for QA architects to understand both software and testing architectures, aligning with Agile team structures that prioritize small, cross-functional units. Resources like a GitHub repository and YouTube channel are highlighted as tools for democratizing advanced QA knowledge, emphasizing community-driven learning and adaptability in a rapidly evolving field.

What If

• What if you built a modular QA framework using the AI-native orchestration pattern?

  • Move: Adopt the “AI Native Scaffold” with Playwright, integrating just-in-time skill loading and a structured orchestrator (summary, constitution, table of contents).
  • Why Now?: The market demands scalable, maintainable test architectures, and this approach directly addresses the “context void” and tribal knowledge problems discussed.
  • Expected Upside: A framework with reduced flakiness, standardized enums/fixtures, and flexibility to evolve with AI models, enabling faster test development and maintenance.

• What if you implemented a “audit-then-edit” workflow for your automation scripts?

  • Move: Formalize a process where AI agents generate audit plans (with confidence levels) and require manual verification before execution.
  • Why Now?: The discussion highlighted the risks of non-strict schema validation and code credentials. This workflow enforces safety and avoids silent test skips.
  • Expected Upside: Higher test reliability, reduced false positives, and clearer accountability for execution decisions, even with AI-crafted scripts.

• What if you contributed to the “AI Native Scaffold” GitHub project to accelerate community growth?

  • Move: Create a TypeScript/Playwright example for a specific QA task (e.g., API testing) and publish it as a YouTube video, linking back to the scaffold repository.
  • Why Now?: The project aims to reach 100 contributors by year-end and positions solo developers as thought leaders in QA automation.
  • Expected Upside: Increased visibility, feedback from the community, and direct influence on a tool that standardizes QA workflows, easing onboarding for new teams.

Takeaway

• Implement structured test framework architecture
  Create a modular framework with a clear “cloud MD” file at the repository root, including a summary, premise, and table of contents to guide agents and maintain a single source of truth for test data, selectors, and page objects.

• Adopt the audit-then-edit workflow for QA automation
  Integrate human verification steps before executing code changes (e.g., require developer review of agent-generated audit plans with a confidence level 5) to prevent errors and ensure accuracy in test automation.

• Leverage the AI Native Scaffold GitHub repository
  Use the provided customizable Playwright/AI integration framework as a foundation, extending its pre-built “skills” (e.g., API testing, UI selectors) to fit specific use cases and reduce flakiness in test execution.

• Design an orchestrator with layered context management
  Build a lightweight orchestrator that provides agents with a summary of the repository, a constitution of rules, and modular skill breakdowns, ensuring agents receive just-in-time context to avoid overwhelming them with information.

• Document tribal knowledge systematically
  Formalize all testing practices, decisions, and edge cases into appendices, references, and code comments within the framework, ensuring teammates and future developers can access standardized guidance instead of relying on informal knowledge sharing.