How do you sleep at night? Arthur Hicken and Igor Kirilenko on testing AI generated code
Categories: Podcasts , Into The MoTaverse
AI enhances software development by automating tasks and improving code quality but faces challenges in reliability, coverage, and cost. While AI aids testing and CI/CD, human oversight remains essential for decision-making, especially in safety-critical fields.
Into The MoTaverse
Rosie Sherry interviews people involved in testing. Video only interviews. Available on youtube or the homepage. Each episode has a full transcript if you find it on the main site.
- https://www.ministryoftesting.com/podcasts/into-the-motaverse
- https://www.youtube.com/playlist?list=PLbdLjg29s9lCY4hspzj3AGdAL7Vr2ys1B
Episode Details
- Show Notes: https://www.youtube.com/watch?v=Vx4Yvj1YIBM
- Published: 2026-10-07T09:48:54Z
- Duration: 00:54:02
- Author: MoTaverse
Overview
The podcast discusses the evolving role of AI in software development and testing, emphasizing both its potential and limitations. AI is highlighted as a tool that can significantly enhance productivity by automating repetitive tasks, generating tests, and improving code quality through static analysis and remediation. However, challenges remain, particularly in ensuring reliability, test coverage, and cost-effectiveness when scaling AI-driven solutions. The discussion underscores that while AI can assist in test creation, security validation, and CI/CD automation, it is not yet capable of fully autonomous development and still requires human oversight for meaningful evaluation and decision-making.
A key theme is the balance between leveraging AI for efficiency and maintaining control through deterministic processes, especially in safety-critical industries where predictability and compliance are paramount. The conversation explores how AI can virtualize unstable dependencies, stabilize testing environments, and enable non-technical users to contribute to testing through natural language commands. At the same time, there is a strong emphasis on the continued importance of human judgment in recognizing anomalous behavior, curating knowledge sources, and guiding AI tools effectively. The overall outlook is one of cautious optimism - encouraging practical experimentation with AI while staying grounded in real-world applicability and maintaining a toolbox approach rather than relying on AI as a one-size-fits-all solution.
What If
-
What if you shifted from reactive bug fixing to proactive failure simulation in your solo projects?
- Move: Implement automated chaos engineering tests (e.g., simulate network failures, disk full errors, API timeouts) using lightweight tools like Toxiproxy or custom fault injection in your CI pipeline.
- Why Now?: As your software scales - even as a solo developer - unexpected edge cases become more likely; catching them early prevents post-deployment fires that disrupt your workflow and erode user trust.
- Expected Upside: Reduce emergency debugging time by 40 - 60%, increase release confidence, and avoid “firefighter mode” during vacations or critical launches.
-
What if you used AI not to generate tests, but to maintain and optimize your existing test suite?
- Move: Integrate AI as a maintenance assistant - use LLMs to auto-update test assertions after refactoring, detect flaky tests, and summarize test results in natural language for quick review.
- Why Now?: AI-generated code changes make traditional test suites brittle; manual upkeep is unsustainable for solo developers facing rapid iteration.
- Expected Upside: Cut test maintenance time by 50%, improve test relevance with test impact analysis, and maintain high coverage without bloating execution time.
-
What if you treated 100% code coverage as a minimum baseline - not a goal - using AI-augmented static analysis?
- Move: Combine free static analyzers (e.g., SonarQube, CodeQL) with AI prompts that suggest edge-case tests for uncovered branches, especially in safety-critical paths (e.g., payment handling, data deletion).
- Why Now?: With AI making thorough testing more feasible, settling for 60 - 80% coverage exposes solo developers to avoidable risk, especially when dependencies or user loads grow unexpectedly.
- Expected Upside: Achieve demonstrable reliability for investors or clients, reduce production incidents by targeting untested code, and build audit-ready quality evidence without team-scale overhead.
Takeaway
- Implement test impact analysis to run only relevant tests after code changes, reducing execution time and costs.
- Use AI to assist with test maintenance and creation but retain human oversight to ensure coverage, repeatability, and alignment with requirements.
- Prioritize deterministic testing methods for critical functionality, avoiding reliance on non-deterministic AI for tasks requiring consistent outcomes.
- Curate and maintain up-to-date internal documentation (specs, notes, test plans) to enable effective AI use in testing and development workflows.
- Experiment with AI tools in daily development tasks to identify practical, cost-effective use cases while avoiding overuse for problems better solved with traditional methods.
For a PDF of longer Software Testing Podcast Episode Summaries with Briefing Notes and more detailed summary notes, visit EvilTester Patreon Podcast Summaries.