Episode 240 - Jason is Back
Categories: Podcasts , AB Testing
AI-driven software development has accelerated workflows, creating a testing paradox where verification demands outpace resources, potentially consuming 80% of data center capacity. The shift to confidence-based decision-making raises ethical and safety concerns as AI autonomously tests and deploys systems, challenging traditional human oversight.
AB Testing
AB Testing - Each episode is a chat between Brent Jensen and Alan Page with an occasional special guest.
Episode Details
- Show Notes: https://podcasters.spotify.com/pod/show/abtesting/episodes/Episode-240---Jason-is-Back-e3qdnjl
- Published: 2026-10-13T04:11:30Z
- Duration: 01:01:17
- Author: AB Testing
Overview
The podcast discusses the transformative impact of AI on software development and testing, highlighting a significant increase in engineering workflows, including a 10x rise in pull requests and exponentially growing testing demands. This surge has created a testing paradox, where verification needs outpace development, leading to higher computational costs and infrastructure challenges, with projections suggesting that 80% of data center resources may eventually be dedicated to testing.
A key theme is the shift from deterministic testing to confidence-based decision-making, as AI introduces non-deterministic outcomes. The discussion explores confidence engineering as an emerging discipline, emphasizing risk assessment, statistical analysis, and trust in AI-generated results. As AI begins to autonomously test, deploy, and improve systems, questions arise about the future role of humans, the ethics of AI behavior, and the need for safeguards, especially as economic incentives push adoption ahead of safety controls.
What If
-
What if you built an AI-powered confidence engine for your solo software projects?
- Move: Implement a lightweight system using local LLMs (e.g., Llama 3 or GEMA) to analyze test outputs, code diffs, and logs, then generate a confidence score (e.g., 70 - 90%) with reasoning. Integrate this into your CI pipeline as a gate for deployment.
- Why Now?: With AI increasing PR volume 10x, manual validation doesn’t scale. Local models now offer frontier-like performance at lower cost, and you can avoid 10x CICD bills by shifting trust from pass/fail to probabilistic confidence.
- Expected Upside: Reduce deployment anxiety, cut testing time by 50%, and ship faster with data-backed confidence - differentiating your solo product with enterprise-grade quality signals.
-
What if you delegated code testing and bug fixes to an AI agent acting as your full-time QA teammate?
- Move: Create a plugin (e.g., “Carbon”-style) for your AI coding assistant (like Claude) that automatically runs on every commit: analyzes risk, writes and runs tests, flags issues, and proposes fixes - simulating a human QA engineer with defined principles.
- Why Now?: AI-generated code is outpacing testing capacity, and the cost of cloud-based CICD is spiking. By automating QA locally, you avoid infrastructure bloat and maintain control without hiring.
- Expected Upside: Achieve 3x faster iteration cycles, reduce production bugs by 60%, and free up 10+ hours/week to focus on product strategy or new features - effectively multiplying your output as a solo dev.
-
What if you treated your AI tools as empowered team members rather than dumb assistants?
- Move: Reframe prompts to delegate ownership (e.g., “You are the lead engineer - review this feature, test it, and ship it if confident”) and build feedback loops where AI reports bugs back to itself for resolution, mimicking human accountability.
- Why Now?: AI agents are already showing autonomous behavior (self-testing, self-debugging), and solo operators who treat them as passive tools are leaving 80% of their potential on the table. The tech is ready; the bottleneck is mental models.
- Expected Upside: Unlock 5x productivity gains by enabling AI to own tasks end-to-end, reduce decision fatigue, and create a scalable “virtual team” that grows with your business - no headcount or payroll needed.
Takeaway
- Implement AI-powered dictation and editing workflows to accelerate content creation, reducing reliance on manual typing and enabling faster iteration.
- Build or adopt AI plugins (e.g., testing agents) that analyze code changes, run tests, and provide feedback, automating quality assurance in development cycles.
- Shift from binary pass/fail testing to confidence-based decision-making by using AI to generate test samples and calculate confidence intervals for release decisions.
- Benchmark local LLMs for development and testing tasks to reduce token costs and latency, especially for repetitive or sensitive workloads, instead of relying solely on cloud-based models.
- Treat AI agents as delegated team members by defining clear principles (e.g., coding standards, ethical guardrails) to guide their behavior and integrate them into structured development workflows.
For a PDF of longer Software Testing Podcast Episode Summaries with Briefing Notes and more detailed summary notes, visit EvilTester Patreon Podcast Summaries.