The conversation examines AI’s role in creativity and efficiency, contrasting it with deterministic automation, while emphasizing risk management through small, frequent deployments and cultural shifts. It highlights the need for real-time feedback, controlled rollouts, and leadership-driven change to improve data-driven decision-making and reduce deployment risks.

AB Testing

AB Testing - Each episode is a chat between Brent Jensen and Alan Page with an occasional special guest.

• https://www.moderntesting.org/

Episode Details

• Show Notes: https://podcasters.spotify.com/pod/show/abtesting/episodes/AB-Testing-237-Another-Show-About-Everything-and-Nothing-e3or81j
• Published: 2026-09-14T15:18:14Z
• Duration: 01:01:04
• Author: AB Testing

Overview

The discussion covers a range of professional and personal topics, beginning with reflections on writing, creativity, and the use of AI tools to improve efficiency. One speaker shares experiences from a recent job transition into a data science role, highlighting organizational inefficiencies, excessive work hours, and an overemphasis on marketing rather than data-driven decisions. The importance of shifting toward better testing practices, telemetry, and continuous deployment is emphasized, with criticism of traditional staging environments in favor of ephemeral environments and controlled rollouts.

Key themes include risk management through small, frequent deployments, the limitations of pre-production testing, and the need for cultural change over tooling. The conversation explores how AI differs from deterministic automation by behaving more like a fallible human, introducing unpredictability that must be managed. Strategies such as flight control, A/B testing, and rollback mechanisms are presented as ways to build team confidence and reduce customer impact. There’s also a strong focus on overcoming resistance to change, treating deployments as experiments, and using real-time feedback - exemplified by Google’s weekly Android updates - to drive improvements. The role of trust, incremental progress, and leadership in fostering adoption of modern practices is underscored throughout.

What If

• What if you treated your next software release as a field experiment instead of a final product?

  • Move: Deploy a small, measurable change (e.g., UI tweak or config update) behind a feature flag with telemetry tracking user behavior and system performance.
  • Why Now?: Delayed releases based on over-testing create calendar-time debt; competitors ship faster using real-world feedback - starting small reduces risk while building deployment muscle.
  • Expected Upside: You gain actionable data within days instead of weeks, build team confidence in rollback systems, and shift culture from “perfect pre-launch” to “learn fast in production.”

• What if you replaced your staging environment with ephemeral, production-like instances for testing?

  • Move: Spin up short-lived environments per pull request using infrastructure-as-code, integrate canary deployment tools, and run smoke tests against live-like data configurations.
  • Why Now?: Traditional staging environments drift from production and give false confidence; modern tooling makes ephemeral environments cheaper and more reliable than maintaining stale sandboxes.
  • Expected Upside: You eliminate environment-specific bugs, reduce cycle time by 30 - 50%, and align development workflow with continuous deployment practices used by high-performing teams.

• What if you delegated repetitive test validation to AI and focused your effort on diagnosing edge cases?

  • Move: Use AI to generate and execute routine regression checks, then dedicate your time to analyzing anomalies, improving observability, and refining failure-detection logic.
  • Why Now?: AI tools now simulate human judgment at scale with consistent error rates - leveraging them frees solo developers to focus on high-leverage debugging and system resilience.
  • Expected Upside: You cut verification time by up to 70% while increasing coverage of real-world failure modes, turning QA from a bottleneck into a strategic advantage.

Takeaway

• Implement ephemeral environments and controlled rollouts (e.g., 5% flighting) to reduce deployment risk and validate changes in production-like conditions.
• Treat every deployment as a measurable experiment by defining clear success/failure criteria, rollback plans, and incremental scaling based on data.
• Focus on telemetry and real-time monitoring in production to identify customer pain points and prioritize fixes, replacing long pre-release testing cycles.
• Shift from tooling-first to culture-first change: build trust through small, visible improvements and avoid imposing top-down overhauls in new roles.
• Self-publish technical books or content directly via platforms like Amazon KDP when traditional publishers fail to deliver, maintaining control over distribution and timeline.