System Failure Risk: The Questions Leaders Aren't Asking
Categories: Podcasts , The Value of Software Testing
Recent outages in travel and banking sectors highlight how prioritizing release speed over rigorous testing exacerbates risks from technical debt and cloud dependencies. Leaders must implement combinatorial testing and certified personnel to effectively detect defects and ensure system resilience.
The Value of Software Testing
Randy Rice has a video Software Testing podcast - solo shows and interviews. Youtube only.
- https://www.youtube.com/playlist?list=PLGrFXPvIwr2WR6wn-Ngw7_9X_Ec3WO4vK
- https://www.riceconsulting.com/
Episode Details
- Show Notes: https://www.youtube.com/watch?v=-cTpZoOkHS8
- Published: 2026-11-20T15:24:07Z
- Duration: 00:39:22
- Author: Rice Consulting Services, Inc.
Overview
A cascade of system failures across multiple industries in a single week - including a Microsoft 365 outage, an FAA ground stop, Workday disruptions, United Airlines delays, major bank online banking failures, an Apple service outage, and NSA internal system degradation - prompted urgent questions about software release practices. Bloomberg reported the FAA ground stop alone could cost airlines more than $1 billion, citing a Raymond James estimate, potentially making it the costliest system failure to date when individual and organizational costs are included. Earlier incidents, such as the Boeing 737 MAX 8 crashes in 2018 and 2019 that killed 346 people due to erroneous sensor readings and inadequate pilot training, and the freezer incidents spoiling food at US military commissaries and European grocery stores between August and November 2026, underscored recurring patterns of insufficient testing and training.
The central argument is that testing has been systematically gutted from software delivery. Companies have fired testers, cut training budgets, and relied on AI and automation under the assumption these can replace rigorous testing. In reality, most CI/DevOps automation is confirmatory rather than exploratory, verifying that a build is stable rather than hunting for defects. A 2026 Black Duck analysis of 947 commercial code bases comprising 2,843 individual projects found open source components in 98% of audited applications, with the average number of open source vulnerabilities per application rising 107% to 581 per code base.
Research findings from NIST (2001) showed that 98% of 109 sufficiently detailed medical device failure reports could have been detected by testing with all pairs of parameter settings. Capers Jones found that certified testers combined with white box structural testing, defect prevention, and pre-test defect removal can achieve over 99% defect removal efficiency. The podcast argues that leaders must ask pointed questions about release criteria, rollback testing, tester certification, and cyber-attack detection before approving any release.
What If
-
What if you treated every release as a potential failure event and rehearsed the rollback before shipping?
- Move: Before your next deploy, actually execute a full rollback in a staging environment timed against a stopwatch. Document who is on call, how long detection takes, and what the customer-facing impact window is.
- Why Now?: The speaker notes that most CI/DevOps pipelines proceed when tests are green, with no rollback rehearsal. The FAA ground stop and bank outages showed that untested recovery paths turn a small defect into a multi-hour public failure.
- Expected Upside: A rehearsed rollback (proposed) could cut your mean-time-to-recovery from hours to minutes, reducing the window in which customers experience broken functionality and limiting reputational damage.
-
What if you ran pairwise combinatorial tests on your top three user workflows instead of relying solely on your existing automation suite?
- Move: Use the NIST ACTS tool (free, runs on Mac, Windows, and Linux) to generate all-pairs test cases for your most critical parameter combinations - login states, payment paths, data-entry boundaries - and execute them manually or scripted before each release.
- Why Now?: The NIST 2001 study found 98% of 109 detailed medical device failures were detectable by testing all pairs of parameter settings. Your current automation is confirmatory; it does not explore interaction defects between conditions.
- Expected Upside: Pairwise coverage (proposed) targets the interaction defects that single-condition tests miss, potentially surfacing the boundary and dependency bugs that cause cascading failures in production.
-
What if you audited your open-source and AI-generated dependencies for known vulnerabilities before every release?
- Move: Run a software composition analysis scan on your dependency tree, flag any components with known CVEs in PHP, Java, or other core languages, and maintain a written risk register showing which vulnerabilities you accepted versus remediated.
- Why Now?: The 2026 Black Duck analysis found 581 average open-source vulnerabilities per code base, up 107%, across 947 commercial code bases. Major PHP and Java vulnerabilities affect most of the software stack most solo developers rely on.
- Expected Upside: A documented vulnerability register (proposed) gives you defensible evidence of due diligence, reduces the chance of shipping a known-exploitable defect, and shortens incident response when a new CVE drops.
Takeaway
- Execute a timed rollback rehearsal in staging before every production release, documenting detection time, on-call response, and customer impact window so recovery is practiced rather than improvised.
- Generate all-pairs test cases for your top three critical user workflows using the free NIST ACTS tool, supplementing your existing green-build automation with interaction-level coverage that catches dependency and boundary defects.
- Run a software composition analysis scan on your full dependency tree before each release, maintain a written risk register of accepted versus remediated open-source vulnerabilities, and track CVEs in your core languages (PHP, Java, C++) between releases.
- Require that at least one tester on your team holds a current recognized certification (e.g., CTFL) and receives meaningful training annually, treating certification as evidence of due diligence rather than optional credentialing.
- Before approving any release, answer in writing: what could stop this release today, who makes that call, what are the release criteria, and what is the level of risk if you ship right now - then share that risk statement with stakeholders so “we assumed someone tested that” never becomes the post-mortem.
For a PDF of longer Software Testing Podcast Episode Summaries with Briefing Notes and more detailed summary notes, visit EvilTester Patreon Podcast Summaries.