AI-powered log mining tools like Tanvis Log Miner automate regression test case generation from structured production logs, using pattern recognition to prioritize high-impact testing in complex systems, while challenges persist in replicating real-world transaction scenarios and full CI/CD integration. The approach emphasizes collaboration between developers and testers to enhance log quality and balance AI-driven automation with human oversight in high-risk domains like finance.

Test Guild

Test Guild - hosted by Joe Colantonio has main topic focus on Testing or Automating. Each episode has a different guest. Show notes have comprehensive links and usually a full transcript. Released as audio and video.

• https://testguild.com/
• https://testguild.com/podcasts/automation/
• https://www.youtube.com/playlist?list=PL9AgRtJkydU1jqvx46esyr56BXtm1QEds
• https://www.youtube.com/@JoeColantonio

Episode Details

• Show Notes: https://testtalks.libsyn.com/ai-testing-from-production-logs-generate-smarter-regression-tests-with-tanvi-mittal
• Published: 2026-03-17T21:56:00Z
• Duration: 27:47
• Author: Unknown

Overview

The podcast discusses leveraging AI for log mining and shift-right testing, emphasizing the underutilized potential of production logs as sources for regression test cases. AI tools, like the Tanvis Log Miner, analyze structured logs (e.g., fraud alerts, transaction sequences) to identify event patterns, generate test scenarios in Gherkin format, and prioritize testing based on high-impact features. This approach addresses challenges in traditional testing, particularly for AI-driven systems, which require continuous monitoring due to their non-deterministic behavior. The tool uses Python and transformer models to process logs, ensuring data privacy through anonymization (e.g., hash tokens), and aims to reduce manual log analysis by converting large datasets into structured test cases. However, it currently lacks full integration into CI/CD pipelines and relies on human validation for automation.

Key challenges highlighted include the need for structured, meaningful log data (e.g., account details, payment info) and the limitations of current AI tools in replicating real-world transaction sequences (e.g., simultaneous deposits and third-party payments). Shift-right testing, which uses production insights to inform test design, helps catch bugs that escape pre-deployment testing. The discussion also underscores the importance of collaboration between testers and developers to improve log quality and test coverage. While AI tools like Log Miner enhance efficiency by automating repetitive tasks, they are not meant to replace human judgment. Testers are advised to focus on understanding AIs probabilistic nature and integrating these tools into existing workflows rather than becoming AI/ML experts.

The podcast emphasizes that AI testing requires updated strategies, balancing automation with human oversight, particularly in high-risk domains like finance. Future efforts aim to expand the tools use cases (e.g., security vulnerability detection) and improve CI/CD compatibility. However, adoption hurdles persist, including concerns over open-source AI tools security and the need for domain-specific customization. The conversation also highlights the role of logs in identifying high-risk areas and prioritizing testing within limited timelines, while reinforcing that AI should augmentnot replacetesters expertise in complex, non-deterministic systems.

What If

• What if you integrated Tanvis Log Miner into your CI/CD pipeline to automatically generate regression tests from production logs?

  • Move: Install the open-source tool, process your existing production logs (e.g., CSV/JSON from Datadog), and generate Gherkin test cases for high-risk scenarios.
  • Why_now: CI/CD pipelines are already set up for automation, and the tools CLI interface allows seamless integration without exposing sensitive data.
  • Expected_upside: Save 10+ hours per sprint on manual log analysis, catch environment-specific bugs (e.g., session expiration), and improve regression test coverage by 30%+.

• What if you used log analysis to identify and prioritize high-risk transaction sequences for testing in your next release cycle?

  • Move: Cluster similar log events (e.g., “deposit” or “third-party payment”) using the tools pattern recognition, then manually validate and add these as high-priority tests.
  • Why_now: Production logs reveal edge cases (e.g., transaction sequences) that isolated tests miss, and your team has limited testing time (e.g., six weeks for a release).
  • Expected_upside: Reduce critical production bugs by 50% by focusing testing on high-impact, real-world scenarios instead of legacy test suites.

• What if you built a custom script to anonymize sensitive data in logs before feeding them into Tanvis Log Miner, ensuring compliance with data privacy standards?

  • Move: Write a pre-processing script that replaces sensitive fields (e.g., account numbers, SSNs) with hash tokens using the tools existing anonymization framework.
  • Why_now: The text emphasizes data security (e.g., P2 compliance), and enterprise-grade log analysis requires sanitization before processing.
  • Expected_upside: Enable safe log analysis for compliance-critical sectors (e.g., finance) while retaining the ability to extract meaningful test cases from production logs.

Takeaway

• Implement structured logging with clear naming and formats (e.g., JSON, CSV) to ensure logs are analyzable by tools like Tanvis Log Miner, enabling automatic test case generation.
• Use the open-source Tanvis Log Miner tool to process production logs and generate Gherkin-formatted regression test cases, focusing on high-impact scenarios identified in logs (e.g., fraud alerts, transaction sequences).
• Anonymize sensitive data in logs using hash tokens to comply with privacy standards (e.g., P2 data handling), ensuring log analysis doesnt expose confidential information like account numbers.
• Create domain-specific JSON mappings to link log events (e.g., “session started,” “user click on payment button”) to actionable test scenarios, improving the accuracy of generated test cases.
• Validate AI-generated test cases manually to ensure they align with real-world edge cases, such as transaction sequences or session expiration risks, which may not be fully modeled by automated tools.