As AI agents take on a growing role in software testing, ensuring transparency, accountability and reliable verification of their actions is becoming a critical challenge for quality engineering teams. TestMu AI is addressing this need through evidence packs in its Kane CLI platform, which capture step-by-step test actions, annotated screenshots, network activity, console logs and failure records in a single auditable file. In this interview with AI Spectrum, Mudit Singh, Co-founder and Head of Growth at TestMu AI, discusses how evidence-based testing can improve the traceability of agentic AI workflows, help engineering teams distinguish application defects from agent errors, and strengthen security and governance. He also shares his perspective on how AI agents will reshape software testing, the evolving role of quality engineering professionals, and the next phase of autonomous testing and failure diagnosis.
As AI agents increasingly write and execute software tests, what are the key challenges around visibility and accountability in agent-driven quality engineering? How does Kane CLI's evidence pack approach address these gaps?
When a person writes and runs a test, you can ask them what they did. When an agent does it, you get a verdict and a transcript the agent wrote about itself. That leaves two gaps. One is visibility: what did the agent do at each step, and what did it see? The other is accountability: who ran the test, in what environment, and can someone else reproduce it?
Evidence packs close both. Every Kane CLI run produces one sealed .evidence file. It holds a screenshot for each step with the element the agent acted on highlighted, a HAR network log and console output tied to the step that produced them, a result summary, and a failure record for anything that broke. The pack is the record, so there’s no separate log folder to hunt through. It’s saved in the project, so teams can keep it alongside their code and diff it in git.
What role do evidence packs play in making agentic AI testing more transparent and auditable? How can engineering teams use the captured test definitions, step-by-step actions, screenshots, and logs to understand an agent's decision-making and outcomes?
Transparency means you don’t have to trust the summary. Open the pack in the viewer and walk through the run step by step: what was asked, what the agent saw, what it did, and what the browser reported at that moment. The trace records the agent’s reasoning alongside its actions, so you can follow why it made each choice. The annotated screenshots are the most useful part, because they show the exact element the agent chose.
Audit starts with the result summary, which records who ran the test, the duration, the host OS, and the browser viewport. Each test step links to the acceptance criterion it proves, so a reviewer can trace any check back to a requirement. Referenced sources travel inside the pack with a content fingerprint that can be checked offline, which makes tampering visible. Teams can validate a pack’s structure and completeness as a CI gate, and merge packs from several runs into one record for a release. Coverage is read off the same evidence, so it reports which requirements were proved instead of how many steps ran.
AI agents can sometimes produce unexpected actions or outcomes during testing. How does TestMu AI ensure that the evidence generated by Kane CLI provides sufficient context to investigate failures and distinguish between an application issue and an agent-related error?
The pack doesn't make that judgment for you. It makes the judgment possible, and quick. On a failed run, each failed step carries the error message, the page state at the point of failure, and references into the console and network logs for that step.
In practice, the pattern is clear within a minute. If the screenshot shows the right element, the agent acted on it, and the network log shows a 500 from the backend, that's the application. If the page was healthy and the annotated screenshot shows the agent picked the wrong element, that's the agent. And because the test definition is in the pack, you can also see whether the instruction itself was ambiguous, which is often the real cause. The point is that the evidence is there for every step, not reconstructed afterwards.
What considerations around data security, privacy, and governance arise when evidence packs capture browser sessions, screenshots, console logs, and network activity? What safeguards are important for organisations using these capabilities in production environments?
Packs capture real pages, so they can contain real data: whatever was on screen, in request payloads, or in console output. We designed around that. Packs are written to the developer's machine and project folder, and the viewer runs from a localhost-only server. Nothing leaves the environment unless the team chooses to upload the run to Test Manager or share the link.
For organisations, three safeguards matter most. Run against test accounts and test data, not production. Treat .evidence files like any other build artifact, with the same access controls and retention rules in CI. And decide deliberately which runs get published to a shared dashboard. The governance upside is that the pack is itself the audit trail: who ran what, when, and what happened.
How do you see agentic AI changing the role of software testers and developers over the next few years? Will AI primarily augment existing quality engineering teams, or could it increasingly take ownership of parts of the testing lifecycle?
Augmentation first, then ownership of the repetitive parts. Agents will take over execution, first-pass authoring, and the initial diagnosis of failures. What stays with people is deciding what matters to test, judging risk, reading the evidence, and signing off.
So the role shifts from writing scripts to setting intent and reviewing evidence. The tester becomes the reviewer. I'd add that this makes quality engineering teams more important, not less. AI is now writing code faster than teams can verify it, and that validation debt lands on the people responsible for quality. Teams that learn to direct agents and audit their evidence will carry far more weight than they do today.
Looking ahead, what are the next capabilities TestMu AI plans to develop for Kane CLI and agentic quality engineering? How do you see AI agents evolving from simply executing tests to independently identifying risks, diagnosing failures, and improving test coverage?
Kane CLI launched in April as a way to run tests on real browsers and devices from the terminal in plain English. Since then, we've added test case generation, the Assurance Lifecycle that ties tests back to requirement documents, and now evidence packs. Each step gives the agent more context and gives the team more proof.
The next direction is agents that use that evidence themselves: reading past runs to spot recurring failures and flaky steps, proposing coverage where requirements have none, and diagnosing a failure before a person opens the pack. The other half is the agents being built by our customers. The same principle applies there: an agent's own report of what it did is the weakest evidence available, and the enterprise needs an independent record of what actually happened.
