Bengaluru-based Drizz is testing the limits of visual AI agents in software quality assurance, revealing significant performance variance across complex environments.
Bengaluru-based startup Drizz is evaluating the efficacy of visual-based AI agents to automate mobile application testing, a domain historically reliant on brittle, identifier-based scripting. The company’s primary offering, Fathom, utilizes vision models to interpret application screens and execute test sequences via natural-language instructions, aiming to mitigate the maintenance burden associated with frequent interface modifications.
Yash Varyani, co-founder and chief technology officer at Drizz, identified that conventional automation frameworks like Appium often fail when UI elements undergo minor structural changes. By shifting the focus from internal element identifiers to visual interpretation, the system attempts to achieve a form of self-healing functionality. This approach allows the agent to navigate unexpected interface interruptions, such as promotional pop-ups, without terminating the test execution flow.
Internal benchmarks conducted by Drizz indicate a reduction in test flakiness from a range of 15% to 50% down to approximately 3%. These figures, however, remain highly sensitive to the specific testing environment and the underlying model architecture employed. The company emphasizes that these results reflect the performance of the entire system, including the agent scaffold and the specific Android image, rather than the isolated capabilities of the frontier models.
Testing across six different model configurations and 20 distinct tasks revealed significant performance disparities between environments. In a standard Android image suite, models achieved a 72.3% pass rate, successfully completing 217 of 300 runs. Conversely, a more rigorous suite utilizing a rooted Android image—which allowed for direct database inspection—saw the pass rate plummet to 28.7%. The variance in model rankings between these two suites highlights the instability of current vision-based agents when confronted with increased environmental complexity.
Performance metrics fluctuated notably, with models such as Gemini 3.5 Flash demonstrating superior resilience in the harder testing suite compared to various GPT configurations. Five of the ten tasks in the advanced suite resulted in zero successful runs across all tested models, a discrepancy that remains under investigation by the development team. These findings suggest that while visual interpretation offers a path toward more robust testing, the reliability of such agents is currently contingent upon the specific task-model pairing.
The company has expanded its reach through multiple deployment channels, including desktop applications, web interfaces, and continuous integration pipelines. By integrating with existing workflows, Drizz aims to provide a more flexible alternative to traditional YAML-based testing tools like Maestro. This multi-modal approach ensures that developers can leverage the agent’s reasoning capabilities within their existing terminal-based environments or through automated CI/CD workflows.
Data privacy remains a critical constraint for the deployment of these agents in enterprise environments. To address concerns regarding the exposure of sensitive financial or personal information, Drizz is prioritizing on-premise model deployments. The company is currently developing a smaller, sub-one-billion parameter model intended to handle a significant portion of the testing workload, thereby reducing reliance on cloud-based frontier models.
The integration of these agents into software quality assurance marks a shift toward more autonomous verification processes. While Drizz currently maintains a command-line tool and a model context protocol interface, the long-term objective involves verifying the actions of other AI agents as they begin to interact with applications on behalf of users. This potential expansion into agent verification represents a significant shift in the company’s strategic roadmap, with leadership projecting that such contracts could eventually exceed current testing revenue by an order of magnitude.
Future iterations of the Fathom agent will need to reconcile the performance gaps observed in high-complexity environments to achieve broader industrial adoption. The reliance on visual cues necessitates a high degree of precision in screen interpretation, particularly as applications move toward more dynamic and non-standardized interface designs. Continued research into smaller, specialized models may provide the necessary balance between computational efficiency and task-specific accuracy required for enterprise-grade software verification.
