Cerebras AI Enhances Software Testing Reliability and Efficiency
Cerebras Systems' new AI-driven QA framework for its Cloud Console allows tests to be expressed in plain English, dramatically reducing the fragility of traditional testing methods. Discover how this innovation transforms software quality assurance.
Key Facts
- Cerebras' AI-driven QA reduces E2E test flakiness by eliminating markup dependencies, enhancing reliability.
- The self-healing tests automate triage, cutting morning bug triage time significantly, boosting efficiency.
- Risk-scoring PRs before release minimizes outages, indicating a proactive approach to software quality.
- Cloud-native architecture scales testing processes, showcasing competitive agility in a fast-paced market.
- Transitioning QA to AI-driven models reflects a strategic shift towards automation, optimizing resource allocation.
Summary
Summary
Cerebras deployed an AI agent to automate quality assurance (QA) for their Cloud Console, addressing the challenge of maintaining test reliability amidst rapid development. By implementing a framework of plain-English tests run by a language model (LLM) agent, they significantly reduced end-to-end test flakiness. This innovation led to a more resilient testing suite that adapts to UI changes and streamlines the release process.
Background
Cerebras Systems operates in the technology sector, specializing in AI hardware and software solutions. As a leader in providing high-speed processing capabilities, the company needed to ensure that their Cloud Console, which developers interact with daily, maintained its reliability and performance. Before the deployment of the AI agent, traditional testing methods were prone to failures due to minor UI changes, resulting in increased maintenance overhead.
Challenge
The primary challenge was to create a robust QA process that could keep pace with the rapid development cycles of the Cloud Console. Traditional testing scripts often broke due to changes in the UI, leading to false positives and increased time spent on triaging test failures.
Solution
Cerebras developed an internal framework called console-prompt-tests, where each test is written as a paragraph in plain English. This framework utilizes an LLM agent that drives a real browser, allowing it to interact with the UI as a human would. The implementation involved three key components: a prompt runner that executes the tests, a Playwright instance for browser control, and a testing environment that uses real user authentication. This setup enables the agent to adapt to UI changes without failing, as it focuses on user intent rather than specific markup.
Results
The deployment of the AI agent resulted in a significant reduction in end-to-end test flakiness. The testing suite now covers dozens of critical user journeys across the console, and the automated generation of tests from both PDF plans and live UI baselines ensures that coverage evolves with the product. The system also includes a triage mechanism that classifies test failures and proposes fixes for common issues, such as UI drift, thereby reducing the need for human intervention.
Key Insights
- Describing tests in terms of user intent rather than specific implementation details can eliminate a major source of testing flakiness.
- Allowing the AI agent to perceive the UI enhances the robustness of the testing process.
- Separating the proposal and application of changes ensures that only verified fixes are implemented, maintaining the integrity of the release process.
- Automating the generation and scoping of tests can yield greater efficiency than merely running existing tests.
Customer Testimonial
No direct quote is available from the source material.
Entities Mentioned
Companies
Products
Technologies
Key Concepts
Definitions
- AI Agent
- An automated system that performs tasks typically done by a human, such as quality assurance testing in this context.
- End-to-End Testing
- A testing methodology that validates the entire application flow from start to finish to ensure all components work together as expected.
- Self-Healing Tests
- Tests that automatically adapt to changes in the user interface, reducing maintenance overhead and improving reliability.
- Continuous Integration
- A software development practice where code changes are automatically tested and merged into a shared repository frequently.
- Cloud-Native Pipeline
- A software delivery pipeline designed to leverage cloud computing resources for scalability and efficiency.
Use Cases
- →Automating QA for the Cerebras Cloud Console
- →Generating tests from PDF test plans
- →Creating sanity tests from live UI baselines
- →Classifying test failures for efficient triage
- →Risk scoring pull requests before releases
- →Rolling out self-healing tests to other platforms
Frequently Asked Questions
What is the main advantage of using an AI agent for QA?
The main advantage is the ability to automate testing processes, making them more resilient to UI changes and reducing the manual effort required for maintenance.
How does the self-healing feature work?
The self-healing feature analyzes test failures, identifies UI drift, and proposes fixes based on evidence from screen recordings and logs, ensuring that tests remain valid without manual intervention.
What technologies are used in this QA automation?
The automation utilizes technologies such as Python for scripting, Playwright for browser automation, and AWS services for cloud orchestration and scaling.
How does the system ensure the safety of releases?
The system scores the risk of changes and requires human approval for merges, ensuring that only thoroughly tested and validated changes are deployed.
Can this framework be applied to other products?
Yes, the framework is designed to be scalable and is already being rolled out to other products like the API Gateway and API Server, extending its benefits beyond the Cloud Console.