Our QA process used to take 3 days per release. Now it takes 4 hours. But the path to get here nearly broke our deployment pipeline twice.
I am not going to pretend this was a smooth transition. In October 2025, we had a client project with 380+ pages, a design system with 47 components, and a QA bottleneck that was bleeding money. Our manual testing checklist was 14 pages long. We had Selenium scripts that broke every time someone changed a button's padding. Our CI pipeline took 45 minutes on a good day, and "good days" were maybe twice a week.
Something had to change. So we spent 6 months systematically replacing every piece of our QA pipeline with AI-powered alternatives. This is the honest account of what worked, what failed spectacularly, and what we would do differently if we started over today.
The Before: A QA Process That Was Slowly Killing Us
Let me paint the picture of what our QA looked like before the switch. It is embarrassing in hindsight, but I suspect a lot of agencies run something similar.
The manual testing checklist. A Google Doc. Fourteen pages. Three columns: test case, expected result, pass/fail. A QA engineer would open every page on desktop, tablet, and mobile, click through every interactive element, and mark checkboxes. For a full regression test on a 380-page site, this took two people roughly 2.5 days.
The Selenium scripts. We had about 200 end-to-end tests written in Selenium. On paper, they covered 60% of our critical flows. In practice, about 30% of them were flaky. They would fail because a CSS animation had not finished, because a third-party script loaded slowly, because the test database had stale fixtures. We spent more time maintaining these tests than writing new features.
The CI pipeline. GitHub Actions running those Selenium tests. Average time: 45 minutes. But it frequently timed out at 60 minutes. Developers would push a fix, wait 45 minutes, find out a flaky test failed, re-run the pipeline, wait another 45 minutes. Some days we burned 3+ hours just waiting for CI.
The bug triage process. Bugs came in through Slack, email, Jira, and sometimes text messages. Someone had to manually categorize each one by severity, affected component, and assign it to the right developer. This was a 30-minute daily ritual that nobody wanted to own.
Here is what that looked like in numbers:
| Metric | Before (Oct 2025) |
|---|---|
| Full regression test time | 2.5 days |
| CI pipeline duration | 45 min avg |
| Test suite flakiness rate | ~30% |
| Test coverage (meaningful) | ~40% |
| Bugs caught pre-release | ~55% |
| QA engineer hours per sprint | 60+ hours |
| Bugs reaching production per month | 12-15 |
Those numbers are not great. But when you are a growing agency handling 6-8 projects simultaneously, QA is always the thing that gets squeezed.
The Decision: Replace It All (But Not at Once)
We did not wake up one morning and flip a switch. We took a phased approach over 6 months, replacing one piece of the pipeline at a time. Here is the order we followed, and why that order matters.
Phase 1 (Month 1-2): Visual regression testing -- highest ROI, easiest to measure
Phase 2 (Month 2-3): AI-generated test scripts -- reduce manual test writing
Phase 3 (Month 3-4): Bug triage automation -- save daily admin time
Phase 4 (Month 4-5): Accessibility auditing -- catch a11y issues early
Phase 5 (Month 5-6): API contract testing with AI-generated edge cases
We deliberately did not try to do everything at once. Each phase had to prove itself before we moved on to the next one.
Phase 1: Visual Regression Testing
This was the biggest win, and it is where I would tell any team to start.
The Problem
Our manual visual QA process was this: open the page, look at it, compare it to the design, check if anything looks off. Human eyes are decent at this, but they miss subtle changes. A 2px shift in button alignment. A font weight that changed from 500 to 400. A color that shifted from #1a1a1a to #1a1a1b. These micro-regressions accumulate and make a site look "off" without anyone being able to pinpoint why.
What We Tried
We evaluated three tools:
Percy (BrowserStack): Cloud-based visual testing. Takes screenshots and diffs them. Good at catching layout shifts. Pricing starts at $399/month for the team plan.
Applitools Eyes: AI-powered visual comparison using their "Visual AI" engine. Goes beyond pixel diffing -- it understands layout, content changes, and dynamic content. Starts at ~$500/month.
Custom solution with Playwright + Claude Vision API: Take screenshots with Playwright, send them to Claude's vision model for comparison. Pay-per-use.
We went with a hybrid: Playwright for screenshot capture + Applitools for AI-powered comparison, with a custom Claude Vision layer for interpreting complex visual issues.
The Setup
// visual-regression.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests/visual',
use: {
baseURL: process.env.STAGING_URL,
screenshot: 'on',
viewport: { width: 1280, height: 720 },
},
projects: [
{ name: 'desktop', use: { viewport: { width: 1280, height: 720 } } },
{ name: 'tablet', use: { viewport: { width: 768, height: 1024 } } },
{ name: 'mobile', use: { viewport: { width: 375, height: 812 } } },
],
});// tests/visual/homepage.spec.ts
import { test, expect } from '@playwright/test';
import { Eyes, Target } from '@applitools/eyes-playwright';
test.describe('Homepage visual regression', () => {
let eyes: Eyes;
test.beforeEach(async () => {
eyes = new Eyes();
eyes.setApiKey(process.env.APPLITOOLS_API_KEY!);
});
test.afterEach(async () => {
await eyes.close();
});
test('hero section renders correctly', async ({ page }) => {
await page.goto('/');
await eyes.open(page, 'Client Site', 'Homepage Hero');
await eyes.check('Hero Section', Target.region('#hero'));
await eyes.check('Navigation', Target.region('nav'));
await eyes.check('Full Page', Target.window().fully());
});
test('component library renders correctly', async ({ page }) => {
await page.goto('/design-system');
await eyes.open(page, 'Client Site', 'Design System');
// Check each component in isolation
const components = await page.$$('[data-testid^="component-"]');
for (const component of components) {
const testId = await component.getAttribute('data-testid');
await eyes.check(testId!, Target.region(component));
}
});
});The Claude Vision Layer
For complex visual issues that Applitools flagged but could not explain well, we added a Claude Vision API step:
// lib/visual-analysis.ts
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
interface VisualAnalysis {
hasIssues: boolean;
issues: Array<{
severity: 'critical' | 'major' | 'minor';
description: string;
location: string;
suggestion: string;
}>;
}
export async function analyzeVisualDiff(
baselineScreenshot: Buffer,
currentScreenshot: Buffer,
context: string
): Promise<VisualAnalysis> {
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 1024,
messages: [
{
role: 'user',
content: [
{
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: baselineScreenshot.toString('base64'),
},
},
{
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: currentScreenshot.toString('base64'),
},
},
{
type: 'text',
text: `Compare these two screenshots. The first is the baseline (expected),
the second is the current version. Context: ${context}
Identify any visual differences that would affect user experience.
Ignore: minor anti-aliasing differences, dynamic content like dates/times.
Flag: layout shifts, missing elements, color changes, font changes,
broken alignments, overlapping elements, responsive issues.
Return JSON: {
"hasIssues": boolean,
"issues": [{ "severity": "critical|major|minor",
"description": "...",
"location": "...",
"suggestion": "..." }]
}`,
},
],
},
],
});
return JSON.parse(
response.content[0].type === 'text' ? response.content[0].text : '{}'
);
}Results from Phase 1
After 2 months of visual regression testing:
- Caught 23 visual regressions that manual testing had missed in the previous 3 months
- Reduced visual QA time from 8 hours to 20 minutes per release
- False positive rate: 12% initially, dropped to 4% after tuning ignore regions for dynamic content
- Cost: $399/month (Applitools) + ~$15/month (Claude Vision API calls) = $414/month
The $414/month paid for itself in the first week. We were spending 16+ hours of QA engineer time per sprint on visual checking alone.
Phase 2: AI-Generated Test Scripts
This is where things got interesting -- and where we made our biggest mistakes.
The Idea
Instead of writing Playwright test scripts by hand, feed user stories and acceptance criteria to Claude and have it generate test code. In theory, this should dramatically increase test coverage with minimal human effort.
The Reality
Here is how we set it up:
// scripts/generate-tests.ts
import Anthropic from '@anthropic-ai/sdk';
import * as fs from 'fs';
const anthropic = new Anthropic();
interface UserStory {
id: string;
title: string;
acceptanceCriteria: string[];
pageUrl: string;
selectors?: Record<string, string>;
}
async function generateTest(story: UserStory): Promise<string> {
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 4096,
system: `You are a senior QA engineer writing Playwright test scripts.
Generate comprehensive, production-ready tests.
Use data-testid selectors when available.
Include negative test cases and edge cases.
Use proper async/await patterns.
Add meaningful assertion messages.
Do NOT generate tests that test implementation details -- test user-visible behavior.`,
messages: [
{
role: 'user',
content: `Generate Playwright tests for this user story:
Title: ${story.title}
Page URL: ${story.pageUrl}
Acceptance Criteria:
${story.acceptanceCriteria.map((ac, i) => `${i + 1}. ${ac}`).join('\n')}
Available selectors:
${JSON.stringify(story.selectors || {}, null, 2)}
Generate a complete test file with describe/test blocks.`,
},
],
});
const text =
response.content[0].type === 'text' ? response.content[0].text : '';
// Extract code block
const codeMatch = text.match(/```typescript\n([\s\S]*?)```/);
return codeMatch ? codeMatch[1] : text;
}
// Usage
const stories: UserStory[] = JSON.parse(
fs.readFileSync('./user-stories.json', 'utf-8')
);
for (const story of stories) {
const testCode = await generateTest(story);
fs.writeFileSync(
`./tests/generated/${story.id}.spec.ts`,
testCode
);
console.log(`Generated test for: ${story.title}`);
}The Problem We Did Not See Coming
The AI-generated tests had a serious issue: they looked correct but tested nothing meaningful.
Here is an example. We had a user story: "User can submit the contact form with valid data." The AI generated this test:
// What the AI generated (looks fine, right?)
test('user can submit contact form', async ({ page }) => {
await page.goto('/contact');
await page.fill('[data-testid="name"]', 'John Doe');
await page.fill('[data-testid="email"]', 'john@example.com');
await page.fill('[data-testid="message"]', 'Hello');
await page.click('[data-testid="submit"]');
// This assertion is the problem
await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
});This test passes. But it does not verify:
- Was the form data actually sent to the server?
- Did the server respond with 200?
- Is the success message showing the right content?
- What happens if the email is malformed?
- What about the rate limiting we added?
The test clicks a button and checks that a div appears. That is it. It would pass even if the form submission was completely broken, as long as the UI showed a success state (which some frontend frameworks do optimistically).
We called these "theater tests" -- they perform the motions of testing without actually verifying anything. About 35% of our AI-generated tests had this problem.
The Fix
We added a review layer. Every AI-generated test goes through a quality check:
// scripts/validate-test-quality.ts
async function validateTestQuality(
testCode: string,
story: UserStory
): Promise<{
quality: 'good' | 'needs-review' | 'reject';
issues: string[];
suggestions: string[];
}> {
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 2048,
messages: [
{
role: 'user',
content: `Review this Playwright test for quality issues:
TEST CODE:
${testCode}
USER STORY:
${story.title}
ACCEPTANCE CRITERIA:
${story.acceptanceCriteria.join('\n')}
Check for:
1. "Theater tests" - tests that click buttons but don't verify outcomes
2. Missing negative test cases
3. Missing edge cases from the acceptance criteria
4. Assertions that only check visibility without verifying content
5. Missing API response validation
6. Missing error state testing
7. Hardcoded waits instead of proper waiting strategies
Return JSON: {
"quality": "good" | "needs-review" | "reject",
"issues": ["list of specific problems"],
"suggestions": ["list of improvements"]
}`,
},
],
});
return JSON.parse(
response.content[0].type === 'text' ? response.content[0].text : '{}'
);
}After adding this validation layer, our test quality improved significantly. But it also meant that about 40% of generated tests needed human revision. The AI generates the skeleton; a human makes it meaningful.
Results from Phase 2
| Metric | Manual Test Writing | AI-Generated (Before Validation) | AI-Generated (After Validation) |
|---|---|---|---|
| Tests written per day | 3-5 | 25-30 | 15-20 (after review) |
| Test quality (% meaningful) | 90% | 65% | 88% |
| Time per test | 45 min | 2 min generate + 10 min review | 12 min total |
| Coverage increase per sprint | +5% | +18% | +15% |
The net result: 3x faster test creation, but only with a human in the loop. Fully automated test generation without review is a trap.
Phase 3: AI-Powered Bug Triage
This one was straightforward and had the most immediate impact on team productivity.
The Problem
Bug reports came in three formats:
- Jira tickets from internal QA
- Slack messages from clients (often just a screenshot and "this is broken")
- Error logs from Sentry
Someone had to read every report, classify it by severity (critical/major/minor), identify the affected component, and assign it to the right developer. This took 30-45 minutes every morning and was everyone's least favorite task.
The Solution
We built a classification pipeline using Claude's API:
// lib/bug-triage.ts
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
interface BugClassification {
severity: 'critical' | 'major' | 'minor' | 'cosmetic';
component: string;
suggestedAssignee: string;
estimatedEffort: 'small' | 'medium' | 'large';
summary: string;
possibleCause: string;
relatedIssues: string[];
}
const TEAM_CONTEXT = `
Team members and their domains:
- Ravi: Authentication, user management, API endpoints
- Priya: Frontend components, CSS, responsive design
- Amit: Database, migrations, data processing
- Sneha: Third-party integrations, payment, email
- Anurag: Architecture, deployment, infrastructure
Components:
- auth: Login, signup, password reset, sessions
- ui: Design system components, layouts, pages
- api: REST endpoints, middleware, validation
- data: Database queries, migrations, caching
- integrations: Stripe, SendGrid, analytics, CMS
- infra: CI/CD, deployment, monitoring, hosting
`;
export async function triageBug(
report: string,
screenshots?: string[]
): Promise<BugClassification> {
const content: any[] = [];
if (screenshots) {
for (const screenshot of screenshots) {
content.push({
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: screenshot,
},
});
}
}
content.push({
type: 'text',
text: `Classify this bug report:
${report}
${TEAM_CONTEXT}
Return JSON: {
"severity": "critical|major|minor|cosmetic",
"component": "auth|ui|api|data|integrations|infra",
"suggestedAssignee": "team member name",
"estimatedEffort": "small|medium|large",
"summary": "one sentence summary",
"possibleCause": "your best guess at the root cause",
"relatedIssues": ["any related issue patterns you notice"]
}
Severity guide:
- critical: Data loss, security vulnerability, complete feature broken for all users
- major: Feature partially broken, workaround exists but it is bad
- minor: Edge case bug, affects few users, easy workaround
- cosmetic: Visual issue, no functional impact`,
});
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 1024,
messages: [{ role: 'user', content }],
});
return JSON.parse(
response.content[0].type === 'text' ? response.content[0].text : '{}'
);
}We integrated this with our Slack bot and Jira webhook. When a bug report comes in, it gets auto-classified and assigned within seconds.
Results from Phase 3
- Triage accuracy: 87% agreement with human classification (based on 200 reports reviewed)
- Time saved: 30+ minutes per day across the team
- Severity accuracy: 92% for critical/major (rarely misses something important), 78% for minor/cosmetic (sometimes over-classifies cosmetic issues as minor)
- Cost: ~$8/month in API calls
The 13% disagreement rate is fine because the developer who gets assigned the ticket reviews the classification anyway. It is a suggestion, not a mandate.
Phase 4: Automated Accessibility Auditing
Accessibility testing is one of those things every team knows they should do but never does thoroughly. AI made it practical for us.
The Setup
We combined automated tools (axe-core) with AI interpretation:
// tests/accessibility/audit.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
interface A11yReport {
criticalIssues: Array<{
element: string;
issue: string;
wcagCriteria: string;
fix: string;
priority: number;
}>;
score: number;
summary: string;
}
test.describe('Accessibility audit', () => {
const pages = ['/', '/about', '/services', '/contact', '/blog'];
for (const pagePath of pages) {
test(`${pagePath} meets WCAG 2.1 AA`, async ({ page }) => {
await page.goto(pagePath);
// Run axe-core
const results = await new AxeBuilder({ page })
.withTags(['wcag2a', 'wcag2aa', 'wcag21aa'])
.analyze();
// If there are violations, get AI interpretation
if (results.violations.length > 0) {
const screenshot = await page.screenshot({ fullPage: true });
const analysis = await interpretA11yResults(
results.violations,
screenshot,
pagePath
);
// Only fail on critical issues
const criticalCount = analysis.criticalIssues.filter(
(i) => i.priority <= 2
).length;
expect(
criticalCount,
`Page ${pagePath} has ${criticalCount} critical a11y issues: ${analysis.summary}`
).toBe(0);
}
});
}
});
async function interpretA11yResults(
violations: any[],
screenshot: Buffer,
pagePath: string
): Promise<A11yReport> {
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 2048,
messages: [
{
role: 'user',
content: [
{
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: screenshot.toString('base64'),
},
},
{
type: 'text',
text: `Analyze these accessibility violations for ${pagePath}:
${JSON.stringify(violations, null, 2)}
For each violation:
1. Explain the real-world impact (who is affected and how)
2. Provide a specific code fix
3. Rate priority 1-5 (1 = blocks users, 5 = nice to have)
4. Reference the WCAG criterion
Return JSON with criticalIssues array, overall score (0-100), and summary.`,
},
],
},
],
});
return JSON.parse(
response.content[0].type === 'text' ? response.content[0].text : '{}'
);
}Why AI Interpretation Matters
Raw axe-core output is noisy. It will flag 50+ issues on a typical page, and most developers do not know which ones matter. The AI layer does three things:
- Prioritizes by real impact. "Missing alt text on a decorative border image" is different from "missing alt text on the only product image."
- Provides actual code fixes. Not just "add alt text" but "add
alt='Product dashboard showing monthly analytics'to the img at line 47." - Explains impact in plain language. "A screen reader user cannot navigate this form because none of the inputs have associated labels" is more actionable than "form-field-no-label: Ensures every form element has a label."
Results from Phase 4
- Issues found on first audit: 147 across 5 key pages (we thought our site was accessible -- it was not)
- Critical issues fixed in first sprint: 23
- Ongoing issues caught per release: 3-5 new violations
- WCAG 2.1 AA compliance: went from roughly 60% to 94% in 3 months
- Cost: ~$12/month in API calls for weekly audits
Phase 5: API Contract Testing with AI Edge Cases
This was the most technically interesting phase. The idea: use AI to generate edge case inputs for our API endpoints that a human tester would never think of.
The Approach
// scripts/generate-api-edge-cases.ts
import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
interface APIEndpoint {
method: string;
path: string;
requestSchema: object;
description: string;
constraints: string[];
}
async function generateEdgeCases(
endpoint: APIEndpoint
): Promise<Array<{ name: string; input: object; expectedBehavior: string }>> {
const response = await anthropic.messages.create({
model: 'claude-sonnet-4-20250514',
max_tokens: 4096,
messages: [
{
role: 'user',
content: `Generate edge case test inputs for this API endpoint:
${endpoint.method} ${endpoint.path}
Description: ${endpoint.description}
Request Schema: ${JSON.stringify(endpoint.requestSchema, null, 2)}
Constraints: ${endpoint.constraints.join(', ')}
Generate 15-20 edge cases including:
- Boundary values (min, max, zero, negative)
- Unicode and special characters in string fields
- Extremely long inputs
- SQL injection attempts
- XSS payloads
- Empty/null/undefined values for required fields
- Type coercion attacks (sending string where number expected)
- Array overflow (if arrays are accepted)
- Nested object depth attacks
- Valid but semantically wrong data
Return JSON array: [{ "name": "test name", "input": { ... }, "expectedBehavior": "should return 400|200|etc and why" }]`,
},
],
});
return JSON.parse(
response.content[0].type === 'text' ? response.content[0].text : '[]'
);
}This found real bugs. In the first run against a client's API, it generated a test case where the email field contained test@example.com\n\nBcc: attacker@evil.com -- an email header injection attempt. The API accepted it and passed it to SendGrid. That is a security vulnerability that no one on our team had thought to test for.
Results from Phase 5
- Edge cases generated: 200+ per API endpoint (we had 15 endpoints = 3,000+ test cases)
- Real bugs found: 14 bugs, including 3 security vulnerabilities
- False positives: ~20% (edge cases that triggered expected error handling)
- Time to generate: 5 minutes vs. 2-3 days of manual edge case brainstorming
The Full Picture: Before vs After
After 6 months of incremental AI automation, here are the numbers:
| Metric | Before (Oct 2025) | After (Apr 2026) | Change |
|---|---|---|---|
| Full regression test time | 2.5 days | 4 hours | -87% |
| CI pipeline duration | 45 min | 12 min | -73% |
| Test suite flakiness rate | ~30% | ~5% | -83% |
| Test coverage (meaningful) | ~40% | ~78% | +95% |
| Bugs caught pre-release | ~55% | ~89% | +62% |
| QA engineer hours per sprint | 60+ hours | 18 hours | -70% |
| Bugs reaching production per month | 12-15 | 3-4 | -73% |
| Monthly tooling cost | $0 (just engineer time) | ~$450 | -- |
| Monthly QA engineer cost saved | -- | ~$3,200 (42 hours at ~$76/hr) | -- |
The math works: $450/month in tools saves $3,200/month in engineer time. And the quality improvements are not even captured in that ROI calculation.
What Still Needs Humans
I want to be brutally honest here because the "AI replaces everything" narrative is dangerous. Here is what we still cannot automate:
Exploratory Testing
AI tests what you tell it to test. It follows scripts -- even if it wrote those scripts itself. What it cannot do is poke around a page and notice "huh, this dropdown feels weird when I scroll fast" or "why does this button flash when I hover near it?"
Our best bug reports still come from a human spending 20 minutes just using the product like a real user would. No script. No checklist. Just trying to break things creatively.
UX Evaluation
AI can tell you that a button exists and is clickable. It cannot tell you that the button is confusing, that the label is ambiguous, or that users will not understand what it does. UX quality is subjective and contextual in ways that AI does not handle well.
Security Testing
AI can generate known attack patterns (SQL injection, XSS, header injection). It cannot discover novel attack vectors specific to your application's business logic. For security-critical applications, a human pentester is still irreplaceable.
Domain-Specific Edge Cases
For a healthcare client, we had a bug where patient dosage calculations were wrong for patients weighing exactly 100kg because of a rounding boundary in the BMI calculation. No AI would generate that test case without deep understanding of the medical domain and the specific formula being used.
The "Feels Right" Test
Sometimes a feature works exactly as specified but feels wrong. The animation is too slow. The loading state is jarring. The error message is technically correct but emotionally cold. This kind of polish requires human judgment.
The False Positive Problem
I mentioned this briefly, but it deserves its own section because it almost made us abandon AI testing entirely.
In month 2, our AI-generated visual regression tests started flagging 15-20 "regressions" per run. Our team would spend 30 minutes reviewing screenshots, only to find that 12 of the 15 flags were false positives -- dynamic content changes, cookie banners appearing inconsistently, third-party widget layout shifts.
The fix was a combination of:
- Ignore regions for dynamic content (timestamps, ad slots, cookie banners)
- Tolerance thresholds -- ignore pixel differences below 0.1%
- Smart baselines -- update baselines automatically when intentional design changes are merged
- Weekly tuning -- spend 15 minutes per week reviewing false positives and adding ignore rules
After 3 months of tuning, our false positive rate dropped from ~25% to ~4%. But that initial 3-month tuning period was painful.
The Cost Breakdown
Here is exactly what we spend monthly on AI-powered QA:
| Tool/Service | Monthly Cost | What It Does |
|---|---|---|
| Applitools Eyes | $399 | Visual regression comparison |
| Claude API (Sonnet) | $25-40 | Test generation, bug triage, a11y interpretation, edge cases |
| Claude API (Vision) | $10-15 | Visual diff analysis |
| GitHub Actions | $0 (free tier) | CI pipeline |
| Playwright | $0 (open source) | Test runner and screenshot capture |
| Total | $434-454 |
Compare that to the alternative: a full-time QA engineer in India costs $2,000-4,000/month. In the US, $6,000-10,000/month. We are not replacing the QA role entirely -- our QA engineers now focus on exploratory testing, UX evaluation, and test strategy instead of clicking through checklists.
Our Current Stack
For anyone who wants to replicate this, here is the exact toolchain:
Test Runner: Playwright (for e2e and visual tests)
Visual Regression: Applitools Eyes + Claude Vision API
Test Generation: Claude Sonnet API + custom validation
Bug Triage: Claude Sonnet API + Slack/Jira webhooks
A11y Auditing: axe-core + Claude Sonnet API
API Edge Cases: Claude Sonnet API + custom runner
CI/CD: GitHub Actions
Reporting: Custom dashboard (Next.js + Supabase)Lessons Learned
1. Start with visual regression. It has the highest ROI, the most measurable results, and the lowest risk. If AI visual testing does not work for your project, nothing else will either.
2. Do not trust AI-generated tests blindly. Every generated test needs human review. The "theater test" problem is real and insidious because the tests pass, so you think you are covered when you are not.
3. Budget 3 months for false positive tuning. Your first month of AI testing will generate so many false positives that your team will want to turn it off. Push through it. The tuning period is worth the investment.
4. Keep humans for strategy, use AI for execution. Humans should decide what to test and why. AI should handle the how and the scale. This division works.
5. Measure everything. You need before-and-after numbers to justify the investment. Track: test count, coverage percentage, bugs in production, CI time, QA engineer hours. Without numbers, you are just guessing.
6. Phase it in. Do not try to replace your entire QA pipeline at once. One phase per month. Each phase earns trust with the team and proves the approach works.
7. AI catches different bugs than humans. AI is great at finding regression, consistency issues, and known attack patterns. Humans are great at finding usability problems, novel bugs, and "this just feels wrong" issues. You need both.
What We Are Working On Next
Our QA pipeline is not done evolving. Here is what is coming in Q3 2026:
- AI-powered test prioritization: Use production error logs and user analytics to decide which tests to run first (test the things that break most often)
- Self-healing tests: When a selector changes, use AI to find the new selector instead of failing the test
- Natural language test authoring: Let product managers write tests in plain English and generate Playwright code automatically
- Production monitoring integration: Connect AI bug triage to production error monitoring for automatic regression detection
Building a product and want to ship faster without sacrificing quality? At CODERCOPS, we set up AI-powered QA pipelines for web applications, SaaS products, and mobile apps. Get in touch and we will audit your current testing setup for free.
Comments