AI QA Bench: 1,396 Real Bugs, $2.67 a Bug and What Automated Testing Really Costs (2026)

The best AI bug hunter in the newest public QA benchmark missed four out of every five bugs it was sent to find, and it cost the most per run.

Quick answer: Momentic's QA Bench, published on 2 November 2026, dropped ten AI testing setups into four running open-source web apps and graded their bug reports against 1,396 labelled bugs. Mo, Momentic's own QA product, found the most, 269 per 60-minute pass (19.3% recall), at $2.67 a bug. OpenAI Codex with GPT-6.1 Sol found 86 at $0.80.

The bill is the more useful half of the story. QA Bench is the first public price list we've seen for automated bug hunting, from $7 to $719 a pass, and it shows the browser bridge and the time budget moving results about as much as the model did.

You're reading this on a directory's blog. SaaSCity is a gamified startup directory with a live city map and human editorial review, and we don't sell testing software. Our interest is selfish: products that launch with us get one week of attention, and a signup button that does nothing that week costs more than any test run.

Every figure below comes from a page we opened on 3 November 2026 and is linked where it appears. QA Bench numbers are from Momentic's write-up of 2 November 2026 (exports taken 31 October, labels pinned to commit e77c72b). Screenshots were captured in Chrome the same day.

Key takeaways

• QA Bench graded 10 configurations on 20 product areas of Gitea, Metabase, Twenty and Saleor, against 1,396 labelled bugs (Momentic, 2 November 2026).
• Mo matched 269.3 bugs per 60-minute pass, 19.3% recall, at $2.67 a bug and about $719 a pass: 3.1 times the bugs of the runner-up for 10.4 times the money, by our arithmetic.
• Codex with GPT-6.1 Sol on Playwright MCP was the most precise setup at 86.0%, finding 86.3 bugs per pass at $0.80 each.
• Playwright MCP found 10% to 58% more bugs than agent-browser for Codex and for Claude Code with Claude Opus 5.5, so the browser bridge mattered as much as the model.
• Doubling the clock lifted Mo's haul by 52% for 5% more per bug, while both GPT-6.1 Terra setups found fewer bugs at 60 minutes than at 30.
• Momentic built the winner, drafted the labels with three of the graded tools and graded the runs with an LLM judge, and its write-up lists each of those as a limitation.
• A US QA automation engineer averages $87,761 in base pay (Salary.com, 1 November 2026), about $125,400 loaded, or roughly 174 Mo passes a year.

What is QA Bench, and what did it measure?

QA Bench measures how many real bugs an AI QA tool finds in a running web app within a fixed time, and how many of the bugs it reports are real. Momentic, a Y Combinator W24 testing company in San Francisco, published it on 2 November 2026 in a write-up by CEO Wei-Wei Wu, and launched its Mo product on Hacker News the same day.

Each run gets a fresh E2B sandbox holding one product area of one app, with identical seeded data, no source code and 30 or 60 minutes on the clock. The tool explores the app in a browser and reports what it thinks is broken, and a grader matches those reports against the labelled bugs.

App	What it is	GitHub stars	Product areas	Labelled bugs
Gitea	Self-hosted Git service	58.8k	11	476
Metabase	Business intelligence	50.1k	4	464
Twenty	Open-source CRM	60.2k	4	374
Saleor	Headless e-commerce	23.5k	1	82
Total		192.6k	20	1,396

Source: Momentic, QA Bench write-up, 2 November 2026.

Ten configurations took part. Mo ran on its own. OpenAI Codex drove GPT-6.1 Sol and GPT-6.1 Terra, and Anthropic's Claude Code drove Claude Opus 5.5 and Claude Sonnet 5.5, each on two browser bridges: Microsoft's Playwright MCP and Vercel Labs' agent-browser. Devin by Cognition used its built-in browser. Codex and Claude Code ran at medium effort on hosted Browserbase browsers, while Mo ran at medium effort for 30 minutes and high effort for 60. Each configuration ran every area three times per budget, except Devin, which ran once.

Bugs carry severity weights of 1 (cosmetic), 3 (functional) or 5 (broken). In Twenty, a Save button that keeps spinning after a rejected role save is a 1, and a settings page that crashes when you sort the AI Skills list by Type is a 5. In Saleor, a voucher rename that sticks after a failed save is a 3. Recall is the share of labelled bugs a tool matched. Precision is the share of its graded claims that were real, with duplicates left out. The score adds the weights of matched bugs and subtracts a point for each invalid claim and each duplicate. GPT-6.1 Sol, at high reasoning effort, judges the matches, and anything it can't match goes to a human review queue.

Every cost figure below depends on what a "pass" means, which is one run through all 20 areas. In Momentic's words, "in a pass through all 20 areas, Mo matches about 269 of the 1,396 labeled bugs." A $719 Mo pass therefore buys 20 sandboxes, each with its own 60-minute clock.

The labels are public. The Hugging Face dataset has 1,917 rows under the MIT licence: the bugs, 501 known-invalid claims and the 20 area definitions.

How many bugs do AI QA tools actually find?

In one 60-minute pass, the best tool in QA Bench found 19.3% of the labelled bugs, and the median configuration found about 4%.

Configuration	Matched bugs, 30 min	Matched bugs, 60 min	Recall, 60 min	Precision, 60 min	Score, 60 min
Mo (Momentic)	176.7	269.3	19.3%	84.2%	588.7
Codex, GPT-6.1 Sol, Playwright MCP	74.7	86.3	6.2%	86.0%	207.0
Codex, GPT-6.1 Terra, Playwright MCP	83.3	82.7	5.9%	81.6%	177.0
Codex, GPT-6.1 Sol, agent-browser	68.0	64.3	4.6%	76.3%	144.3
Claude Code, Opus 5.5, Playwright MCP	38.0	59.0	4.2%	79.5%	122.7
Codex, GPT-6.1 Terra, agent-browser	59.3	57.7	4.1%	76.2%	126.3
Claude Code, Opus 5.5, agent-browser	24.0	41.0	2.9%	77.0%	81.0
Devin (Cognition)	16.0	27.0	1.9%	84.4%	65.0
Claude Code, Sonnet 5.5, Playwright MCP	12.0	15.0	1.1%	72.6%	26.7
Claude Code, Sonnet 5.5, agent-browser	8.7	13.0	0.9%	65.0%	20.0

Source: Momentic QA Bench, published 2 November 2026, exports taken 31 October 2026. Per-pass averages over three runs per area (one for Devin).

Mo had the best three-run average in 19 of the 20 areas. It lost only Gitea notifications, a 16-bug area where Claude Code with Opus 5.5 on Playwright MCP averaged 3.0 matches to Mo's 2.3. Its recall varied by app: 26% in Gitea, 23% in Saleor, 22% in Twenty and 9% in Metabase, which holds a third of all the labels.

On precision, Codex with GPT-6.1 Sol on Playwright MCP led at 86.0%, ahead of Devin (84.4%) and Mo (84.2%). Per 60-minute pass, Mo produced 50.7 known-invalid claims and 15.3 duplicates, against 14.0 invalid claims and no duplicates for Codex Sol. Someone on your team reads every one of those.

The severity data argues against the idea that Mo pads its count with trivial bugs. Matched bugs averaged a weight between 2.3 and 2.6 for every configuration except Sonnet, with Mo at 2.4 and Codex at 2.6.

For a founder, 19.3% recall works out like this: if your billing flow holds 50 real bugs, the best tool here finds about ten of them in a pass. A green report means the pass found nothing, and the app behind it can still be full of bugs.

What does AI QA cost per bug?

AI QA cost between $0.27 and $4.51 per matched bug in QA Bench, and between $7 and $719 per pass, depending on the model, the browser bridge and the time budget.

Configuration	Cost per bug, 30 min	Cost per bug, 60 min	Cost per pass, 30 min	Cost per pass, 60 min
Mo (Momentic)	$2.54	$2.67	$448	$719
Codex, GPT-6.1 Sol, Playwright MCP	$0.79	$0.80	$59	$69
Codex, GPT-6.1 Terra, Playwright MCP	$0.56	$0.53	$47	$44
Codex, GPT-6.1 Sol, agent-browser	$0.83	$0.96	$57	$62
Claude Code, Opus 5.5, Playwright MCP	$2.86	$4.09	$109	$241
Codex, GPT-6.1 Terra, agent-browser	$0.29	$0.27	$17	$16
Claude Code, Opus 5.5, agent-browser	$1.73	$2.45	$42	$101
Devin (Cognition)	$4.51	$3.89	$72	$105
Claude Code, Sonnet 5.5, Playwright MCP	$1.66	$2.08	$20	$31
Claude Code, Sonnet 5.5, agent-browser	$0.77	$0.68	$7	$9

Source: Momentic QA Bench, 2 November 2026. Devin's cost is its on-demand session billing. The repo notes that E2B sandbox and grading costs are left out of reported cost.

Mo costs 3.3 times as much per bug as Codex with GPT-6.1 Sol on Playwright MCP, and 10.4 times as much per pass. Put per area-hour, a 60-minute Mo pass costs about $36 and the Codex Sol pass about $3.45, by our arithmetic from the table. Hard areas cost everyone more: Mo paid $1.94 a bug in Saleor vouchers and $10.69 in Metabase querying, while Codex Sol paid $0.27 in the Gitea wiki and $1.75 in Metabase querying.

The browser barely registers. Browserbase's pricing page lists $0.12 per browser hour after the 100 hours in its $20 Developer plan (checked 3 November 2026), so twenty browser-hours cost about $2.40 against a $69 Codex pass, and almost all of the bill is the model at work.

Is AI QA cheaper than hiring a QA engineer?

Per hour it can be, though the two do different jobs and QA Bench measured only one of them.

Salary.com put the average US QA automation engineer's base salary at $87,761 on 1 November 2026, with the middle half between $79,870 and $94,857. PayScale gives $85,578 for a QA or test automation engineer (141 profiles, updated 29 September 2026). The US Bureau of Labor Statistics reported on 9 September 2026 that wages were 70.0% of private-industry compensation costs in June 2026, with benefits making up the rest.

Option	Cost	What it buys	Source
QA automation engineer, base pay	$87,761 a year	One person with product context and judgement	Salary.com, 1 Nov 2026
Same engineer, loaded	About $125,400 a year, about $60 per paid hour	Base pay at a 70% wage share, over 2,080 hours	Our arithmetic from BLS, June 2026
Mo, one 60-minute pass of 20 areas	$719	269.3 matched bugs, each with a video and report	QA Bench, 2 Nov 2026
Mo, weekly for a year	$37,388	52 passes	Our arithmetic
Claude Code Opus 5.5 + Playwright MCP, weekly for a year	$12,532	52 passes of about 59 bugs	Our arithmetic
Codex GPT-6.1 Sol + Playwright MCP, weekly for a year	$3,588	52 passes of about 86 bugs	Our arithmetic

One Mo pass costs about as much as 12 loaded hours of a QA engineer, and a loaded year of that engineer would pay for about 174 Mo passes or 1,800 Codex passes. The engineer also writes the regression suite, triages reports and knows which bugs customers actually hit. Your app isn't 20 areas of four open-source projects, so your cost per pass will differ. Nobody ran a human through QA Bench either, and one Hacker News commenter, adwad, wrote that it "would be even more valuable with a human benchmark."

What Momentic charges

Momentic bills Mo and its test platform separately. Mo sessions, per the Mo page, bill "active agent runtime, hosted runtime, and inference" against a separate Mo balance, and the qa cost command shows what a session cost. The founders' launch post says every new signup gets $250 of Mo spend, which is about seven area-hours at QA Bench's average Mo rate.

The test platform runs on credits. The pricing page, checked on 3 November 2026, lists Free with 2,000 credits a month, Pay-as-you-go at $125 a month with 10,000 credits, and custom Enterprise pricing, with no seat fees. A normal step costs 1 credit, an AI-generated step 2 and a hosted browser 1 a minute. Overage is $0.01875 a credit, while a 10,000-credit top-up costs $125, or $0.0125 a credit, so buying blocks comes out a third cheaper than running over.

Why do Codex and Claude Code trail a purpose-built harness?

Codex and Claude Code are general coding tools pointed at a browser, and in QA Bench the setup around the model (the browser bridge, the time budget, whether anything plans and splits up the exploration) moved results about as much as the model did.

Mo does far more work per minute. The Launch HN post says it builds a directed acyclic graph of the app as it explores, forks large areas into parallel sub-agents with their own browsers, and uses separate locator agents to work out what to click. The Mo page adds that another agent tries to reproduce each functional bug before it gets reported, and the post says a single bug bash "can spin up 100s of concurrent browsers and devices." Momentic's own limitations section concedes that the time budget "doesn't cap spend or compute," so sixty minutes of Mo and sixty minutes of Codex are different amounts of work.

The founders also argue that coding models check their own work badly, writing that LLMs "reinforce their own assumptions" and "write a ton of useless tests." Their post cites a DeepSWE v1.1 result to back it, which we haven't checked.

The browser bridge was the other big lever. Playwright MCP is Microsoft's Model Context Protocol server, which gives the model accessibility-tree snapshots and needs no vision model. agent-browser is Vercel Labs' "Browser automation CLI for AI agents," which returns a text snapshot with references like @e1. With the same models on the same apps, Playwright MCP found 10% to 58% more bugs for Codex and for Claude Code with Opus 5.5, and Sonnet found 38% more at 30 minutes and 15% more at 60. agent-browser was cheaper per bug for Terra, Sonnet and Opus, 5% to 20% dearer for Sol, and gave Codex Sol 1.4 to 1.8 times as many known-invalid claims. Its runs also had open web access through the shell, and the README says --prevent-cheating "is not a network firewall for agent-browser runs."

Time split the tools most sharply:

• Mo found 52% more bugs at 60 minutes than at 30, paid 5% more per bug and improved in 19 of 20 areas.
• Claude Code with Opus 5.5 found 55% more on Playwright MCP and 71% more on agent-browser, with cost per bug up 43% and 42%.
• Codex with GPT-6.1 Sol on Playwright MCP gained 16% for a 1% rise in cost per bug.
• Both GPT-6.1 Terra setups found fewer bugs at 60 minutes than at 30.
• Claude Sonnet 5.5 "stops exploring early," per the write-up, and matched 8.7 to 15 bugs a pass.

For a Codex user, the cheapest improvement is a config change: moving from agent-browser to Playwright MCP was worth up to 58% more bugs here. We've covered newer models getting worse at following tool schemas, and QA Bench shows the same problem from the testing side.

Should you trust a benchmark run by the company that won?

Trust it as a careful vendor report: the method is published, the labels and harness are MIT-licensed, and Momentic lists its own conflicts. We found no independent rerun as of 3 November 2026.

Momentic has a lot riding on Mo. It announced a $3.7 million seed round on 4 March 2025 from FundersClub, General Catalyst, Y Combinator and AI Grant, with angels including Box's Aaron Levie. TechCrunch reported a $15 million Series A on 24 November 2025, led by Standard Capital with Dropbox Ventures, and named Notion, Xero, Bilt, Webflow and Retool as customers. A Microsoft customer story puts the total raised at $19.2 million. When Mo launched on 28 September 2026, Let's Data Science noted that "those performance figures are vendor claims, rather than independently verified benchmarks." QA Bench reads like Momentic's reply to that line.

The write-up names six limitations of its own:

• Momentic built Mo and also designed, ran and graded the benchmark.
• Candidate bugs came from bug bashes by Mo, Claude Code and Codex before reproduction and human review, so the labels can share those tools' blind spots. The results page and launch post call them "human-labeled," though human-reviewed is closer.
• A GPT-6.1 Sol judge does the matching, and its mistakes can move scores either way. The README warns that regrading "can change a few matches."
• The corpus is uneven. Eleven areas come from Gitea and one from Saleor, and Metabase holds a third of the bugs in four areas.
• Only three-run means are reported, with no error bars, so a gap of a bug or two in an area may be noise.
• Mo had to record a reproduction video and write a report for every bug while the others only named bugs, and Mo ran at high effort with far more spend.

We'd add three. Codex had web search switched on. Devin got one run per budget instead of three, so its numbers are the shakiest in the table, and they say nothing about its coding work, which we covered in our look at SWE-2 on Devin's $20 plan. And results age quickly. We've tracked how a model's behaviour can drift from week to week, and the repo already carries a commit titled "Publish the v2 rerun results."

The benchmark also prints the numbers that make Mo look worse, including the area Mo lost and the precision lead Codex holds. The founders even put the cost gap near the top of their launch post: "Mo costs $2.67 per bug vs $0.80 for Codex." Anyone with an E2B account and API keys can rerun it from the GitHub repo with commands such as pnpm eval --agent codex-playwright-mcp --minutes 30.

What did Hacker News readers object to?

Hacker News readers pushed on four points: paying again for tokens, Momentic's moat, whether Codex is the better deal, and whether a homemade harness does the same job. The thread had 143 points and about 95 comments when we captured it on 3 November 2026.

On bring-your-own-agent, aliasxneo wrote that "BYOA should be a default." Co-founder Jeff An, posting as jeffan, said the team "wrestled a lot" with it and that "there was no configuration with off the shelf models that we were happy with performance-wise." When adwn asked what Momentic will sell "when Opus 6.0 swallows Mo's edge in 5 months from now," Jeff An replied: "To be honest I don't know if there is or will be any kind of moat in SaaS."

Smoofer asked, "Is the takeaway from that not just that Codex is better for cheaper?" Momentic engineer Henry (momentichenry) answered with the recall case: "the worst thing you can do in QA is miss a real bug and ship broken code to prod." On building your own, Fordec wrote, "I just built mine in two months," and Jeff An agreed for small projects that "a single sub-agent that focuses on vibe testing is enough."

StilesCrisis argued that "finding 1000 miniscule bugs isn't helpful," which the severity averages above partly answer. daniiel went further, writing that "a silent pass is worse than a flaky test," and suggested letting the tool answer "I can't tell." Henry replied that "if Mo can't reproduce a suspected bug it gets discarded." That rule protects precision, and it also means a real but intermittent bug may never reach your report.

How to run your own bug-set eval before you pay

Run your own eval by seeding 30 to 50 known bugs into a staging copy of your app, giving every tool the same brief and the same clock, and scoring the reports with QA Bench's formulas. No public benchmark covers your app, so this is the one that counts.

1. Freeze one build and seed its data. Deploy a release candidate to a staging URL with test accounts and seeded data that resets between runs. QA Bench gave every run a fresh sandbox with identical data, and your eval needs the same fairness or the scores mean nothing.
2. Pick three to five product areas that make you money. Signup, billing, the core workflow, settings and imports are typical. QA Bench scored 20 areas separately, and per-area scores show you where a tool is blind instead of averaging the blindness away.
3. Build a labelled set of 30 to 50 known bugs. Revert a few old fixes on a branch, add the open bugs you already know about, and write each one up with a name, expected and actual behaviour, repro steps and a severity of 1, 3 or 5. That is the same shape as QA Bench's published labels.
4. Write one brief and hide the answers. Every tool gets the same prompt, the same URL, no source code and no access to your bug list. If a tool can browse the web or read your repo, write that down, since QA Bench flagged exactly that for its Codex and agent-browser runs.
5. Run each tool three times at 30 and 60 minutes. One run is an anecdote. Three runs at two budgets show whether more time buys more bugs or just a bigger invoice, which was the sharpest split between tools in QA Bench.
6. Grade every claim yourself. Mark each reported bug as matched, invalid, duplicate or new. A new bug that reproduces is a real find, so add it to your labelled set and the next eval gets harder.
7. Compute recall, precision, score and cost per matched bug. Recall is matched bugs divided by labelled bugs. Precision is matches divided by matches plus invalid claims. Score is the sum of matched severity weights minus invalid claims and duplicates. Cost per matched bug is total spend divided by matches.
8. Decide which failure costs you more, then rerun before renewals. If a missed checkout bug loses customers, weight recall. If your two engineers lose a morning to bogus reports, weight precision. Models and harnesses change every few weeks, so rerun the same set before every renewal.

The QA Bench repo's fixture layout makes a decent template if you'd rather not design your own. Each area folder holds a fixture.json with scope and entry points, a ground-truth.json of confirmed bugs and a known-invalid.json of rejected claims. The README doesn't explain how to add a new tool, so expect to read the code under src/agents/.

What should a bootstrapped SaaS team do this quarter?

Keep a small deterministic test suite for the flows that make money, run one cheap exploratory pass before each release, and pay for a high-recall pass before launches and big rewrites.

A tool that finds 19% of bugs per pass can't stand in for a regression suite. If you don't have one, Playwright ships three test agents (a planner, a generator and a healer), set up with npx playwright init-agents for Claude Code, Codex, VS Code or OpenCode. They turn a plain-English plan into test files you own and run in CI.

The cheap exploratory pass really is cheap. Codex with GPT-6.1 Sol on Playwright MCP averaged about $3.45 per area-hour in QA Bench, so an hour on five areas before each release comes to under $20. Before a launch, buy recall: one high-recall pass, triaged by a person, with its findings added to your labelled set. If you'd rather buy coverage than run tools, managed services exist too. QA Wolf, for one, charges per test per month, with creation, infrastructure, triage and maintenance included.

None of these tools is a security review, so run a pre-launch security checklist as well. And count the bugs customers report each month, because that number shows whether your QA spend is working.

Launches have the same recall problem. One channel reaches a slice of the people who'd want your product, the way one pass finds a slice of the bugs, so you run several. A free SaaSCity listing gets a page and a building on our live city map after human editorial review. Add the SaaSCity badge to your site and you get a dofollow backlink from a domain rated DR 70 at our latest Ahrefs refresh (November 3, 2026), plus a Monday launch slot. Quick Pass ($19.99) goes live within 24 hours, Premium ($39.99) adds a written launch post with three dofollow links, and Max ($99.99) adds 30 days of billboard, ticker and homepage pin. For QA and dev tools, a listing is one of the cheapest ways to reach the teams who read benchmarks like this one.

The best tool in QA Bench finds about one bug in five per pass, and its maker printed that number itself. So how many passes is your launch worth? Run the 50-bug eval and find out before the first invoice.

FAQ

What is Momentic's QA Bench?

QA Bench is a benchmark Momentic published on 2 November 2026 to measure how many real bugs an AI QA tool finds in a running web app within 30 or 60 minutes. It covers 20 product areas of four open-source apps (Gitea, Metabase, Twenty and Saleor) with 1,396 labelled bugs, and grades ten configurations, including Momentic's Mo, OpenAI Codex, Claude Code and Devin. The labels, harness and results are public under the MIT licence on GitHub and Hugging Face, pinned to commit e77c72b.

How many bugs can AI find in a web app?

In QA Bench, published by Momentic on 2 November 2026, the best tool found 19.3% of 1,396 labelled bugs in one 60-minute pass: Momentic's Mo matched 269.3 on average. The runner-up, OpenAI Codex with GPT-6.1 Sol on Playwright MCP, found 86.3 (6.2%), Claude Code with Claude Opus 5.5 found 59.0 (4.2%) and Devin found 27.0 (1.9%). Every configuration missed most of the labelled bugs in a single pass.

How much does AI QA cost per bug?

QA Bench's 60-minute results, published 2 November 2026, put the cost per matched bug between $0.27 and $4.09. Codex with GPT-6.1 Terra on agent-browser was cheapest at $0.27, Codex with GPT-6.1 Sol on Playwright MCP cost $0.80, Momentic's Mo $2.67, Devin $3.89 and Claude Code with Opus 5.5 on Playwright MCP $4.09. A full pass over 20 product areas cost from $9 (Claude Code with Sonnet 5.5 on agent-browser) to about $719 (Mo).

Is Codex or Claude Code better for AI browser testing?

In Momentic's QA Bench (2 November 2026), OpenAI Codex beat Claude Code on bugs found and on cost. Codex with GPT-6.1 Sol on Playwright MCP matched 86.3 bugs per 60-minute pass at $0.80 each and 86.0% precision. Claude Code with Claude Opus 5.5 on the same browser bridge matched 59.0 at $4.09 each, and Claude Sonnet 5.5 matched 15.0. Both trailed Momentic's purpose-built Mo, which matched 269.3.

Is Mo worth $2.67 a bug when Codex costs $0.80?

It depends on what a missed bug costs you. In QA Bench (2 November 2026), Mo found 3.1 times as many labelled bugs as Codex with GPT-6.1 Sol per 60-minute pass, for 10.4 times the money ($719 against $69 a pass). Codex was slightly more precise, at 86.0% against 84.2%. If a missed bug in checkout or signup loses customers, recall is worth paying for. If your team drowns in triage, the cheaper and more precise option wins.

Is AI QA cheaper than hiring a QA engineer?

Per hour of bug hunting it can be, but the two do different work. Salary.com put the average US QA automation engineer base salary at $87,761 on 1 November 2026, about $125,400 a year loaded if wages are 70% of compensation, as the US Bureau of Labor Statistics reported for June 2026. That equals about 174 Mo passes or about 1,800 Codex passes at QA Bench's 2 November 2026 prices. QA Bench has no human baseline to compare against.

Can you trust a benchmark run by Momentic?

Treat it as a careful vendor report. Momentic built Mo, drafted the labels from bug bashes by Mo, Claude Code and Codex, ran every test and graded them with a GPT-6.1 Sol judge, and lists all of that as limitations in its 2 November 2026 write-up. It reports no error bars, and Mo ran at higher effort with far more spend. The labels and harness are open under the MIT licence, so anyone can rerun it.

What is the difference between Playwright MCP and agent-browser?

Both let a coding model drive a real browser. Microsoft's Playwright MCP is a Model Context Protocol server that hands the model accessibility-tree snapshots of the page. Vercel Labs' agent-browser is a command-line tool that returns a text snapshot with element references such as @e1. In QA Bench (2 November 2026), Playwright MCP found 10% to 58% more bugs for Codex and Claude Code with Opus 5.5, while agent-browser was usually cheaper per bug.

How much does Momentic cost?

Momentic's pricing page, checked on 3 November 2026, lists a Free plan with 2,000 credits a month, Pay-as-you-go at $125 a month with 10,000 credits and overage at $0.01875 per credit, and custom Enterprise pricing. A normal test step costs 1 credit, an AI step 2 and a hosted browser 1 credit a minute, with no seat fees. Mo sessions bill runtime and inference against a separate Mo balance, and new signups got $250 of Mo spend at launch.

Who founded Momentic and how much has it raised?

Wei-Wei Wu (CEO) and Jeff An (CTO) founded Momentic in San Francisco in late 2023, and the company went through Y Combinator's Winter 2024 batch. It announced a $3.7 million seed round on 4 March 2025 from FundersClub, General Catalyst, Y Combinator and AI Grant, then a $15 million Series A on 24 November 2025 led by Standard Capital. A Microsoft customer story puts total funding at $19.2 million.

Does more testing time help AI QA tools find more bugs?

For some tools. In QA Bench (2 November 2026), doubling the budget from 30 to 60 minutes raised Mo's matched bugs by 52% while its cost per bug rose 5%. Claude Code with Opus 5.5 found 55% to 71% more bugs but paid 42% to 43% more per bug. Codex with GPT-6.1 Sol on Playwright MCP gained 16%, and both GPT-6.1 Terra setups found fewer bugs at 60 minutes than at 30.

How do you evaluate an AI QA tool before paying for it?

Seed 30 to 50 known bugs into a staging copy of your app, spread across the three to five flows that make you money. Give every tool the same written brief, no source code and the same 30- or 60-minute clock, and run each one at least three times. Grade every claim as matched, invalid, duplicate or new, then compute recall, precision and cost per matched bug with the formulas Momentic published for QA Bench on 2 November 2026.