7.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Rank
#20
#24
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 4Did not follow instructions: 1Response Time (avg)1.37sResponse Time (max)3.39sResponse Time (total)20.53sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Timed out: 3API error: 1Wrong answer: 1Response Time (avg)72.86sResponse Time (max)234.29sResponse Time (total)1092.84sA test is fully passed only if every run passed for that test.…
Consistency
9.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
7.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Cost per result
0.142Shows the average cost per correct benchmark answer in cents (lower is better).…
0.565Shows the average cost per correct benchmark answer in cents (lower is better).…
Total Cost
$0.015Total Cost…
$0.057Total Cost…
Attempt pass rate
71.1%Attempt pass rate = passed attempts / total attempts across runs.…
82.2%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
4Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
common.totalRuns
45 (15 x 3)common.totalRuns…
45 (15 x 3)common.totalRuns…
Output Tokens
4,646Output Tokens…
1,708Output Tokens…
Reasoning Tokens
0Reasoning Tokens…
131,466Reasoning Tokens…
Response Time (avg)
1.37sResponse Time (avg)…
72.86sResponse Time (avg)…
Response Time (max)
3.39sResponse Time (max)…
234.29sResponse Time (max)…
Response Time (total)
20.53sResponse Time (total)…
1092.84sResponse Time (total)…
Top Models by Score
Score vs Total Cost
Response Time (avg)
Avg Score vs Response Time (avg)
Category Breakdown
Anti-AI Tricks
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
55.6%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)1.16sResponse Time (max)1.47sResponse Time (total)3.49sA test is fully passed only if every run passed for that test.…
1.16sResponse Time (avg)…
1,086Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)71.35sResponse Time (max)168.31sResponse Time (total)214.06sA test is fully passed only if every run passed for that test.…
71.35sResponse Time (avg)…
363Output Tokens…
23,645Reasoning Tokens…
Combined
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.20sResponse Time (max)3.20sResponse Time (total)3.20sA test is fully passed only if every run passed for that test.…
3.20sResponse Time (avg)…
339Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)17.78sResponse Time (max)17.78sResponse Time (total)17.78sA test is fully passed only if every run passed for that test.…
17.78sResponse Time (avg)…
483Output Tokens…
8,270Reasoning Tokens…
Data parsing and extraction
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)1.22sResponse Time (max)1.33sResponse Time (total)2.44sA test is fully passed only if every run passed for that test.…
1.22sResponse Time (avg)…
399Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
5.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
5.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
83.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.API error: 1Response Time (avg)56.99sResponse Time (max)80.14sResponse Time (total)113.98sA test is fully passed only if every run passed for that test.…
56.99sResponse Time (avg)…
235Output Tokens…
16,237Reasoning Tokens…
Domain specific
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)942msResponse Time (max)1.12sResponse Time (total)2.83sA test is fully passed only if every run passed for that test.…
942msResponse Time (avg)…
568Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.4%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Timed out: 1Wrong answer: 1Response Time (avg)146.50sResponse Time (max)234.29sResponse Time (total)439.49sA test is fully passed only if every run passed for that test.…
146.50sResponse Time (avg)…
58Output Tokens…
43,615Reasoning Tokens…
Instructions following
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)1.13sResponse Time (max)1.14sResponse Time (total)2.27sA test is fully passed only if every run passed for that test.…
1.13sResponse Time (avg)…
574Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)63.49sResponse Time (max)111.61sResponse Time (total)126.98sA test is fully passed only if every run passed for that test.…
63.49sResponse Time (avg)…
98Output Tokens…
14,139Reasoning Tokens…
Puzzle Solving
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)972msResponse Time (max)1.13sResponse Time (total)2.92sA test is fully passed only if every run passed for that test.…
972msResponse Time (avg)…
898Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
4.4Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
77.8%Attempt pass rate = passed attempts / total attempts across runs.…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Timed out: 2Response Time (avg)56.74sResponse Time (max)115.01sResponse Time (total)170.23sA test is fully passed only if every run passed for that test.…
56.74sResponse Time (avg)…
162Output Tokens…
24,276Reasoning Tokens…
Tool Calling
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.39sResponse Time (max)3.39sResponse Time (total)3.39sA test is fully passed only if every run passed for that test.…
3.39sResponse Time (avg)…
782Output Tokens…
0Reasoning Tokens…
Qwen: Qwen3.5-Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)10.33sResponse Time (max)10.33sResponse Time (total)10.33sA test is fully passed only if every run passed for that test.…