10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Consistency
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Cost per result
1.025Shows the average cost per correct benchmark answer in cents (lower is better).…
0.000Shows the average cost per correct benchmark answer in cents (lower is better).…
Total Cost
$0.164Total Cost…
$0.000Total Cost…
Tests Correct
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)12.36sResponse Time (max)50.16sResponse Time (total)111.21sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Did not follow instructions: 3Wrong answer: 3Response Time (avg)29.10sResponse Time (max)170.45sResponse Time (total)290.96sA test is fully passed only if every run passed for that test.…
Attempt pass rate
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
68.8%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
48 (16 x 3)Total Runs…
48 (16 x 3)Total Runs…
Output Tokens
1,634Output Tokens…
71,452Output Tokens…
Reasoning Tokens
47,907Reasoning Tokens…
155,147Reasoning Tokens…
Response Time (avg)
12.36sResponse Time (avg)…
29.10sResponse Time (avg)…
Response Time (max)
50.16sResponse Time (max)…
170.45sResponse Time (max)…
Response Time (total)
111.21sResponse Time (total)…
290.96sResponse Time (total)…
Top Models by Score
Score vs Total Cost
Response Time (avg)
Avg Score vs Response Time (avg)
Category Breakdown
Anti-AI Tricks
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)5.61sResponse Time (max)5.61sResponse Time (total)5.61sA test is fully passed only if every run passed for that test.…
5.61sResponse Time (avg)…
299Output Tokens…
3,127Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)18.54sResponse Time (max)32.30sResponse Time (total)37.07sA test is fully passed only if every run passed for that test.…
18.54sResponse Time (avg)…
13,924Output Tokens…
17,208Reasoning Tokens…
Combined
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)50.16sResponse Time (max)50.16sResponse Time (total)50.16sA test is fully passed only if every run passed for that test.…
50.16sResponse Time (avg)…
351Output Tokens…
12,645Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)29.57sResponse Time (max)29.57sResponse Time (total)29.57sA test is fully passed only if every run passed for that test.…
29.57sResponse Time (avg)…
1,176Output Tokens…
12,984Reasoning Tokens…
Data parsing and extraction
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.72sResponse Time (max)4.72sResponse Time (total)4.72sA test is fully passed only if every run passed for that test.…
4.72sResponse Time (avg)…
279Output Tokens…
5,333Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)15.01sResponse Time (max)15.01sResponse Time (total)15.01sA test is fully passed only if every run passed for that test.…
15.01sResponse Time (avg)…
600Output Tokens…
13,886Reasoning Tokens…
Domain specific
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)21.12sResponse Time (max)21.12sResponse Time (total)21.12sA test is fully passed only if every run passed for that test.…
21.12sResponse Time (avg)…
12Output Tokens…
14,908Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.4%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)170.45sResponse Time (max)170.45sResponse Time (total)170.45sA test is fully passed only if every run passed for that test.…
170.45sResponse Time (avg)…
45,350Output Tokens…
90,436Reasoning Tokens…
General Intelligence
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.09sResponse Time (max)4.09sResponse Time (total)4.09sA test is fully passed only if every run passed for that test.…
4.09sResponse Time (avg)…
111Output Tokens…
1,285Reasoning Tokens…
StepFun: Step 3.5 Flash
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)6.54sResponse Time (max)6.54sResponse Time (total)6.54sA test is fully passed only if every run passed for that test.…
6.54sResponse Time (avg)…
2,214Output Tokens…
2,584Reasoning Tokens…
Instructions following
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)6.10sResponse Time (max)6.10sResponse Time (total)6.10sA test is fully passed only if every run passed for that test.…
6.10sResponse Time (avg)…
72Output Tokens…
4,558Reasoning Tokens…
StepFun: Step 3.5 Flash
9.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
6.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
83.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)4.98sResponse Time (max)4.98sResponse Time (total)4.98sA test is fully passed only if every run passed for that test.…
4.98sResponse Time (avg)…
2,284Output Tokens…
3,412Reasoning Tokens…
Puzzle Solving
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.43sResponse Time (max)4.68sResponse Time (total)8.85sA test is fully passed only if every run passed for that test.…
4.43sResponse Time (avg)…
276Output Tokens…
4,921Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)7.72sResponse Time (max)10.60sResponse Time (total)15.44sA test is fully passed only if every run passed for that test.…
7.72sResponse Time (avg)…
5,629Output Tokens…
10,835Reasoning Tokens…
Tool Calling
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3 Flash Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)10.55sResponse Time (max)10.55sResponse Time (total)10.55sA test is fully passed only if every run passed for that test.…
10.55sResponse Time (avg)…
234Output Tokens…
1,130Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)11.91sResponse Time (max)11.91sResponse Time (total)11.91sA test is fully passed only if every run passed for that test.…