7.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Consistency
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Cost per result
0.443Shows the average cost per correct benchmark answer in cents (lower is better).…
0.000Shows the average cost per correct benchmark answer in cents (lower is better).…
Total Cost
$0.049Total Cost…
$0.000Total Cost…
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 4Did not follow instructions: 1Response Time (avg)3.83sResponse Time (max)14.93sResponse Time (total)61.25sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Did not follow instructions: 3Wrong answer: 3Response Time (avg)29.10sResponse Time (max)170.45sResponse Time (total)290.96sA test is fully passed only if every run passed for that test.…
Attempt pass rate
68.8%Attempt pass rate = passed attempts / total attempts across runs.…
68.8%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
48 (16 x 3)Total Runs…
48 (16 x 3)Total Runs…
Output Tokens
1,731Output Tokens…
71,452Output Tokens…
Reasoning Tokens
25,821Reasoning Tokens…
155,147Reasoning Tokens…
Response Time (avg)
3.83sResponse Time (avg)…
29.10sResponse Time (avg)…
Response Time (max)
14.93sResponse Time (max)…
170.45sResponse Time (max)…
Response Time (total)
61.25sResponse Time (total)…
290.96sResponse Time (total)…
Top Models by Score
Score vs Total Cost
Response Time (avg)
Avg Score vs Response Time (avg)
Category Breakdown
Anti-AI Tricks
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
9.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)2.53sResponse Time (max)3.89sResponse Time (total)7.58sA test is fully passed only if every run passed for that test.…
2.53sResponse Time (avg)…
564Output Tokens…
3,780Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)18.54sResponse Time (max)32.30sResponse Time (total)37.07sA test is fully passed only if every run passed for that test.…
18.54sResponse Time (avg)…
13,924Output Tokens…
17,208Reasoning Tokens…
Combined
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)14.93sResponse Time (max)14.93sResponse Time (total)14.93sA test is fully passed only if every run passed for that test.…
14.93sResponse Time (avg)…
327Output Tokens…
7,347Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)29.57sResponse Time (max)29.57sResponse Time (total)29.57sA test is fully passed only if every run passed for that test.…
29.57sResponse Time (avg)…
1,176Output Tokens…
12,984Reasoning Tokens…
Data parsing and extraction
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.29sResponse Time (max)2.31sResponse Time (total)4.59sA test is fully passed only if every run passed for that test.…
2.29sResponse Time (avg)…
279Output Tokens…
2,952Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)15.01sResponse Time (max)15.01sResponse Time (total)15.01sA test is fully passed only if every run passed for that test.…
15.01sResponse Time (avg)…
600Output Tokens…
13,886Reasoning Tokens…
Domain specific
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Response Time (avg)4.21sResponse Time (max)5.86sResponse Time (total)12.62sA test is fully passed only if every run passed for that test.…
4.21sResponse Time (avg)…
18Output Tokens…
5,325Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.4%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)170.45sResponse Time (max)170.45sResponse Time (total)170.45sA test is fully passed only if every run passed for that test.…
170.45sResponse Time (avg)…
45,350Output Tokens…
90,436Reasoning Tokens…
General Intelligence
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.16sResponse Time (max)3.16sResponse Time (total)3.16sA test is fully passed only if every run passed for that test.…
3.16sResponse Time (avg)…
96Output Tokens…
1,488Reasoning Tokens…
StepFun: Step 3.5 Flash
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)6.54sResponse Time (max)6.54sResponse Time (total)6.54sA test is fully passed only if every run passed for that test.…
6.54sResponse Time (avg)…
2,214Output Tokens…
2,584Reasoning Tokens…
Instructions following
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)1.91sResponse Time (max)1.93sResponse Time (total)3.82sA test is fully passed only if every run passed for that test.…
1.91sResponse Time (avg)…
72Output Tokens…
2,121Reasoning Tokens…
StepFun: Step 3.5 Flash
9.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
6.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
83.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)4.98sResponse Time (max)4.98sResponse Time (total)4.98sA test is fully passed only if every run passed for that test.…
4.98sResponse Time (avg)…
2,284Output Tokens…
3,412Reasoning Tokens…
Puzzle Solving
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
7.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.58sResponse Time (max)4.41sResponse Time (total)10.75sA test is fully passed only if every run passed for that test.…
3.58sResponse Time (avg)…
141Output Tokens…
1,896Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)7.72sResponse Time (max)10.60sResponse Time (total)15.44sA test is fully passed only if every run passed for that test.…
7.72sResponse Time (avg)…
5,629Output Tokens…
10,835Reasoning Tokens…
Tool Calling
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
Google: Gemini 3.1 Flash Lite Preview
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.80sResponse Time (max)3.80sResponse Time (total)3.80sA test is fully passed only if every run passed for that test.…
3.80sResponse Time (avg)…
234Output Tokens…
912Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)11.91sResponse Time (max)11.91sResponse Time (total)11.91sA test is fully passed only if every run passed for that test.…