7.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Consistency
9.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Cost per result
2.440Shows the average cost per correct benchmark answer in cents (lower is better).…
0.000Shows the average cost per correct benchmark answer in cents (lower is better).…
Total Cost
$0.269Total Cost…
$0.000Total Cost…
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 4Did not follow instructions: 1Response Time (avg)7.03sResponse Time (max)38.52sResponse Time (total)112.51sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Did not follow instructions: 3Wrong answer: 3Response Time (avg)29.10sResponse Time (max)170.45sResponse Time (total)290.96sA test is fully passed only if every run passed for that test.…
Attempt pass rate
75.0%Attempt pass rate = passed attempts / total attempts across runs.…
68.8%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
48 (16 x 3)Total Runs…
48 (16 x 3)Total Runs…
Output Tokens
15,845Output Tokens…
71,452Output Tokens…
Reasoning Tokens
0Reasoning Tokens…
155,147Reasoning Tokens…
Response Time (avg)
7.03sResponse Time (avg)…
29.10sResponse Time (avg)…
Response Time (max)
38.52sResponse Time (max)…
170.45sResponse Time (max)…
Response Time (total)
112.51sResponse Time (total)…
290.96sResponse Time (total)…
Top Models by Score
Score vs Total Cost
Response Time (avg)
Avg Score vs Response Time (avg)
Category Breakdown
Anti-AI Tricks
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.97sResponse Time (max)4.78sResponse Time (total)11.90sA test is fully passed only if every run passed for that test.…
3.97sResponse Time (avg)…
1,651Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)18.54sResponse Time (max)32.30sResponse Time (total)37.07sA test is fully passed only if every run passed for that test.…
18.54sResponse Time (avg)…
13,924Output Tokens…
17,208Reasoning Tokens…
Combined
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)9.12sResponse Time (max)9.12sResponse Time (total)9.12sA test is fully passed only if every run passed for that test.…
9.12sResponse Time (avg)…
1,243Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)29.57sResponse Time (max)29.57sResponse Time (total)29.57sA test is fully passed only if every run passed for that test.…
29.57sResponse Time (avg)…
1,176Output Tokens…
12,984Reasoning Tokens…
Data parsing and extraction
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.05sResponse Time (max)3.33sResponse Time (total)6.10sA test is fully passed only if every run passed for that test.…
3.05sResponse Time (avg)…
980Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)15.01sResponse Time (max)15.01sResponse Time (total)15.01sA test is fully passed only if every run passed for that test.…
15.01sResponse Time (avg)…
600Output Tokens…
13,886Reasoning Tokens…
Domain specific
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)17.78sResponse Time (max)38.52sResponse Time (total)53.33sA test is fully passed only if every run passed for that test.…
17.78sResponse Time (avg)…
7,810Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.4%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)170.45sResponse Time (max)170.45sResponse Time (total)170.45sA test is fully passed only if every run passed for that test.…
170.45sResponse Time (avg)…
45,350Output Tokens…
90,436Reasoning Tokens…
General Intelligence
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
3.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)3.20sResponse Time (max)3.20sResponse Time (total)3.20sA test is fully passed only if every run passed for that test.…
3.20sResponse Time (avg)…
335Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)6.54sResponse Time (max)6.54sResponse Time (total)6.54sA test is fully passed only if every run passed for that test.…
6.54sResponse Time (avg)…
2,214Output Tokens…
2,584Reasoning Tokens…
Instructions following
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
6.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
83.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)5.46sResponse Time (max)6.45sResponse Time (total)10.92sA test is fully passed only if every run passed for that test.…
5.46sResponse Time (avg)…
1,528Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
9.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
6.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
83.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)4.98sResponse Time (max)4.98sResponse Time (total)4.98sA test is fully passed only if every run passed for that test.…
4.98sResponse Time (avg)…
2,284Output Tokens…
3,412Reasoning Tokens…
Puzzle Solving
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
7.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)4.42sResponse Time (max)5.04sResponse Time (total)13.27sA test is fully passed only if every run passed for that test.…
4.42sResponse Time (avg)…
1,743Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)7.72sResponse Time (max)10.60sResponse Time (total)15.44sA test is fully passed only if every run passed for that test.…
7.72sResponse Time (avg)…
5,629Output Tokens…
10,835Reasoning Tokens…
Tool Calling
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Output Tokens
Reasoning Tokens
OpenAI: GPT-5.2 Chat
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.68sResponse Time (max)4.68sResponse Time (total)4.68sA test is fully passed only if every run passed for that test.…
4.68sResponse Time (avg)…
555Output Tokens…
0Reasoning Tokens…
StepFun: Step 3.5 Flash
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)11.91sResponse Time (max)11.91sResponse Time (total)11.91sA test is fully passed only if every run passed for that test.…