The average score is effectively tied at 9.9 vs 9.9. Gemini 3.8 Flash (high) has the lower benchmark cost at $1.927 vs $6.213. GPT-6 Astra (xhigh) is faster at 29.67s vs 40.60s, with pass rates of 98.6% vs 97.1%.
Last updated at:
2026-10-19
Compared models
Rank
#2
Total Output Tokens
446,728
Response Time (avg)
40.60s
Total Cost
$1.927
Rank
#8
Total Output Tokens
90,095
Response Time (avg)
29.67s
Total Cost
$6.213
Recommended modelGemini 3.8 Flash (high)
It has the best score here (9.9), while costing about 3.2x less than GPT-6 Astra (xhigh).
GPT-6 AstraGPT-6 AstraxhighArchived model: this model is no longer updated or tested on new tests.Release: 2026-09-04
Score
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
9.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Rank
#2
#8
Reliability
10.0First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
9.9First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
Consistency
9.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Attempts
69/69
69/69
Tests Correct
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)40.60sResponse Time (max)177.08sResponse Time (total)933.73sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)29.67sResponse Time (max)381.11sResponse Time (total)682.42sA test is fully passed only if every run passed for that test.…
Attempt pass rate
98.6%Attempt pass rate = passed attempts / total attempts across runs.…
97.1%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
69Total Runs…
69Total Runs…
Cost per result
8.756
28.237
Total Cost
$1.927
$6.213
Input Price
$0.750 / 1MInput Price…
$10.000 / 1MInput Price…
Output Price
$3.750 / 1MOutput Price…
$50.000 / 1MOutput Price…
Cache Read Price
$0.075 / 1MCache Read Price…
$1.000 / 1MCache Read Price…
Cache Write Price
$0.042 / 1MCache Write Price…
$12.500 / 1MCache Write Price…
Total Input Tokens
334,568Total Input Tokens…
170,731Total Input Tokens…
Output Tokens
11,394Output Tokens…
5,865Output Tokens…
Reasoning Tokens
435,334Reasoning Tokens…
84,230Reasoning Tokens…
Response Time (avg)
40.60sResponse Time (avg)…
29.67sResponse Time (avg)…
Response Time (max)
177.08sResponse Time (max)…
381.11sResponse Time (max)…
Response Time (total)
933.73sResponse Time (total)…
682.42sResponse Time (total)…
Parameters
~500B total (~20B active)
~2T total (~150B active)
Availability
Closed
Closed
Cache prices apply to input tokens. Reads reuse cached prompts; writes store them and can cost extra. Output tokens use the output price.
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)115.54sResponse Time (max)115.54sResponse Time (total)115.54sA test is fully passed only if every run passed for that test.…
115.54sResponse Time (avg)…
249,975Total Input Tokens…
877Output Tokens…
37,663Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)56.73sResponse Time (max)56.73sResponse Time (total)56.73sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.92sResponse Time (max)3.96sResponse Time (total)11.69sA test is fully passed only if every run passed for that test.…
2.92sResponse Time (avg)…
492Total Input Tokens…
172Output Tokens…
4,363Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.71sResponse Time (max)5.18sResponse Time (total)14.85sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)29.30sResponse Time (max)55.87sResponse Time (total)87.90sA test is fully passed only if every run passed for that test.…
29.30sResponse Time (avg)…
8,118Total Input Tokens…
462Output Tokens…
50,196Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.27sResponse Time (max)11.08sResponse Time (total)24.81sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)115.73sResponse Time (max)177.08sResponse Time (total)231.45sA test is fully passed only if every run passed for that test.…
115.73sResponse Time (avg)…
60,514Total Input Tokens…
4,032Output Tokens…
132,696Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)41.20sResponse Time (max)63.24sResponse Time (total)82.41sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.23sResponse Time (max)8.26sResponse Time (total)16.46sA test is fully passed only if every run passed for that test.…
8.23sResponse Time (avg)…
7,548Total Input Tokens…
279Output Tokens…
7,902Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.34sResponse Time (max)3.64sResponse Time (total)6.69sA test is fully passed only if every run passed for that test.…
8.2Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
88.9%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)131.31sResponse Time (max)157.23sResponse Time (total)393.92sA test is fully passed only if every run passed for that test.…
131.31sResponse Time (avg)…
649Total Input Tokens…
4,972Output Tokens…
167,505Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
7.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
77.8%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)146.97sResponse Time (max)381.11sResponse Time (total)440.90sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.06sResponse Time (max)4.06sResponse Time (total)4.06sA test is fully passed only if every run passed for that test.…
4.06sResponse Time (avg)…
486Total Input Tokens…
75Output Tokens…
1,576Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)7.68sResponse Time (max)7.68sResponse Time (total)7.68sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.73sResponse Time (max)5.47sResponse Time (total)9.46sA test is fully passed only if every run passed for that test.…
4.73sResponse Time (avg)…
615Total Input Tokens…
72Output Tokens…
4,241Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.91sResponse Time (max)5.08sResponse Time (total)9.82sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.44sResponse Time (max)3.62sResponse Time (total)10.31sA test is fully passed only if every run passed for that test.…
3.44sResponse Time (avg)…
558Total Input Tokens…
204Output Tokens…
3,704Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.48sResponse Time (max)5.60sResponse Time (total)13.44sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)42.65sResponse Time (max)42.65sResponse Time (total)42.65sA test is fully passed only if every run passed for that test.…
42.65sResponse Time (avg)…
5,457Total Input Tokens…
238Output Tokens…
20,359Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)11.04sResponse Time (max)11.04sResponse Time (total)11.04sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)10.30sResponse Time (max)10.30sResponse Time (total)10.30sA test is fully passed only if every run passed for that test.…
10.30sResponse Time (avg)…
156Total Input Tokens…
11Output Tokens…
5,129Reasoning Tokens…
GPT-6 AstraArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)14.06sResponse Time (max)14.06sResponse Time (total)14.06sA test is fully passed only if every run passed for that test.…