Gemini 3.5 Flash vs Grok 4.1 Fast benchmark comparison: Gemini 3.5 Flash leads on average score with 9.8 vs 4.0. Grok 4.1 Fast has the lower benchmark cost at $0.008 vs $1.115. Grok 4.1 Fast is faster at 1.62s vs 8.84s, with pass rates of 96.8% vs 20.6%.
Recommended model: Gemini 3.5 Flash - It has the strongest score in this comparison (9.8) and the best overall balance of cost and response time across all 2 models.
Grok 4.1 FastGrok 4.1 FastnoneArchived model: this model is no longer updated or tested on new tests.Release: 2025-11-19
Score
9.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
4.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Rank
#1
#166
Reliability
10.0First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
10.0First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
Consistency
9.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
7.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)8.84sResponse Time (max)34.82sResponse Time (total)185.57sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Wrong answer: 13Did not follow instructions: 3Response Time (avg)1.62sResponse Time (max)5.51sResponse Time (total)19.48sA test is fully passed only if every run passed for that test.…
Attempt pass rate
96.8%Attempt pass rate = passed attempts / total attempts across runs.…
20.6%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
3Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
63Total Runs…
57Total Runs…
Cost per result
5.575Shows the average cost per correct benchmark answer in cents (lower is better).…
0.273Shows the average cost per correct benchmark answer in cents (lower is better).…
Total Cost
$1.115Total Cost (Current Price)…
$0.008Total Cost (Current Price)…
Input Price
$1.500 / 1MInput Price…
$0.202 / 1MInput Price…
Output Price
$9.000 / 1MOutput Price…
$0.202 / 1MOutput Price…
Total Input Tokens
37,594Total Input Tokens…
36,608Total Input Tokens…
Output Tokens
1,975Output Tokens…
1,723Output Tokens…
Reasoning Tokens
115,638Reasoning Tokens…
0Reasoning Tokens…
Response Time (avg)
8.84sResponse Time (avg)…
1.62sResponse Time (avg)…
Response Time (max)
34.82sResponse Time (max)…
5.51sResponse Time (max)…
Response Time (total)
185.57sResponse Time (total)…
19.48sResponse Time (total)…
Generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#1 Gemini 3.5 Flash
high
Cost
$0.208
Time
118.2s
Tokens
23,158 tok
#166 Grok 4.1 Fast
none
Grok 4.1 Fast is deprecated. xAI recommends switching to Grok 4.3 (https://openrouter.ai/x-ai/grok-4.3)
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.57sResponse Time (max)3.60sResponse Time (total)10.27sA test is fully passed only if every run passed for that test.…
2.57sResponse Time (avg)…
492Total Input Tokens…
174Output Tokens…
4,997Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
3.2Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Did not follow instructions: 1Response Time (avg)1.07sResponse Time (max)1.73sResponse Time (total)2.15sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)22.96sResponse Time (max)34.82sResponse Time (total)68.88sA test is fully passed only if every run passed for that test.…
22.96sResponse Time (avg)…
8,118Total Input Tokens…
456Output Tokens…
47,129Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
1.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
1.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
11.1%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)1.79sResponse Time (max)1.79sResponse Time (total)1.79sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)22.37sResponse Time (max)22.37sResponse Time (total)22.37sA test is fully passed only if every run passed for that test.…
22.37sResponse Time (avg)…
12,873Total Input Tokens…
351Output Tokens…
16,323Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.33sResponse Time (max)3.33sResponse Time (total)3.33sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)6.43sResponse Time (max)8.51sResponse Time (total)12.87sA test is fully passed only if every run passed for that test.…
6.43sResponse Time (avg)…
7,548Total Input Tokens…
279Output Tokens…
8,466Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)943msResponse Time (max)943msResponse Time (total)943msA test is fully passed only if every run passed for that test.…
7.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
77.8%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)14.09sResponse Time (max)22.00sResponse Time (total)42.27sA test is fully passed only if every run passed for that test.…
14.09sResponse Time (avg)…
633Total Input Tokens…
12Output Tokens…
24,721Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
5.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
55.6%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)1.06sResponse Time (max)1.06sResponse Time (total)1.06sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.63sResponse Time (max)3.63sResponse Time (total)3.63sA test is fully passed only if every run passed for that test.…
3.63sResponse Time (avg)…
486Total Input Tokens…
115Output Tokens…
1,650Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
4.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
9.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)1.08sResponse Time (max)1.08sResponse Time (total)1.08sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.35sResponse Time (max)3.42sResponse Time (total)6.69sA test is fully passed only if every run passed for that test.…
3.35sResponse Time (avg)…
615Total Input Tokens…
70Output Tokens…
3,799Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)685msResponse Time (max)685msResponse Time (total)685msA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.23sResponse Time (max)3.68sResponse Time (total)9.69sA test is fully passed only if every run passed for that test.…
3.23sResponse Time (avg)…
558Total Input Tokens…
241Output Tokens…
4,940Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Response Time (avg)1.10sResponse Time (max)1.36sResponse Time (total)2.21sA test is fully passed only if every run passed for that test.…
9.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.96sResponse Time (max)4.96sResponse Time (total)4.96sA test is fully passed only if every run passed for that test.…
4.96sResponse Time (avg)…
6,115Total Input Tokens…
265Output Tokens…
1,608Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
2.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
1.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)5.51sResponse Time (max)5.51sResponse Time (total)5.51sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.94sResponse Time (max)3.94sResponse Time (total)3.94sA test is fully passed only if every run passed for that test.…
3.94sResponse Time (avg)…
156Total Input Tokens…
12Output Tokens…
2,005Reasoning Tokens…
Grok 4.1 FastArchived model: this model is no longer updated or tested on new tests.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)731msResponse Time (max)731msResponse Time (total)731msA test is fully passed only if every run passed for that test.…