DeepSeek V4.1 Flash (low) leads on average score with 8.5 vs 8.4. DeepSeek V4.1 Flash (low) has the lower benchmark cost at $0.224 vs ~$0.287. DeepSeek V4.1 Flash (low) is faster at 26.77s vs 167.89s, with pass rates of 74.2% vs 77.3%.
Last updated at:
2026-09-10
Compared models
Rank
#51
Total Output Tokens
347,963
Response Time (avg)
26.77s
Total Cost
$0.224
Candidate responses were generated on local hardware. OpenRouter was used only for judging.
Rank
#59
Total Output Tokens
656,635
Response Time (avg)
167.89s
Total Cost
~$0.287Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.8999 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.286715.
Recommended modelDeepSeek V4.1 Flash (low)
It has the best score here (8.5), while responding about 6.3x faster than Qwen3.8 27B (high).
Qwen3.8 27BQwen3.8 27BhighCandidate responses were generated on local hardware. OpenRouter was used only for judging.Release: 2026-08-14
Score
8.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
8.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Rank
#51
#59
Reliability
9.5First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
9.6First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
Consistency
8.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Attempts
66/66
66/66
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 6API error: 1Response Time (avg)26.77sResponse Time (max)149.84sResponse Time (total)589.01sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Did not follow instructions: 2No answer: 2Wrong answer: 2Response Time (avg)167.89sResponse Time (max)640.85sResponse Time (total)3693.54sA test is fully passed only if every run passed for that test.…
Attempt pass rate
74.2%Attempt pass rate = passed attempts / total attempts across runs.…
77.3%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
3Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
66Total Runs…
66Total Runs…
Cost per result
1.492
~1.792Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.8999 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.286715.
Total Cost
$0.224
~$0.287Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.8999 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.286715.
Input Price
$0.150 / 1MInput Price…
N/AInput Price…
Output Price
$0.600 / 1MOutput Price…
N/AOutput Price…
Total Input Tokens
99,715Total Input Tokens…
100,179Total Input Tokens…
Output Tokens
6,381Output Tokens…
904Output Tokens…
Reasoning Tokens
341,582Reasoning Tokens…
655,731Reasoning Tokens…
Response Time (avg)
26.77sResponse Time (avg)…
167.89sResponse Time (avg)…
Response Time (max)
149.84sResponse Time (max)…
640.85sResponse Time (max)…
Response Time (total)
589.01sResponse Time (total)…
3693.54sResponse Time (total)…
Parameters
748B total (16B active)
27.3B
Availability
Open source
Weights available
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#51 DeepSeek V4.1 Flash
low
Cost
$0.017
Time
43.6s
Tokens
14,069 tok
#59 Qwen3.8 27B
highCandidate responses were generated on local hardware. OpenRouter was used only for judging.
Cost
~$0.008Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0750 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.007419.
8.3Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
75.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.32sResponse Time (max)5.98sResponse Time (total)13.28sA test is fully passed only if every run passed for that test.…
3.32sResponse Time (avg)…
852Total Input Tokens…
171Output Tokens…
4,328Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.10sResponse Time (max)21.48sResponse Time (total)32.40sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)40.66sResponse Time (max)56.87sResponse Time (total)121.97sA test is fully passed only if every run passed for that test.…
40.66sResponse Time (avg)…
7,509Total Input Tokens…
378Output Tokens…
81,978Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
8.1Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
88.9%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)248.62sResponse Time (max)458.25sResponse Time (total)745.85sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)24.73sResponse Time (max)34.30sResponse Time (total)49.47sA test is fully passed only if every run passed for that test.…
24.73sResponse Time (avg)…
77,368Total Input Tokens…
4,906Output Tokens…
24,016Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)185.57sResponse Time (max)333.21sResponse Time (total)371.15sA test is fully passed only if every run passed for that test.…
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.API error: 1Response Time (avg)3.80sResponse Time (max)7.39sResponse Time (total)7.61sA test is fully passed only if every run passed for that test.…
3.80sResponse Time (avg)…
2,397Total Input Tokens…
120Output Tokens…
1,149Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)34.85sResponse Time (max)57.71sResponse Time (total)69.71sA test is fully passed only if every run passed for that test.…
3.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
4.4Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Response Time (avg)95.58sResponse Time (max)149.84sResponse Time (total)286.75sA test is fully passed only if every run passed for that test.…
95.58sResponse Time (avg)…
909Total Input Tokens…
27Output Tokens…
184,701Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
5.3Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Wrong answer: 1Response Time (avg)526.74sResponse Time (max)640.85sResponse Time (total)1580.21sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.34sResponse Time (max)4.34sResponse Time (total)4.34sA test is fully passed only if every run passed for that test.…
4.34sResponse Time (avg)…
549Total Input Tokens…
117Output Tokens…
1,319Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
5.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)8.45sResponse Time (max)8.45sResponse Time (total)8.45sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.38sResponse Time (max)14.30sResponse Time (total)16.75sA test is fully passed only if every run passed for that test.…
8.38sResponse Time (avg)…
783Total Input Tokens…
63Output Tokens…
1,546Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
9.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.24sResponse Time (max)10.18sResponse Time (total)16.49sA test is fully passed only if every run passed for that test.…
8.2Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.7Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
77.8%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)9.39sResponse Time (max)20.40sResponse Time (total)28.16sA test is fully passed only if every run passed for that test.…
9.39sResponse Time (avg)…
828Total Input Tokens…
273Output Tokens…
14,502Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
7.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
77.8%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)199.90sResponse Time (max)569.33sResponse Time (total)599.69sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)13.83sResponse Time (max)13.83sResponse Time (total)13.83sA test is fully passed only if every run passed for that test.…
13.83sResponse Time (avg)…
8,259Total Input Tokens…
313Output Tokens…
284Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)5.62sResponse Time (max)5.62sResponse Time (total)5.62sA test is fully passed only if every run passed for that test.…
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)46.86sResponse Time (max)46.86sResponse Time (total)46.86sA test is fully passed only if every run passed for that test.…
46.86sResponse Time (avg)…
261Total Input Tokens…
13Output Tokens…
27,759Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)263.98sResponse Time (max)263.98sResponse Time (total)263.98sA test is fully passed only if every run passed for that test.…