The average score is effectively tied at 5.6 vs 5.5. Qwen3.8 27B has the lower benchmark cost at ~$0.006 vs $0.048. Qwen3.8 27B is faster at 3.12s vs 8.65s, with pass rates of 45.5% vs 40.9%.
Last updated at:
2026-09-28
Compared models
Rank
#272
Total Output Tokens
66,576
Response Time (avg)
8.65s
Total Cost
$0.048
Candidate responses were generated on local hardware. OpenRouter was used only for judging.
Rank
#277
Total Output Tokens
11,445
Response Time (avg)
3.12s
Total Cost
~$0.006Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0572 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.005659.
Recommended modelQwen3.8 27B
It has the best score here (5.5), while costing about 8.4x less than KAT-Coder-Air V2.5 (medium).
A test is fully passed only if every run passed for that test.Wrong answer: 11Did not follow instructions: 1Invalid tool call: 1Response Time (avg)3.12sResponse Time (max)44.76sResponse Time (total)68.69sA test is fully passed only if every run passed for that test.…
Attempt pass rate
45.5%Attempt pass rate = passed attempts / total attempts across runs.…
40.9%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
3Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
66Total Runs…
66Total Runs…
Cost per result
0.596
~0.063Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0572 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.005659.
Total Cost
$0.048
~$0.006Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0572 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.005659.
Input Price
$0.150 / 1MInput Price…
N/AInput Price…
Output Price
$0.600 / 1MOutput Price…
N/AOutput Price…
Total Input Tokens
51,369Total Input Tokens…
122,463Total Input Tokens…
Output Tokens
8,008Output Tokens…
11,445Output Tokens…
Reasoning Tokens
58,568Reasoning Tokens…
0Reasoning Tokens…
Response Time (avg)
8.65sResponse Time (avg)…
3.12sResponse Time (avg)…
Response Time (max)
48.19sResponse Time (max)…
44.76sResponse Time (max)…
Response Time (total)
190.35sResponse Time (total)…
68.69sResponse Time (total)…
Parameters
~35B total (~3B active)
27.3B
Availability
Closed
Weights available
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#272 KAT-Coder-Air V2.5
medium
Cost
$0.003
Time
23.7s
Tokens
4,924 tok
#277 Qwen3.8 27B
noneCandidate responses were generated on local hardware. OpenRouter was used only for judging.
Cost
~$0.001Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0066 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.000657.
8.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
8.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
91.7%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.79sResponse Time (max)9.97sResponse Time (total)15.17sA test is fully passed only if every run passed for that test.…
3.79sResponse Time (avg)…
672Total Input Tokens…
549Output Tokens…
4,194Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)828msResponse Time (max)1.95sResponse Time (total)3.31sA test is fully passed only if every run passed for that test.…
3.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
22.2%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Extra formatting: 1Response Time (avg)23.37sResponse Time (max)48.19sResponse Time (total)70.10sA test is fully passed only if every run passed for that test.…
23.37sResponse Time (avg)…
7,893Total Input Tokens…
5,199Output Tokens…
29,973Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
5.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)1.19sResponse Time (max)1.97sResponse Time (total)3.56sA test is fully passed only if every run passed for that test.…
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.API error: 1Response Time (avg)19.56sResponse Time (max)27.68sResponse Time (total)39.11sA test is fully passed only if every run passed for that test.…
19.56sResponse Time (avg)…
23,854Total Input Tokens…
435Output Tokens…
6,811Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Invalid tool call: 1Wrong answer: 1Response Time (avg)24.10sResponse Time (max)44.76sResponse Time (total)48.21sA test is fully passed only if every run passed for that test.…
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)3.65sResponse Time (max)4.88sResponse Time (total)7.31sA test is fully passed only if every run passed for that test.…
3.65sResponse Time (avg)…
7,776Total Input Tokens…
268Output Tokens…
1,881Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)1.14sResponse Time (max)1.18sResponse Time (total)2.27sA test is fully passed only if every run passed for that test.…
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Timed out: 1Response Time (avg)6.58sResponse Time (max)7.90sResponse Time (total)19.73sA test is fully passed only if every run passed for that test.…
6.58sResponse Time (avg)…
668Total Input Tokens…
673Output Tokens…
4,670Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
7.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)552msResponse Time (max)675msResponse Time (total)1.66sA test is fully passed only if every run passed for that test.…
5.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)9.81sResponse Time (max)9.81sResponse Time (total)9.81sA test is fully passed only if every run passed for that test.…
9.81sResponse Time (avg)…
516Total Input Tokens…
189Output Tokens…
557Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
4.2Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
9.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)1.31sResponse Time (max)1.31sResponse Time (total)1.31sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)1.50sResponse Time (max)1.88sResponse Time (total)3.00sA test is fully passed only if every run passed for that test.…
1.50sResponse Time (avg)…
699Total Input Tokens…
93Output Tokens…
730Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
6.3Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)582msResponse Time (max)633msResponse Time (total)1.16sA test is fully passed only if every run passed for that test.…
3.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
22.2%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Response Time (avg)1.87sResponse Time (max)2.41sResponse Time (total)5.61sA test is fully passed only if every run passed for that test.…
1.87sResponse Time (avg)…
696Total Input Tokens…
265Output Tokens…
1,184Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
6.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)1.25sResponse Time (max)1.84sResponse Time (total)3.76sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)4.77sResponse Time (max)4.77sResponse Time (total)4.77sA test is fully passed only if every run passed for that test.…
4.77sResponse Time (avg)…
8,391Total Input Tokens…
317Output Tokens…
906Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.75sResponse Time (max)2.75sResponse Time (total)2.75sA test is fully passed only if every run passed for that test.…
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)15.74sResponse Time (max)15.74sResponse Time (total)15.74sA test is fully passed only if every run passed for that test.…
15.74sResponse Time (avg)…
204Total Input Tokens…
20Output Tokens…
7,662Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)703msResponse Time (max)703msResponse Time (total)703msA test is fully passed only if every run passed for that test.…