Muse Spark 1.1 (high) leads on average score with 8.0 vs 7.9. Qwen3.8 27B (low) has the lower benchmark cost at ~$0.071 vs $1.709. Muse Spark 1.1 (high) is faster at 30.84s vs 39.11s, with pass rates of 68.2% vs 68.2%.
Last updated at:
2026-08-29
Rank
#53
Total Output Tokens
363,989
Response Time (avg)
30.84s
Total Cost
$1.709
Candidate responses were generated on local hardware. OpenRouter was used only for judging.
Rank
#62
Total Output Tokens
169,329
Response Time (avg)
39.11s
Total Cost
~$0.071Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.7171 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.070898.
Recommended modelQwen3.8 27B (low)
Its score stays close to the best score here (7.9 vs 8.0), while costing about 24.1x less than Muse Spark 1.1 (high).
A test is fully passed only if every run passed for that test.Wrong answer: 4No answer: 2Did not follow instructions: 1Response Time (avg)39.11sResponse Time (max)376.04sResponse Time (total)860.51sA test is fully passed only if every run passed for that test.…
Attempt pass rate
68.2%Attempt pass rate = passed attempts / total attempts across runs.…
68.2%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
6Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
64Total Runs…
66Total Runs…
Cost per result
14.236
~0.473Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.7171 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.070898.
Total Cost
$1.709
~$0.071Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.7171 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.070898.
Input Price
$1.250 / 1MInput Price…
N/AInput Price…
Output Price
$4.250 / 1MOutput Price…
N/AOutput Price…
Total Input Tokens
129,003Total Input Tokens…
99,705Total Input Tokens…
Output Tokens
8,340Output Tokens…
1,545Output Tokens…
Reasoning Tokens
355,649Reasoning Tokens…
167,784Reasoning Tokens…
Response Time (avg)
30.84sResponse Time (avg)…
39.11sResponse Time (avg)…
Response Time (max)
196.03sResponse Time (max)…
376.04sResponse Time (max)…
Response Time (total)
647.67sResponse Time (total)…
860.51sResponse Time (total)…
Parameters
~1T total (~50B active)
27.3B
Availability
Closed
Weights available
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#53 Muse Spark 1.1
high
Cost
$0.039
Time
50.1s
Tokens
9,423 tok
#62 Qwen3.8 27B
lowCandidate responses were generated on local hardware. OpenRouter was used only for judging.
Cost
~$0.003Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0258 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.002549.
7.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
8.4Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)8.60sResponse Time (max)15.63sResponse Time (total)34.39sA test is fully passed only if every run passed for that test.…
8.60sResponse Time (avg)…
2,334Total Input Tokens…
516Output Tokens…
13,023Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)3.02sResponse Time (max)5.68sResponse Time (total)12.07sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)22.92sResponse Time (max)27.22sResponse Time (total)68.75sA test is fully passed only if every run passed for that test.…
22.92sResponse Time (avg)…
8,562Total Input Tokens…
349Output Tokens…
36,039Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
7.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)140.92sResponse Time (max)376.04sResponse Time (total)422.75sA test is fully passed only if every run passed for that test.…
5.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
2.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Invalid tool call: 2Response Time (avg)70.25sResponse Time (max)78.10sResponse Time (total)140.51sA test is fully passed only if every run passed for that test.…
70.25sResponse Time (avg)…
90,082Total Input Tokens…
5,535Output Tokens…
68,439Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)90.63sResponse Time (max)143.71sResponse Time (total)181.27sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.40sResponse Time (max)10.84sResponse Time (total)16.81sA test is fully passed only if every run passed for that test.…
8.40sResponse Time (avg)…
7,827Total Input Tokens…
240Output Tokens…
5,907Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)8.03sResponse Time (max)9.09sResponse Time (total)16.06sA test is fully passed only if every run passed for that test.…
2.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
4.4Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
22.2%Attempt pass rate = passed attempts / total attempts across runs.…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.API error: 1Extra formatting: 1Wrong answer: 1Response Time (avg)60.59sResponse Time (max)67.02sResponse Time (total)121.18sA test is fully passed only if every run passed for that test.…
60.59sResponse Time (avg)…
852Total Input Tokens…
329Output Tokens…
67,684Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
5.3Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 2Response Time (avg)60.78sResponse Time (max)120.94sResponse Time (total)182.34sA test is fully passed only if every run passed for that test.…
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)7.73sResponse Time (max)7.73sResponse Time (total)7.73sA test is fully passed only if every run passed for that test.…
7.73sResponse Time (avg)…
906Total Input Tokens…
116Output Tokens…
3,390Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
5.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)5.37sResponse Time (max)5.37sResponse Time (total)5.37sA test is fully passed only if every run passed for that test.…
6.4Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
6.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)7.81sResponse Time (max)9.13sResponse Time (total)15.62sA test is fully passed only if every run passed for that test.…
7.81sResponse Time (avg)…
1,527Total Input Tokens…
193Output Tokens…
7,158Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.43sResponse Time (max)2.87sResponse Time (total)4.86sA test is fully passed only if every run passed for that test.…
7.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
9.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)69.96sResponse Time (max)196.03sResponse Time (total)209.89sA test is fully passed only if every run passed for that test.…
69.96sResponse Time (avg)…
1,938Total Input Tokens…
225Output Tokens…
140,780Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
7.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)4.91sResponse Time (max)8.81sResponse Time (total)14.74sA test is fully passed only if every run passed for that test.…
9.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)9.88sResponse Time (max)9.88sResponse Time (total)9.88sA test is fully passed only if every run passed for that test.…
9.88sResponse Time (avg)…
14,363Total Input Tokens…
795Output Tokens…
2,938Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)8.42sResponse Time (max)8.42sResponse Time (total)8.42sA test is fully passed only if every run passed for that test.…
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)22.93sResponse Time (max)22.93sResponse Time (total)22.93sA test is fully passed only if every run passed for that test.…
22.93sResponse Time (avg)…
612Total Input Tokens…
42Output Tokens…
10,291Reasoning Tokens…
Qwen3.8 27BCandidate responses were generated on local hardware. OpenRouter was used only for judging.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)12.64sResponse Time (max)12.64sResponse Time (total)12.64sA test is fully passed only if every run passed for that test.…