Qwen3.8 Flash Next (low) vs Inkling 2 Mini (local Q4) (low)
Qwen3.8 Flash Next (low) leads on average score with 8.0 vs 7.6. Qwen3.8 Flash Next (low) has the lower benchmark cost at ~$0.045 vs ~$0.292. Qwen3.8 Flash Next (low) is faster at 23.66s vs 153.99s, with pass rates of 79.7% vs 66.7%.
Last updated at:
2026-11-22
Compared models
Candidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
Rank
#171
Total Output Tokens
135,295
Response Time (avg)
23.66s
Total Cost
~$0.045Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.4535 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.044834.
Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
Rank
#202
Total Output Tokens
129,600
Response Time (avg)
153.99s
Total Cost
~$0.292Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.9515 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.291818.
Recommended modelQwen3.8 Flash Next (low)
It has the best score here (8.0), while costing about 6.5x less than Inkling 2 Mini (local Q4) (low).
Detailed comparison
Metric
Qwen3.8 Flash NextQwen3.8 Flash NextlowCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.Release: 2026-10-04
Inkling 2 Mini (local Q4)Inkling 2 Mini (local Q4)lowCandidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.Release: 2026-11-20
Metric
Qwen3.8 Flash NextQwen3.8 Flash NextlowCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.Release: 2026-10-04
Inkling 2 Mini (local Q4)Inkling 2 Mini (local Q4)lowCandidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.Release: 2026-11-20
Score
8.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.6Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
Rank
#171
#202
Reliability
10.0First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
9.7First-attempt success score: 10.0 means no retryable target API or rate-limit failures before successful calls; tracked failures lower the score.…
Consistency
8.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
9.6Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
Attempts
69/69
69/69
Tests Correct
A test is fully passed only if every run passed for that test.Wrong answer: 6Did not follow instructions: 1Invalid tool call: 1Response Time (avg)23.66sResponse Time (max)166.72sResponse Time (total)544.16sA test is fully passed only if every run passed for that test.…
A test is fully passed only if every run passed for that test.Wrong answer: 5Did not follow instructions: 1No answer: 1Timed out: 1Response Time (avg)153.99sResponse Time (max)1963.67sResponse Time (total)3541.85sA test is fully passed only if every run passed for that test.…
Attempt pass rate
79.7%Attempt pass rate = passed attempts / total attempts across runs.…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
Flaky tests
6Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
Total Runs
69Total Runs…
69Total Runs…
Cost per result
~0.299Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.4535 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.044834.
~1.946Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.9515 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.291818.
Total Cost
~$0.045Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.4535 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.044834.
~$0.292Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 2.9515 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.291818.
Input Price
N/AInput Price…
N/AInput Price…
Output Price
N/AOutput Price…
N/AOutput Price…
Cache Read Price
N/ACache Read Price…
N/ACache Read Price…
Cache Write Price
N/ACache Write Price…
N/ACache Write Price…
Total Input Tokens
342,163Total Input Tokens…
312,366Total Input Tokens…
Output Tokens
1,518Output Tokens…
1,631Output Tokens…
Reasoning Tokens
133,777Reasoning Tokens…
127,969Reasoning Tokens…
Response Time (avg)
23.66sResponse Time (avg)…
153.99sResponse Time (avg)…
Response Time (max)
166.72sResponse Time (max)…
1963.67sResponse Time (max)…
Response Time (total)
544.16sResponse Time (total)…
3541.85sResponse Time (total)…
Parameters
180B total (6B active)
26.4B total (3.4B active)
Availability
Weights available
Open source
Cache prices apply to input tokens. Reads reuse cached prompts; writes store them and can cost extra. Output tokens use the output price.
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#171 Qwen3.8 Flash Next
lowCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
Cost
~$0.002Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0131 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.001295.
Time
47.2s
Tokens
5,270 tok
#202 Thinking Machines: Inkling 2 Mini (local Q4)
lowCandidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
Cost
~$0.003Estimated GPU electricity cost only. It excludes the rest of the computer and hardware ownership. NVIDIA GeForce RTX 3090: 0.0237 h × 0.300 kW × €0.2896/kWh × 1.138 USD/EUR = ~$0.002347.
Time
85.5s
Tokens
2,977 tok
Rank
-
Cost
-
Time
-
Tokens
-
Top Models by Score
Score vs Total Cost
Response Time (avg)
Score vs Response Time (avg)
Total Output Tokens
Score vs Total Output Tokens
Category Breakdown
Agentic
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
6.1Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
3.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)137.50sResponse Time (max)137.50sResponse Time (total)137.50sA test is fully passed only if every run passed for that test.…
137.50sResponse Time (avg)…
194,811Total Input Tokens…
72Output Tokens…
28,519Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)58.97sResponse Time (max)58.97sResponse Time (total)58.97sA test is fully passed only if every run passed for that test.…
58.97sResponse Time (avg)…
138,225Total Input Tokens…
129Output Tokens…
3,192Reasoning Tokens…
Anti-AI Tricks
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.03sResponse Time (max)2.72sResponse Time (total)8.13sA test is fully passed only if every run passed for that test.…
2.03sResponse Time (avg)…
984Total Input Tokens…
233Output Tokens…
1,773Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)5.40sResponse Time (max)8.07sResponse Time (total)21.60sA test is fully passed only if every run passed for that test.…
5.40sResponse Time (avg)…
732Total Input Tokens…
183Output Tokens…
2,778Reasoning Tokens…
Coding
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)29.46sResponse Time (max)49.90sResponse Time (total)88.39sA test is fully passed only if every run passed for that test.…
29.46sResponse Time (avg)…
8,127Total Input Tokens…
397Output Tokens…
24,147Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
5.3Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.2Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.4%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Timed out: 1Wrong answer: 1Response Time (avg)831.97sResponse Time (max)1963.67sResponse Time (total)2495.90sA test is fully passed only if every run passed for that test.…
831.97sResponse Time (avg)…
5,668Total Input Tokens…
524Output Tokens…
34,490Reasoning Tokens…
Combined
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
6.9Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
5.9Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Invalid tool call: 1Response Time (avg)98.91sResponse Time (max)166.72sResponse Time (total)197.82sA test is fully passed only if every run passed for that test.…
98.91sResponse Time (avg)…
118,012Total Input Tokens…
60Output Tokens…
51,088Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
6.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
50.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No answer: 1Response Time (avg)123.84sResponse Time (max)227.77sResponse Time (total)247.69sA test is fully passed only if every run passed for that test.…
123.84sResponse Time (avg)…
145,811Total Input Tokens…
48Output Tokens…
29,790Reasoning Tokens…
Data parsing and extraction
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)6.20sResponse Time (max)6.23sResponse Time (total)12.40sA test is fully passed only if every run passed for that test.…
6.20sResponse Time (avg)…
8,253Total Input Tokens…
267Output Tokens…
2,783Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)20.83sResponse Time (max)21.70sResponse Time (total)41.66sA test is fully passed only if every run passed for that test.…
20.83sResponse Time (avg)…
9,657Total Input Tokens…
207Output Tokens…
4,340Reasoning Tokens…
Domain specific
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
4.1Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
4.4Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
44.5%Attempt pass rate = passed attempts / total attempts across runs.…
2Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 3Response Time (avg)23.34sResponse Time (max)42.75sResponse Time (total)70.03sA test is fully passed only if every run passed for that test.…
23.34sResponse Time (avg)…
1,014Total Input Tokens…
12Output Tokens…
18,966Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
7.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
66.7%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)198.65sResponse Time (max)335.23sResponse Time (total)595.96sA test is fully passed only if every run passed for that test.…
198.65sResponse Time (avg)…
825Total Input Tokens…
12Output Tokens…
43,200Reasoning Tokens…
General Intelligence
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
4.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
3.1Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)5.00sResponse Time (max)5.00sResponse Time (total)5.00sA test is fully passed only if every run passed for that test.…
5.00sResponse Time (avg)…
594Total Input Tokens…
130Output Tokens…
1,171Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
5.5Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)5.55sResponse Time (max)5.55sResponse Time (total)5.55sA test is fully passed only if every run passed for that test.…
5.55sResponse Time (avg)…
558Total Input Tokens…
60Output Tokens…
672Reasoning Tokens…
Instructions following
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
9.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)2.32sResponse Time (max)3.23sResponse Time (total)4.64sA test is fully passed only if every run passed for that test.…
2.32sResponse Time (avg)…
855Total Input Tokens…
48Output Tokens…
1,050Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
9.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)5.10sResponse Time (max)5.52sResponse Time (total)10.21sA test is fully passed only if every run passed for that test.…
5.10sResponse Time (avg)…
738Total Input Tokens…
42Output Tokens…
1,305Reasoning Tokens…
Puzzle Solving
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
8.7Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
7.8Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
88.9%Attempt pass rate = passed attempts / total attempts across runs.…
1Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Response Time (avg)1.72sResponse Time (max)2.26sResponse Time (total)5.15sA test is fully passed only if every run passed for that test.…
1.72sResponse Time (avg)…
930Total Input Tokens…
272Output Tokens…
1,001Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
5.8Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
33.3%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Did not follow instructions: 1Wrong answer: 1Response Time (avg)10.51sResponse Time (max)14.31sResponse Time (total)31.52sA test is fully passed only if every run passed for that test.…
10.51sResponse Time (avg)…
756Total Input Tokens…
396Output Tokens…
3,885Reasoning Tokens…
Tool Calling
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)5.85sResponse Time (max)5.85sResponse Time (total)5.85sA test is fully passed only if every run passed for that test.…
5.85sResponse Time (avg)…
8,301Total Input Tokens…
21Output Tokens…
816Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
10.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
100.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.No failed answers.Response Time (avg)7.53sResponse Time (max)7.53sResponse Time (total)7.53sA test is fully passed only if every run passed for that test.…
7.53sResponse Time (avg)…
9,180Total Input Tokens…
27Output Tokens…
822Reasoning Tokens…
Trivia
Score
Consistency
Attempt pass rate
Flaky tests
Tests Correct
Response Time (avg)
Input Tokens
Output Tokens
Reasoning Tokens
Qwen3.8 Flash NextCandidate responses were generated on local hardware. OpenRouter was used only for judging. qwen3.8-flash-next-iq3_s · IQ3_S · Strata 0.1.39 · NVIDIA GeForce RTX 3090.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)9.26sResponse Time (max)9.26sResponse Time (total)9.26sA test is fully passed only if every run passed for that test.…
9.26sResponse Time (avg)…
282Total Input Tokens…
6Output Tokens…
2,463Reasoning Tokens…
Inkling 2 Mini (local Q4)Candidate responses were generated on local hardware. OpenRouter was used only for judging. inkling-2-mini-ud-q4_k_xl · UD-Q4_K_XL · llama.cpp b12631-e300919cc · NVIDIA GeForce RTX 3090.
3.0Summarizes broad quality across our full private benchmark suite, so ranking reflects consistent performance.…
10.0Consistency score reflects run-to-run stability (10 = very consistent, even if consistently wrong).…
0.0%Attempt pass rate = passed attempts / total attempts across runs.…
0Flaky tests had mixed outcomes across runs (at least one pass and one fail).…
A test is fully passed only if every run passed for that test.Wrong answer: 1Response Time (avg)25.28sResponse Time (max)25.28sResponse Time (total)25.28sA test is fully passed only if every run passed for that test.…