- Rank
- #294
- Total Output Tokens
- 66,576
- Response Time (avg)
- 8.32s
- Total Cost
- $0.048
KAT Coder AIR V2.5 (medium) vs Mistral Small 4
The average score is effectively tied at 5.1 vs 5.1. Mistral Small 4 has the lower benchmark cost at $0.047 vs $0.048. Mistral Small 4 is faster at 3.12s vs 8.32s, with pass rates of 43.5% vs 24.6%.
Compared models
- Rank
- #292
- Total Output Tokens
- 10,913
- Response Time (avg)
- 3.12s
- Total Cost
- $0.047
Recommended model
Mistral Small 4
It has the best score here (5.1), while responding about 2.7x faster than KAT Coder AIR V2.5 (medium).
Detailed comparison
| Metric | KAT Coder AIR V2.5 KAT Coder AIR V2.5 medium | Mistral Small 4 Mistral Small 4 none |
|---|---|---|
| Score | 5.1 | 5.1 |
| Rank | #294 | #292 |
| Reliability | 9.6 | 10.0 |
| Consistency | 8.9 | 9.6 |
| Attempts | 69/69 | 69/69 |
| Tests Correct | ||
| Attempt pass rate | 43.5% | 24.6% |
| Flaky tests | 3 | 1 |
| Total Runs | 69 | 69 |
| Cost per result | 0.596 | 0.928 |
| Total Cost | $0.048 | $0.047 |
| Input Price | $0.000 / 1M | $0.150 / 1M |
| Output Price | $0.000 / 1M | $0.600 / 1M |
| Total Input Tokens | 51,369 | 265,602 |
| Output Tokens | 8,008 | 10,913 |
| Reasoning Tokens | 58,568 | 0 |
| Response Time (avg) | 8.32s | 3.12s |
| Response Time (max) | 48.19s | 45.46s |
| Response Time (total) | 191.40s | 71.87s |
| Parameters | ~35B total (~3B active) | 119B total (6B active) |
| Availability | Closed | Open source |
Model generation showcase
Hamster playing table tennis
Prompt: Create a detailed SVG illustration of a hamster playing table tennis.
#294 KAT Coder AIR V2.5
medium- Cost
- $0.003
- Time
- 23.7s
- Tokens
- 4,924 tok
#292 Mistral Small 4
none- Cost
- $0.002
- Time
- 10.4s
- Tokens
- 2,370 tok
Top Models by Score
Score vs Total Cost
Response Time (avg)
Score vs Response Time (avg)
Total Output Tokens
Score vs Total Output Tokens
Category Breakdown
| Agentic | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 3.0 | 10.0 | 0.0% | 0 | 1.05s | 0 | 0 | 0 | |
| Mistral Small 4 | 5.0 | 10.0 | 0.0% | 0 | 45.46s | 160,885 | 1,101 | 0 |
| Anti-AI Tricks | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 8.7 | 8.0 | 91.7% | 1 | 3.79s | 672 | 549 | 4,194 | |
| Mistral Small 4 | 3.4 | 7.9 | 16.7% | 1 | 395ms | 708 | 182 | 0 |
| Coding | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 3.6 | 7.0 | 22.2% | 1 | 23.37s | 7,893 | 5,199 | 29,973 | |
| Mistral Small 4 | 3.7 | 9.7 | 0.0% | 0 | 901ms | 7,636 | 619 | 0 |
| Combined | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 6.5 | 10.0 | 50.0% | 0 | 19.56s | 23,854 | 435 | 6,811 | |
| Mistral Small 4 | 3.0 | 10.0 | 0.0% | 0 | 7.44s | 79,039 | 8,107 | 0 |
| Data parsing and extraction | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 6.5 | 10.0 | 50.0% | 0 | 3.65s | 7,776 | 268 | 1,881 | |
| Mistral Small 4 | 10.0 | 10.0 | 100.0% | 0 | 822ms | 7,914 | 261 | 0 |
| Domain specific | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 3.0 | 10.0 | 0.0% | 0 | 6.58s | 668 | 673 | 4,670 | |
| Mistral Small 4 | 5.3 | 10.0 | 33.3% | 0 | 377ms | 807 | 28 | 0 |
| General Intelligence | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 5.0 | 10.0 | 0.0% | 0 | 9.81s | 516 | 189 | 557 | |
| Mistral Small 4 | 4.0 | 10.0 | 0.0% | 0 | 729ms | 519 | 205 | 0 |
| Instructions following | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 10.0 | 10.0 | 100.0% | 0 | 1.50s | 699 | 93 | 730 | |
| Mistral Small 4 | 6.5 | 10.0 | 50.0% | 0 | 380ms | 729 | 69 | 0 |
| Puzzle Solving | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 3.6 | 7.2 | 22.2% | 1 | 1.87s | 696 | 265 | 1,184 | |
| Mistral Small 4 | 3.1 | 9.9 | 0.0% | 0 | 399ms | 735 | 111 | 0 |
| Tool Calling | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 10.0 | 10.0 | 100.0% | 0 | 4.77s | 8,391 | 317 | 906 | |
| Mistral Small 4 | 10.0 | 10.0 | 100.0% | 0 | 1.40s | 6,420 | 213 | 0 |
| Trivia | Score | Consistency | Attempt pass rate | Flaky tests | Tests Correct | Response Time (avg) | Input Tokens | Output Tokens | Reasoning Tokens |
|---|---|---|---|---|---|---|---|---|---|
| KAT Coder AIR V2.5 | 3.0 | 10.0 | 0.0% | 0 | 15.74s | 204 | 20 | 7,662 | |
| Mistral Small 4 | 3.0 | 10.0 | 0.0% | 0 | 397ms | 210 | 17 | 0 |
Quick Compare
Switch Comparison Pair
Mercury 2.5lowvsKAT Coder AIR V2.5mediumKAT Coder AIR V2.5mediumvsMiMo-V2.5-PrononeMercury 2.5lowvsMistral Small 4noneKAT Coder AIR V2.5highvsMistral Small 4noneMistral Small 4nonevsNemotron 3.5 LightninghighFree AvailableMistral Small 4nonevsLaguna S 2.1mediumFree AvailableKAT Coder AIR V2.5mediumvsNemotron 3.5 LightninghighFree AvailableMercury 2.5 PreviewlowvsKAT Coder AIR V2.5mediumMistral Small 4nonevsInkling SmalllowFree AvailableLing 3.0 FlashnonevsKAT Coder AIR V2.5mediumMercury 2.5 PreviewlowvsMistral Small 4noneKAT Coder AIR V2.5mediumvsInkling SmalllowFree Available