AI BENCHY Compare

Google: Gemma 4 26B A4B vs Kwaipilot: KAT-Coder-Air V2.5

Summary

KAT-Coder-Air V2.5 (low) leads on average score with 6.0 vs 5.5. Gemma 4 26B A4B has the lower benchmark cost at $0.004 vs $0.030. Gemma 4 26B A4B is faster at 5.91s vs 6.47s, with pass rates of 44.4% vs 46.0%.

Recommended modelKAT-Coder-Air V2.5 (low)It has the strongest score in this comparison (6.0) and the best overall balance of cost and response time across all 2 models.

Last updated at: 2026-07-14

Metric	Gemma 4 26B A4B Gemma 4 26B A4B none Release: 2026-04-03 Free Available	KAT-Coder-Air V2.5 KAT-Coder-Air V2.5 low Release: 2026-07-14

Metric	Gemma 4 26B A4B Gemma 4 26B A4B none Release: 2026-04-03 Free Available	KAT-Coder-Air V2.5 KAT-Coder-Air V2.5 low Release: 2026-07-14
Score	5.5	6.0
Rank	#146	#124
Reliability	10.0	9.7
Consistency	9.2	7.6
Tests Correct
Attempt pass rate	44.4%	46.0%
Flaky tests	2	6
Total Runs	63	63
Cost per result	0.068	0.427
Total Cost	$0.004	$0.030
Input Price	$0.060 / 1M	$0.150 / 1M
Output Price	$0.330 / 1M	$0.600 / 1M
Total Input Tokens	40,038	42,464
Output Tokens	1,824	4,227
Reasoning Tokens	0	34,864
Response Time (avg)	5.91s	6.47s
Response Time (max)	57.10s	25.49s
Response Time (total)	124.05s	135.80s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#146 Gemma 4 26B A4B

none

Cost: $0.001
Time: 39.5s
Tokens: 790 tok

#124 KAT-Coder-Air V2.5

low

Cost: $0.004
Time: 29.9s
Tokens: 6,008 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Category:

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	8.3	10.0	75.0%	0		1.28s	852	230	0
KAT-Coder-Air V2.5	7.3	5.8	83.3%	2		3.50s	672	422	1,548

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.7	7.2	22.2%	1		4.16s	7,736	476	0
KAT-Coder-Air V2.5	3.5	7.3	11.1%	1		11.06s	7,893	2,001	13,623

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.0	10.0	0.0%	0		30.53s	13,650	309	0
KAT-Coder-Air V2.5	10.0	10.0	100.0%	0		25.49s	15,132	368	5,707

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	10.0	10.0	100.0%	0		1.70s	8,352	285	0
KAT-Coder-Air V2.5	6.5	10.0	50.0%	0		2.82s	7,776	261	1,069

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.6	7.2	22.2%	1		2.49s	903	27	0
KAT-Coder-Air V2.5	2.9	7.2	11.1%	1		4.99s	553	310	3,332

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	4.0	10.0	0.0%	0		3.54s	576	85	0
KAT-Coder-Air V2.5	5.0	10.0	0.0%	0		10.11s	516	142	596

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	6.3	10.0	50.0%	0		690ms	795	75	0
KAT-Coder-Air V2.5	9.8	10.0	100.0%	0		1.64s	699	89	640

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	6.2	10.0	33.3%	0		744ms	828	114	0
KAT-Coder-Air V2.5	3.1	4.6	22.2%	2		1.57s	696	294	796

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	10.0	10.0	100.0%	0		57.10s	6,123	210	0
KAT-Coder-Air V2.5	10.0	10.0	100.0%	0		4.47s	8,391	330	1,065

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.0	10.0	0.0%	0		778ms	223	13	0
KAT-Coder-Air V2.5	3.0	10.0	0.0%	0		19.98s	136	10	6,488

Quick Compare

Switch Comparison Pair