AI BENCHY Compare

Google: Gemma 4 26B A4B vs Kwaipilot: KAT-Coder-Air V2.5

Summary

KAT-Coder-Air V2.5 (medium) leads on average score with 6.2 vs 5.5. Gemma 4 26B A4B has the lower benchmark cost at $0.004 vs $0.045. Gemma 4 26B A4B is faster at 5.91s vs 8.28s, with pass rates of 44.4% vs 47.6%.

Recommended modelKAT-Coder-Air V2.5 (medium)It has the strongest score in this comparison (6.2) and the best overall balance of cost and response time across all 2 models.

Last updated at: 2026-07-14

Metric	Gemma 4 26B A4B Gemma 4 26B A4B none Release: 2026-04-03 Free Available	KAT-Coder-Air V2.5 KAT-Coder-Air V2.5 medium Release: 2026-07-14

Metric	Gemma 4 26B A4B Gemma 4 26B A4B none Release: 2026-04-03 Free Available	KAT-Coder-Air V2.5 KAT-Coder-Air V2.5 medium Release: 2026-07-14
Score	5.5	6.2
Rank	#146	#115
Reliability	10.0	9.7
Consistency	9.2	8.8
Tests Correct
Attempt pass rate	44.4%	47.6%
Flaky tests	2	3
Total Runs	63	63
Cost per result	0.068	0.562
Total Cost	$0.004	$0.045
Input Price	$0.060 / 1M	$0.150 / 1M
Output Price	$0.330 / 1M	$0.600 / 1M
Total Input Tokens	40,038	42,750
Output Tokens	1,824	7,759
Reasoning Tokens	0	56,357
Response Time (avg)	5.91s	8.28s
Response Time (max)	57.10s	48.19s
Response Time (total)	124.05s	173.80s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#146 Gemma 4 26B A4B

none

Cost: $0.001
Time: 39.5s
Tokens: 790 tok

#115 KAT-Coder-Air V2.5

medium

Cost: $0.003
Time: 23.7s
Tokens: 4,924 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Category:

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	8.3	10.0	75.0%	0		1.28s	852	230	0
KAT-Coder-Air V2.5	8.7	8.0	91.7%	1		3.79s	672	549	4,194

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.7	7.2	22.2%	1		4.16s	7,736	476	0
KAT-Coder-Air V2.5	3.6	7.0	22.2%	1		23.37s	7,893	5,199	29,973

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.0	10.0	0.0%	0		30.53s	13,650	309	0
KAT-Coder-Air V2.5	10.0	10.0	100.0%	0		27.68s	15,132	372	4,816

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	10.0	10.0	100.0%	0		1.70s	8,352	285	0
KAT-Coder-Air V2.5	6.5	10.0	50.0%	0		3.65s	7,776	268	1,881

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.6	7.2	22.2%	1		2.49s	903	27	0
KAT-Coder-Air V2.5	3.0	10.0	0.0%	0		4.87s	771	487	4,454

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	4.0	10.0	0.0%	0		3.54s	576	85	0
KAT-Coder-Air V2.5	5.0	10.0	0.0%	0		9.81s	516	189	557

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	6.3	10.0	50.0%	0		690ms	795	75	0
KAT-Coder-Air V2.5	10.0	10.0	100.0%	0		1.50s	699	93	730

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	6.2	10.0	33.3%	0		744ms	828	114	0
KAT-Coder-Air V2.5	3.6	7.2	22.2%	1		1.87s	696	265	1,184

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	10.0	10.0	100.0%	0		57.10s	6,123	210	0
KAT-Coder-Air V2.5	10.0	10.0	100.0%	0		4.77s	8,391	317	906

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 26B A4B	3.0	10.0	0.0%	0		778ms	223	13	0
KAT-Coder-Air V2.5	3.0	10.0	0.0%	0		15.74s	204	20	7,662

Quick Compare

Switch Comparison Pair