AI BENCHY Compare

Kwaipilot: KAT-Coder-Pro V2.5 vs OpenAI: GPT-5.4 Mini

Summary

The average score is effectively tied at 8.0 vs 8.0. KAT-Coder-Pro V2.5 (low) has the lower benchmark cost at $0.220 vs $0.526. KAT-Coder-Pro V2.5 (low) is faster at 10.44s vs 22.34s, with pass rates of 71.4% vs 73.0%.

Recommended modelKAT-Coder-Pro V2.5 (low)It has the best score here (8.0), while costing about 2.4x less than GPT-5.4 Mini (medium).

Last updated at: 2026-07-14

Metric	KAT-Coder-Pro V2.5 KAT-Coder-Pro V2.5 low Release: 2026-07-14	GPT-5.4 Mini GPT-5.4 Mini medium Release: 2026-03-17

Metric	KAT-Coder-Pro V2.5 KAT-Coder-Pro V2.5 low Release: 2026-07-14	GPT-5.4 Mini GPT-5.4 Mini medium Release: 2026-03-17
Score	8.0	8.0
Rank	#35	#33
Reliability	9.9	10.0
Consistency	7.3	8.0
Tests Correct
Attempt pass rate	71.4%	73.0%
Flaky tests	7	5
Total Runs	63	63
Cost per result	1.991	4.381
Total Cost	$0.220	$0.526
Input Price	$0.740 / 1M	$0.750 / 1M
Output Price	$2.960 / 1M	$4.500 / 1M
Total Input Tokens	42,750	34,116
Output Tokens	2,167	2,181
Reasoning Tokens	61,133	108,937
Response Time (avg)	10.44s	22.34s
Response Time (max)	51.34s	138.75s
Response Time (total)	219.17s	469.20s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#35 KAT-Coder-Pro V2.5

low

Cost: $0.016
Time: 47.6s
Tokens: 5,182 tok

#33 GPT-5.4 Mini

medium

Cost: $0.056
Time: 95.5s
Tokens: 12,464 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Category:

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	6.9	5.8	75.0%	2		4.20s	672	277	4,228
GPT-5.4 Mini	8.6	7.9	91.7%	1		4.05s	606	296	2,876

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	7.8	10.0	66.7%	0		24.87s	7,893	402	24,945
GPT-5.4 Mini	8.4	7.4	88.9%	1		57.87s	7,305	467	40,902

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		14.59s	15,132	394	3,391
GPT-5.4 Mini	10.0	10.0	100.0%	0		17.81s	11,019	317	4,317

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		4.19s	7,776	288	1,917
GPT-5.4 Mini	10.0	10.0	100.0%	0		2.43s	7,140	234	650

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	4.1	4.4	44.5%	2		17.52s	771	35	16,877
GPT-5.4 Mini	4.1	4.4	44.5%	2		65.31s	619	60	43,286

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	4.1	2.7	33.3%	1		2.32s	516	99	412
GPT-5.4 Mini	4.5	10.0	0.0%	0		3.72s	477	150	510

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		2.53s	699	83	1,008
GPT-5.4 Mini	9.8	10.0	100.0%	0		2.13s	660	96	1,185

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	6.4	4.4	77.8%	2		3.11s	696	289	1,820
GPT-5.4 Mini	7.8	10.0	66.7%	0		4.37s	642	278	2,443

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		18.45s	8,391	282	834
GPT-5.4 Mini	4.7	1.6	66.7%	1		9.62s	5,453	251	2,594

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	3.0	10.0	0.0%	0		17.06s	204	18	5,701
GPT-5.4 Mini	3.0	10.0	0.0%	0		30.10s	195	32	10,174

Quick Compare

Switch Comparison Pair