AI BENCHY Compare

Kwaipilot: KAT-Coder-Pro V2.5 vs OpenAI: GPT-5.5

Summary

GPT-5.5 (low) leads on average score with 9.3 vs 7.5. KAT-Coder-Pro V2.5 (medium) has the lower benchmark cost at $0.275 vs $0.907. GPT-5.5 (low) is faster at 9.76s vs 12.95s, with pass rates of 65.1% vs 85.7%.

Recommended modelGPT-5.5 (low)It has the strongest score in this comparison (9.3) and the best overall balance of cost and response time across all 2 models.

Last updated at: 2026-07-14

Metric	KAT-Coder-Pro V2.5 KAT-Coder-Pro V2.5 medium Release: 2026-07-14	GPT-5.5 GPT-5.5 low Release: 2026-04-24

Metric	KAT-Coder-Pro V2.5 KAT-Coder-Pro V2.5 medium Release: 2026-07-14	GPT-5.5 GPT-5.5 low Release: 2026-04-24
Score	7.5	9.3
Rank	#60	#7
Reliability	10.0	10.0
Consistency	7.7	10.0
Tests Correct
Attempt pass rate	65.1%	85.7%
Flaky tests	6	0
Total Runs	63	63
Cost per result	2.498	5.035
Total Cost	$0.275	$0.907
Input Price	$0.740 / 1M	$5.000 / 1M
Output Price	$2.960 / 1M	$30.000 / 1M
Total Input Tokens	42,750	34,209
Output Tokens	2,098	2,046
Reasoning Tokens	80,027	22,460
Response Time (avg)	12.95s	9.76s
Response Time (max)	81.45s	56.19s
Response Time (total)	271.92s	204.92s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#60 KAT-Coder-Pro V2.5

medium

Cost: $0.009
Time: 25.6s
Tokens: 2,939 tok

#7 GPT-5.5

low

Cost: $0.068
Time: 37.0s
Tokens: 2,339 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Category:

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	8.2	7.9	83.3%	1		3.90s	672	208	3,868
GPT-5.5	10.0	10.0	100.0%	0		4.41s	606	238	1,020

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	7.8	10.0	66.7%	0		33.10s	7,893	416	33,658
GPT-5.5	10.0	10.0	100.0%	0		15.04s	7,302	423	6,402

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		14.50s	15,132	363	3,367
GPT-5.5	10.0	10.0	100.0%	0		9.56s	11,019	303	717

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	7.3	5.9	83.3%	1		4.70s	7,776	275	2,122
GPT-5.5	10.0	10.0	100.0%	0		3.28s	7,140	228	157

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	2.9	7.2	11.1%	1		29.03s	771	36	29,584
GPT-5.5	5.3	10.0	33.3%	0		28.05s	723	69	11,609

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	4.7	3.1	33.3%	1		2.35s	516	92	381
GPT-5.5	10.0	10.0	100.0%	0		5.17s	477	133	245

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	9.9	10.0	100.0%	0		2.58s	699	83	971
GPT-5.5	9.9	10.0	100.0%	0		3.74s	660	93	415

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	5.9	4.5	66.7%	2		3.20s	696	325	1,867
GPT-5.5	10.0	10.0	100.0%	0		4.74s	642	279	954

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	10.0	10.0	100.0%	0		19.02s	8,391	282	1,010
GPT-5.5	10.0	10.0	100.0%	0		4.96s	5,445	250	101

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
KAT-Coder-Pro V2.5	3.0	10.0	0.0%	0		9.87s	204	18	3,199
GPT-5.5	3.0	10.0	0.0%	0		10.06s	195	30	840

Quick Compare

Switch Comparison Pair