AI BENCHY Compare

OpenAI: GPT-4o-mini vs Qwen: Qwen3.5-9B

Last updated at: 2026-03-12

Metric	GPT-4o-mini GPT-4o-mini none Release: 2024-07-18	Qwen3.5-9B Qwen3.5-9B medium Release: 2026-03-02

Metric	GPT-4o-mini GPT-4o-mini none Release: 2024-07-18	Qwen3.5-9B Qwen3.5-9B medium Release: 2026-03-02
Rank	#55	#66
Avg Score	4.0	2.6
Consistency	10.0	7.4
Cost per result	0.114	0.779
Total Cost	$0.005	$0.024
Tests Correct
Attempt pass rate	25.0%	35.4%
Flaky tests	0	5
Total Runs	48	48
Output Tokens	1,594	17,930
Reasoning Tokens	0	139,706
Response Time (avg)	2.07s	71.44s
Response Time (max)	7.58s	226.38s
Response Time (total)	18.60s	928.77s

Top Models by Score

Score vs Total Cost

Response Time (avg)

Avg Score vs Response Time (avg)

Total Output Tokens

Avg Score vs Total Output Tokens

Category Breakdown

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	4.0	10.0	33.3%	0		1.83s	180	0
Qwen3.5-9B	4.0	7.2	55.6%	1		31.54s	2,410	10,913

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	10.0	10.0	0.0%	0		7.58s	568	0
Qwen3.5-9B	10.0	10.0	0.0%	0		0ms	0	0

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	9.9	10.0	100.0%	0		1.27s	183	0
Qwen3.5-9B	5.0	5.6	33.3%	1		87.31s	1,383	32,113

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	10.0	10.0	0.0%	0		637ms	15	0
Qwen3.5-9B	10.0	7.2	22.2%	1		137.75s	11,549	48,475

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	3.0	10.0	0.0%	0		909ms	66	0
Qwen3.5-9B	10.0	1.6	33.3%	1		226.38s	0	30,695

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	4.5	10.0	0.0%	0		1.27s	69	0
Qwen3.5-9B	5.5	5.8	66.7%	1		17.15s	599	4,517

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	2.3	10.0	0.0%	0		1.30s	308	0
Qwen3.5-9B	10.0	10.0	0.0%	0		33.38s	1,545	11,844

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
GPT-4o-mini	10.0	10.0	100.0%	0		2.51s	205	0
Qwen3.5-9B	10.0	10.0	100.0%	0		4.31s	444	1,149

Quick Compare

Switch Comparison Pair

Qwen3.5-9BmediumvsMiMo-V2-Flashnone Qwen3.5-9BmediumvsGrok 4.1 Fastnone GPT-4o-mininonevsQwen3 Coder Nextmedium MiniMax M2.5mediumvsGPT-4o-mininone Mercury 2nonevsQwen3.5-9Bmedium Nemotron 3 Super 120b A12bnoneFree AvailablevsQwen3.5-9Bmedium GPT-4o-mininonevsGLM 4.7 Flashmedium GPT-4o-mininonevsGrok 4.20 Multi-Agent Betamedium Qwen3.5-9BmediumvsGLM 4.7 Flashnone Mercury 2mediumvsGPT-4o-mininone Kimi K2.5nonevsQwen3.5-9Bmedium GPT-4o-mininonevsQwen3.5-35B-A3Bmedium