AI BENCHY Compare

Mistral: Mistral Small 4 vs OpenAI: GPT-4o-mini

Last updated at: 2026-03-17

Metric	Mistral Small 4 Mistral Small 4 none Release: 2026-03-16	GPT-4o-mini GPT-4o-mini none Release: 2024-07-18

Metric	Mistral Small 4 Mistral Small 4 none Release: 2026-03-16	GPT-4o-mini GPT-4o-mini none Release: 2024-07-18
Rank	#61	#63
Score	5.3	5.0
Consistency	9.5	10.0
Cost per result	0.108	0.115
Total Cost	$0.006	$0.005
Tests Correct
Attempt pass rate	33.3%	23.5%
Flaky tests	1	0
Total Runs	51	51
Output Tokens	1,624	1,600
Reasoning Tokens	0	0
Response Time (avg)	629ms	1.94s
Response Time (max)	1.72s	7.58s
Response Time (total)	10.70s	19.44s

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	3.4	7.9	16.7%	1		395ms	182	0
GPT-4o-mini	4.8	10.0	25.0%	0		1.34s	186	0

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	3.0	10.0	0.0%	0		1.72s	496	0
GPT-4o-mini	3.0	10.0	0.0%	0		7.58s	568	0

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	10.0	10.0	100.0%	0		822ms	261	0
GPT-4o-mini	10.0	10.0	100.0%	0		1.27s	183	0

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	5.3	10.0	33.3%	0		367ms	28	0
GPT-4o-mini	3.0	10.0	0.0%	0		637ms	15	0

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	4.0	10.0	0.0%	0		729ms	205	0
GPT-4o-mini	4.0	10.0	0.0%	0		909ms	66	0

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	6.5	10.0	50.0%	0		380ms	69	0
GPT-4o-mini	4.8	10.0	0.0%	0		1.27s	69	0

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	3.1	9.9	0.0%	0		589ms	170	0
GPT-4o-mini	3.7	10.0	0.0%	0		1.30s	308	0

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Output Tokens	Reasoning Tokens
Mistral Small 4	10.0	10.0	100.0%	0		1.40s	213	0
GPT-4o-mini	10.0	10.0	100.0%	0		2.51s	205	0

Quick Compare

Switch Comparison Pair

GPT-4o-mininonevsQwen3 Coder Nextmedium GPT-4o-mininonevsGLM 4.7 Flashmedium Mistral Small 4nonevsQwen3 Coder Nextmedium Mistral Small 4mediumvsGPT-4o-mininone GPT-4o-mininonevsQwen3.5-9Bmedium Mistral Small 4nonevsGLM 4.7 Flashmedium MiniMax M2.5mediumFree AvailablevsMistral Small 4none Mistral Small 4nonevsgpt-oss-120bmediumFree Available Mistral Small 4nonevsQwen3.5-9Bmedium MiniMax M2.5mediumFree AvailablevsGPT-4o-mininone Mistral Small 4nonevsGrok 4.20 Multi-Agent Betamedium Mistral Small 4nonevsGPT-5 Nanomedium