AI BENCHY Compare

Google: Gemma 4 31B vs OpenAI: GPT-5 Nano

Last updated at: 2026-06-04

Metric	Gemma 4 31B Gemma 4 31B none Release: 2026-04-02 Free Available	GPT-5 Nano GPT-5 Nano medium Release: 2025-08-07

Metric	Gemma 4 31B Gemma 4 31B none Release: 2026-04-02 Free Available	GPT-5 Nano GPT-5 Nano medium Release: 2025-08-07
Score	6.5	6.3
Rank	#85	#94
Reliability	10.0	10.0
Consistency	10.0	7.3
Tests Correct
Attempt pass rate	47.6%	57.1%
Flaky tests	0	7
Total Runs	63	63
Cost per result	0.034	0.897
Total Cost	$0.004	$0.081
Input Price	$0.120 / 1M	$0.050 / 1M
Output Price	$0.370 / 1M	$0.400 / 1M
Total Input Tokens	20,911	34,108
Output Tokens	1,407	5,464
Reasoning Tokens	0	192,064
Response Time (avg)	4.05s	42.51s
Response Time (max)	26.13s	204.02s
Response Time (total)	76.87s	595.09s

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	6.5	10.0	50.0%	0		1.85s	852	45	0
GPT-5 Nano	6.5	7.9	58.3%	1		25.50s	606	1,221	21,184

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	5.5	10.0	33.3%	0		11.19s	8,381	735	0
GPT-5 Nano	7.0	7.7	55.6%	1		41.62s	7,305	740	41,152

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	3.0	10.0	0.0%	0		0ms	0	0	0
GPT-5 Nano	10.0	10.0	100.0%	0		65.96s	11,019	578	17,984

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	10.0	10.0	100.0%	0		2.25s	8,352	285	0
GPT-5 Nano	3.7	1.7	50.0%	2		21.42s	7,140	453	10,560

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	7.7	10.0	66.7%	0		3.22s	903	27	0
GPT-5 Nano	5.2	4.4	55.6%	2		204.02s	619	237	64,448

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	10.0	10.0	100.0%	0		2.09s	576	117	0
GPT-5 Nano	4.1	10.0	0.0%	0		17.51s	477	202	4,608

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	6.5	10.0	50.0%	0		2.84s	795	78	0
GPT-5 Nano	9.8	10.0	100.0%	0		15.64s	660	312	4,736

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	6.5	10.0	33.3%	0		4.23s	828	108	0
GPT-5 Nano	5.3	7.2	44.4%	1		20.63s	642	929	14,272

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	3.0	10.0	0.0%	0		0ms	0	0	0
GPT-5 Nano	10.0	10.0	100.0%	0		33.30s	5,445	558	6,976

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Gemma 4 31B	3.0	10.0	0.0%	0		1.25s	224	12	0
GPT-5 Nano	3.0	10.0	0.0%	0		20.13s	195	234	6,144

Quick Compare

Switch Comparison Pair

GPT-5 NanomediumvsQwen3.5 Plus 2026-02-15none Ring-2.6-1TnonevsGPT-5 Nanomedium Gemini 2.5 FlashnonevsGPT-5 Nanomedium Gemini 3.1 Flash LitenonevsGPT-5 Nanomedium Gemma 4 31BnoneFree AvailablevsMercury 2medium GPT-5 NanomediumvsQwen3.7 Plusnone Gemini 3.1 Flash LiteminimalvsGPT-5 Nanomedium GPT-5 NanomediumvsGLM 5none Gemma 4 31BnoneFree AvailablevsMimo V2 Omnimedium Gemma 4 31BnoneFree AvailablevsQwen3.6 27Bmedium GPT-5 NanomediumvsMimo V2 Omninone Gemma 4 31BnoneFree AvailablevsKimi K2.5medium