Ember-1 (low) vs Gemma 4 31B (medium)

Compared models

Recommended model Gemma 4 31B (medium)

Its score stays close to the best score here (6.9 vs 7.0), while costing about 12.7x less than Ember-1 (low).

Metric	Ember-1 Ember-1 low Release: 2026-09-28	Gemma 4 31B Gemma 4 31B medium Release: 2026-04-02 Free Available

Metric	Ember-1 Ember-1 low Release: 2026-09-28	Gemma 4 31B Gemma 4 31B medium Release: 2026-04-02 Free Available
Score	7.0	6.9
Rank	#157	#164
Reliability	8.8	10.0
Consistency	8.2	8.7
Attempts	69/69	69/69
Tests Correct
Attempt pass rate	65.2%	66.7%
Flaky tests	5	3
Total Runs	69	69
Cost per result	11.441	1.195
Total Cost	$1.488	$0.117
Input Price	$3.000 / 1M	$0.090 / 1M
Output Price	$15.000 / 1M	$0.340 / 1M
Total Input Tokens	211,908	251,164
Output Tokens	14,158	36,112
Reasoning Tokens	42,608	241,203
Response Time (avg)	34.90s	78.76s
Response Time (max)	121.29s	437.40s
Response Time (total)	802.79s	1653.96s
Parameters	~2.8T total (~104B active)	30.7B
Availability	Closed	Open source

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

low

medium

Category:

Agentic	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	4.7	3.1	33.3%	1		121.29s	178,897	9,378	9,921
Gemma 4 31B	10.0	10.0	100.0%	0		82.13s	156,195	1,713	8,181

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	8.2	7.9	83.3%	1		22.18s	1,402	571	613
Gemma 4 31B	10.0	10.0	100.0%	0		12.89s	816	962	2,046

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	10.0	10.0	100.0%	0		37.91s	8,097	2,518	13,142
Gemma 4 31B	4.3	5.8	22.2%	1		219.76s	5,568	11,098	33,212

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	6.5	10.0	50.0%	0		51.45s	12,828	381	689
Gemma 4 31B	2.9	5.8	16.7%	1		433.11s	77,035	12,112	157,552

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	7.3	5.9	83.3%	1		34.82s	6,076	223	778
Gemma 4 31B	10.0	10.0	100.0%	0		21.11s	8,334	1,822	2,951

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	2.9	7.2	11.1%	1		56.17s	1,154	234	12,476
Gemma 4 31B	5.3	7.2	44.4%	1		59.92s	853	4,280	18,729

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	10.0	10.0	100.0%	0		5.67s	744	139	462
Gemma 4 31B	10.0	10.0	100.0%	0		9.57s	567	105	888

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	6.2	5.8	66.7%	1		32.73s	781	121	466
Gemma 4 31B	10.0	10.0	100.0%	0		12.76s	777	533	2,035

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	10.0	10.0	100.0%	0		3.27s	1,452	393	595
Gemma 4 31B	9.9	10.0	100.0%	0		26.91s	801	1,795	5,595

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	3.0	10.0	0.0%	0		26.43s	0	0	0
Gemma 4 31B	3.0	10.0	0.0%	0		0ms	0	0	0

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	3.0	10.0	0.0%	0		30.60s	477	200	3,466
Gemma 4 31B	3.0	10.0	0.0%	0		90.14s	218	1,692	10,014

Switch Comparison Pair