Ember-1 (high) vs Gemma 4 31B (medium)

Compared models

Recommended model Gemma 4 31B (medium)

It has the best score here (6.9), while costing about 20.6x less than Ember-1 (high).

Metric	Ember-1 Ember-1 high Release: 2026-09-28	Gemma 4 31B Gemma 4 31B medium Release: 2026-04-02 Free Available

Metric	Ember-1 Ember-1 high Release: 2026-09-28	Gemma 4 31B Gemma 4 31B medium Release: 2026-04-02 Free Available
Score	6.9	6.9
Rank	#164	#165
Reliability	8.6	10.0
Consistency	7.2	8.7
Attempts	69/69	69/69
Tests Correct
Attempt pass rate	71.0%	66.7%
Flaky tests	8	3
Total Runs	69	69
Cost per result	20.041	1.195
Total Cost	$2.405	$0.117
Input Price	$3.000 / 1M	$0.090 / 1M
Output Price	$15.000 / 1M	$0.340 / 1M
Total Input Tokens	195,489	251,164
Output Tokens	12,475	36,112
Reasoning Tokens	108,748	241,203
Response Time (avg)	58.14s	78.76s
Response Time (max)	255.06s	437.40s
Response Time (total)	1337.19s	1653.96s
Parameters	~2.8T total (~104B active)	30.7B
Availability	Closed	Open source

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

high

medium

Category:

Agentic	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	6.1	3.1	66.7%	1		88.31s	146,912	5,130	5,968
Gemma 4 31B	10.0	10.0	100.0%	0		82.13s	156,195	1,713	8,181

Anti-AI Tricks	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	8.7	7.9	91.7%	1		24.05s	1,544	886	992
Gemma 4 31B	10.0	10.0	100.0%	0		12.89s	816	962	2,046

Coding	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	8.2	7.2	88.9%	1		72.23s	7,144	1,281	23,093
Gemma 4 31B	4.3	5.8	22.2%	1		219.76s	5,568	11,098	33,212

Combined	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	6.4	5.8	66.7%	1		79.89s	28,994	1,667	6,614
Gemma 4 31B	2.9	5.8	16.7%	1		433.11s	77,035	12,112	157,552

Data parsing and extraction	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	7.3	5.9	83.3%	1		36.10s	6,076	230	1,228
Gemma 4 31B	10.0	10.0	100.0%	0		21.11s	8,334	1,822	2,951

Domain specific	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	2.9	4.4	22.2%	2		175.49s	1,154	1,023	61,582
Gemma 4 31B	5.3	7.2	44.4%	1		59.92s	853	4,280	18,729

General Intelligence	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	10.0	10.0	100.0%	0		4.49s	744	194	304
Gemma 4 31B	10.0	10.0	100.0%	0		9.57s	567	105	888

Instructions following	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	7.1	5.8	83.3%	1		33.38s	992	162	784
Gemma 4 31B	10.0	10.0	100.0%	0		12.76s	777	533	2,035

Puzzle Solving	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	10.0	10.0	100.0%	0		4.84s	1,452	390	848
Gemma 4 31B	9.9	10.0	100.0%	0		26.91s	801	1,795	5,595

Tool Calling	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	3.0	10.0	0.0%	0		26.45s	0	0	0
Gemma 4 31B	3.0	10.0	0.0%	0		0ms	0	0	0

Trivia	Score	Consistency	Attempt pass rate	Flaky tests	Tests Correct	Response Time (avg)	Input Tokens	Output Tokens	Reasoning Tokens
Ember-1	3.0	10.0	0.0%	0		65.34s	477	1,512	7,335
Gemma 4 31B	3.0	10.0	0.0%	0		90.14s	218	1,692	10,014

Switch Comparison Pair