Navigate
AI BENCHY
Advertise here

AI BENCHY Compare

Google: Gemma 4 31B vs Mistral: Mistral Small 4

Summary

Gemma 4 31B vs Mistral Small 4 benchmark comparison: Gemma 4 31B leads on average score with 6.3 vs 5.1. Mistral Small 4 has the lower benchmark cost at $0.007 vs $0.033. Mistral Small 4 is faster at 630ms vs 56.55s, with pass rates of 69.8% vs 27.0%.

Recommended model: Mistral Small 4 - It offers the best overall trade-off: a competitive score (5.1), lower cost than Gemma 4 31B, and balanced response time.

Last updated at: 2026-06-12

Metric Gemma 4 31B Gemma 4 31B medium Release: 2026-04-02 Free Available Mistral Small 4 Mistral Small 4 none Release: 2026-03-16
Score 6.3 5.1
Rank #87 #136
Reliability 10.0 10.0
Consistency 9.4 9.5
Tests Correct
Attempt pass rate 69.8% 27.0%
Flaky tests 1 1
Total Runs 63 63
Cost per result 0.257 0.139
Total Cost $0.033 $0.007
Input Price $0.120 / 1M $0.150 / 1M
Output Price $0.350 / 1M $0.600 / 1M
Total Input Tokens 17,957 37,309
Output Tokens 22,356 2,201
Reasoning Tokens 65,726 0
Response Time (avg) 56.55s 630ms
Response Time (max) 437.40s 1.72s
Response Time (total) 1074.41s 13.22s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#87 Gemma 4 31B

medium
Cost
$0.002
Time
45.7s
Tokens
2,696 tok

#136 Mistral Small 4

none
Cost
$0.002
Time
10.4s
Tokens
2,370 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Anti-AI Tricks Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 10.0 10.0 100.0% 0 12.89s 816 962 2,046
Mistral Small 4 3.4 7.9 16.7% 1 395ms 708 182 0
Coding Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 4.3 5.8 22.2% 1 219.76s 5,568 11,098 33,212
Mistral Small 4 3.7 9.7 0.0% 0 901ms 7,636 619 0
Combined Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 3.0 10.0 0.0% 0 0ms 0 0 0
Mistral Small 4 3.0 10.0 0.0% 0 1.72s 11,640 496 0
Data parsing and extraction Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 10.0 10.0 100.0% 0 21.11s 8,334 1,822 2,951
Mistral Small 4 10.0 10.0 100.0% 0 822ms 7,914 261 0
Domain specific Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 7.7 10.0 66.7% 0 38.48s 876 4,349 8,985
Mistral Small 4 5.3 10.0 33.3% 0 367ms 798 28 0
General Intelligence Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 10.0 10.0 100.0% 0 9.57s 567 105 888
Mistral Small 4 4.0 10.0 0.0% 0 729ms 519 205 0
Instructions following Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 10.0 10.0 100.0% 0 12.76s 777 533 2,035
Mistral Small 4 6.5 10.0 50.0% 0 380ms 729 69 0
Puzzle Solving Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 9.9 10.0 100.0% 0 26.91s 801 1,795 5,595
Mistral Small 4 3.1 9.9 0.0% 0 399ms 735 111 0
Tool Calling Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 3.0 10.0 0.0% 0 0ms 0 0 0
Mistral Small 4 10.0 10.0 100.0% 0 1.40s 6,420 213 0
Trivia Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Gemma 4 31B 3.0 10.0 0.0% 0 90.14s 218 1,692 10,014
Mistral Small 4 3.0 10.0 0.0% 0 397ms 210 17 0

Quick Compare

Switch Comparison Pair

Gemma 4 31BmediumFree AvailablevsGPT-5.5noneMiniMax M2.7mediumvsMistral Small 4noneDeepSeek V4 ProhighvsGemma 4 31BmediumFree AvailableDeepSeek V4 PrononevsGemma 4 31BmediumFree AvailableSeed-2.0-LitenonevsGemma 4 31BmediumFree AvailableGemma 4 31BmediumFree AvailablevsQwen3.5-FlashnoneCobuddymediumvsMistral Small 4noneGemma 4 31BmediumFree AvailablevsNemotron 3 Ultra 550b A55bnoneFree AvailableGemma 4 31BmediumFree AvailablevsQwen3.6 Max PreviewnoneGemma 4 31BmediumFree AvailablevsGLM 5noneGemma 4 31BmediumFree AvailablevsQwen3.6 FlashnoneGemma 4 31BmediumFree AvailablevsQwen3.5-35B-A3Bnone