Navigate
AI BENCHY
Advertise here

AI BENCHY Compare

Nemotron 3 Ultra 550b A55b vs Grok 4.20 Multi Agent Beta

Summary

Nemotron 3 Ultra 550b A55b vs Grok 4.20 Multi Agent Beta benchmark comparison: Nemotron 3 Ultra 550b A55b leads on average score with 7.5 vs 6.6. Nemotron 3 Ultra 550b A55b has the lower benchmark cost at $0.177 vs $5.599. Grok 4.20 Multi Agent Beta is faster at 9.69s vs 15.05s, with pass rates of 69.8% vs 59.3%.

Recommended model: Nemotron 3 Ultra 550b A55b - It has the best score here (7.5), while costing about 31.8x less than Grok 4.20 Multi Agent Beta.

Last updated at: 2026-06-10

Metric Nemotron 3 Ultra 550b A55b Nemotron 3 Ultra 550b A55b medium Release: 2026-06-04 Free Available Grok 4.20 Multi Agent Beta Grok 4.20 Multi Agent Beta medium Release: 2026-03-12
Score 7.5 6.6
Rank #42 #85
Reliability 9.7 N/A
Consistency 8.8 7.9
Tests Correct
Attempt pass rate 69.8% 59.3%
Flaky tests 3 5
Total Runs 63 52
Cost per result 0.000 62.923
Total Cost $0.177 $5.599
Input Price $0.500 / 1M $4.235 / 1M
Output Price $2.500 / 1M $4.235 / 1M
Total Input Tokens 46,813 721,952
Output Tokens 18,002 294,668
Reasoning Tokens 53,091 305,374
Response Time (avg) 15.05s 9.69s
Response Time (max) 43.93s 35.28s
Response Time (total) 316.09s 155.07s

Generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#42 Nemotron 3 Ultra 550b A55b

medium
No showcase result has been generated for this model yet.
Cost
$0.000
Time
-
Tokens
0 tok

#85 Grok 4.20 Multi Agent Beta

medium
Cost
$0.261
Time
123.4s
Tokens
199,344 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Anti-AI Tricks Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 10.0 10.0 100.0% 0 8.62s 780 835 1,485
Grok 4.20 Multi Agent Beta 6.9 5.8 75.0% 2 3.46s 90,925 33,706 33,077
Coding Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 8.4 7.4 88.9% 1 26.53s 7,686 2,854 17,725
Grok 4.20 Multi Agent Beta 10.0 10.0 100.0% 0 27.11s 13,212 86 13,141
Combined Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 9.8 10.0 100.0% 0 43.93s 17,574 1,040 3,590
Grok 4.20 Multi Agent Beta 3.0 10.0 0.0% 0 0ms 0 0 0
Data parsing and extraction Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 10.0 10.0 100.0% 0 5.68s 7,989 473 1,285
Grok 4.20 Multi Agent Beta 10.0 10.0 100.0% 0 5.54s 97,232 25,306 25,051
Domain specific Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 3.5 4.4 33.3% 2 24.90s 858 11,169 16,249
Grok 4.20 Multi Agent Beta 2.9 7.2 11.1% 1 24.67s 328,253 164,609 163,647
General Intelligence Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 3.7 9.5 0.0% 0 2.52s 360 70 235
Grok 4.20 Multi Agent Beta 5.8 2.8 66.7% 1 6.40s 41,387 15,848 15,746
Instructions following Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 9.8 10.0 100.0% 0 6.35s 765 182 1,243
Grok 4.20 Multi Agent Beta 9.8 10.0 100.0% 0 3.52s 43,923 19,752 19,617
Puzzle Solving Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 5.5 9.9 33.3% 0 3.54s 792 771 2,055
Grok 4.20 Multi Agent Beta 6.7 7.9 55.6% 1 5.19s 107,020 35,361 35,095
Tool Calling Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 10.0 10.0 100.0% 0 7.72s 9,781 304 984
Grok 4.20 Multi Agent Beta 3.0 10.0 0.0% 0 0ms 0 0 0
Trivia Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Nemotron 3 Ultra 550b A55b 3.0 10.0 0.0% 0 38.47s 228 304 8,240
Grok 4.20 Multi Agent Beta - - - - - - - - -

Quick Compare

Switch Comparison Pair

Gemini 3 Flash PreviewnonevsNemotron 3 Ultra 550b A55bmediumFree AvailableGemini 3.1 Flash Lite PreviewlowvsNemotron 3 Ultra 550b A55bmediumFree AvailableNemotron 3 Ultra 550b A55bmediumFree AvailablevsQwen3.7 MaxnoneGemini 3.5 FlashminimalvsNemotron 3 Ultra 550b A55bmediumFree AvailableNemotron 3 Ultra 550b A55bmediumFree AvailablevsStep 3.7 FlashlowDeepSeek V4 FlashhighvsNemotron 3 Ultra 550b A55bmediumFree AvailableGemini 3.1 Flash Lite PreviewnonevsNemotron 3 Ultra 550b A55bmediumFree AvailableGemini 3.1 Flash LitelowvsNemotron 3 Ultra 550b A55bmediumFree AvailableNemotron 3 Ultra 550b A55bmediumFree AvailablevsGPT-5.3 ChatnoneNemotron 3 Ultra 550b A55bmediumFree AvailablevsGPT-5.2 ChatnoneClaude Opus 4.8nonevsNemotron 3 Ultra 550b A55bmediumFree AvailableNemotron 3 Ultra 550b A55bmediumFree AvailablevsStep 3.7 Flashhigh