Navigate
Advertise here

Mercury 2.5 Preview (default) vs Nemotron 3.5 Lightning (medium)

The average score is effectively tied at 4.7 vs 4.7. Mercury 2.5 Preview (default) has the lower benchmark cost at $0.023 vs $0.195. Mercury 2.5 Preview (default) is faster at 7.05s vs 81.93s, with pass rates of 29.0% vs 47.8%.

Last updated at: 2026-11-16

Compared models

Rank
#419
Total Output Tokens
86,367
Response Time (avg)
7.05s
Total Cost
$0.023
Rank
#420
Total Output Tokens
732,696
Response Time (avg)
81.93s
Total Cost
$0.195
Recommended model Mercury 2.5 Preview (default)

It has the best score here (4.7), while costing about 8.5x less than Nemotron 3.5 Lightning (medium).

Detailed comparison

Metric Mercury 2.5 Preview Mercury 2.5 Preview default Release: 2026-09-02 Nemotron 3.5 Lightning Nemotron 3.5 Lightning medium Release: 2026-08-11 Free Available
Score 4.7 4.7
Rank #419 #420
Reliability 9.9 9.8
Consistency 8.3 5.8
Attempts 69/69 69/69
Tests Correct
Attempt pass rate 29.0% 47.8%
Flaky tests 5 12
Total Runs 69 69
Cost per result 0.559 0.000
Total Cost $0.023 $0.195
Input Price $0.040 / 1M $0.060 / 1M
Output Price $0.150 / 1M $0.160 / 1M
Cache Read Price N/A $0.030 / 1M
Cache Write Price N/A N/A
Total Input Tokens 250,541 195,780
Output Tokens 86,367 167,746
Reasoning Tokens 0 598,384
Response Time (avg) 7.05s 81.93s
Response Time (max) 84.96s 437.11s
Response Time (total) 162.25s 1884.31s
Parameters ~100B 30B total (3B active)
Availability Closed Weights available

Cache prices apply to input tokens. Reads reuse cached prompts; writes store them and can cost extra. Output tokens use the output price.

Model generation showcase

Hamster playing table tennis

Prompt: Create a detailed SVG illustration of a hamster playing table tennis.

#419 Mercury 2.5 Preview

default
Invalid SVG
Cost
$0.001
Time
3.3s
Tokens
3,122 tok

#420 Nemotron 3.5 Lightning

medium
Cost
$0.000
Time
130.5s
Tokens
6,750 tok

Top Models by Score

Score vs Total Cost

Response Time (avg)

Score vs Response Time (avg)

Total Output Tokens

Score vs Total Output Tokens

Category Breakdown

Coding Score Consistency Attempt pass rate Flaky tests Tests Correct Response Time (avg) Input Tokens Output Tokens Reasoning Tokens
Mercury 2.5 Preview 4.7 7.9 22.2% 1 459ms 7,850 650 0
Nemotron 3.5 Lightning 4.4 5.1 33.3% 2 285.47s 7,623 78,065 297,359

Switch Comparison Pair