AI BENCHY
Advertise here
#189

Nemotron 3.5 Lightning

NVIDIA Release: 2026-08-11 Tested on: 2026-08-11 18:28 nvidia/nemotron-3.5-lightning::high
(high) (medium) (low) (none)

Summary

Nemotron 3.5 Lightning scores 5.6 on AI BENCHY and ranks #189. It has 9.9 reliability, a 48.5% pass rate, $0.167 total cost, and 57.99s average response time.

What makes Nemotron 3.5 Lightning unique: It stands out most in Combined, where it ranks #3, while General Intelligence is its weakest area at #10. It uses unusually many reasoning tokens, which can help explain its slower or more expensive runs.

Score

5.6

Consistency

6.0

Reliability

9.9

Total Cost (Current Price)

$0.167 ↕ New

Tested at: $0.000

Total Output Tokens

647,615

Total Input Tokens

118,007

Input Price

$0.100 / 1M

Output Price

$0.250 / 1M

Tests Correct

Wrong Tests: 17

Attempt pass rate: 48.5%

Flaky tests

11

Flaky tests had mixed outcomes across runs (at least one pass and one fail).

Response Time (avg)

57.99s

Response Time (max): 285.26s

Response Time (total): 1275.68s

Wrong answer: 11 Did not follow instructions: 4 Extra formatting: 1 Invalid tool call: 1

Price History

Historical pricing data for this model from OpenRouter.

Date Input Price Output Price
2026-08-11 17:51 $0.000 / 1M $0.000 / 1M
2026-08-11 18:26 $0.100 / 1M $0.250 / 1M

Charts

Choose the first model, then click a second model to open a side-by-side page.

Total Output Tokens

Score vs Total Output Tokens

Quick Compare

Nemotron 3.5 LightninghighFree AvailablevsDeepSeek V4 FlashnoneNemotron 3.5 LightninghighFree AvailablevsGLM 5.1noneNemotron 3.5 LightninghighFree AvailablevsKAT-Coder-Air V2.5mediumNemotron 3.5 LightninghighFree AvailablevsQwen3.6 27BnoneNemotron 3.5 LightninghighFree AvailablevsKAT-Coder-Air V2.5highNemotron 3.5 LightninghighFree AvailablevsMiMo-V2.5-PrononeNemotron 3.5 LightninghighFree AvailablevsQwen3.5-122B-A10BnoneNemotron 3.5 LightninghighFree AvailablevsKimi K2.5noneNemotron 3.5 LightninghighFree AvailablevsLing 3.0 FlashnoneFree AvailableNemotron 3.5 LightninghighFree AvailablevsGemma 4 26B A4BnoneFree Available

Category Breakdown

Category Score Consistency Tests Correct
Anti-AI Tricks 5.5 3.7
Coding 4.4 5.1
Combined 6.4 10.0
Data parsing and extraction 7.3 5.9
Domain specific 2.9 7.2
General Intelligence 3.7 9.5
Instructions following 8.5 6.8
Puzzle Solving 3.6 1.8
Tool Calling 10.0 10.0
Trivia 3.0 10.0

Compared models