AI BENCHY Category Failures
Coding: Wrong answer
Coding
Wrong answer
See which AI models are most likely to hit Wrong answer on Coding, so you can spot weak points faster. Sort by: Total Cost ↓.
Failure Reasons
134/134
Filter models
No models match the current search and filters.
| Rank | Model | Company | Wrong answer Count | Category Score | Total Cost | Tests Correct | Response Time (avg) |
|---|---|---|---|---|---|---|---|
| #31 | Nemotron 3 Ultra 550b A55b medium | NVIDIA | 1 | 8.4 | $0.158 | 2/3 | 26.5s |
| #108 | GPT-5.6 Luna low | OpenAI | 2 | 5.5 | $0.141 | 1/3 | 4.61s |
| #117 | GPT-5.6 Terra none | OpenAI | 2 | 5.5 | $0.130 | 1/3 | 1.00s |
| #129 | GPT-5.4 none | OpenAI | 2 | 5.5 | $0.122 | 1/3 | 1.62s |
| #65 | Gemini 3 Flash Preview low | 2 | 5.8 | $0.111 | 1/3 | 6.00s | |
| #85 | Gemini 3.5 Flash minimal | 1 | 5.6 | $0.108 | 1/3 | 2.75s | |
| #59 | GPT-5.4 Nano medium | OpenAI | 2 | 6.1 | $0.107 | 1/3 | 19.1s |
| #169 | Grok 4.20 Beta none | X AI | 1 | 1.8 | $0.087 | 0/1 | 1.14s |
| #93 | GPT-5 Nano medium | OpenAI | 2 | 7.0 | $0.081 | 1/3 | 41.6s |
| #84 | Qwen3.5-Flash medium | Qwen | 2 | 3.7 | $0.080 | 0/3 | 58.9s |
| #128 | Kimi K2.6 none | Moonshot AI | 1 | 5.5 | $0.078 | 1/3 | 82.6s |
| #116 | Qwen3.6 Max Preview none | Qwen | 3 | 3.8 | $0.075 | 0/3 | 3.12s |
| #43 | Gemini 3.1 Flash Lite medium | 2 | 5.5 | $0.071 | 1/3 | 3.81s | |
| #41 | Gemini 3.1 Flash Lite Preview medium | 2 | 5.5 | $0.068 | 1/3 | 4.09s | |
| #153 | Mistral Small 4 medium | Mistral | 3 | 4.4 | $0.068 | 0/3 | 40.0s |