AI BENCHY
Benchmark Paddhati
He page amchi benchmarking approach high-level var samjavte. Test integrity tikvnyasathi amhi exact prompts ani grading internals private thevto.
Tests
Prashna bahutek random padhatine, vegveglya tasks ani domains madhun nivadle jatat. Statistical drushtine baghitla tar, ek stronger model-ne average madhye weaker model peksha random, non-cherry-picked task var better karayla hava. Maza background competitive programming madhye aahe, mhanun tests ani edge cases vicharane mala naturally jamte.
He kontehi standardized "IQ" value nahi. Score la kahi unit nahi; to fakta ek arbitrary value aahe jo dakhavto ki model sampurn test suite var kiti changla karto (correct answers + consistency). Mi models cherry-pick kart nahi ani kontyahi model la suit honyasathi tests modify pan kart nahi. Mala nava test sujla ki mi to add karto, sagle models punha test karto, ani scores recalculate karto.
Prashna sahasa simple ideas varun yetat, jase: "Models na X, Y, kiwa Z karayla sangitla tar te changle kartat ka?" Udaaharanarth: "Respond with the two equal natural numbers, a and b, that when added together have the total = 2. Respond in this exact format: a,b". Kahi AI chukicha answer deu shaktat, jase "2,2". Kahi numbers equal asnyachi requirement follow karat nahi, jase "0,2". Kahi output format ignore kartat, jase "The answer is a = 1 and b = 1". Ani kahi saral barobar "1,1" mhanun answer detat.
Kahi tests ya peksha adhik complex astat, pan gist lakshat yete. He kontyahi specific model la favor kart nahi, ani he prashna manasansathi sarvasadharanpane khup sope astat. Claude ne "**1**, **1**" sarkha markdown highlighting sah output dila, ani bahutek itar models required format barobar follow kart astil, tar tyat maza dosh nahi.
He Kase Kaam Karte (High Level)
- Private tests: Amhi exact test content, prompts, kiwa full grading details publish kart nahi.
- Repeated runs: Pratyek model anek vela chalavla jato jene karun result stability disel, fakt ekda milalela lucky attempt nahi.
- Reasoning modes: Jithe support aahe, tithe models na multiple reasoning configurations madhye evaluate kele jate.
- OpenRouter execution: Benchmark requests OpenRouter madhun run hotat.
- Real-world reliability: Timeout, downtime, ani API errors failed attempts mhanun count hotat.
- Fast coverage with evolving suite: Amcha suite lahan aslyamule amhi nave models lavkar test karto ani tests satat add kiwa remove karto.
- Generic intelligence signal: Score ekach category purta maryadit nahi. To eka practical prashnacha indicator aahe: tumhi AI la kahi hi vicharle tar yogya uttar milnyachi shakyaata kiti?
Transparency sathi amhi broad methodology share karto, pan sensitive benchmark details private thevto.