ninjahawk

Has Opus 5.5 Been Nerfed? LiveNerf Update: Day 38 Over a month ago, we began a quest in trying to answer a simple question: Does model performance degrade silently without the user being aware of this change? We’re now 38 days into collecting data through LiveNerf’s methodology, so far, no degradation has been detected. We’re able to detect a 7% change or more as of now but I’m aiming to get that to under 1% for Fable 5.5, not a promise, just a goal. If you’re interested in tracking model performance of Opus 5.5, here’s the repo: ninjahawk/livenerf To do that, it would require 44x as much data to be collected, and we’re already consuming one pro plan per week for the data being collected. I’m trying to organize a group of people who’d be willing to join as contributors as we expand, I should be making a Google form within a few days which will be posted to the discord. It wouldn’t be technical at all, the idea is that for us to secure enough accuracy in v2 of LiveNerf we’ll need a much larger amount of compute to ensure data accuracy. I’m very thankful to everyone for everything, I would’ve never expected for us to already be at over 1,600 stars on Github. You’re all awesome. I’m hoping to bring a level of transparency that consumer plans and enterprise systems currently lack in AI.

You’re thinking of: ninjahawk/livenerf I’ll ask the same question as when it came out: if you don’t trust the provider how are you trust them not to look for and cheat on repeat benchmarks like this.

@Gonzalo_stba Pas sûr de ça... Source : ninjahawk/livenerf t.co

@_svs_ Doesn't look like so yet on nerf benchmark - ninjahawk/livenerf

Did they nerf Opus 5.5? Doesn’t look like it. LiveNerf day 29 update About a month ago we started LiveNerf as a way to track model capabilities over time. Since then it’s gotten bigger than I’d expected. We’ve accumulated around 1.5k stars on Github and now we’ve collected 29 consecutive days worth of data on the model capabilities of Opus 5.5. If anyone is interested in tracking model performance of Opus 5.5, here’s the repo: ninjahawk/livenerf As you can see from the 29 collected days of data, no nerf was detected. This weakens the idea that Anthropic nerfs models, or at least that they did not nerf Opus 5.5 within a range that LiveNerf can detect which would be by 6.6 points or more. We set out to produce unbiased data and determine if Anthropic nerfs their models, I believe we’ve done that. But the job is not finished. While Anthropic may not decrease the capabilities of their models over time, other companies might. I plan to expand LiveNerf to independently track models and APIs through this same scientific approach in the future. Currently we’re in talks for potential funding but nothing is definite yet. If funding is secured, I plan to make this much bigger. We should know in the next few days. I want to thank everyone for your community involvement so far, I’ll definitely be tracking Fable 5.5 when it’s released in the same way, and probably other models in due time. It’s meant the world to me seeing the community rally behind this in an effort to give independent transparency. I can’t thank you all enough. Give me some feedback in the comments, do you think that we could’ve done things differently? Maybe you had a problem with the way certain things were performed?

Timing An Anthropic AI Nerf on a Github project #technews #anthropic There's a conspiracy that Anthropic ships powerful AI models, then secretly makes them worse days later. Now someone's running a public experiment to prove it. Source: Hacker News Front Page Original: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [de-DE] GitHub Man plant einen anthropischen KI-Nerf #technews #anthropic Es kursiert die Verschwörungstheorie, dass Anthropic leistungsstarke KI-Modelle ausliefert und sie dann heimlich Tage später verschlechtert. Nun führt jemand ein öffentliches Experiment durch, um dies zu beweisen. Quelle: Hacker News Startseite Original: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [es-US] Un desarrollador de GitHub está cronometrando un nerf de la IA de Anthropic #noticiastecnológicas... Existe una teoría conspirativa que afirma que Anthropic lanza potentes modelos de IA y, días después, los empeora en secreto. Ahora, alguien está realizando un experimento público para demostrarlo. Fuente: Página principal de Hacker News Original: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [hi] GitHub पर मौजूद डेवलपर Anthropic के AI को धीमा करने की जांच कर रहा है #technews #anthropic एक साजिश चल रही है कि एंथ्रोपिक शक्तिशाली एआई मॉडल भेजता है, और फिर कुछ दिनों बाद चुपके से उन्हें खराब कर देता है। अब कोई इसे साबित करने के लिए सार्वजनिक प्रयोग कर रहा है। स्रोत: हैकर न्यूज़ का मुख्य पृष्ठ मूल: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [ja] GitHubのエンジニアがAnthropicのAI性能低下を検証中 #テクノロジーニュース #anthropic Anthropic社が強力なAIモデルを出荷した後、数日後に密かに性能を低下させているという陰謀論が囁かれている。そして今、誰かがそれを証明するための公開実験を行っている。 出典：Hacker News トップページ オリジナル：ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [ko] GitHub Man은 Anthropic AI Nerf를 타이밍하고 있습니다 #technews #anthropic 앤트로픽이 강력한 AI 모델을 출시한 후 며칠 뒤 몰래 성능을 저하시킨다는 음모론이 제기되었습니다. 이제 누군가가 이를 증명하기 위한 공개 실험을 진행하고 있습니다. 출처: Hacker News 메인 페이지 원본: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [ml] ഗിറ്റ്‌ഹബ് ഡെവലപ്പർ ആന്ത്രോപ്പിക് എഐ-യുടെ പ്രകടനം കുറയുന്നത് പരിശോധിക്കുന്നു #സാങ്കേതികവാർത്തകൾ #... ആന്ത്രോപിക് ശക്തമായ AI മോഡലുകൾ പുറത്തിറക്കുകയും പിന്നീട് ദിവസങ്ങൾക്കുള്ളിൽ അവയെ രഹസ്യമായി മോശമാക്കുകയും ചെയ്യുന്ന ഒരു ഗൂഢാലോചനയുണ്ട്. ഇപ്പോൾ അത് തെളിയിക്കാൻ ആരോ ഒരു പൊതു പരീക്ഷണം നടത്തുകയാണ്. ഉറവിടം: ഹാക്കർ ന്യൂസ് ഫ്രണ്ട് പേജ് ഒറിജിനൽ: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [nl-NL] GitHub Man is bezig met het timen van een antropische AI ​​Nerf #technews #anthropic Er gaat een complottheorie rond dat Anthropic krachtige AI-modellen verzendt en ze vervolgens dagen later stiekem slechter maakt. Nu voert iemand een openbaar experiment uit om dit te bewijzen. Bron: Hacker News Front Page Origineel: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [pl] GitHub Man planuje wprowadzenie antropicznego osłabienia sztucznej inteligencji #technews #anthropic Istnieje spisek, według którego Anthropic dostarcza potężne modele sztucznej inteligencji, a następnie potajemnie je pogarsza kilka dni później. Teraz ktoś przeprowadza publiczny eksperyment, aby to udowodnić. Źródło: Strona główna Hacker News Oryginał: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [ru] Человек с GitHub подбирает время для антропного ослабления ИИ #technews #anthropic Существует теория заговора, согласно которой Anthropic выпускает мощные модели ИИ, а затем тайно ухудшает их через несколько дней. Теперь кто-то проводит публичный эксперимент, чтобы это доказать. Источник: Hacker News Front Page Оригинал: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts [ta] GitHub டெவலப்பர் Anthropic AI-ன் தரத்தை குறைப்பதை கண்காணிக்கிறார் #தொழில்நுட்பசெய்திகள் #anthropic ஆந்த்ரோபிக் நிறுவனம் சக்திவாய்ந்த செயற்கை நுண்ணறிவு மாதிரிகளை வெளியிட்டு, சில நாட்களுக்குப் பிறகு அவற்றை இரகசியமாக மேலும் மோசமாக்குகிறது என்ற ஒரு சதி கோட்பாடு உள்ளது. இப்போது அதை நிரூபிக்க ஒருவர் ஒரு பொது சோதனையை நடத்தி வருகிறார். ஆதாரம்: ஹேக்கர் நியூஸ் முகப்புப் பக்கம் மூலம்: ninjahawk/livenerf #hindsite20xx #claude #anthropic #ainerf #shorts

@Agff_ @Coach_ChrisUK @Da7_Tech All of the temporal based benchmarks disagree. I suspect every time this happens we've just become acclimated to it's abilities. Your sense of judgement isn't objective, it's subjective. ninjahawk/livenerf

@ivanainai Someone made this so you can track if the nerfed it: ninjahawk/livenerf t.co

there are some efforts to track it: ninjahawk/livenerf

Are They Nerfing Opus 5.5? Livenerf Day 20 update Three weeks ago we began tracking model capability over time with LiveNerf. The goal was to prove from an unbiased set of data if Anthropic is nerfing their models after release. I’ve continued measuring it everyday since then and the results haven’t necessarily been what I think the community expected to see. If you’re interested in tracking the performance of Opus 5.5, here’s the repo: ninjahawk/livenerf If you take a look at the left 1/3, you’ll see that the variance was higher for the first 10 days when the baseline was being established. Then it appears like that in the following 11 days we’ve actually see a decreasing variance and potentially a regression to the mean. This means that if a nerf is happening or has happened, it has not happened within the data we’ve collected so far. This doesn’t necessarily mean that Anthropic isn’t decreasing capability in other ways, it also doesn’t mean that what we’re measuring establishes that day per day the result is always the same per 24-hour period. They could very well be throttling at specific times in the day and not during other periods. This brings me to where the community needs to get involved in a couple ways, we now do have some solid data that’s been accumulated over the course of three weeks, what I need from you guys now is to ask your own models or access for yourself the following question: “Did I experience a perceived decrease in capability at X time on Y day? If so, what was effected?” Testimonies won’t necessarily be more than anecdotal but I think it would be interesting if as a community we see that on certain days there’s perhaps different relations to the data that’s been collected so far. I want to thank everyone so much for the all the support over this past month, I certainly didn’t expect for us to surpass 1.5k stars in a few weeks and would’ve never expected people to donate to the project, I’d like to say as well that all the funds donated will go into further compute so we can scale further with model tracking. I do not plan on stopping with Opus 5.5 or with only tracking nerfs. In 10 days, I will return with the final verdict to lay down the hammer on tentatively proving or disproving model capabilities decreasing under the specific set of criteria LiveNerf collects data on. See you soon!