@heyyalexwang not with that specific nGPT recipe, but yes, I do think there is something to get out of nGPT, all the recent stream of work around Hyperball/MuonMD are essentially nGPT (it's being tested live in the Marin hero run) marin#8435
1 like · 16 views
We’re Marin, the details are pretty much always available!
marin#9960
This time we’re deploying Triton router top-k and short conv, leaner ragged a2a transport, fused expert backward, and XLA scheduling fixes
26 likes · 3 reposts · 1,142 views
marin's pre-switch one-rack test kept tokens/step fixed: 4k and 8k measured 301.9k and 297.0k tokens/s, with mfu at 28.25% and 28.47%
longer attention changes the work counted by mfu, so read it alongside throughput
marin#9615
10 views
If you would like to learn more, everything is on GitHub (marin-community/marin), hang out with us in Discord (discord.com), where we have active discussions. We also have biweekly meetings where we deep dive into various parts of the stack, open to anyone in the Marin community.
28 likes · 2 reposts · 1,820 views
We then noticed a big increase in our gated attention norm, which was off trend compared to our scaling ladder. Loss was going down and on track, but just to be safe, we added a touch of weight decay mid-run, which dampened the growth: marin#8818 t.co
30 likes · 1 repost · 1,901 views
Mentions
@heyyalexwang not with that specific nGPT recipe, but yes, I do think there is something to get out of nGPT, all the recent stream of work around Hyperball/MuonMD are essentially nGPT (it's being tested live in the Marin hero run) marin#8435
We’re Marin, the details are pretty much always available! marin#9960 This time we’re deploying Triton router top-k and short conv, leaner ragged a2a transport, fused expert backward, and XLA scheduling fixes
marin's pre-switch one-rack test kept tokens/step fixed: 4k and 8k measured 301.9k and 297.0k tokens/s, with mfu at 28.25% and 28.47% longer attention changes the work counted by mfu, so read it alongside throughput marin#9615
If you would like to learn more, everything is on GitHub (marin-community/marin), hang out with us in Discord (discord.com), where we have active discussions. We also have biweekly meetings where we deep dive into various parts of the stack, open to anyone in the Marin community.
We then noticed a big increase in our gated attention norm, which was off trend compared to our scaling ladder. Loss was going down and on track, but just to be safe, we added a touch of weight decay mid-run, which dampened the growth: marin#8818 t.co