fp4Attn

CroissAttn reproduces for stable NVFP4 training: github.com/alrojo/fp4Attn/blob/main/Croissattn_FP… Slight ppl degradation: 3.4765 vs 3.4585 for BF16 for 1B at 254M model. Trains 6% slower. So why use it? Because the final model ends up being 4-bit precision with the best possible ppl.

Incredible momentum in betterment we are seeing. alrojo/fp4Attn

@askalphaxiv This is THE fix for running attention backward pass in FP4 training. We can finally do end to end FP4 training! alrojo/fp4Attn

@petrenko_ai This line of work is incredibly interesting. We are very interested in quantizing RL rollouts. We use the same method here to run the full attention backward pass in FP4. alrojo/fp4Attn I believe exact replay might be the fix here.

🥐 Paper: alrojo/fp4Attn If you work on attention kernels, low-precision training, or RL rollouts, come talk to us!