toshllm

RX580(Polaris AMD) on a MAC with near Vulkan performance! This is my fork of ToshLLM, who brought modern AMD's back to metal acceleration for llama.cpp , But his fork was about 17toks on Ornith 9b when I started my project vs Linux's Vulkan @ 32toks on MTP. Now I've seen up to 29toks , which to me is usable considering this old hardware. (My cpu is 15 years old and doesn't even have AVX !) GLM-5.3 flash and Qwen3.8 Flash did the heavy tuning work, I just directed and cracked the whip at them for a few days. There's a new Qwen 3.5 4b coder that can do 3d models , I'll be playing with that next, it looks very promising and starts at about 35toks on my card.

RX580 now getting usable performance for llm inference on Hackintosh! 2 years ago this wasn't even worth the hours of experiments to try running anything on the rx580, it was slow, models were a toy and any quantization of kv cache would give you garbage results. Now TOSHllm has brought modern AMD Metal support to llama.cpp , which had always focused on Apple silicon previously. And I extended his work to improve the kernels on these old Polaris RX580's that are a great card on Hackintosh. graydini/toshllm , I've reached with in spitting distance of modern linux Vulkan , and I think I tie with ROCm but the ancient machine I'm on now (a xeon westemere) has a hardware conflict with ROCm so I can't try it and I'm not going to put it back in my old newer rig to test that. Up to 29Toks decode for Ornith 1.5 9b which is the best model I found so far to fit on 8gb with good enough context to run Hermes if I want. I was getting about 17 toks on his base line when I started this.