Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

Anemll @anemll Confirmed 24K context on iPhone with both K/V on int8. Delta net stays FP16. Decoding also peaks at 70 tok/s with easy to predict context. Apple Neural Engine on A20 is fast 🚀 00:00 Anemll @anemll 12h Qwen 3.8- 27B on iPhone 18 Apple Neural Engine. Added draft model , dflash-2 at ~ 2.8 bpw ( mixed 2-4 vector quantization with small drop in acceptance rate) Base is @ PrismML Bonsai-2, 1.585 bpw requantized to 1.5 bpw ( VQ 6 bits per 4 weights) Decoding speed is 15-60 tps. 00:00 Last edited 5:58 PM · Oct 17, 2026 Opens edit history · 714 Views 5

Anemll @anemll 4m Please upvote 🙏 news.ycombinator.com/item?id=501335…

David Dalcu @ddalcu 31m Amazing!🤩