That’s exactly what I thought about. Getting my hands on an M5 Max this week and... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

Yukonv 3 months ago | parent | context | favorite | on: iPhone 17 Pro Demonstrated Running a 400B LLM

That’s exactly what I thought about. Getting my hands on an M5 Max this week and going to see hows Dan’s experiment performs with faster I/O. Also going to experiment with running active parameters at Q6 or Q8 since output is I/O bottlenecked there should room for higher accuracy compute.

anemll 3 months ago | [–]

Check my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8 https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...

3abiton 3 months ago | [–]

To be fair, it's "possible" to run such setup with llama.cpp with ssd offload. It's just abysmal TG speeds. But it's possible.

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact