The best local LLM inference setup:
4x Mac Studio (M3 Ultra, 512 GB of unified RAM), 2 TB of UMA RAM with RDMA
EXO 1.0 tooling for clustering, now with tensor parallelism enabled!
RDMA (Remote Direct Memory Access) though Thunderbolt 5 - clustering bottleneck eliminated
MLX inference acceleration (now with RDMA support!)
And... Mac OS 26.2
https://www.youtube.com/watch?v=A0onppIyHEg&t=3m10s
DeepSeek v3.2 8 bit quantization (original training quantization) at 25 tokens per second! Wow!
516 Watts at the peak of power usage!
Downside: cost of 50K USD for hardware. Still better than one or several H100/H200/B200 with limited non unified discrete memory architecture! =)
And such setup will work for way much cheaper Mac Minis (no RDMA yet and Thunderbolt 5, but will be added to new generations of M chips, now available in M4 Pro and higher and M3 Ultra)!
Apple way ahead of all again!
In couple of years this will be a common consumer setup for local LLM inference, using conventional hardware, an APUs from AMD and Intel+NVidia (with integrated CPU+GPU NVLink bus - an upcoming APU architecture), while Apple and NVidia will use Intel Fabs and TSMC fabrication.
The enclaves/TEE for hardware memory encryption will be part of such setups for confidential computing over confidential sensitive data.
#CPU
#GPU
#LLM
#TEE
4x Mac Studio (M3 Ultra, 512 GB of unified RAM), 2 TB of UMA RAM with RDMA
EXO 1.0 tooling for clustering, now with tensor parallelism enabled!
RDMA (Remote Direct Memory Access) though Thunderbolt 5 - clustering bottleneck eliminated
MLX inference acceleration (now with RDMA support!)
And... Mac OS 26.2
https://www.youtube.com/watch?v=A0onppIyHEg&t=3m10s
DeepSeek v3.2 8 bit quantization (original training quantization) at 25 tokens per second! Wow!
516 Watts at the peak of power usage!
Downside: cost of 50K USD for hardware. Still better than one or several H100/H200/B200 with limited non unified discrete memory architecture! =)
And such setup will work for way much cheaper Mac Minis (no RDMA yet and Thunderbolt 5, but will be added to new generations of M chips, now available in M4 Pro and higher and M3 Ultra)!
Apple way ahead of all again!
In couple of years this will be a common consumer setup for local LLM inference, using conventional hardware, an APUs from AMD and Intel+NVidia (with integrated CPU+GPU NVLink bus - an upcoming APU architecture), while Apple and NVidia will use Intel Fabs and TSMC fabrication.
The enclaves/TEE for hardware memory encryption will be part of such setups for confidential computing over confidential sensitive data.
#CPU
#GPU
#LLM
#TEE
YouTube
Apple JUST Dropped a Game-Changer
Every M3 Ultra Mac Studio and Mac Mini cluster I’ve built hit the same wall… until Apple quietly removed the bottleneck—and suddenly, we're running the real deal.
👀 Start learning cyber security with TryHackMe: https://tryhackme.com/ziskind Use my code "ZISKIND25"…
👀 Start learning cyber security with TryHackMe: https://tryhackme.com/ziskind Use my code "ZISKIND25"…