MLX and RDMA implement local and distributed large language model inference on Mac Studio
On a cluster of four Mac Studio machines, Muse Spark Open (816B parameters) generates at a speed of over 36 tokens/second. The Mac Studio M5 Ultra version offers 4.6x faster inference speed (TTFT) and 1.6x faster tokens/second compared to the M3 Ultra, benefiting from the GPU neural accelerator and 1.2 TB/s memory bandwidth.






















