When evaluating hardware for machine learning and artificial intelligence, the conversation usually revolves around raw compute—teraflops and clock speeds. But for running Large Language Models (LLMs) locally, the true bottleneck isn't compute. It's memory bandwidth.
This is exactly why Apple Silicon has inadvertently become the ultimate architecture for local AI.
The PCIe Bottleneck
On traditional PC architectures, the system RAM and GPU Video RAM (VRAM) are physically separated. The CPU loads the model into system RAM, and then has to shuttle chunks of that massive model across the PCIe bus to the GPU for it to do the actual math.
When you are dealing with a 32-billion parameter model, constantly moving gigabytes of tensor data back and forth across a PCIe bridge introduces catastrophic latency. This is why traditional PCs require extremely expensive, dedicated Nvidia GPUs with their own discrete VRAM pools just to run inference smoothly.
Enter Unified Memory
Apple discarded this paradigm. With the M-series chips, the CPU and the GPU share the exact same physical pool of memory on the die.
There is no copying. There is no PCIe bus transfer.
When Mochi loads a language model into memory, the GPU can instantly access those weights at staggering speeds—upwards of 800GB/s on a Mac Studio. This is why a relatively standard MacBook Pro with 36GB of unified memory can run complex, highly-quantized models that would otherwise require thousands of dollars in discrete server GPUs on a PC.
Powering Mochi
Because Mochi is built specifically for Apple Silicon, it bypasses traditional memory abstractions. We map the model weights directly into unified memory, allowing the Metal GPU cores to scream through inference calculations.
The result? Answers that stream onto your screen instantly, feeling less like software, and more like thought.