🔍 Read the full analysis: Can Transformers Handle 180B Parameters With Just 32GB Of RAM? on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Hugging Face developer proposed an experimental Transformers option that loads selected mixture-of-experts model weights from storage into a bounded device cache. The developer reported 34.6 tokens per second on OLMoE, near the stated 38.1-token-per-second standard result, but did not report a completed test of a 180B model on a 32 GB Mac.
A Hugging Face developer has proposed an experimental change to Transformers that streams selected mixture-of-experts (MoE) model weights from storage into a bounded device cache, as described in the original analysis. In a reported test on OLMoE, it reached 34.6 tokens per second, compared with 38.1 for the standard Transformers path; the source does not report a completed test of a 180B model on a Mac with 32 GB of memory.
The approach targets a memory problem in MoE models: their expert weights can exceed available GPU or unified memory, even though a router selects only some experts for each token. The developer says experts account for more than 90% of the model weights in the cases discussed. The proposed system aims to keep selected experts in a device cache and fetch others from storage when required.
In the implementation work described, a first version ran experts one at a time in a Python loop. On the OLMoE test it reached 10.1 tokens per second, even with experts already cached, compared with about 32 for vanilla Transformers. The developer then used a shared, preallocated device tensor and a table connecting expert IDs to cache slots. That let the existing grouped matrix-multiplication kernel handle selected experts together; the reported result was 34.6 tokens per second versus 38.1 for the standard path.
The developer says the option was submitted to Transformers’ from_pretrained loading flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes work spanning Transformers, a model-specific package and Metal kernels, and says AI was used in building the code. These are details reported by the project author, not an independent performance assessment.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
Lower Memory Use, With Speed Trade-Offs
If the method works on larger models, it could let people run some MoE models on machines that cannot hold every expert in fast memory. That may widen local inference options for users with a laptop or a single GPU, where memory capacity can prevent loading a model even if each token uses only a subset of its experts.
The central question is whether fetching experts from storage can keep pace with generation. The OLMoE result suggests the cache design can preserve much of the standard path’s speed in that reported test. It does not show what happens when a much larger model requires frequent SSD reads, nor does it establish the memory use or speed of the proposed 180B-on-32-GB setup.
Products Worth Considering

Samsung 32 Odyssey G5 QHD Gaming Monitor

Bose QuietComfort Wireless Noise-Cancelling Headphones, Moonlight Grey

Sonos Arc Ultra Dolby Atmos Soundbar, Black
As an affiliate, we earn on qualifying purchases.
How Expert Streaming Is Designed
MoE models use a router to select a subset of experts for each token. In the proposed design, selected experts are placed in a bounded device cache while others can remain in RAM or on disk, depending on the system. The intended difference from a simple offload approach is to fetch only the experts selected by the router, then continue using Transformers’ grouped expert computation.
The source also reports a separate comparison using Qwen3.6-35B-A3B on an A10G GPU with 24 GB of memory and ample system RAM. In that setup, it says Strata decoded at 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those figures concern a different model and configuration; they are not direct results for the proposed Transformers option or a 32 GB Mac. The source characterizes other offload approaches differently, including CPU expert execution in llama.cpp and whole-layer expert-bank movement with Transformers and Accelerate.
The 180B Mac Benchmark Is Missing
The source frames a 180B MoE model on a 32 GB Mac as a motivating use case, but gives no measured result for that model, device or setup. It also does not provide storage type, cache size, prompt conditions or repeated runs for the OLMoE comparison, limiting what can be concluded about performance across workloads.
The reported figures do not establish how the cache performs when the router’s selected experts change rapidly, or how it compares with Strata on identical hardware and a matching model. The source says the work was submitted as pull request #50717 but does not say whether it was accepted, merged or released. The cited results and implementation details are developer-reported, not independently verified in the material provided.
Pull Request and Device Tests
The next concrete development is the status of pull request #50717 and whether the option becomes part of a published Transformers version. The project Space offers a way to inspect the experimental code, according to the source, but its existence does not confirm that the change has been accepted or packaged for general use.
To test the headline use case, further reporting would need to show speed and memory use for the intended 180B model on a 32 GB Mac, with storage, cache settings and workload identified. Repeated tests and a comparison with Strata under matching conditions would help show whether the approach can maintain useful generation speed when experts must be loaded from storage.
Key Questions
Has a 180B model been shown running on 32 GB of RAM with this method?
Not in the source material. The 180B model and 32 GB Mac are presented as a target, not a completed benchmark.
What performance did the developer report?
The reported OLMoE test reached 34.6 tokens per second, compared with 38.1 for the standard Transformers path. The source does not provide enough test details to assess variation across workloads.
How does the proposed method reduce memory requirements?
It aims to load only the experts selected for a token into a bounded device cache, while leaving other expert weights in RAM or on storage. The method is experimental, and its performance on the larger target model is unreported.
Is the change available in Transformers?
The developer says it was submitted as pull request #50717. The source does not state whether it was accepted, merged or released.
Primary source: Hugging Face · via ThorstenMeyerAI.com
