Janas: Run LLMs Larger Than RAM By Streaming Experts From SSD

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Janas-LLM, an early-stage C project, streams mixture-of-experts model weights from NVMe storage and caches frequently used experts in memory. Its developer reports running 48.4 GB models on a 32 GB laptop and faster results than llama.cpp across most listed tests on that machine; the benchmarks are developer-run, hardware-specific, and not independent validation.

Janas-LLM, an experimental language-model runtime by developer Maurizio “camauri” Cammalleri, is designed to run mixture-of-experts models larger than a computer’s RAM by loading the needed experts from an NVMe SSD and keeping frequently used ones in memory. The project reports that a 48.4 GB model generated text on a laptop with 32 GB of RAM, but its published speed figures are developer-run tests on one machine, not independent benchmarks.

In measurements posted with the project, Janas-LLM ran on a laptop with an Intel Core Ultra 9 185H, integrated Arc graphics, 32 GB of memory and an NVMe SSD. The developer reported 24 tokens per second generating with Qwen3-Next-80B-A3B, listed as a 48.4 GB model, compared with 2.8 tokens per second for llama.cpp on that setup. With speculative draft decoding enabled, Janas’ reported generation result was 37 tokens per second. The repository also reports 62 tokens per second for processing a 1,024-token prompt, against 16 for llama.cpp.

The published table covers a range of models and separates prompt processing from text generation. Janas reported higher rates than llama.cpp on most listed tests, including smaller models; for Qwen3-4B, for example, it recorded 27 tokens per second generating versus 18. The comparison is not uniformly in Janas’ favor: with a three-bit Qwen3-30B-A3B build, Janas read the prompt more slowly than llama.cpp, though its reported generation speed was three times higher. The figures depend on the model file, settings and hardware, and should not be treated as universal performance estimates.

The project describes Janas as a family of AI tools written in C, with no dependencies beyond the C library. It says the models are supplied in a converted format and that the runtime chooses its behavior based on measurements of the computer. The project offers Linux x86-64 and Windows builds, but warns that the software is in early development and its command-line interface and public API can change frequently.

Running Oversized Models on Laptops

Janas’ approach targets a practical constraint for local AI: a model’s stored weights may exceed available system memory, limiting which computers can run it. Streaming only the experts needed for a token from an SSD could let some mixture-of-experts models run on lower-memory systems, rather than requiring all model weights to stay in RAM at once. That could widen access to local inference for people with capable storage but without workstation-scale memory.

The trade-off is that SSD access, memory pressure and GPU support all affect real-world performance. The reported results suggest the method can be fast on the developer’s laptop, but they do not establish that it will perform similarly on other computers or with other storage devices. The project itself warns that its cache can consume free memory and that a GPU run has previously reset a driver, underscoring that this is an experiment rather than a stable consumer product.

How Janas Loads Model Experts

Mixture-of-experts models contain multiple specialist components, or experts, while using only some of them for a given token. Janas says it keeps model weights on disk, streams the experts needed for each token from an NVMe SSD, and retains repeatedly used experts in a memory cache. This differs from approaches that try to keep the entire model resident in RAM, although the project’s benchmark results alone do not establish how the method compares across a broad range of hardware.

The published Linux results were gathered on Debian 13 between October 9 and 12, 2026, with the laptop’s integrated GPU. The repository also lists Windows 11 tests on the same hardware dated October 12. Those results show why platform and configuration matter: on Windows, the project reported 25.6 tokens per second generating with Qwen3-4B and 34.2 with Qwen3-30B-A3B, while prompt processing for Qwen3-4B was reported at 346 tokens per second. These are measurements from the project, not tests by an outside lab.

Limits of the Published Tests

The benchmark results were produced by the Janas developer on one laptop; the source material does not include independent replication or tests across a broad range of computers. It is also not clear how the runtime will perform with different SSD speeds, memory capacities, GPU drivers, or sustained workloads. The project notes that it is early-stage and unverified, and says a GPU run has once caused a driver reset. Its warning that the cache can push a machine into swap means a model fitting through SSD streaming does not guarantee a responsive desktop.

The tables report tokens per second under specified tests, but those figures do not by themselves describe answer quality, power use, reliability over long sessions or the full setup required to reproduce each result. The repository’s tests include different model quantizations and, in some cases, draft decoding, so comparisons depend on matching those conditions. The available source does not establish a product release schedule or independent assessment of the claims.

Testing Beyond One Laptop

The project invites users on supported Linux and Windows x86-64 systems to run Janas and use its janas-bench utility to measure performance on their own hardware. Those results could show how much the SSD-streaming approach varies across machines, but the repository does not describe a formal external testing program or a timetable for a stable release.

For now, the next meaningful developments are broader reproducible benchmarks, clearer guidance on memory and GPU risks, and updates to the runtime as it remains under active development. Until those are available, the reported ability to run a 48.4 GB model on 32 GB of RAM is a notable project result, not a general guarantee for users.

Key Questions

What is Janas-LLM?

Janas-LLM is an experimental runtime that runs language models by streaming needed mixture-of-experts weights from storage and caching frequently used experts in memory.

How can it run a model larger than RAM?

According to its documentation, Janas keeps model weights on disk and reads the experts needed for each token from an NVMe SSD, rather than requiring all weights to reside in system memory at once.

What performance did the project report?

On a laptop with 32 GB of RAM, the developer reported 24 tokens per second generating with a 48.4 GB Qwen3-Next-80B-A3B model, or 37 with draft decoding. These are project-reported results from one machine, not independent benchmarks.

Is Janas ready for everyday use?

The project describes itself as an experiment, not a product. It warns that the software is unverified, can use substantial memory and disk bandwidth, and may behave differently on hardware other than the developer’s laptop.

Source: hn