把 7440 亿参数的大模型塞进笔记本:Calibri 到底做对了什么?

2026年11月8日 · AOAI AI

Running a 744-Billion-Parameter AI Model on Your Laptop: What Did Calibri Get Right?

过去两年,AI 圈里流传着一个默认的共识:想跑最前沿的大模型,要么租云端算力,要么砸钱买一排高端 GPU。几百亿参数的模型已经让普通玩家望而却步,更别说参数规模逼近万亿的”巨无霸”了。但最近,Better Stack 的一期视频介绍了一个让人眼前一亮的开源工具 Calibri,它由开发者 Vincenzo Fornaro 开发,目标非常直接:让普通的消费级硬件也能在本地跑起像 GLM 5.2 这样拥有 7440 亿参数的超大模型。这件事听上去像天方夜谭,但它确实被做出来了。

For the past two years, the AI world has lived with an unspoken assumption: if you want to run a frontier model, you either rent cloud compute or buy a rack of high-end GPUs. Models with a few dozen billion parameters already strain ordinary hobbyists, let alone giants approaching a trillion. But a recent Better Stack video introduced an open-source tool that challenges this assumption. It is called Calibri, built by developer Vincenzo Fornaro, and its goal is bold: let ordinary consumer hardware run a model like GLM 5.2, which has 744 billion parameters. It sounds absurd, yet it actually works.


先说清楚一件事:Calibri 并不是什么”魔法”,它没有让你的笔记本凭空多出几百 GB 显存。它所做的,是把软件工程里最朴素的思想——分层缓存、按需加载——用到了极致。理解它为什么可行,需要先理解这个模型本身的结构,也就是如今大模型领域最热门的架构之一:MoE,混合专家模型。

Let us be clear about one thing first: Calibri is not magic. It does not conjure hundreds of gigabytes of VRAM out of thin air. What it does is take two of the oldest ideas in systems engineering, tiered caching and loading on demand, and push them to the limit. To see why this is possible, we first need to understand the model itself and one of the hottest architectures in modern AI: Mixture of Experts, or MoE.


传统的稠密模型,每处理一个 token,几乎所有参数都要参与计算。参数越多,算得越慢,吃的显存越多,这条线性关系几乎没有捷径。MoE 的思路完全不同:它把模型的大部分参数拆分成成千上万个小型子网络,也就是所谓的”专家”。每处理一个 token,一个路由机制只会挑选其中很小的一部分专家来参与计算,其余的专家这一轮”休息”。

In a traditional dense model, nearly every parameter participates in computing every token. The more parameters you have, the slower and hungrier the model becomes, and there are few shortcuts. MoE takes a very different approach. It splits most of the model into thousands of small sub-networks called experts. For each token, a routing mechanism picks only a small subset of them to do the work, while the rest sit idle for that step.


GLM 5.2 正是这样的设计。它的总参数是 7440 亿,但对于任何一个给定的 token,真正被激活的参数只有约 40 亿,占比大约 5%。这意味着,虽然模型”体积”庞大,但每一步的瞬时计算量,其实只相当于一个中等规模的模型。换句话说,MoE 把”模型有多大”和”每一步要算多少”这两件事解耦了。Calibri 正是抓住了这个解耦带来的空间。

GLM 5.2 is built exactly this way. It has 744 billion parameters in total, but for any given token only about 4 billion are active, roughly 5 percent. So although the model is enormous on disk, the per-step computation resembles that of a mid-sized model. In effect, MoE decouples how big a model is from how much work each step requires. Calibri exploits the room that this decoupling creates.


不过,计算量降下来了,存储问题依然存在。7440 亿个参数,即使经过量化压缩,体积仍然是数百 GB 的量级,绝不可能完整塞进一台普通电脑的显存,甚至塞不进内存。这时候 Calibri 的第二个核心思想登场:把电脑的存储系统看成一个多层级的缓存体系。GPU 显存最快但最小,系统内存次之,NVMe 固态硬盘最慢但最大。三者像金字塔一样,各司其职。

However, less computation does not mean less storage. Even after quantization, 744 billion parameters occupy hundreds of gigabytes. That will never fit in the VRAM of an ordinary machine, and often not even in its RAM. This is where Calibri’s second core idea comes in: treat the computer’s storage as a layered cache. GPU VRAM is the fastest but smallest, system RAM sits in the middle, and the NVMe SSD is the slowest but largest. Like a pyramid, each layer has its own job.


具体怎么分工?模型里有一部分组件是每个 token 都要用到的稠密部分,约 170 亿参数,量化之后体积不到 10GB。这部分被 Calibri 常驻在内存里,随时待命。而那座庞大的”专家库”则安静地躺在固态硬盘上。当模型的路由机制决定需要调用某几个专家时,Calibri 会实时把这些专家从硬盘流式加载到内存中,用完之后保留在缓存里,以备后续复用。

How is the work divided? Some components of the model are used by every single token. This dense portion is about 17 billion parameters, and after quantization it takes under 10 GB. Calibri keeps it permanently in memory, ready at any moment. The vast library of experts, meanwhile, rests quietly on the SSD. When the router decides that certain experts are needed, Calibri streams them from disk into memory in real time, then keeps them cached in case they are needed again.


这个设计的精妙之处在于,它承认了一个事实:在任何一个时间点,你并不需要整个模型,你只需要”此刻要用的那一小部分”。这很像操作系统的虚拟内存,也像我们日常使用的流媒体——你不必先下载完整部电影才能开始观看。Calibri 把这种思路搬到了大模型推理上,让硬盘从单纯的”仓库”,变成了推理链路中一个活跃的环节。

The elegance of this design lies in accepting a simple truth: at any moment, you do not need the whole model, only the small slice you are using right now. It resembles virtual memory in an operating system, or video streaming, where you do not download an entire film before pressing play. Calibri brings that mindset to LLM inference, turning the disk from a passive warehouse into an active link in the inference chain.


除了核心的分层与流式加载,Calibri 在工程细节上也下了功夫。首先,它完全用纯 C 语言编写,并且零外部依赖。这意味着没有庞杂的 Python 环境,没有层层叠叠的框架版本冲突,编译之后就能跑,部署门槛极低。对于想在自己机器上折腾本地模型的人来说,这种”干净”本身就是一种吸引力。

Beyond tiering and streaming, Calibri also pays attention to engineering details. It is written in pure C with zero external dependencies. That means no sprawling Python environment and no tangled framework versions: compile it and run it. For anyone who likes to tinker with local models, that cleanliness is an attraction in itself, and it keeps the barrier to entry low.


其次,它内置了几项针对性的优化。一是智能缓存学习:它会记录哪些专家被频繁调用,并提前把它们放进内存,减少临时从硬盘读取的次数。二是推测解码,用较低的成本先”猜”出后续的若干个 token,再批量验证,从而提升生成速度。三是高效的 KV 缓存压缩,降低长上下文对内存的占用。这几项技术单独看都不新鲜,但在这样一个极端受限的环境里组合使用,价值被放大了。

It also includes several targeted optimizations. The first is smart cache learning: it tracks which experts are called most often and preloads them into memory, reducing slow disk reads. The second is speculative decoding, which cheaply guesses several upcoming tokens and verifies them in a batch to speed up generation. The third is efficient KV-cache compression, which reduces memory use for long contexts. None of these ideas is new on its own, but in such a constrained environment, combining them multiplies their value.


理论讲完了,关键还是实测。视频在两套差异很大的硬件上跑了 Calibri。第一套是一台 MacBook,搭载 M2 Max 芯片、32GB 内存,模型文件则放在一块外接的 USB 固态硬盘上。这几乎是”普通用户能拿出的最常见高配笔记本”的代表。结果怎么样?一句话概括:能跑,但很慢。

Theory is nice, but benchmarks matter. The video tested Calibri on two very different machines. The first was a MacBook with an M2 Max chip and 32 GB of RAM, with the model stored on an external USB SSD. This is roughly the best laptop setup that a typical enthusiast might own. The result, in one sentence: it works, but it is slow.


先说好的一面:它确实成功回答了问题,比如”比利时的首都是哪里”,并给出了正确答案。这本身就是一个里程碑——一台笔记本,在没有任何云端支持的情况下,真的把一个 7440 亿参数级别的模型”跑起来”了。过去这几乎是不可想象的。从”不可能”到”可行”,这一步的意义远大于速度本身。

Start with the good news. The model successfully answered questions such as “What is the capital of Belgium?” and got them right. That alone is a milestone: a laptop, with no cloud support at all, actually ran a model in the 744-billion-parameter class. Not long ago this would have been unthinkable. Going from impossible to possible matters far more than the raw speed at this stage.


但现实也很骨感。生成第一个词,用户需要等待约两分钟;之后的生成速度大约只有每秒 0.1 个 token,也就是大约十秒钟才蹦出一个字。按这个节奏,写一段稍长的回答要耗上很久。这种速度显然无法支撑日常的对话或编程辅助,更多只能算作一次”概念验证”。

But reality bites. The user waited about two minutes for the first word, and after that generation crawled along at roughly 0.1 tokens per second, which is about ten seconds per word. At that pace, even a short answer takes a very long time. It clearly cannot support everyday chat or coding assistance. It is better understood as a proof of concept.


为什么这么慢?瓶颈的原因并不神秘。32GB 的内存无法容纳足够多的专家,大部分专家调用都需要从外接硬盘现读;而 USB 接口的外置固态硬盘,传输速度远低于机器内部的存储。视频的分析显示,超过 80% 的时间都耗在了等待硬盘数据加载上,真正用于数学运算的时间只占一小部分。换句话说,这台 MacBook 的算力并没有被充分利用,它大多数时候是在”等货”。

Why so slow? The bottleneck is not mysterious. With only 32 GB of RAM, the machine cannot hold enough experts, so most expert lookups require reading from the external drive. And a USB-attached SSD is much slower than internal storage. According to the video’s analysis, over 80 percent of the time was spent waiting for data to load from disk, with only a small slice devoted to actual math. In other words, the MacBook’s compute was mostly idle, waiting for deliveries.


第二套测试环境则强得多:一台高性能工作站,搭载 RTX 5090 显卡、64GB 内存,模型放在速度更快的内部固态硬盘上。结果出现了明显改善:生成第一个词只需约 17 秒,整体生成速度提升到每秒约 0.8 个 token,并且能够比较清晰地解释量子纠缠这类复杂概念。从两分钟到十几秒,从 0.1 到 0.8,进步是实打实的。

The second test machine was far stronger: a high-end workstation with an RTX 5090, 64 GB of RAM, and the model on a faster internal SSD. The improvement was clear. The first word arrived in about 17 seconds, generation speed rose to roughly 0.8 tokens per second, and the model could explain a complex idea such as quantum entanglement coherently. Going from two minutes to seventeen seconds, and from 0.1 to 0.8 tokens per second, is real progress.


值得注意的是,这次提升的主要原因并不是那块顶级显卡。更快的内部硬盘、更大的内存,让等待硬盘加载的时间大约减少了一半,机器因此有更多时间去做真正的计算。这是整个视频最有启发性的发现之一:在这类超大模型的本地推理中,决定体验的往往不是 GPU 有多强,而是数据能多快地被送到 GPU 面前。

Notably, the gain did not mainly come from the flagship GPU. A faster internal drive and more RAM cut the waiting time on disk loads roughly in half, leaving the machine more time for real computation. This is one of the most instructive findings of the whole video: for local inference of such huge models, the experience is often decided not by how strong the GPU is, but by how quickly data can be delivered to it.


这个结论会改变我们对”本地 AI 硬件”的一些直觉。过去讨论本地大模型,大家几乎都在比较显卡型号和显存容量。但在 MoE 加存储流式加载的路线下,系统内存容量和 SSD 的读写速度同样关键,甚至更关键。也就是说,一块高速的 NVMe 固态硬盘和一条足够大的内存,带来的收益,可能比多加一块昂贵的显卡更直接。

That conclusion should change some of our intuitions about local AI hardware. Discussion of local models has long centered on GPU models and VRAM capacity. But on the MoE-plus-streaming path, system memory and SSD read speed matter just as much, perhaps more. A fast NVMe drive and a generous amount of RAM may bring more direct benefits than adding another expensive graphics card.


当然,我们也要保持冷静。视频最终的结论很诚实:把前沿大模型流畅地装进普通笔记本的理想,目前还没有完全实现。想要在实用的速度下运行这类模型,依然需要配置极高的设备,比如拥有 1TB 内存的服务器,或是多 GPU、多块大内存组合的架构。每秒 0.8 个 token,对于技术演示来说令人兴奋,但对于日常使用来说仍然偏慢。

Still, we should stay level-headed. The video’s final verdict is honest: the dream of running a frontier model smoothly on an ordinary laptop is not fully realized yet. To get practical speeds, you still need very capable machines, such as servers with 1 TB of RAM or multi-GPU, multi-RAM setups. Roughly 0.8 tokens per second is thrilling as a demo, but still too slow for daily use.


那么,Calibri 的价值究竟在哪里?我认为有三点。第一,它验证了路径:证明在极端受限的硬件上运行万亿级模型是可行的,只是速度问题。第二,它指明了瓶颈:让社区清楚地看到,下一步该优化的是数据搬运,而不是一味堆算力。第三,它降低了门槛:开源、纯 C、零依赖,让更多开发者可以在此基础上继续改进。

So where does Calibri’s value lie? I see three points. First, it validates the path: running a model of this scale on severely limited hardware is feasible, and what remains is a speed problem. Second, it identifies the bottleneck: the community can now see that the next focus should be data movement rather than simply stacking more compute. Third, it lowers the barrier: being open source, pure C, and dependency-free invites more developers to build on it.


从更大的视角看,这件事也折射出 AI 发展的一个趋势:模型架构和系统工程正在互相靠拢。MoE 让模型天然具备”稀疏”的特性,而稀疏性恰好是存储分层、按需加载最理想的土壤。未来,我们很可能会看到更多专门为”稀疏激活加分层存储”设计的模型与推理引擎,甚至连硬件厂商也会针对这种访问模式去优化 SSD 与内存的带宽。

From a wider angle, this also reflects a trend in AI: model architecture and systems engineering are growing closer. MoE gives models a naturally sparse character, and sparsity is the perfect soil for tiered storage and on-demand loading. We will likely see more models and inference engines designed specifically for sparse activation plus layered storage, and even hardware vendors may begin optimizing SSD and memory bandwidth for these access patterns.


对普通用户来说,短期内的建议也很清楚:如果你只是想体验本地大模型,不妨先选择中小规模的模型,它们在消费级设备上已经相当好用。如果你是爱折腾的开发者,想亲眼看看 7440 亿参数的模型在自己机器上”吐字”的感觉,Calibri 是一个有趣的实验场。准备好一块高速的内部固态硬盘,并给自己多一点耐心。

For ordinary users, the short-term advice is simple. If you only want to experience local AI, choose small or mid-sized models, which already work well on consumer devices. If you are a tinkerer who wants to watch a 744-billion-parameter model produce text on your own machine, Calibri is a fascinating playground. Bring a fast internal SSD, and bring some patience.


回到最开始的问题:大模型会不会真的走进每个人的笔记本?Calibri 给出的答案是:门已经被推开了一道缝,但还没有完全敞开。速度、内存、硬盘带宽,每一项都还有提升空间。可贵的是,推开这道门的,不是一家大公司的资源堆砌,而是一位独立开发者对工程细节的执着。这正是开源社区最迷人的地方。

Back to the opening question: will giant models truly make their way onto everyone’s laptop? Calibri’s answer is that the door has been pushed open a crack, but not yet thrown wide. Speed, memory, and disk bandwidth all have room to improve. What is remarkable is that the person who pushed that door is not a big company with piles of resources, but an independent developer with a passion for engineering details. That is precisely what makes open source so compelling.


最后,不妨留给自己一个思考:当未来某一天,固态硬盘的速度再翻几倍,内存价格再降下来,今天这个”十秒一个字”的演示,也许就会变成流畅的日常体验。到那时,我们回头再看 Calibri,大概会把它视为本地大模型从”不可能”走向”可能”的一个关键注脚。让我们拭目以待。

Finally, a thought to leave with. Someday, when SSDs get several times faster and memory gets cheaper, today’s “one word every ten seconds” demo may turn into a smooth everyday experience. Looking back then, we will probably see Calibri as a key footnote in how local large models went from impossible to possible. We will be watching.