A Whole Book in 16 Tokens? Notes on Memorilla: Fast, but Fuzzy on Details
最近有个说法在圈子里传得很快:斯坦福推出了一个叫 Memorilla 的 AI 记忆研究项目,据称能把整本书或海量的聊天记录,压缩成区区 16 个记忆 Token,再交给大模型去读。听上去很像一个”无限上下文”的捷径,所以我专门看了一位博主做的公开实测,把原理、速度和翻车的地方整理成这篇笔记。先说明一点:文中的数字全部来自那次实测和项目的官方说法,我自己没有复现,请当作参考。我想回答的问题只有一个:它到底是能替代长上下文,还是只是一个看上去很美的压缩技巧。这类”把长内容压短再交给模型”的思路,本来就是当下大模型领域最热的问题之一,因为上下文越长,速度越慢、成本越高,谁能用更少的 Token 带更多的信息,谁就占了便宜。Memorilla 把这个想法推到了一个很极端的位置,16 这个数字小得让人不敢相信,也正因为如此,它值得被认真检验,而不是被一句标题带着走。下面按”原理、速度、成功的例子、翻车、原因、建议”的顺序展开,你可以直接跳到自己关心的部分。
A claim has been spreading fast lately: Stanford has released an AI memory research project called Memorilla that reportedly compresses an entire book or a mountain of chat logs into just 16 memory tokens for a large model to read. It sounds like a shortcut to “unlimited context,” so I went through a blogger’s public test and organized the principle, the speed and the failures into this note. One thing up front: the numbers below all come from that test and the project’s own claims, and I have not reproduced them myself, so treat them as reference. I want to answer just one question: can it replace long context, or is it a compression trick that only looks good? The idea of “shrinking long content before handing it to the model” is one of the hottest problems in large models today, because the longer the context, the slower and costlier it gets, and whoever carries more information in fewer tokens gains an edge. Memorilla pushes this idea to an extreme position; the number 16 is so small it is hard to believe, and for exactly that reason it deserves serious testing rather than being carried along by a headline. What follows runs in the order of principle, speed, the successful example, the failures, the reasons and the advice, so you can jump straight to the part you care about.
先看原理,其实分三步。第一步,模型不把长文本直接塞进提示词,而是先把每一份文档转成单个向量。第二步,一个约 2 亿参数的小型训练模块登场:它会拿你的问题去评估所有文档向量,经过两轮筛选与融合,最后提炼出 16 个记忆 Token,也就是 16 个向量。第三步,这 16 个向量被直接输入到一个保持冻结、没有做任何修改的 Qwen3-8B 语言模型,由它生成回答。换句话说,主模型一点没动,真正被训练的只是中间那个小模块,它负责回答”这么多文档里,针对这个问题,哪些信息值得带进去”。这一步有点像一位反应很快的筛选员:它不会把所有文档原样端给主模型,而是先看问题,再决定把哪些文档的意思、以多大的比例融合进那 16 个向量里。所以同样一批文档,换一个问题,得到的 16 个向量也会不同。这个思路的好处很明显:主模型可以随时换,记忆部分单独更新。它也解释了为什么这个项目会强调”冻结”:不改动主模型的参数,就不会破坏它原有的能力,也省掉了为每个新场景微调一整个大模型的成本。代价则是所有的信息都必须挤过那 16 个向量组成的窄口,这一点后面会反复出现。给不熟悉的朋友补一句:向量可以理解为一长串数字,模型把一段文字的含义”折叠”进这串固定长度的数字里,意思相近的文字,数字也相近。正因为长度固定,一份文档不管多长,压缩出来的大小都一样,这既是它省空间的原因,也是它丢细节的根源。
First the principle, which has three steps. Step one: instead of stuffing long text into the prompt, the model converts each document into a single vector. Step two: a small trained module of about 200 million parameters steps in. It evaluates all the document vectors against your question, goes through two rounds of filtering and fusion, and distills 16 memory tokens, that is, 16 vectors. Step three: those 16 vectors are fed directly into a frozen, unmodified Qwen3-8B language model, which generates the answer. In other words, the main model is untouched, and what gets trained is only the small module in the middle, which answers the question “out of all these documents, what is worth carrying in for this question?” This step is a bit like a quick-reacting screener: it does not serve all the documents to the main model as they are, but first looks at the question and then decides which documents’ meaning to blend into those 16 vectors, and in what proportion. So for the same batch of documents, a different question yields different 16 vectors. The benefit is clear: the main model can be swapped at will while the memory part is updated separately. It also explains why the project stresses “frozen”: leaving the main model’s parameters alone does not damage its existing abilities, and it saves the cost of fine-tuning a whole large model for every new scenario. The price is that all information must squeeze through the narrow opening of those 16 vectors, a point that will keep returning below. A word for readers new to this: a vector can be understood as a long string of numbers into which the model “folds” the meaning of a piece of text, with similar meanings giving similar numbers. Because the length is fixed, a document comes out the same size however long it is, which is both why it saves space and why it loses detail.
速度是它最亮眼的地方。据实测,处理 512 份文档,大约 12.8 万个 Token 的内容,模型的回答时间稳定在 0.12 秒左右;如果把向量化那一步也算进去,总共是 3.2 秒。作为对比,直接把全文输进去需要 19.3 秒,所以端到端大约快了 6 倍。这个差距的来源不难理解:全文输入意味着主模型要读十几万个 Token,而这里它只需要读 16 个向量。需要留意的是,3.2 秒里包含的向量化,是对文档提前做一次就可以复用的工作,所以文档库不变、问题很多的场景,体感上会更接近 0.12 秒那个数字。不过这是我的推断,实测里并没有专门拆开讨论。另外要留意,这组速度数字是在 512 份文档的规模下测的,文档再多时,第二步里”评估所有文档向量”的开销会不会明显增长,实测没有给出答案。所以”快了 6 倍”这个结论,更适合理解为”在这个规模下的一次测量”,而不是一条可以随意外推的规律。话说回来,对实时对话或需要频繁查询的应用来说,等待时间从将近 20 秒降到几秒,体验上是质变,哪怕准确率不变,这样的速度也足以改变一个产品的形态,这正是人们对压缩记忆抱有期待的原因。
Speed is its brightest point. In the test, with 512 documents, roughly 128,000 tokens of content, the model’s answer time held steady at about 0.12 seconds, or 3.2 seconds in total once the vectorization step is counted. By comparison, feeding in the full text took 19.3 seconds, so end to end it was about six times faster. The source of the gap is easy to understand: full-text input makes the main model read over a hundred thousand tokens, while here it reads only 16 vectors. Note that the vectorization inside the 3.2 seconds is work you can do once per document and reuse, so with a fixed document store and many questions, the felt speed would be closer to that 0.12-second figure. That is my inference, though; the test did not discuss it separately. Note too that these speed figures were measured at 512 documents, and whether the cost of “evaluating all document vectors” in step two grows noticeably with more documents, the test does not say. So “six times faster” is better read as “one measurement at this scale” than as a rule to extrapolate freely. That said, for real-time conversation or applications queried often, cutting the wait from nearly 20 seconds to a few seconds is a qualitative change in experience, and even with accuracy unchanged, such speed is enough to alter the shape of a product, which is exactly why people hold hopes for compressed memory.
好的一面还有官方示例的复现。在官方提供的 107 条历史对话测试里,它成功把用户早先提过的”母系糖尿病风险”,和当下的”早餐推荐”联系了起来,给出了控糖的饮食建议。这个例子很能说明记忆压缩的价值:两条信息相隔很远,问题里甚至没有提到糖尿病,但它还是把背景带了进来。这正是向量压缩擅长的事,保留整体的语境和倾向,而不是逐字记住。可以拿人的记忆来类比:你记得朋友”家里有糖尿病史、饮食要注意”,却未必记得他具体是哪天说的、原话是什么。这种”印象式”的记忆,在给建议、做推荐这类任务里已经够用了。不过也要看到,官方示例是挑过的、对它有利的场景,能复现成功,只能说明它在这种设定下确实有效,不能说明它在任何设定下都有效。同一份实测里,博主自己设计的测试结果就差得多,两相对照,恰好说明了”官方场景”和”自己场景”之间那道缝。所以我的习惯是,看到任何”官方演示成功”的消息,都顺手问三句:这个例子是谁挑的,有没有对照组,换一批数据还成立吗?问完这三句,多数宣传的分量就能掂量出来了。
There is also a good point in reproducing the official example. In the official test of 107 past conversations, it successfully connected the “maternal diabetes risk” the user had mentioned earlier with the present “breakfast recommendation,” giving blood-sugar-friendly dietary advice. The example shows well what memory compression is worth: the two pieces of information sit far apart, and the question does not even mention diabetes, yet it still brought the background in. That is exactly what vector compression does well, keeping overall context and leaning rather than remembering word for word. Human memory offers an analogy: you remember that a friend “has a family history of diabetes and should watch his diet,” yet not necessarily which day he said it or his exact words. This kind of “impression” memory is already enough for tasks like giving advice and making recommendations. But note that official examples are chosen, favorable scenes; reproducing one only shows it works under that setup, not under every setup. In that same test, the blogger’s own designs did far worse, and the contrast shows exactly the gap between “the vendor’s scenario” and “your own scenario.” My habit, on seeing any “official demo succeeded” news, is to ask three questions: who chose this example, is there a control group, and does it still hold on a different batch of data? After asking those three, you can weigh most promotional claims.
然后是翻车的部分,这也是整个实测最有价值的地方。博主自己搭了一个 150 条虚拟对话的”埋藏线索”测试,又拿《巴斯克维尔的猎犬》这本小说来考它,结果都很糟糕,小说那一项 8 道题只答对了 1 道。它不仅忘了人名、生日这类具体细节,还出现了严重的幻觉:比如凭空编出一个并不存在的海鲜过敏,甚至把不同福尔摩斯故事里的反派张冠李戴。这里的关键在于,错误不是”不知道”,而是”自信地说错”。对记忆系统来说,后者危险得多,因为用户很难察觉那条被编出来的信息其实不在原始数据里。试想一下,如果这样的记忆被用在健康、法律或客户资料这类场景,一个凭空出现的”过敏”或张冠李戴的人物,后果就不只是答错一道题了。这也是我认为实测里最该被记住的一条:评价记忆系统,不能只看它记住了多少,还要看它在记不住的时候会不会编造。博主用的”埋藏线索”测试其实是个很好的方法:在数据里故意埋几条具体的细节,再提问看模型能不能找回来,会不会编出不存在的内容。你也可以照这个思路,用自己的数据做一个小测试,比读十篇评测都管用。
Then come the failures, which are the most valuable part of the whole test. The blogger built a 150-conversation “buried clue” test of his own and also quizzed it on the novel The Hound of the Baskervilles; both went badly, and on the novel it answered only 1 of 8 questions correctly. It not only forgot specifics like names and birthdays but also hallucinated seriously: for example, it invented a seafood allergy that did not exist, and even mixed up villains from different Sherlock Holmes stories. The key point is that the error is not “I do not know” but “confidently wrong.” For a memory system the latter is far more dangerous, because users can hardly tell that the made-up item was never in the original data. Imagine such memory used in health, legal or client-record settings: a made-up “allergy” or a mixed-up character would then cost more than one wrong quiz answer. That is also the point I think should be remembered most from the test: when judging a memory system, look not only at how much it remembers but at whether it makes things up when it cannot remember. The blogger’s “buried clue” test is in fact a very good method: deliberately bury a few concrete details in the data, then ask questions and see whether the model can retrieve them and whether it invents things that are not there. You can follow the same idea with your own data in a small test, which beats reading ten reviews.
为什么会这样?原文给的解释很直接:把整段对话压缩成一个向量,虽然保留了整体语境和极高的推理速度,但精确事实会随着数据量增加而丢失。这很符合直觉:一个向量的容量是固定的,装一两段话绰绰有余,装一整本书就只能保留”大意”,人名、日期、数字这类需要一字不差的信息,最先被挤掉。16 个记忆 Token 同理,它是一个有限的出口,数据越多,每条信息能分到的位置就越少。所以与其说它”记性差”,不如说它的设计目标本来就是抓大意。拿它去做需要精确引用的事,就像用地图的缩略图去找一户人家的门牌号。反过来说,如果任务本身就只需要”大意”,比如回忆一个人的整体偏好、一段对话的基调,这种压缩反而是合适的,速度优势也能派上用场。选不选它,取决于你的任务容不容得下细节的模糊。还有一层要补充:压缩得越极端,损失就越难避免。原文把问题归因于”数据量增加”,也就是说,数据少的时候表现可能不错,真正的考验在规模上来之后。107 条对话的官方示例成功,150 条自建对话和整本小说却翻车,这个对比很诱人,但两者在内容类型和测试设计上也不同,我不想把它简单归结为数量问题。
Why does this happen? The source’s explanation is direct: compressing a whole conversation into one vector keeps the overall context and gives very high inference speed, but precise facts get lost as the data grows. That fits intuition: a vector has fixed capacity, ample for a paragraph or two but able to keep only the “gist” of a whole book, and information that must be exact, like names, dates and numbers, is squeezed out first. The same holds for 16 memory tokens: a limited outlet, where the more data there is, the less room each item gets. So rather than “poor memory,” its design goal was to catch the gist in the first place. Using it for tasks that need exact quotation is like using a thumbnail map to find one family’s house number. Conversely, if the task needs only the “gist,” such as recalling a person’s overall preferences or the tone of a conversation, this compression is actually suitable, and the speed advantage can come into play. Whether to choose it depends on whether your task can tolerate fuzziness in the details. One more layer to add: the more extreme the compression, the harder the loss is to avoid. The source attributes the problem to “growing data,” which means it may do fine on small data, with the real test coming once scale rises. The official example of 107 conversations succeeded while 150 self-built conversations and a whole novel failed, a tempting contrast, but the two also differ in content type and test design, and I do not want to reduce it simply to a question of quantity.
最后说我的判断和建议。第一,Memorilla 目前仍处于早期研究阶段,值得关注的是方向:冻结主模型、单独训练记忆模块,而且支持用户基于自己的数据重新训练,这条路线很灵活。如果你手里有一批自己的数据,而且对”大意”的需求大于对”细节”的需求,它值得拿来做一次小规模的试验,但请务必把”会不会编造”也纳入你的检查清单,而不是只看速度和答对率。第二,如果你现在要做的是高准确度的生产级应用,传统的 RAG(检索增强生成)依然是更稳妥的选择,因为它每次都回到原文取证据,不会把细节压没。第三,两者并不矛盾,一个自然的想法是让压缩记忆负责”大意和语境”,让检索负责”精确事实”,各取所长,不过原文只提了方向,没有给出验证结果,这一点我也没有测试过。如果你打算上手试一试,我的建议是按这样的顺序来:先用少量自己的数据;再准备一组带有精确细节的问题;然后把它的回答和全文输入、RAG 的结果放在一起对照;最后记下它编造内容的次数。这几步做完,你对它适不适合自己的任务,心里就有数了。说到底,压缩记忆和检索原文不是谁取代谁的关系,而是在”快”与”准”之间的两种取舍,选哪一种,要看你的用户能容忍什么样的错误。如果答错一个细节只是小麻烦,速度可以优先;如果答错就要付出代价,就让原文留在手边。你更看重回答的速度,还是细节的准确?欢迎在评论区聊聊你的场景。最后再压成一句话:Memorilla 把”记住大意”这件事做得又快又省,却还没有证明自己能”记住细节”,而且在记不住的时候会自信地说错。在这三件事被验证清楚之前,把它当成一个值得跟踪的研究方向,会比把它当成现成的产品更合适。
Finally, my judgment and advice. First, Memorilla is still at an early research stage, and what is worth watching is the direction: freezing the main model, training the memory module separately, and supporting retraining on users’ own data, a flexible route. If you have a batch of your own data and need the “gist” more than the “details,” it is worth a small-scale trial, but please put “does it make things up?” on your checklist too, rather than looking only at speed and the correct rate. Second, if what you are building now is a high-accuracy production application, traditional RAG (retrieval-augmented generation) remains the safer choice, because it goes back to the source text for evidence each time and does not squash the details. Third, the two do not conflict; a natural idea is to let compressed memory handle “gist and context” and retrieval handle “exact facts,” each doing what it does best, though the source only mentions the direction without verified results, and I have not tested it either. If you plan to try it, my suggestion is to go in this order: start with a small amount of your own data; prepare a set of questions with exact details; compare its answers side by side with full-text input and with RAG results; and finally note how many times it makes things up. After these steps you will have a good sense of whether it suits your task. In the end, compressed memory and retrieving the source are not a matter of one replacing the other but two trade-offs between “fast” and “accurate,” and which to pick depends on what kind of error your users can tolerate. If a wrong detail is only a small nuisance, speed can come first; if a wrong answer carries a cost, keep the source text at hand. Do you value answer speed more, or accuracy of detail? Tell me about your use case in the comments. To squeeze it into one sentence: Memorilla does the job of “remembering the gist” quickly and cheaply, but has not yet proven it can “remember details,” and when it cannot remember it confidently gets things wrong. Until those three matters are verified, treating it as a research direction worth following suits better than treating it as a ready-made product.