An AI-MI–supported study proposes a new architecture for language models that externalizes factual knowledge to a knowledge base rather than storing it in model weights. “Co-LMLM: Continuous-Query Limited Memory Language Models” by Yair Feldman, Linxi Zhao, Nathan Godey, Dongyoung Go, Yilun Hua, Kilian Q. Weinberger, Jennifer J. Sun, and Yoav Artzi — three of whom are AI-MI senior personnel — pairs continuous vector queries with textual knowledge values so the model generates efficient queries while retrieved knowledge remains human-readable and attributable. Across Wikipedia and FineWeb-Edu pretraining at multiple scales, Co-LMLM outperforms prior memory-augmented and standard LLMs; at 360M parameters it reaches lower perplexity than models pretrained on 40× more data.
From Feldman et al., “Co-LMLM: Continuous-Query Limited Memory Language Models,” arXiv:2607.07707 (2026). Used under arXiv non-exclusive license.
Source: arXiv:2607.07707