Google’s new technique gives LLMs infinite context

Infini-attention is a technique introduced by Google researchers that allows large language models (LLMs) to process text of infinite length by extending their context window while keeping memory and compute requirements constant. It works by keeping the classic attention mechanism in the transformer block and adding a compressive memory module. This module stores old attention states once the input grows beyond its context length, maintaining a constant number of memory parameters for computational efficiency. The final output is computed by aggregating the compressive memory and the local attention contexts.

Increasing context length is important for LLMs because it allows the models to handle longer sequences of text without a drop in performance. This is crucial for organizations customizing LLMs for their applications, as they often insert bespoke documents and knowledge into their prompts. A longer context length enables LLMs to maintain quality over longer inputs, which is a significant advantage in improving models and gaining a competitive edge.

Infini-attention has shown to outperform other long-context transformer models in experiments. For instance, in long-context language modeling, it maintained lower perplexity scores, which is a measure of the model’s coherence, while requiring 114x less memory. Additionally, in the 'passkey retrieval' test, Infini-attention successfully returned a random number from a text of up to one million tokens and outperformed other techniques in summarizing tasks for texts of up to 500,000 tokens.

LLMs with infinite context have several potential applications, including the creation of custom applications without extensive engineering efforts. They could allow users to insert all relevant documents into the prompt, letting the model choose the most pertinent parts for each query. Additionally, they could be used to provide the model with a long list of examples to improve its performance on specific tasks without the need for fine-tuning. This could lower the barrier to entry for developing prototypes and applications quickly.