US patent US12769728
Better inference pattern for long context retrieval
Abstract
The present disclosure relates to a method and system for enhancing inference in large language models (LLMs) over long input sequences. A segmented inference strategy may be employed, wherein the long context can be divided and sequentially processed through a key-value (KV) cache of the LLM. At each step, the model may generate auxiliary outputs (or margins), which may include extractive summaries or intermediate signals based on the segment's relevance to an instruction. These margins may then be classified and selectively retained to guide final inference on the instruction. The retained margins may be prepended to the instruction to facilitate improved generation without modifying the model's internal weights. The disclosed approach provides efficient localization of relevant content, improves comprehension of extended contexts, and reduces computational overhead. Moreover, the disclosed technique is particularly effective for retrieval-based NLP tasks and supports long-context reasoning in LLMs while enhancing inference efficiency and user experience.