Managing Small Context Windows in Language Models

In this article, you will learn three practical strategies for managing small context windows in large language models, along with working Python examples that demonstrate how two of those strategies are implemented.

Topics we will cover include:

How context truncation via the sliding window approach keeps token usage flat and predictable.
How token budgeting combined with retrieval-augmented generation ensures only the most relevant context fits within a prompt.
A concise overview of additional strategies for more specialized use cases — rolling summaries, prompt compression, and observation masking.

Introduction

Top-tier AI industries have become somewhat obsessed with language models capable of ingesting massive context windows, e.g. an entire book in a single prompt. However, what they won’t admit easily is that in real-world LLM applications, these massive context windows come with various limitations and challenges, including soaring API costs, unacceptable response times, and even worse, the so-called “lost in the middle” problem whereby a model ignores data deeply buried in the middle of the giant prompt. No surprise, then, that working with small yet smartly managed context windows could yield superior outcomes, reducing latency, minimizing costs, and forcing the model to concentrate on what truly matters to generate its response.

This article unveils three of the most widely adopted practical strategies for managing and mastering small context windows in language models, along with examples that mimic the implementation of some of them for better understanding.

Context Truncation: Sliding Window

There is a consensus that sliding windows are arguably the most common and simplest strategy for managing shortened context windows in language models. Instead of providing an entire user conversation history to the model, the context is treated as a FIFO (First-In-First-Out) queue: as new interactions (exchanged messages) come in, the oldest ones are simply dropped. All it takes is defining the size of the context window and striking a balance between sufficient past context retention and latency-cost control.

The main advantage of truncating the context via sliding windows is absolute control and predictability over token usage and computing overhead. The maximum number of interactions dealt with by the model at a given time remains fixed, keeping latency flat and surprise-free.

To better understand how this approach works, let’s look at the following Python code in which you can freely adjust the value of max_turns (context window size) and see how it affects the “memory” injected into the current prompt:

| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | class SlidingWindowMemory: def init(self, max_turns=3): """Keep only the last max_turns of a conversation.""" self.max_turns = max_turns self.history = [] def add_interaction(self, user_text, ai_text): self.history.append({"user": user_text, "ai": ai_text}) # The logic behind a sliding window: drop the oldest turns if limits are surpassed if len(self.history) > self.max_turns: self.history = self.history[-self.max_turns:] def build_prompt(self, new_query): prompt = "System: Answer concisely based on recent context.\n\n" for turn in self.history: prompt += f"User: {turn['user']}\nAI: {turn['ai']}\n" prompt += f"User: {new_query}\nAI:" return prompt # --- Testing the Sliding Window mechanism: feel free to adjust the value of max_turns --- memory = SlidingWindowMemory(max_turns=2) # Simulating a long conversation memory.add_interaction("Hi, I'm learning Python.", "Great choice!") memory.add_interaction("What are lists?", "Lists are mutable arrays.") memory.add_interaction("Can they hold mixed types?", "Yes, they can.") # The prompt will only contain the last 'max_turns' interactions, saving tokens print(memory.build_prompt("How do I append to one?")) |

Output:

| 1 2 3 4 5 6 7 8 | System: Answer concisely based on recent context. User: What are lists? AI: Lists are mutable arrays. User: Can they hold mixed types? AI: Yes, they can. User: How do I append to one? AI: |

You can also try extending the conversation history by appending new memory.add_interaction() calls with extra query-response pairs of your own, to test the mechanism for larger context windows.

Token Budgeting and RAG (Retrieval-Augmented Generation)

RAG systems supplement LLMs with engines that reference and retrieve external documents to enrich the original user prompt with founded, relevant context. Small context windows may intuitively force a ruthless attitude toward the data to include in the context. To address this, token budgeting splits the context window into zones with strict limits per zone. For instance, a token budgeting criterion could allow up to 20% of the context for system instructions, 20% for the chat history (including the latest user query), and the remaining 60% for retrieved data. This incorporates a more dynamic retrieval and data chunking behavior, halting insertion as soon as budget limits are hit.

The main advantage of token budgeting is preventing unduly large retrieved documents from quickly exhausting the prompt and ensuring only highly relevant, concentrated information is included, thus avoiding side issues like the aforementioned “lost in the middle” problem.

This code excerpt exemplifies the use of the mechanism in Python, using a simple word count as a free, lightweight proxy for token budgeting — to make it more realistic, you could consider the commonly accepted heuristic of 1 word = 1.3 tokens on average. The loop inside the function shows how to reliably pack a prompt without surpassing enforced limits:

| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | def build_budgeted_prompt(system_prompt, retrieved_chunks, user_query, max_words=50): """Packs context chunks into a prompt until a strict word budget is hit.""" # Calculating the fixed cost of mandatory elements base_words = len(system_prompt.split()) + len(user_query.split()) current_words = base_words included_chunks = [] for chunk in retrieved_chunks: chunk_words = len(chunk.split()) # Only add the chunk if it fits within the strict budget if current_words + chunk_words <= max_words: included_chunks.append(chunk) current_words += chunk_words else: print(f"Budget hit! Left out {len(retrieved_chunks) - len(included_chunks)} chunks.") break context_str = "\n---\n".join(included_chunks) return f"{system_prompt}\n\nContext:\n{context_str}\n\nUser: {user_query}" # --- Testing the Budgeted Prompt Mechanism --- system_msg = "Use the context to answer." query = "What is the capital of Spain?" docs = [ "Seville is a city in Andalusia, Spain.", "Madrid is the capital of Spain.", # We want this to fit "Spain is located in Southwestern Europe.", # This might get cut off "The population of Spain is roughly 47 million." ] # Setting a very small budget to see the cutoff in action print(build_budgeted_prompt(system_msg, docs, query, max_words=30)) |

Output:

| 1 2 3 4 5 6 7 8 9 10 11 | Budget hit! Left out 1 chunks. Use the context to answer. Context: Seville is a city in Andalusia, Spain. --- Madrid is the capital of Spain. --- Spain is located in Southwestern Europe. User: What is the capital of Spain? |

Beyond the Basics: Other Strategies

To close out, let’s quickly outline some other strategies for managing small context windows, particularly for specialized use cases. Be aware that some of these strategies typically require live API calls or additional external dependencies for their implementation.

Rolling Summaries:This method uses an auxiliary LLM for summarization that condenses older conversation history into a compact paragraph, replacing the raw prompt text. It helps retain long-term memory without token bloat, but requires extra API calls to request and obtain the summaries, introducing added overhead and potential costs.
Prompt Compression:Instead of resorting to an auxiliary model, an algorithm is invoked to strip out filler words, redundant data, and stop words from the raw context before feeding it to the main model. This can drastically reduce latency without compromising input quality or semantic intent, but if applied too aggressively, it could strip away subtle yet valuable nuances needed by the model to generate an acceptable response.
Observation Masking:This approach evaluates the context to hide or mask older, structural noise — such as database queries in agent-based systems or intermediate code execution logs — while the core logic remains intact. It is a popular technique in autonomous agents fueled by LLMs, allowing them to stay focused on their immediate goal without being distracted by past internal steps. However, it is more complex to implement, as it requires determining which observations are safe to mask without compromising the agent’s reasoning chain.

Closing Remarks

Small context windows shouldn’t be regarded as a limitation but rather as an architectural feature for preventing major issues like excessive cost and latency. This article presented a number of strategies for effectively managing small context windows in LLMs to yield faster and cheaper solutions without compromising accuracy.