Beyond Prompting: A Dive into Context in AI

When AI tools gained popularity, there was one big concern — can AI do all of software engineering on its own? I think the answer is still no.

When I first tried using an LLM, it felt like using an oracle. The responses were concise, clear, and saved time. I soon realized I felt that way mainly because I used it for simple queries — syntax, minor bugs in a short snippet, or writing small functions. Eventually, I started seeing the true fallacies in it when I began using it in larger, more creative systems.

Have you ever felt like LLM responses get more inaccurate or slower as a chat gets longer? If so, you've experienced large contexts and context degradation — some of the biggest bottlenecks in AI today. In real apps and systems, projects are large, creating demand for ever larger context. In this article, we'll discuss the need for context and what's hindering it today.

01 / The Basics

What is context?

Context is the working memory of an AI model. It contains the relevant background information and memory that a model can use to generate the response to a specific query. While using an LLM, this context can entail the current prompt, the previous conversation history, external sources, and other system instructions. In some models, context may also include video, audio, math, and images, which are encoded and tokenized in a way specific to the model. Relevant external sources for the most up-to-date information are injected into the context through Retrieval-Augmented Generation (RAG), by pulling from a vector database.

The context window, measured in tokens, is the maximum amount of relevant data that can be held at a time in the LLM's working memory. Context window sizes vary between LLMs, with some high-performing models from Claude and Gemini offering 1M+ token context windows.

Context is the working memory of an AI model — and like any working memory, it is finite, noisy, and easy to overwhelm.

02 / The Issues

Where context breaks down

Context overflow. In complex problems, conversations often get long and the context window capacity is exceeded, causing the model to drop older information. This can lead the model to provide incorrect responses or drift away from the originally intended goals.

The “lost in the middle” issue. Models usually don't rely on all parts of the context equally, and instead focus on the beginning and end of the context to generate responses. This is because the middle of the context often contains noise, with deviations and irrelevant information. It can cause the model to overlook important information that would have been relevant to the current response.

Unwritten context. AI models don't have access to any unwritten context — historical decisions, informal workarounds, or anything the user hasn't told the AI. This often causes agents to produce results that aren't aligned with what the user actually wants, even when the results are functionally correct. This is where models can lack human nuance: not every thought or detail can realistically be provided to an AI to make personalized decisions.

Performance costs. Long contexts can cause slow performance. Large contexts require large amounts of VRAM, often putting real strain on GPUs.

03 / Mitigations

How we work around it today

Context filtering. Removing noise from the context so it holds only the most relevant information, while also shortening the context. This keeps responses focused and helps the model stay on goal.

Grounding in human workflows. Using messages and specific code documentation to bring fragmented human workflows into the context while generating code, pushing the models to stay aligned with historical practices.

04 / Conclusion

AI still needs us

Today, AI is much deeper than prompts, and hence cannot do software engineering without human knowledge. AI is not perfect — like humans, AI has only finite memory, and it needs to make decisions where there isn't just one right answer. Human input is still needed to verify results and make informed choices about which AI outputs to keep, so that products stay aligned with our values.