Context Engineering for AI Agents: Managing the Finite Resource That Determines Success

Andrej Karpathy described LLMs as "a new kind of operating system," with context functioning as RAM. Poorly managed context causes hallucinations and forgotten instructions--just like a computer thrashing with insufficient memory.

This guide covers four battle-tested patterns, real cost analysis, working code, and metrics frameworks.

Understanding Context as a Resource

Context isn't a single thing--it's a complex allocation problem. Every agent juggles four distinct types, each competing for the same finite token budget:

The Context Budget Mental Model

Think of context as a budget, not a container. Every token has a cost--both literal (API pricing) and functional (attention dilution). The question isn't "does this fit?" but "is this worth its cost?"

Model	Input (per 1M tokens)	Output (per 1M tokens)	100K Context Cost
Claude Opus 4.5	$5.00	$25.00	$0.50 input
Claude Sonnet 4	$3.00	$15.00	$0.30 input
Claude Haiku 4.5	$1.00	$5.00	$0.10 input
GPT-4o	$5.00	$20.00	$0.50 input
GPT-4o mini	$0.60	$2.40	$0.06 input
DeepSeek-V3	$0.28	$1.10	$0.03 input

Pricing as of January 2026. Cache hits typically reduce input costs by 90%.

A single conversation that accumulates 100,000 tokens of context costs between $0.03 and $0.50 per turn on input alone. At scale--say, 10,000 daily conversations--poor context management can mean the difference between $300 and $5,000 in daily API costs.

Context Rot: The Hidden Performance Killer

Contrary to what tutorials suggest, larger context windows don't mean better performance. Chroma's research on "Context Rot" shows LLM accuracy degrades significantly as input length increases.

Bottom line: Stuffing more context into prompts isn't a strategy--it's a liability.

Context Engineering Patterns

Four essential patterns have emerged from production deployments:

Pattern 1: Context Compression

Compression reduces context size while preserving essential information. The goal is maintaining signal while reducing tokens.

When to use: Long-running conversations, extensive tool outputs, accumulated history.

When to avoid: When exact wording matters (legal documents, code review, precise quotes).

The most practical implementation combines a buffer for recent messages with summarization for older history:

from langchain.memory import ConversationSummaryBufferMemory
from langchain_anthropic import ChatAnthropic

def create_compressed_memory(max_recent_tokens: int = 4000):
    # Keep recent messages verbatim, summarize older ones
    llm = ChatAnthropic(
        model="claude-sonnet-4-20250514",
        max_tokens=500
    )

    memory = ConversationSummaryBufferMemory(
        llm=llm,
        max_token_limit=max_recent_tokens,
        return_messages=True,
        human_prefix="User",
        ai_prefix="Assistant"
    )
    return memory

# Usage
memory = create_compressed_memory(max_recent_tokens=4000)

Production tip: Use a cheaper, faster model for summarization. Summarizing with Haiku at $1/million tokens instead of Opus at $5/million tokens reduces compression overhead by 80% with minimal quality loss.

Pattern 2: Context Prioritization

Not all context is equally valuable. Context value is dynamic--what's relevant changes based on the current query.

When to use: Multi-turn conversations, multiple retrieved documents, competing context sources.

A robust priority system scores items on multiple dimensions:

from dataclasses import dataclass, field
import time

@dataclass
class ContextItem:
    content: str
    source: str   # "conversation", "tool", "retrieval", "system"
    token_count: int
    created_at: float = field(default_factory=time.time)
    relevance_score: float = 0.5
    importance: float = 0.5

    @property
    def recency_score(self) -> float:
        age_minutes = (time.time() - self.created_at) / 60
        return max(0.1, 1.0 - (age_minutes / 60))

    @property
    def priority(self) -> float:
        return (self.relevance_score * 0.5 +
                self.recency_score * 0.3 +
                self.importance * 0.2)

Production tip: Track which context items are excluded and why. This data reveals patterns--if conversation history is consistently dropped, your system prompt might be too long. If retrieved documents are excluded, your retrieval is returning too much.

Pattern 3: Context Externalization

Move information outside the context window while keeping it accessible--the foundation of RAG systems, but extends beyond document retrieval.

When to use: Information occasionally needed but not required every turn, historical data, reference material.

Pattern 4: Context Segmentation

Break complex tasks into context-bounded subtasks, each handled by a specialized agent with a clean context window.

When to use: Multi-step workflows, tasks requiring different expertise, long-running agents.

Key principle: Each segment should complete its task with minimal context from previous segments. If extensive prior context is needed, you've drawn boundaries wrong.

Production Challenges and Solutions

Theory is clean; production is messy. Here are common challenges:

Measuring Context Effectiveness

What gets measured gets managed. Here are the core metrics:

Implementation Guide

A practical 5-step path to implementing context engineering:

Common Implementation Mistakes

Conclusion

Context engineering is the primary determinant of whether your agent succeeds or fails in production. The four patterns--compression, prioritization, externalization, and segmentation--transform context from a source of mysterious failures into a well-understood, optimizable system.

Context engineering is an ongoing discipline with substantial payoffs: agents that maintain coherence, costs that scale predictably, and failures that trace to understandable causes.