The Architecture Tax: How Solution and Context Engineering Shape Your AI Costs
Why the biggest driver of your AI spend isn’t the model you choose. It’s the architecture behind it.
ABOUT THIS PAPER
Most organizations evaluating AI costs compare model prices when the bigger lever is architecture. This paper breaks down the two disciplines that set your AI bill — Solution Engineering and Context Engineering — and grounds the argument in an internal Radiant Digital benchmark that found a 3.8× cost gap and a 40-point accuracy gap between the best and worst architecture, on the exact same model.
Executive Summary
Most organizations evaluating AI costs start in the wrong place: they compare model prices, not architectures.
But two decisions made long before a single prompt is sent, how information is stored and retrieved, and how it is shaped for each request, often account for a bigger swing in cost than the model itself. Get these wrong, and even the cheapest model becomes expensive at scale. Get them right, and even a premium model stays affordable.
This paper breaks down the two disciplines behind that decision, Solution Engineering and Context Engineering, and shows how choices like RAG, KAG, or full-metadata retrieval directly translate into your AI bill. We back the argument with results from an internal Radiant Digital benchmark, a 20-question, three-architecture test of an LLM-powered SQL generation system, that found a 3.8x token cost gap and a 40-percentage-point accuracy gap between the best and worst architecture, using the exact same underlying model. Anyone can deploy AI. Deploying AI that is economical at scale takes deliberate Solution and Context Engineering.
The Problem: Comparing Prices, Not Architectures
When enterprises budget for AI, the conversation almost always starts with a spreadsheet comparing per-token prices across model providers, sorted by dollars per million tokens. It is a natural starting point: the numbers are public, easy to compare, and feel like the lever most directly under a buyer’s control.
It is also the wrong place to start. Model price is one input in a much larger equation. The bigger variable is how many tokens your system sends to the model in the first place, and that number is set almost entirely by architecture decisions made during design, not by which vendor you picked.
Consider two systems built on the identical model, answering the identical question. One retrieves a tightly scoped, relevant slice of context before generating a response. The other dumps everything it might conceivably need and lets the model sort it out. In Radiant Digital’s own internal benchmark of an LLM-based SQL generation system, that difference alone produced a 3.8x gap in tokens consumed per query before accuracy is even considered. At scale, that gap is not a rounding error; it is millions of dollars a year.
This is the pattern behind most AI cost overruns: teams optimize the wrong variable. They negotiate a better per-token rate, or downgrade to a cheaper model, and are surprised when the bill barely moves because the volume of tokens flowing through the system was never addressed. The architecture set the ceiling long before the model was chosen.
Two Decisions That Set Your Ceiling
Two architectural decisions determine AI spend more than model selection does. Understanding them, and the difference between them, is the foundation for controlling AI costs at scale.
Solution Engineering is the upfront architectural decision of how information is stored, indexed, and made retrievable for an AI system (RAG, KAG, full-metadata approaches). It sets the cost ceiling for every query that follows.
Context Engineering is the practice of shaping what gets sent to the model per request, including trimming, formatting, and prioritizing so the model gets only what it needs. It determines how close spend gets to the ceiling.
Think of Solution Engineering as choosing the size of the pipe, and Context Engineering as deciding how much water flows through it on any given day. A well-designed retrieval architecture (a narrow, efficient pipe) stays cheap even under sloppy prompt construction. A poorly designed one (a wide, wasteful pipe) stays expensive no matter how carefully individual prompts are trimmed.
The Bottom Line
Get both wrong, and even a cheap model gets expensive at scale. Get both right, and even a premium model stays affordable.
Deep Dive: Solution Engineering — RAG, KAG, and Full-Metadata Retrieval
Solution Engineering answers one question: what should the AI know before it responds, and how does it get that information? There are three common approaches in production systems today.
Retrieval-Augmented Generation (RAG)
Stores information as vector embeddings and retrieves the chunks that are semantically similar to the incoming query. It is fast to implement and cheap to run, because it only pulls a narrow slice of content. Its weakness is that similarity is not the same as relevance. RAG can miss information that is structurally necessary to answer a question but doesn’t share obvious vocabulary with it, particularly in multi-step or multi-table reasoning.
Knowledge-Augmented Generation (KAG)
Stores information as a graph of explicit relationships, with entities connected to related entities and records connected to the records they depend on, rather than as isolated similarity clusters. When a query arrives, the system traverses the graph to pull exactly the connected information relevant to that query, including relationships that would never surface through similarity search alone.
Full-Metadata Retrieval
Takes the brute-force approach: pass the AI everything, including every record, every field, and every relationship on every request, and let the model figure out what matters. It requires the least design effort up front, which is precisely why it is so common in early-stage systems, and precisely why it becomes the most expensive one to operate.
The cost and accuracy differences between these three approaches are not theoretical. Radiant Digital ran all three architectures on the same underlying model, against the same 20 benchmark questions, inside Talk2Data, our LLM-powered SQL generation system, using the BIRD EX (Execution Match) standard, where generated SQL must return the exact same result set as a human-written reference query.
Three architectures, one model, 20 benchmark questions — BIRD EX (Execution Match) standard.
Three findings stand out. First, full-metadata retrieval is the worst of both worlds: 3.8 times the token cost of the knowledge-graph approach, for roughly half the accuracy. Giving a model “everything” does not make it smarter. It creates context overload, where the signal is buried in undifferentiated volume and the model cannot reliably tell what matters. Second, RAG is nearly as cheap as the knowledge-graph approach on a per-token basis, but its retrieval ceiling shows up directly in accuracy: at 35%, similarity search reliably misses schema relationships that are not lexically obvious from the question text. Third, the knowledge-graph approach won on both dimensions simultaneously. It was the cheapest-per-accurate-answer architecture by a wide margin, and it uniquely answered nine questions that neither of the other two approaches got right under any circumstance.
That last point matters most for how you should think about architecture selection: the knowledge-graph advantage was not marginal or incremental. It was structural. Cost and accuracy are frequently treated as a trade-off: pay more, get a better answer. This benchmark is a reminder that a well-chosen architecture can beat that trade-off entirely, delivering both a lower ceiling and a higher accuracy floor at once.
The practical implication is not “always use knowledge graphs.” It is that Solution Engineering decisions must be evaluated on both axes together, cost and accuracy, because default choices like full-metadata retrieval that feel simplest to build are frequently the most expensive to run and the least reliable to use.
Deep Dive: Context Engineering — Closing the Gap to the Ceiling
Solution Engineering sets the ceiling. Context Engineering determines how close actual spend gets to it, and it operates at the level of every individual request, not the system design as a whole.
Trimming. Even a well-architected retrieval system can over-fetch. Trimming means pruning retrieved content down to what is demonstrably needed for the specific question being asked, rather than passing along an entire retrieved block because it was convenient to fetch that way.
Prioritization. Not all context is equally valuable to a given query. Prioritization ranks retrieved content and puts the highest-value information where the model is most likely to weight it heavily, instead of treating every retrieved element as equally important.
Formatting. How context is structured affects both token count and model comprehension. Dense, well-structured formats, such as tables, key-value pairs, and compact schemas, consistently outperform verbose prose representations of the same information. They use fewer tokens in and often produce better comprehension out.
Caching. Static or slowly changing context, such as system instructions, schema definitions, and reference documentation, does not need to be resent and reprocessed on every single call. Caching this content is one of the highest-leverage cost levers available, because it converts a per-query cost into a largely fixed one.
Prompt compression. Techniques that summarize, deduplicate, or otherwise compress context before it reaches the model reduce token volume directly, at the cost of some engineering complexity to implement well.
The Talk2Data benchmark also illustrates a subtler point about Context Engineering: more tokens spent is not automatically waste, and fewer tokens is not automatically better. It depends on where those tokens go. The knowledge-graph pipeline produced noticeably longer, richer answers than the other two architectures, with 544 output tokens on average, versus 195 for RAG and 181 for full-metadata, because its structured, relationship-aware input context gave the model better material to reason with. That is Context Engineering working as intended: additional tokens spent on the part of the exchange that improves the answer, not on undifferentiated bulk.
The Order of Operations
Context Engineering cannot fix a poor Solution Engineering decision. No amount of prompt trimming will make full-metadata retrieval cheap. But on top of a well-designed retrieval architecture, disciplined context engineering is what determines whether an organization operates near its cost ceiling or comfortably below it.
Putting It Together: A Cost Framework
Before scaling an AI system, five questions surface most of the architecture decisions that will determine your operating cost:
- What does our retrieval architecture actually fetch per query, and is that scoped to relevance or to convenience? If the honest answer is “everything we have,” that is a full-metadata pattern, and the cost consequences compound with every query thereafter.
- Does our retrieval method understand relationships, or only similarity? If your data has meaningful structure, including dependencies, hierarchies, and joins, similarity-based retrieval alone will systematically miss the queries that depend on that structure.
- What fraction of the context we send is static, and is it cached? Repeated system instructions, schema, or reference material sent fresh on every call is money left on the table.
- Are we measuring cost and accuracy together, or separately? A cheaper-per-token approach that requires more retries, more escalations, or produces wrong answers more often is not actually cheaper.
- Would this architecture’s cost curve still make sense at ten times today’s query volume? Architecture decisions compound; test them against the volume you expect to have, not the volume you have today.
An architecture that answers these five questions well sets a low ceiling. Disciplined context engineering practices then keep actual spend close to that ceiling, rather than letting it drift upward as the system scales.
Case Example: Same Model, Different Architecture, Different Bill
To make the framework concrete, consider Radiant Digital’s own internal benchmark for Talk2Data, our LLM-powered SQL generation product. The test held the model constant and varied only the Solution Engineering architecture, including full-metadata retrieval, similarity-based RAG, and a knowledge-graph (KAG) approach, across the same 20 real-world business questions.
At varying operating volumes, the SQL-generation-stage token cost alone (before accounting for downstream answer generation) works out as follows:
SQL-generation-stage token cost only. Reference pricing: GPT-4o input, $2.50 / million tokens.
At every volume tested, the full-metadata architecture cost 3.8 times more and delivered 40 percentage points less accuracy than the knowledge-graph architecture, using the identical underlying model in every case. This was not a performance trade-off where the more expensive option bought better results. It was strictly worse on both dimensions, and the gap was entirely attributable to a Solution Engineering decision made once, at design time.
The lesson generalizes well beyond SQL generation: the architecture decided before a single production query runs is the single largest lever an organization has over its AI operating cost. In this case, it was larger than a nearly 4x difference in model list price would have been.
Why Radiant Digital
Anyone can call a model API. Fewer teams have the discipline to design the retrieval architecture and context-shaping practices that make that API call economical at scale, and fewer still have the benchmark data to prove which choices actually work rather than which choices merely sound reasonable.
Radiant Digital’s Tokenomics framework treats LLM token consumption as a first-class commercial variable, a measurable, optimizable cost unit alongside compute and storage, in the design of enterprise AI systems, rather than an afterthought discovered after a surprising invoice. Talk2Data’s knowledge-graph architecture is a direct, production application of that principle, validated against a rigorous internal benchmark rather than a vendor’s marketing claim.
If your organization is running or piloting AI systems and finding that costs are scaling faster than expected, the underlying cause is very often architectural, not the model. Radiant Digital specializes in diagnosing that gap, auditing Solution Engineering decisions, redesigning retrieval architectures, and implementing the context-engineering discipline that keeps actual spend close to the ceiling those decisions set.
Conclusion
Anyone can deploy AI. Deploying AI that is economical at scale takes deliberate Solution and Context Engineering. The architecture decided before a single prompt is sent, including how information is stored and retrieved, and how it is shaped for each request, sets the ceiling on what your AI systems will cost, far more than the model you choose to run them on. Get these two decisions right, and a premium model stays affordable. Get them wrong, and even the cheapest model becomes expensive at scale. That is the architecture tax, and it is paid whether or not you choose to see it.
Radiant Digital, AI Practice. This paper draws on Radiant Digital’s internal Talk2Data benchmark study (20 golden questions, BIRD EX evaluation standard, May 2026).
▶▶ Anyone can deploy AI. Deploying AI that’s economical at scale takes deliberate Solution and Context Engineering — and that’s exactly where Radiant Digital specializes.
