The Running Costs of AI That Nobody Puts in the Business Case

The Number That Fools Everyone

Between November 2022 and October 2024, the cost of running a model at GPT-3.5's level of capability fell from $20 per million tokens to $0.07. That's Stanford's figure, from the 2025 AI Index, measured at fixed performance. A 280-fold drop in under two years.

Over the same period, enterprise AI bills went up. Spending on generative AI software jumped from $11.5 billion in 2024 to $37 billion in 2025, and the companies posting those numbers were mostly the same ones benefiting from the price collapse.

Both things are true at once, and the gap between them is where AI budgets die. Unit prices are falling. Total consumption is growing faster than prices fall. Plan around the falling number and ignore the growing one, and you'll be over budget by spring without understanding why.

That's the trap. Not whether AI is worth doing — it usually is. The problem is that the cost that gets approved is the pilot's cost, and the cost that arrives is production's cost, and those two numbers live in different universes.

Why the Pilot Number Is Always Wrong

A pilot runs a few hundred requests through a model while someone watches and tunes it, then reports back that it works and the bill was trivial. That report is accurate and nearly useless, because almost none of the real cost was present during the pilot.

A planning rule keeps surfacing across enterprise cost research: the first year of running an AI system costs roughly what building it cost, and three-year total cost of ownership lands at two to three times the build. Independent 2026 analyses converge on that same two-to-three-times figure for the gap between what organizations budgeted at greenlight and what the system actually cost live.

The build is the cheap half. The expensive half is inference, maintenance, governance, evaluation, and the human labor around the system — none of which shows up in a demo, because a demo doesn't run for three years against real traffic.

The deeper reason is that AI inverted the normal software cost curve. Traditional software is expensive to build and cheap to run: you pay engineers once, then it executes for near-zero marginal cost. AI flips that. The foundation models did the expensive part already, so building is cheaper than ever — but every request costs money, forever, and that cost scales with usage. Morgan Stanley projects inference will be 70 to 80 percent of AI compute spending by 2027. You aren't buying a tool. You're signing up for a utility bill that grows with adoption.

The Five Costs That Arrive After Launch

They cluster into five categories. None is the model, and all are routinely left out of the business case.

Inference at real volume. A conversational system serving a million requests a month runs $5,000 to $15,000 monthly. Push it to a hundred million requests and you're past $100,000. The per-request cost is tiny; the volume is not, and volume is exactly what you were hoping for. Success is what makes this line expensive.

The agent tax. This is the one that surprises people most, so here's the mechanism rather than hand-waving about "token multiplication." A chatbot sends one message and gets one answer — bounded and cheap. An agent runs a loop: plan, retrieve, call a tool, read the result, check its work, often several times over. And a language model reprocesses the entire conversation on every step. So by the tenth turn, each call carries the original documents, every tool result, and every intermediate thought before it. Cost doesn't add up turn by turn. It compounds. A team modeling agent cost as "steps times cost per step" underprices their system by three to five times. Published estimates put the agent tax at 5x to 50x the token cost of a simple query. EY's illustration: a customer service interaction that cost about four cents in a 2023-style workflow runs around $1.20 as a 2026 agentic system doing tools, reasoning, and retries — roughly thirty times more for what looks like the same interaction. This isn't only a small-company problem. Uber's CTO said the company's AI coding budget was blown well ahead of schedule. Microsoft reportedly pulled back AI coding licenses months after rolling them out. The head of the FinOps Foundation said companies started calling in April 2026 to report they were already three times over their full-year AI budget.

Maintenance and model churn. Prompts drift, integrations break when an API changes, and models get deprecated or quietly updated behind a stable name — so the system you tested in January isn't the one running in June, even if your code never changed. Budget 20 to 30 percent of the build cost every year just to hold performance steady. A $500,000 system implies $100,000 to $150,000 annually. That's the cost of standing still.

Evaluation and monitoring. The line most often cut, and the one most tied to failure. Without an evaluation harness you can't tell when a model silently degrades, and silent degradation is the default failure mode of production AI. Tooling runs $20,000 to $100,000 a year. It feels skippable; it's actually the cheapest insurance in the stack, because the alternative is learning about a regression from a customer instead of a dashboard. I've written about why systems break at this exact seam in [Why Your LLM Works in the Demo and Breaks in Production](https://nexhubai.com/insights/llm-production-failure-lessons).

Governance, security, and the human in the loop. Model risk, compliance, and security monitoring typically run $30,000 to $100,000+ a year, scaling with how regulated your data is rather than your headcount. The subtler cost is the human-in-the-loop tax: any system whose output carries real consequences needs a person reviewing the consequential actions, and that review is a permanent operating cost that eats directly into the labor savings the project was justified on. If an agent handles part of a person's job but you still employ the person to check it, the honest ROI is the difference, not the whole. That's the calculation that gets skipped in the enthusiasm of a green light.

The Cost Nobody Owns

There's a whole category of AI spend that never appears as an AI line item, because it's hiding inside software you already pay for. Your CRM added AI features and raised its price. Your help desk tool has an AI tier. Your analytics platform charges for AI-assisted queries. None of it shows up when someone asks "what are we spending on AI," because it's baked into SaaS renewals different teams own.

Two consequences. Your true AI spend is almost certainly higher than your AI budget, because a chunk is dispersed across contracts nobody classifies as AI. And it's usage-driven rather than seat-driven, which breaks the mental model finance teams use for software — you're used to paying per person, but AI costs scale with how much work the software does. Ramp's data showed heavy corporate spenders seeing monthly costs spike more than 50 percent in roughly one month out of four. Try forecasting against that.

The move is to stop asking "what's our AI budget" and start asking "what's our all-in cost per outcome" — per ticket resolved, per claim processed, per call deflected. That's the number that tells you whether the economics work, and the one vendors would rather you didn't calculate.

What Good Cost Discipline Looks Like

The point isn't to scare you off AI. The unit economics are genuinely improving, and for the right workflow the returns are real. The point is that the difference between a project that pays off and one that quietly bleeds money is almost never the model — it's whether anyone modeled the running cost honestly before committing.

Model cost per outcome, not cost per token. Tokens are an input. The business cares about the fully loaded cost of one resolved ticket, inference plus review plus maintenance amortized in. If you can't state that number, you can't tell whether the project works.

Match the model to the task. Not every step needs a frontier model. Routing the easy 80 percent to a cheap, fast model and escalating only the hard 20 percent routinely cuts costs by more than half with little quality loss. Classification, extraction, and summarization run fine on budget-tier models. Defaulting everything to your most expensive model is the single most common source of waste.

Put a ceiling on the loop. The agent tax scales with how much looping and context re-injection you allow. Cap iterations. Trim what context carries forward instead of blindly accumulating it. Much of a runaway agent bill is re-sent context that didn't need re-sending.

Instrument spend before you scale it. You can't govern what you can't see. Attribution — knowing which workflow or agent is burning tokens — turns a shocking monthly bill into a specific, fixable line. Ninety-eight percent of FinOps practitioners now manage AI spend as part of their remit, up from under a third two years ago. That discipline arrived because the bills forced it.

Budget the run, not just the build. First-year run cost roughly equals build cost; three-year total is two to three times the build. If a proposal shows you a build number and goes quiet on the run, it's not a complete proposal. Ask what inference, maintenance, evaluation, and review cost at the volume you're actually hoping for. If nobody can answer, that's the finding.

The Bottom Line

AI is getting cheaper per unit and more expensive per organization at the same time, and both trends will hold. The falling price is real and it's why building on AI makes sense now in ways it didn't three years ago. But cheaper tokens don't make cheaper systems if you use dramatically more of them, and the shift toward agents and retrieval means you almost certainly will.

The businesses that come out ahead won't be the ones that found the cheapest model or the ones scared off by a runaway-bill horror story. They'll be the ones that treated AI as what it is — not a tool you buy once, but a utility you run continuously, with a meter attached and a cost per outcome that either works or doesn't. Get that number right before you scale and most of the nasty surprises never happen. Skip it and you'll meet all five of these costs anyway. Just later, and in front of your CFO.

The demo is the easy part; everything expensive lives in the gap between the demo and the durable system. [Why Most AI Pilots Never Reach Production](https://nexhubai.com/insights/why-ai-pilots-fail-production) covers the operational side of that same divide, and [A Practical Guide to AI Automation for Service Businesses](https://nexhubai.com/insights/ai-automation-practical-guide-service-businesses) covers where to start so the economics are on your side from the beginning.

Frequently Asked Questions

Why do AI bills keep rising if inference costs are falling?

Unit price and total spend are different numbers moving in opposite directions. The cost to run a fixed amount of AI dropped 280-fold for GPT-3.5-level performance between late 2022 and late 2024, but total consumption is growing faster than prices fall, and reasoning and agentic models burn far more tokens per task than simple chat. Cheaper per token doesn't mean cheaper overall. Budgeting on the falling unit price alone is the most common way teams end up over budget.

What is the "agent tax"?

The sharp jump in token cost when you wrap a task in an agentic workflow instead of a single query. An agent plans, retrieves, calls tools, checks its work, and retries, and because the model reprocesses the whole context on every step, cost compounds rather than adding up linearly. Estimates put the multiplier at 5x to 50x the cost of a simple query. Modeling agent cost as steps times cost-per-step underprices the real bill by three to five times.

How should I estimate the true cost of an AI system before building it?

Anchor on run cost, not build cost. First-year running cost roughly equals build cost, and three-year total cost of ownership lands at two to three times the build. Inference at real volume, maintenance, evaluation, governance, and human review make up the rest, and they recur every year. If a quote only covers the build, it's describing about half of what you'll actually pay.

What's the single most effective way to control AI running costs?

Match the model to the task instead of defaulting everything to your most expensive model. Routing routine work to cheaper budget-tier models and escalating only genuinely hard steps routinely cuts costs by more than half with little quality impact. Pair that with a cap on agent loop iterations and real visibility into which workflows are spending, and you've addressed most of the runaway-cost risk.

Does AI spend include costs outside of API bills?

Yes, and this is where most estimates fall short. Beyond direct inference: annual maintenance (20 to 30 percent of build cost), evaluation and monitoring tooling ($20,000 to $100,000+ a year), governance and security ($30,000 to $100,000+ a year), the human labor to review consequential outputs, and AI features bundled into SaaS tools you already pay for. That last category often doesn't appear in any "AI budget" at all, which means your true spend is usually higher than your tracked spend.