Every team in your company is paying the same hidden AI tax for the same reason, and the CFO is often the only person positioned to see the whole bill.
The tax is context. Each time someone re-explains their business to a model, pastes the same background again, regenerates an answer that came back slightly off, or checks a figure the model had no way of knowing, the meter runs. The work looks productive, and every line of the invoice is justified.
The CFO is usually the one who has to answer for it. In Datarails’ 2026 CFO Sentiments Survey of 270 CFOs and finance leaders at US companies with more than 1,000 employees, 84% said they own their organization’s AI budget, and almost a third said their organization overspent it by at least 10% in the past year. This article explains where AI token costs really come from, the three controls that make them more predictable, how finance applies them first, and how the same approach extends to every other team.
Why cheaper tokens mean bigger AI bills
The price of AI keeps falling: in a 2025 analysis, Epoch AI found that the price of reaching a given level of model performance was falling between 9x and 900x a year, depending on the performance level. Yet many companies’ AI bills keep rising. On Palantir’s first-quarter 2026 earnings call, CTO Shyam Sankar put it in one line: “Tokens are the new coal.” He was invoking the Jevons paradox. When the Victorians built more efficient steam engines, many expected coal consumption to fall. Instead it soared, because cheaper power made new uses worthwhile.
AI is following the same curve. As each token gets cheaper, teams give AI more tasks, longer documents, and more attempts at the same answer. On the same call he added a line finance should hold onto: “More tokens means more slop.” Cheap tokens make it easy to generate output faster than anyone can check it, so spend rises fastest where the work is least controlled.
The way tokens are billed explains why. A token is a small chunk of text, often part of a word. Providers charge per million tokens, with separate rates for the input a model reads and the output it writes, and output costs more: on Anthropic’s published API pricing in October 2026, five times the input rate. On OpenAI’s API, reasoning models also bill their hidden reasoning as output tokens. In a chat or an agent, each new turn sends the conversation thus far back to the model. So the cost of a piece of work depends on how much context travels with it, how long the answers run, which model handles it, and how many attempts it takes.
That makes AI token costs a behavior problem as much as a pricing problem. Negotiating a better rate per token helps at the margin. Changing how many tokens each piece of work consumes is where the money is. For the mechanics of how usage compounds, see The CFO’s Guide to AI Cost Savings.
Where the waste comes from: the context tax
Finance shows the pattern clearly, because finance work runs on rules a general-purpose model doesn’t know. Take a monthly variance explanation:
- An analyst asks the model to explain the month’s variances.
- The answer uses a different definition of margin from the one the board sees, so the analyst explains the definition and asks again.
- The numbers refresh after a late journal entry, so the work is regenerated.
- One driver still looks invented, so someone checks it against the ledger by hand.
Every loop costs twice: once in tokens, and again in the time spent reviewing and reworking. Nothing in the loop is irrational. It is what happens when AI runs outside the business’s rules, so the team ends up serving the model instead of the other way round.
The review half of that cost is already large. Finance leaders in Datarails’ survey said their teams spend 26% of the working week verifying or correcting AI output. Among those whose data sat in disconnected systems, 76% spent more than a quarter of their time on it, compared with 31% of teams whose data was mostly centralized. Context the model lacks becomes both a token cost and a labor cost, and the second never appears on the AI invoice.
The three controls that make AI spend more predictable
Token tracking and usage caps are necessary, but they act after the fact. The controls that do more address the cause: the context the model is missing and the work it repeats.
| Control | What it means | What it removes |
|---|---|---|
| A governed core | The approved context a team works from, defined once: its data, definitions, rules, and permissions | Re-explaining the business in every prompt, and answers built on the wrong definition |
| Locked workflows | Recurring work set up once and run the same way every time, with the same inputs, format, and checks | Rebuilding the request from scratch and regenerating until it looks right |
| Tiered intelligence | Matching the model to the task: lighter models for routine work, more capable ones where the reasoning needs it | Paying top-tier rates for formatting, lookups, and first drafts |
The three reinforce each other. A governed core makes outputs consistent enough to lock into a workflow. A locked workflow makes the task predictable enough to route to the right model. And predictable tasks on governed context make AI spend far easier for the CFO to plan for. A governed core is not free: whatever context the model uses still reaches it as input tokens. It is worth paying for when it replaces longer loops of re-explaining and regenerating, and context that stays the same between calls can be cached: on Anthropic’s API, reading cached input costs a tenth or less of the standard input rate.
How finance applies the three controls
Finance is the natural place to start. Its context is among the most rule-bound in the company, a wrong answer is expensive, and the CFO who owns the AI budget sits on top of it.
- The governed core comes from FinanceOS, the data foundation of Datarails. FinanceOS connects to 600+ systems and data sources and applies consolidation, account mappings, FX conversion, and intercompany eliminations before any model sees a number. Metrics such as revenue and gross margin are defined once, and the same permissions apply to every report and AI tool. Users no longer have to explain the business in every prompt.
- Locked workflows are agents that sit where the work happens. Datarails’ Inflow Agents run defined jobs in place: Flux Analysis in Datarails Month-End Close quantifies reconciliation movements and drafts the explanation for the team to review. Custom Agents, built by Datarails’ Forward-Deployed Financial Engineers, can run recurring work on a schedule, such as a Monday review of departmental overspend.
- Tiered intelligence comes from keeping the model interchangeable. The FinanceOS AI Connector gives tools such as ChatGPT, Claude, and Copilot the same governed data and definitions. Because the context is maintained in FinanceOS rather than retyped into each prompt, a team can pick the model that fits the task without rebuilding that context.
The result is fewer loops. The model starts from consolidated, defined data, the recurring work runs the same way each cycle, and review gets faster because every figure traces back to its source.
The same controls in every other department
The context tax isn’t unique to finance. Every team that uses AI has rules a general model doesn’t know, and every team pays to explain them again. The governed core looks different in each department, but the three controls carry over.
Marketing is a good example. A marketing team’s rules are its positioning: how the company describes itself, the approved product names, the claims it can back with proof, the phrases it never uses. Without those rules in front of the model, every draft comes back generic or off-message, and someone rewrites it. A marketing team that writes those rules down once, as a saved set of instructions the assistant loads automatically, gets drafts that more often start on-message. If the same instructions also run a fixed review on every piece, the check is the same whoever writes the brief. Routine checks can then run on a lighter model, with the capable one kept for the writing.
| Department | Governed core | Locked workflow | Where tiering helps |
|---|---|---|---|
| Marketing | Positioning, approved claims, product names, terminology | A fixed review that every draft runs through | Lighter models for checks and formatting |
| Sales | Pricing rules, discount limits, approved customer proof | Proposal and follow-up drafts built the same way each time | Lighter models for call summaries |
| Customer support | Policies, product documentation, escalation rules | Response drafts grounded in the policy library | Lighter models for triage and tagging |
| People and HR | Policies, compensation bands, role definitions | Job descriptions and policy answers from one source | Lighter models for routine questions |
None of this requires Datarails outside finance. Each team owns its own context. Each team’s context also needs its permissions: an assistant that can read compensation bands or discount limits should show them only to people allowed to see them in the source system. What carries over is the discipline: define the rules once, lock the recurring work, and match the model to the job.
A CFO playbook for company-wide AI spend
The CFO already owns the budget. These steps turn that ownership into control, without slowing any team down.
- Map AI use by team and task. Pull every AI invoice, seat subscription, and API usage report into one view, allocate the spend to teams, and compare it with budget each month. Find out which teams use AI for what, which tasks recur every week or month, and where the same request gets rebuilt from scratch.
- Ask each team for its governed core. Every department should be able to name the rules it keeps re-explaining to the model: definitions, policies, approved language. Get them written down once, in a place the AI can use.
- Lock the recurring work first. The tasks that repeat on a schedule cost the most over a year and gain the most from a fixed setup. Start there, not with one-off questions.
- Set a routing rule. Agree which kinds of work justify the most capable models and which can run on lighter ones, and make that the default rather than a per-user choice. Check the rule against results: as BCG points out, a lighter model that needs several retries can cost more per successful outcome than a stronger one that gets there quickly, and lookups, arithmetic, and fixed policy checks belong in ordinary software rather than a model.
- Measure cost per accepted output. Cost per token tells you the price. Cost per accepted output, including the rework and review time, tells you whether the controls are working.
- Start with finance. Finance has high-stakes context and clear rules, and Datarails provides its governed core through FinanceOS. A working finance setup also gives every other team a model to copy.
Own the AI budget? Start where the stakes are highest. See how Datarails FinanceOS gives finance a governed core that every AI tool can work from.
AI token costs FAQs
Why are AI token costs rising when token prices are falling?
Because usage grows faster than prices fall. Cheaper tokens make it worthwhile to give AI more tasks, longer inputs, and more attempts at the same answer, so total spend rises even as the unit price drops. This is the Jevons paradox applied to AI.
Who should own AI spend in a company?
In most large companies, the CFO already does. Owning the budget is different from controlling the spend, though. Control comes from setting company-wide standards for how teams give AI their context, which recurring work gets a fixed setup, and which tasks justify the most capable models.
What drives AI token waste?
The main drivers are long prompts and conversation histories, longer outputs than the task needs, premium or high-reasoning models used for simple work, agent loops, and retries. Missing context feeds several of them. When a model doesn’t know a team’s definitions, rules, or approved language, people re-explain them, regenerate answers that come back wrong, and spend time checking the output. Each loop costs tokens and review time.
Do usage caps control AI costs?
Caps limit how much a team can spend, but they don’t change the behavior behind the spend. A team that hits its cap still has the same loops of re-explaining and regenerating. Reducing the context the model is missing, and locking recurring work into a fixed setup, lowers the tokens each task needs.
What is tiered intelligence?
Tiered intelligence means matching the model to the task. Routine work such as formatting, tagging, and summaries can run on lighter, cheaper models, while complex analysis and reasoning use the more capable ones. Deterministic lookups and calculations are better left to ordinary software. Setting that as a default stops teams paying top rates for simple work.
How does Datarails help control AI costs in finance?
Datarails, through FinanceOS, gives every connected AI tool the same consolidated data, shared definitions, and permissions, so models don’t need the business explained to them in every prompt. Inflow Agents and scheduled Custom Agents run recurring finance work in a fixed way, and the FinanceOS AI Connector lets teams choose the model that fits the task without rebuilding the context.
