A portfolio company launched an AI customer-support agent in Q2 2024. The proof-of-concept cost $240 in API tokens across three weeks. Production deployment cost $14,000 in month one, $22,000 in month two, and $31,000 by month four with no plateau in sight. The CFO flagged the overage during a monthly close. The team had not modeled token volume at production scale, had not set cost alerts, and had shipped the agent with no limits on input length or retry loops.
This is not an edge case. PE-backed companies deploying AI agents without production token cost forecasts routinely face surprise six-figure annual bills that compress EBITDA, require board explanation, and cannot be accurately attributed to revenue or saved headcount. The economic model breaks when the cost to serve one automated interaction exceeds what a human would have charged, and you discover that fact only after the contract is live and the user base has scaled.
Token cost forecasting is engineering work, not strategy work, and it must precede deployment. The returns depend on it.
Token cost is variable cost tied to volume and model choice
An AI agent’s operating cost is overwhelmingly token cost: the per-call charge from the API provider for input tokens (the prompt, context, and user message) and output tokens (the model’s response). OpenAI GPT-4 charges roughly $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens as of mid-2024. Claude 3.5 Sonnet runs $3 per million input tokens, $15 per million output tokens. Gemini 1.5 Pro sits between them. The rates compress annually, but the volume scales faster.
A single customer-support interaction averages 800 input tokens (prompt template, conversation history, knowledge-base context) and 400 output tokens (the agent’s reply). That is 1,200 tokens per turn. If the conversation runs four turns, one support case consumes 4,800 tokens. At GPT-4 rates, that case costs $0.24. For 10,000 cases monthly, the bill is $2,400. For 100,000 cases, $24,000. The cost scales linearly with volume and model selection, and neither variable is negotiable once the agent is live.
The financial mistake is treating token cost as negligible during proof-of-concept and discovering the production number only when the invoice arrives. A $300 pilot misleads because it ran on 50 test cases with short prompts. A 100,000-case production load is 2,000 times larger, and the cost moves proportionally unless the architecture is rebuilt to compress it.
Most implementations ship without cost governors or input limits
The portfolio company mentioned above had no maximum token limit per request, no retry cap, and no alert threshold. A user could paste a 15,000-word document into the input field, the agent would process it, and the API would charge for every token. One support case hit 47,000 tokens because a customer uploaded a PDF transcript that the agent attempted to summarize. That single case cost $2.80. The median case cost $0.18, but the distribution had a long tail, and the tail consumed 40% of the monthly budget.
Production deployments require four governors before launch. First, a hard input token limit enforced at the application layer, typically 4,000 to 8,000 tokens depending on use case, with longer inputs truncated or rejected. Second, a retry cap of two or three attempts for failed API calls, to prevent infinite loops when the model returns an error or malformed JSON. Third, a monthly cost ceiling with automatic suspension when the threshold is hit, set at 120% of the forecasted budget to allow variance without runaway spend. Fourth, logging of per-interaction token counts and costs, written to a database or observability platform, so the finance team can attribute spend to user cohorts, case types, or agent features.
None of these require ML expertise. They are application-engineering work, and the absence of any one creates financial exposure that only a portco CFO or operating partner will notice, months into deployment.
Context windows drive cost more than conversation length
A common assumption is that longer conversations cost more. That is true only if every prior turn is appended to the prompt. In practice, token cost is driven by context window size, the total amount of text sent to the model on each API call, which includes the system prompt, any retrieved knowledge-base documents, conversation history, and the user’s latest message.
A customer-support agent pulling five help-center articles into every prompt to ground its responses will send 3,000 tokens of context per turn, regardless of whether the user wrote ten words or two hundred. If the agent retrieves 15 documents instead of five, context balloons to 9,000 tokens, and cost triples. The retrieval strategy, how many documents, at what length, and whether they are chunked or sent whole, determines cost structure more than user behavior does.
The CEO-level decision to expand agent knowledge base from 50 articles to 500 articles can increase token cost by 40% if the retrieval system is not re-tuned to select fewer, shorter passages. That is an engineering constraint that must be surfaced before the content team scales documentation, not discovered when the monthly bill doubles.
Model selection carries a three-fold cost difference
GPT-4 costs roughly ten times more per token than GPT-3.5 and twice as much as Claude 3.5 Sonnet. For identical workloads, model selection determines whether monthly token spend is $5,000 or $50,000, which makes it a financial decision as much as a quality one.
A support agent handling common questions, password resets, billing inquiries, order status, can run on GPT-3.5 or a fine-tuned Llama 3 8B model with equivalent accuracy and 90% lower cost. A legal contract analysis agent summarizing clauses and flagging non-standard terms requires GPT-4 or Claude 3 Opus for reliability, and the cost difference is the price of correctness. The use case determines the minimum acceptable model, and anything beyond that minimum is margin compression.
The financial discipline is to define acceptable accuracy before selecting the model, then choose the cheapest model that meets the threshold. Most implementations do the opposite: default to GPT-4 because it is the flagship, discover that it costs too much, and only then evaluate cheaper alternatives. By that point, the agent is live, users expect a certain response quality, and downgrading the model risks user complaints and churn.
Model selection is a one-way door. Forecast cost at the target accuracy level during proof-of-concept, not after deployment.
Rate limits and throttling prevent cost spikes but require user-experience trade-offs
API providers impose rate limits: maximum requests per minute, maximum tokens per minute, maximum concurrent requests. OpenAI’s standard tier allows 3,500 requests per minute and 90,000 tokens per minute for GPT-4. Exceed that, and the API returns a 429 error, which the application must handle by queuing the request, retrying after a delay, or rejecting it.
A portfolio company scaling a chatbot to 50,000 daily users will hit rate limits during peak hours if every user triggers one request per session. The application must implement a queue, and users will wait. The alternative is to upgrade to a higher API tier, which raises the rate limit but also raises the per-token cost or requires a minimum monthly spend commitment, often $50,000 to $100,000 annually.
The trade-off is cost predictability versus user experience. Throttling keeps token spend within budget but introduces latency. Upgrading the tier removes latency but locks in a cost floor. The financial forecast must account for both, and the operating partner must decide which constraint, cost or speed, is binding. That decision cannot be deferred to post-launch.
Attribution to revenue or saved headcount is required to justify the spend
A $30,000 monthly token bill is defensible if it eliminates $60,000 in support-team payroll or increases conversion by 200 basis points on a $10 million revenue base. It is indefensible if neither number can be shown. PE operating partners evaluating AI agent business cases need three figures before deployment: forecasted monthly token cost, the revenue or cost impact the agent is expected to deliver, and the payback period in months.
Most business cases present the productivity benefit and treat token cost as negligible or omit it entirely. When monthly spend reaches $20,000 and the CFO asks what the company is getting for it, the attribution model must already exist, because reverse-engineering it from production logs rarely produces a defensible number.
The cleanest attribution is saved headcount. If a support agent handles 10,000 cases monthly that would otherwise require two full-time employees at $50,000 annual salary each, the agent saves $100,000 annually and costs $30,000 in tokens, a $70,000 net gain. The weaker attribution is incremental revenue, which requires proving that the agent caused the revenue lift and isolating it from other growth levers. The weakest attribution is ‘improved user experience’, which is qualitative and cannot be used to justify six-figure spend in a board deck.
Build the attribution model into the pilot. If you cannot prove the financial return in a controlled 500-user test, you will not be able to prove it at 50,000 users, and the operating partner will pull the budget.
Forecasting requires production-load simulation, not pilot extrapolation
A proof-of-concept running on 200 test cases is not a valid basis for production cost forecasting. Test cases are shorter, cleaner, and more uniform than real user inputs. Real users paste error logs, upload documents, ask multi-part questions, retry failed requests, and trigger edge cases that the pilot never saw. Token consumption in production is 30% to 60% higher than in testing, and the variance is wider.
The correct forecasting method is to simulate production load during the pilot phase. Take 10,000 historical support cases, customer emails, or logged user queries, and run them through the agent in a batch process. Measure total token count, cost per case, and distribution of case costs. Identify the 95th percentile case and the maximum case, because those are the ones that will blow the budget. Multiply the median cost by projected monthly volume to get the baseline forecast, then add 40% as a variance buffer.
If the forecast exceeds the budget, the architecture must change: compress context windows, truncate inputs, switch to a cheaper model, or reduce retrieval depth. Those changes require engineering time, and they must happen before deployment. Discovering the cost overrun in month two means the agent is already live, users are depending on it, and the business marketing world has communicated it as a capability. Rolling it back is a user-experience failure and a credibility loss.
Embedded retainers make cost forecasting a portfolio-wide capability
A portfolio company deploying its first AI agent does not have the in-house expertise to build token cost forecasts, implement governors, or simulate production load. The engineering team knows how to call an API, but they do not know how to model variable cost at scale, and the finance team does not understand token economics well enough to challenge the assumptions.
This is where an embedded technical partner operating across the portfolio builds leverage. The same forecasting model, the same governor templates, the same load simulation scripts, can be deployed to five portfolio companies in parallel. The second deployment takes 40% less time than the first. The fifth takes 60% less. The operating partner gets consistent cost visibility across the portfolio, and every portco CFO sees the same line item in the P&L with the same attribution model behind it.
The embedded model makes cost forecasting a repeatable capability rather than a one-time project, which is what creates value at the portfolio level. The alternative is that each portfolio company builds forecasting from scratch or skips it and discovers the cost in production, both of which waste either time or capital.
We run this as a standing check on our own company: the AI Ops Hive keeps model spend under a daily ceiling and reconciles it against receipts rather than estimates.
Conclusion
AI agents are production systems, and production systems have operating costs that must be forecasted before deployment. Token cost is the largest variable expense, it scales linearly with volume, and it cannot be negotiated down after the fact. The discipline is to model cost under production load, implement governors to cap spend, select the cheapest model that meets the accuracy threshold, and build attribution to revenue or saved headcount into the business case. Companies that skip this work discover six-figure annual bills with no clear return, and operating partners who fund those deployments face board questions they cannot answer with data.
The capability to forecast and govern token cost is built once, deployed across the portfolio, and improves with every implementation. That is the embedded operating-unit model, and it makes AI agent deployments defensible at the EBITDA line because it turns a per-company build into a reusable asset.
If your portfolio is evaluating AI agent deployments and needs production cost forecasting, attribution modeling, or embedded engineering capacity to implement governors across multiple portcos, reach out and we will walk through the model.
## Related Reading
