This Executive Impact Series article is a collaboration with Anush Naghshineh and David Turner.
Introduction: Beyond the Hype – Understanding Real AI Costs
Every leader today feels the same pressure: “We need an AI strategy.”
But too many are learning the hard way that AI strategy without cost reality is just wishful thinking.
Over the past two years, AI has shifted from curiosity to core strategy. Boardrooms are abuzz, LinkedIn is lively, and pilot projects are everywhere. Yet according to McKinsey, 78% of companies using AI report no significant financial gains, even as compute costs have soared nearly 90% since 2023.
So what’s happening? The hype is hiding the cost. I saw the same thing firsthand when evaluating AI for one of our projects, what looked like a smart, fast adoption quickly turned into a slow, expensive rebuild of our data foundation and workflows.
And here’s the uncomfortable truth: most AI projects end up costing 10–15x more than the original estimate once you include integration, talent, and maintenance. The reason isn’t failure, it’s misunderstanding. Predictive models, generative tools, and agentic systems live in entirely different financial realities. Each demands its own infrastructure, skill sets, and ROI curve.
This article breaks those differences down. It’s not about scaring you off, it’s about helping you invest smarter.
Because in 2025, the most valuable leaders won’t be the ones who launched AI first; they’ll be the ones who understood its true cost before anyone else did.
This is Part 1 of a two-part series exploring The True Cost of AI Implementation.
Predictive AI and Machine Learning – The Proven Tech
Machine learning (ML) and predictive AI have been widely used for over 20 years, providing valuable insights into costs and ROI.
According to recent industry analysis, standing up a traditional ML system in a mid-sized company usually costs somewhere between $50,000 and $500,000. That range depends on the complexity of the problem and the quality of the available data.
The investments are typically spread across multiple domains. One area is Data scientists and machine learning engineers, with average US salaries of around $140,000 a year. Another primary area is data work, which typically exceeds initial estimates. Cleaning up data so that it is ready to train models can cost up to 25 percent of a project’s budget. One analysis found that more than 95% of companies lack quality training data when they begin projects, and generating 50,000 – 100,000 data samples through services like Amazon’s Mechanical Turk can require $70,000.
While initially determining the level of effort and spend for data is challenging, the hardware requirements are more straightforward. Predictive AI and ML don’t require massive GPU clusters, which are more commonly used for large language models (LLMs). These models typically run well on standardized cloud infrastructure, with well-defined and stable costs.
One aspect of data that companies often overlook is the cost of ongoing maintenance and data quality. While not a cost of implementation, it is a cost that must be borne to keep a model operational. Typically, companies need to budget 15 to 30 percent of the implementation cost annually toward model upkeep. The money is required for retraining as data patterns shift, ongoing monitoring to identify performance issues, and adjusting the model as business needs change.
Using predictive AI and ML is most effective when the available data is well-structured and has patterns that are well understood. These models help identify events and trends that require informed decisions from humans. The models are focused on identifying these key points and delivering accurate data, rather than creating creative content or messaging. Typical use cases include:
- Predicting customers who are most likely to leave
- Identifying fraudulent transactions
- Forecasting how much inventory you’ll need over time
- Predicting equipment maintenance before failure
While the technology for ML and predictive AI is well known and proven, implementation still has substantive risks that need to be managed. As we touched upon, data is a key driver of success, meaning data must be available and of sufficient quality. Data availability becomes a significant challenge the more it is spread across multiple disparate systems. In situations like this, companies must prioritize addressing the problem before implementing a full model.
The good news is that this is a solvable problem; it just requires effort, focus, and realistic expectations.
Generative AI – A Multiplier of Productivity and Cost
The track record for Gen AI implementation is significantly shorter and more divergent than that of ML and predictive AI. Being available for only around five years, and having undergone such rapid evolution, effectively leveraging it has been like “nailing jelly to a tree.” Additionally, Gen AI and LLMs often require substantial computing resources, which can be exceedingly expensive and complex to manage.
Models have become more capable and complex, primarily due to the increasing amount and quality of training they undergo. The growth in model complexity has led to a massive increase in the capital required to train newer generations of models. For example, according to Lambda Labs’ calculations, the training cost for OpenAI’s GPT-3 in 2020 was estimated to be over $4 million. Training for GPT-4, released in 2023, reportedly cost over $100 million. Some estimates for Google’s Gemini Ultra model estimate $191 million in compute costs alone. Industry reports from 2024 projected that OpenAI spent $3 billion training new models, with the runtime cost for ChatGPT reaching $4 billion that year.
While most companies are not building models like ChatGPT from scratch, the data highlights the tremendous cost burden required to train custom models. Additionally, it indicates the amount of money that platform providers need to charge companies using APIs to access complex models. API costs are based on the number of text chunks, or tokens, exchanged with the model. These token costs multiply quickly when processing documents, performing multi-turn conversations, and analyzing large text blocks. Managing these costs is challenging due to the variability of the text data and results, with companies reporting monthly usage fees in tens of thousands of dollars.
For companies that require complex models but also need them to be tailored to their unique needs, fine-tuning existing models may be an option. Fine-tuning typically costs between $80,000 and $190,000 for the initial process, with a recurring cost of between $5,000 and $15,000 every month to keep it up-to-date. As a result, companies can create a model that addresses their specific business better. However, they are still relying on an external company’s infrastructure, security, and controls.
Companies looking to host complex LLMs with tens of billions of parameters or more must understand the infrastructure costs that they must incur. A single NVIDIA H100 GPU required to run large language models efficiently can cost $40,000. Most serious applications require multiple GPUs clustered together. Mission-critical applications will require extra hardware for continuity, as well as for development and testing. Add to this the significant cost of energy needed to run these systems. As a result, companies hosting models are seeing their cloud computing bills jump 70% or more after deploying GenAI applications.
Given the different nature of GenAI from traditional ML and predictive AI, the costs are variable and harder to predict. With conventional software, adding more users doesn’t significantly increase costs. However, with GenAI, every conversation, document processed, and even API calls can create a non-trivial spike in cost.
The research on the ROI for GenAI implementations is mixed. Microsoft’s research suggests that these investments average a return of 3.5 times. However, recent research from McKinsey shows that while 78% of companies are using Gen AI in some form, 3/4 report no significant return. Research from MIT backs up the McKinsey finding.
GenAI provides robust functionality in various use cases, including content generation, customer service automation, coding assistance, and document summarization. Companies can also deploy GenAI to achieve rapid productivity gains for knowledge workers. However, the gains are spread thinly across a broad range of employees and are not focused on a measurable business process. This dispersion makes calculating a meaningful ROI a significant challenge.
To control costs, many companies are adopting hybrid approaches, routing simple tasks to less expensive models and utilizing more costly models only for complex tasks where the higher costs are justified. This type of approach requires companies to thoroughly understand their use cases and provide employees with proper training and clear direction.
Companies that are seeing a positive ROI from GenAI:
- Focus on it as a strategic capability
- Created requirements and plans around realistic expectations
- Designed their infrastructure and controls well
- Trained their staff on how to use GenAI
- Implemented proper change and risk management practices
Agentic AI – The Autonomous Future at Enterprise Scale
Agentic AI systems utilize LLMs, but do so differently than GenAI chatbots. Chatbots utilize LLMs to respond to user prompts. In contrast, agentic AI employs LLMs to plan its steps, analyze data to make decisions, and act without requiring constant human supervision. These systems are autonomous assistants equipped with the tools and guidance needed to complete complex tasks.
Agentic AI is more capable than standard GenAI systems; however, this increased capability comes with higher implementation costs. Basic reactive agents that handle simple workflows cost between $15,000 and $50,000 to build. More complex agents, focused on a single business domain and capable of planning actions, remembering context, and integrating with other systems, will often cost more than $120,000 to create. Complex multi-agent systems composed of several AI agents working together on complex processes can cost well in excess of $200,000. For highly specialized agents in regulated industries, such as healthcare or finance, costs can approach $1 million.
Ongoing operations costs are also significant, and like GenAI, can be challenging to predict. For companies using commercial models from major providers, token costs remain a primary cost driver. Agentic systems have internal conversations, encounter unexpected situations and errors that necessitate retries, and ultimately require a high number of iterations to complete a task. This iterative and variable nature requires substantial design and testing to create an optimized solution. As a result, industry analysts have estimated that mid-sized deployments will use between 5 and 10 million tokens each month.
Even for companies that use models from providers, internal infrastructure is still required and contributes to the total cost. For example, many agents utilize vector databases to handle memory and context, which incur monthly expenses of $500 to $2,500. Monitoring tools are essential for tracking agent activity and identifying issues before they escalate. Additional security layers and monitors are needed to monitor and protect data as agents are taking actions. The added monitoring and security will likely cost in excess of $2,500 monthly.
Another cost dimension to consider for Agentic AI is personnel costs. Companies need well-trained staff to build and maintain these systems. Because agents interact with changing business systems and requirements, maintenance is a continual process. These costs vary widely based on the business domain and the skills and availability of internal resources. In best-case scenarios, added costs may only be a few thousand dollars per month. In other cases, they can reach into the tens of thousands of dollars.
Beyond the personnel costs for system maintenance are the broader costs of engaging cross-functional teams, which include not only AI engineers but also business analysts, financial analysts, domain experts, lawyers, and compliance personnel. The time investment required by these broader staffing domains needs to be planned for and managed.
According to a May 2025 PwC survey, 88% of executives plan to increase AI budgets specifically for agentic AI. As a result of this trend, Gartner predicts that by 2028, 15% of routine business decisions will be made by autonomous AI agents. However, Gartner also predicts that over 40% of agentic AI projects will be canceled due to increasing costs, unclear business value, or unmitigated risk.
The bottom line is that agentic AI costs 2 to 3 times more than basic GenAI implementations, but it’s targeting high-value solutions. ROI is best realized by fundamentally transforming a process for the better, rather than achieving small cost efficiencies by having employees become a bit more productive. Companies focusing on substantive process improvements are seeing significant ROI. Dow Chemical is utilizing agents to analyze shipping invoices to identify and resolve billing errors across hundreds of thousands of annual shipments. Their projected savings are in the millions and far exceed the cost of implementation and maintenance. They are an example of how autonomous agents can create a positive impact, focusing on high-volume, detail-oriented work that requires discernment but without needing human judgment for each decision.
Companies seeing success with agentic AI share key traits:
- They have executive sponsorship for a multi-year transformation
- They commit to organizational change
- They have well-defined use cases with clear value
- They budget and plan realistically
- They have processes to deal with unexpected costs, risks, and roadblocks
Conclusion: Making Informed AI Investments in 2025
The deeper you look into AI implementation, the clearer one truth becomes: AI doesn’t fail because it’s overhyped, it fails because it’s misunderstood.
Predictive, generative, and agentic systems each bring value, but they live in different cost realities. Without understanding that, leaders mistake complexity for progress and end up funding experiments that never reach scale. The organizations winning today aren’t the ones experimenting fastest, they’re the ones budgeting smarter, structuring data early, and aligning each AI type to measurable business outcomes.
If Part 1 showed you where the money really goes, Part 2 takes it further, into how to make those investments pay off. It’s about frameworks, prioritization, and the disciplined decision-making that turns AI from a cost sink into a competitive asset.
Because in the next phase of this journey, success isn’t about what you build with AI, it’s about how you invest in it.