By: Anand Rao, Associate Vice President & Portfolio Head (UKI & Europe), LTIMindtree Consulting

Many industries are keen to leverage artificial intelligence (AI) to boost productivity or enhance creative output. However, several studies show that nearly three-quarters of AI scaling efforts fail to make a real impact. So, why do so many of these initiatives fall short? There are several reasons to that. The few primary reasons are legacy infrastructure and higher than expected costs and ‘disengagement’ by unexpected stakeholders. This article explores some basic problems in scaling AI and strategies to overcome them AI system integration.

Here are four key data points highlighting challenges in scaling AI projects:

The scaling dilemma: from pilot to enterprise

Many businesses start with AI projects designed to augment decision-making and automate tasks. For example, an AI-powered customer service assistant might manage routine tasks, suggesting responses and improving efficiency. While such pilots often show promising results—boosting customer satisfaction and efficiency, reducing workloads by about 30% or speed of response by about 40%—scaling them to manage millions of interactions daily requires substantial infrastructure upgrades, effective AI system integration, cost management, and stakeholder collaboration.

So, why does scaling from a limited pilot to an enterprise-wide solution pose such challenges? The answer lies largely in the infrastructure. Pilots usually involve simple systems with limited data and processing needs, but expanding these systems to enterprise level—encompassing customer service, supply chains, and financial systems—requires significant technological upgrades. Picture a small, efficient assistant evolving into a data-intensive system managing millions of interactions. This demands powerful processors, vast storage, and sophisticated data pipelines to handle the increased load.

For example, imagine a company scaling AI to handle a million daily interactions. This shift, from hundreds of interactions to millions, needs robust processing power, rapid data storage, and real-time monitoring. Specialized chips and cloud-based graphics processing unit (GPU) clusters are often necessary to manage the demands of Large Language Models (LLMs). Scaling AI could cost anywhere from US $380,000 to US $630,000 per year, depending on infrastructure complexity. For AI to scale effectively, organizations must establish AI system integration that supports fast data processing, secure data storage, and a strong legal and governance framework for accountability.

Scaling can quickly become expensive. If these needs aren’t met, the system may slow down, user satisfaction might decline, and operational savings may disappear. So, what does it take to ensure an AI system scales smoothly?

Scaling with confidence: balancing business and technology transformation

For AI to deliver measurable value, businesses must have a clear plan to scale their AI initiatives[i]. Success requires aligning AI initiatives with business goals, choosing strategic use cases, monitoring return on investment (ROI), and preparing for future advancements. Business transformation with AI entails a holistic approach that combines strategic goals and technological upgrades, ensuring that AI adds value enterprise-wide.

At LTIMindtree, we’ve found that blending business and technology transformations is key to helping organizations scale AI effectively. This approach allows enterprises to mature their AI capabilities while balancing technology with business outcomes.

Risk and governance: the compliance challenge

A significant hurdle in scaling AI is ensuring AI solutions are compliant with regulations. Both the UK and EU mandate that AI systems be robust, accurate, reliable, and trustworthy, with respect for privacy and human rights. The EU AI Act and the General Data Protection Regulation (GDPR) require transparency, fairness, accountability, and robustness in AI. How do businesses navigate these stringent standards?

For example, if a company collects sensitive customer data, it must comply with GDPR to avoid fines up to US $22 million or 4% of annual global turnover. The EU AI Act further imposes fines for violations, reaching US $38 million or 7% of annual turnover. Non-compliance not only risks severe fines but can also erode customer trust and dilute the financial benefits of AI.

Building trustworthy AI involves more than technology—it’s also about human oversight. Effective governance frameworks that ensure your AI models are:

1. Explainable (clear about how they reach conclusions)
2. Auditable (capable of providing evidence of their operations) and
3. Unbiased (fair to all user groups)

Organizations can implement several key actions, such as:

1. Regular auditing of models to spot changes in the data distribution or illegalities in the model
2. Bias checks to ensure fair treatment across demographics, and
3. Secure data management practices

Although building these systems can be costly, the long-term benefits like lower compliance expenses and stronger customer trust outweigh the initial investment. What governance steps would help scale your AI responsibly?

Hidden AI scaling costs: token usage

As AI systems scale, token usage costs can accumulate quickly. Each time a Large Language Model (LLM) processes data, it consumes tokens. For instance, handling 100,000 conversations daily, with each conversation using 500 tokens at US $0.0006 per token, would cost US $30,000 daily or US $600,000 annually. Without careful cost management, token expenses alone can wipe out the financial benefits of scaling AI.

How can companies reduce these scaling costs? Implementing feedback loops for continuous AI fine-tuning and evolutionary algorithms to optimize energy usage and reduce token consumption are both effective strategies. For instance, a 30% reduction in token usage could save approximately US $420,000 annually. Additionally, choosing cloud services that offer cost-efficient storage can also aid in scaling AI without excessive costs.

Profitability is maintained by inexpensive measures, and therefore, AI transformation of technology to activities allow businesses to reduce operating costs by balancing AI innovation with cost reduction for positive ROI. Further, AI usage and performance monitoring enables firms to keep track of operating costs while reducing them on part of executive industry.

Note:

Cumulative Costs

• AI-driven use case: Higher initial costs but slower cumulative growth due to automation, thereby reducing operational inefficiencies
• Non-AI use case: Lower initial costs but faster cumulative cost growth over time is driven by manual labor and higher ongoing expenses

Scaling Costs and Token Usage

• Scaling AI has a high token-usage cost. Each query processed by LLMs incurs token fees. Therefore, without careful control, token fees can erode profitability

Cost Control and Optimisation

• Constant feedback loops for fine-tuning and evolutionary algorithms can reduce these costs. Additionally, optimizing cloud services can further lower expenses

ROI

• AI-driven use case: Shows rapidly increasing ROI over time, as AI systems become more efficient and deliver greater value
• Non-AI use case: ROI grows more slowly, reflecting the limitations of scaling manual processes

This balance between AI innovation and cost reduction ensures that AI systems deliver positive ROI over time, despite higher initial investments, with operational monitoring helping keep costs manageable while increasing efficiency.

Drive AI adoption across the organisation by engaging stakeholders

It’s not just the technology that’s important for AI scaling to succeed; it’s also the people you engage with. A lack of buy-in from key business functions is a common reason for failed AI projects. Without early stakeholder involvement, AI initiatives may face resistance or be seen as threats to core business objectives.

If stakeholders don’t fully support AI, systems may be underutilized, leading to lower impact and reduced ROI. To avoid this, business leaders should champion AI adoption, forming networks of advocates who promote AI across departments. Early engagement, AI literacy programs, and positioning AI as a partner that enhances human work can foster company-wide support and drive meaningful AI outcomes.

Final thoughts on pervasive AI

Scaling AI involves more than just technological advancements; it requires a fundamental shift in business culture. A truly successful AI journey bridges the gap between teams, fostering collaboration, and ensuring that every part of the organization derives meaningful value from the transformation.

Key takeaways

1. Business model innovation: Align AI projects with measurable business goals to maximize impact and achieve meaningful outcomes
2. Technology transformation: Adopt scalable AI infrastructure that provides a secure foundation for growth and seamless integration
3. Risk and change management: Develop strategies that ensure compliance and transparency, while effectively managing risk and change
4. Cost optimization: Keep AI operating costs in check by leveraging feedback loops and optimizing cloud resources for sustainable ROI

Organizations must have the right infrastructure, governance, cost management, and team alignment to move AI from pilot to enterprise-wide use. Those that carefully navigate this transition will unlock the full potential of business transformation with AI for lasting enterprise success

References

i Economic Impact of AI: Evaluating ROI for Gen AI Use Cases

ⁱⁱ EU AI Act: https://artificialintelligenceact.eu/

ⁱⁱⁱ GDPR: https://gdpr-info.eu/

^iv O’Reilly in 2024: https://www.oreilly.com/pub/pr/3441

^v BCG 2023: https://www.bcg.com/capabilities/artificial-intelligence

^vi Gartner Says More Than 80% of Enterprises Will Have Used Generative AI APIs or Deployed Generative AI-Enabled Applications by 2026, Stamford, CT, Gartner, October 11, 2023: https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026

^vii Keep Your AI Projects on Track, Harvard Business Review, November-December, 2023: https://hbr.org/2023/11/keep-your-ai-projects-on-track