The LLM lock-in: changing a model isn't that easy, after all

During years leading product and digital transformations across major organizations, vendor lock-in has always been an area of attention and careful consideration.

AI implementations are creating a new type of vendor lock-in that's often overlooked, and with which many have little experience: swapping Large Language Models isn't simply a matter of changing an API key.

The myth of model interchangeability

The assumption that Claude, GPT, or Gemini models are interchangeable reflects a fundamental misunderstanding that can derail even the most strategic AI initiatives. As someone who has led complex data transformations, I've seen firsthand how technical decisions at this level can impact business outcomes. Each model family interprets prompts uniquely, with distinct "personalities" that affect everything from reasoning to output formatting. Recent research shows enterprise teams who treat model migration as plug-and-play face significant regressions in quality, reliability, and costs.

Understanding model differences

The differences manifest in several critical areas:

Cost structures. Different models count and charge for words differently, potentially doubling your costs for the same tasks. For example, Anthropic's models often tokenize text much more expansively than OpenAI's, resulting in higher operational costs for identical inputs.

Processing capacity. While some models advertise handling massive documents, their performance often deteriorates with longer inputs. Gemini might boast a 2M token context window compared to GPT-4o's 128K, but actual performance degradation curves vary significantly across these models.

Communication preferences. Each model responds better to different formatting styles — essentially speaking different "dialects." OpenAI's models typically perform better with markdown-formatted prompts, while Anthropic's Claude series responds more effectively to XML tags and structured instructions.

Output styles. Some excel at creative, flowing text while others perform better with structured, predictable formats. This directly impacts application architecture decisions, especially for systems requiring consistent, parseable outputs.

The real-world cost of model swapping

When organizations swap models without accounting for these differences, the consequences can be severe.

Broken integrations and customer experiences. Applications that worked perfectly with one model can suddenly fail when prompts are misinterpreted or responses are formatted differently, damaging customer trust. Output structures can completely break systems after a model migration, requiring urgent engineering intervention.

Budget overruns. What seemed like a cost-saving move can quickly turn into a financial nightmare when token counts unexpectedly balloon, or when more complex prompting techniques become necessary to achieve the same results.

Lost productivity. Engineering teams can spend weeks troubleshooting issues that stem from subtle model differences rather than focusing on innovation and value creation. At dunnhumby, we learned early to document model behaviour thoroughly to avoid this exact scenario during our early AI initiatives.

Imperceptible quality degradation. Without comprehensive benchmarking data from your previous model, quality regressions can go undetected until they affect business outcomes. What appears to work in limited testing might perform differently at scale or on edge cases, requiring extensive historical performance records to properly evaluate.

The vendor lock-in reality

These implementation challenges represent a sophisticated form of vendor lock-in that's often underappreciated in procurement discussions, and that you might sleep-walk into. Unlike traditional software where migration costs are primarily technical, LLM switching costs are deeply embedded in both technical implementations and organizational knowledge.

The strength of this lock-in varies based on several factors:

1. Application complexity. Systems with simple, straightforward prompts experience less lock-in than those leveraging complex prompt engineering techniques optimized for a specific model.
2. Integration depth. The more deeply an LLM is integrated into critical business processes, the stronger the lock-in becomes.
3. Knowledge investment. Organizations that have invested heavily in understanding the quirks and capabilities of a specific model face substantial knowledge transfer costs when switching.
4. Prompt IP. The intellectual property developed in prompt engineering for one model represents significant value that doesn't necessarily transfer cleanly to another provider.

A strategic approach to model migration

The good news is that these challenges can be mitigated with proper planning and systematic evaluation. Based on my experience working with data innovation at scale, here are key recommendations for organizations contemplating a model switch.

1. Build comprehensive evaluation frameworks

Don't rely on anecdotal testing or small sample sets. Develop a robust evaluation framework that covers all use cases your application serves:

• Create a diverse test dataset that represents real-world usage patterns
• Define clear metrics for both technical performance and business outcomes
• Establish baseline performance with your current model before testing alternatives
• Test at scale to uncover edge cases that might not appear in limited trials

2. Document model behaviours extensively

Do not underestimate the value of comprehensive documentation. For AI models, this means maintaining detailed records of:

• Prompt templates that work effectively with each model
• Observed response patterns and formatting preferences
• Performance characteristics across different input lengths and complexities
• Any observed limitations or quirks specific to each model family

3. Implement progressive rollout strategies

Rather than switching all traffic to a new model overnight, consider:

• A/B testing approaches where a percentage of traffic routes to the new model
• Shadow deployments where the new model runs in parallel without affecting user experience
• Canary releases that gradually increase traffic to the new model while monitoring key metrics

4. Invest in model-agnostic infrastructure

The most effective mitigation against vendor lock-in is building abstraction layers between your applications and underlying models:

• Create standardized prompt templates that can be transformed based on the target model
• Implement middleware that normalizes outputs from different models into consistent formats
• Develop multi-model routing capabilities that allow seamless failover between providers
• Consider open-source models as fallback options for critical applications

5. Negotiate strategic contracts

From my experience negotiating enterprise agreements at various organizations:

• Include detailed terms governing price increases and performance guarantees
• Secure data rights that allow you to use outputs from one model to fine-tune another
• Negotiate transition assistance in the event you decide to switch providers
• Consider multi-vendor strategies for mission-critical applications

Looking forward

The LLM landscape continues to evolve rapidly, with new models and capabilities emerging regularly. Organizations that develop systematic approaches to evaluating and integrating these models will maintain competitive advantage through more reliable AI systems and optimized operational costs.

In my career, I learned that successful AI integration requires systematic evaluation and continuous learning — much like our approach to data monetisation capabilities. This systematic approach to data and AI is essential for enterprises seeking to derive sustainable value from these technologies.

Enterprise leaders must invest in proper evaluation frameworks, maintain detailed documentation of model behaviours, and collaborate deeply with product teams to align outputs with business objectives.

AI initiatives stall on data foundations more often than on models. The Data Monetisation Readiness assessment scores those foundations across five capabilities.