AI Vendor Lock-In Is the Cheap Kind: Your Prompts and Evaluation Set Are Not

Which layer of AI vendor lock-in actually costs money?

I have been brought in to unpick AI systems that a company wanted to leave, and the pattern is consistent: the argument is always about the model, and the money is always somewhere else.

Model choice is the cheapest part of AI vendor lock-in. It is also the part that gets most of the diligence, because it is the only layer whose name a buyer recognises. The expensive lock-in is quieter: your prompts, your evaluation set and your data schema sitting inside someone else's product, in a form only that product can read.

Model lock-in is the cheapest kind of AI vendor lock-in; the lock that costs you real money is your prompts, your evaluation set and your data schema living inside someone else's product.

The commercial difference is what each layer takes from you. Losing a model costs you a migration. Losing the evaluation set costs you the ability to make the comparison at all, which is a permanent position rather than a project.

Why is the model the cheapest layer to replace?

Because the model layer is the one place in this stack with genuine competition, and the interface to it is almost trivial.

If your code calls a provider directly, swapping models is an endpoint change and a re-run of your graded cases. Days, not quarters. List prices for a given capability level have fallen repeatedly over the years I have been building on these APIs, so the layer buyers worry about most is the layer getting cheaper to leave.

Two exceptions matter. Fine-tuned weights are not portable: they live with whoever trained them, in their format, and a competitor cannot serve them. And a prompt that was hand-tuned to one model's quirks over three months of iteration will not produce the same behaviour elsewhere, which is why the prompt layer belongs in the next section rather than this one.

There is also an organisational exception. If the model is reached through a vendor's builder rather than your own code, the model is not the lock at all. The builder is, and it will be described to you as convenience.

Why does an evaluation set decide whether you can leave?

Leaving requires evidence, and the evaluation set is the evidence. That is the whole mechanism.

Nobody with a P&L approves a replacement on the strength of a demo. They approve it because somebody ran two hundred real cases from last quarter through the alternative and showed that it was no worse, with the failures listed. Without that, the decision becomes a matter of faith, and faith gets deferred indefinitely. You are not locked in by the API. You are locked in because you cannot prove the alternative works.

The evaluation set is also the only asset in this stack that compounds. Every exception you grade teaches the system where its judgement fails, so the next change is safer than the last. A team with four hundred graded cases can move model vendors, rewrite prompts and change a schema in a fortnight. A team with none will re-run the same pilot argument every eighteen months.

This is where the vendor's own evaluation feature becomes the trap. It is convenient, and it is designed to keep the cases. The cases are inputs, expected outputs, acceptance criteria and the reason for each grade, each attached to a real case reference. What many vendors export is a score summary, which is a report about your evaluation rather than the evaluation itself. The question to ask is exact: can I export every graded case, including the reason, in a format I can run against a second provider without your cooperation? Ask it in writing, before signature, and ask them to attach a sample export file.

What does the exit actually cost, layer by layer?

The two middle rows are where the invoices are. The top row is what gets negotiated.

Here is the arithmetic, with every input stated so you can substitute yours. Take a workflow with 400 graded cases. Assume a fully loaded $150 an hour for platform engineering and $90 for analyst time, which are my working figures, not a benchmark. Rebuilding prompts is about three days, roughly $3,600. Re-grading 400 cases at six minutes each is 40 hours, about $3,600, and it assumes the original graders are still employed and still remember why each case was graded as it was. Rediscovering a schema and remapping it is three to six weeks of an architect's time, $18,000 to $36,000, because the semantics of a status field are not recoverable from a database dump. Add the two weeks of retraining that happens either way.

Call it $25,000 to $45,000 of labour, and three to four months during which nobody can safely change anything, against roughly one week if the artifacts are yours. The ratio is the point, and it holds at any rate you prefer to use. I would rather you treat these as assumptions to argue with than as findings, because I do not know your rates, your case mix or how much of the schema your team already carries in their heads.

What has to live in your repository, not the vendor's?

Four artifacts, each with a named owner. If you can name the person and the location in under ten seconds, you are not locked in. If you have to check, you are.

Prompt files. Plain text, version controlled, with the model and parameters recorded in the same commit. A prompt that exists only as a field in a web form is configuration, and configuration is not an asset.

The graded evaluation set. Input, expected output or acceptance criteria, the grade, and the reason for the grade. Plain JSONL or CSV, one row per case, case references intact. This is the artifact that takes longest to rebuild and the only one that makes future decisions provable, so it is the one to secure first.

The schema and mapping document. Entities, statuses, who may write what, and which system of record wins. Written in your vocabulary, not the vendor's.

The run log. Per run: input, output, model version, and what a human did with it. Retained long enough to answer an auditor, and readable without the vendor's software.

Ownership alone is not enough. Once a year, take your highest-volume workflow and have somebody export the artifacts and run the evaluation set against a second provider without telling the incumbent. If it takes more than a week, you have just measured your switching cost, and it is now a number rather than an opinion.

Which contract clauses settle this before you sign?

Six terms do most of the work, and all six are cheaper to negotiate now than later.

Two of these get missed constantly. The benchmarking clause matters because a vendor that forbids comparative testing on your own data has locked the evaluation layer by contract even if the export works. Model routing matters because a bundled fee hides the one cost that falls every year, and you want that decline to reach you.

When is lock-in the right purchase?

Sometimes it is. If the process is table stakes, every competitor runs it, and the vendor's scale genuinely beats yours, deliberate lock-in at a known price is a rational purchase rather than a failure of nerve. That is a build-versus-buy decision that turns on whether the process is a source of advantage, and it is worth settling before you argue about architecture.

What is not rational is doing it without a price. The failure I see is not a company that chose a platform; it is a company that chose a platform, never measured the exit, and then discovered at renewal that it had no comparison to bring to the table. Lock-in is a cost. It is only fatal when it is an unknown one.

What should you do before the next renewal?

Pick your highest-volume workflow and answer two questions in writing this quarter: where do the graded cases live, and can you export them in a format a competitor could read. If the answer to the second is no, that is your project, ahead of any new automation, because everything else you build on top of it inherits the constraint. Start with the evaluation set rather than the model, since the model swap is the part that was never going to hurt you, and the graded cases are the part that decides whether you can ever prove a better option exists. The exit you can price is an option you hold; the exit you cannot price is a subscription with your operating knowledge as collateral.