The demo took an afternoon. The feature took a quarter.
That gap is the most consistent thing I see in AI projects right now, and it has almost nothing to do with the model. Calling an LLM is four lines of code. What takes the quarter is everything wrapped around those four lines — and none of it is discussed in the tutorial you followed to build the demo.
This is what actually has to exist before an AI feature is safe to put in front of paying customers.
Why the demo lies
A demo has properties production never has. You typed the input yourself, so it was well-formed. You ran it once, so cost was invisible. You waited eight seconds without minding, because you were watching it work. Nothing else was happening at the same time. And when it produced something wrong, you re-ran it.
Every one of those becomes a hard engineering constraint at scale:
- Real users type garbage, paste 40 pages, and try to jailbreak it
- One-off cost becomes a per-request cost with no ceiling
- Eight seconds is an abandoned session
- Concurrency turns rate limits into outages
- "Just re-run it" is not available when it is generating a customer's invoice
I made this argument at a higher level in Beyond the Hype: The Engineering Reality of Running Generative AI in Production. Here is the concrete version.
1. Retrieval is the product
If your feature answers questions about your data — and nearly every useful one does — then the model is not the interesting part. Retrieval is.
The quality ceiling of an AI feature is set almost entirely by what you put in the context window, not by which model reads it. A strong model with bad retrieval produces confident nonsense. A modest model with excellent retrieval produces something useful. Teams consistently spend their effort on the wrong side of that trade.
Practically that means: chunking that respects document structure instead of splitting mid-sentence, embeddings refreshed when source data changes, hybrid keyword-plus-vector search because pure vector search misses exact identifiers, and re-ranking before you spend context on marginal results.
This is also where architecture bites. If your data is scattered across nine services with nine schemas, assembling coherent context becomes its own project — the point I made in The 'Context Window' Paradox: Why Fragmented Microservices Are Sabotaging Your AI Strategy.
2. Cost has no natural ceiling
Traditional features have roughly predictable unit economics. AI features do not. Cost scales with tokens, tokens scale with user behaviour, and user behaviour has a long tail. One user pasting a large document repeatedly can cost more than a hundred normal users.
Before launch you need, at minimum:
- A per-user and per-tenant budget, enforced in code, not monitored in a dashboard after the fact
- Token accounting per request, stored — otherwise you cannot tell which feature is expensive
- A cheaper model for the easy majority, reserving the expensive one for requests that need it
- Caching for repeated questions, which in most products is a large share of traffic
That last one is the highest-leverage and most-skipped. The caching discipline is the same one I described for conventional backends in the 4-layer caching strategy — the layers change, the thinking does not.
3. Latency changes the architecture
An LLM call takes seconds. That single fact invalidates the request/response shape most web features are built on.
You have three options and should pick deliberately: stream the response so perceived latency drops even though total time does not; make it asynchronous — job queued, user notified — which is right for anything over a few seconds; or precompute during ingestion rather than at request time, which is the best answer whenever it is available.
What you must not do is hold an HTTP request open for eight seconds under concurrency. That is how one slow dependency exhausts your connection pool and takes down endpoints that have nothing to do with AI. The back-pressure machinery in Handling High Concurrency in Node.js is not optional here — it is the difference between a slow feature and an outage.
4. Failure modes you have not met before
Conventional dependencies fail loudly — timeout, 500, connection refused. You handle those already. LLMs add a category most teams have never designed for: confident, well-formed, wrong output.
Nothing throws. The response is valid JSON. It is simply incorrect. Which means:
- Validate structure, always — schema-check every response, never trust shape
- Constrain the output space — enums over free text wherever possible
- Make the boundary explicit to the user — say what is generated
- Keep a human in the loop anywhere the cost of being wrong is real
- Log inputs and outputs, or you cannot investigate the complaint you will get
Plus the ordinary ones: rate limits under concurrency, provider outages, silent model deprecations, and behaviour drift when a provider updates a model you pinned by name but not by version.
5. Evaluation, or you are flying blind
This is the one teams skip entirely and regret most.
You cannot improve what you cannot measure, and "it seemed better in testing" is not measurement. Before launch you need a set of real inputs with known-good outputs — even thirty is transformative — and a way to run them on every prompt change. Without it, every prompt tweak is a coin flip, and you will eventually make the feature worse while believing you improved it.
The structural version of this problem is what I was getting at in Scaling AI-Driven Engineering: Why Unstructured Frameworks Are Costing Your Team 30% in Velocity.
Adding AI to something that already exists
Retrofitting is a different job from greenfield, and usually harder. The constraints are real: your data was not modelled for retrieval, your latency budget is already spent, and your users have expectations about how the product behaves.
The sequence that works:
- Pick one narrow, high-value use case. Not "AI assistant". Something like "summarise this specific record type."
- Check whether your data can answer it. Usually the honest answer is "not yet" — and that data work is the actual project.
- Ship it read-only first. Suggestions before actions. Let it be wrong cheaply while you learn the failure modes.
- Instrument acceptance. How often do users take the suggestion? That is your real quality metric.
- Only then let it write, act, or spend money.
The infrastructure implications are in Adding AI to Your Existing Product? Here's the Backend Infrastructure Nobody Talks About.
The honest cost picture
What actually drives the number:
- Retrieval quality required. Casual summarisation is cheap; anything users will act on financially is not.
- Whether your data is ready. This is usually the majority of the work and is almost always underestimated.
- Latency tolerance. Async is much cheaper to build than fast-and-synchronous.
- Blast radius of a wrong answer. Determines how much validation and human review you must build.
What makes it expensive is shipping the demo, discovering these one at a time in production, and retrofitting each under pressure.
The short version
The model is a commodity and getting cheaper. Your advantage is in the parts nobody demos: retrieval quality, cost control, latency shape, failure handling, and an evaluation loop that tells you whether changes help.
Teams that build those ship AI features that survive contact with users. Teams that skip them ship an impressive demo and spend the following quarter discovering, one incident at a time, what this article listed.
If you are about to commit a sprint to an AI feature, the cheapest hour you can spend is mapping the backend work it actually implies — before the sprint, not during it.
Read next
- Beyond the Hype: The Engineering Reality of Running Generative AI in Production — the strategic version of this argument.
- Adding AI to Your Existing Product? — the retrofit path in detail.
- Building AI-Powered Web Applications: A Complete Guide — end-to-end walkthrough for a greenfield build.
- Implementing the ChatGPT API in Your Application — the integration mechanics, concretely.
- Scaling AI-Driven Engineering — why unstructured AI adoption costs velocity.
- How Agentic AI Is Transforming the Way We Code and Review — AI inside the engineering process rather than the product.
- Transforming Your Business with Claude Sonnet — model choice and what it changes in practice.
- Machine Learning for Developers: Getting Started — the fundamentals, if this is new ground.
- Computer Vision Applications in Modern Apps — where the same production constraints apply to vision.
