Almost every product roadmap now has an "add AI" item on it. Some of those features become the reason customers stay. Many become a chat box nobody uses, a surprising monthly bill, or a support headache when the model confidently says something wrong. The difference is rarely the model. It is the scoping, the plumbing around the model, and whether anyone measured if it works.
This is how I approach adding a large language model (LLM) feature to an existing web or mobile product.
Start from a task, not from "a chatbot"
The best LLM features remove a specific, repetitive piece of work that users already do. Good candidates usually involve reading or writing unstructured text:
- Summarizing long records, tickets, notes or reports into something a person can scan.
- Turning free text into structured data: extracting fields from an email, a form or a document.
- Drafting replies, descriptions or reports that a human reviews and sends.
- Searching your own content by meaning rather than exact keywords.
- Classifying and routing incoming requests.
A general-purpose chat assistant is the hardest version to get right, because the space of questions is unbounded. A narrow feature, such as "draft a job summary from these technician notes", is easier to evaluate, cheaper to run and easier for users to trust.
A quick test: can you describe the input, the expected output, and how a human would judge if the output is good? If not, the feature is not ready to build yet.
Pick the simplest architecture that works
There are roughly four levels, and most products should start at the first or second:
- Prompting a hosted model with clear instructions and the relevant data inserted into the prompt. Fast to build, often good enough.
- Retrieval-augmented generation (RAG): when the model needs knowledge from your documents or database, you search for the relevant pieces first and pass them in. This keeps answers grounded in your data, and you can show sources. Postgres with the pgvector extension is often enough for search at the scale of a typical product; you may not need a separate vector database.
- Tool use / function calling: the model can call specific, permissioned functions in your backend (look up an order, create a draft). Powerful, but every tool is an action you must validate and authorize as if a user had called the API directly.
- Fine-tuning a model on your examples. Useful for consistent style or format at high volume, rarely the first step, and it does not reliably teach a model new facts. Retrieval does that better.
Ask for structured output (JSON matching a schema) whenever the result feeds into your application rather than straight to a person's eyes, and validate it before you use it.
Build an evaluation set before you polish prompts
This is the step most teams skip, and it is the one that matters most. Collect a few dozen to a few hundred real examples of inputs, with what a good output looks like or the criteria it must meet. Then every change to a prompt, model or retrieval setting gets run against that set.
Without it, you are tuning on vibes: a prompt change fixes the example you were looking at and quietly breaks five others. With it, you can switch model providers, cut costs with a smaller model, or upgrade versions with evidence instead of hope.
Evaluation can be simple: exact checks where possible (did it extract the right date?), rule-based checks (is the output valid JSON, under a length, in the right language?), and human review for the rest. Using a second model as a judge can help at scale, but spot-check the judge too.
Design for being wrong
LLMs produce fluent, plausible text whether or not it is correct. The product design has to assume some outputs will be wrong:
- Keep a human in the loop for anything consequential. Drafts that a person approves are far safer than actions taken automatically.
- Show sources for answers based on your data, so users can verify.
- Make it easy to edit, retry or ignore the output.
- Constrain the scope: tell the model what it should refuse, and enforce limits in code rather than only in the prompt.
- Have a fallback when the provider is slow or down: a timeout, a clear message, and the feature degrading gracefully instead of breaking the page.
Security: treat model input and output as untrusted
Prompt injection is the defining security problem of LLM features. If the model reads content that a third party controls, such as an email, a web page or an uploaded document, that content can contain instructions, and the model may follow them. Practical defences:
- Never give the model more permissions than the user it is acting for.
- Validate every tool call server-side as a normal API request, with authorization checks.
- Require confirmation for actions with side effects, like sending, deleting or paying.
- Do not render model output as raw HTML; escape it like any user input.
- Keep secrets and other customers' data out of prompts entirely.
Privacy and data handling
Know what data goes to the model provider and under what terms. Check whether the provider retains or trains on API data, which region processes it, and whether that is compatible with your contracts and your users' expectations. Strip or mask personal data you do not need for the task. Update your privacy policy to say what you actually do. For sensitive workloads, a self-hosted open-weights model is an option, but it moves the operational burden, including GPUs, scaling and updates, onto you.
Cost and latency
LLM costs scale with tokens, so they scale with usage in a way normal hosting does not. Keep them predictable:
- Send only the context the task needs; long prompts are the biggest cost driver.
- Use the smallest model that passes your evaluation set, and reserve larger models for the hard cases.
- Cache results for identical inputs.
- Set per-user and per-account limits, and alert on spend.
- Stream responses for anything user-facing, so the interface feels responsive even when generation takes seconds.
- Move long jobs, like processing a batch of documents, to a background queue instead of an HTTP request.
Observe it in production
Log the prompt version, model, latency, token counts, cost and outcome of each call (with personal data handled appropriately). Capture user feedback such as thumbs up or down, or whether the draft was edited heavily before sending. Those signals tell you where the feature is failing and feed new cases back into your evaluation set. Dedicated LLM observability tools exist, but even structured logs in your existing monitoring are a big step up from nothing.
Keep the provider swappable
Models and prices change quickly. Put the model call behind a small internal interface in your backend, keep prompts versioned in code, and avoid spreading provider-specific details through the codebase. With an evaluation set in place, switching providers or models becomes a measured decision instead of a risky rewrite.
A sensible first project
- Pick one narrow, frequent task with clear success criteria.
- Collect real examples and define what good looks like.
- Build the simplest version: a prompt, or retrieval plus a prompt, returning structured output.
- Ship it to a few users behind a feature flag, with human review and feedback built in.
- Measure quality, cost and usage, then decide whether to expand.
AI features are still software features: they need clear requirements, a backend that validates everything, monitoring and a way to roll back. That is the same discipline behind the products in my work, such as the NeonTask field-service platform. If you are deciding whether and how to add an LLM feature to your product, get in touch and we can scope it properly.