Adding an AI feature to a product starts with a deceptively simple question: which model should we use? There are several major providers, each with families of models at different sizes and prices, plus open models you can host yourself. New versions appear every few months, and benchmark tables change just as often.
The good news is that you do not need to follow every release. You need a way to choose for your feature, test the choice, and switch later if something better comes along. This post is the framework I use.
Start from the task, not the leaderboard
Public benchmarks measure general abilities. Your feature has a specific job: summarizing support tickets, extracting fields from invoices, drafting replies in Arabic and English, classifying requests, answering questions from your own documents.
Write down:
- The input: what text, documents or images the model receives, and how long they are.
- The output: free text, a short label, or structured data such as JSON.
- The quality bar: what a good answer looks like, and what a harmful mistake looks like.
- The volume: how many requests per day you expect, and how fast responses must be.
- The data: whether inputs contain personal or confidential information.
These answers narrow the field quickly. A classification task with short inputs and high volume needs something very different from a long-document analysis run a few times a day.
Build a small evaluation set first
Before comparing providers, collect real examples: thirty to a hundred inputs from your own domain, with the expected output or at least notes on what a good answer contains. Include the hard cases: messy inputs, mixed languages, edge cases, attempts to misuse the feature.
Then run each candidate model on the same set with the same prompt and compare. This is the most reliable way to choose, and the same set later protects you when you change prompts or models. I wrote about this in detail in how to evaluate LLM output before your users do.
The criteria that matter
Quality on your task
The only quality measure that counts is performance on your evaluation set. Larger models are usually better at complex reasoning and nuanced writing; smaller ones are often just as good at narrow, well-defined tasks such as classification or extraction, at a fraction of the cost and latency.
Cost per request
Providers charge per token of input and output, with large differences between model sizes. Estimate cost per request from your real inputs, then multiply by expected volume. Check prices on the provider's current pricing page, because they change often. Watch the input side: if every request includes long instructions or documents, input tokens dominate the bill. Prompt caching, offered by several providers, can reduce this for repeated content.
Latency
For interactive features, such as a user waiting for a draft, response time matters as much as quality. Measure time to first token and total time on your real inputs. Streaming the response makes waiting feel shorter. For background tasks, latency matters less, and batch processing options can lower costs.
Context size
If your inputs are long documents or long conversations, check the model's context window. A bigger window is not always the answer, though: retrieving only the relevant parts of your documents is usually cheaper and often more accurate than sending everything.
Structured output and tools
If your feature needs reliable JSON, or needs the model to call functions in your system, check how well each provider supports structured outputs and tool calling. Native support saves a lot of fragile parsing code.
Languages
If your users write in Arabic, French or a mix of languages, test those specifically. Quality varies by language and dialect more than general benchmarks suggest. This matters for products serving users in Lebanon, the GCC and the wider Middle East, where messages often mix Arabic, English and transliterated Arabic.
Data handling and privacy
Read the provider's terms for API usage:
- Is your data used for training? Under which settings?
- How long are inputs and outputs retained?
- Where is the data processed and stored? Some clients require a specific region.
- What certifications and agreements does the provider offer?
Send only what the task needs. Remove personal details that do not affect the result. For sensitive workloads, consider providers offering regional hosting or open models hosted on your own infrastructure.
Reliability and limits
Check rate limits for your account tier, published status history and how errors are reported. Plan for timeouts and outages: retries with backoff, a fallback model, and a graceful message in the product when the AI part is unavailable.
Hosted APIs or open models?
Open models you host yourself give you full control over data and no per-token fees. In exchange, you take on GPU infrastructure, scaling, updates and usually some loss of quality compared to the strongest hosted models. For most small and medium products, a hosted API is the practical starting point. Self-hosting becomes attractive with very high volume, strict data requirements, or a narrow task where a smaller open model performs well enough.
Avoiding lock-in
Models improve and prices change. Design so switching is easy:
- Wrap model calls behind a small internal interface in your backend, so the rest of the code does not know which provider is used.
- Keep prompts in one place, versioned like code.
- Keep your evaluation set, so you can test a new model in an afternoon.
- Avoid depending on provider-specific features unless they bring clear value.
- Log inputs and outputs (with privacy in mind) to compare models on real traffic.
With this in place, moving to a newer or cheaper model is a measured decision, not a migration project.
Security considerations
Whatever the provider, any text the model reads can contain instructions meant to manipulate it. If the model can call tools or see sensitive data, design for that risk from the start. See prompt injection risks in LLM apps. Keep API keys on the server, never in the mobile app or browser, and set spending limits with the provider.
A simple selection process
- Define the task, inputs, outputs, quality bar, volume and data constraints.
- Build an evaluation set from real examples.
- Shortlist two or three models across providers and sizes.
- Run the evaluation, and measure cost and latency on the same inputs.
- Pick the smallest model that meets the quality bar.
- Wrap it behind an interface, add monitoring, and launch to a small group.
- Re-run the evaluation when new models are released or prompts change.
Common mistakes
- Choosing the biggest model by default, and paying for capability the task does not need.
- Choosing from benchmarks without testing on real inputs.
- Ignoring language quality for Arabic or mixed-language users.
- Hardcoding one provider throughout the code.
- Sending far more context than needed.
- No plan for outages or rate limits.
Summary
The right model is the smallest one that does your task well, at a cost and speed that suit your product, under data terms you are comfortable with. Find it with your own evaluation set, and keep your code ready to switch.
For the wider picture of building AI into a product, see adding an AI feature to your product and AI agents for a small business. If you want help choosing and integrating a model, read about me and get in touch.