The same model: how providers differ
Same name, different service. The same open-source model can be offered by many providers. The previous lesson checked "is it really this model"; this lesson checks "is this provider's service good to use": how fast the first token arrives, how many tokens per second, how long an input it accepts, whether tool calling and JSON work out of the box, and whether it times out all the time.
What you'll be able to do after this lesson
- Name the aspects in which "the same model" differs across providers
- Use a checklist to run an evaluation of candidate providers
- Choose a provider based on your own task, not on the model name
What the research says: the model name isn't enough
The 2026 paper "When Is the Same Model Not the Same Service?" (arXiv 2605.02821) measured real-world usage of hosted open-source model APIs in the fourth quarter of 2025. Its core claim: in production, the unit you actually consume is not the "model" but the service, jointly determined by these factors:
| Component | Meaning |
|---|---|
| Model variant | Which exact version it is |
| Protocol behavior | Which interface formats are supported, e.g., OpenAI-compatible, Anthropic-compatible |
| Context capacity | How long an input and output it can actually accept |
| List price | Cost per million tokens |
| Latency and throughput | How long until the first token (TTFT), how many tokens per second (TPS) |
| Reliability | Whether it times out, and whether it can recover after errors |
| Task feasibility | Whether your task can actually run through this provider |
And all of these change over time.
The paper's three findings:
- List prices are relatively consistent; other dimensions vary more. Providers' list prices are fairly close, while latency, throughput, context length, protocol support, and failure modes differ much more
- Being listed doesn't mean being used. A model may be offered by many providers, but actual usage is concentrated in a few
- The task determines the choice. Input and output lengths vary greatly across applications, so choosing a provider depends on "which model, which provider, what task, what time"—you can't just look it up by model name
The paper also worked two examples: with feasibility constraints satisfied, routing requests to suitable providers reduced the cost of Qwen3-32B by 37.8%; for DeepSeek-V3.2, average throughput improved by about 90% over connecting directly to the official API.
Quiz timeTwo providers charge the same for DeepSeek. Does that make them the same for you?
- I know the main aspects in which providers differ
Step 1: Filter hard requirements first
Some conditions rule a provider out immediately if unmet—no need to compare scores. The paper's advice: when your application requires strict output formats or tool-calling compatibility, treat these as must-meet conditions rather than bonus points.
Check each item against your task:
- Supports the interface format your code uses (OpenAI Chat Completions, Anthropic Messages, or OpenAI Responses)
- Context length, maximum input length, and maximum output length are enough for your longest request
- If you need caching, caching is supported
You should end up with: a shortlist of remaining candidate providers.
Step 2: Test across five dimensions
Some evaluation platforms break providers' capabilities into five dimensions. You can borrow this framework and test with your own requests:
| Dimension | What to test | How to test |
|---|---|---|
| Response quality | Following instructions, reasoning questions, domain questions, image-text questions | Use your own eval set and record accuracy |
| Response efficiency | First response, body wait time, generation rate, total task duration | Streaming requests; record time to first token and tokens per second |
| Tool cooperation | Picking the right tool, multi-tool calls, not misusing tools, continuing after receiving tool results | Prepare a few requests with tool definitions and check whether the calls are correct |
| Format compatibility | Whether it complies when JSON is specified, whether common parameters work on the first try, whether complex parameters work | Send the real request parameters from your code once |
| Cache reuse | Whether repeated requests hit the cache, input reuse ratio in multi-turn conversations | Send the same long prefix multiple times and check cached tokens in the bill |
When testing, use your own requests: the same prompts, the same parameters, the same streaming settings, sent from the same region.
Quiz timeYour app is a customer service bot—the longer users wait, the more likely they leave. Which of the five dimensions should you look at first?
- I've tested every candidate provider with my own requests
Step 3: Test more than once
The paper counts measurement time as part of the service: latency, throughput, and reliability all change over time. Testing once only shows you that single moment.
- Test during the hours you actually use it
- Test at least once across several days and several time slots
- Record timeout and error counts, not just average speed
Evaluation platforms' reports usually also note: only the most recent valid sample is shown, and scores are affected by output length, thinking toggles, and so on—for reference only. When you test yourself, record these settings as well.
Step 4: Factor in the risks too
The paper breaks "the risk of using one provider" into several categories, which you can use directly as a checklist:
| Risk | What you'll encounter |
|---|---|
| Timeouts | Requests hang and need retries |
| Format errors | JSON or tool-call parsing failures |
| Context truncation | Long inputs get cut off, answers miss information |
| Billing | Actual charges don't match expectations |
| Identity or quality drift | The underlying model changed, see the previous lesson |
The paper also reminds: a complete comparison looks not only at cost and speed, but also at retry costs, failure rates, truncation rates, tool-call and JSON format compliance rates, and task quality.
Checkpoints
- The same model is a different service at different providers
- List prices are similar; the differences are in latency, throughput, context, protocols, and failure modes
- Filter hard requirements first, then test across five dimensions, multiple times across time slots
- Risk checklist: timeouts, format errors, truncation, billing, quality drift
Common misconceptions
"Just pick the cheapest one." Prices are usually similar across providers. The costs from failed retries, truncation, and format errors can exceed the per-token price differences.
"The fastest one in a single test is the fastest." Speed varies by time of day. Testing once may catch that provider at an idle moment.
Next section
If the model seems to be "getting dumber" as you use it, is that real? What's going on with model degradation.