How to Verify Whether an API Is the Real Thing
Methods available to ordinary users can only uncover large-scale substitutions, not subtle changes such as quantization. The researchers' conclusion: relying on software detection alone is fundamentally unreliable; truly reliable guarantees require hardware-level trusted execution environments (TEEs). Below, from simplest to strictest, we look at each of the five methods and how far it can go.
What You Will Be Able to Do After Learning This
- Know the capability limits of each of the five detection methods
- Do a rough screening with your own evaluation set
- Know what kind of guarantee to ask the provider for
The Five Methods in One Table
The conclusions come from UC Berkeley's "Are You Getting What You Pay For?" (arXiv 2504.04715) and Stanford's "Model Equality Testing" (arXiv 2410.20247).
| Method | How to Do It | Can Detect | Cannot Detect |
|---|---|---|---|
| Ask about identity | Ask "what model are you" | Detects almost nothing | Parameter count, quantization; correct-answer rate on spec-type questions is under 5% |
| Run benchmarks | Run public benchmarks on the API and compare with official scores | Large-scale substitutions | Small changes such as quantization (differences within one standard deviation) |
| Compare greedy outputs | Set temperature to 0 and compare word for word with official weights run locally | Different outputs | Cannot distinguish substitution from different inference settings; prone to false positives |
| Two-sample test | Sample from both the API and a reference model and test whether the distributions match | Various changes, median power 77.4% | Requires access to the reference model's outputs |
| Trusted execution environment | The provider runs the model in a TEE and issues an attestation | Cryptographic guarantees | Requires the provider's cooperation |
Method 1: Ask About Identity
Easiest, and least useful. Results from the paper's testing: models usually only name their family or developer, rarely their parameter count or quantization status; when asked directly about specs, the success rate is under 5%. When the system prompt is rewritten to impersonate another identity, most models fail to recognize themselves.
Conclusion: Do not use this.
Method 2: Run Benchmarks, or Run Your Own Evaluation Set
Run public benchmarks on the API and compare with officially published scores. The paper's conclusion: benchmark-based detection can expose large-scale substitutions, such as swapping in a much smaller model; but the differences between quantized versions and the original on MMLU, GSM8K, MATH, and GPQA Diamond are usually within one standard deviation of repeated runs and can be masked by natural fluctuation. Factors like temperature and caching also make judgment harder.
You can substitute the evaluation set you built in Part 3 for public benchmarks: run the same set of questions on both the official channel and the API you want to verify, and use error bars to judge whether the gap is real.
Conclusion: Suitable for ordinary users doing rough screening—it can detect "a smaller model was swapped in," but not "a quantized version was swapped in."
Quiz timeYou ran your own benchmark on a relay API and the score differed from the official channel by only 1 point, within the margin of error. What does this tell you?
- I understand that matching benchmark scores does not prove authenticity
Method 3: Compare Greedy Outputs
Set the temperature to 0 so the model always picks the most likely next token, and the output should be deterministic. Compare the API's output word for word against the output of the official weights running locally.
The problem: inference nondeterminism across providers can make exactly the same model produce different outputs, causing false positives. Also, this method requires you to run the official weights locally, so it only applies to open-weight models.
Conclusion: Matching outputs are a good sign; a mismatch alone does not settle the matter.
Method 4: Two-Sample Test
Stanford's approach treats the question as a statistical test: draw batches of samples from both the API and a reference model, and test whether the two come from the same distribution. Their MMD-based test, with a simple string kernel, needs only about 10 samples per prompt on average, with a median detection power of 77.4% across various modifications.
Using this method, they audited commercial APIs offering 4 Llama models in the summer of 2024, and found that 11 of 31 endpoints produced output distributions different from the reference weights released by Meta.
Conclusion: The most effective software-based method today, but it requires samples from a reference model, making it suitable for technically capable teams auditing open-weight models.
Method 5: Trusted Execution Environment (TEE)
The Berkeley paper argues that purely software-based methods are fundamentally unreliable: statistical tests based on output text require large numbers of queries and are ineffective against subtle substitutions; methods based on log probabilities are defeated by the inference nondeterminism inherent in production environments.
They advocate using trusted execution environments: the model runs in a hardware-isolated environment that can issue a cryptographic attestation of which weights are running inside. The paper argues the added overhead of doing this is small.
Conclusion: The only method that can provide a reliable guarantee, but it requires provider support. What you can do is ask the provider whether they offer this kind of attestation.
How Far Can an Ordinary User Go
- Run your own evaluation set on both the official channel and the API you want to verify
- Use error bars to judge the gap; a clear gap means it is very likely not the same model
- No clear gap only means there was no large-scale substitution
- For quality-critical scenarios, prefer official channels, or ask the provider to state the deployed version (original or quantized) and whether they can provide an attestation
The paper's code is publicly available at github.com/sunblaze-ucb/llm-api-audit for those with the technical skills to reference.
Before Concluding "Same Model," Do These Steps First
The 2026 paper "When Is the Same Model Not the Same Service?" (arXiv 2605.02821) provides an audit procedure in its appendix. Before claiming that two providers offer the same model, first do the following:
- Fix the prompt set, decoding parameters, streaming mode, tokenizer, request format, and the region requests are sent from
- Sample repeatedly from each endpoint; when official weights are available, also sample from the reference implementation
- Compare distributions with a two-sample test (i.e., Method 4)
- Report separately: exact version match, match with the provider's claimed version, and merely the same family—these three conclusions are different
Checkpoints
- Asking about identity is useless; correct-answer rate on spec-type questions is under 5%
- Benchmarks can only detect large-scale substitutions
- Greedy output comparison is prone to false positives
- The two-sample test is the most effective, with median power 77.4%
- Only trusted execution environments can provide reliable guarantees
Common Misconceptions
"There's a tool that can detect whether an API is genuine with one click." Research shows purely software-based methods have fundamental limitations. Any detection can only say "no problems found," never "guaranteed to be fine."
"If the scores match, I can relax." Quantized versions score almost identically to the original on benchmarks.
Next Section
After confirming authenticity, you still need to check whether the service is good to use: Same Model, Different Providers: Where They Differ.