Measure the workload. Not the hype.
A benchmark method and downloadable manifest for comparing useful work, response time, and failure behavior under stated conditions.
Start with an acceptance rule
Define what a useful answer must contain and how quickly the workflow needs it. A chat interface may care about first readable content. An extraction pipeline may need a complete, validated object. A background batch may prioritize accepted results over an entire processing window.
Keep quality fixed while comparing performance. An answer that finishes early because it omits required information is not equivalent work. HFAPI.com publishes a measurement method, not provider rankings, live telemetry, or invented speed claims.
Name each timing boundary
| Measurement | What it tells you | What it does not establish |
|---|---|---|
| First content | Time from your chosen start boundary to meaningful generated content. | Completion, correctness, or total response time. |
| Complete output | Time until generation is reported complete. | That the answer passes your parser or task checks. |
| Validated completion | Time until the application accepts the finished result. | Performance for a different task or traffic pattern. |
| Queue time | Time waiting for admission or an available worker. | The generation speed after work begins. |
| Accepted-task throughput | Tasks meeting the defined quality and deadline rules per interval. | An unlimited sustainable service capacity. |
Use a monotonic clock for durations measured in one process. Do not mix client-side and server-side boundaries without explanation. The vLLM metrics documentation describes separate server metrics for first-token, queue, generation, and end-to-end behavior.
Record the experiment conditions
The manifest contains fields for task, model revision, provider, region, input fixture revision, generation settings, arrival pattern, concurrency, warmup procedure, cache state, and retry policy. Its measurement arrays are empty by design. They are a place for your observations, not sample results presented as real data.
Record prompt and output lengths along with the counting method. Separate startup-sensitive runs from steady-state runs. State whether repeated inputs or shared prefixes benefit from a warm cache. Keep the traffic generator's own limitations visible so it does not silently become the bottleneck.
Keep every attempt in view
Count rejected requests, timeouts, invalid outputs, partial streams, and retries. Report successful-request latency alongside those counts rather than silently dropping the failures. For cost comparisons, include all attempts needed to obtain an accepted result.
Use an adequate sample for the percentile you report, and preserve the sample count and run duration beside it. A high percentile from a tiny experiment should not be presented as a stable description of rare behavior. Do not average unrelated percentile values and call the result a combined percentile.
Run a controlled comparison
Begin at low concurrency and increase it in bounded steps. Keep the model, prompt set, generation settings, and acceptance rules fixed while testing one change. Use only authorized traffic within your spending and rate limits. Stop the run when its budget or safety boundary is reached.
Write a conclusion that states what changed, what improved, what worsened, and what remains uncertain. A measurement describes the tested conditions, not a permanent property of a brand. Repeat the relevant test after a meaningful model, provider, or configuration change.
Read the methodology in depth
The latency benchmarking article develops the full process. The caching and batching guide explains why an optimization needs both performance and correctness tests. Use the reliability checklist before raising load.
Do not place credentials or private prompts in the manifest. Store your completed research record in an appropriate internal location and label any remaining estimates as estimates.
Download benchmark manifest