Build the failure path first.
Deadlines, bounded queues, careful retries, and observable recovery turn a working demo into a more understandable integration.
Define useful success
Write down what the workflow needs: required fields, supporting evidence, a completion state, and an acceptable deadline. Track accepted tasks separately from successful network exchanges. A response that arrives after its consumer has abandoned the task is not an on-time success.
Decide how the user sees a rejected request, unavailable provider, invalid answer, and interrupted stream. Those states should not collapse into an endlessly spinning interface. Preserve enough context for a person to understand what can safely happen next.
Put a bound on every waiting room
Set an end-to-end deadline and carry the remaining budget through the request path. Include time spent waiting for a connection, worker, or downstream capacity. A fresh timeout at every stage can add up to an unexpectedly long total wait.
Bound queued work and concurrency. Consider payload sizes and task cost as well as request count. Reject work you cannot complete within the contract rather than accepting an unbounded backlog. Keep an independent path available for health checks and operating controls.
Match recovery to the failure
| Failure | A starting response | Avoid |
|---|---|---|
| Invalid credential | Stop and review permissions safely. | Repeated authentication attempts with the same invalid token. |
| Unsupported request | Inspect the route, task, and schema. | Retrying without changing the cause. |
| Rate limit | Apply documented guidance within the original deadline. | A fleet of clients immediately retrying together. |
| Timeout | Preserve uncertainty about upstream processing. | Assuming the original request never happened. |
| Partial stream | Mark the result incomplete and apply a task-specific recovery rule. | Displaying partial output as a complete validated answer. |
| Unknown external state | Reconcile through the supported operation mechanism. | Blindly repeating a state-changing instruction. |
This table is a design starting point. A specific provider or external system may expose additional semantics that your adapter needs to preserve.
Budget retries explicitly
Use a small, bounded retry budget only for failures classified as plausibly transient. Keep the original deadline, add a controlled delay where appropriate, and account for all attempts in cost and success reporting.
The HTTP semantics standard distinguishes safe and idempotent operations. Your application must still determine whether its particular request can be repeated. An arbitrary idempotency header does not add deduplication to an endpoint that does not support it.
Test degraded operation
Simulate a slow dependency, worker restart, interrupted connection, full queue, and malformed response. Decide whether stale cached data is acceptable for each task. A labeled older reference result may be useful in one workflow and unacceptable in a permission or account-state decision.
Make fallback behavior visible and keep it within approved processing routes. Record the fallback used and the reason. Test that monitoring can distinguish intentional rejection from an application crash or silent quality failure.
Create a small operating record
Record the configuration revision, owner, limits, expected failures, recovery steps, and stop condition. Include enough identifiers to reconstruct an operation without unnecessarily storing private content. Revisit this record after changing the model, provider, prompt, or output parser.
Use the API engineering article for the architecture, the streaming guide for partial-result handling, and the benchmark manifest to evaluate whether useful work remains within the stated deadline.