When an LLM Request Fails

An LLM request can fail after the model has already returned: a tool times out, a database write fails, or the client disconnects. Retrying the whole request may repeat an action that already succeeded.

These are the checks I'd make before putting an LLM-backed service in front of users.

Bound the waiting

Set a timeout for each provider and tool call, plus a deadline for the complete request. The individual timeouts need to fit inside that deadline. Stop retrying when there isn't enough time left to finish.

If the client leaves, cancel work that is no longer useful. Record any action whose outcome is still unknown so it can be checked later.

Make retries safe

Give actions a stable request ID and persist their result. A repeated request should return the previous outcome instead of creating another booking, message, or payment.

When a provider doesn't support idempotency, decide how to reconcile an uncertain result before enabling automatic retries. A timeout tells you that you didn't receive an answer; it doesn't prove nothing happened.

Limit the queue

A queue needs a maximum size, expiry, and a policy for overload. Otherwise, a brief slowdown can leave users waiting for work that is already irrelevant.

Some optional steps can be skipped under load. Others, such as authorization checks, can't. Define that distinction explicitly instead of letting failures choose it for you.

Leave a trace you can follow

Carry the same request ID through retrieval, model calls, tools, and writes. Record durations, retries, provider errors, and the final outcome. Keep secrets and unnecessary personal data out of logs.

The trace should answer a concrete question: did the request fail during retrieval, generation, a tool call, or saving the result?

Retest the failure

Add reproduced failures to a regression set. Include slow tools, duplicate requests, malformed output, missing retrieval evidence, and attempts to access another user's data.

Run these scenarios when changing prompts, models, or orchestration. A response that reads better is still a regression if it repeats an action or crosses an access boundary.