r/sre
Spent days debugging a "30 minute timeout" that was actually a silent retry loop. What I'd do differently.
- upvotes
- 2
- comments
- 7
Post
Highlighted: the lines this signal was extracted from
We run LLM agents as Temporal workflows (durable execution engine; think step functions, but in code). The loop is: call a tool, append the result to the message list, hand the whole list to the next inference step, repeat. Nothing exotic. One of these kept dying at 30 minutes. Not roughly 30. Exactly 30:00, which is its execution timeout. So naturally I assumed a slow tool. Pulled activity latencies, nothing. Checked the upstream API rate limits, fine. Added more logging, restarted things, the usual. Still died at 30:00 every single time. I was pretty convinced something upstream was hanging on us. What finally cracked it was giving up on the dashboard and reading the raw workflow history. And there it was, repeating every few seconds for half an hour: WorkflowTaskFailed cause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE 3.9 MB against a 2 MB limit. The workflow was trying to schedule the next step with a ~3.9 MB input. Server rejects it. SDK retries. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it’s indistinguishable from a hang, which is why I spent so long looking for one. Root cause was dumb in hindsight. One tool fetched commit details, and the service behind it returned the diff plus the full before and after content of every changed file...
Keep reading with a free account
The rest of this post, and every signal for Temporal, is in your free account.