Skip to main content
TemporalCustomer feedback

An SRE running LLM agent workflows on Temporal details a multi-day debugging issue where exceeding the 2 MB payload limit triggered deterministic silent retry loops that manifested externally as...

Source: Autobound Signal API · 56 signals tracked for Temporal · updated

Get Temporal's live signals in Claude or ChatGPT, or by API

AskWhat changed at Temporal this week? Use Autobound signals for temporal.io.

What happened

An SRE running LLM agent workflows on Temporal details a multi-day debugging issue where exceeding the 2 MB payload limit triggered deterministic silent retry loops that manifested externally as execution timeouts.

Source

Post

Highlighted: the lines this signal was extracted from

We run LLM agents as Temporal workflows (durable execution engine; think step functions, but in code). The loop is: call a tool, append the result to the message list, hand the whole list to the next inference step, repeat. Nothing exotic. One of these kept dying at 30 minutes. Not roughly 30. Exactly 30:00, which is its execution timeout. So naturally I assumed a slow tool. Pulled activity latencies, nothing. Checked the upstream API rate limits, fine. Added more logging, restarted things, the usual. Still died at 30:00 every single time. I was pretty convinced something upstream was hanging on us. What finally cracked it was giving up on the dashboard and reading the raw workflow history. And there it was, repeating every few seconds for half an hour: WorkflowTaskFailed cause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE 3.9 MB against a 2 MB limit. The workflow was trying to schedule the next step with a ~3.9 MB input. Server rejects it. SDK retries. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it’s indistinguishable from a hang, which is why I spent so long looking for one. Root cause was dumb in hindsight. One tool fetched commit details, and the service behind it returned the diff plus the full before and after content of every changed file...

Keep reading with a free account

The rest of this post, and every signal for Temporal, is in your free account.

Comments on the post

4 of 7 comments
  • “I’d say the dashboard wasn’t watching the right things. Maybe adding something like: Count of rejected or failed steps per workflow. A spike here would have shown the loop in minutes. Total size of what’s passed between steps, compared with the 2 MB limit, not just each item. Jobs that end by hitting their timeout instead of finishing. Honestly though, it’s a bit of trial and error until y”

    u/Ok_Pipe_96312 points · Nov 11, 2026View

  • “queues show the same shape with a poison message. A payload that can never succeed keeps getting redelivered until a max-receive count moves it to a dead-letter queue, like an SQS redrive policy, and that turns a silent loop into a visible failure.”

    u/Future_AGI2 points · Nov 11, 2026View

  • “the per item checks passing while the total kept growing is the bit that stood out to me here . did u add a size check before scheduling the next step too”

    u/Ok-Listen-21621 points · Nov 13, 2026View

  • “Reading the raw history instead of the dashboard is the lesson I keep relearning too. Did you end up capping tool results before they get appended, or storing big ones outside the workflow and passing a reference?”

    u/Accurate_Bag90121 points · Nov 11, 2026View

Extracted by Autobound

From the Signal API record
Signal
Customer feedback

What this signalsUser posts often show product pain before it reaches reviews or churn.

Subreddit
r/sre
Event date
Nov 2026

The full record

From the Signal API record

Numbers

Mentions
1

Details

Timing
Completed
Category
Usability
Virality
Low
Post kind
Text
Prominence
Core
Company's role
Vendor

Topics and mentions

Topics

  • orchestration
  • debugging
  • reliability
  • llm

Flair

  • DISCUSSION

Products named

  • Temporal

Extraction

Sentiment
Neutral
Detected
Nov 10, 2026
signal_type
reddit-company
signal_subtype
customerFeedback

Use this data

Get every Reddit signal for Temporal and the companies you sell to, in the tools you already use.

  1. Ask Claude about it

    Connect Autobound to Claude, Claude Code or Cursor with MCP. Then ask: “What changed at Temporal this week?”

  2. Send it to your own tools

    The Signal API returns Reddit signals for any list of companies as JSON, for your CRM, warehouse or app.

  3. Try it free

    Sign up and spend your free credits on the companies you sell to.

    Start Free1,000 free credits

The API returns more than this page shows

This page shows a preview. The full reddit-company record in the Signal API and MCP can also have these 8 fields. Some fields are empty for some signals.

Company

  • linkedin_urlValue in the API
  • industriesValue in the API
  • employee_count_lowValue in the API
  • employee_count_highValue in the API
  • revenueValue in the API
  • descriptionValue in the API

Signal

  • signal_nameValue in the API
  • associationValue in the API
Show the full JSONThe record on this page and the API request

GET /v1/signals/402511ac-5162-5cc2-add4-deb2c9ebbb2e returns this record as JSON. POST /v1/companies/enrich returns every signal for temporal.io.

{
  "signal_id": "402511ac-5162-5cc2-add4-deb2c9ebbb2e",
  "signal_type": "reddit-company",
  "signal_subtype": "customerFeedback",
  "detected_at": "2026-11-10T18:25:22+00:00",
  "company": {
    "name": "Temporal",
    "domain": "temporal.io"
  },
  "data": {
    "nsfw": false,
    "stage": "none",
    "awards": 0,
    "timing": "completed",
    "topics": [
      "orchestration",
      "llm",
      "debugging",
      "reliability"
    ],
    "post_id": "1xsuit8",
    "summary": "An SRE running LLM agent workflows on Temporal details a multi-day debugging issue where exceeding the 2 MB payload limit triggered deterministic silent retry loops that manifested externally as execution timeouts.",
    "category": "usability",
    "comments": [
      {
        "url": "https://www.reddit.com/r/sre/comments/1xsuit8/comment/19sgjoy9kdrdv/",
        "depth": 0,
        "score": 2,
        "author": "Ok_Pipe_9631",
        "excerpt": "I’d say the dashboard wasn’t watching the right things. Maybe adding something like:\n\n Count of rejected or failed steps per workflow. A spike here would have shown the loop in minutes.\n\n Total size of what’s passed between steps, compared with the 2 MB limit, not just each item.\n\n Jobs that end by hitting their timeout instead of finishing.\n\n Honestly though, it’s a bit of trial and error until y",
        "posted_at": "2026-11-11T05:00:23.000Z",
        "author_url": "https://www.reddit.com/user/Ok_Pipe_9631/"
      },
      {
        "url": "https://www.reddit.com/r/sre/comments/1xsuit8/comment/19she5tkb93ad/",
        "depth": 0,
        "score": 2,
        "author": "Future_AGI",
        "excerpt": "queues show the same shape with a poison message. A payload that can never succeed keeps getting redelivered until a max-receive count moves it to a dead-letter queue, like an SQS redrive policy, and that turns a silent loop into a visible failure.",
        "posted_at": "2026-11-11T16:23:29.000Z",
        "author_url": "https://www.reddit.com/user/Future_AGI/"
      },
      {
        "url": "https://www.reddit.com/r/sre/comments/1xsuit8/comment/19skkbci2ff1g/",
        "depth": 0,
        "score": 1,
        "author": "Ok-Listen-2162",
        "excerpt": "the per item checks passing while the total kept growing is the bit that stood out to me here . did u add a size check before scheduling the next step too",
        "posted_at": "2026-11-13T11:02:50.000Z",
        "author_url": "https://www.reddit.com/user/Ok-Listen-2162/"
      },
      {
        "url": "https://www.reddit.com/r/sre/comments/1xsuit8/comment/19shlipbmbopw/",
        "depth": 0,
        "score": 1,
        "author": "Accurate_Bag9012",
        "excerpt": "Reading the raw history instead of the dashboard is the lesson I keep relearning too. Did you end up capping tool results before they get appended, or storing big ones outside the workflow and passing a reference?",
        "posted_at": "2026-11-11T19:08:27.000Z",
        "author_url": "https://www.reddit.com/user/Accurate_Bag9012/"
      }
    ],
    "evidence": [
      "[post] We run LLM agents as Temporal workflows (durable execution engine; think step functions, but in code).",
      "[post] And there it was, repeating every few seconds for half an hour: WorkflowTaskFailed cause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE 3.9 MB against a 2 MB limit.",
      "[post] The workflow was trying to schedule the next step with a ~3.9 MB input. Server rejects it. SDK retries. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it’s indistinguishable from a hang"
    ],
    "virality": "low",
    "post_date": "2026-11-10T18:25:22.000Z",
    "post_kind": "text",
    "post_text": "We run LLM agents as Temporal workflows (durable execution engine; think step functions, but in code). The loop is: call a tool, append the result to the message list, hand the whole list to the next inference step, repeat. Nothing exotic.\n\nOne of these kept dying at 30 minutes. Not roughly 30. Exactly 30:00, which is its execution timeout. So naturally I assumed a slow tool. Pulled activity latencies, nothing. Checked the upstream API rate limits, fine. Added more logging, restarted things, the usual. Still died at 30:00 every single time. I was pretty convinced something upstream was hanging on us.\n\nWhat finally cracked it was giving up on the dashboard and reading the raw workflow history. And there it was, repeating every few seconds for half an hour:\n\nWorkflowTaskFailed\n\ncause: WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE\n\n3.9 MB against a 2 MB limit.\n\nThe workflow was trying to schedule the next step with a ~3.9 MB input. Server rejects it. SDK retries. Workflow replays deterministically to the exact same state, emits the exact same command with the exact same payload, gets rejected again. Forever. From the outside it’s indistinguishable from a hang, which is why I spent so long looking for one.\n\nRoot cause was dumb in hindsight. One tool fetched commit details, and the service behind it returned the diff plus the full before and after content of every changed file...",
    "sentiment": "neutral",
    "subreddit": "sre",
    "event_date": "2026-11",
    "post_flair": [
      "DISCUSSION"
    ],
    "post_title": "Spent days debugging a \"30 minute timeout\" that was actually a silent retry loop. What I'd do differently.",
    "prominence": "core",
    "source_url": "https://www.reddit.com/r/sre/comments/1xsuit8/spent_days_debugging_a_30_minute_timeout_that_was/",
    "entity_role": "vendor",
    "post_author": "Horror_Farmer_2875",
    "upvote_ratio": 0.625,
    "mention_count": 1,
    "mention_surge": false,
    "subreddit_url": "https://www.reddit.com/r/sre/",
    "total_upvotes": 2,
    "comments_total": 8,
    "total_comments": 7,
    "post_author_url": "https://www.reddit.com/user/Horror_Farmer_2875/",
    "signal_category": "feedback",
    "comments_included": 4,
    "products_mentioned": [
      "Temporal"
    ]
  }
}

Long text fields are shortened on this page.

Looking up one signal by its id is free. Enrich costs 2 credits per signal returned; a call with no results is free.