fix(agent-server): survive webhook rate limiting and reap stale run tasks

[shanemort1982]

HUMAN:

Follow-up to my report in #3842, which VascoSch92 reproduced. We caught the trigger in the wild on our fleet: the app server rate-limits its own sandbox's webhook posts (429), the stock poster gives up, the app-side transcript freezes, and the lingering run task answers 409 forever with no self-heal. Fix validated end to end on a self-hosted box (evidence below) on top of the unit suites.

---

AGENT:

Why

Fixes #3842.

Two defects compound into the wedge:

1. WebhookSubscriber._post_events retries on a fixed delay with no backoff, does not honour Retry-After, fires overlapping volleys (each buffer-full flush posts concurrently), and after exhausting retries only tries again when a NEW event arrives. Against a rate-limited downstream (the OpenHands app server applies InMemoryRateLimiter(requests=10, seconds=1) per client IP to ALL routes, including /api/v1/webhooks/events/...) the concurrent volleys trip the limiter, keep it tripped, and delivery stalls; the queue trim then drops events permanently. Production signature: hundreds of Webhook post attempt 4 failed: 429 / Failed to post events ... after 4 attempts per second for 90+ minutes while the app-side transcript sat frozen (the agent's local store reached 8,622 events; the app side had 270).

2. EventService.run() raises conversation_already_running whenever _run_task is alive, even while execution_status reports non-RUNNING. A run task that never completes therefore wedges the conversation forever - the self-heal added in fix(agent-server): re-run input stranded in a finished run's cleanup tail #3364 never fires because the task never reaches its cleanup tail.

Summary

• Webhook poster: serialise flushes behind a lock; exponential backoff between attempts, honouring Retry-After; after a failed volley enter an exponentially growing cooldown (1s floor, 60s cap) instead of hammering; re-arm the flush timer on failure so delivery resumes WITHOUT requiring a new event; preserve delivery order by prepending re-queued events.
• EventService.run(): when a live _run_task coexists with non-RUNNING execution_status, start a staleness clock; if the SAME task is still lingering on a later attempt past STALE_RUN_TASK_REAP_SECONDS (120s, comfortably beyond the 30s-bounded wait_for_pending tail), cancel and reap it, then start the requested run. The task's finally now only clears its own task reference so a reaped task cannot clobber a replacement run.

How to Test

Commands run in a clean checkout with the patch applied:

• uv run pytest tests/agent_server/test_webhook_subscriber.py tests/agent_server/test_event_service.py -> 150 passed. 5 new tests cover Retry-After/backoff with zero event loss and order preservation, cooldown queueing without posts, and the stale-run-task reaper (first observation never reaps; same lingering task past the grace period is cancelled and cleared; a different task restarts the clock). 5 existing assertions updated where the fixed-delay retry cadence intentionally changed to exponential.
• uv run ruff check / uv run ruff format on the four touched files: clean.

End to end on a self-hosted deployment (agent-server image built from this branch's changes applied to v1.30.0, --target source; OpenHands app at current main). The app's global rate limit was tightened to requests=1/second to force the storm, then a long multi-step streaming conversation was driven through the full app -> sandbox -> webhook path:

• Stock agent-server (1.30.0): webhook volleys tripped repeated 429s and logged the permanent give-up (Failed to post events to webhook ... after 4 attempts); the app-side transcript stalls until unrelated later events happen to re-trigger a flush - or forever when the storm persists (the production case above).
• Patched agent-server: the serialised, paced poster stayed under the SAME limiter - zero 429s - and the full transcript including the final assistant message reached the app store; the conversation completed normally. Under a separately induced sustained-failure scenario, the cooldown machinery held: re-arms progressed 10s -> 20s -> 40s -> 60s (capped), no hot loop, no drops within the queue bound.

[all-hands-bot]

[Automatic Post]: It has been a while since there was any activity on this PR. @shanemort1982, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up.

This comment was created by an AI agent (OpenHands) on behalf of the user.