[Bug] Graceful BE shutdown can hang 10 minutes: StreamLoadRecorderManager is stopped after the HTTP server, its in-flight audit-table stream load blocks the join

[morningman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master (observed on commit 0b3406309dc-based CI run of #66789, 2026-08-15; the involved code paths are unchanged on current master and the bug is independent of that PR's changes)

What's Wrong?

A graceful BE shutdown (stop_be.sh --grace) can hang for up to 10 minutes and
then fail the pipeline's "stop grace" check. The hang is a shutdown-ordering
problem around StreamLoadRecorderManager:

1. doris_main stops the servers first — at 16:45:10-15 the log shows
   Http service stopped -> Brpc service stopped -> Backend Service stopped — and only then calls ExecEnv::destroy().
2. destroy() reaches SAFE_STOP(_stream_load_recorder_manager).
   StreamLoadRecorderManager::stop() sets the (atomic) _stop flag and
   join()s its worker std::thread.
3. The worker polls _stop once per second, but each iteration may call
   _load_if_necessary() -> _send_stream_load(), which PUTs the buffered
   audit records to the FE as an HTTP stream load with
   client.set_timeout_ms(DEFAULT_STREAM_LOAD_TIMEOUT_SEC * 1000) where
   DEFAULT_STREAM_LOAD_TIMEOUT_SEC = 600.
4. If such a load is in flight (or starts) in the window after the BE's own
   HTTP server has already been torn down, the load can never complete — the
   redirect target of the stream load is this BE's http endpoint — so the
   request only returns when the 600 s curl timeout expires.
5. stop()'s join() therefore blocks for up to 600 s. The regression
   pipeline's grace budget is timeout 10m and starts earlier, so the stop
   check always loses the race: Stop grace fail, and the pipeline
   kill -ABRTs the BE.

Evidence from the failing run (NonConcurrentRegression build, PR #66789 round;
474/474 test cases had passed, the only failure was the stop):

• ABRT-time stack of the main thread:

  2# __pthread_clockjoin_ex
  3# __interceptor_pthread_join
  4# std::thread::join()
  5# doris::ExecEnv::destroy()
  6# main
  

• be.INFO: main thread's last destroy-progress line is
  16:45:15.813 GroupCommitMgr is stopped, then silence for 10 minutes.
  In ExecEnv::destroy()'s order, the next steps after that stop are
  _routine_load_task_executor (ThreadPool-based; a stuck doris Thread::join
  prints Waited for ...ms trying to join warnings every second — none
  appear) and then _stream_load_recorder_manager, which is the only object
  in that stretch owning a raw std::thread (raw std::thread::join is
  silent, matching the log).

• The recorder worker was active during the run
  (15:47:52 Failed to load stream load records to audit log table: ...),
  and there is no completion/failure log for its last send after 16:45 —
  the final request never returned before the ABRT.

The window is narrow (an audit batch must be in flight within roughly one
worker iteration of the service teardown), which is why CI rarely hits it —
in the last 30 NonConcurrentRegression runs it fired once. But any BE with a
pending audit-log batch at stop time can hang its graceful shutdown this way.

What You Expected?

Graceful shutdown completes within seconds regardless of whether an
audit-table stream load is in flight.

How to Reproduce?

Race-dependent; the deterministic recipe is:

1. Enable the stream load audit recorder and run enough stream loads that the
   recorder buffers a batch.
2. Arrange for _send_stream_load() to be in flight when stop_be.sh --grace
   runs (e.g. pause the FE side of the audit load, or inject a sync point
   between the HTTP-server stop and ExecEnv::destroy()).
3. Observe the BE blocked in ExecEnv::destroy -> std::thread::join for up to
   DEFAULT_STREAM_LOAD_TIMEOUT_SEC (600 s).

Anything Else?

Possible fixes, any one of which suffices (they compose too):

1. Ordering: stop StreamLoadRecorderManager (and other workers that
   depend on live FE/BE services) before tearing down the HTTP/brpc
   servers, instead of inside ExecEnv::destroy() after them.
2. Interruptible send: give the shutdown path a way to abort the in-flight
   HttpClient request (or check _stop and use a short timeout, e.g. a few
   seconds, once shutdown has begun).
3. Bounded join: make stop() cap how long it waits for the worker and
   detach/abandon the final request past the cap, so graceful stop stays
   bounded even if the request hangs.

The 600 s request timeout being exactly equal to the common 10-minute grace
budget of deployment scripts makes this failure mode particularly unlucky:
whenever the race fires, the grace check is guaranteed to fail.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater initial analysis

Disposition: This is a valid BE graceful-shutdown ordering defect on the reported pre-fix master revision. It has already been fixed on master by merged PR #66797 (commit b119273e3f06b3425a09908fc0ac65742e6a1b96), so the issue is correctly closed.

Verified facts

• Before the fix, doris_main() stopped and joined the HTTP service before calling ExecEnv::destroy().
• ExecEnv::destroy() then called SAFE_STOP(_stream_load_recorder_manager), whose stop() set _stop and synchronously joined its raw std::thread.
• The worker checked _stop only between iterations. Once it entered _send_stream_load(), the flag did not interrupt HttpClient::execute_put_request().
• That request targets this BE's own HTTP stream-load endpoint and had a 600-second client timeout. An in-flight request that lost its serving HTTP service could therefore keep the join blocked until curl returned.
• The reported main-thread stack and shutdown log position are consistent with this path. The source mechanism is confirmed; a worker-thread stack inside curl would have made attribution of the individual CI occurrence fully conclusive, but it is not needed for the current disposition because the defect has been fixed and the fix was merged.

Resolution review

The merged change addresses all relevant race windows:

1. It stops StreamLoadRecorderManager before tearing down the BE HTTP service.
2. It prevents the worker from starting another audit stream load after shutdown begins and replaces the fixed one-second sleep with a condition-variable wake-up.
3. It adds an HttpClient abort callback backed by libcurl's transfer-progress callback, allowing an already in-flight request to terminate after _stop is set rather than waiting for the 600-second timeout.

The patch also adds HttpClientTest.abort_in_flight_request, which covers interruption of a request whose peer accepts the connection but never responds. Keeping the normal stream-load timeout at 600 seconds is reasonable because shutdown now has a separate cancellation path.

Remaining gap / information needed only if the problem recurs

• There is no deterministic end-to-end test shown for the complete graceful-shutdown sequence with an audit batch in flight; the added unit test validates the critical curl cancellation primitive.
• For any recurrence on a build containing b119273, collect the exact BE commit/version, the complete shutdown log, an all-thread stack dump (especially the recorder worker), the stream-load audit recorder and batch configuration, and the final curl status/timing. Those data would distinguish this path from another component blocked in ExecEnv::destroy().

Recommended next steps

1. No additional fix is required for master based on the current evidence.
2. Decide whether supported release branches contain the affected recorder implementation; if so, backport [fix](load) Keep graceful BE stop bounded when an audit stream load is in flight #66797 using each branch's actual code history rather than assuming branch/tag naming.
3. If the shutdown test infrastructure permits deterministic synchronization, add an integration test that stops the BE while the audit request is deliberately held in flight and asserts that graceful shutdown remains bounded.

Breakwater-GitHub-Analysis-Slot: slot_c3d2749c4d05