in_forward: fix 100% CPU hang and connection teardown use-after-free [Backport to 4.2]#12053
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a499c837f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
A connection destroyed by the downstream timeout/pending-destroy machinery can still have its event linked in the priority bucket queue after mk_event_inject (only FLB_ENGINE_LOOP_MAX_ITER events are processed per loop round). destroy_conn freed the connection unconditionally, leaving a dangling _priority_head in the bucket queue. On the next loop iteration flb_bucket_queue_pop_min dereferenced the freed node (use-after-free), and the stale self-referential linkage made flb_event_load_bucket_queue_event skip the event forever, so a socket in CLOSE_WAIT kept waking epoll while its handler was never dispatched, spinning the input thread at 100% CPU. Mirror the guard already used by upstream destroy_conn: keep the connection in the destroy_queue until its event has drained from the bucket queue, then free it on a later sweep. Fixes #12025 Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
The plugin kept a fw_conn wrapper in ctx->connections for every active connection but never registered a drop notification callback. When the engine destroyed the underlying connection on its own (e.g. an IO timeout via prepare_destroy_conn), the wrapper was left pointing at a freed connection. The next fw_conn_del_all (backpressure pause or shutdown) then dereferenced it, causing a use-after-free crash. Register a drop_notification_callback that detaches and frees the wrapper when the engine reclaims the connection, mirroring the http_server pattern. Split the wrapper cleanup (fw_conn_release) from the downstream release so fw_conn_del and the drop path don't double-release, and guard fw_conn_event against a wrapper already dropped by an injected event that fires after teardown. Fixes #12025 Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
cosmo0920
force-pushed
the
fix-12025-priority-queue-stale-event-4.2
branch
from
a499c83 to
9ca5886
Compare
2 participants
Partially backporting of #12044 due to lack of integration testing mechanism on 4.2 branch.
Plus, also backporting connections and downstream hardening from #11742 due to plugging lack of notification mechanism for a deletion of connections.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.