Skip to content

feat(event): run native queue and event plugin simultaneously#6852

Open
h45hc47 wants to merge 1 commit into
tronprotocol:developfrom
h45hc47:feat(event)-run-native-queue-and-event-plugin-simultaneously
Open

feat(event): run native queue and event plugin simultaneously#6852
h45hc47 wants to merge 1 commit into
tronprotocol:developfrom
h45hc47:feat(event)-run-native-queue-and-event-plugin-simultaneously

Conversation

@h45hc47

@h45hc47 h45hc47 commented Jun 22, 2026

Copy link
Copy Markdown

What does this PR do?

Makes the native ZMQ message queue and the event plugin (e.g. the MongoDB plugin) two independent event sinks, so a node can publish triggers to both at the same time.

Previously they were mutually exclusive: when useNativeQueue = true, the plugin's path/server/dbconfig were never loaded and only the native queue started; the plugin only ran when useNativeQueue = false.

Key changes:

  • Args: always copy path/server/dbconfig into EventPluginConfig, regardless of useNativeQueue.
  • EventPluginLoader: add a separate useEventPlugin flag (derived from a non-empty plugin path). start() launches the plugin first — so the plugin's listeners exist before any topic is registered — and then the native queue. Each post*Trigger is dispatched to every active sink; setPluginTopic is null-guarded for the now-possible double trigger-config pass; isBusy() is gated on the plugin being active instead of on the native queue being off.
  • HistoryEventService: honor the plugin's back-pressure (isBusy) during historical sync even when the native queue is enabled, so the plugin queue is not flooded. Native-only pacing is unchanged.
  • config.conf / reference.conf: document that useNativeQueue and a plugin path can be combined.

Why are these changes required?

Operators often want both transports at once: low-latency ZMQ delivery for real-time consumers AND durable persistence in MongoDB for querying and indexing. Today they must pick one. With a single event.subscribe block (native queue on + a plugin path set), both now receive every trigger.

Example configuration enabling both:

event.subscribe = {
  enable = true
  native   = { useNativeQueue = true, bindport = 5555, sendqueuelength = 1000 }
  path     = "/home/trx/event-plugin/plugin-mongodb-1.0.0.zip"
  server   = "127.0.0.1:27017"
  dbconfig = "db|log|pass"
  ...
}

This PR has been tested by:

  • Unit Tests
    • Built on JDK 8 (jdk1.8.0_271) (from GreatVoyage-v4.8.1.1 tag): :common and :framework compileJava succeed.
    • EventPluginLoaderTest#testIsBusy — updated for the new useEventPlugin semantics — PASSED.
    • HistoryEventServiceTest#test — covers the reordered back-pressure path — PASSED.
  • Manual Testing
    • Verified in a Docker deployment: a mainnet full node running with useNativeQueue = true and a MongoDB plugin path set starts cleanly, writes events to the MongoDB collections, and binds the native ZMQ queue on port 5555 simultaneously.

Follow up

A future enhancement could add an explicit useEventPlugin config key instead of inferring plugin activation from a non-empty path.

Extra details

In combined mode the trigger-config pass runs once per launcher; this is idempotent (boolean flags and setTopic overwrite with the same values). Every trigger is now serialized and dispatched to both sinks, a modest overhead versus single-sink mode. Behavior is unchanged for the two existing single-sink configurations (native-only, or plugin-only).

@github-actions github-actions Bot requested review from 0xbigapple and xxo1shine June 22, 2026 13:21
@h45hc47 h45hc47 closed this Jun 22, 2026
@h45hc47 h45hc47 reopened this Jun 22, 2026
this.triggerConfigList = config.getTriggerConfigList();

useNativeQueue = config.isUseNativeQueue();
useEventPlugin = StringUtils.hasText(config.getPluginPath());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SHOULD] @h45hc47, switching the "plugin enabled" condition from !useNativeQueue to "path is non-empty" introduces an upgrade regression: a node currently running with useNativeQueue = true that happens to still have a non-empty path in its config (the old docs said path is ignored when useNativeQueue = true, so such leftover config does exist) boots fine today as native-only; after the upgrade it will try to load that plugin because useEventPlugin = true, and if the plugin zip is missing/invalid, launchEventPlugin returns false -> start() returns false -> Manager.startEventSubscribing() throws TronError(EVENT_SUBSCRIBE_INIT), so the node fails to start outright.

Overall I'd recommend keeping the current behavior (native queue and plugin mutually exclusive); the existing design is fine as-is:

  • The "deliver to both" need can be solved cleanly outside the node: enable the native ZMQ queue and run a lightweight consumer that subscribes to ZMQ and persists to MongoDB. The node keeps a single sink and minimal responsibility, instead of coupling dual delivery into the core.
  • Mutual exclusivity is a safety property here, not a defect: it keeps plugin/MongoDB failures out of the node, so a failed plugin load or a MongoDB outage cannot affect the block-producing node. This PR removes that isolation (the startup failure above is the direct consequence) and also couples the plugin queue's back-pressure into the block-loading path (HistoryEventService / BlockEventLoad start throttling on the plugin queue even when the native queue is on), adding failure surface and coupling to a subsystem that is sensitive to block processing.
  • The trade-off doesn't pay off: the benefit is a niche scenario that already has a clean external alternative, while the cost is added coupling in a core subsystem, a new failure path that can prevent the node from starting, and the per-trigger overhead of an extra serialization plus dual dispatch on a hot path.

So I'd suggest not merging this for now and keeping the current mutually-exclusive semantics. If there's a strong need later, it should be redone as an explicit opt-in flag that guarantees zero behavior change for existing configs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants