[MongoDB Replication] Raw Change Streams

[rkistner]

Supersedes #309.

This switches to using aggregate/getMore commands directly, instead of the built-in watch methods.

Motivations:

1. This gives us much more control over the entire process, instead of fighting the mongodb driver.
2. By getting data back in batches, we can avoid the overhead from repeatedly calling tryNext() for every individual change (impact not measured yet).
3. By parsing documents as we're processing them (instead of a batch at a time), we can slightly reduce memory usage.

This removes hacks previously using private APIs from:

• [MongoDB] Track bytes, transactions and chunks replicated #575
• [MongoDB] Fix write checkpoint throughput #573

This is also the first step in writing a more efficient conversion for raw BSON -> serialized JSON. We can't do that with the driver's change stream implementation, since that doesn't support getting results as raw BSON.

TODO:

• Handle automatic retries according to the specification.
• Confirm that we're following the remainder of the specification exactly.
• Make sure we have proper unit tests, especially for cluster-level streams.

[changeset-bot]

🦋 Changeset detected

Latest commit: d04024f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-mongodb	Minor
@powersync/service-schema	Patch
@powersync/service-image	Patch
@powersync/service-core	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR