Skip to content

feat(sync-service): Add exclusive mode to support AWS EFS#3859

Open
magnetised wants to merge 1 commit intomainfrom
magnetised/shape-db-exclusive-mode
Open

feat(sync-service): Add exclusive mode to support AWS EFS#3859
magnetised wants to merge 1 commit intomainfrom
magnetised/shape-db-exclusive-mode

Conversation

@magnetised
Copy link
Contributor

NFS filesystems, such as AWS EFS, do not support SQLite WAL mode or simultaneous access by multiple processes, due to the limitiations of NFS locking semantics.

So to support AWS EFS this pr adds a new "exclusive mode" to the shape db configuration, enabled by setting ELECTRIC_SHAPE_DB_EXCLUSIVE_MODE=true, which multiplexes all reads and writes through a single SQLite connection and configures the database with journal_mode=DELETE as recommended by the SQLite docs.

There is also a new ELECTRIC_SHAPE_DB_STORAGE_DIR configuration option which allows for hosting the shape db file in a different location from the shape data, e.g. on a non-networked filesystem.

These two options open up the possibility of an in-memory shape database, so enabling exclusive mode and setting ELECTRIC_SHAPE_DB_STORAGE_DIR=:memory: will do exactly that.

Since I was wiring up shapedb configuration, I've also added env-vars and configuration to set page cache size and durabilty (PRAGMA cache_size=... and PRAGMA synchronous=...).

Follow up work:

  • Start some shape cleanup task to remove all orphaned shape data when the shape db is empty, to fix the situation when an ephemeral shape db is used alongside persistent shape data.

@codecov
Copy link

codecov bot commented Feb 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.18%. Comparing base (dfa6301) to head (59ba3da).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #3859       +/-   ##
===========================================
+ Coverage   76.66%   87.18%   +10.52%     
===========================================
  Files          37       25       -12     
  Lines        1611     2325      +714     
  Branches      171      587      +416     
===========================================
+ Hits         1235     2027      +792     
+ Misses        375      296       -79     
- Partials        1        2        +1     
Flag Coverage Δ
elixir ?
elixir-client ?
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 92.03% <ø> (?)
packages/y-electric 56.05% <ø> (ø)
typescript 87.18% <ø> (+11.42%) ⬆️
unit-tests 87.18% <ø> (+10.52%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blacksmith-sh

This comment has been minimized.

@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch 2 times, most recently from 50a1bcd to 986229f Compare February 17, 2026 14:48
@netlify
Copy link

netlify bot commented Feb 17, 2026

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 986229f
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/69947fbf855dfa0008e7400e
😎 Deploy Preview https://deploy-preview-3859--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@blacksmith-sh

This comment has been minimized.

@blacksmith-sh

This comment has been minimized.

@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch from 986229f to 2cdda99 Compare February 17, 2026 15:24
@magnetised magnetised marked this pull request as ready for review February 17, 2026 16:02
@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch from 2cdda99 to c8e3cea Compare February 18, 2026 09:34
@blacksmith-sh

This comment has been minimized.

@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch from c8e3cea to 75f6997 Compare February 18, 2026 10:48
@blacksmith-sh

This comment has been minimized.

@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch 2 times, most recently from d8d8e2f to 12495f2 Compare February 18, 2026 14:27
this is done by registering the write pool twice when in exclusive_mode,
so the runtime cost is equivalent to the version without the mode tweak.

while we're there allow for configuring other bits of the sqlite
behaviour via config and env-vars
@magnetised magnetised force-pushed the magnetised/shape-db-exclusive-mode branch from 12495f2 to 59ba3da Compare February 18, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments