feat(world-postgres-redis): add PostgreSQL + Redis world implementation#155
feat(world-postgres-redis): add PostgreSQL + Redis world implementation#155kschmelter13 wants to merge 0 commit intovercel:mainfrom
Conversation
|
|
@kschmelter13 is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
|
@kschmelter13 Thanks for contributing! We love supporting community-created worlds as much as we can, while also making sure the community isn't bottlenecked by us reviewing/maintaining all the worlds. Since a world is just an npm package, do you want to take a stab at publishing your world as an npm package yourself? If you provide us with a link, we'll add your package to the list of community worlds on useworkflow.dev If you think this should be part of the main workflow repository instead, please let us know your reasoning. Thanks! |
|
@VaguelySerious This should be an official World. The Whereas Redis is a battletested for realtime/near realtime streaming. I understand the ask for reasoning, but this one is so obviously needed. |
|
I'm happy to replace To do this, we'd need to ensure this world passes all relevant e2e tests and can replace the Postgres world. How about this:
|
|
Also note that if the intention is to make this a first-party package, it's in competition with #467, which does not depend on Redis and similarly solves the performance issues of pg-boss |
|
@VaguelySerious We're working off of our own fork of the postgres world where we implemented Graphile Worker. The performance is excellent, so FWIW, I fully support that. Redis performance will likely be better, but it's not suitable for storing workflow state for historical purposes. We'll likely end up with a fork that uses Redis for queues/workers, and Postgres for state. Happy to try and contribute a PR, but I'm seeing you already have a team member who's working on it. |
|
Great - if Graphile workers work well, then it'll be nice to stick to a pg-only solution for simplicity, with an optional world-redis that can be added for streaming. The redis-world won't be first-party (in this repo), but we do plan on creating a list of community packages and a way to benchmark and e2e test them, to broadcast community efforts. @arielweinberger Also, we don't have anyone working on the Graphile implementation right now - #467 is a community PR. If you want to give a simple migration of the postgres world to Graphile a shot, that'd be awesome |
|
@arielweinberger How stable is your Graphile implementation? Do you think it's production-ready? For a chatbot, having pg-boss delay each step (and there are a few of them) adds quite some latency. From what I understood in this and other discussions, the the team seems to be more than willing to switch the queue implementation in world-postgres, and the fact that Graphile doesn't require additional infrastructure is definitely a plus. |
|
@VaguelySerious If there is still interest in moving this forward I would love to help and talk with anyone needed to get it done! p.s. doesnt have to be redis, I would be happy to work on a production ready Graphile worker implementation. Just anything better than pg-boss lol |
|
@kschmelter13 We'd be happy to accept a PR changing the world-postgres implementation to use Graphile workers (without using Redis). The main criteria are:
Separately, if you do end up publishing your world with Graphile workers as its own package, or Graphile+Redis, you can also add your world to this json file, which adds it to https://useworkflow.dev/worlds |
|
@VaguelySerious What about a migration path for existing users? Must queued items be transferred from the pgboss tables or is that not necessary? |
|
@rovo89 Existing users of It would be nice, but not required, if the PR came with migrations from the old postgres table schema to the new, or if that's too difficult, with a script that pulls the old data and re-ingests it into a new table.AI should be able to do this pretty well given the old/new schema definitions. I'll probably add something like that and double check it against an existing deploy before we release the world-postgres package. |
|
@VaguelySerious sounds good ill work on the Graphile implementation. might need authorization for deployment |
|
@kschmelter13 Awesome! Deployment authorization is for |
|
@kschmelter13 Thanks for working on this! Latency improvements are very welcome, especially for chatbots. I asked @arielweinberger about his integration of Graphile Worker, which is at https://github.com/appwrite-labs/vercel-workflow-fork/pull/1/changes. Wasn't used in production, but stress tests were promising. They have moved on since, so don't expect a PR from his side, but maybe you can use it as inspiration. He is fine with that. |
|
@VaguelySerious @rovo89 I opened up a PR for the replacement of pg-boss with graphile worker. I am going to continue with the postgres redis world as a community npm package and create a PR to add it to the worlds site. Thanks for your help! #1124 |
4 participants
Implement workflow world using PostgreSQL for durable storage and Redis for queue processing and streaming. Uses Redis Lists with Pub/Sub for job queues and Redis Streams for real-time data streaming, providing 3-6x faster performance than pg-boss while maintaining the same guarantees. #106