Postgres replication issue on Fly.io since latest release (3.13.4)

[capoccias]

Elixir version

1.19.2

Database and Version

PostgreSQL 17.2

Ecto Versions

ecto 3.13.5

Database Adapter and Versions (postgrex, myxql, etc)

postgrex 0.21.1

Current behavior

1. Create an Elixir app and Postgres app on Fly.io with the fly_postgres dependency
2. Scale to multiple regions
3. Bump to ecto_sql 3.13.4

Saw a bunch of errors immediately following a deployment that bumped ecto_sql from 3.13.3 to 3.13.4
These errors went away after downgrading back to 3.13.3
Note: our primary region is iad

This can be seen in the following screenshot from Sentry:

01:31 ACDT: Deployed ecto_sql 3.13.4 (shown by the little purple dot under the bar graph)
01:48 ACDT: The first error
15:28 ACDT: Deployed ecto_sql 3.13.3
15:29 ACDT: The latest error
16:37 ACDT: Took this screenshot

Stack trace in Sentry:

ArgumentError: no node found running in region "iad"
    lib/fly_rpc.ex:165: Elixir.Fly.RPC.Fly.RPC.rpc_region/3
    lib/task/supervised.ex:105: Elixir.Task.Supervised.Task.Supervised.invoke_mfa/2

Expected behavior

I'd have hoped everything would keep working as it did prior to the upgrade

From what I can tell, this is the only code change in 3.13.4: 1883163

[greg-rychlewski]

By any chance did any other dependency get upgraded at the same time or it was completely isolated to Ecto SQL?

[josevalim]

Yeah, from the data so far, this seems to be unlikely to be an ecto_sql issue :)

[capoccias]

Hey, thanks for the replies!

I didn't include it as it seemed irrelevant but I bumped swoosh from 1.19.9 to 1.20.0 at the same time which did have a few changes (https://github.com/swoosh/swoosh/commits/v1.20.0/):

• Bumped bandit from 1.8.0 to 1.9.0 but I was already on 1.9.0
• Bumped ex_doc which surely this isn't the cause 🤷
• Bumped ex_aws but I don't use aws
• Added Resend adapter but I don't use resend

Also, as mentioned what has "fixed" it is reverting back to ecto_sql 3.13.3

I say "fixed" as it has happened 4 times in the last 5 hours:

However this is much less frequent that the 5 hours prior to that:

Sorry for the noisy graph (and Activate Windows) but this is the Grafana dashboard showing that there were definitely machines running in iad (our primary region) for the last 10 hours - only dropping at around 15:30 ACDT when I deployed the change to revert ecto_sql

This could indicate that it was the redeploy that fixed the issue but I had actually restarted the running machines via the Fly UI (see below) a bit prior to the deployment which didn't help.

A deploy and restart are different though (as deploying spins up new machines) so perhaps I'll deploy ecto_sql 3.13.4 again and see if the issue comes back - if not (🤞) I guess it could just be that the machines were in a funky state.
I'll try this soon but please let me know if you can think of a better way to test the theory :)

Have a good one!

[greg-rychlewski]

I haven’t used Fly before but one thing I am thinking is if the application is not starting at all then it might crash and the logs do not go to the place where your log collector is reading from before it sends them to grafana or whichever other service you are using to read your logs.

In that case what you can do is go into the machine that was supposed to have the app deployed and look for the logs manually. For example if you are using docker then using the docker logs command on the failed container. Whether or not this makes sense for fly though I’m not sure.

Right now it seems like the logs you are seeing are from the perspective of a machine that is working fine trying to talk to a machine that is supposed to be fine but is not. So I would try to see what happened on the machine that is not fine.

[greg-rychlewski]

Another thing I can think of is if you have a race condition. So your nodes are being redeployed and something is trying to talk to an iad node but the iad nodes are being redeployed so not currently up. Then one of these things is happening

• it tries once and gives up
• it tries again and eventually succeeds once the iad nodes are up. So your logs are not indicating a permanent problem
• it tries again but it is using a stale list of nodes and doesn’t ever “see” the iad nodes are up so it keeps failing

If you have any other metrics to indicate whether your app is working fine or not it might be able to help you determine if any of these are happening

[capoccias]

Closing this as it seems to be fine since I deployed

Thanks for the responses, if I see this happen again I'll look into some of your suggestions, @greg-rychlewski

Happy new year and don't forget your crowns, kings 👑 👑