Enable the bsddb deadlock detector

[otargowski]

bsddb seems to use page-level locks, which necessitate running a deadlock detector, as two parallel transactions making e.g. two writes each can deadlock regardless of whether any two of those four writes access the same key, since they may be on the same page.
From the documentation:

"[deadlock detection] is necessary for almost all applications in
which more than a single thread of control will be accessing the
database at one time. Even when Berkeley DB automatically handles
database locking, it is normally possible for deadlock to occur."

Almost all of my encounters with this issue were on fresh (a few days old) SIO2 instances.
My guess as to why it didn't (to my best knowledge) happen on SZKOpuł is that when the migration from filetracker 1 was done, the database was already quite big, so the chances of two keys being on the same page were quite low, especially since usually not many PUTs are done in parallel.

I added a test demonstrating the issue. On my machine, it reproduced the deadlocks quite reliably.

The fix involves running a deadlock detector pass every time a lock isn't immediately available, which perhaps kills one of the transactions. This required adding retries to transactions.
The performance penalty from this should be negligible, especially since we don't have many conflicts anyway (thanks to using per-link and per-blob file locks).
Documentation of the needed setting: https://bsdwatch.net/docs/sharedocs/db5/api_reference/C/envset_lk_detect.html.

[otargowski]

My guess as to why it didn't (to my best knowledge) happen on SZKOpuł

Nvm, SZKOpuł's filetracker just went down and at the start of the unusual part of the logs is:

2026-02-27 20:47:05 CRITICAL (126976) gunicorn.error  WORKER TIMEOUT (pid:127034)
2026-02-27 20:47:05 CRITICAL (126976) gunicorn.error  WORKER TIMEOUT (pid:127042)
2026-02-27 20:47:06 INFO     (126976) gunicorn.config child_exit() hook: sending SIGTERM to ourselves
2026-02-27 20:47:06 INFO     (126976) gunicorn.config child_exit() hook: sending SIGTERM to ourselves
2026-02-27 20:47:06 INFO     (28777) gunicorn.error  Booting worker with pid: 28777
2026-02-27 20:47:07 INFO     (28778) gunicorn.error  Booting worker with pid: 28778

These WORKER TIMEOUTs look just like a deadlock... and cause those processes to exit ungracefully, which corrupts the bsddb and requires a restart and an absurdly slow recovery.

Start of previous downtime:

2026-01-27 12:55:03 CRITICAL (5152 ) gunicorn.error  WORKER TIMEOUT (pid:5179)
2026-01-27 12:55:03 CRITICAL (5152 ) gunicorn.error  WORKER TIMEOUT (pid:5250)
2026-01-27 12:55:04 CRITICAL (5152 ) gunicorn.error  WORKER TIMEOUT (pid:5180)
2026-01-27 12:55:04 CRITICAL (5152 ) gunicorn.error  WORKER TIMEOUT (pid:5232)
2026-01-27 12:55:04 INFO     (5152 ) gunicorn.config child_exit() hook: sending SIGTERM to ourselves
2026-01-27 12:55:04 INFO     (5152 ) gunicorn.config child_exit() hook: sending SIGTERM to ourselves
2026-01-27 12:55:04 INFO     (126115) gunicorn.error  Booting worker with pid: 126115
2026-01-27 12:55:04 INFO     (126116) gunicorn.error  Booting worker with pid: 126116