Occasionally unable to download geopackage after synchronization with work-packages.

[ronnie-ma]

I am using media-sync, work-packages and db-sync (in this order) together with CE version of server (with storage mounted on S3). The data I'm working with is created with a custom Python script using python-api-client. I understand this may involve several repositories, so please let me know if I should take this issue to another repository's issues. *Note: I did minor modification to the media-sync and work-packages script and recreated the images to fit them to my workflow. The changes were mostly to take in the config parameters in a custom way and to loop through work-packages projects --> not the specific logic in the scripts.

Occasionally, when it gets to the work-packages or db-sync step, the project is stuck. This means that a new version is created and visible in the CE server web, however, the downloading of the geopackages is unavailable. In the following case, the "stuck" version is v9, while v8 client.gpkg can be downloaded. This causes my work-packages and db-sync scripts to be stuck.

I would appreciate it, if you may provide:
1 - A suggested workaround or quick fix such as rollback to the previous version.
2 - Whether there is a similar previous issue that I may have overlooked.

Below are details about my project and specific errors. Let me know if you may need more information:

The project structure is as follows, the older .json, .yml and qgz files can be downloaded, while recently updated .gpkg and work-packages content updated 52 minutes ago in the screenshot cannot:

Error when downloading .gpkg from web, work-packages or QGIS:

{ "detail": "The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "status": 404, "title": "Not Found", "type": "about:blank" }

Log on Server:

[WARNING] [7] [ACCESS] 50.68.26.72 GET /v1/project/version/337a0505-d340-4391-a752-7cc20c72a053/v9 HTTP/1.0 200 212372 http://<URL>/projects/mergin/<PROJECT_NAME>/history?version_id=v9 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36" 238605 <7> [ERROR] [8] Missing file mergin/<PROJECT_NAME>/v9/client_data.gpkg [ACCESS] 50.68.26.72 GET /v1/project/raw/mergin/<PROJECT_NAME> file=client_data.gpkg&random=0.27613017972683396 HTTP/1.0 404 205 http://<URL>/projects/mergin/<PROJECT_NAME>/tree?file_path=client_data.gpkg "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36" 11243 <8> [ERROR] [7] Missing file mergin/<PROJECT_NAME>/v9/client_data.gpkg [ACCESS] 50.68.26.72 GET /v1/project/raw/mergin/<PROJECT_NAME> file=client_data.gpkg&version=v9 HTTP/1.0 404 205 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36" 26033 <7>

Upon further investigation, I've noticed that the changesets for v8 (v9 is the latest version) seems to also be corrupt (see below image)?

I then went into our S3 folders and found this difference in files:

v7

v8 - Previous data sync trial (3 hours ago) diff-summary is missing

v9 - Most recent data sync trial (1 hour ago) client_data.gpkg is missing