Skip to content

feat(emulator): adds resumable upload API support for multiple chunk and cancelling#10325

Open
jakeisonline wants to merge 6 commits intofirebase:mainfrom
jakeisonline:emulated-resumable-uploads
Open

feat(emulator): adds resumable upload API support for multiple chunk and cancelling#10325
jakeisonline wants to merge 6 commits intofirebase:mainfrom
jakeisonline:emulated-resumable-uploads

Conversation

@jakeisonline
Copy link
Copy Markdown

@jakeisonline jakeisonline commented Apr 12, 2026

Description

This is a follow up to #9047 which was unfortunately reverted in #9107 due to issues in failing integration tests. While it wasn't revealed what the specific issue were, I'm hoping this new approach might avoid those same issues. Given the complexity of resumable uploads, I figured this was an important piece of coverage for the emulator to have.

This PR pays particular attention to the validation of the Content-Range header sent by the client, in addition to ensuring the status codes and response bodies (and therefore the underlying logic) coming from the emulator match live GCS as much as possible.

Approach

  • Matching documentation per https://docs.cloud.google.com/storage/docs/performing-resumable-uploads
  • Created a set of tests to ensure the correct responses of the API (not included in this PR, as I was unsure what best practises for testing endpoints in this repo are. Happy to include at request, though)
  • Created a thin node app which helped me manually test and compare responses in the emulator vs live GCS, including testing edge cases (e.g. malformed Content-Range, non-existent upload IDs, etc).
  • Using my lived experience working with GCS resumable uploads in a production environment with real end-users

Scenarios Tested

  • PUT
    • single chunk uploads
      • returns 200 + metadata when complete
    • multiple chunk uploads
      • returns 308 + range header when uploading a partial chunk
      • returns 200 + metadata when complete
      • returns 200 when a single chunk uses unknown total (*)
      • returns 308 then 200 when unknown total and the last chunk is not a full 256 KiB multiple
      • returns 308 (not 200) when unknown total and chunk is an exact 256 KiB multiple
    • status checking
      • returns 308 with no Range header when active and no bytes persisted yet
      • returns 308 with Range when active and some bytes are already persisted
      • returns 200 with object metadata when the resumable session is already finished
      • returns * in the Range suffix when bytes / is used
      • finalises when bytes */total exactly matches bytes received
      • finalises an exact 256 KiB upload via a zero-content bytes */total request
    • error handling
      • returns 400 when content-range is invalid
      • returns 400 when chunk position is invalid
      • returns 400 when chunk size is invalid
      • returns 404 when the upload is not found
      • returns 499 when the upload is cancelled
  • DELETE
    • cancels an upload and returns 499
    • returns 404 when upload_id is not found
    • returns 405 when upload_id is missing

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the storage emulator's resumable upload API by adding support for multiple chunk uploads and upload cancellation. The changes include a more robust handling of the Content-Range header, status check requests, and the implementation of a DELETE endpoint for cancelling active uploads. Feedback focuses on optimizing redundant header parsing, correcting an offset calculation in an error message, and expanding the Content-Range regex to support all valid Cloud Storage formats.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants