Turbopack: bincode: Implement bincode Encode/Decode traits on all turbo task values

[bgw]

It looks like bincode should reduce the filesystem cache size by ~10% versus pot, because its not a self-describing format. This attempts to add the bincode::Encode/bincode::Decode traits everywhere, so that we can use those instead of serde.

Why not use bincode's serde compatibility feature?

• Unfortunately, serde has some bugs/incompatibilities/footguns with non-self-describing formats. bincode's traits avoid this by not exposing features that would be incompatible with a self-describing format. https://docs.rs/bincode/latest/bincode/serde/index.html#known-issues
• bincode::Encode/Decode are much simpler traits than serde's equivalents, so we'll probably get some rustc build performance and turbopack binary size benefits from using the bincode versions. In cases where we do have to implement these traits by hand, it's much easier than implementing the serde equivalents.
• I'd like to kill erased-serde anyways because we're probably paying some performance cost from the large amounts of dynamic dispatch it performs, and we're not getting any benefits from it if we only ever serialize to a single format.

Forks of Upstream Crates

• I've forked bincode here: https://github.com/bgw/bincode/commits/bgw/patches/
  • I submitted a patch via email to the maintainer via sourcehut
    • Upstream: https://git.sr.ht/~stygianentity/bincode
    • Fork: https://git.sr.ht/~bgw/bincode
  • This adds a few extra derive features and fixes type bounds on PhantomData.
  • The type bounds on PhantomData aren't very important, and we could work around it in our code.
  • We could just fork the derive crate if upstream doesn't want to accept the patches.
• And virtue here: bgw/virtue@e386f35
  • I've submitted a PR to virtue here: Fix parsing of defaults in generics bincode-org/virtue#94
  • This fixes a bug in the parsing of const enums and generics with defaults.
  • We don't hit this very often as most of our types are not generics, so we could work around it in our code if upstream doesn't want to accept the patch.

Other Notes

• Depends on a patched version of the bincode and virtue crates to fix a few bugs and add a few new features. I'll work on upstreaming these changes.
• This implements BorrowDecode everywhere because the bincode::Decode derive macro requires it. I plan to add an attribute in my bincode fork to allow disabling this, and then we can get rid of all of the BorrowDecode impls. We don't want/need BorrowDecode for turbo task values because everything needs to be owned anyways.

[bgw]

• Turbopack: bincode: Remove automatic serde derives from #[turbo_tasks::value] macro #86634
• Turbopack: bincode: Migrate TaskInput serialization to bincode #86631
• Turbopack: bincode: Use bincode to store the contents of value cells #86338
• Turbopack: bincode: Add bincode encode/decode wrappers for serde_json::Value when stored in a cell #86749
• Turbopack: bincode: Implement bincode Encode/Decode traits on all turbo task values #85580 👈 (View in Graphite)
• Turbopack: bincode: Add traits for types that require TurboBincodeEncoder or TurboBincodeDecoder #86633
• Turbopack: bincode: Add custom bincode reader/writer implementations #86632
• Turbopack: bincode: Add crate with bincode serialization helpers #85878
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

[bgw]

Merge activity

• Dec 5, 1:47 AM UTC: @bgw merged this pull request with Graphite.