This is a follow-up to the discussion in #9935.
As discussed there, Borg 2 currently uses sha256(content) as the storage key (object name) for packs and indexes. The rationale makes sense: simpler code, a stable algorithm, broad ecosystem support, verification without Borg (just check the sha256sum against the file name), and content-addressable storage.
However, while SHA hardware acceleration is widely available today, there is still a significant amount of hardware in active use that does not support SHA extensions (see docs; most notably, Intel only introduced SHA extensions in 2021). On such systems, sha256 performance is expected to be significantly lower, which is why Borg 1.1 added support for blake2b as ID hash, and Borg 2 recently switched that to blake3.
I wonder whether the mandatory use of sha256 for pack and index IDs has a measurable performance impact on systems without SHA hardware acceleration, and whether that impact could justify letting repositories created with borg2 repo-create --id-hash blake3 use blake3 for pack and index IDs instead. As discussed in #9935, this should be possible in principle, with the main downside being the additional code maintenance burden. For example, verification without Borg would still be possible using the official b3sum tool.
As always, this depends on whether sha256 is actually the limiting factor in a given workload. As Thomas pointed out in #9935, it is unlikely to matter when writing to slow remote storage such as cloud providers, where other bottlenecks dominate. However, it could become relevant on fast local storage.
I'm currently migrating a Borg 1 repository on an Intel Core i5-8265U (8th generation, no SHA extensions), where borg transfer --from-borg1 appears noticeably slower than expected. I haven't yet isolated whether sha256 contributes significantly to that slowdown, but I plan to run some benchmarks when I have the time and will post the results here if they provide useful data.
This is a follow-up to the discussion in #9935.
As discussed there, Borg 2 currently uses
sha256(content)as the storage key (object name) for packs and indexes. The rationale makes sense: simpler code, a stable algorithm, broad ecosystem support, verification without Borg (just check thesha256sumagainst the file name), and content-addressable storage.However, while SHA hardware acceleration is widely available today, there is still a significant amount of hardware in active use that does not support SHA extensions (see docs; most notably, Intel only introduced SHA extensions in 2021). On such systems,
sha256performance is expected to be significantly lower, which is why Borg 1.1 added support forblake2bas ID hash, and Borg 2 recently switched that toblake3.I wonder whether the mandatory use of
sha256for pack and index IDs has a measurable performance impact on systems without SHA hardware acceleration, and whether that impact could justify letting repositories created withborg2 repo-create --id-hash blake3useblake3for pack and index IDs instead. As discussed in #9935, this should be possible in principle, with the main downside being the additional code maintenance burden. For example, verification without Borg would still be possible using the officialb3sumtool.As always, this depends on whether
sha256is actually the limiting factor in a given workload. As Thomas pointed out in #9935, it is unlikely to matter when writing to slow remote storage such as cloud providers, where other bottlenecks dominate. However, it could become relevant on fast local storage.I'm currently migrating a Borg 1 repository on an Intel Core i5-8265U (8th generation, no SHA extensions), where
borg transfer --from-borg1appears noticeably slower than expected. I haven't yet isolated whethersha256contributes significantly to that slowdown, but I plan to run some benchmarks when I have the time and will post the results here if they provide useful data.