join: consider locale collation in field comparison

[WaterWhisperer]

Fixes: #9971

GNU join uses LC_COLLATE for field comparison. This PR (ref expr) implements locale-aware string comparison using uucore's i18n::collator module.

Reproduce:

coreutils$ export LC_ALL=en_US.UTF-8
coreutils$ cat > f1 <<'EOF'
ab:d  1
abc:d 2
EOF
coreutils$ cat > f2 <<'EOF'
ab:d  x
abc:d y
EOF
coreutils$ sort -k1,1 f1 > f1.sorted
coreutils$ sort -k1,1 f2 > f2.sorted
coreutils$ /usr/bin/join --check-order f1.sorted f2.sorted
abc:d 2 y
ab:d 1 x
coreutils$ ./target/release/join --check-order f1.sorted f2.sorted 
abc:d 2 y
ab:d 1 x

[sylvestre]

did you run some benchmarks ?

[WaterWhisperer]

did you run some benchmarks ?

Thanks for reminding me. Here are the results of benchmark test.


It seems there's a significant performance drop :( So I'm trying to find ways to improve it.

[WaterWhisperer]

I have a few optimization ideas:

• Cache the locale check at startup: Currently checking locale on every comparison. We could check once and store a flag in the Input struct.
• Fast path for C locale: Detect C/POSIX locale early and use direct byte comparison.

Initial testing shows this could bring the C locale overhead from ~16% down to ~1%.
Is this performance trade-off acceptable for the project? Would you prefer a different approach?

Happy to iterate on this based on your guidance! @sylvestre

[sylvestre]

Why not both :)

[WaterWhisperer]

Why not both :)

Yeah, that's exactly what I did.

[sylvestre]

I don't see the change :)

Note that we will need a benchmark for join
In a separatepr if you are interested in doing it

[WaterWhisperer]

I don't see the change :)

Sorry, I just pushed.

Note that we will need a benchmark for join
In a separatepr if you are interested in doing it

Sure!