numfmt: don't panic on a multi-byte suffix under a multi-byte-separat… - #14065
numfmt: don't panic on a multi-byte suffix under a multi-byte-separat…#14065nagendramohan wants to merge 3 commits into
Conversation
…or locale
Under a locale whose decimal separator is multi-byte (e.g. ar_SA.UTF-8,
where it is the Arabic '٫', U+066B, 2 bytes), an input like '1٫€K'
aborted with a char-boundary panic in find_valid_number_with_suffix.
The numeric part was iterated with s.chars().skip(numeric_part.len()) —
using the *byte* length as a *character* count — which skipped past the
multi-byte separator incorrectly, and the result was then sliced with
&s[..=numeric_part.len()], landing inside a multi-byte suffix character.
Iterate the remainder from the numeric part's byte offset (a valid char
boundary, since it is a prefix of the input) and slice using each suffix
character's actual UTF-8 length. Malformed input is now rejected as GNU
does ('invalid suffix in input'), and valid multi-byte-separator input
such as '1٫5K' still parses.
Adds a regression test (verified failing before the fix). Closes uutils#13937.
Signed-off-by: Nagendra Mohan <nagendramohan1990@gmail.com>
|
Duplicated PR. Please close this. |
|
GNU testsuite comparison: |
Merging this PR will degrade performance by 3.84%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | mv_directory |
7.3 ms | 8.2 ms | -11.06% |
| ⚡ | Simulation | du_deep_tree[(100, 3)] |
2.1 ms | 2.1 ms | +3.96% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing nagendramohan:fix/numfmt-multibyte-suffix-panic-13937 (24161d4) with main (d9c5343)2
Footnotes
-
291 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
main(728388e) during the generation of this report, so d9c5343 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
4 participants
…or locale
Under a locale whose decimal separator is multi-byte (e.g. ar_SA.UTF-8, where it is the Arabic '٫', U+066B, 2 bytes), an input like '1٫€K' aborted with a char-boundary panic in find_valid_number_with_suffix.
The numeric part was iterated with s.chars().skip(numeric_part.len()) — using the byte length as a character count — which skipped past the multi-byte separator incorrectly, and the result was then sliced with &s[..=numeric_part.len()], landing inside a multi-byte suffix character.
Iterate the remainder from the numeric part's byte offset (a valid char boundary, since it is a prefix of the input) and slice using each suffix character's actual UTF-8 length. Malformed input is now rejected as GNU does ('invalid suffix in input'), and valid multi-byte-separator input such as '1٫5K' still parses.
Adds a regression test (verified failing before the fix). Closes #13937.
Closes #13937.
Under a locale whose decimal separator is multi-byte (e.g. ar_SA.UTF-8, where it's the Arabic ٫, U+066B, 2 bytes), an input like 1٫€K aborted with a char-boundary panic:
Root cause: the numeric part was iterated with s.chars().skip(numeric_part.len()) — using its byte length as a character count — so a multi-byte separator was mis-skipped, and the result was then sliced with &s[..=numeric_part.len()], landing inside a multi-byte suffix char.
Fix: iterate the remainder from the numeric part's byte offset (a valid char boundary) and slice using each suffix char's actual len_utf8(). Malformed input is now rejected like GNU (invalid suffix in input), and valid multi-byte-separator input like 1٫5K still parses.
Adds a regression test (verified failing before the fix). cargo fmt, cargo clippy, and the full numfmt test suite pass.