Skip to content

Fix tokenize_and_concatenate splitting tokens across chunk boundaries#1201

Open
brainsnog wants to merge 2 commits intoTransformerLensOrg:mainfrom
brainsnog:fix/tokenize-and-concatenate-word-boundaries
Open

Fix tokenize_and_concatenate splitting tokens across chunk boundaries#1201
brainsnog wants to merge 2 commits intoTransformerLensOrg:mainfrom
brainsnog:fix/tokenize-and-concatenate-word-boundaries

Commits

Commits on Mar 12, 2026

Commits on Mar 14, 2026