| | --- |
| | |
| |
|
| | license: mit |
| | library_name: tokenizers |
| | tags: |
| | - tokenizer |
| | - bpe |
| | - byte-level |
| | - multilingual |
| | - code |
| | - unitronx |
| | --- |
| | |
| | # {DISPLAY_NAME} |
| | |
| | **UnitronX** is a 32k byte-level BPE tokenizer optimized for English, multilingual (ru/ar/de/es/fr/it/cs/hr/sr), and code. |
| | It enforces safe merge boundaries (script changes, ZWJ, letter↔digit), preserves code identifiers, and uses |
| | placeholder tokens for URLs/emails/paths/hashes/UUIDs/handles/hashtags. |
| | |
| | ## Files |
| | - `tokenizer.json`, `merges.txt`, `vocab.json` |
| | - `tokenizer_config.json`, `special_tokens_map.json` |
| | - `meta.json` |
| | - *(optional)* `unitronx.tiktoken.json` (tiktoken-compatible) |
| |
|
| | ## Load with Transformers |
| |
|
| | ```python |
| | from transformers import AutoTokenizer |
| | tok = AutoTokenizer.from_pretrained("UnitronX-Tokenizer-32k-v1") |
| | print(tok.encode("don't split hyphen-words or fooBar123_id in code!")) |
| | |
