Token microscope

Inspect WordPiece subwords, IDs, special tokens and a chosen context budget.

How this experiment works

The actual MiniLM tokenizer normalises text and splits words into vocabulary pieces. No inference weights are required.

What it does not prove

These are MiniLM’s uncased WordPiece tokens. Other models have different vocabularies and context limits.

Try this challenge

Compare punctuation, an unfamiliar name and non-English text. Which consume the most tokens?

Related Trust Agent courses · Sources and licences

Enable JavaScript to run the experiment on your device.