Token workshop

Train a tiny character-pair tokeniser. Predict the next merge, then inspect the result.

How this experiment works

Count adjacent token pairs, merge the most frequent repeated pair, and repeat. Ties use a stable ordering. The initial tokens are Unicode code points.

What it does not prove

This small BPE teaching model is not the tokeniser used by a commercial model. Its vocabulary and merge boundaries depend on this exact text.

Try this challenge

Repeat one word more often. How does that change which pair gets merged first?

Related Trust Agent courses · Sources and licences

Enable JavaScript to run the experiment on your device.