Train a tiny language model

Start with random weights. Train a real neural character model, then see what it invents.

How this experiment works

A one-hot context feeds a tanh hidden layer and a softmax next-character predictor. Each step updates weights with stochastic gradient descent. The last 15% of the text is held out.

What it does not prove

A tiny character model learns local patterns. It is not a conversational assistant. Loss on this small held-out sample is not a general intelligence benchmark.

Try this challenge

Compare three-character and five-character contexts. Which produces better names at the same number of training steps?

Related Trust Agent courses · Sources and licences

Enable JavaScript to run the experiment on your device.