Exam · Tokenisation: train and inspect a small tokenizer

Take the exam and earn your certificate.

A real, server-scored exam for this course. Score strictly above 80% to pass and get a genuine, verifiable certificate.

Before you start

We ask for your name and email first, the same as a PDF download, so we know you are here and your certificate can carry the right name.

We privately record your name, email and request IP for this exam request. Read the privacy notice and retention policy.

Visit Mickai on LinkedIn

Complete the security check before submitting.

Cloudflare Turnstile helps prevent automated submissions. How this check uses data.

You can retake the exam any time. Each attempt draws a fresh set of questions.

Written practice, ungraded

Write your response in your own notes, then compare it with the sample answer. This practice is separate from the scored exam and does not affect your certificate.

Practice 1

Use three fictional strings: `naïve`, `£10` and `a b `. Predict each string's Unicode-character count, whitespace-split word count and UTF-8 byte count. Then identify which count loses the repeated or trailing spaces when the text is split and joined.

Check your answer against these points:

  • Distinguishes Unicode-character count from UTF-8 byte count.
  • Counts the examples with their spaces as written.
  • Identifies that whitespace splitting can lose exact spacing.
Compare with a sample answer

`naïve`: 5 Unicode characters, 1 word, 6 UTF-8 bytes. `£10`: 3 characters, 1 word, 4 bytes. `a b `: 5 characters, 2 words, 5 bytes. Splitting and joining the last example normalises its spaces, so it does not reproduce the original string.

Ungraded paper practice. Counts are for these supplied strings and UTF-8; the task does not compare commercial tokenisers.

Practice 2

On paper, run three BPE merges for this fictional corpus: `cat` appears 4 times, `cats` 2 times, and `scat` 1 time. Start with single characters, count adjacent pairs weighted by word frequency, and record each winning pair and its count. State one tie rule you would need if two pairs shared the highest count.

Check your answer against these points:

  • Weights each adjacent pair by the number of times its word occurs.
  • Recounts pairs after each merge rather than choosing the longest string.
  • States a deterministic tie rule.
Compare with a sample answer

Initially `c+a` occurs 7 times, `a+t` 7 times, `t+s` 2 times and `s+c` once. With a stated first-counted-pair tie rule, `c+a` wins first. Recount after the merge before choosing the next pair, and repeat for three merges. A different documented tie rule can produce a different ordered merge list.

Ungraded paper practice. The sample illustrates the starting counts and the need to recompute after each merge; learners should show their own ordered recounts.

Practice 3

A teammate runs the same tiny BPE training corpus twice and gets different merge lists. Write a short diagnostic checklist using the source code's behaviour: include pair counting, tie handling and the order in which learned merges are replayed. Suggest one controlled check before changing the implementation.

Check your answer against these points:

  • Checks corpus and chunk inputs before attributing a result to the algorithm.
  • Mentions the deterministic tie rule and learned merge order.
  • Proposes a controlled comparison with the same inputs.
Compare with a sample answer

Check that both runs use the identical corpus and chunk boundaries, that pair counts weight repeated chunks, that ties use the same first-counted rule, and that the merge list is replayed in learned order. First compare the exact corpus and printed first few merges from two clean runs with the same requested merge count.

Ungraded paper practice. This is a reproducibility diagnostic for the supplied toy implementation, not a claim about every tokeniser.

Practice 4

For a fictional message containing `café`, a tab and the typed text `<|end|>`, describe two checks that distinguish exact text round-tripping from a real special-token id. Explain why decoded text alone cannot prove that the end marker was inserted as a special id.

Check your answer against these points:

  • Checks equality against the original text, including whitespace.
  • Inspects ids separately from their decoded spelling.
  • Explains that typed marker text and a special id can look alike after decoding.
Compare with a sample answer

Compare the decoded text with the original string exactly, including the tab. Separately inspect the encoded id sequence and verify that the special id is added only by the document-wrapping function. A typed `<|end|>` can be ordinary text that decodes to the same visible spelling, so printed text alone cannot distinguish the ids.

Ungraded paper practice. Treat the example as a test-design exercise; malformed byte streams and actual model interfaces are outside its scope.

Practice 5

For a fictional model with width 64, compare embedding-table entry counts at vocabulary sizes 256 and 452. Then explain in one or two sentences why the shorter sequence from the larger vocabulary is only one side of the trade-off.

Check your answer against these points:

  • Calculates vocabulary size multiplied by width for both cases.
  • States at least one sequence-side and one vocabulary-side cost.
  • Limits the conclusion to the supplied calculation.
Compare with a sample answer

At width 64, the tables contain 256 x 64 = 16,384 and 452 x 64 = 28,928 numbers. A larger vocabulary can shorten sequences and reduce sequence-related work, while making embedding and output tables larger and giving rare tokens fewer examples. These toy counts alone do not establish the cost or quality of a complete model.

Ungraded paper practice. The calculation uses the course's toy vocabulary sizes and a fictional width; it is not a model price estimate.

Practice 6

Design a five-row probe sheet for one toy tokeniser. Include an ordinary English string, a string with changed spaces, a number, text in another writing system or with a combining mark, and a short code fragment. For each row record the exact input, corpus and tokeniser version, observed token count, and one limit on what the result can show.

Check your answer against these points:

  • Includes all five requested input categories.
  • Records the tokeniser and corpus conditions beside each observed count.
  • States a limit that prevents a broad language or task-performance claim.
Compare with a sample answer

A useful sheet keeps the exact text unchanged in each row and records the same corpus, pre-tokenisation rule and merge count. It reports counts for the specific toy implementation only, notes that five examples are not a language benchmark, and does not infer arithmetic ability from digit counts.

Ungraded paper practice. Use invented or public example text; a small probe set cannot establish general language or model performance.

About this exam

Is this exam graded?

Yes. Every question is scored on our server, never in your browser, and you need a score strictly above 80% to pass. The questions are a real, reviewed bank for this course, drawn fresh for each attempt.

Is it only multiple choice?

Yes. This is a multiple-choice knowledge assessment, with at least one question from each chapter in every attempt. Written exercises are separate practice and are not automatically scored. Passing does not establish practical professional competence or independently verify identity.

Is there a time limit?

Yes, 30 minutes, shown as a countdown once you start. If time runs out, whatever you have answered is submitted automatically.

Do I need to download the PDF first?

No. This course's full text is on its own page, so you can read it online and come straight here when you feel ready.

What do you do with my name and email?

We privately record your name, email, request IP, course and privacy acknowledgement for this request. Newsletter consent is separate and optional. See /privacy.html for storage and retention. A passed exam creates a server-stored named certificate. An email confirmation is not identity assurance.

Can I retake it if I fail?

Yes, straight away. Each attempt draws a fresh set of questions and shuffles the answer order, so it will not be identical to your last attempt, and an attempt can only be submitted once.

Is the certificate real?

It is a genuine, server-issued certificate with its own verification page and a unique ID, not a self-declared one. It records that you passed this course's exam on this date. It is not an accredited qualification, a professional licence or proof of employment.