Write your response in your own notes, then compare it with the sample answer. This practice is separate from the scored exam and does not affect your certificate.
Practice 1
Write the shape contract for L = 3, S = 5, dk = 2 and dv = 4: give the Q, K, V, weight and output shapes, and say which dimensions must agree.
Compare with a sample answer
Q is 3 by 2, K is 5 by 2 and V is 5 by 4. Scores and weights are 3 by 5 and the output is 3 by 4. Q and K must share the feature width 2 so that dot products are defined, and K and V must share the 5 source positions; the value width 4 may differ. Different query and source lengths are valid here because the call is noncausal.
A good answer uses the L by dk, S by dk and S by dv convention and names both agreement rules.
Practice 2
For Q and K both [[1, 0], [0, 1]] and V = [[10, 0], [0, 20]], calculate the first noncausal attention row, then explain why a row-sum check would not catch a missing scale.
Compare with a sample answer
The dot products are 1 and 0, so the scaled scores are 1/sqrt(2) (about 0.707107) and 0. The first weight is exp(1/sqrt(2)) / (exp(1/sqrt(2)) + 1), about 0.669762, and the second is about 0.330238. The output is about [6.697615, 6.604769]. Without the scale the first weight becomes about 0.731059, yet both versions sum to one, so only comparison with a known calculation reveals the defect.
A good answer keeps full precision, shows the scale and contrasts the scaled and unscaled weights.
Practice 3
Evaluate the two-position fixture with allow=[[False, True], [False, True]] first with causal=False and then with causal=True, and explain each result.
Compare with a sample answer
With causal=False both queries can see only key one, so each has weights [0, 1] and output [0, 20]. With causal=True the first query cannot see position one and allow excludes position zero, so its effective visibility is empty and the function raises ValueError instead of returning a uniform or zero row.
A good answer combines the allow mask with the causal condition and reports the error case rather than inventing an output.
Practice 4
Trace the third row of X, [1, 1, 0, 0], through both fixed heads, explain the uniform row in head two, and list the operations after head concatenation in order.
Compare with a sample answer
Head one selects features zero and one, giving the query [1, 1]; head two selects features two and three, giving [0, 0]. The zero query has a zero dot product with every key, so head two gives equal weights of one third to its three visible positions. After concatenation the block multiplies by WO, adds X, normalises, multiplies by W1, applies ReLU, multiplies by W2, adds the hidden representation and normalises again.
A good answer gives both projected queries, explains the zero dot products and lists the post-normalisation sequence.
Practice 5
Describe the future-independence test for the complete block and explain why per-row normalisation matters to it.
Compare with a sample answer
The test changes only the last row of X, runs the causal block on the original and altered inputs, and checks that the first two output rows are identical while the third differs. It runs the whole block, including both residual paths and normalisation, because a passing attention function alone would not catch a later step that mixes positions. Normalising within each row keeps that boundary; normalising across positions could let the later row change earlier outputs. The test checks information flow only, not language quality or gradients.
A good answer describes the perturbation, the expected comparison and why a passing attention primitive alone is not enough.
Practice 6
Write a short plan for extending the fixture into a tiny language model: give shifted targets for a four-token example, list the missing trainable components and name three separate checks.
Compare with a sample answer
For [a, b, c, d], use inputs [a, b, c] and targets [b, c, d] with causal diagonal visibility. Add token embeddings, a positional scheme, distinct trainable projections, normalisation parameters as chosen, a vocabulary output layer and a next-token loss with optimisation. First reproduce this fixed forward fixture in the chosen framework, then check gradients on a tiny example, then evaluate a trained model on held-out data. None of these steps has been executed in the current lab, so each needs its own evidence.
A good answer separates what has been executed from proposed work and names each evidence stage.