Write your response in your own notes, then compare it with the sample answer. This practice is separate from the scored exam and does not affect your certificate.
Practice 1
Using the invented library facts in this course, write a four-row contract table for: a supported weekday-hours question, an unsupported fee question, an empty question, and a request for another reader's private history. For each row give the expected status, required evidence or boundary, and one observable scoring rule. Add one sentence explaining why the release gate should be fixed before seeing candidate results.
Check your answer against these points:
- Defines expected behaviour for all four supplied situations.
- Makes each scoring rule observable and ties supported claims to evidence.
- Keeps the critical privacy boundary and predeclared gate explicit.
Compare with a sample answer
The weekday-hours row should answer 09:00 to 18:00 and cite the supplied source. The fee row should say the notes do not establish a fee and should not invent one. The empty-question row should ask for a question. The private-history row should refuse without disclosing private information. Each rule should name an observable status and content check. Set the pass threshold, no-regression rule and critical privacy requirement before comparing results.
Ungraded paper practice using fictional course fixtures. It does not test a real library or live assistant.
Practice 2
Design a small versioned case set for a fictional appointment-booking assistant. Include one ordinary successful request, one missing-evidence question, one request that needs clarification, and one critical boundary case. For every case specify a stable id, expected behaviour, whether it is development or held-out evaluation data, and the source version that supports the expected behaviour. State how you would minimise example data and record a changed policy.
Check your answer against these points:
- Covers success, missing evidence, clarification and one critical boundary.
- Separates development cases from held-out evaluation cases and versions the expected evidence.
- Explains minimisation and how a policy change is recorded without overwriting history.
Compare with a sample answer
A suitable set has stable ids such as appt-01 to appt-04, with the four required categories and explicit expected status and content. Keep exploratory examples in a development set and reserve a separate held-out set from tuning. Use invented or minimised fields, and record the reason and source version for each expected response. A policy change creates a new dataset version while preserving the older evidence for comparison.
Ungraded design exercise. The fictional examples are planning fixtures, not operational appointment guidance.
Practice 3
A six-case comparison reports four passes for both baseline and candidate. The candidate fixes two cases but breaks a previously passing loan-period case and a critical privacy case. Write a case-level change table with at least two gains and both regressions, then state the scorer checks you would run before trusting the summary.
Check your answer against these points:
- Reports gains and regressions by stable case id instead of only giving an aggregate.
- Includes the critical privacy regression and explains its effect on the predeclared gate.
- Names meaningful tests of scorer structure and failure handling.
Compare with a sample answer
The aggregate is 4/6 for each version, but it hides the trade. Record the ids and expected and observed outcomes for the two gains, the loan-period regression and the privacy regression. Before relying on the report, test a known wrong response, a missing source, malformed response shape and an absent case id; missing responses remain failures rather than disappearing from the denominator. Under the stated gate the critical privacy regression means HOLD.
Ungraded analysis of the course's fictional scorer fixtures. The result is a teaching example, not evidence about a deployed system.
Practice 4
A team runs the same 8 cases three times each and records 19 successful trials out of 24. Explain what is being repeated and how many distinct cases were tested. Then write a short reporting note that includes the result, a simple interval caveat, and one reason this does not establish performance across all user tasks.
Check your answer against these points:
- Distinguishes 24 repeated trials from 8 distinct cases.
- Reports the observed proportion accurately and qualifies a simple interval.
- Names dependence or case-selection limits on broad generalisation.
Compare with a sample answer
There are 24 trials on 8 distinct cases, not 24 independent cases. The observed trial rate is 19/24, about 79.2%. An interval based on a binomial model would describe uncertainty only if its sampling assumptions fit; repeated attempts on the same cases are dependent, and the hand-picked set may not represent user tasks. Report case coverage separately and avoid a population-wide claim.
Ungraded interpretation prompt. It does not require calculating an interval or inferential claim beyond the course example.
Practice 5
A proposed model judge prefers a longer unsupported answer over a concise answer with allowed source evidence. Draft a rule that separates correctness from style, then specify a blinded comparison with both answer orders and qualified human labels. List the judge prompt and version, the observed order effects, uncertainty or disagreement, and the findings that would trigger investigation before a release decision.
Check your answer against these points:
- Separates evidence-backed correctness from verbosity or style.
- Records judge version and prompt and describes blinded, order-varied comparison with qualified human labels.
- Retains uncertainty or disagreement and identifies findings that affect a release decision.
Compare with a sample answer
Require each factual claim to be supported by allowed evidence and score clarity separately from length. Record the judge version and prompt. Hide system identities, compare both presentation orders where relevant, and compare judge decisions with qualified human labels while retaining uncertainty and disagreement. Order-sensitive decisions, or repeated preference for unsupported detail, are reasons to investigate and record as a release limitation.
Ungraded design prompt about evaluating a judge. It does not establish that any particular judging model is accurate.
Practice 6
Write a compact run-record and release note for a fictional evaluation in which canned responses on six invented cases produce a candidate HOLD. List the versions and hashes or identifiers another reviewer would need, the result and gate, one remaining risk, one untested area, and who must make the release decision. State what passing this paper exercise does and does not establish.
Check your answer against these points:
- Includes reproducibility details, per-case evidence and the predeclared decision gate.
- States a defensible HOLD plus a remaining risk and an untested area.
- Limits claims to the fictional exercise and identifies the decision owner.
Compare with a sample answer
Record the dataset and source version, application commit, prompt and tool settings, model and judge identifiers or 'none; canned responses', scorer and rubric version, trial and retry policy, per-case results, timestamp and content hashes. Report the six-case result and the failed predeclared gate, with decision HOLD. Name limited case coverage as a risk and production integrations as untested. Identify the authorised decision owner. This paper exercise documents a reproducible teaching example; it is not evidence of live-system behaviour, accreditation or release approval.
Ungraded documentation exercise. Use fictional identifiers and no personal or customer data.