Exam · Design a repeatable evaluation suite for an AI application

Take the exam and earn your certificate.

A real, server-scored exam for this course. Score strictly above 80% to pass and get a genuine, verifiable certificate.

Before you start

We ask for your name and email first, the same as a PDF download, so we know you are here and your certificate can carry the right name.

We privately record your name, email and request IP for this exam request. Read the privacy notice and retention policy.

Visit Mickai on LinkedIn

Complete the security check before submitting.

Cloudflare Turnstile helps prevent automated submissions. How this check uses data.

You can retake the exam any time. Each attempt draws a fresh set of questions.

Written practice, ungraded

Write your response in your own notes, then compare it with the sample answer. This practice is separate from the scored exam and does not affect your certificate.

Practice 1

Using the invented library facts in this course, write a four-row contract table for: a supported weekday-hours question, an unsupported fee question, an empty question, and a request for another reader's private history. For each row give the expected status, required evidence or boundary, and one observable scoring rule. Add one sentence explaining why the release gate should be fixed before seeing candidate results.

Check your answer against these points:

  • Defines expected behaviour for all four supplied situations.
  • Makes each scoring rule observable and ties supported claims to evidence.
  • Keeps the critical privacy boundary and predeclared gate explicit.
Compare with a sample answer

The weekday-hours row should answer 09:00 to 18:00 and cite the supplied source. The fee row should say the notes do not establish a fee and should not invent one. The empty-question row should ask for a question. The private-history row should refuse without disclosing private information. Each rule should name an observable status and content check. Set the pass threshold, no-regression rule and critical privacy requirement before comparing results.

Ungraded paper practice using fictional course fixtures. It does not test a real library or live assistant.

Practice 2

Design a small versioned case set for a fictional appointment-booking assistant. Include one ordinary successful request, one missing-evidence question, one request that needs clarification, and one critical boundary case. For every case specify a stable id, expected behaviour, whether it is development or held-out evaluation data, and the source version that supports the expected behaviour. State how you would minimise example data and record a changed policy.

Check your answer against these points:

  • Covers success, missing evidence, clarification and one critical boundary.
  • Separates development cases from held-out evaluation cases and versions the expected evidence.
  • Explains minimisation and how a policy change is recorded without overwriting history.
Compare with a sample answer

A suitable set has stable ids such as appt-01 to appt-04, with the four required categories and explicit expected status and content. Keep exploratory examples in a development set and reserve a separate held-out set from tuning. Use invented or minimised fields, and record the reason and source version for each expected response. A policy change creates a new dataset version while preserving the older evidence for comparison.

Ungraded design exercise. The fictional examples are planning fixtures, not operational appointment guidance.

Practice 3

A six-case comparison reports four passes for both baseline and candidate. The candidate fixes two cases but breaks a previously passing loan-period case and a critical privacy case. Write a case-level change table with at least two gains and both regressions, then state the scorer checks you would run before trusting the summary.

Check your answer against these points:

  • Reports gains and regressions by stable case id instead of only giving an aggregate.
  • Includes the critical privacy regression and explains its effect on the predeclared gate.
  • Names meaningful tests of scorer structure and failure handling.
Compare with a sample answer

The aggregate is 4/6 for each version, but it hides the trade. Record the ids and expected and observed outcomes for the two gains, the loan-period regression and the privacy regression. Before relying on the report, test a known wrong response, a missing source, malformed response shape and an absent case id; missing responses remain failures rather than disappearing from the denominator. Under the stated gate the critical privacy regression means HOLD.

Ungraded analysis of the course's fictional scorer fixtures. The result is a teaching example, not evidence about a deployed system.

Practice 4

A team runs the same 8 cases three times each and records 19 successful trials out of 24. Explain what is being repeated and how many distinct cases were tested. Then write a short reporting note that includes the result, a simple interval caveat, and one reason this does not establish performance across all user tasks.

Check your answer against these points:

  • Distinguishes 24 repeated trials from 8 distinct cases.
  • Reports the observed proportion accurately and qualifies a simple interval.
  • Names dependence or case-selection limits on broad generalisation.
Compare with a sample answer

There are 24 trials on 8 distinct cases, not 24 independent cases. The observed trial rate is 19/24, about 79.2%. An interval based on a binomial model would describe uncertainty only if its sampling assumptions fit; repeated attempts on the same cases are dependent, and the hand-picked set may not represent user tasks. Report case coverage separately and avoid a population-wide claim.

Ungraded interpretation prompt. It does not require calculating an interval or inferential claim beyond the course example.

Practice 5

A proposed model judge prefers a longer unsupported answer over a concise answer with allowed source evidence. Draft a rule that separates correctness from style, then specify a blinded comparison with both answer orders and qualified human labels. List the judge prompt and version, the observed order effects, uncertainty or disagreement, and the findings that would trigger investigation before a release decision.

Check your answer against these points:

  • Separates evidence-backed correctness from verbosity or style.
  • Records judge version and prompt and describes blinded, order-varied comparison with qualified human labels.
  • Retains uncertainty or disagreement and identifies findings that affect a release decision.
Compare with a sample answer

Require each factual claim to be supported by allowed evidence and score clarity separately from length. Record the judge version and prompt. Hide system identities, compare both presentation orders where relevant, and compare judge decisions with qualified human labels while retaining uncertainty and disagreement. Order-sensitive decisions, or repeated preference for unsupported detail, are reasons to investigate and record as a release limitation.

Ungraded design prompt about evaluating a judge. It does not establish that any particular judging model is accurate.

Practice 6

Write a compact run-record and release note for a fictional evaluation in which canned responses on six invented cases produce a candidate HOLD. List the versions and hashes or identifiers another reviewer would need, the result and gate, one remaining risk, one untested area, and who must make the release decision. State what passing this paper exercise does and does not establish.

Check your answer against these points:

  • Includes reproducibility details, per-case evidence and the predeclared decision gate.
  • States a defensible HOLD plus a remaining risk and an untested area.
  • Limits claims to the fictional exercise and identifies the decision owner.
Compare with a sample answer

Record the dataset and source version, application commit, prompt and tool settings, model and judge identifiers or 'none; canned responses', scorer and rubric version, trial and retry policy, per-case results, timestamp and content hashes. Report the six-case result and the failed predeclared gate, with decision HOLD. Name limited case coverage as a risk and production integrations as untested. Identify the authorised decision owner. This paper exercise documents a reproducible teaching example; it is not evidence of live-system behaviour, accreditation or release approval.

Ungraded documentation exercise. Use fictional identifiers and no personal or customer data.

About this exam

Is this exam graded?

Yes. Every question is scored on our server, never in your browser, and you need a score strictly above 80% to pass. The questions are a real, reviewed bank for this course, drawn fresh for each attempt.

Is it only multiple choice?

Yes. This is a multiple-choice knowledge assessment, with at least one question from each chapter in every attempt. Written exercises are separate practice and are not automatically scored. Passing does not establish practical professional competence or independently verify identity.

Is there a time limit?

Yes, 30 minutes, shown as a countdown once you start. If time runs out, whatever you have answered is submitted automatically.

Do I need to download the PDF first?

No. This course's full text is on its own page, so you can read it online and come straight here when you feel ready.

What do you do with my name and email?

We privately record your name, email, request IP, course and privacy acknowledgement for this request. Newsletter consent is separate and optional. See /privacy.html for storage and retention. A passed exam creates a server-stored named certificate. An email confirmation is not identity assurance.

Can I retake it if I fail?

Yes, straight away. Each attempt draws a fresh set of questions and shuffles the answer order, so it will not be identical to your last attempt, and an attempt can only be submitted once.

Is the certificate real?

It is a genuine, server-issued certificate with its own verification page and a unique ID, not a self-declared one. It records that you passed this course's exam on this date. It is not an accredited qualification, a professional licence or proof of employment.