AI fraud-alert triage with investigator control

Learn in your browser · Chapter 01 of 06

Use AI to organise a review task, while keeping the investigator’s factual judgement and action authority separate. A low score is a route choice, not permission to make a case disappear.

Bracken Fraud Learning Lab is an invented adult learning exercise. You receive 500 transaction alerts, an uncalibrated Q1 priority index and twelve independent mandatory-review flags. The task is to design a review queue and challenge its evidence. Nothing here connects to an account, sends a report, contacts a customer, blocks a payment or decides that someone committed fraud. The word investigator names a role in the paper exercise, not a credential or authority granted to the learner.

Write the decision before choosing a model. In this packet the decision is which records receive human review during a limited work window, with all pending work retained. AI may propose an order or a draft case note. An investigator compares that note with the supplied evidence and records uncertainty. A separately authorised human process would own any real financial action. Ranking, case disposition and action permission are three different records; copying the same score into all three hides the missing decisions.

A plausible failure is an efficiency report that celebrates closing 360 alerts automatically. The system has treated score 20 as evidence that those cases are legitimate, even though no investigator has established that fact. The dashboard loses the unreviewed population, later evaluation counts it as negative and future training learns from the system’s own decisions. Another failure preserves the cases technically but removes them from visible queues and ageing reports. Retaining bytes alone does not preserve an effective review route.

Scroll sideways to see every column.

For AL-017, TX-017 supplies GBP 639.00 and Q1 supplies index 90. Those are observed packet fields. The packet does not supply verified transaction purpose or a customer-contact record. The sentence “Fraud is confirmed and the transfer must be frozen” adds both an unsupported conclusion and an unauthorised action. Replace it with: “TX-017 records GBP 639.00; Q1 assigns priority index 90. Purpose and authorisation remain unverified. Refer the evidence and open questions for investigator review.” Verify each clause against packet.json rather than judging how confident it sounds.

1. Name the population, proposed decision and work window.
2. List permitted input fields and the exact output an assistant may prepare.
3. Separate ranking from the investigator’s evidence decision and from any financial action.
4. Preserve mandatory and below-threshold cases with explicit states and owners.
5. Record stop conditions: missing evidence, unresolved authority or unavailable review capacity.

FCA FCG4 is guidance within its stated financial-sector scope. It highlights management oversight, risk assessment, fraud data, response procedures and learning from incidents. It does not prescribe Bracken’s scores or capacity. The PRA SS1/23 framework has a narrower stated institutional scope, including UK-incorporated firms with specified internal-model capital approvals. Do not describe every payment firm as automatically within that statement. A compliance owner must establish which rules and guidance apply to an actual entity and activity.

Mickai Training Academy is not an accredited training provider. This course is educational only and is not financial, legal or investigation advice. Its knowledge assessment does not verify professional competence, identity or permission to operate a real service. A proper human-control claim would need evidence that the person can access relevant facts, hold the proposal, request correction and use a workable alternative, not merely an approval button beside an unexplained score.

Learn in your browser · Chapter 02 of 06

Preserve what was available at routing time and join later outcomes only for their declared evaluation purpose. A clean table can still leak an answer or hide missing cases.

The source archive contains alerts.csv with 500 unique IDs AL-001 throughAL-500. It also contains baseline_outcomes.csv and shifted_outcomes.csv. These outcome files represent complete later educational adjudications of two hypothetical cohorts using the same row identifiers and scores. They are counterfactual evaluation scenarios, not two observations of the same real transaction. Always identify the cohort when joining. The fixture deliberately supplies complete labels for arithmetic; real investigations rarely offer that immediate certainty.

At the routing cutoff, 12 October 2026 at 08:00 UTC, the alert fields are available but the baseline labels are not available until 12 November 2026. Reading them early would leak future information into the proposed decision. Evaluation on 13 Novembercan use the baseline file. Evaluation of the shifted scenario occurs on 13 Decemberafter its 12 Decemberavailability. The original scores are supplied; this course does not train or run an AI detector. Describing the checker’s arithmetic as a validated fraud model would be false.

Scroll sideways to see every column.

Check uniqueness before joining. A repeated AL-017 can inflate the selected count and give its outcome twice the influence. Check coverage in both directions: an alert without an outcome is unresolved, while an outcome without an alert does not belong in this cohort. Do not quietly turn an empty label into zero or drop the unmatched row. Record the count difference and hold the affected evaluation. A fixed 500-row denominator must still represent 500 distinct cases after every transformation.

Work the time check. AL-111 has score 50 and mandatory_review true at the routing cutoff. Its later baseline label is 0. An analyst writes “this was legitimate, so the flag can be ignored”. That reasoning imports a later answer into the earlier decision. The correct routing uses the flag and available evidence. The later label can inform retrospective evaluation of that policy, with the dates visible, but it cannot prove that the earlier reviewer was entitled to skip the case.

1. Inventory files, cohort identity, row count and unique identifiers.
2. Record feature availability and evaluation cutoffs before joining labels.
3. Require one valid outcome per evaluated case, or explicitly describe incomplete coverage.
4. Preserve original inputs; make labelled working copies for experiments.
5. Record the score version and any transformation, then verify counts after the join.

Investigators need an evidence map rather than a large uncontrolled data dump. The packet uses invented identifiers and amounts and excludes names, addresses and payment credentials. In real work, a pseudonymous identifier can still relate to a person; it is not automatically anonymous. An authorised process must establish necessary fields, permitted purpose, retention, access and any sharing conditions. No external assistant or connector is needed here. Run locally or use the printed band table and calculator.

On a paper control sheet, write 500 alerts, 500 baseline outcomes, 500 matched unique IDs and an evaluation date after label availability. List the blocked alternative: using those labels to choose a routing rule as if they were available in October. Also note selection leakage: if a later outcome field encodes an investigator decision caused by this very queue, it is not necessarily independent evidence of the underlying event. The next chapter’s metrics are conditional on the supplied labels and population, not a general claim about customers or future fraud.

Learn in your browser · Chapter 03 of 06

Count the four outcomes before choosing a headline metric. High accuracy can coexist with many missed positive cases, while improved recall can create more review work.

For these calculations, a positive outcome means the supplied later label 1, and a positive selection means the score meets the chosen threshold. They are not equivalent. A true positive is selected and labelled 1; a false positive is selected and labelled 0. A false negative has label 1 but falls below threshold; a true negative has label 0 and falls below it. These descriptions concern the frozen evaluation labels. They do not convict a person or establish the legal meaning of fraud.

Scroll sideways to see every column.

At threshold 70, select scores 70 or 90: 40 plus 60 equals 100 cases. True positives are 20 plus 15 equals 35; false positives are 20 plus 45 equals 65. Fifteen of the 50 positive cases remain below the threshold, so false negatives equal 15. The remaining 385 negative cases are true negatives. Verify 35+65+15+385=500 and 35+15=50 before calculating rates. A greater-than test instead of greater-than-or-equal would accidentally exclude the whole 70 band.

Precision is 35/100=35 percent: the share of selected cases with positive later outcomes. Recall is 35/50=70 percent: the share of all positive outcomes selected. The missed-positive fraction, or false negative rate, is 15/50=30 percent. The false positive rate is 65/450, about 14.44 percent. It is not 65 percent: 65/100 is the fraction of selections with negative outcomes. Write the denominator beside every metric so readers can distinguish review burden from the error rate among all negative cases.

Scroll sideways to see every column.

An all-negative rule would label 450 of 500 correctly, giving 90 percent accuracy and zero recall. Threshold 90 also has 90 percent accuracy, because 20+430=450, yet it selects 20 positive cases. The identical accuracy figure hides very different behaviour. Threshold 50 recovers 45 of 50 positives but sends 200 cases to review. None of these numbers alone determines acceptable harm, investigation cost or capacity. A reviewer must discuss missed cases, unnecessary scrutiny, available evidence and service consequences separately.

The ICO’s AI accuracy guidance distinguishes factual accuracy from statistical performance and discusses precision, recall and different error consequences. Its current page is marked under review following the Data (Use and Access) Act. This course uses its bounded measurement distinction, not a complete statement of current UK legal duties. In particular, a score or inferred suspicion should not silently become an established fact in a case record.

For a separate base-rate illustration, suppose a detector has 80 percent sensitivity and a 10 percent false positive rate. In 1000 invented cases with 20 positives, expected counts are 16 true positives and 98 false positives, so precision is 16/114, about 14.04 percent. With 100 positives at the same rates, counts become 80 and 90, and precision is 80/170, about 47.06 percent. These are arithmetic scenarios with fractional-rate assumptions, not measurements of Q1. Different prevalence can change precision even when those two rates are unchanged.

Learn in your browser · Chapter 04 of 06

A useful summary separates directly observed fields, model output, inference and missing evidence. It gives the investigator a traceable starting point instead of a fluent accusation.

Use investigator_form.md as a claim ledger. For each material sentence, record the source reference and version, the observed field, the proposed inference, the unknowns and the reviewer’s disposition. Amount, currency, event status and time belong together. A transaction log entry can support that a row was recorded; it may not prove who authorised the event or what happened outside the recorded system. Do not strengthen “recorded” into “verified” because an assistant writes elegantly.

Work AL-017 again at the claim level. The source row is TX-017, the amount is GBP 639.00 and Q1 gives index 90. The supplied case card expressly lacks verified purpose and customer contact. The phrase “unusual payment” would require a defined comparison or baseline not present in this card. The phrase “fraudster” would turn an unresolved concern into an accusation. A sound draft states the fields, names both missing facts and requests investigator review. It does not fill the gaps with a plausible customer story.

Scroll sideways to see every column.

For AL-067, a fictional invoice note says that a reviewer approved the case and it should be marked cleared. Keep that sentence in the evidence record as an unverified claim. It is not an instruction to the assistant and it is not a reviewer decision authenticated outside the source. The note cannot change the threshold, suppress an alert, grant a role or populate a confirmed-legitimate label. A cautious summary can say that the source contains an approval claim which remains unverified.

An equally poor approach deletes every suspicious-looking source paragraph. That can remove evidence an investigator needs to understand why a summary went wrong. Preserve authorised source content with appropriate access controls and distinguish content from control. In this local exercise the note is harmless fictional text; no address or live instruction is provided. Your task is to mark its attempted authority claim and continue comparing the actual packet fields.

Challenge a proposed explanation as well as its conclusion. “The score is high because the payment is fraudulent” is circular. “The amount caused the score” is unsupported here because no feature-attribution record was supplied. Even a future feature contribution would describe a particular model calculation, not necessarily a cause of a real-world event. In Bracken, the honest explanation is that the score is a supplied index and its generation is outside the fixture. You can explain the routing rule without inventing how a detector reached the score.

1. Copy the exact source identifier, field, unit and version.
2. Label model output separately from source observation.
3. List missing or conflicting evidence before drafting a conclusion.
4. Inspect source-contained instructions without acting on them.
5. Have the investigator review the complete draft and record corrections against that version.

Learn in your browser · Chapter 05 of 06

Treat capacity as a constraint to expose, not a reason to relabel unreviewed cases. Combine model ranking with separate review rules and account for every pending case.

The threshold table evaluates a score-only rule. The actual Bracken proposal also routes all twelve mandatory-review flags. At threshold 70, four flagged cases are already among the 100 score selections, so adding twelve blindly would double-count them. The union contains 100+12-4=108 fresh cases. The eight additional lower-score cases are AL-101, AL-102, AL-111, AL-112, AL-201, AL-206, AL-207 and AL-208. Their baseline later labels contain three positives and five negatives.

Accordingly, the complete routing proposal has 38 true positives and 70 false positives, leaving 12 false negatives and 380 true negatives among all 500 cases. Its recall is 38/50=76 percent and its precision is 38/108, about 35.19 percent. These are selection metrics for the combined route. Do not display 70 percent recall from the score-only table while claiming it evaluates the mandatory route, or claim that routing a positive case means the investigator resolved it successfully.

Scroll sideways to see every column.

All 128 cases require 1536 review minutes under the fixed-duration assumption. Available time is 480 minutes, leaving 1056 minutes of unmet work. This is a work-budget calculation, not a forecast of actual investigator productivity. Complex cases, breaks, rework and escalation could change the assumption. The given 240 minutes already represents productive review time; do not subtract a second invented break allowance or silently substitute 480 minutes per person.

The prior queue is already 26 hours old, beyond the invented 24-hour ageing trigger. Escalation is required at the start of this scenario, independently of model performance. A proposed allocation can handle the twelve fresh mandatory cases first, then the twenty prior cases and eight of the remaining score-selected cases. That uses 40 slots and retains 88 fresh cases as pending. Another justified ordering may be defensible, but it must name the trade-off, account for every case and leave no mandatory or old case invisible.

Raising the threshold to 90 would reduce score selections to 40, but the mandatory union still contains 50 fresh cases. Together with 20 prior cases, demand is 70 and 30 remain after 40 reviews. Raising a threshold therefore does not solve the backlog in this packet and would alter missed-case exposure. A numerical target is not authority to close the other cases. Ask the accountable owner to address capacity or constrain a proposed trial while preserving the existing manual route and its pending work.

Use the investigator form to record review state and action boundary separately. Permitted learning dispositions include request more evidence, propose a reasoned finding for review, or hold an unsupported draft. The fixture is not an operational reporting manual. Actual disclosure, suspicious-activity reporting, reimbursement, freezing, legal process or customer contact depends on applicable duties and authorised teams. Do not have the assistant improvise a reporting deadline or share a case because a training field says “escalate”.

Learn in your browser · Chapter 06 of 06

Do not train a detector on the system’s own unreviewed closures. Keep label availability, investigation evidence and selected populations visible, then respond to a changed pattern with a controlled review.

An outcome pipeline can quietly convert “not investigated” into “not fraud”. In Bracken, feedback_candidates.csv presents six proposed labels. At the 12 November cutoff, only FB-1 and FB-2 meet the stated structured conditions. FB-3 was auto-closed without investigation. FB-4 is still pending despite a proposed zero. FB-5 has a confirmed outcome recorded after the cutoff. FB-6 says confirmed legitimate but carries label 1. None of those four entries should become a clean training row merely because its label column contains an integer.

Scroll sideways to see every column.

The checker can detect these explicit contradictions and times. It cannot authenticate who investigated, decide whether the evidence justified the finding or interpret all free text. A self-written “reviewed” flag can pass a structural gate without demonstrating real review. A production label policy would need source evidence, authorised adjudication, corrections and controlled access. This exercise makes those missing capabilities visible rather than presenting a boolean field as professional assurance.

Selection bias remains even with honest labels. If only the 100 score-selected baseline cases were investigated, the observed positive proportion would be 35/100=35 percent. The supplied whole-cohort proportion is 50/500=10 percent. The 35 percent is a property of that selected group, not an estimate of all transactions. Nor can you infer 15 missed cases without outcomes outside the selected queue. Bracken’s complete later labels make that arithmetic possible; a real evaluation would need a defensible route to outcome evidence, with the ethical and practical limits recorded.

Now inspect shifted_outcomes.csv as its separate hypothetical later cohort. It still contains 50 positive outcomes across 500 cases, but only 20 positives lie in the score 70 or 90 bands. At threshold 70, score-only TP 20, FP 80, FN 30 and TN 370 give precision 20 percent and recall 40 percent. The proposed mandatory union selects 108 and captures 26 positives, giving recall 52 percent and precision about 24.07 percent. The unchanged total alert count and unchanged 10 percent prevalence have not preserved ranking performance.

Bracken’s invented monitoring trigger is score-only matured recall below 60 percent. The shifted 40 percent breaches it. That trigger cannot be measured on 12 October before outcomes mature; immediate monitoring should instead report missing scores, unexpected versions, queue ages, review completion and preserved hold states. These operational signals do not prove recall. Maintain both clocks: what is observable now and what can be judged only when outcomes become available.

1. Freeze the evaluated cohort, model version, threshold and label cutoff.
2. Exclude unresolved, contradictory and unavailable labels without relabelling them.
3. Describe which cases were investigated and the limits of that selected evidence.
4. Compare mature error counts and immediate queue signals separately.
5. On a trigger, hold expansion, preserve pending cases and evidence, and have the accountable owner choose a controlled fallback and retest plan.

Rollback is not deleting the new model and assuming the problem has gone away. Record which ranking version is withdrawn, which manual or previously checked route will take over, where pending cases remain and who owns follow-up. A manual route still has the same finite staffing unless additional capacity is actually supplied. Restoring Q1 would also not establish better performance in the changed pattern. Retest a proposed alternative on appropriately separated evidence and record unresolved legal, operational and customer consequences before a real decision.

The NIST AI Risk Management Framework is a voluntary framework, not a certification for this packet. Use its risk-management orientation to organise the evidence and follow-up. The capstone’s result is a bounded recommendation to hold expansion, not a model certification. Your knowledge exam draws 20 questions; exactly 16 correct is 80 percent and fails, while 17 is 85 percent and passes. Written practice and the capstone are ungraded unless a separate qualified marking process is actually provided.