How to evaluate a new frontier model the day it lands

Learn in your browser · Chapter 01 of 07

Models land every week, each with a confident announcement. Your job is to turn the claim into a decision you can defend.

Three rules make the method repeatable. Start from primary sources: the publisher's own model card, release notes and technical report. Write down what you verified and when, because facts about models change. Trust your own tests over anyone's leaderboard, and keep those tests private.

Scroll sideways to see every column.

Then write a decision record (chapter 7) so the next person can see what you knew, and when.

Learn in your browser · Chapter 02 of 07

Most launch-day mistakes come from secondhand facts. Go to the source, then read what it actually says.

Step 1: primary sources. Use the publisher's announcement page, the model card in its official repository, and the technical report. Follow the link from the publisher's own page to the repository rather than searching, because third-party copies and conversions are common. Save the URL, the date and a copy of the text.

Step 2: licence. Open-weight means the trained weights can be downloaded. Open source, as the Open Source Initiative's Open Source AI Definition 1.0 sets it out, asks for more: freedoms to use, study, modify and share, plus information about the training data, the code and the parameters. Open-weight is not the same as open source, and a permissive licence is a third thing again. Read the actual licence text. Look for what it covers (weights, code, outputs), use restrictions and attribution rules.

Step 3: architecture and capabilities. Read these from the card and the configuration file.

Scroll sideways to see every column.

Learn in your browser · Chapter 03 of 07

Now turn the claims into numbers you can plan with.

Step 4: hardware. Apply the memory sum from the workbook on inference engines. The most direct number is the total size of the weight files on the repository, because it already reflects the published precision. Add the KV cache for the context you need, plus an allowance. A mixture-of-experts model must hold every expert in memory. Then check your engine supports the architecture: new designs often need engine updates, so read its release notes that day.

Step 5: cost and latency. For hosted use, read the pricing page for the exact model name. Look for separate rates for cached and uncached input, output rates, peak and off-peak pricing, how thinking tokens are billed and rate limits. For local use, count hardware, power and your time. Measure time to first token and streaming speed on your own prompts, at the hours you will use it.

Step 6: benchmarks. A benchmark table is a claim until you can see how it was made. Ask these questions of every number.

Scroll sideways to see every column.

Treat published numbers as a reason to test, not as a result.

Learn in your browser · Chapter 04 of 07

This is the step that matters most, and the one people skip. About twenty real tasks and an hour of scoring will tell you more about your use than any leaderboard can.

Choose about twenty real tasks from your own work: easy, typical and hard. Include cases that failed before, tasks with a known right answer and tasks that need judgement. Keep the set private, so it is less likely to leak into anyone's training or tuning (a hosted model's provider still sees the prompts you send it), and give it a version number.

Write the rubric first, before you see any output.

Score each criterion 0 to 2
Correct   0 wrong, 1 partly right, 2 fully right
Complete  0 key parts missing, 1 minor gaps, 2 complete
Format    0 ignores the requested format, 1 minor slips, 2 exact
Safe      0 unsafe or leaks data, 1 borderline, 2 appropriate
Pass = Correct 2 and no zeros

Run it blind and side by side. Give the same prompts, at the same settings, to your current model and the candidate. Shuffle the outputs, label them A and B, and score without knowing which is which. Run each prompt more than once, because outputs vary. Record time and cost per task.

Scroll sideways to see every column.

Learn in your browser · Chapter 05 of 07

A model can win your tests and still be the wrong choice. These steps catch what accuracy scores do not.

Step 8: safety and policy behaviour. Behaviour depends on the model plus your system prompt, tools and filters, so test the whole stack. Try requests it should refuse, requests it should not, attempts to make it leak its instructions or private data and, if it uses tools, instructions planted in a document or page it reads (prompt injection). Read the publisher's usage policy.

Step 9: data residency and jurisdiction. For a hosted service, find out where prompts are processed and stored, who controls the data, how long it is kept, whether it may be used to improve the service, and which law applies. Read the privacy policy and the terms covering the API, which may differ from those covering the consumer app. Running open weights yourself keeps prompts with you, but security is then your job. Involve your data protection lead.

Step 10: supply chain. Weights are software you are about to run.

1. Reach the repository from the publisher's own announcement.
2. Prefer safetensors. Avoid pickle-based files unless you trust the source completely.
3. Note the exact revision (commit) you downloaded, so a later change cannot surprise you.
4. Verify each file's checksum against the hash the hub shows.
5. Read any code the repository ships before enabling a flag that allows it to run, often called trust remote code.
6. Treat third-party conversions and quantisations as separate downloads with their own publisher to check.

# Compute a file's SHA-256, then compare it with the hash the hub shows.
# The file name is an example.
sha256sum model-00001-of-00046.safetensors
# Windows: certutil -hashfile model-00001-of-00046.safetensors SHA256

Learn in your browser · Chapter 06 of 07

This is an educational analysis of a public model family, not advice or an endorsement, and it says nothing about any Mickai product. Every fact below was verified on 25 September 2026 from official pages and model cards, listed under sources. Anything not confirmed is marked verify before relying on this.

Scroll sideways to see every column.

Steps 2 and 3, from the model cards:

Scroll sideways to see every column.

• Licence (step 2). Each card says the repository and weights are licensed under the MIT Licence, which is permissive. That answers what you may do with the weights. Whether the release is open source under the Open Source AI Definition turns on published data information and code, which the licence alone does not settle. Verify before relying on this.
• Total versus active (step 3). The preview Pro holds 1.6T parameters but uses 49B per token. Memory follows the first number, compute per token the second. The V4.1 card goes further: its active count differs between reading and writing, and it also lists a 196B-parameter memory component. The text I read does not say whether that sits inside the 552B figure. Verify before relying on this.
• Hardware (step 4). The preview instruct cards give mixed precision: 4-bit for expert parameters, 8-bit for most others. The arithmetic below uses the file sizes the hub lists.
• Serving (steps 3 and 4). The 0731 and 0813 cards give launch examples for two open-source serving engines, attach a speculative-decoding module, and ship their own prompt-encoding scripts instead of a standard chat template. Check your engine supports all of this. The examples also use a flag (trust remote code) that lets code shipped in the repository run, so apply step 10.
• Reasoning control. The preview announcement describes thinking and non-thinking modes, while the preview card lists three modes: non-think, think high and think max. The 0731 and 0813 cards, and the change log from 13 August, give three effort levels: low, high and max. The V4.1 card documents an integer scale from 1 to 100.
• Cost (step 5). The pricing page lists separate rates for cached and uncached input and for output, with off-peak at half the peak rate. I have copied no prices, because they change. It maps the API name deepseek-v4-pro to V4-Pro-0813 and deepseek-flash to V4.1-Flash.
• Aliases and change (steps 1 and 5). The 10 September announcement said that from 14 September all deepseek-v4-pro requests would route to V4.1-Flash. The change log entry of the same date says Pro service continues after 14 September. Two official pages appear to conflict, and I could not tell which is later. Verify before relying on this.
• Jurisdiction (step 9). The privacy policy on DeepSeek's site (last updated 10 February 2026) says the services are provided by a company registered in China and that personal data is stored in the People's Republic of China. Applications developers build on the platform are outside that policy. Read the API terms too: verify before relying on this. Running the open weights yourself is different: prompts stay on your hardware.
• Benchmarks (step 6). The cards and announcements contain benchmark tables and strong claims. I have reproduced none. Read them with the questions from chapter 3, then run your own twenty tasks.

ILLUSTRATIVE CHECK, decimal units. Sizes are my sums of each repository's listed files.

V4-Pro: 1.6 trillion parameters x 0.5 byte (4-bit) = 0.8 TB
        1.6 trillion parameters x 1 byte   (8-bit) = 1.6 TB
        Instruct repository (mixed 4-bit and 8-bit): about 865 GB
        Base repository (8-bit mixed): about 1,606 GB
V4-Flash: 284 billion x 0.5 byte = 142 GB. Repository: about 160 GB.
V4.1-Flash repository: about 510 GB.

None of this includes the KV cache. The preview card suggests a context
window of at least 384K tokens for its deepest reasoning mode.

Every figure sits well beyond a typical single desktop graphics card, before any KV cache. The 8-bit base repository matches one byte per parameter, and the instruct repository sits between the 4-bit and 8-bit sums, as mixed precision predicts.

Learn in your browser · Chapter 07 of 07

A decision you cannot reconstruct is one you will have to make again. Write it down on one page.

DECISION RECORD
Model and exact revision:
Date evaluated, and by whom:
Purpose:
Decision: adopt / trial / reject / revisit on (date)

Evidence, each with source and date
1 Primary sources:
2 Licence and restrictions:
3 Architecture (dense or mixture of experts, total and active parameters,
  context, modalities):
4 Hardware needed:
5 Cost and latency:
6 Published benchmarks (settings, gaps):
7 Our evaluation (set version, scores, blind results):
8 Safety and policy findings:
9 Jurisdiction and data handling:
10 Supply chain (revision, checksums, format):

Unverified items (verify before relying on this):
Risks and mitigations:
Re-check trigger (new release, licence change, 90 days):

Two habits make it work. Mark unverified items plainly so nobody mistakes them for facts, and set a re-check trigger, because facts move.