Transfer learning with frozen features and PyTorch

Learn in your browser · Chapter 01 of 08

Separate source knowledge, target labels and the evidence needed to justify reuse.

Transfer learning starts with a representation learned for a source task and reuses it for a target task. In this lab an encoder learns vertical, horizontal and cross patterns, then supports a new line-versus-cross decision. The target task merges vertical and horizontal into line. Reusing the encoder is a hypothesis that its features remain useful under a changed label space and image style. It is not a promise that a pretrained network always beats a fresh model. The comparison can reveal useful reuse, harmful reuse or insufficient evidence. [S4]

Name exactly where the weights came from. Here pretrained means the supplied program first trains its own encoder on 480 original generated source images. It does not mean ImageNet, a downloaded foundation model, NVIDIA weights or a large photographic dataset. The source and target share a deliberately simple drawing family. This creates a tractable transfer experiment on CPU, while sharply limiting the distance of transfer demonstrated. A strong result on these drawings cannot establish useful features for medical images, camera feeds or arbitrary objects.

The source classifier has three logits in order vertical, horizontal, cross. The target classifier has two logits in order line, cross. Keep the encoder and replace the source head with a freshly initialised linear head. Merely renaming the original three columns leaves the wrong output contract. A new head must learn the target boundary from target training labels. The executable separates encoder and head as named modules, making the replacement and the subset of trainable parameters visible rather than hiding them in a large sequential model.

Existing MICKAI courses teach tensors and gradients, training a CNN, forward attention and a tiny character language model. This course adds provenance of source weights, reusable features, full-state freezing and a controlled adaptation comparison. Complete Train and test an image classifier with PyTorch first, or independently demonstrate its tensor, loss and split concepts. We do not repeat its CNN walkthrough as a substitute for transfer. Our central question is whether a particular reused representation helps a specified target experiment under a declared adaptation policy.

Three strategies answer different questions. Scratch starts from a new encoder and target head. Frozen loads the source encoder and updates only its target head. Fine-tune loads the same source encoder and permits encoder parameter updates as well. All receive the same 32 target training views and 48 validation views, the same target head initialisation and 60 update epochs. Their learning rates and BatchNorm policy are fixed before final testing. Comparing them is evidence about these recipes on this dataset, not a universal ranking of transfer methods.

NVIDIA's Fundamentals of Deep Learning repository includes pretrained-model and advanced-architecture modules. This original course responds to its transfer-learning topic using a different manuscript, synthetic data, code and assessment. The repository separates Apache 2.0 code from CC BY 4.0 educational materials and preserves separate third-party terms. We copy no notebook, code cell, imagery, dataset or question. Microsoft stock imagery and NVIDIA marks are not included. Attribution records the curriculum inspiration without implying endorsement, affiliation, an NVIDIA certificate or equivalence to NVIDIA's GPU course. [S1] [S2] [S3]

Learn in your browser · Chapter 02 of 08

Make related views, preprocessing and a final-test boundary explicit before adapting weights.

The generator first assigns a source group to a split, then creates two noisy views of its base drawing. Both views remain together. There are 240 source-training groups and 36 source-validation groups, producing 480 and 72 images. The target has 16 training groups, 24 validation groups and 40 final-test groups, producing 32, 48 and 80 images. Group namespaces are disjoint across all five splits. Different view names alone would not establish independence if two views of one drawing were put on opposite sides of a split.

Target negatives combine vertical and horizontal patterns; positives are crosses. The target training groups contain four vertical, four horizontal and eight cross bases. Two views per base give 16 line and 16 cross images. Validation and final test are also balanced between the two target labels. A training-label majority tie is resolved as class 0, line. That constant prediction gets 24 of 48 validation images and 40 of 80 final images correct. This is a task-specific reference, not evidence that chance is always one half in binary data.

Source drawings use a darker background and stronger strokes. Target drawings use a brighter background, weaker strokes and more view noise. Stroke locations come from four positions in both domains. Split-specific local random generators create disjoint base instances, but all still share the same simple pattern grammar. This shift makes source accuracy an unreliable substitute for target validation. It does not create unrelated photographic domains or establish that the representation survives transformations the generator never implements, such as occlusion, rotation or a new object category.

Fit the scalar pixel mean and population standard deviation on source-training pixels only. Store them with the encoder and apply them unchanged to every source and target split. This is the preprocessing contract of these learned weights. Refitting on target validation leaks development inputs into the transformation, while silently using a new scale breaks the comparison with the source interface. This lab deliberately does not fit a separate target normaliser. Such adaptation could be another declared candidate, but it would require its own training-only fit and validation comparison.

The fit command constructs source training, source validation, target training and target validation pixels. It never calls generate("target_test") to construct target-test pixel or label tensors. A behavioural test replaces the generator with a guard that raises if fit asks for target_test. Only the separate test command opens that split after the strategy, state, class order and normaliser have been saved. The manifest constructs final group IDs, kinds and declared integer label metadata for counting, but does not construct final pixel or label tensors. These educational checks expose accidental access; a learner who reads or edits the generator still has access, so this is not a secure sealed examination service.

Validate raw inputs before normalisation: float32 tensors shaped N,1,8,8; finite pixels between zero and one; long targets shaped N; and indices in the declared class range. After normalisation, values need not stay between zero and one. Do not apply the raw-range check to already standardised values or divide these unit-range floats by 255 again. Reject malformed inputs rather than quietly repairing them under the old run identity. Input checks establish this program's interface, not the truth of a human label or suitability for a new dataset.

Scroll sideways to see every column.

Learn in your browser · Chapter 03 of 08

Connect architecture, source selection and provenance to the exact target interface.

The encoder maps N,1,8,8 pixels through a one-to-six-channel padded 3 by 3 convolution, BatchNorm and ReLU. Average pooling halves each spatial axis, yielding N,6,4,4. Flatten gives 96 values per image; a linear layer and ReLU produce 12 features. The source head maps 12 features to 3 logits; the target head maps 12 to 2. Preserve the batch dimension through flatten. A head that expects 96 inputs is attached at the wrong point if it receives the encoder's final 12-value representation. Inspect shapes before looking at accuracy.

Count state rather than guessing model size from a name. The convolution has 60 parameters, BatchNorm has 12 affine parameters, and the 96-to-12 linear layer has 1,164. The encoder therefore has 1,236 learned values. A 12-to-2 target head adds 26, giving 1,262 total target parameters. The three BatchNorm running buffers are state but are not learned parameters. Consequently a parameter count alone cannot prove that every state value was frozen, and counting a checkpoint's dictionary entries does not equal counting its learned scalar values.

Pretraining uses source labels only, seed 17, full-batch Adam at learning rate 0.01 and 80 fixed epochs. Each epoch updates on 480 source-training images and evaluates 72 source-validation images in evaluation mode. Preserve the state with the lowest source-validation cross-entropy, keeping the earliest exact tie. This selects the source representation before target adaptation. In the recorded CPU run, source-training loss before the first update was 1.118185 and before the last update 0.000184; selected epoch 80 had source-validation loss 0.000291. A low source loss does not settle the target comparison.

Save a deep copy of the selected state rather than retaining a live reference that later updates can modify. The source state includes encoder parameters and BatchNorm buffers. When creating a target model, load only the encoder portion. Initialise a new head using seed 91 for every strategy. This makes its starting weights identical across the target recipes; loading the source head would preserve an irrelevant three-class output. The scratch candidate uses the same new model seed but deliberately does not load the source encoder, which the tests check directly.

Treat a feature extractor as an interface with provenance, not a bag of interchangeable numbers. The saved target artifact names its architecture, source-training normaliser and ordered target labels. Its state dictionary must fit that architecture exactly. A two-output head with labels reversed can load successfully yet be interpreted incorrectly, so label order is separately checked. The final loader uses the checkpoint digest recorded when selection was frozen. A digest detects a later byte change against that record; it does not authenticate who trained the artifact or turn unknown weights into trusted content.

The source model is not distributed as someone else's pretrained download. Learners regenerate it from the original source task in fit. The ZIP contains source, tests and instructions; the author-run checkpoint is separate evidence. In a real model acquisition, investigate the model card, training-data rights, licence, expected preprocessing and prohibited uses before loading. This lab teaches what to record, but it does not audit any external weights. Keep that distinction visible when replacing its tiny encoder with a larger model from a model hub.

features = encoder(images)  # [N, 12]
head = torch.nn.Linear(12, 2)
logits = head(features)      # [N, 2]
loss = torch.nn.functional.cross_entropy(logits, labels)

Learn in your browser · Chapter 04 of 08

Make a full-state freezing claim that survives a deliberate failure demonstration.

Freezing parameters means disabling their gradient accumulation and excluding them from the intended optimiser update. The frozen recipe calls encoder.requires_grad_(False) and gives Adam only the new head's parameters. The head retains gradients. Calling no_grad around the entire target forward would also suppress the head's learning graph, preventing the intended update. Instead this recipe computes encoder features once under no_grad, then trains the head on those features in ordinary gradient mode. The saved test checks all encoder state after adaptation. [S5]

Evaluation mode is separate from gradient tracking. encoder.eval() changes modules with mode-dependent behaviour; it does not make encoder parameters immutable or forbid backpropagation. Conversely requires_grad=False does not set evaluation mode. The target recipes intentionally keep their encoders in evaluation mode so BatchNorm uses fixed running statistics on this small dataset. Fine-tuning still computes encoder gradients and changes weights. The policy is an explicit experiment choice, not a general instruction to run every network in evaluation mode during training. [S5]

With default tracking enabled, BatchNorm keeps running mean, running variance and a batch counter during training-mode forward passes. These buffers can change without an optimiser step and even inside no_grad. In evaluation mode it uses stored running statistics rather than updating them. The lab's source stage learns those statistics; the frozen target stage retains them. The scratch candidate begins with initial running statistics and keeps those fixed under the same target policy. That declared asymmetry follows from different initial states and is one reason the recipes do not isolate every possible cause of a score difference. [S6]

Run the deliberate bad-freeze demonstration on a copy of the source encoder. Disable parameter gradients, leave it in training mode and pass a target-training batch through it. The actual run reports changes to 1.running_mean,1.running_var and 1.num_batches_tracked, with no changed parameters. The full-state freeze check therefore fails. This is an observed implementation failure, not an invented error message or a failed training claim. It shows why checking only weights, gradients or optimiser membership misses a changing feature extractor.

Cache features only while the input transformation and encoder state remain fixed. The frozen recipe computes 32 training feature rows once and reuses them while optimising the head. If you later unfreeze a layer, alter normalisation or change augmentation, those cached rows no longer represent the current forward function. Recompute features or train end to end. Keeping an old cache while updating the encoder can create a misleading comparison because training and evaluation then use different feature functions. An ID and shape check alone cannot detect that semantic staleness.

A useful freeze test snapshots state_dict, performs the operation that is supposed to preserve it, and compares every tensor with exact equality. The good frozen adaptation reports no changed keys. The bad version changes buffers while parameter tensors remain equal. A test that merely checks requires_grad flags would pass both versions. Likewise model.eval() alone would allow ordinary gradient updates if an optimiser still owns its parameters. State comparison tests the promised outcome while flags and optimiser inspection explain how the implementation reaches it.

Learn in your browser · Chapter 05 of 08

Declare adaptation recipes before selection and interpret their differences honestly.

The frozen candidate loads the selected source encoder and trains its 26-parameter head with Adam at learning rate 0.02. The encoder stays in evaluation mode and its entire state is invariant. The scratch candidate trains 1,262 parameters using a new encoder with learning rate 0.01 and head rate 0.02. Fine-tune also trains 1,262 parameters, starting from source encoder state, but uses encoder rate 0.001 and head rate 0.02. These rates are fixed recipe choices. Their difference means this is a comparison of declared strategies, not an isolated causal test of initialisation alone.

All candidates use the same 32 target training views,48 target validation views, initial head state and 60 full-batch update epochs. Their encoders retain fixed BatchNorm running statistics during target adaptation. They may nevertheless require different work: frozen features are computed once; end-to-end recipes repeatedly calculate encoder forwards and gradients. An equal epoch count is therefore not equal compute. The exercise records no controlled timing benchmark. To compare efficiency, separately measure preprocessing, training and inference under a declared machine and measurement protocol rather than inferring a speed ratio from parameter counts.

Within each candidate, select the epoch with the lowest target-validation cross-entropy and keep the earliest exact tie. Across candidates, select the smallest such loss, resolving an exact tie in order scratch, frozen, fine_tune. Validation labels do not supply gradient updates in this code, but they influence selection, so their resulting score is development evidence. Searching more recipes on the same validation set can overfit that selection process. Record the whole predeclared set rather than reporting only the winner or changing the criterion after seeing accuracy.

The actual CPU run selected scratch epoch 1 with loss 0.688436 and 26/48 correct, frozen epoch 9 with loss 0.591373 and 30/48 correct, and fine_tune epoch 60 with loss 0.518323 and 35/48 correct. The fine-tuned recipe won by the declared loss criterion. Its validation accuracy was approximately 72.9 percent, against a 50 percent majority baseline. Scratch's selected early epoch does not mean it stopped running after one epoch; all 60 epochs ran and the earlier state was restored. Preserve this distinction when interpreting a saved selection history.

Negative transfer means reuse harms the relevant target result against an appropriate alternative under the stated comparison. This run does not show negative transfer: both reused recipes exceeded the selected scratch recipe on the recorded target-validation counts and loss. It also cannot establish that source pretraining is always beneficial. One seed, a small labelled target sample, different encoder rates and a related synthetic source leave many explanations open. Repeated independent development experiments could assess stability; the already opened final test cannot become a reusable tuning set for that investigation.

Smaller encoder updates are a common way to limit how aggressively fine-tuning changes a useful representation, but the learning rate is not a measure of semantic retention. A small rate can still accumulate many updates or move sensitive features; a larger rate can occasionally work well. Inspect actual changed state, target validation and the intended source-retention requirement. This exercise does not evaluate the adapted encoder on source labels after every update, so it does not quantify catastrophic forgetting. Make that an explicit additional experiment if retention matters.

Scroll sideways to see every column.

Learn in your browser · Chapter 06 of 08

Use counts, errors and a destructive control to identify limits in a promising result.

The selected fine-tuned model's target-validation confusion matrix has true labels as rows and predicted labels as columns in order line,cross: row line is 19,5 and row cross is 8,16. Accuracy is(19+16)/48=35/48. Line recall is 19/24, approximately 79.2 percent; cross recall is 16/24, approximately 66.7 percent. Accuracy hides that difference. Reversing row and column meaning changes the interpretation of errors, even though the diagonal total stays the same. Save the orientation and label order with the matrix rather than relying on a picture caption.

Two checkpoints can have the same number of correct argmax predictions and different cross-entropy losses. Loss responds to how their logits distribute probability to the true classes; accuracy only checks the winning column. The lab supplies raw logits directly to cross_entropy with long class indices. Do not add softmax before this loss. A higher predicted probability is also not a validated estimate of reliability on a new population. This course does not perform calibration or choose a risk-based acceptance threshold from confidence scores. [S7]

Error IDs connect aggregate metrics to concrete cases. The selected model's first validation error is target_val-0-000-v1. Its other view is not listed, so one view of that base was correct and one was wrong. Later errors include both target_val-1-000-v0 and target_val-1-000-v1, exposing a base that failed in both views. Inspect whether weaker strokes, location or noise coincide with mistakes, but do not assert a cause merely from one example. Preserve all 13 error IDs and the generator so another reader can reconstruct the actual arrays.

The declared corruption replaces every validation pixel by zero before applying the saved normaliser. Every resulting image is identical, so this deterministic evaluation produces one constant label. Because validation is balanced, it gets 24/48 correct and misses all 24 crosses in the recorded run. The first two error IDs are target_val-2-000-v0 and target_val-2-000-v1. This is a destructive control showing that clean performance depends on information erased by blanking. It does not measure robustness to ordinary blur, lighting, sensor noise or malicious inputs that were never tested.

Use validation diagnostics to decide what experiment to run next, then explicitly recognise the resulting development reuse. If you change preprocessing, architecture or rates after studying these errors, the old validation result helped design a new candidate. Record the new comparison and obtain a genuinely fresh final set when a new independent estimate is needed. The supplied sequence makes one selection and one final evaluation. Looking at a final result and then changing the recipe would invalidate a claim that the same test remained untouched, even if every file still had the word test in its name.

For error counts, the observation unit matters. The 48 validation views come from 24 bases and related view errors can co-occur. Treating them as 48 independent source drawings would exaggerate the amount of evidence. This course reports raw view counts and group counts but does not calculate an uncertainty interval or population certainty. A useful extension would repeat the whole development procedure across independent source-group samples and report variation, with a fresh evaluation plan. That is separate work, not evidence already supplied by one deterministic rerun.

Learn in your browser · Chapter 07 of 08

Bind the artifact before a single final evaluation and explain the result without retrospective selection.

Fit saves selected.pt containing the chosen target state_dict, normaliser, architecture identifier and class order. Its fit.json record stores the selected strategy and checkpoint SHA-256. The test loader compares the checkpoint bytes to that digest, checks the architecture and ordered labels, and uses torch.load with map_location=cpu and weights_only=True. A mismatch stops evaluation. These are consistency checks against the local record, not proof of authorship or a general security assessment of arbitrary model files. Only load artifacts whose provenance and trust you have separately assessed. [S8]

The separate test command opens 80 target views from 40 previously unconstructed groups after the selection is fixed. The recorded selected fine-tuned checkpoint gets 68/80 correct, or 85 percent, with loss 0.354516. The training-majority line baseline gets 40/80. This final number is numerically higher than the 35/48 validation count converted to a percentage, but it comes from another small sample. It does not mean the test trained the model or that final evaluation improved the weights. No optimiser or selection step runs in final_test.

The final confusion rows are line[30,10] and cross[2,38]. Thus line recall is 30/40=75 percent and cross recall 38/40=95 percent. Twelve view errors belong to six base groups. The first two error IDs are target_test-0-000-v0 and target_test-0-000-v1. Do not describe 68 correct views as 68 independent successful drawings. The saved final.json retains all errors and binds the checkpoint digest so its counts can be compared to the exact selected artifact, rather than a later model with the same filename.

If final.json already exists, the test command refuses another run in that directory and explains that a repeated evaluation is not a fresh untouched test. This guard makes the intended workflow visible; it does not prevent copying a directory or editing code. The automated suite intentionally exercises the final command in its own temporary run to verify this guard, and a ZIP replay repeats the declared author experiment to check reproducibility. Neither replay supplies an additional independent test sample, and neither author result is a learner study.

The defensible conclusion is that the selected declared fine-tuning recipe achieved 68/80 on this fixed synthetic final set after selection on target validation. Only that recipe was evaluated on final data; we did not rank all three by final accuracy or claim a statistically established final-test advantage over frozen or scratch. The task has two labels, one small drawing grammar and a controlled style shift. No real-world operating threshold, unknown-class handling, fairness analysis, adversarial security, domain-specific suitability or production monitoring has been established.

For a real reuse decision, preserve a model card describing intended inputs, source provenance, preprocessing, target labels, split construction, candidate search, known failures and review responsibilities. Add deployment-specific evaluation before making consequential decisions. A knowledge exam can assess explanations of freezing and selection; it does not observe implementation competence, authenticate identity or authorise professional deployment. Its honest limitations are part of the teaching result, not an optional disclaimer to remove when a score looks strong.

Learn in your browser · Chapter 08 of 08

Execute the package and diagnose failures without replacing actual evidence with a copied answer.

Use Python 3.12 with the official CPU torch 2.14.1 wheel for the tested configuration. The author reused an existing isolated environment with torch 2.14.1+cpu; no packages were installed for this course. The program uses the standard library and torch, no torchvision, external model, GPU, account or network call. The local torch environment emits a warning that NumPy is absent, but this lab does not use NumPy and all 14 tests plus both commands passed. On another machine follow official installation instructions and record the actual versions rather than assuming the author environment was reproduced. [S9]

From the extracted exercise folder, run python -B -m unittest -v test_transfer_lab, then python -B transfer_lab.py fit --out run-01. Fit requires a new output directory and writes selected.pt plus fit.json. Inspect the complete candidate histories, normaliser and bad-freeze result before running python -B transfer_lab.py test --out run-01. That command writes final.json. Treat run names as evidence identifiers, not as a way to erase earlier experiments. The roughly four-and-a-half-hour learning estimate includes reading and written analysis and has not been measured with learners.

The 14 tests cover input rejection, disjoint groups and paired views, source-only normalisation, feature/head shapes, source-loss reduction, equal initial heads, frozen full-state invariance, a failing BatchNorm freeze, fine-tuning gradients in evaluation mode, fresh scratch state, destructive-control counts, final-access guarding and selection, digest/class-order rejection, and matrix orientation. Passing them demonstrates these checked behaviours for this implementation. It does not prove every possible bug absent, independence from all generator similarities or correctness after an untested code modification. Read each test's intervention and expected observation.

Three useful failure investigations differ. If the input guard rejects a missing channel axis, repair the caller's tensor contract and preserve the changed run identity. If the digest check rejects a modified selected.pt, do not replace the expected hash just to continue; reconstruct why the artifact changed and freeze a valid selection record. If the bad-freeze demonstration reports changed running buffers, that is its intended observed failure, while the good frozen candidate must still report no changed encoder keys. A test failure is evidence to diagnose, not a reason to rewrite the expected answer around an unexplained result.

The author extracted the final ZIP into a new task-local directory, checked every member for absolute paths, traversal and unexpected files, then reran its tests, fit and final commands using the same CPU interpreter. The replay matched selected strategy, losses and counts. Checkpoint byte hashes can differ between separately saved archives, so compare each checkpoint to its own recorded digest and compare semantic results separately. This is an author reproducibility check, not an independent reviewer approval, a measured learner outcome or proof of identical results across PyTorch releases and hardware. [S10]

The package's README explains commands and scope; ANSWER-KEY gives worked interpretations, not private scored exam questions. Keep your own run logs and reasoning before reading it. The eight workbook activities and separate written practices are ungraded. The 40-question bank assesses knowledge across all eight chapters. An assessment draws 20 questions, with a pass threshold strictly above 80 percent, so at least 17 correct answers out of 20 are required. Its educational certificate records that knowledge result; it does not verify identity, observe implementation competence, provide NVIDIA certification or authorise professional deployment. Check the published course and assessment pages for current availability and terms; local files and an exam URL do not establish that an exam or certificate is publicly available. Publication, accessible print layout, independent semantic/code review and integration checks are separate responsibilities.

For handover, include the exact source and package hashes, Python and torch versions, manifest counts, selected strategy, validation rule, final matrix and failure observations. Say whether source code, generator, rates or candidates changed after inspecting development results. If you extend the source domain or target labels, revise the architecture and class-order contract, regenerate the experiment and define new evaluation data. The supplied files provide a small, inspectable starting point for reasoning about reuse, not a vendor runtime, a pretrained general visual system or a credentialing service.

python -B -m unittest -v test_transfer_lab
python -B transfer_lab.py fit --out run-01
python -B transfer_lab.py test --out run-01