ViT Retinal Screening Presentation

• New lesson editor

MAAI901 · DEEP LEARNING FOR VISUAL DATA

Vision Transformer-Based Retinal Disease Screening

Performance, bias and ethical analysis on the RFMiD dataset

Zainab Fathima Mohammed · ID 1710746

University of Wollongong in Dubai · Instructor: Dr. Milan Dordevic

THE PROJECT

Screening, not diagnosis

The task

Look at a retinal photograph and decide: healthy or abnormal. A clinician then examines the flagged cases. The model does not name the disease.

Why it matters

A screening model that misses disease, or works worse on some kinds of images, can delay care for the people who need it.

What I did

Fine-tuned a ViT-B/16, evaluated it with clinical metrics, tested for bias across subgroups, and discussed the ethics.

DATASET

RFMiD: 3,200 fundus photographs

3,200

photographs

45

disease categories

79.1%

abnormal

3

camera groups

• Predefined split: 1,920 train, 640 validation, 640 test

• Train: 1,519 abnormal and 401 healthy. Test: 506 abnormal and 134 healthy

• Camera is inferred from the original image size: a proxy, not a recorded label

METHOD

Pre-processing and model

Pre-processing

• Crop black borders, pad to a square, resize to 256x256 (done once)

• Training only: random crop to 224x224, flips, rotation up to 15°, brightness and contrast jitter

• Validation and test: resize to 224x224 only

• Class imbalance handled in the loss, not by resampling

Model

• ViT-B/16, about 86 million parameters, pre-trained on ImageNet-21k then ImageNet-1k

• Splits the image into 196 patches of 16x16 and uses 12 self-attention layers

• One output: the probability that the photograph is abnormal

• Weighted binary cross-entropy, AdamW, one-cycle schedule, mixed precision, batch size 32 on a Colab T4

METHOD

Training and tuning

• Four settings, 4 epochs each, one run each. I chose 1e-5 and 0.05. The differences were small, so the choice is only weakly supported

• Trained 12 epochs and saved epoch 2 (validation AUC 0.9513)

• Overfitting: clean training loss fell from 0.144 to 0.035, while validation loss rose to 0.162

• Threshold 0.1011 chosen on the validation set, fixed before testing

RESULTS

Test set: 640 photographs

0.970

AUC

91.5%

sensitivity (463 of 506 abnormal found)

91.8%

specificity (123 of 134 healthy cleared)

Accuracy alone would mislead: 79% of the photographs are abnormal. Missing an abnormal photograph is the more serious error.

BIAS ANALYSIS

Performance depends on the camera

• Sensitivity ranges from 24% to 96% across cameras, hidden by the overall 91.5%

• The smallest group has only 30% abnormal in training: consistent with, but not proof of, a camera shortcut

• Small sample (21 abnormal photographs), wide confidence intervals: it needs a camera-balanced experiment

BIAS ANALYSIS

Other subgroups and the threshold

Brightness and contrast

No clear bias: the confidence intervals overlap. One group has only one healthy photograph, so its specificity cannot be read.

Disease

Lowest sensitivity: ODE 70.6% (n = 17), CSR 76.9% (n = 13), ERM 80.0% (n = 5). ODC, DN, ODE and CSR account for 29 of the 43 misses.

Disease rarity

Detection is similar for rare and common diseases: 91.1%, 90.9% and 91.8%. Rarity does not explain the misses.

Threshold trade-off

Threshold 0.0570: sensitivity 0.945, specificity 0.851. 28 missed and 20 healthy flagged, against 43 and 11 at 0.1011.

EXPLAINABILITY

Attention maps are not a reliable explanation

• Attention rollout combines the self-attention maps from all 12 layers into one map

• Many maps show strong spots in the black corners, outside the retina. Photo 124 (probability 0.99) attends to a corner, not the lesion

• The optic disc stands out in healthy and abnormal photographs. Only some true positives attend to a visible lesion

• The four missed cases I examined had probabilities of 0.02 to 0.06 and looked like healthy maps

A visualisation, not proof that the model looks at the right place.

ETHICS

Ethical and societal implications

Missed disease

43 of 506 abnormal photographs missed (8.5%), including 16 of 21 in one camera group.

Unequal devices

A patient screened with an uncommon camera could get worse care unnoticed.

Fairness untested

No age, sex or ethnicity labels, and data from one country.

Over-trust

Convincing but unreliable heatmaps could encourage automation bias.

Human oversight

A trained professional makes the final decision. This is not a medical device.

Privacy and consent

RFMiD is de-identified. Real use would need consent and secure storage.

WRAP-UP

Conclusion, limits and next steps

Conclusion

The ViT separates healthy from abnormal photographs well (AUC 0.970), but sensitivity ranges from 24% to 96% by camera. A high overall score can make a model look safe. Subgroup reporting shows where it fails.

Limits

• One dataset, split and seed; no external validation

• Camera type is inferred; some subgroups are small

• Probabilities are uncalibrated; attention rollout is unreliable

Next

External validation, camera-balanced training, multi-label disease classification

DEMO

Live demo

1. Open the Streamlit app and upload a test photograph

2. The app shows the probability of abnormal and the decision at threshold 0.101

3. It only says healthy or abnormal. It does not name a disease

Research prototype, not a medical device. Thank you. Questions?