​​Self-training with Noisy Student improves ImageNet classification

Using unlabeled data with pseudolabeling improves accuracy in ImageNet.

In the work uses self-training on unlabeled data to achieve 87.4% top-1 on ImageNet, 1% better than SOTA. Huge gains are seen on harder benchmarks (ImageNet-A, C and P).

Method is super simple:

1) Train a classifier on ImageNet
2) Infer labels on a much larger unlabeled dataset
3) Train a larger classifier on the combined set
4) Iterate the process, adding noise

For the beginning they use EfficientNet-B0, pretrained on ImageNet, and predict JFT dataset. Then use predicted classes with confidence > 0.3 on each classes 130k images and get 130М images, delete duplicates, so left 81M.

Architecture:

EfficeintNet, student model much bigger than teacher.

Learning process:

Batch size of 2048
Sota model L2 learned 350 epochs.
The learning rate starts at 0.128 for labeled batch size 2048 and decays by 0.97 every 2.4 epochs if trained for 350 epochs or every 4.8 epochs if trained for 700 epochs :alchemy:

The biggest model L2 have learned 3.5 days on Cloud TPU v3 Pod with 2048 cores.

To start they learned B7 as a student and a teacher. Then, using B7 like teacher learned L0 student. Then they learned L1 and so on to L2 and L2 as the teacher learned L2 student.

Result:
SOTA with 2 times fewer params then last SOTA (FixRes ResNeXt-101 WSL 829M par)


paper: https://arxiv.org/abs/1911.04252
tweet: https://twitter.com/quocleix/status/1194334947156193280?s=20

#cv #selfTraining