On the Regularization of Convolutional Neural Networks and Transformers under Distribution Shifts

On the Regularization of Convolutional Neural Networks and Transformers under Distribution Shifts

Assini, Leo (TU Delft Electrical Engineering, Mathematics and Computer Science)

Bohmer, Wendelin (mentor)
Poulsen, C.B. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2022-06-24

The use of Transformers outside the realm of natural language processing is becoming more and more prevalent. Already in the classification of data sets such as CIFAR-100 it has shown to be able to perform just as well as the much more established Convolutional Neural Network. This paper investigates the possible out-of-distribution capabilities of the multi-head attention mechanism, through the classification of the MNIST data set with added backgrounds. Additionally, various regularization techniques are applied to increase the generalization capabilities even more. Regularization is shown to be an important tool to improve out-of-distribution accuracy, though it might imply some trade offs for in-distribution settings.

http://resolver.tudelft.nl/uuid:a5eb41ea-c6b4-4035-a482-fa9bd74d42f7

Student theses

bachelor thesis

© 2022 Leo Assini