Can Invariant Risk Minimization resist the temptation of learning spurious correlations?

Can Invariant Risk Minimization resist the temptation of learning spurious correlations?

van Lith, Jochem (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Pattern Recognition and Bioinformatics)

Karlsson, R.K.A. (mentor)
Bongers, S.R. (mentor)
Krijthe, J.H. (mentor)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2022-06-24

Learning algorithms can perform poorly in unseen environments when they learn
spurious correlations. This is known as the out-of-domain (OOD) generalization problem. Invariant Risk Minimization (IRM) is a method that attempts to solve this problem by learning invariant relationships. Motivating examples as well as counterexamples have been proposed about the performance of IRM. This work aims to clarify when the method works well and when it fails by testing its ability to learn invariant relationships. Therefore, experiments are done on a synthetic data model which simulates four data distribution shifts: covariate shift (CS), confounder based shift (CF), anti-causal shift (AC), and hybrid shift (HB). The experiments exploit IRM’s behaviour with respect to hetero- and homoskedasticity and adaptation of the training environments. We measure the error with regards to the optimal invariant predictor and compare to the non invariant Empirical Risk Minimization (ERM). The results show that IRM is generally able to learn invariance for the CS and CF shifts, especially when the deviation between the training environments is large. In the AC and HB shifts, this strongly depends on the values of the training environments.

invariance principle
Machine learning
Generalization

http://resolver.tudelft.nl/uuid:60339154-302c-407a-9047-05d9b3b21f57

Student theses

bachelor thesis

© 2022 Jochem van Lith