Print Email Facebook Twitter Towards a Safer and More Reliable Selective Classifier Title Towards a Safer and More Reliable Selective Classifier: With Human Knowledge and Value Incorporated Author Chen, Xinyue (TU Delft Electrical Engineering, Mathematics and Computer Science) Contributor Yang, J. (mentor) Houben, G.J.P.M. (graduation committee) Khademi, S. (graduation committee) Degree granting institution Delft University of Technology Programme Computer Science Date 2022-08-31 Abstract While the performance of traditional confidence-based rejectors is heavily dependent on the calibration of the pretrained model, this study proposes the concept of feature-based rejectors and the whole pipeline where such rejector can be used in. Multiple design and development decisions along the implementation are discussed in the paper - involving humans in the loop of creating the feature set that machine should use and defining value, using interpretability methods such as saliency map to extract the features that the machine actually uses, and SceneRejector is the end product of the whole process. It is relevant to the field of computer vision, and it is applied to the task of scene classification within the scope of this paper. Its performance is evaluated against baselines with accuracy, rejection rate, and a new metric named value, which more comprehensively measures the practical value of machine learning with a reject option in different use cases. Experiments have proved the concept of a feature-based rejector works and it is able to filter out unknown unknowns, which is a challenge to the confidence-based rejectors. It also creates better value than the traditional confidence-based rejectors in some cases, especially when the machine learning model is not well calibrated. Further analysis is conducted to understand the behavior of SceneRejector as well. It is discovered that SceneRejector brings better value than confidence-based rejectors when the pretrained model is not well calibrated, and that rejection rate and accuracy of the rejector is also correlated with the value. Subject selective classifierrejectorresponsible machine learningvalueCrowdsourcing To reference this document use: http://resolver.tudelft.nl/uuid:9553608a-4faf-4ea7-997c-72e730b01fba Part of collection Student theses Document type master thesis Rights © 2022 Xinyue Chen Files PDF Master_Thesis_Xinyue_Chen.pdf 1.18 MB Close viewer /islandora/object/uuid:9553608a-4faf-4ea7-997c-72e730b01fba/datastream/OBJ/view