Register
Registering takes a minute and helps plan the apéro. If you attend on site, you will receive a calendar invitation. If you join online, an email with the Teams link will follow.
Summary
- The thesis combines tabular data with high‑dimensional sources such as images and text, known as multi‑modal learning, and surrounds each prediction with an interval that holds the real value with some mathematical guarantees (e.g., 95 times in 100). Seven papers were authored or co‑authored during the PhD (five published, two preprints). Five form the core of the thesis; the other two are related.
- Part I measures what photos and text add when a table already describes each case. In predicting rental prices, apartment photos and satellite views cut a neural network’s error by about a tenth. For hurricane intensity one day ahead, the forecasters’ written discussions lower another network’s error.
- Part II replaces the single prediction with an interval. The first method, SEMF, builds such an interval around models like XGBoost that do not give one on their own. The second, CLEAR, learns how much of the width should come from noise in the data and how much from what the model does not know.
- Part III combines the two. An interval calibrated on complete data can be too narrow when the sources disagree, or when one is missing at prediction time. In the experiments, calibrating separately for cases where the sources strongly disagree, or for the set of sources present, helped recover intervals that hold the real value close to 95 times in 100.
Animations
Abstract
Humans use various sources of information to make decisions and can convey the uncertainty associated with those decisions. These types of information (tabular data, images, textual descriptions, etc.) are called modalities. Machine learning models can similarly benefit from combining these modalities, and it is equally valuable to convey how trustworthy their predictions are. The first part of the thesis measures what images and text add to tabular data in two applications. First, for 11,000 rental listings in Switzerland, combining photos of the apartment and satellite images of the area with the information traditionally used, such as size and location, lowers the average rent prediction error by roughly a tenth. Second, for hurricanes, where lives and property are at stake, adding the reports written by forecasters improves intensity predictions a day ahead. The second part introduces new methods that quantify the uncertainty of any predictor. The goal is to showcase methods that produce a range of plausible values instead of a single number, with the aim of having this range contain the true outcome for instance 95 times out of 100, some providing mathematical guarantees. The last part combines the first two for multi-modal uncertainty quantification, where it is shown that the range can become too narrow when the predictions of individual modalities disagree or when one type of information is missing, and how to correct it so that it remains reliable. These findings pave the way for scientific and commercial applications of multi-modal data, particularly in high-stakes domains that require trustworthy predictions.
Les humains utilisent diverses sources d’information pour prendre des décisions et peuvent exprimer l’incertitude de ces décisions. Ces différentes sources d’information (données tabulaires, images, descriptions textuelles, etc.) sont appelées modalités. Les modèles d’apprentissage automatique peuvent aussi bénéficier de la combinaison de ces modalités, et il est tout aussi utile d’exprimer à quel point leurs prédictions sont fiables. La première partie de la thèse mesure ce que les images et le texte apportent aux données tabulaires dans deux applications. Pour 11 000 annonces de location en Suisse, combiner les photos de l’appartement et les images satellites du quartier avec les informations traditionnellement utilisées, comme la surface et l’emplacement, diminue l’erreur moyenne de prévision du loyer d’environ un dixième. Pour les ouragans, où des vies et des biens sont en jeu, utiliser les rapports écrits par les prévisionnistes améliore les prévisions d’intensité un jour à l’avance. La deuxième partie introduit de nouvelles méthodes qui quantifient l’incertitude de tout modèle prédictif. Ces méthodes donnent une fourchette de valeurs plausibles au lieu d’un seul nombre, avec dans certains cas la garantie mathématique que la vraie valeur est contenue dans cette fourchette, par exemple, 95 fois sur 100. Dans la dernière partie, qui combine les deux premières, il est montré que la fourchette peut devenir trop étroite quand les prédictions des différentes modalités sont en désaccord ou qu’un type d’information manque, et comment la corriger pour qu’elle reste fiable. Ces résultats ouvrent la voie à des applications scientifiques et commerciales des données multimodales, surtout dans les domaines à forts enjeux qui nécessitent des prédictions fiables.
Papers
The thesis rests on five papers, four published and one preprint. The last two below are related papers that it refers to but that are not part of it.