Public PhD defense of Ilia Azizi

An Evaluation of Supervised Multi‑modal Learning
and Uncertainty Quantification for Tabular Data

PhD in Business Analytics, HEC, University of Lausanne

When
Tuesday 24 November 2026, 17:00 CET
Where
On site at Extranef 109, UNIL‑Dorigny (map), or online (info below)

Register

Registering takes a minute and helps plan the apéro. If you attend on site, you will receive a calendar invitation. If you join online, an email with the Teams link will follow.

How will you attend? (required)
Will you stay for the apéro? (required)
Data privacy, retention and protection

Your answers are stored by Netlify on servers in Frankfurt, with a copy in a private Google Sheet that only I can open. They are used only to plan the apéro and to email you the calendar invitation or the Teams link, sent through Brevo, an EU email service. The dietary field is optional. To filter spam, Cloudflare Turnstile checks your browser once you start filling in the form, and Cloudflare may use this to improve its bot detection; Netlify, Google and Cloudflare are US companies. Your answers are deleted after the defense. To change or remove them sooner, write to ilia.azizi@unil.ch.

Summary

  1. The thesis combines tabular data with high‑dimensional sources such as images and text, known as multi‑modal learning, and surrounds each prediction with an interval that holds the real value with some mathematical guarantees (e.g., 95 times in 100). Seven papers were authored or co‑authored during the PhD (five published, two preprints). Five form the core of the thesis; the other two are related.
  2. Part I measures what photos and text add when a table already describes each case. In predicting rental prices, apartment photos and satellite views cut a neural network’s error by about a tenth. For hurricane intensity one day ahead, the forecasters’ written discussions lower another network’s error.
  3. Part II replaces the single prediction with an interval. The first method, SEMF, builds such an interval around models like XGBoost that do not give one on their own. The second, CLEAR, learns how much of the width should come from noise in the data and how much from what the model does not know.
  4. Part III combines the two. An interval calibrated on complete data can be too narrow when the sources disagree, or when one is missing at prediction time. In the experiments, calibrating separately for cases where the sources strongly disagree, or for the set of sources present, helped recover intervals that hold the real value close to 95 times in 100.

Animations

Rents. On one test split, adding photos and satellite images cut the neural network’s mean absolute error on log rent by 11%, from 0.113 to 0.101. XGBoost on the table alone is slightly lower at 0.099, but the image network is best on RMSE and R² (Chapter 2).
Hurricanes. Forecasters’ notes, like the 2013 example shown, cut the neural network’s wind-speed error one day ahead by 7%. This is the mean of five runs on the 2022 and 2023 seasons. With the hybrid XGBoost models the gain is about 4%, and the official National Hurricane Center forecast is still more accurate (Chapter 3).
Intervals. SEMF gave better intervals than quantile regression on 10 of 11 benchmark datasets, both run with XGBoost at the same settings (Chapter 4). Both were calibrated to 95 times in 100. Better means more coverage per unit of width, counted in the rerun with the corrected code. With two other model types SEMF was better on 8 and 7 of 11. The rent example is made up.
Two kinds of uncertainty. CLEAR accounts for both the randomness in the data and the model’s own limits, which grow where past cases are few or models disagree. It had the lowest average quantile loss on 15 of 17 datasets, against three methods that each use only one kind of uncertainty. This is the main configuration of Chapter 5, with 10 random splits per dataset. The top plot is a sketch, not data.
Missing sources. In the hardest case studied, with the photo and name missing, intervals calibrated on complete data held the real age only 76 times in 100. Recalibrating for the sources present, without retraining, brought it back to 95 with intervals 2.25 times as wide. Results are the mean of five runs on IMDB-WIKI. There, age is predicted from a face photo, the person’s name and table columns (Chapter 6).

Abstract

Humans use various sources of information to make decisions and can convey the uncertainty associated with those decisions. These types of information (tabular data, images, textual descriptions, etc.) are called modalities. Machine learning models can similarly benefit from combining these modalities, and it is equally valuable to convey how trustworthy their predictions are. The first part of the thesis measures what images and text add to tabular data in two applications. First, for 11,000 rental listings in Switzerland, combining photos of the apartment and satellite images of the area with the information traditionally used, such as size and location, lowers the average rent prediction error by roughly a tenth. Second, for hurricanes, where lives and property are at stake, adding the reports written by forecasters improves intensity predictions a day ahead. The second part introduces new methods that quantify the uncertainty of any predictor. The goal is to showcase methods that produce a range of plausible values instead of a single number, with the aim of having this range contain the true outcome for instance 95 times out of 100, some providing mathematical guarantees. The last part combines the first two for multi-modal uncertainty quantification, where it is shown that the range can become too narrow when the predictions of individual modalities disagree or when one type of information is missing, and how to correct it so that it remains reliable. These findings pave the way for scientific and commercial applications of multi-modal data, particularly in high-stakes domains that require trustworthy predictions.

Papers

The thesis rests on five papers, four published and one preprint. The last two below are related papers that it refers to but that are not part of it.

Part I. Supervised multi-modal learning for tabular data

Ch. 2
Improving real estate rental estimations with visual data
BDCC 2022Azizi and Rudnytskyi, Big Data and Cognitive Computing 6(3)
Ch. 3
Quantifying the added value of human forecaster discussions for tropical cyclone intensity prediction
arXiv 2026Azizi, Tam, Boldi, Chavez-Demoulin and Beucler, preprint

Part II. Uncertainty quantification for tabular regression

Ch. 4
SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals
COPA 2025Azizi, Boldi and Chavez-Demoulin, PMLR 266
Ch. 5
CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk
ICLR 2026Azizi, Bodik, Heiss and Yu

Part III. Bridging multi-modality and uncertainty

Ch. 6
Conformal calibration for multi-modal regression with missing modalities
COPA 2026Azizi, PMLR 329

Related papers outside the thesis

Matchability-aware conformal prediction for open-ended language model generation
NeurIPS 2026Azizi. Extends CLEAR to text generated by language models.
TCBench: A benchmark for tropical cyclone track and intensity forecasting at the global scale
arXiv 2026Gomez, McGraw, Ganesh S., Tam, Azizi et al., preprint. Its storm identifiers are used in Chapter 3.