Conformal Predictive Decision Making: A Comparative Study with Bayesian and Point-Predictive Methods

Paper i proceeding, 2025

In many real-world settings, machine learning predictions serve as intermediate outputs used to inform decision-making. However, quantifying and accounting for uncertainty in these decisions remains a fundamental challenge. Conformal Predictive Decision Making is a framework for decision-making under uncertainty that leverages Conformal Predictive Distributions to optimize outcomes over a specified utility function. In this work, we evaluate Conformal Predictive Decision Making on synthetic datasets in both online and inductive settings, and compare its performance to two alternative approaches: Bayesian Decision Theory and Point Predictive Decision Making. Online Conformal Predictive Decision Making showed signs of greater robustness than Bayesian Decision Theory and Point Predictive Decision Making in scenarios involving noisy data and skewed utility functions, suggesting it may be a suitable option in more complex settings. However, it generally performed worse than the two alternative methods. In contrast, inductive Conformal Predictive Decision Making consistently outperformed the alternatives. This, combined with its computational advantages, makes it a promising approach for larger real-world decision-making applications where well-calibrated uncertainty quantification is needed for robustness.