Confidence can be compelling, but in AI systems, it can also mislead. The most advanced reasoning models today often express unwavering certainty in their answers, regardless of accuracy. Researchers at MIT’s CSAIL have identified a flaw in the training of these models that leads to overconfidence and devised a solution that retains accuracy. The new method, RLCR (Reinforcement Learning with Calibration Rewards), enables models to generate calibrated confidence estimates with their responses. This approach involves the model evaluating its uncertainty and providing a confidence score alongside its answer.<\/p>\n
Experiments demonstrated that RLCR reduced calibration error by up to 90 percent while maintaining or improving accuracy, even on tasks the model had not been specifically trained for. The findings will be presented at the International Conference on Learning Representations. The issue arises from the reinforcement learning methods that reward models solely for correct answers without considering the process or certainty of arriving at those answers. This can lead models to confidently provide answers even when unsure, which is problematic in fields like medicine or finance where decisions rely on AI outputs.<\/p>\n
According to Mehul Damani, an MIT PhD student and co-lead author, the standard training method lacks incentives for models to express uncertainty. RLCR addresses this by incorporating a Brier score into the reward function, penalizing discrepancies between stated confidence and actual accuracy. During training, models learn to assess both the problem and their uncertainty, resulting in simultaneous generation of answers and confidence estimates. Incorrect confident answers and overly uncertain correct ones are both penalized.<\/p>\n
The math confirms that this reward structure ensures models are accurate and well-calibrated. Tests on a 7-billion-parameter model across various benchmarks, including unfamiliar datasets, showed that RLCR significantly improved calibration without sacrificing accuracy. Unlike standard RL training, which worsens calibration, RLCR reversed this effect and outperformed post-hoc methods that assign confidence scores after the fact. MIT PhD student and co-lead author Isha Puri noted that typical RL training not only fails to aid calibration but also worsens it.<\/p>\n
RLCR\u2019s confidence estimates proved beneficial during inference, enhancing accuracy and calibration by selecting answers based on self-reported confidence. The researchers also discovered that models reasoning about uncertainty improved classifier performance, especially in smaller models. This self-reflective reasoning offers valuable information. The research team included Stewart Slocum, Idan Shenfeld, Leshem Choshen, and senior authors Jacob Andreas and Yoon Kim, along with Damani and Puri.<\/p>\n