View in EDS

Partial identifiability and misspecification in inverse reinforcement learning.

Saved in:
Bibliographic Details
Title: Partial identifiability and misspecification in inverse reinforcement learning.
Authors: Skalse, Joar1 (AUTHOR) joar.mvs@gmail.com, Abate, Alessandro1 (AUTHOR)
Source: Artificial Intelligence. Jul2026, Vol. 356, pN.PAG-N.PAG. 1p.
Subjects: Reinforcement learning, Human behavior models, Quasi-metric spaces, Utility functions, Robust statistics, Mathematical analysis
Abstract: • We present frameworks for reasoning about partial identifiability and misspecification in inverse reinforcement learning. • We introduce pseudometrics that quantify the differences between reward functions in an informative way. • We show that the partial identifiability of common behavioural models is unproblematic when there is no misspecification. • We show that many behavioural models are highly robust to certain forms of misspecification. • We show that many behavioural models can be highly sensitive to certain forms of misspecification. The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only partially identifiable , and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π , an IRL algorithm must have a behavioural model of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be misspecified , which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms. [ABSTRACT FROM AUTHOR]
Copyright of Artificial Intelligence is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:• We present frameworks for reasoning about partial identifiability and misspecification in inverse reinforcement learning. • We introduce pseudometrics that quantify the differences between reward functions in an informative way. • We show that the partial identifiability of common behavioural models is unproblematic when there is no misspecification. • We show that many behavioural models are highly robust to certain forms of misspecification. • We show that many behavioural models can be highly sensitive to certain forms of misspecification. The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only partially identifiable , and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π , an IRL algorithm must have a behavioural model of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be misspecified , which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms. [ABSTRACT FROM AUTHOR]
ISSN:00043702
DOI:10.1016/j.artint.2026.104525