Partial identifiability and misspecification in inverse reinforcement learning.
Saved in:
| Title: | Partial identifiability and misspecification in inverse reinforcement learning. |
|---|---|
| Authors: | Skalse, Joar1 (AUTHOR) joar.mvs@gmail.com, Abate, Alessandro1 (AUTHOR) |
| Source: | Artificial Intelligence. Jul2026, Vol. 356, pN.PAG-N.PAG. 1p. |
| Subjects: | Reinforcement learning, Human behavior models, Quasi-metric spaces, Utility functions, Robust statistics, Mathematical analysis |
| Abstract: | • We present frameworks for reasoning about partial identifiability and misspecification in inverse reinforcement learning. • We introduce pseudometrics that quantify the differences between reward functions in an informative way. • We show that the partial identifiability of common behavioural models is unproblematic when there is no misspecification. • We show that many behavioural models are highly robust to certain forms of misspecification. • We show that many behavioural models can be highly sensitive to certain forms of misspecification. The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only partially identifiable , and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π , an IRL algorithm must have a behavioural model of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be misspecified , which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms. [ABSTRACT FROM AUTHOR] |
| Copyright of Artificial Intelligence is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 194045770 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Partial identifiability and misspecification in inverse reinforcement learning. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Skalse%2C+Joar%22">Skalse, Joar</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> joar.mvs@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Abate%2C+Alessandro%22">Abate, Alessandro</searchLink><relatesTo>1</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink>. Jul2026, Vol. 356, pN.PAG-N.PAG. 1p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Reinforcement+learning%22">Reinforcement learning</searchLink><br /><searchLink fieldCode="DE" term="%22Human+behavior+models%22">Human behavior models</searchLink><br /><searchLink fieldCode="DE" term="%22Quasi-metric+spaces%22">Quasi-metric spaces</searchLink><br /><searchLink fieldCode="DE" term="%22Utility+functions%22">Utility functions</searchLink><br /><searchLink fieldCode="DE" term="%22Robust+statistics%22">Robust statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+analysis%22">Mathematical analysis</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: • We present frameworks for reasoning about partial identifiability and misspecification in inverse reinforcement learning. • We introduce pseudometrics that quantify the differences between reward functions in an informative way. • We show that the partial identifiability of common behavioural models is unproblematic when there is no misspecification. • We show that many behavioural models are highly robust to certain forms of misspecification. • We show that many behavioural models can be highly sensitive to certain forms of misspecification. The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only partially identifiable , and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π , an IRL algorithm must have a behavioural model of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be misspecified , which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Artificial Intelligence is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=194045770 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.artint.2026.104525 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 1 StartPage: N.PAG Subjects: – SubjectFull: Reinforcement learning Type: general – SubjectFull: Human behavior models Type: general – SubjectFull: Quasi-metric spaces Type: general – SubjectFull: Utility functions Type: general – SubjectFull: Robust statistics Type: general – SubjectFull: Mathematical analysis Type: general Titles: – TitleFull: Partial identifiability and misspecification in inverse reinforcement learning. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Skalse, Joar – PersonEntity: Name: NameFull: Abate, Alessandro IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 07 Text: Jul2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 00043702 Numbering: – Type: volume Value: 356 Titles: – TitleFull: Artificial Intelligence Type: main |
| ResultId | 1 |
