PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments.
Saved in:
| Title: | PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments. |
|---|---|
| Authors: | Lev, Omer1 (AUTHOR) omerlev@bgu.ac.il, Mattei, Nicholas2 (AUTHOR) nsmattei@tulane.edu, Turrini, Paolo3 (AUTHOR) p.turrini@warwick.ac.uk, Zhydkov, Stanislav1,4 (AUTHOR) s.zhydkov@warwick.ac.uk |
| Source: | Artificial Intelligence. Mar2023, Vol. 316, pN.PAG-N.PAG. 1p. |
| Subjects: | Algorithms, Computer simulation, Peers |
| Abstract: | In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem of impartial (or strategyproof) peer selection – how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm, PeerNomination , that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall of PeerNomination and prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness of PeerNomination to various levels of noise in the reviews. In order to maintain good performance under such conditions, we extend PeerNomination by using weights for reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance.1 [ABSTRACT FROM AUTHOR] |
| Copyright of Artificial Intelligence is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 161627663 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Lev%2C+Omer%22">Lev, Omer</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> omerlev@bgu.ac.il</i><br /><searchLink fieldCode="AR" term="%22Mattei%2C+Nicholas%22">Mattei, Nicholas</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> nsmattei@tulane.edu</i><br /><searchLink fieldCode="AR" term="%22Turrini%2C+Paolo%22">Turrini, Paolo</searchLink><relatesTo>3</relatesTo> (AUTHOR)<i> p.turrini@warwick.ac.uk</i><br /><searchLink fieldCode="AR" term="%22Zhydkov%2C+Stanislav%22">Zhydkov, Stanislav</searchLink><relatesTo>1,4</relatesTo> (AUTHOR)<i> s.zhydkov@warwick.ac.uk</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink>. Mar2023, Vol. 316, pN.PAG-N.PAG. 1p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+simulation%22">Computer simulation</searchLink><br /><searchLink fieldCode="DE" term="%22Peers%22">Peers</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem of impartial (or strategyproof) peer selection – how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm, PeerNomination , that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall of PeerNomination and prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness of PeerNomination to various levels of noise in the reviews. In order to maintain good performance under such conditions, we extend PeerNomination by using weights for reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance.1 [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Artificial Intelligence is the property of Elsevier B.V. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=161627663 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1016/j.artint.2022.103843 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 1 StartPage: N.PAG Subjects: – SubjectFull: Algorithms Type: general – SubjectFull: Computer simulation Type: general – SubjectFull: Peers Type: general Titles: – TitleFull: PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Lev, Omer – PersonEntity: Name: NameFull: Mattei, Nicholas – PersonEntity: Name: NameFull: Turrini, Paolo – PersonEntity: Name: NameFull: Zhydkov, Stanislav IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 03 Text: Mar2023 Type: published Y: 2023 Identifiers: – Type: issn-print Value: 00043702 Numbering: – Type: volume Value: 316 Titles: – TitleFull: Artificial Intelligence Type: main |
| ResultId | 1 |
