View in EDS HTML Full Text PDF Full Text

Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement.

Saved in:
Bibliographic Details
Title: Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement.
Authors: Alarfaj, Maher M.1 (AUTHOR) malarfaj@kfu.edu.sa
Source: Learned Publishing. Jan2026, Vol. 39 Issue 1, p1-8. 8p.
Subjects: Statistics, Uniformity, Evaluation methodology, Scholarly publishing, Peer review of students, Contradiction
Geographic Terms: Arab countries
Abstract: This paper investigates the inconsistencies present in peer review by analysing the evaluation patterns of reviewers involved in an educational award in the Arab Gulf Country States. A statistical approach was used to assess the degree of variation in scores assigned to 270 manuscripts reviewed by three different groups of reviewers. The study revealed significant differences in the evaluations, suggesting that at least two reviewers often showed discrepancies in their assessments despite using the standardised evaluation form. The observed discrepancies appear to reflect underlying complexities related to reviewer perspectives and evaluation standards, highlighting challenges in achieving uniformity across assessments. Additionally, it proposes a model to enhance peer review consistency, including methods for score adjustment and calibration to mitigate reviewer differences. The goal is to offer practical recommendations for improving the fairness, transparency and reliability of peer review systems, contributing to the ongoing development of academic publishing practices. For audiences beyond the academic community including publishers, editors and academic librarians, these findings show how practical statistical tools can strengthen peer review and build greater trust in academic publishing. Summary: Peer review often exhibits not able inconsistency, with considerable variation in how reviewers evaluate the same manuscript.Discrepancies may arise from variations in reviewer expertise, subjective biases, and differing expectations regarding research quality.Implementing structured and standardized review measures can help minimize subjectivity.Blending narrative comments with numerical scores supports evaluations that balance rigor with depth.Statistical calibration methods can help adjust reviewer scores to better reflect objective evaluation standards.Using anchor scores or incorporating a third reviewer's assessment can enhance comparability across reviewers and strengthen the reliability of the review process.The study's limitations—including the scope of the dataset, potential unexamined biases, and partial testing of calibration approaches—highlight important directions for future research. [ABSTRACT FROM AUTHOR]
Copyright of Learned Publishing is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 191182520
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Alarfaj%2C+Maher+M%2E%22">Alarfaj, Maher M.</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> malarfaj@kfu.edu.sa</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Learned+Publishing%22">Learned Publishing</searchLink>. Jan2026, Vol. 39 Issue 1, p1-8. 8p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Statistics%22">Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Uniformity%22">Uniformity</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+methodology%22">Evaluation methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Scholarly+publishing%22">Scholarly publishing</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+review+of+students%22">Peer review of students</searchLink><br /><searchLink fieldCode="DE" term="%22Contradiction%22">Contradiction</searchLink>
– Name: SubjectGeographic
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Arab+countries%22">Arab countries</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper investigates the inconsistencies present in peer review by analysing the evaluation patterns of reviewers involved in an educational award in the Arab Gulf Country States. A statistical approach was used to assess the degree of variation in scores assigned to 270 manuscripts reviewed by three different groups of reviewers. The study revealed significant differences in the evaluations, suggesting that at least two reviewers often showed discrepancies in their assessments despite using the standardised evaluation form. The observed discrepancies appear to reflect underlying complexities related to reviewer perspectives and evaluation standards, highlighting challenges in achieving uniformity across assessments. Additionally, it proposes a model to enhance peer review consistency, including methods for score adjustment and calibration to mitigate reviewer differences. The goal is to offer practical recommendations for improving the fairness, transparency and reliability of peer review systems, contributing to the ongoing development of academic publishing practices. For audiences beyond the academic community including publishers, editors and academic librarians, these findings show how practical statistical tools can strengthen peer review and build greater trust in academic publishing. Summary: Peer review often exhibits not able inconsistency, with considerable variation in how reviewers evaluate the same manuscript.Discrepancies may arise from variations in reviewer expertise, subjective biases, and differing expectations regarding research quality.Implementing structured and standardized review measures can help minimize subjectivity.Blending narrative comments with numerical scores supports evaluations that balance rigor with depth.Statistical calibration methods can help adjust reviewer scores to better reflect objective evaluation standards.Using anchor scores or incorporating a third reviewer's assessment can enhance comparability across reviewers and strengthen the reliability of the review process.The study's limitations—including the scope of the dataset, potential unexamined biases, and partial testing of calibration approaches—highlight important directions for future research. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Learned Publishing is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=191182520
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/leap.2033
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 8
        StartPage: 1
    Subjects:
      – SubjectFull: Statistics
        Type: general
      – SubjectFull: Uniformity
        Type: general
      – SubjectFull: Evaluation methodology
        Type: general
      – SubjectFull: Scholarly publishing
        Type: general
      – SubjectFull: Peer review of students
        Type: general
      – SubjectFull: Contradiction
        Type: general
      – SubjectFull: Arab countries
        Type: general
    Titles:
      – TitleFull: Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Alarfaj, Maher M.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan2026
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 09531513
          Numbering:
            – Type: volume
              Value: 39
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Learned Publishing
              Type: main
ResultId 1