View in EDS

Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques

Saved in:
Bibliographic Details
Title: Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques
Language: English
Authors: Ramachandran, Lakshmi, Gehringer, Edward F., Yadav, Ravi K.
Source: International Journal of Artificial Intelligence in Education. Sep 2017 27(3):534-581.
Availability: Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
Peer Reviewed: Y
Page Count: 48
Publication Date: 2017
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: 0942279
Document Type: Journal Articles
Reports - Research
Descriptors: Natural Language Processing, Peer Evaluation, Educational Quality, Meta Analysis, Automation, Computer Software, Feedback (Response), Authors, Writing for Publication, Plagiarism, Evaluation Methods
DOI: 10.1007/s40593-016-0132-x
ISSN: 1560-4292
Abstract: A "review" is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through "metareview" or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors' submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review's coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.
Abstractor: As Provided
Number of References: 67
Entry Date: 2017
Accession Number: EJ1148519
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1148519
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ramachandran%2C+Lakshmi%22">Ramachandran, Lakshmi</searchLink><br /><searchLink fieldCode="AR" term="%22Gehringer%2C+Edward+F%2E%22">Gehringer, Edward F.</searchLink><br /><searchLink fieldCode="AR" term="%22Yadav%2C+Ravi+K%2E%22">Yadav, Ravi K.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Artificial+Intelligence+in+Education%22"><i>International Journal of Artificial Intelligence in Education</i></searchLink>. Sep 2017 27(3):534-581.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: service-ny@springer.com; Web site: http://www.springerlink.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 48
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2017
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 0942279
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+Evaluation%22">Peer Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Quality%22">Educational Quality</searchLink><br /><searchLink fieldCode="DE" term="%22Meta+Analysis%22">Meta Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software%22">Computer Software</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Authors%22">Authors</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+for+Publication%22">Writing for Publication</searchLink><br /><searchLink fieldCode="DE" term="%22Plagiarism%22">Plagiarism</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s40593-016-0132-x
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1560-4292
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: A "review" is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through "metareview" or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors' submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review's coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 67
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2017
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1148519
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1148519
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s40593-016-0132-x
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 48
        StartPage: 534
    Subjects:
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Peer Evaluation
        Type: general
      – SubjectFull: Educational Quality
        Type: general
      – SubjectFull: Meta Analysis
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Computer Software
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Authors
        Type: general
      – SubjectFull: Writing for Publication
        Type: general
      – SubjectFull: Plagiarism
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
    Titles:
      – TitleFull: Automated Assessment of the Quality of Peer Reviews Using Natural Language Processing Techniques
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ramachandran, Lakshmi
      – PersonEntity:
          Name:
            NameFull: Gehringer, Edward F.
      – PersonEntity:
          Name:
            NameFull: Yadav, Ravi K.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2017
          Identifiers:
            – Type: issn-print
              Value: 1560-4292
          Numbering:
            – Type: volume
              Value: 27
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: International Journal of Artificial Intelligence in Education
              Type: main
ResultId 1