Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment.
Saved in:
| Title: | Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment. |
|---|---|
| Authors: | Lin, Zirou, Yan, Hanbing, Zhao, Li |
| Source: | Journal of Computer Assisted Learning. Aug2024, Vol. 40 Issue 4, p1535-1551. 17p. |
| Subjects: | High schools, Research funding, Affinity groups, Educational outcomes, High school students, Educational tests & measurements, Descriptive statistics, Teachers, Middle school students, Deep learning, Online education, Artificial neural networks, Web development, Automation, Computer assisted instruction, Short-term memory, Middle schools, Computer assisted testing (Education) |
| Geographic Terms: | China |
| Abstract: | Background: Peer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning. However, with the emergence of numerical grades and textual feedback generated by peers, it is necessary to detect the reliability of the large amount of peer assessment data, and then develop an effective automated grading model to analyse the data and predict learners' learning results. Objectives: The present study aimed to propose an automated grading model with reliability detection. Methods: A total of 109,327 instances of peer assessment from a large‐scale teacher online learning program were tested in the experiments. The reliability detection approach included three steps: recurrent convolutional neural networks (RCNN) was used to detect grade consistency, bidirectional encoder representations from transformers (BERT) was used to detect text originality, and long short‐term memory (LSTM) was used to detect grade‐text consistency. Furthermore, the automated grading was designed with the BERT‐RCNN model. Results and Conclusions: The effectiveness of the automated grading model with reliability detection was shown. For reliability detection, RCNN performed best in detecting grade consistency with an accuracy rate of 0.889, BERT performed best in detecting text originality with an improvement of 4.47% compared to the benchmark model, and LSTM performed best with an accuracy rate of 0.883. Moreover, the automated grading model with reliability detection achieved good performance, with an accuracy rate of 0.89. Compared to the absence of reliability detection, it increased by 12.1%. Implications: The results strongly suggest that the automated grading model with reliability detection for large‐scale peer assessment is effective, with the following implications: (1) The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results. (2) This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. (3) This solution could shift educational institutions from labour‐intensive grading procedures to a more efficient educational assessment pattern, allowing for more investment in supporting instructors and learners to improve the quality of peer feedback. Lay Description: What is already known about this topic: Peer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning.Issues such as disagreement between peer assessors, rough assessment, and plagiarism in large‐scale online learning can decrease peer assessment reliabilityIncorporating extensive data into a training model may result in grading uncertainties. What this paper adds: Detecting the peer assessment reliability before grading is essential in the context of large‐scale online learning.This study aimed to propose and validate an automated grading model with reliability detection for the large‐scale online peer assessment, which will help improve the effectiveness of automated grading, combining the advantages of computer technology and human expertise. Implications for practice and/or policy: The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results.This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Computer Assisted Learning is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Psychology and Behavioral Sciences Collection |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: pbh DbLabel: Psychology and Behavioral Sciences Collection An: 178531903 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Lin%2C+Zirou%22">Lin, Zirou</searchLink><br /><searchLink fieldCode="AR" term="%22Yan%2C+Hanbing%22">Yan, Hanbing</searchLink><br /><searchLink fieldCode="AR" term="%22Zhao%2C+Li%22">Zhao, Li</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Journal+of+Computer+Assisted+Learning%22">Journal of Computer Assisted Learning</searchLink>. Aug2024, Vol. 40 Issue 4, p1535-1551. 17p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22High+schools%22">High schools</searchLink><br /><searchLink fieldCode="DE" term="%22Research+funding%22">Research funding</searchLink><br /><searchLink fieldCode="DE" term="%22Affinity+groups%22">Affinity groups</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+outcomes%22">Educational outcomes</searchLink><br /><searchLink fieldCode="DE" term="%22High+school+students%22">High school students</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+tests+%26+measurements%22">Educational tests & measurements</searchLink><br /><searchLink fieldCode="DE" term="%22Descriptive+statistics%22">Descriptive statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Teachers%22">Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+school+students%22">Middle school students</searchLink><br /><searchLink fieldCode="DE" term="%22Deep+learning%22">Deep learning</searchLink><br /><searchLink fieldCode="DE" term="%22Online+education%22">Online education</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+neural+networks%22">Artificial neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Web+development%22">Web development</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+assisted+instruction%22">Computer assisted instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Short-term+memory%22">Short-term memory</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+schools%22">Middle schools</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+assisted+testing+%28Education%29%22">Computer assisted testing (Education)</searchLink> – Name: SubjectGeographic Label: Geographic Terms Group: Su Data: <searchLink fieldCode="DE" term="%22China%22">China</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Background: Peer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning. However, with the emergence of numerical grades and textual feedback generated by peers, it is necessary to detect the reliability of the large amount of peer assessment data, and then develop an effective automated grading model to analyse the data and predict learners' learning results. Objectives: The present study aimed to propose an automated grading model with reliability detection. Methods: A total of 109,327 instances of peer assessment from a large‐scale teacher online learning program were tested in the experiments. The reliability detection approach included three steps: recurrent convolutional neural networks (RCNN) was used to detect grade consistency, bidirectional encoder representations from transformers (BERT) was used to detect text originality, and long short‐term memory (LSTM) was used to detect grade‐text consistency. Furthermore, the automated grading was designed with the BERT‐RCNN model. Results and Conclusions: The effectiveness of the automated grading model with reliability detection was shown. For reliability detection, RCNN performed best in detecting grade consistency with an accuracy rate of 0.889, BERT performed best in detecting text originality with an improvement of 4.47% compared to the benchmark model, and LSTM performed best with an accuracy rate of 0.883. Moreover, the automated grading model with reliability detection achieved good performance, with an accuracy rate of 0.89. Compared to the absence of reliability detection, it increased by 12.1%. Implications: The results strongly suggest that the automated grading model with reliability detection for large‐scale peer assessment is effective, with the following implications: (1) The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results. (2) This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. (3) This solution could shift educational institutions from labour‐intensive grading procedures to a more efficient educational assessment pattern, allowing for more investment in supporting instructors and learners to improve the quality of peer feedback. Lay Description: What is already known about this topic: Peer assessment has played an important role in large‐scale online learning, as it helps promote the effectiveness of learners' online learning.Issues such as disagreement between peer assessors, rough assessment, and plagiarism in large‐scale online learning can decrease peer assessment reliabilityIncorporating extensive data into a training model may result in grading uncertainties. What this paper adds: Detecting the peer assessment reliability before grading is essential in the context of large‐scale online learning.This study aimed to propose and validate an automated grading model with reliability detection for the large‐scale online peer assessment, which will help improve the effectiveness of automated grading, combining the advantages of computer technology and human expertise. Implications for practice and/or policy: The introduction of reliability detection is necessary to help filter out low reliability data in peer assessment, thus promoting effective automated grading results.This solution could assist assessors in adjusting the exclusion threshold of peer assessment reliability, providing a controllable automated grading tool to reducing manual workload with high quality. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Journal of Computer Assisted Learning is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=178531903 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1111/jcal.12970 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 17 StartPage: 1535 Subjects: – SubjectFull: High schools Type: general – SubjectFull: Research funding Type: general – SubjectFull: Affinity groups Type: general – SubjectFull: Educational outcomes Type: general – SubjectFull: High school students Type: general – SubjectFull: Educational tests & measurements Type: general – SubjectFull: Descriptive statistics Type: general – SubjectFull: Teachers Type: general – SubjectFull: Middle school students Type: general – SubjectFull: Deep learning Type: general – SubjectFull: Online education Type: general – SubjectFull: Artificial neural networks Type: general – SubjectFull: Web development Type: general – SubjectFull: Automation Type: general – SubjectFull: Computer assisted instruction Type: general – SubjectFull: Short-term memory Type: general – SubjectFull: Middle schools Type: general – SubjectFull: Computer assisted testing (Education) Type: general – SubjectFull: China Type: general Titles: – TitleFull: Exploring an effective automated grading model with reliability detection for large‐scale online peer assessment. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Lin, Zirou – PersonEntity: Name: NameFull: Yan, Hanbing – PersonEntity: Name: NameFull: Zhao, Li IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 08 Text: Aug2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 02664909 Numbering: – Type: volume Value: 40 – Type: issue Value: 4 Titles: – TitleFull: Journal of Computer Assisted Learning Type: main |
| ResultId | 1 |
