View in EDS

Neural Networks or Linguistic Features? -- Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits among L1- and L2-Learners' Argumentative Essays

Saved in:
Bibliographic Details
Title: Neural Networks or Linguistic Features? -- Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits among L1- and L2-Learners' Argumentative Essays
Language: English
Authors: Julian F. Lohmann (ORCID 0000-0002-5864-9692), Fynn Junge (ORCID 0009-0009-4834-8325), Jens Möller (ORCID 0000-0003-1767-5859), Johanna Fleckenstein (ORCID 0000-0003-4488-1455), Ruth Trüb (ORCID 0000-0002-9915-8611), Stefan Keller (ORCID 0000-0003-0115-5314), Thorben Jansen (ORCID 0000-0001-9714-6505), Andrea Horbach (ORCID 0009-0004-3680-3304)
Source: International Journal of Artificial Intelligence in Education. 2025 35(3):1178-1217.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 40
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Artificial Intelligence, Natural Language Processing, Essays, Writing Evaluation, Native Speakers, English Learners, Second Language Learning, Distinctive Features (Language)
DOI: 10.1007/s40593-024-00426-w
ISSN: 1560-4292
1560-4306
Abstract: Recent investigations in automated essay scoring research imply that hybrid models, which combine feature engineering and the powerful tools of deep neural networks (DNNs), reach state-of-the-art performance. However, most of these findings are from holistic scoring tasks. In the present study, we use a total of four prompts from two different corpora consisting of both L1 and L2 learner essays annotated with trait scores (e.g., content, organization, and language quality). In our main experiments, we compare three variants of trait-specific models using different inputs: (1) models based on 220 linguistic features, (2) models using essay-level contextual embeddings from the distilled version of the pre-trained transformer BERT (DistilBERT), and (3) a hybrid model using both types of features. Results imply that when trait-specific models are trained based on a single resource, the feature-based models slightly outperform the embedding-based models. These differences are most prominent for the organization traits. The hybrid models outperform the single-resource models, indicating that linguistic features and embeddings indeed capture partially different aspects relevant for the assessment of essay traits. To gain more insights into the interplay between both feature types, we run addition and ablation tests for individual feature groups. Trait-specific addition tests across prompts indicate that the embedding-based models can most consistently be enhanced in content assessment when combined with morphological complexity features. Most consistent performance gains in the organization traits are achieved when embeddings are combined with length features, and most consistent performance gains in the assessment of the language traits when combined with lexical complexity, error, and occurrence features. Cross-prompt scoring again reveals slight advantages for the feature-based models.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1488317
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1488317
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Neural Networks or Linguistic Features? -- Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits among L1- and L2-Learners' Argumentative Essays
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Julian+F%2E+Lohmann%22">Julian F. Lohmann</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-5864-9692">0000-0002-5864-9692</externalLink>)<br /><searchLink fieldCode="AR" term="%22Fynn+Junge%22">Fynn Junge</searchLink> (ORCID <externalLink term="http://orcid.org/0009-0009-4834-8325">0009-0009-4834-8325</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jens+Möller%22">Jens Möller</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-1767-5859">0000-0003-1767-5859</externalLink>)<br /><searchLink fieldCode="AR" term="%22Johanna+Fleckenstein%22">Johanna Fleckenstein</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-4488-1455">0000-0003-4488-1455</externalLink>)<br /><searchLink fieldCode="AR" term="%22Ruth+Trüb%22">Ruth Trüb</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-9915-8611">0000-0002-9915-8611</externalLink>)<br /><searchLink fieldCode="AR" term="%22Stefan+Keller%22">Stefan Keller</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-0115-5314">0000-0003-0115-5314</externalLink>)<br /><searchLink fieldCode="AR" term="%22Thorben+Jansen%22">Thorben Jansen</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-9714-6505">0000-0001-9714-6505</externalLink>)<br /><searchLink fieldCode="AR" term="%22Andrea+Horbach%22">Andrea Horbach</searchLink> (ORCID <externalLink term="http://orcid.org/0009-0004-3680-3304">0009-0004-3680-3304</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Artificial+Intelligence+in+Education%22"><i>International Journal of Artificial Intelligence in Education</i></searchLink>. 2025 35(3):1178-1217.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 40
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Native+Speakers%22">Native Speakers</searchLink><br /><searchLink fieldCode="DE" term="%22English+Learners%22">English Learners</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Distinctive+Features+%28Language%29%22">Distinctive Features (Language)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s40593-024-00426-w
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1560-4292<br />1560-4306
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Recent investigations in automated essay scoring research imply that hybrid models, which combine feature engineering and the powerful tools of deep neural networks (DNNs), reach state-of-the-art performance. However, most of these findings are from holistic scoring tasks. In the present study, we use a total of four prompts from two different corpora consisting of both L1 and L2 learner essays annotated with trait scores (e.g., content, organization, and language quality). In our main experiments, we compare three variants of trait-specific models using different inputs: (1) models based on 220 linguistic features, (2) models using essay-level contextual embeddings from the distilled version of the pre-trained transformer BERT (DistilBERT), and (3) a hybrid model using both types of features. Results imply that when trait-specific models are trained based on a single resource, the feature-based models slightly outperform the embedding-based models. These differences are most prominent for the organization traits. The hybrid models outperform the single-resource models, indicating that linguistic features and embeddings indeed capture partially different aspects relevant for the assessment of essay traits. To gain more insights into the interplay between both feature types, we run addition and ablation tests for individual feature groups. Trait-specific addition tests across prompts indicate that the embedding-based models can most consistently be enhanced in content assessment when combined with morphological complexity features. Most consistent performance gains in the organization traits are achieved when embeddings are combined with length features, and most consistent performance gains in the assessment of the language traits when combined with lexical complexity, error, and occurrence features. Cross-prompt scoring again reveals slight advantages for the feature-based models.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1488317
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1488317
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s40593-024-00426-w
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 40
        StartPage: 1178
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Native Speakers
        Type: general
      – SubjectFull: English Learners
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Distinctive Features (Language)
        Type: general
    Titles:
      – TitleFull: Neural Networks or Linguistic Features? -- Comparing Different Machine-Learning Approaches for Automated Assessment of Text Quality Traits among L1- and L2-Learners' Argumentative Essays
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Julian F. Lohmann
      – PersonEntity:
          Name:
            NameFull: Fynn Junge
      – PersonEntity:
          Name:
            NameFull: Jens Möller
      – PersonEntity:
          Name:
            NameFull: Johanna Fleckenstein
      – PersonEntity:
          Name:
            NameFull: Ruth Trüb
      – PersonEntity:
          Name:
            NameFull: Stefan Keller
      – PersonEntity:
          Name:
            NameFull: Thorben Jansen
      – PersonEntity:
          Name:
            NameFull: Andrea Horbach
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1560-4292
            – Type: issn-electronic
              Value: 1560-4306
          Numbering:
            – Type: volume
              Value: 35
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: International Journal of Artificial Intelligence in Education
              Type: main
ResultId 1