A DQF-MQM Evaluation of Machine Translation in English-Arabic Customs Law.
Saved in:
| Title: | A DQF-MQM Evaluation of Machine Translation in English-Arabic Customs Law. |
|---|---|
| Authors: | Damseh, Karam1, Ghassemiazghandi, Mozhgan1 mozhgan.ghassemi@gmail.com |
| Source: | Journal of Language Teaching & Research. May2026, Vol. 17 Issue 3, p1019-1027. 9p. |
| Subject Terms: | *Translating & interpreting, Tariff laws, Machine translating |
| Reviews & Products: | Google Translate (Web resource) |
| Abstract: | This study presents a corpus-based diagnostic evaluation of Google Translate's performance in translating the Revised Kyoto Convention (RKC) from English into Arabic. Building on the "Adequacy-Fluency Tradeoff" observed in recent literature, researchers employ an explanatory sequential triangulation design to assess whether high-resource NMT models can satisfy the strict compliance requirements of Customs law. The methodology applies the COMET metric to a stratified test suite of the RKC corpus (N = 1,000 segments) to establish a quantitative baseline. To diagnose potential metric discordance, researchers conducted a targeted human evaluation: a Likert-scale assessment of the highest-scoring segments (n = 100) to verify the "performance ceiling," followed by a granular DQF-MQM error analysis of the lowest-scoring segments (n = 200) conducted by a panel of Experts. The analysis identifies a profound discordance within the RKC corpus: while the system achieved a high mean COMET score of 87.70, the detailed analysis of the "performance floor" revealed 1,579 discrete errors. The resulting penalty score (1,292.5) exceeded the professional acceptance threshold for this text type by a factor of 5.7. These findings indicate that, within this specific legal corpus, Google Translate exhibits "specification-blindness," systematically failing to disambiguate polysemous "Terms of Art" required for international trade compliance. The research concludes that for the RKC, automated metrics do not yet serve as a reliable proxy for legal review, and that a Human-in-the-Loop framework remains an absolute prerequisite for the deployment of Google Translate in Customs administration. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Language Teaching & Research is the property of Academy Publication Co., LTD and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 193700239 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: A DQF-MQM Evaluation of Machine Translation in English-Arabic Customs Law. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Damseh%2C+Karam%22">Damseh, Karam</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Ghassemiazghandi%2C+Mozhgan%22">Ghassemiazghandi, Mozhgan</searchLink><relatesTo>1</relatesTo><i> mozhgan.ghassemi@gmail.com</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Journal+of+Language+Teaching+%26+Research%22">Journal of Language Teaching & Research</searchLink>. May2026, Vol. 17 Issue 3, p1019-1027. 9p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Translating+%26+interpreting%22">Translating & interpreting</searchLink><br /><searchLink fieldCode="DE" term="%22Tariff+laws%22">Tariff laws</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+translating%22">Machine translating</searchLink> – Name: SubjectProduct Label: Reviews & Products Group: Su Data: <searchLink fieldCode="PS" term="%22Google+Translate+%28Web+resource%29%22">Google Translate (Web resource)</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This study presents a corpus-based diagnostic evaluation of Google Translate's performance in translating the Revised Kyoto Convention (RKC) from English into Arabic. Building on the "Adequacy-Fluency Tradeoff" observed in recent literature, researchers employ an explanatory sequential triangulation design to assess whether high-resource NMT models can satisfy the strict compliance requirements of Customs law. The methodology applies the COMET metric to a stratified test suite of the RKC corpus (N = 1,000 segments) to establish a quantitative baseline. To diagnose potential metric discordance, researchers conducted a targeted human evaluation: a Likert-scale assessment of the highest-scoring segments (n = 100) to verify the "performance ceiling," followed by a granular DQF-MQM error analysis of the lowest-scoring segments (n = 200) conducted by a panel of Experts. The analysis identifies a profound discordance within the RKC corpus: while the system achieved a high mean COMET score of 87.70, the detailed analysis of the "performance floor" revealed 1,579 discrete errors. The resulting penalty score (1,292.5) exceeded the professional acceptance threshold for this text type by a factor of 5.7. These findings indicate that, within this specific legal corpus, Google Translate exhibits "specification-blindness," systematically failing to disambiguate polysemous "Terms of Art" required for international trade compliance. The research concludes that for the RKC, automated metrics do not yet serve as a reliable proxy for legal review, and that a Human-in-the-Loop framework remains an absolute prerequisite for the deployment of Google Translate in Customs administration. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Journal of Language Teaching & Research is the property of Academy Publication Co., LTD and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=193700239 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.17507/jltr.1703.26 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 9 StartPage: 1019 Subjects: – SubjectFull: Translating & interpreting Type: general – SubjectFull: Tariff laws Type: general – SubjectFull: Machine translating Type: general – SubjectFull: Google Translate (Web resource) Type: general Titles: – TitleFull: A DQF-MQM Evaluation of Machine Translation in English-Arabic Customs Law. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Damseh, Karam – PersonEntity: Name: NameFull: Ghassemiazghandi, Mozhgan IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: May2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 17984769 Numbering: – Type: volume Value: 17 – Type: issue Value: 3 Titles: – TitleFull: Journal of Language Teaching & Research Type: main |
| ResultId | 1 |
