Can ChatGPT Translate Like a Pro? A Pilot Benchmarking Study of English-Malay Translation Quality.
Saved in:
| Title: | Can ChatGPT Translate Like a Pro? A Pilot Benchmarking Study of English-Malay Translation Quality. |
|---|---|
| Authors: | SULAIMAN, M. ZAIN1 zain@ukm.edu.my, ZAINUDIN, INTAN SAFINAZ1, HAROON, HASLINA2 |
| Source: | 3L: Southeast Asian Journal of English Language Studies. Dec2025, Vol. 31 Issue 4, p259-278. 20p. |
| Subject Terms: | *Translating & interpreting, *Certification, ChatGPT, Professional standards, Machine translating, Malay language |
| Abstract: | Artificial intelligence (AI) tools such as ChatGPT have significantly advanced machine translation, yet their performance in low-resource language pairs, particularly English-Malay, lags behind. While existing studies have compared AI and human translation quality, most have relied on academic assessment frameworks, leaving a gap in evaluating AI translation through professional certification standards. From a professional standpoint, translation competence is most reliably assessed through formal certification frameworks that combine analytic rubrics, performance descriptors, and expert judgment. To determine whether AI systems can perform at a professional standard, they must be evaluated using the same criteria applied to human translators. This pilot study addresses that gap by benchmarking ChatGPT's English-Malay translation performance against a novice and a professional translator using the National Accreditation Authority for Translators and Interpreters (NAATI) Certified Translator examination framework. Thirteen professional raters from the Malaysian Translators Association assessed the translations based on Meaning Transfer, Textual Norms and Conventions, and Language Proficiency. Findings revealed a clear performance hierarchy--Professional Translator > ChatGPT > Novice Translator--indicating that while ChatGPT achieved near-professional competence in fluency and meaning accuracy, it remained limited in idiomatic precision and cultural adaptation. The study highlights ChatGPT's potential as an assistive tool for translation and training, while reaffirming the need for human oversight. It also validates the NAATI framework as a robust benchmark for evaluating AI translation quality. As AI models continue to evolve, future research involving larger translator samples and a wider range of language pairs is essential to evaluate ongoing progress and ensure the responsible integration of AI translation into professional practice. [ABSTRACT FROM AUTHOR] |
| Copyright of 3L: Southeast Asian Journal of English Language Studies is the property of 3L: Language, Linguistics, Literature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 190781655 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Can ChatGPT Translate Like a Pro? A Pilot Benchmarking Study of English-Malay Translation Quality. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22SULAIMAN%2C+M%2E+ZAIN%22">SULAIMAN, M. ZAIN</searchLink><relatesTo>1</relatesTo><i> zain@ukm.edu.my</i><br /><searchLink fieldCode="AR" term="%22ZAINUDIN%2C+INTAN+SAFINAZ%22">ZAINUDIN, INTAN SAFINAZ</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22HAROON%2C+HASLINA%22">HAROON, HASLINA</searchLink><relatesTo>2</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%223L%3A+Southeast+Asian+Journal+of+English+Language+Studies%22">3L: Southeast Asian Journal of English Language Studies</searchLink>. Dec2025, Vol. 31 Issue 4, p259-278. 20p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Translating+%26+interpreting%22">Translating & interpreting</searchLink><br />*<searchLink fieldCode="DE" term="%22Certification%22">Certification</searchLink><br /><searchLink fieldCode="DE" term="%22ChatGPT%22">ChatGPT</searchLink><br /><searchLink fieldCode="DE" term="%22Professional+standards%22">Professional standards</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+translating%22">Machine translating</searchLink><br /><searchLink fieldCode="DE" term="%22Malay+language%22">Malay language</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Artificial intelligence (AI) tools such as ChatGPT have significantly advanced machine translation, yet their performance in low-resource language pairs, particularly English-Malay, lags behind. While existing studies have compared AI and human translation quality, most have relied on academic assessment frameworks, leaving a gap in evaluating AI translation through professional certification standards. From a professional standpoint, translation competence is most reliably assessed through formal certification frameworks that combine analytic rubrics, performance descriptors, and expert judgment. To determine whether AI systems can perform at a professional standard, they must be evaluated using the same criteria applied to human translators. This pilot study addresses that gap by benchmarking ChatGPT's English-Malay translation performance against a novice and a professional translator using the National Accreditation Authority for Translators and Interpreters (NAATI) Certified Translator examination framework. Thirteen professional raters from the Malaysian Translators Association assessed the translations based on Meaning Transfer, Textual Norms and Conventions, and Language Proficiency. Findings revealed a clear performance hierarchy--Professional Translator > ChatGPT > Novice Translator--indicating that while ChatGPT achieved near-professional competence in fluency and meaning accuracy, it remained limited in idiomatic precision and cultural adaptation. The study highlights ChatGPT's potential as an assistive tool for translation and training, while reaffirming the need for human oversight. It also validates the NAATI framework as a robust benchmark for evaluating AI translation quality. As AI models continue to evolve, future research involving larger translator samples and a wider range of language pairs is essential to evaluate ongoing progress and ensure the responsible integration of AI translation into professional practice. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of 3L: Southeast Asian Journal of English Language Studies is the property of 3L: Language, Linguistics, Literature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=190781655 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.17576/3L-2025-3104-17 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 20 StartPage: 259 Subjects: – SubjectFull: Translating & interpreting Type: general – SubjectFull: Certification Type: general – SubjectFull: ChatGPT Type: general – SubjectFull: Professional standards Type: general – SubjectFull: Machine translating Type: general – SubjectFull: Malay language Type: general Titles: – TitleFull: Can ChatGPT Translate Like a Pro? A Pilot Benchmarking Study of English-Malay Translation Quality. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: SULAIMAN, M. ZAIN – PersonEntity: Name: NameFull: ZAINUDIN, INTAN SAFINAZ – PersonEntity: Name: NameFull: HAROON, HASLINA IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 12 Text: Dec2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 01285157 Numbering: – Type: volume Value: 31 – Type: issue Value: 4 Titles: – TitleFull: 3L: Southeast Asian Journal of English Language Studies Type: main |
| ResultId | 1 |
