View in EDS

An analysis of ChatGPT's performance in predicting countries and regions of origin from personal names.

Saved in:
Bibliographic Details
Title: An analysis of ChatGPT's performance in predicting countries and regions of origin from personal names.
Authors: Sebo, Paul1 (AUTHOR) paul.seboe@unige.ch
Source: Scientometrics. Mar2026, Vol. 131 Issue 3, p1731-1754. 24p.
Subjects: ChatGPT, Classification, Linguistic complexity, Language models, Demography, Predictive tests
Abstract: This study evaluated the performance of ChatGPT-4o in predicting the country and region of origin from personal names. A cross-sectional analysis was conducted using 2515 names from 63 countries of origin, generated by randomly combining popular first and last names from a public GitHub dataset. Predictions were obtained in two rounds, conducted five days apart (25 and 30 November 2024). Performance metrics included errorCoded (proportion of incorrect or non-classifications), errorCodedWithoutNA (proportion of incorrect classifications, excluding non-classifications), and naCoded (proportion of non-classifications). Cohen's Kappa was used to assess agreement between rounds. ChatGPT correctly identified the country of origin for 69.7% and 70.3% of names in the first and second rounds, respectively (errorCoded = 0.303 and 0.297). Regional predictions were more accurate, with 95.8% of names correctly assigned in both rounds (errorCoded = 0.042). No non-classifications occurred (naCoded = 0), and agreement between rounds was high (Cohen's Kappa: 0.841 for countries and 0.986 for regions). Performance varied by linguistic complexity, with high accuracy for linguistically distinct countries (e.g., Japan) and lower accuracy for countries with overlapping linguistic, cultural, or geographic features (e.g., English, French, and Spanish-speaking countries). In conclusion, ChatGPT-4o demonstrated high accuracy in predicting regions of origin and moderate performance at the country level when classifying names from a diverse international dataset. These findings highlight the potential and current limitations of large language models for name-based origin detection and suggest promising avenues for future applications in research, demography, and public health. [ABSTRACT FROM AUTHOR]
Copyright of Scientometrics is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Description
Abstract:This study evaluated the performance of ChatGPT-4o in predicting the country and region of origin from personal names. A cross-sectional analysis was conducted using 2515 names from 63 countries of origin, generated by randomly combining popular first and last names from a public GitHub dataset. Predictions were obtained in two rounds, conducted five days apart (25 and 30 November 2024). Performance metrics included errorCoded (proportion of incorrect or non-classifications), errorCodedWithoutNA (proportion of incorrect classifications, excluding non-classifications), and naCoded (proportion of non-classifications). Cohen's Kappa was used to assess agreement between rounds. ChatGPT correctly identified the country of origin for 69.7% and 70.3% of names in the first and second rounds, respectively (errorCoded = 0.303 and 0.297). Regional predictions were more accurate, with 95.8% of names correctly assigned in both rounds (errorCoded = 0.042). No non-classifications occurred (naCoded = 0), and agreement between rounds was high (Cohen's Kappa: 0.841 for countries and 0.986 for regions). Performance varied by linguistic complexity, with high accuracy for linguistically distinct countries (e.g., Japan) and lower accuracy for countries with overlapping linguistic, cultural, or geographic features (e.g., English, French, and Spanish-speaking countries). In conclusion, ChatGPT-4o demonstrated high accuracy in predicting regions of origin and moderate performance at the country level when classifying names from a diverse international dataset. These findings highlight the potential and current limitations of large language models for name-based origin detection and suggest promising avenues for future applications in research, demography, and public health. [ABSTRACT FROM AUTHOR]
ISSN:01389130
DOI:10.1007/s11192-026-05587-0