Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory.
Saved in:
| Title: | Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory. |
|---|---|
| Authors: | Hu, Tien-Ling1,2 (AUTHOR) tienhu@iu.edu, Valdivia, Dubravka Svetina3 (AUTHOR) |
| Source: | Research in Higher Education. Dec2024, Vol. 65 Issue 8, p1965-1991. 27p. |
| Subject Terms: | *Item response theory, *Exploratory factor analysis, *Educational outcomes, Psychometrics, Quality factor |
| Abstract: | Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O'Donnell's (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement's (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE's HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students' undergraduate research experiences, thereby fostering their academic success. [ABSTRACT FROM AUTHOR] |
| Copyright of Research in Higher Education is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O'Donnell's (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement's (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE's HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students' undergraduate research experiences, thereby fostering their academic success. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 03610365 |
| DOI: | 10.1007/s11162-024-09814-6 |
