Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory.
Saved in:
| Title: | Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory. |
|---|---|
| Authors: | Hu, Tien-Ling1,2 (AUTHOR) tienhu@iu.edu, Valdivia, Dubravka Svetina3 (AUTHOR) |
| Source: | Research in Higher Education. Dec2024, Vol. 65 Issue 8, p1965-1991. 27p. |
| Subject Terms: | *Item response theory, *Exploratory factor analysis, *Educational outcomes, Psychometrics, Quality factor |
| Abstract: | Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O'Donnell's (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement's (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE's HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students' undergraduate research experiences, thereby fostering their academic success. [ABSTRACT FROM AUTHOR] |
| Copyright of Research in Higher Education is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 180628550 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Hu%2C+Tien-Ling%22">Hu, Tien-Ling</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)<i> tienhu@iu.edu</i><br /><searchLink fieldCode="AR" term="%22Valdivia%2C+Dubravka+Svetina%22">Valdivia, Dubravka Svetina</searchLink><relatesTo>3</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Research+in+Higher+Education%22">Research in Higher Education</searchLink>. Dec2024, Vol. 65 Issue 8, p1965-1991. 27p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Item+response+theory%22">Item response theory</searchLink><br />*<searchLink fieldCode="DE" term="%22Exploratory+factor+analysis%22">Exploratory factor analysis</searchLink><br />*<searchLink fieldCode="DE" term="%22Educational+outcomes%22">Educational outcomes</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Quality+factor%22">Quality factor</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O'Donnell's (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement's (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE's HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students' undergraduate research experiences, thereby fostering their academic success. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Research in Higher Education is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=180628550 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s11162-024-09814-6 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 27 StartPage: 1965 Subjects: – SubjectFull: Item response theory Type: general – SubjectFull: Exploratory factor analysis Type: general – SubjectFull: Educational outcomes Type: general – SubjectFull: Psychometrics Type: general – SubjectFull: Quality factor Type: general Titles: – TitleFull: Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Hu, Tien-Ling – PersonEntity: Name: NameFull: Valdivia, Dubravka Svetina IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 12 Text: Dec2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 03610365 Numbering: – Type: volume Value: 65 – Type: issue Value: 8 Titles: – TitleFull: Research in Higher Education Type: main |
| ResultId | 1 |
