Planning Missing Data Designs for Human Ratings in Creativity Research: A Practical Guide
Saved in:
| Title: | Planning Missing Data Designs for Human Ratings in Creativity Research: A Practical Guide |
|---|---|
| Language: | English |
| Authors: | Boris Forthmann (ORCID |
| Source: | Creativity Research Journal. 2025 37(1):167-178. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 12 |
| Publication Date: | 2025 |
| Sponsoring Agency: | National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL) National Science Foundation (NSF), Division of Undergraduate Education (DUE) |
| Contract Number: | 1920653 2155070 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Creativity, Research, Researchers, Research Methodology, Psychometrics, Simulation, Measurement Techniques, Item Response Theory, Evaluators, Models, Matrices, Data, Computation, Cost Effectiveness |
| DOI: | 10.1080/10400419.2023.2250976 |
| ISSN: | 1040-0419 1532-6934 |
| Abstract: | Human ratings are ubiquitous in creativity research. Yet, the process of rating responses to creativity tasks -- typically several hundred or thousands of responses, per rater -- is often time-consuming and expensive. Planned missing data designs, where raters only rate a subset of the total number of responses, have been recently proposed as one possible solution to decrease overall rating time and monetary costs. However, researchers also need ratings that adhere to psychometric standards, such as a certain degree of reliability, and psychometric work with planned missing designs is currently lacking in the literature. In this work, we introduce how judge response theory and simulations can be used to fine-tune planning of missing data designs. We provide open code for the community and illustrate our proposed approach by a cost-effectiveness calculation based on a realistic example. We clearly show that fine-tuning helps to save time (to perform the ratings) and monetary costs, while simultaneously targeting expected levels of reliability. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1458414 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Human ratings are ubiquitous in creativity research. Yet, the process of rating responses to creativity tasks -- typically several hundred or thousands of responses, per rater -- is often time-consuming and expensive. Planned missing data designs, where raters only rate a subset of the total number of responses, have been recently proposed as one possible solution to decrease overall rating time and monetary costs. However, researchers also need ratings that adhere to psychometric standards, such as a certain degree of reliability, and psychometric work with planned missing designs is currently lacking in the literature. In this work, we introduce how judge response theory and simulations can be used to fine-tune planning of missing data designs. We provide open code for the community and illustrate our proposed approach by a cost-effectiveness calculation based on a realistic example. We clearly show that fine-tuning helps to save time (to perform the ratings) and monetary costs, while simultaneously targeting expected levels of reliability. |
|---|---|
| ISSN: | 1040-0419 1532-6934 |
| DOI: | 10.1080/10400419.2023.2250976 |
