Heterogeneity in effect size estimates.
Saved in:
| Title: | Heterogeneity in effect size estimates. |
|---|---|
| Authors: | Holzmeister, Felix1 felix.holzmeister@uibk.ac.at, Johannesson, Magnus2, Böhm, Robert3,4, Dreber, Anna1,2, Huber, Jürgen5, Kirchler, Michael5 |
| Source: | Proceedings of the National Academy of Sciences of the United States of America. 8/6/2024, Vol. 121 Issue 32, p1-10. 10p. |
| Subjects: | Path analysis (Statistics), Error rates, Heterogeneity, Experimental design, Empirical research |
| Abstract: | A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies. [ABSTRACT FROM AUTHOR] |
| Copyright of Proceedings of the National Academy of Sciences of the United States of America is the property of National Academy of Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 178896008 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Heterogeneity in effect size estimates. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Holzmeister%2C+Felix%22">Holzmeister, Felix</searchLink><relatesTo>1</relatesTo><i> felix.holzmeister@uibk.ac.at</i><br /><searchLink fieldCode="AR" term="%22Johannesson%2C+Magnus%22">Johannesson, Magnus</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Böhm%2C+Robert%22">Böhm, Robert</searchLink><relatesTo>3,4</relatesTo><br /><searchLink fieldCode="AR" term="%22Dreber%2C+Anna%22">Dreber, Anna</searchLink><relatesTo>1,2</relatesTo><br /><searchLink fieldCode="AR" term="%22Huber%2C+Jürgen%22">Huber, Jürgen</searchLink><relatesTo>5</relatesTo><br /><searchLink fieldCode="AR" term="%22Kirchler%2C+Michael%22">Kirchler, Michael</searchLink><relatesTo>5</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America%22">Proceedings of the National Academy of Sciences of the United States of America</searchLink>. 8/6/2024, Vol. 121 Issue 32, p1-10. 10p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Path+analysis+%28Statistics%29%22">Path analysis (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Error+rates%22">Error rates</searchLink><br /><searchLink fieldCode="DE" term="%22Heterogeneity%22">Heterogeneity</searchLink><br /><searchLink fieldCode="DE" term="%22Experimental+design%22">Experimental design</searchLink><br /><searchLink fieldCode="DE" term="%22Empirical+research%22">Empirical research</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Proceedings of the National Academy of Sciences of the United States of America is the property of National Academy of Sciences and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=178896008 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1073/pnas.2403490121 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 10 StartPage: 1 Subjects: – SubjectFull: Path analysis (Statistics) Type: general – SubjectFull: Error rates Type: general – SubjectFull: Heterogeneity Type: general – SubjectFull: Experimental design Type: general – SubjectFull: Empirical research Type: general Titles: – TitleFull: Heterogeneity in effect size estimates. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Holzmeister, Felix – PersonEntity: Name: NameFull: Johannesson, Magnus – PersonEntity: Name: NameFull: Böhm, Robert – PersonEntity: Name: NameFull: Dreber, Anna – PersonEntity: Name: NameFull: Huber, Jürgen – PersonEntity: Name: NameFull: Kirchler, Michael IsPartOfRelationships: – BibEntity: Dates: – D: 06 M: 08 Text: 8/6/2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 00278424 Numbering: – Type: volume Value: 121 – Type: issue Value: 32 Titles: – TitleFull: Proceedings of the National Academy of Sciences of the United States of America Type: main |
| ResultId | 1 |
