Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)
Saved in:
| Title: | Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND) |
|---|---|
| Language: | English |
| Authors: | Bell, Stephen H. (ORCID |
| Source: | American Journal of Evaluation. Mar 2023 44(1):118-132. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 15 |
| Publication Date: | 2023 |
| Document Type: | Journal Articles Reports - Descriptive |
| Descriptors: | Public Policy, Policy Formation, Federal Programs, Social Services, Disabilities, Insurance, Research Design, Statistical Analysis, Costs, Volunteers, Program Effectiveness, Evidence, Program Evaluation |
| DOI: | 10.1177/10982140211006786 |
| ISSN: | 1098-2140 1557-0878 |
| Abstract: | A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning--relative to costs--from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment. |
| Abstractor: | As Provided |
| Entry Date: | 2023 |
| Accession Number: | EJ1374477 |
| Database: | ERIC |
| Abstract: | A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning--relative to costs--from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment. |
|---|---|
| ISSN: | 1098-2140 1557-0878 |
| DOI: | 10.1177/10982140211006786 |
