View in EDS

Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)

Saved in:
Bibliographic Details
Title: Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)
Language: English
Authors: Bell, Stephen H. (ORCID 0000-0002-2286-5887), Stapleton, David C. (ORCID 0000-0001-9147-3046), Wood, Michelle, Gubits, Daniel
Source: American Journal of Evaluation. Mar 2023 44(1):118-132.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 15
Publication Date: 2023
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Public Policy, Policy Formation, Federal Programs, Social Services, Disabilities, Insurance, Research Design, Statistical Analysis, Costs, Volunteers, Program Effectiveness, Evidence, Program Evaluation
DOI: 10.1177/10982140211006786
ISSN: 1098-2140
1557-0878
Abstract: A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning--relative to costs--from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1374477
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1374477
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Bell%2C+Stephen+H%2E%22">Bell, Stephen H.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2286-5887">0000-0002-2286-5887</externalLink>)<br /><searchLink fieldCode="AR" term="%22Stapleton%2C+David+C%2E%22">Stapleton, David C.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9147-3046">0000-0001-9147-3046</externalLink>)<br /><searchLink fieldCode="AR" term="%22Wood%2C+Michelle%22">Wood, Michelle</searchLink><br /><searchLink fieldCode="AR" term="%22Gubits%2C+Daniel%22">Gubits, Daniel</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22American+Journal+of+Evaluation%22"><i>American Journal of Evaluation</i></searchLink>. Mar 2023 44(1):118-132.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Public+Policy%22">Public Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Policy+Formation%22">Policy Formation</searchLink><br /><searchLink fieldCode="DE" term="%22Federal+Programs%22">Federal Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Social+Services%22">Social Services</searchLink><br /><searchLink fieldCode="DE" term="%22Disabilities%22">Disabilities</searchLink><br /><searchLink fieldCode="DE" term="%22Insurance%22">Insurance</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Design%22">Research Design</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Costs%22">Costs</searchLink><br /><searchLink fieldCode="DE" term="%22Volunteers%22">Volunteers</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence%22">Evidence</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Evaluation%22">Program Evaluation</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/10982140211006786
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1098-2140<br />1557-0878
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower "proof-of-concept" evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning--relative to costs--from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1374477
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1374477
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/10982140211006786
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
        StartPage: 118
    Subjects:
      – SubjectFull: Public Policy
        Type: general
      – SubjectFull: Policy Formation
        Type: general
      – SubjectFull: Federal Programs
        Type: general
      – SubjectFull: Social Services
        Type: general
      – SubjectFull: Disabilities
        Type: general
      – SubjectFull: Insurance
        Type: general
      – SubjectFull: Research Design
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Costs
        Type: general
      – SubjectFull: Volunteers
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Evidence
        Type: general
      – SubjectFull: Program Evaluation
        Type: general
    Titles:
      – TitleFull: Embedding a Proof-of-Concept Test in an At-Scale National Policy Experiment: Greater Policy Learning but at What Cost to Statistical Power? The Social Security Administration's Benefit Offset National Demonstration (BOND)
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Bell, Stephen H.
      – PersonEntity:
          Name:
            NameFull: Stapleton, David C.
      – PersonEntity:
          Name:
            NameFull: Wood, Michelle
      – PersonEntity:
          Name:
            NameFull: Gubits, Daniel
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 1098-2140
            – Type: issn-electronic
              Value: 1557-0878
          Numbering:
            – Type: volume
              Value: 44
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: American Journal of Evaluation
              Type: main
ResultId 1