Can Quasi-Experimental Evaluations That Rely on State Longitudinal Data Systems Replicate Experimental Results?
Saved in:
| Title: | Can Quasi-Experimental Evaluations That Rely on State Longitudinal Data Systems Replicate Experimental Results? |
|---|---|
| Language: | English |
| Authors: | Fatih Unlu, Douglas Lee Lauen (ORCID |
| Source: | Grantee Submission. 2021. |
| Peer Reviewed: | Y |
| Page Count: | 70 |
| Publication Date: | 2021 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305A150477 |
| Document Type: | Reports - Research |
| Education Level: | High Schools Secondary Education |
| Descriptors: | Quasiexperimental Design, Longitudinal Studies, Databases, Statistical Bias, Replication (Evaluation), Pretesting, Benchmarking, Scores, Comparative Analysis, High School Students, Dual Enrollment, Attendance |
| DOI: | 10.1002/pam.22295 |
| Abstract: | Do quasi-experimental (QE) studies conducted with baseline covariates that are typically available in the longitudinal administrative state databases yield unbiased effect estimates? This paper conducts a within-study comparison (WSC) study that compares experimental impacts of early college high school (ECHS) attendance with QE impacts drawn from the state and locales. We find that (1) QE models for outcomes with natural (matching) pretests replicated the randomized benchmarks quite well; (2) the replication bias is not sensitive to type of propensity score model or method; and (3) imposing locational restrictions (i.e., local matching) on the comparison students--specifically choosing them from among non-treatment students who came from the same feeder middle schools as the treatment students--does not decrease the QE bias; on the contrary, it performed worse than the models that did not impose this restriction for most outcomes. The first two findings are generally consistent with other education WSCs while the third one is not, suggesting that in cases where selection may be driven by individual-level factors, such as this one, local matching may yield biased treatment effect estimates by greatly reducing the pool of potential comparison units and distorting balance on unobservable confounders while prioritizing balance on observable factors. [This paper was published in "Journal of Policy Analysis and Management" v40 n2 p572-613 2021.] |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2024 |
| Accession Number: | ED643736 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Do quasi-experimental (QE) studies conducted with baseline covariates that are typically available in the longitudinal administrative state databases yield unbiased effect estimates? This paper conducts a within-study comparison (WSC) study that compares experimental impacts of early college high school (ECHS) attendance with QE impacts drawn from the state and locales. We find that (1) QE models for outcomes with natural (matching) pretests replicated the randomized benchmarks quite well; (2) the replication bias is not sensitive to type of propensity score model or method; and (3) imposing locational restrictions (i.e., local matching) on the comparison students--specifically choosing them from among non-treatment students who came from the same feeder middle schools as the treatment students--does not decrease the QE bias; on the contrary, it performed worse than the models that did not impose this restriction for most outcomes. The first two findings are generally consistent with other education WSCs while the third one is not, suggesting that in cases where selection may be driven by individual-level factors, such as this one, local matching may yield biased treatment effect estimates by greatly reducing the pool of potential comparison units and distorting balance on unobservable confounders while prioritizing balance on observable factors. [This paper was published in "Journal of Policy Analysis and Management" v40 n2 p572-613 2021.] |
|---|---|
| DOI: | 10.1002/pam.22295 |
