View in EDS HTML Full Text PDF Full Text

Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study

Saved in:
Bibliographic Details
Title: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
Language: English
Authors: Sales, Adam C. (ORCID 0000-0003-0416-0610), Pane, John F. (ORCID 0000-0001-5155-2436)
Source: Journal of Research on Educational Effectiveness. 2021 14(1):241-269.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 29
Publication Date: 2021
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: 1420374
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Educational Technology, Use Studies, Randomized Controlled Trials, Mathematics Curriculum, Curriculum Evaluation, Algebra, Intelligent Tutoring Systems, Cues, Problem Solving, Statistical Analysis, Causal Models
DOI: 10.1080/19345747.2020.1823538
ISSN: 1934-5747
Abstract: Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.
Abstractor: As Provided
Entry Date: 2021
Accession Number: EJ1293487
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwG7R51gY-qZKZJJHPBTjHShAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDPWLutooVF1Ub94-6gIBEICBm2fgAdr_dWbx7U_rmUcL4QWFXK3Xyi3Ga9lMWOUwGDRn2HCdrDzzZ9wTqlCR5dcj5tNGYs0t0ojaa4QWM04d8NXutnnhKTyQCJ37bXf7SnaCTtXKCkqjd9t9K8ds_6kwajTHtKKS5kRiaoBZ9vVuz9XBjU0kD_Q-J0vQYRjuvB57aTfp3U2J2RYY-Ttq1r-_QGRo710jLQbI3-sQ
Text:
  Availability: 1
  Value: <anid>AN0149920084;[5ew9]01jan.21;2021Apr23.02:53;v2.2.500</anid> <title id="AN0149920084-1">Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study </title> <p>Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.</p> <p>Keywords: Causal inference; educational technology; mediation analysis; principal stratification; propensity</p> <hd id="AN0149920084-2">Introduction</hd> <p>From 2007 through 2010, the RAND corporation conducted one of the first effectiveness trials awarded through competitive grant programs sponsored by the US Department of Education's Institute of Education Sciences. The randomized controlled trial (RCT) was designed to estimate the effect of school-wide adoption of the Cognitive Tutor Algebra I (CTA1) curriculum, whose centerpiece is a computerized "tutor" that uses cognitive science principles to teach Algebra I (Anderson et al., [<reflink idref="bib3" id="ref1">3</reflink>]). The study (Pane et al., [<reflink idref="bib47" id="ref2">47</reflink>]) found no effects in its first year, but in the second year of implementation students in high schools randomized to the CTA1 condition outperformed the control group on the posttest by a fifth of a standard deviation. As educational technology (EdTech) booms, so do RCTs in the mold of the RAND CTA1 study. A recent systematic review (Escueta et al., [<reflink idref="bib11" id="ref3">11</reflink>]) cites 29 published reports of RCTs of "computer assisted learning" programs, all but one of which was published since 2001. We are aware of numerous other such RCTs planned or ongoing.</p> <p>RCTs are invaluable tools for estimating these programs' efficacy. However, the interventions they study are highly multifaceted and complex. Each classroom, and each student within those classrooms, may use an EdTech application in a different way; presumably, the application's efficacy depends on these usage decisions. Similarly, EdTech applications often include a number of optional features—to what extent do each of these drive the program's effectiveness? The data collected during typical education RCTs—data on treatment assignments, outcomes, a standard set of demographics, and possibly a pretest—offer limited options for studying treatment effect heterogeneity, and offer no information on how the intervention was implemented.</p> <p>However, with little extra effort, researchers studying EdTech can gather rich implementation data over the course of an RCT. For instance, the software administrators running the CTA1 RCT gathered computer log data for students in the treatment group. They assembled a dataset that recorded which problems each student worked, along with timestamps, the numbers of hints requested, and the number of errors committed for each problem. Analogous log data is (or may be) collected from other EdTech RCTs. As one example, RAND recently completed an efficacy study of a different algebra tutoring system, ALEKS, and gathered student log data in order to study implementation.</p> <p>Analysis of log data from EdTech RCTs could, in principle, lead to insights on the relationship between implementation and effectiveness. Using log data, researchers could investigate how the product was actually implemented, as well as what aspects of implementation correlate with higher (or lower) treatment effects, or which features of the program drive its effects. For instance, Sales et al. ([<reflink idref="bib62" id="ref4">62</reflink>]) and Sales and Pane ([<reflink idref="bib60" id="ref5">60</reflink>], [<reflink idref="bib58" id="ref6">58</reflink>]) investigated aspects of mastery learning with CTA1. An ongoing research project uses the ALEKS log data to study its instructional material: when students begin working on a particular topic in ALEKS, the system first presents instructional material, followed by problem solving activities, and then, if students are unsuccessful at solving the problems, more instructional material. Some students diligently read the initial instructional material while others skip it and dive directly into problem solving. Is skipping the material a productive strategy for students in terms of learning algebra, or should ALEKS be redesigned to prevent such skipping?</p> <p>Answering questions like these calls for new statistical tools, which must simultaneously surmount three challenges: first, implementation is not typically randomized, so the relationship between log data and outcomes is likely confounded. Second, log data is only gathered from students assigned to use the intervention, and not from the control group. Third, log datasets have a complex structure—often a large number of granular observations of several types gathered over the course of each student's engagement with the program.</p> <p>This article will use log data from the CTA1 study—specifically, data on students' use of hints within the software—to illustrate three approaches to these challenges. We will attempt to answer two related questions: first, what is the role of the availability of hints in driving the overall CTA1 treatment effect? Second, can requesting hints more often lead to a larger treatment effect? The first approach we consider discards the control group, and analyzes hint-requesting in the treatment group as an observational study, using a matching design (c.f. Rosenbaum, [<reflink idref="bib54" id="ref7">54</reflink>]). The second approach applies the framework of causal mediation analysis (Hong, [<reflink idref="bib27" id="ref8">27</reflink>]; Imai et al., [<reflink idref="bib29" id="ref9">29</reflink>]; VanderWeele, [<reflink idref="bib72" id="ref10">72</reflink>]) to estimate the role of hint-requesting in the overall CTA1 treatment effect. However, since students in the control group were unable to request hints, not all mediational estimands are identified, and estimation methods must be modified. Finally, we conduct a principal stratification analysis (Frangakis & Rubin, [<reflink idref="bib15" id="ref11">15</reflink>]) to learn if students requesting hints at different rates experienced different treatment effects.</p> <p>Our intention is to develop, demonstrate, and contrast these three approaches to modeling log data. A rich, and often polemic, literature surrounds principal stratification and mediation analysis (e.g., Mealli & Mattei, [<reflink idref="bib40" id="ref12">40</reflink>]; Pearl, [<reflink idref="bib50" id="ref13">50</reflink>]; Rubin, [<reflink idref="bib57" id="ref14">57</reflink>]; VanderWeele, [<reflink idref="bib70" id="ref15">70</reflink>], [<reflink idref="bib71" id="ref16">71</reflink>]). VanderWeele ([<reflink idref="bib69" id="ref17">69</reflink>]), Jo ([<reflink idref="bib32" id="ref18">32</reflink>]) and others have given conditions under which certain principal stratification and mediational estimands coincide. Our contribution will build on this literature in two ways: first, by providing a detailed demonstration and comparison of matching, mediation analysis, and principal stratification on the same dataset, and second by focusing on the applicability of these three methods in the specific context of studying implementation data from an randomized experiment studying educational technology.</p> <p>The following section will provide background for the case-study, describing the CTA1 curriculum, the RAND study, and our focus on hints. The next three sections will present the observational study, causal mediation, and principal stratification, respectively. As it turns out, these analyses appear to give contradictory results—the observational study and mediation analysis estimate a negative effect of requesting hints, while the principal stratification analysis finds that students who requested more hints may have experienced larger overall treatment effects. Section "Comparing Strategies" will contrast these findings (including a possible reconciliation), compare the three methods generally, and discuss how they may be applied to data from other educational technology RCTs. Section "Discussion: Causal Inference & Measurement" will conclude.</p> <p>The R and Stan code used for all of the analyses in the paper can be found in a github repository, https://github.com/adamSales/logDataCaseStudy.</p> <hd id="AN0149920084-3">Case Study: The Role of Hints in the Cognitive Tutor Effect</hd> <p></p> <hd id="AN0149920084-4">The Cognitive Tutor</hd> <p>CTA1 is one of a series of complete mathematics curricula developed by Carnegie Learning, Inc., which includes both textbook materials and an automated computer-based Cognitive Tutor (Anderson et al., [<reflink idref="bib4" id="ref19">4</reflink>]; Pane et al., [<reflink idref="bib47" id="ref20">47</reflink>]).</p> <p>The computerized tutor was originally built to test the ACT and ACT-R theories of cognition (Anderson, [<reflink idref="bib2" id="ref21">2</reflink>]). These are elaborate theories that describe, among other things, the necessary components of cognitive skills in, say, mathematics or computer programming, and the process of acquiring those skills. The Algebra I tutor guides students through a sequence of Algebra I problems nested in sections within units and organized by the specific sets of skills they require. Students move through the curriculum as the tutor's internal learning model determines that they have mastered the requisite skills.</p> <hd id="AN0149920084-5">The Role of Hints 1</hd> <p>Students using the Cognitive Tutor Algebra software work algebra problems on a computer. Each problem in the software is associated with a set of skills that the student must master in order to solve the problem. The software also specifies a "solution path" for each problem—a sequence of actions (such as "multiply both sides of the equation by 3") students must take in order to solve the problem. If there is more than one way to solve a problem, the software specifies multiple correct solution paths students may follow. As soon as a student errs in working through a problem—that is, departs from one of the solution paths—the software shows the student an error message. This feedback is tailored to the specific mistake and designed to guide the student back to a correct solution path. Similarly, when a student gets stuck and is unable to determine the next step in solving the problem, he or she can ask for a hint. Like the error feedback, these hints are tailored to the specific algebra skills corresponding to the next step on the solution path. Thus, hints and error feedback are crucial aspects of the tutor's pedagogy.</p> <p>There are good reasons to believe that the availability of hints—help on demand—may increase student learning, but there is also reason to be skeptical (see Aleven et al., [<reflink idref="bib1" id="ref22">1</reflink>], for an overview of the theory). First of all, as, e.g., Anderson et al. ([<reflink idref="bib4" id="ref23">4</reflink>]) points out, "if students never receive help of any sort, they are in danger of becoming permanently stuck on some problem" (p. 190). That is, hints give students who are stuck a way to continue working. Hints may also boost learning by "helping students identify relevant features in problems" (Aleven et al., [<reflink idref="bib1" id="ref24">1</reflink>], p. 6), that is, point students toward aspects of the problem that are "most important for the goals of learning" (Koedinger et al., [<reflink idref="bib35" id="ref25">35</reflink>], p. 782). Hints can help improve students' self-awareness and problem-solving strategy, such as by clarifying which skills they have and have not mastered.</p> <p>Most problems are associated with several hints, arranged in a sequence so that a student who remains stuck after one hint may request another. The last hint in the sequence will show a complete solution to the problem—essentially showing students a worked example, which can be an effective tool for learning (e.g., Sweller & Cooper, [<reflink idref="bib68" id="ref26">68</reflink>]).</p> <p>On the other hand, all of these theoretical mechanisms rely on students to play their part. In order for hints to be instructive, students need to reflect on their content in productive ways. This suggests that hints may be helpful for some students but not for others. In the same vein, hints need to be well-designed in order to be effective (McKendree, [<reflink idref="bib39" id="ref27">39</reflink>]), so some hints may be more helpful than others.</p> <p>Hints may even be harmful: as Koedinger and Aleven ([<reflink idref="bib34" id="ref28">34</reflink>]) put it (p. 241), "many lines of research and theory suggest the importance of ...withholding information from students so that they can exercise, test, or reason toward new knowledge on their own." Fred and Van Merriënboer ([<reflink idref="bib16" id="ref29">16</reflink>]) interpret this dilemma in terms of balancing "cognitive load," that is, optimally allocating a student's attention to the most relevant aspects of a problem.</p> <p>The empirical literature on the value of hints in intelligent tutors is mixed (see Goldin and Koedinger ([<reflink idref="bib20" id="ref30">20</reflink>]), for a summary). Anderson et al. ([<reflink idref="bib4" id="ref31">4</reflink>]) compared the performance of students randomly assigned to different versions of the Cognitive Tutor with different feedback structures and found that students who were able to request hints finished the lesson faster than those who were not, but could not detect an effect on learning. Singh et al. ([<reflink idref="bib66" id="ref32">66</reflink>]) used a similar design, comparing two versions of the ASSISTments intelligent tutor, one in which hints and immediate error messages were available, and one which did not provide feedback. Students in the feedback condition showed larger gains on a posttest, though it is unclear whether these gains were due to hints or error feedback. Beck et al. ([<reflink idref="bib6" id="ref33">6</reflink>]) and Goldin and Koedinger ([<reflink idref="bib20" id="ref34">20</reflink>]), did not randomize hint availability, but instead used statistical models to control for observed confounding. Both papers used students' subsequent performance within the tutor to estimate hint effects. Beck et al. ([<reflink idref="bib6" id="ref35">6</reflink>]) found conflicting results using different analytical methods to control for confounding. Goldin and Koedinger ([<reflink idref="bib20" id="ref36">20</reflink>]) investigated two sources of heterogeneity in the effect of a hint request: by student and by the "level" of the hint (the first, second, or final hint available for a given problem). They found that the effect of asking for a hint varies between students, but on average it is negative for the first two hints requested on a problem, but positive for the final hint.</p> <hd id="AN0149920084-6">Measuring Hint Usage in the CTA1 Data</hd> <p>Since the purpose of the CTA1 study was to estimate the overall effect of the CTA1 curriculum—including the associated textbook and recommended classroom practices—access to the full CTA1 curriculum was randomized, rather than any specific aspect of usage. Effectiveness was measured with a standardized posttest covering a broad range of Algebra I skills. In this regard, the CTA1 study is similar to other large-scale evaluations of educational technology which do not randomize usage but do include a posttest.</p> <p>The CTA1 study was of a much longer duration than any of the studies of hint effects we are aware of: students had access to the tutor for an entire school year before the posttest. This factor allows for much more data for each student than shorter studies, as well as more variability in the types of problems and contexts in which students may request hints.</p> <p>In order to align with our research focus, we narrow the type of hint data we model. Since our posttest measured Algebra I skills, we only considered worked problems from the Algebra I curriculum. Next, although Goldin and Koedinger ([<reflink idref="bib20" id="ref37">20</reflink>]) demonstrated the importance of hint level, our focus is on the overall average effect of requesting any hints; therefore, the variable measuring a student's hint request on a problem was set to one if the student requested any hints on that problem, and zero otherwise.</p> <p>Lastly, any observational study of hint usage must contend with the fact that students are unlikely to request hints on problems that they already know how to solve. Hence, any measure of hint usage is necessarily also a measure of student ability—students who know more algebra will almost certainly request fewer hints. This leads to two related problems: first, student ability confounds any statistical relationship between hint request frequency and posttest scores. Secondly, a student's decision to request a hint implies both that she found the problem at least somewhat difficult, and that, in this case, she responded to that difficulty by requesting a hint. Our interest is in the second implication, not the first. Conversely, the significance of a student's decision to forgo a hint depends in part on how challenging he or she finds the problem. Hint request frequency combines students' hint requests (or non-requests) from both problems they find challenging and trivial, so is a poor measure of students' disposition to use the hint feature.</p> <p>Ideally, we would solve this problem by only including data on problem-student pairs in which the problem was challenging to the student. Of course, no direct measure of problem challenge is available. Instead, we considered a problem to be challenging if the student either requested a hint or made an error on that problem. Then, let</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover></mrow><mi>i</mi></msub><mo>=</mo><mfrac><mrow><mo>\#</mo><mtext>problems</mtext><mi mathvariant="normal" /><mtext>on</mtext><mi mathvariant="normal" /><mtext>which</mtext><mi mathvariant="normal" /><mi>i</mi><mi mathvariant="normal" /><mtext>requested</mtext><mtext> a </mtext><mtext>hint</mtext></mrow><mrow><mo>\#</mo><mtext>problems\ on</mtext><mi mathvariant="normal" /><mtext>which</mtext><mi mathvariant="normal" /><mi>i</mi><mi mathvariant="normal" /><mtext>either</mtext><mi mathvariant="normal" /><mtext>requested</mtext><mtext> a </mtext><mtext>hint</mtext><mi mathvariant="normal" /><mtext>or</mtext><mi mathvariant="normal" /><mtext>made</mtext><mi mathvariant="normal" /><mtext>an</mtext><mi mathvariant="normal" /><mtext>error</mtext></mrow></mfrac></mrow></math> </ephtml> </p> <p>measure hint frequency.</p> <p>This measure is inversely related to the ratio of the percentage (of all problems) a student makes an error without having requested a hint, to the percentage of problems a student requests a hint.[<reflink idref="bib2" id="ref38">2</reflink>] Making an error on a problem without having requested a hint reflects a hesitance to request a hint when one might have been warranted (or a careless mistake).</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> ignores the class of challenging problems on which students are challenged and figure the answer out (or guess correctly) without requesting a hint. This class of problems is inherently interesting to our question; however, it is unidentified.</p> <hd id="AN0149920084-7">Dichotomizing Hint Usage</hd> <p>The observational study and mediation analysis below require a dichotomous measurement of hint usage: is a student a high or low hint user? To this end, we dichotomize</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> by comparing it to a cutoff value, so that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><msub><mrow><mn>1</mn></mrow><mrow><mrow><mo stretchy="true">[</mo><mrow><msub><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover></mrow><mi>i</mi></msub><mo>></mo><mi>c</mi></mrow><mo stretchy="true">]</mo></mrow></mrow></msub></mrow><mo>,</mo></math> </ephtml> where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mn>1</mn></mrow><mrow><mrow><mo>[</mo><mi>x</mi><mo>]</mo></mrow></mrow></msub><mo>=</mo><mn>1</mn></mrow></math> </ephtml> if <emph>x</emph> is true and 0 otherwise.</p> <p>To choose <emph>c</emph>, we first fit a modified Rasch mixture model (Rasch, [<reflink idref="bib53" id="ref39">53</reflink>]) to hint request data. Specifically, we modeled the probability of at least one hint request in each problem <emph>p</emph>, worked by student <emph>i</emph>, in which <emph>i</emph> requested a hint and/or committed an error, as</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><msub><mrow><mi>h</mi></mrow><mrow><mi>i</mi><mi>p</mi></mrow></msub><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="italic">logi</mi><msup><mrow><mi>t</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo stretchy="false">(</mo><msub><mrow><mi>η</mi></mrow><mi>i</mi></msub><mo>+</mo><msub><mrow><mi>δ</mi></mrow><mrow><mi>s</mi><mo stretchy="true">[</mo><mi>p</mi><mo stretchy="true">]</mo></mrow></msub><mo stretchy="false">)</mo></mrow></math> </ephtml> (<reflink idref="bib1" id="ref40">1</reflink>)</p> <p>where <emph>h<subs>ip</subs></emph> is the event that student <emph>i</emph> requests at least one hint on problem <emph>p</emph>, <emph>η<subs>i</subs></emph> is a student parameter,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>s</mi><mo stretchy="false">[</mo><mi>p</mi><mo stretchy="false">]</mo></mrow></msub></mrow></math> </ephtml> is a section-level parameter for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>s</mi><mo stretchy="false">[</mo><mi>p</mi><mo stretchy="false">]</mo></mrow><mo>,</mo></math> </ephtml> the section that problem <emph>p</emph> was drawn from, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">logi</mi><msup><mrow><mi>t</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mrow><mrow><mo>{</mo><mn>1</mn><mo>+</mo><mo /><mtext>exp</mtext><mo /><mo stretchy="false">(</mo><mo>−</mo><mi>x</mi><mo stretchy="false">)</mo><mo>}</mo></mrow></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml> is the inverse logit function.</p> <p>Typically, the student parameter <emph>η<subs>i</subs></emph> measures student ability; here, instead, it measures a student's proclivity to request a hint. The typical Rasch model includes a problem-level "difficulty" parameter; here, it would measure the likelihood an individual problem elicits a hint request. That parameter is important in our context because the specific problems a student works on may influence his or her hint requests. For instance, two students with the same underlying proclivity to request hints η may actually differ in their observed hint requests, because one worked on harder problems than the other. However, in our dataset many problems were worked by very few students, so problem-level parameters would be hard to estimate. Instead, we included the section-level parameter</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>s</mi><mo stretchy="false">[</mo><mi>p</mi><mo stretchy="false">]</mo></mrow></msub></mrow><mo>,</mo></math> </ephtml> which measures the extent to which students tend to request hints on problems in section <emph>s</emph>. Ignoring the differences between problems within a section could allow for wider variance in the average problem difficulty each student experiences than is captured by the section level parameters</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>s</mi><mo stretchy="false">[</mo><mi>p</mi><mo stretchy="false">]</mo></mrow></msub></mrow><mo>.</mo></math> </ephtml> That could, in turn, induce a difference in hint requests between two students with similar <emph>η</emph> or vice-versa, and lead to bias in our estimate of <emph>η</emph>. Since the Cognitive Tutor selects problems for students based on estimates of their current skill mastery (Fancsali et al., [<reflink idref="bib12" id="ref41">12</reflink>]), this possibility is not out of the question.</p> <p>With even more granular data, (<reflink idref="bib1" id="ref42">1</reflink>) could be further elaborated, avoiding this assumption. In the Cognitive Tutor, problems often have multiple parts, and each part offers its own hints. Researchers in possession of data on each part of each worked problem could specify a model at the part-level, instead of at the problem-level like (<reflink idref="bib1" id="ref43">1</reflink>). Perhaps an even better option (suggested by an anonymous reviewer) would incorporate data on the specific skills required to solve each problem. While sections of the tutor differ from each other in the skills they teach, not every problem in each section includes the same set of skills; different problems requiring the same skill set will be similar in difficulty (see, e.g., Goldin & Koedinger, [<reflink idref="bib20" id="ref44">20</reflink>]; Pavlik et al., [<reflink idref="bib48" id="ref45">48</reflink>]).</p> <p>In order to dichotomize hint request behavior, we modeled student effects <emph>η</emph> as</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo>∼</mo><msub><mrow><mi>p</mi></mrow><mn>0</mn></msub><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><msub><mrow><mi>μ</mi></mrow><mn>0</mn></msub><mo>,</mo><msub><mrow><mi>σ</mi></mrow><mn>0</mn></msub><mo stretchy="false">)</mo><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msub><mrow><mi>p</mi></mrow><mn>0</mn></msub><mo stretchy="false">)</mo><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><msub><mrow><mi>μ</mi></mrow><mn>1</mn></msub><mo>,</mo><msub><mrow><mi>σ</mi></mrow><mn>1</mn></msub><mo stretchy="false">)</mo></mrow></math> </ephtml> (<reflink idref="bib2" id="ref46">2</reflink>)</p> <p>and constrained</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>μ</mi></mrow><mn>0</mn></msub><mo><</mo><msub><mrow><mi>μ</mi></mrow><mn>1</mn></msub></mrow><mo>.</mo></math> </ephtml> This mixture model clusters students into two categories: high hint users have <emph>η</emph> drawn from a normal distribution with mean <emph>μ</emph><subs>1</subs> and standard deviation <emph>σ</emph><subs>1</subs>, and low hint users have <emph>η</emph> drawn from a normal distribution with mean <emph>μ</emph><subs>0</subs> and standard deviation <emph>σ</emph><subs>0</subs>.</p> <p>Our main parameter of interest here is <emph>p</emph><subs>0</subs>, the proportion of students who are low hint users. We estimated</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>p</mi></mrow><mn>0</mn></msub><mo>≈</mo></mrow></math> </ephtml> 0.7, classifying 30% of students as high hint users, so we chose <emph>c</emph> as the 70th percentile of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo>,</mo></math> </ephtml> approximately 0.6. This definition agreed with the model's classification (based on <emph>η</emph> as opposed to</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> ) in approximately 93% of cases.</p> <p>The Rasch model for hint usage (<reflink idref="bib1" id="ref47">1</reflink>), but not the mixture model (<reflink idref="bib2" id="ref48">2</reflink>) will play an important role in the principal stratification analysis of Section "Principal Stratification."</p> <hd id="AN0149920084-8">Data</hd> <p>Pane et al. ([<reflink idref="bib47" id="ref49">47</reflink>]) estimated the effects separately in middle schools and high schools in each year of implementation; the only statistically significant treatment effect was in the high school sample in the second year. Since our goal was to better understand the CTA1 treatment effect, we focused our analysis on data from high school students in the second year of the CTA1 trial, for whom the treatment effect was most evident.</p> <p>We merged data from two sources: computerized log data gathered by Carnegie Learning, and covariate, treatment and outcome data gathered by RAND.</p> <p>The log data lists the problem name, section, and unit of each problem, the numbers of hints requested and errors committed, and time-stamps. Log data were missing for some students, either because the log files were not retrievable, or because of an imperfect ability to link log data to other student records. Further, log data for sections that were not part of the standard CTA1 Algebra I curriculum and sections worked by fewer than 100 students were omitted from the dataset.[<reflink idref="bib3" id="ref50">3</reflink>]</p> <p>To construct our analysis dataset, we first dropped treatment schools with log data missing for 90% or more students. Prior to randomization, schools were stratified into pairs or triples based on baseline covariates, and randomized between treatment and control condition within those randomization blocks. When we dropped a treatment school from our analysis, we also dropped the control schools in its randomization block. Of the remaining 2,390 students, 88% (<reflink idref="bib2" id="ref51">2</reflink>,<reflink idref="bib108" id="ref52">108</reflink>) had log data. Then, for the sake of simplicity, the treatment students without log data were dropped from the study.[<reflink idref="bib4" id="ref53">4</reflink>]</p> <p>All told, the analyses presented here were all based on 2,108 students assigned to the CTA1 condition; the estimation of direct effects in Section "Mediation Analysis" and the principal stratification also relied on the 2,918 students assigned to control. Together, the 5,026 students were nested within 116 teachers, in 43 schools across five states.</p> <p>Table 1 describes the covariates we used, including missingness information, control and treatment means, and standardized differences (c.f. Kalton, [<reflink idref="bib33" id="ref54">33</reflink>]) from the final analysis sample. We singly-imputed missing values with the Random Forest routine implemented by the missForest package in R (R Core Team, [<reflink idref="bib52" id="ref55">52</reflink>]; Stekhoven & Bühlmann, [<reflink idref="bib67" id="ref56">67</reflink>]), which estimated "out of box" imputation error rates as part of the random forest regression, also shown in Table 1.</p> <p>Table 1. Missingness information, control ("BaU" or "Business as Usual") and treatment ("CTA1") means, and balance for the covariates included in this study, from the high school year two stratum of CTA1 Effectiveness experiment.</p> <p> <ephtml> <table><thead><tr><td /><td>% Miss.</td><td>Imp. Err.</td><td>Levels</td><td>BaU</td><td>CTA1</td><td>Std. Diff.</td></tr></thead><tbody valign="top"><tr><td>Grade</td><td>3%</td><td char=".">0.01</td><td>9th</td><td>91%</td><td char=".">90%</td><td char=".">−0.04</td></tr><tr><td /><td /><td char="." /><td>>9th</td><td>9%</td><td char=".">10%</td><td char=".">0.04</td></tr><tr><td>ELL</td><td>4%</td><td char=".">0.01</td><td>No</td><td>95%</td><td char=".">96%</td><td char=".">0.09</td></tr><tr><td /><td /><td char="." /><td>Yes</td><td>5%</td><td char=".">4%</td><td char=".">−0.09</td></tr><tr><td>FRL</td><td>29%</td><td char=".">0.29</td><td>No</td><td>26%</td><td char=".">27%</td><td char=".">0.02</td></tr><tr><td /><td /><td char="." /><td>Yes</td><td>74%</td><td char=".">73%</td><td char=".">−0.02</td></tr><tr><td>Ethnicity</td><td>6%</td><td char=".">0.23</td><td>White/Asian</td><td>47%</td><td char=".">52%</td><td char=".">0.16</td></tr><tr><td /><td /><td char="." /><td>Black/Multi</td><td>32%</td><td char=".">26%</td><td char=".">−0.14</td></tr><tr><td /><td /><td char="." /><td>Hispanic/Nat.Am.</td><td>21%</td><td char=".">22%</td><td char=".">−0.03</td></tr><tr><td>Sex</td><td>5%</td><td char=".">0.35</td><td>Female</td><td>51%</td><td char=".">49%</td><td char=".">−0.04</td></tr><tr><td /><td /><td char="." /><td>Male</td><td>49%</td><td char=".">51%</td><td char=".">0.04</td></tr><tr><td>Sp. Ed.</td><td>1%</td><td char=".">0.11</td><td>Typical</td><td>87%</td><td char=".">86%</td><td char=".">−0.00</td></tr><tr><td /><td /><td char="." /><td>Sp. Ed.</td><td>8%</td><td char=".">8%</td><td char=".">−0.02</td></tr><tr><td /><td /><td char="." /><td>Gifted</td><td>5%</td><td char=".">6%</td><td char=".">0.03</td></tr><tr><td>Pretest</td><td>16%</td><td char=".">0.20</td><td /><td>−0.33</td><td>−0.36</td><td char=".">−0.05</td></tr><tr><td /><td /><td char="." /><td>Overall Covariate Balance: <italic>p</italic> = 0.55</td><td /><td char="." /><td char="." /></tr></tbody></table> </ephtml> </p> <p>1 Imputation error is percent falsely classified for categorical variables (Race/Ethnicity, Sex, and Special Education) and standardized root mean squared error for Pretest, which is continuous. Analysis done in R via RItools (Bowers et al., [<reflink idref="bib7" id="ref57">7</reflink>]).</p> <hd id="AN0149920084-9">An Observational Study Within an Experiment</hd> <p>How does requesting hints within CTA1 affect learning? More precisely, would low hint users have achieved higher posttest scores had they requested hints more often? Would high hint users have achieved higher posttest scores had the requested hints less often?</p> <p>This section will illustrate an observational study to answer these questions. Generally speaking, the observational study approach discards the control group entirely and uses statistical confounder control (in our case, propensity score matching) to estimate causal effects of a usage parameter on posttest scores. Implementing the method requires data from members of the treatment group: usage (in our case, <emph>H</emph>), posttest scores or another outcome of interest, and a set of baseline covariates to control for confounding. Since the control group is not used, identification does not depend on randomization—in fact, this same approach could be (and is) used with non-experimental data. However, outside of a planned experiment, posttest scores may not be available. Our usage variable <emph>H</emph> is binary, which is necessary for classical propensity score matching; alternative methods are available for usage variable that are continuous (e.g., Hirano & Imbens, [<reflink idref="bib25" id="ref58">25</reflink>]), ordered (e.g., Leon & Hedeker, [<reflink idref="bib36" id="ref59">36</reflink>]), categorical (e.g., Michael & Gutman, [<reflink idref="bib42" id="ref60">42</reflink>]) or other types. Like all observational studies, it depends on the strong, untestable assumption of no unmeasured confounding.</p> <p>The approach we take here, following Rosenbaum ([<reflink idref="bib54" id="ref61">54</reflink>]), Hansen ([<reflink idref="bib22" id="ref62">22</reflink>]), and Ho et al. ([<reflink idref="bib26" id="ref63">26</reflink>]), has three broad steps. In the first, we construct a "match," identifying groups of students, some of whom have <emph>H</emph> = 0 and others with <emph>H</emph> = 1, but who are otherwise comparable. The hope is that the distribution of <emph>H</emph> conditional on this match resembles what might have been seen if (counterfactually) <emph>H</emph> had been randomized within matched groups. Thus we evaluate the success of the match by checking if matched students have similar baseline covariate distributions; if necessary, we may revise match. In the second step, we estimate the average effect of <emph>H</emph> on posttest scores, by comparing posttest scores between matched students with <emph>H</emph> = 1 and <emph>H</emph> = 0. In the third and final step, we account for the possibility of unmeasured confounding in a sensitivity analysis.</p> <p>The next subsection will give more formal background on propensity score matching, including notation and identification assumptions. Following subsections will illustrate each of the three steps of propensity score matching.</p> <hd id="AN0149920084-10">Observational Study: Background</hd> <p>Let <emph>Y<subs>i</subs></emph> represent subject <emph>i</emph>'s posttest score, and let <emph>Z<subs>i</subs></emph> represent <emph>i</emph>'s treatment status (i.e., the CTA1 or control group). Following Neyman ([<reflink idref="bib44" id="ref64">44</reflink>]) and Rubin ([<reflink idref="bib56" id="ref65">56</reflink>]), let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> be the posttest score student <emph>i</emph> would achieve were <emph>i</emph> (perhaps counterfactually) assigned to the treatment condition, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> the score were <emph>i</emph> assigned to control. Then the CTA1 treatment effect for student <emph>i</emph> is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> For students in the treatment group, define potential outcomes</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> or</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> for short, corresponding to <emph>i</emph>'s posttest score were <emph>H<subs>i</subs></emph> = 1 or 0, respectively. Then the effect of <emph>H</emph> on <emph>Y</emph> for student <emph>i</emph> is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> This structure implicitly assumes "non-interference," that one student's treatment assignment <emph>Z</emph> or hint proclivity <emph>H</emph> does not affect another student's posttest scores.</p> <p>Individual treatment effects are (typically) unidentified, since only one of the relevant potential outcomes is ever observed for each subject. For instance,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is unobserved for members of the treatment group with <emph>H</emph> = 0. If</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> were known for each <emph>i</emph>, and bounded away from 1 and 0, (as would be the case if <emph>H</emph> were randomized) then the average treatment effect (ATE) of <emph>H</emph> on <emph>Y</emph>,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>−</mo><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><mo>,</mo></math> </ephtml> could be estimated without bias or further assumptions. Of course, the distribution of <emph>H<subs>i</subs></emph> is unknown and is presumably a function of <emph>i</emph>'s individual characteristics (which, as we shall see, differed at baseline between those students with <emph>H</emph> = 1 and those with <emph>H</emph> = 0).</p> <p>Let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub></mrow></math> </ephtml> represent a vector of baseline covariates for subject <emph>i</emph>. In our study of hints, <bold><emph>x</emph></bold> includes indicators for state, school, and classroom, and the variables described in Table 1. Then we assume (c.f. Rosenbaum & Rubin, [<reflink idref="bib55" id="ref66">55</reflink>]):</p> <hd id="AN0149920084-11">Assumption: Strong Ignorability</hd> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>{</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo><mo>}</mo><mo>⊥</mo><mi mathvariant="normal">​</mi><mi mathvariant="normal">​</mi><mo>⊥</mo><mi>H</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi></mrow></math> </ephtml> </p> <p>that conditional on <bold><emph>x</emph></bold>, potential outcomes</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> are independent of realized <emph>H</emph>. Under strong ignorability, one may compare subjects with identical covariates <bold><emph>x</emph></bold> to estimate causal effects.</p> <p>Unfortunately, this sort of exact matching is impossible in our finite sample. Instead, we estimated "propensity scores" (Rosenbaum & Rubin, [<reflink idref="bib55" id="ref67">55</reflink>]):</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>π</mi></mrow><mi>i</mi></msub><mo>=</mo><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>the probability of a treatment-group subject requesting frequent hints, conditional on his or her covariate vector <bold><emph>x</emph></bold>. Rosenbaum and Rubin ([<reflink idref="bib55" id="ref68">55</reflink>]) shows that conditioning on <emph>π</emph> is equivalent to conditioning on <bold><emph>x</emph></bold>. We estimated <emph>π</emph> using a multilevel logistic regression, using the lme4 package in R (Bates et al., [<reflink idref="bib5" id="ref69">5</reflink>]). The propensity score model was:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>π</mi></mrow><mi>i</mi></msub><mo>=</mo><mi mathvariant="italic">logi</mi><msup><mrow><mi>t</mi></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup><mo stretchy="false">(</mo><mi>α</mi><mo>+</mo><mover accent="true"><mi mathvariant="bold-italic">x</mi><mo>˜</mo></mover><mi mathvariant="bold-italic">β</mi><mo>+</mo><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">state</mi></mrow></msup><mo>+</mo><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">school</mi></mrow></msup><mo>+</mo><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">class</mi></mrow></msup><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi mathvariant="bold-italic">x</mi><mo>˜</mo></mover></math> </ephtml> is a vector of covariates including the variables in Table 1, missingness indicators for grade, race, sex, and economic disadvantage, and a natural spline with five degrees of freedom for pretest. State, school, and class were each included using normally distributed random intercepts,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">state</mi></mrow></msup></mrow><mo>,</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">school</mi></mrow></msup></mrow><mo>,</mo></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">class</mi></mrow></msup></mrow><mo>.</mo></math> </ephtml> </p> <hd id="AN0149920084-12">Constructing and Evaluating a Match</hd> <p>We constructed an optimal full matching design (Hansen, [<reflink idref="bib21" id="ref70">21</reflink>]) to condition on estimated propensity scores</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>π</mi><mo>̂</mo></mover></math> </ephtml> using the optmatch package in R (Hansen & Klopfer, [<reflink idref="bib24" id="ref71">24</reflink>]). We matched <emph>H</emph> = 0 subjects to <emph>H</emph> = 1 subjects in such a way as to minimize the overall distance between subjects in matched sets in the logit-transformed propensity scores,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="italic">logit</mi><mo stretchy="false">(</mo><mover accent="true"><mi>π</mi><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> We constrained the match so that students could only be matched within schools, since schools determine a number of factors that in turn may impact posttest scores, including baseline student characteristics, pedagogical styles, and CTA1 usage patterns (see Israni et al., [<reflink idref="bib30" id="ref72">30</reflink>]). Each matched set was allowed to contain any positive number of <emph>H</emph> = 1 and any positive number of <emph>H</emph> = 0 subjects, resulting in matches of variable size and composition. For instance, one matched set included 44 <emph>H</emph> = 1 subjects matched to a single <emph>H</emph> = 0 subject, and another matched 117 <emph>H</emph> = 0 subjects to a single <emph>H</emph> = 1 subject.</p> <p>This strategy allowed every student in the CTA1 arm our dataset to be included in the match. In contrast, many matching studies discard subjects in the data sample with estimated propensity scores close to 1 or 0, focusing on the "region of common support" (e.g., Caliendo & Kopeinig, [<reflink idref="bib8" id="ref73">8</reflink>]; Shadish & Steiner, [<reflink idref="bib65" id="ref74">65</reflink>]). Doing so requires modifying the causal estimand—studies can only estimate the ATE for included subjects and those who resemble them. We chose, instead, to include every subject in order to simplify the mediation analysis in the following section, and validated the match by inspecting covariate balance. This came at the cost of imprecision in estimating the overall ATE, which we discuss in more detail below.</p> <p>Figure 2 and Table 4 in the online appendix show that matching largely eliminated mean differences in pretreatment covariates between <emph>H</emph> = 1 and <emph>H</emph> = 0 students. The only notable exception is in pretest scores: <emph>H</emph> = 1 students tended to have slightly higher pretest scores than their matched comparisons with <emph>H</emph> = 0. The <emph>p</emph>-value from an omnibus covariate balance test (Hansen & Bowers, [<reflink idref="bib23" id="ref75">23</reflink>]) is 0.39.</p> <hd id="AN0149920084-13">Estimating the Effects of H on Posttest Scores</hd> <p>Let <emph>M<subs>i</subs></emph> be a categorical variable denoting the matched set that <emph>i</emph> belongs to. Then we may re-state the ignorability condition with reference to this match (c.f. Sales et al., [<reflink idref="bib59" id="ref76">59</reflink>]):</p> <hd id="AN0149920084-14">Assumption: Matched Ignorability</hd> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>{</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo><mo>}</mo><mo>⊥</mo><mi mathvariant="normal">​</mi><mi mathvariant="normal">​</mi><mo>⊥</mo><mi>H</mi><mo>|</mo><mi>M</mi></mrow></math> </ephtml> </p> <p>Under matched ignorability, the difference of mean outcomes between <emph>H</emph> = 1 and <emph>H</emph> = 0 students within each match is unbiased for the average treatment effect for students in that match. In other words, if</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>τ</mi></mrow><mi>m</mi></msub><mo>=</mo><mover accent="true"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">¯</mo></mover><mo>−</mo><mover accent="true"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">¯</mo></mover></mrow></math> </ephtml> is the average effect in match <emph>m</emph>, then</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow><mi>m</mi></msub><mo>=</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><mi>m</mi><mo>,</mo><mi>H</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>−</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><mi>m</mi><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></math> </ephtml> is an unbiased estimate of <emph>τ<subs>m</subs></emph>, where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><mi>m</mi><mo>,</mo><mi>H</mi><mo>=</mo><mi>h</mi></mrow></msub></mrow></math> </ephtml> is the average observed outcome for subjects in match <emph>m</emph> with hint level</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>h</mi><mo>∈</mo><mo>{</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>}</mo></mrow><mo>.</mo></math> </ephtml> </p> <p>Weighted means of those estimated treatment effects, of the form</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow><mi>w</mi></msub><mo>=</mo><munder><mo>∑</mo><mi>m</mi></munder><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub></mrow><msub><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow><mi>m</mi></msub><mo>/</mo><munder><mo>∑</mo><mi>m</mi></munder><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib3" id="ref77">3</reflink>)</p> <p>estimate aggregate treatment effects in the sample. To estimate the average effect of <emph>H</emph> for all subjects, set</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub><mo>=</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub><mo>+</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>0</mn><mi>m</mi></mrow></msub></mrow><mo>,</mo></math> </ephtml> where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>n</mi></mrow><mrow><mn>0</mn><mi>m</mi></mrow></msub></mrow></math> </ephtml> are the number of <emph>H</emph> = 1 and <emph>H</emph> = 0 subjects in match <emph>m</emph>, respectively. Alternatively, setting</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub><mo>=</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow></math> </ephtml> estimates the the "treatment on the treated," or TOT, effect—the average effect of <emph>H</emph> on those subjects for whom <emph>H</emph> = 1. Weights</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub><mo>=</mo><msup><mrow><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>/</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub><mo>+</mo><mn>1</mn><mo>/</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>0</mn><mi>m</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mrow><mrow><mo>−</mo><mn>1</mn></mrow></msup></mrow></math> </ephtml> are "OLS" or "precision" (c.f. Schochet, [<reflink idref="bib63" id="ref78">63</reflink>]); the estimate using these weights is equal to the coefficient on <emph>H</emph> from an ordinary least squares (OLS) regression of <emph>Y</emph> on <emph>H</emph> and a set of dummy variables for <emph>M</emph>. Under standard OLS assumptions, precision weights minimize the standard error of the weighted mean estimator.</p> <p>Table 2 gives treatment effect estimates, standard errors, and confidence intervals under these three weighting schemes. All three sets of standard errors and confidence intervals used the heteroskedasticity-robust "HC3" sandwich estimator (Zeileis, [<reflink idref="bib73" id="ref79">73</reflink>]). The three estimates all roughly agree that being a high hint user decreases posttest scores by around 0.15 standard deviations. That is, they suggest that requesting hints hurts posttest scores.</p> <p>Table 2. Estimates, standard errors, and sensitivity analysis of the weighted average effect of hint usage on posttest scores, under different weighting schemes.</p> <p> <ephtml> <table><thead><tr><td /><td /><td /><td /><td>Sensitivity Intervals</td></tr><tr><td>Weights</td><td>Estimate</td><td>Std. Error</td><td>CI</td><td>[Pretest]</td><td>[Ethnicity]</td></tr></thead><tbody valign="top"><tr><td>OLS</td><td>−0.18</td><td>0.04</td><td>[−0.25, −0.10]</td><td>[−0.41, 0.06]</td><td>[−0.30, −0.05]</td></tr><tr><td>ATE</td><td>−0.12</td><td>0.05</td><td>[−0.23, −0.02]</td><td>[−0.45, 0.20]</td><td>[−0.29, 0.04]</td></tr><tr><td>TOT</td><td>−0.14</td><td>0.06</td><td>[−0.25, −0.03]</td><td>[−0.47, 0.19]</td><td>[−0.31, 0.03]</td></tr></tbody></table> </ephtml> </p> <p>The similarity of the three estimates boosts our confidence, if only slightly. The ATE and TOT estimates refer to different groups of students—ATE estimates an average effect for the entire treatment group, whereas TOT estimates an effect for high hint users. Had these been very different, one might wonder why the effect of hint requests varies so much between high and low hint users, and if that suggests selection bias. The OLS estimate downweights matched sets with many more <emph>H</emph> = 1 than <emph>H</emph> = 0 students or vice-versa, relative to matched sets with several of both. (Recall that such imbalanced matched sets were the result of the decision to include every subject in the analysis.) It is larger in magnitude and much less noisy than the other two estimates.</p> <hd id="AN0149920084-15">Sensitivity Analysis: Assessing the Role of Unmeasured Confounding</hd> <p>None of these estimates addressed the possibility of confounding from unmeasured covariates; instead, they relied on matched ignorability. To account for the possibility of unmeasured confounding, we conducted a sensitivity analysis following the method of Hosman et al. ([<reflink idref="bib28" id="ref80">28</reflink>]). This method imagines a hypothetical missing covariate <emph>U</emph> and estimates how the the omission of <emph>U</emph> alters the estimate and its standard error. The result is a "sensitivity interval" (c.f. Rosenbaum, [<reflink idref="bib54" id="ref81">54</reflink>]) that, with 95% confidence, contains the true effect accounting for both sampling uncertainty and uncertainty due to possible confounding. <emph>U</emph> is characterized by two sensitivity parameters:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ρ</mi></mrow><mn>2</mn></msup></mrow></math> </ephtml> is the squared partial correlation between <emph>U</emph> and <emph>Y</emph>, after accounting for the observed covariates in the model, and <emph>T<subs>Z</subs></emph> is the <emph>t</emph>-statistic for the coefficient on <emph>U</emph> from an ordinary least squares regression of <emph>H</emph> on <emph>U</emph> and the observed covariates. These sensitivity parameters may be benchmarked by estimating their counterparts when observed covariates are left out of the model.</p> <p>The last two columns of Table 2 give sensitivity intervals for each of the three weighted average treatment effects. The column of Table 2 labeled "[Pretest]" accounts for the possible omission of a covariate <emph>U</emph> that, at most, predicts <emph>Y</emph> and <emph>H</emph> as well as do pretest scores. The resulting sensitivity intervals are quite wide, including both large negative effects as well as substantial positive effects. In fact, pretest is the most important of our observed covariates; it is the observed covariate whose omission would cause the most bias. This is unsurprising, since pretests are generally considered to be the most important covariate to measure, and studies that include pretest measures often reproduce experimental estimates (e.g., Cook et al., [<reflink idref="bib9" id="ref82">9</reflink>], [<reflink idref="bib10" id="ref83">10</reflink>]). Therefore, a sensitivity analysis considering an omitted covariate as important as pretest scores may be too pessimistic. On the other hand, it is likely that pretest scores do not fully capture between-student variance in academic ability; that is, there may be important components of prior ability (such as the ability to learn algebra) that are correlated with hint requests and posttest scores but not reflected in pretest scores. Hypothetical measurements of these components could, perhaps, constitute unobserved covariates with similar or greater importance than observed pretest scores.</p> <p>A more optimistic scenario is reflected in the final column of Table 2. That column gives sensitivity intervals for the omission of a covariate that, at most, predicts <emph>Y</emph> and <emph>H</emph> as well as ethnicity dummy variables, the second most important of our observed covariates. These sensitivity intervals are also wide, but the interval corresponding to OLS weights is entirely negative.</p> <p>The conclusion is that the omission of a confounder as important as ethnicity indicators cannot explain the estimated OLS-weighted ATE. Allowing for the possible omission of such a covariate, the results suggest that requesting a large number of hints hurts students' posttest scores. The size of the effect may be as small as 5% of a standard deviation or as large as 30% of a standard deviation.</p> <hd id="AN0149920084-16">Mediation Analysis</hd> <p>The goal of causal mediation analysis is to decompose a treatment effect into an "indirect" or "mediated" effect and a "direct" effect. The indirect effect captures the component of the effect that is due to the mediator: the treatment affected the outcome by affecting the mediator, which itself, in turn, affected the outcome. Assignment to the treatment condition allowed students to request many hints during their work—how did the wide availability of hints affect their posttest scores? The direct effect is the component of the overall effect that operates via other mechanisms. For instance, how would assignment to CTA1 affect posttest scores were hints more limited?</p> <p>The method we present here builds on the results from the observational study in the previous section. In fact, in our analysis estimating indirect effects requires no more data than the observational study. Estimating direct effects requires, additionally, data from the control group: posttest scores and (ideally) baseline covariates. The randomization of the intervention in the RCT now plays an important role in identification since the effects of the intervention on both hint usage and posttest scores are at issue; the fact that the control group does not have access to the program likewise plays a crucial role. Finally, the assumption of no unmeasured confounding remains important.</p> <p>It is worth noting here that conducting mediation analysis requires measuring or positing a value of the mediator in the control arm of the study. We know that CTA1 hints were not available to the control group, so we set <emph>H</emph> = 0 for all members of the control group. However, other facets of log data may not be coherently defined in the control group; for instance, students' proclivity to master the skills of one CTA1 section before moving on to the next (Sales & Pane, [<reflink idref="bib60" id="ref84">60</reflink>]) or teachers' practice of reassigning students to new CTA1 sections (Sales & Pane, [<reflink idref="bib58" id="ref85">58</reflink>]). In cases such as these, the mediation framework may not apply.</p> <p>The following subsection reviews the formal definitions of direct and indirect effects. Subsection "Identification of Direct and Indirect Effects" discusses identification, and "Estimating Log Data Indirect Effects" and "Estimating Log Data Direct Effects" discuss estimation of indirect and direct effects, respectively.</p> <hd id="AN0149920084-17">Review: Defining Direct and Indirect Effect</hd> <p>Express a subject's potential outcomes as a function of two variables, treatment assignment <emph>Z</emph> and hint requests <emph>H</emph>: <emph>Y</emph>(<emph>Z</emph>, <emph>H</emph>). Now, if <emph>Z</emph> and <emph>H</emph> are both binary, as in our example, subjects each have four potential outcomes:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>;</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>;</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo stretchy="false">)</mo><mo>;</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>representing the outcomes they would exhibit were they assigned to control and requested few hints, assigned to treatment and requested few hints, assigned to control and requested many hints, or assigned to treatment and requested many hints, respectively. In the CTA1 trial, however, the potential outcome <emph>Y</emph>(0, 1) is meaningless: students in the control group had no access to CTA1, and therefore could not request hints. This will be an important factor going forward.</p> <p>Since <emph>H</emph> is itself affected by treatment assignment, it too has potential values:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>;</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>representing the hints that would be requested under treatment and control. In the CTA1 study,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow></math> </ephtml> for all subjects.</p> <p>Combining potential values for <emph>Y</emph> and <emph>H</emph> yields the fundamental building blocks of causal mediation analysis (e.g., VanderWeele, [<reflink idref="bib72" id="ref86">72</reflink>]). Let</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mi>z</mi><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mi>z</mi><mo>′</mo><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> represent the outcome that a subject would express given treatment assignment <emph>z</emph>, but the hint behavior that he or she would have expressed under assignment</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>z</mi><mo>′</mo></mrow><mo>.</mo></math> </ephtml> When</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>z</mi><mo>=</mo><mi>z</mi><mo>′</mo></mrow><mo>,</mo></math> </ephtml> the original two potential outcomes for <emph>Y</emph> emerge:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is the outcome exhibited under the treatment, when <emph>H</emph> takes its potential treatment value; similarly,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> On the other hand,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> are strictly counterfactual.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> gives the outcome value that would result in a subject assigned to the treatment condition but who nevertheless requested hints as he or she would have under control.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> gives the potential outcome for a subject assigned to control but who nevertheless requested hints as he or she would have under treatment. This last quantity is problematic in our case since <emph>H</emph>(<reflink idref="bib1" id="ref87">1</reflink>) can take the value 1, and as we have seen <emph>Y</emph>(0, 1) is undefined.</p> <p>This framework facilitates precise definitions of direct and indirect effects; in fact, there are two versions of each (e.g., Imai et al., [<reflink idref="bib29" id="ref88">29</reflink>]):</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mtable><mtr><mtd><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mi>δ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mtd></mtr></mtable></math> </ephtml> </p> <p>are the indirect effects, contrasting potential outcomes when <emph>H</emph> varies as it would with varying treatment assignment, but holding the assignment itself constant at either 1 or 0.</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mtable><mtr><mtd><mi>ξ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mtd></mtr></mtable></math> </ephtml> </p> <p>represent the direct effects: holding the value of <emph>H</emph> constant, either at its potential value under treatment or control, while varying treatment assignment. The total treatment effect,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> can then be decomposed in two ways: as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>+</mo><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> or as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>δ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>+</mo><mi>ξ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> These decompositions can potentially reveal the role hints play in the CTA1 treatment effect:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> gives the extent to which assignment to CTA1 affects a student's posttest score by (possibly) causing him or her to request many hints.</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> gives the effect of assignment to CTA1 if hints were, perhaps counterfactually, held at their control level, <emph>H</emph> = 0.</p> <p>Rather than attempt to estimate individual direct and indirect effects <emph>ξ<subs>i</subs></emph> and <emph>δ<subs>i</subs></emph>, our goal will be to estimate their means,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>ξ</mi></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi></mrow><mo>.</mo></math> </ephtml> </p> <hd id="AN0149920084-18">Identification of Direct and Indirect Effects</hd> <p>Since the potential outcome <emph>Y</emph>(0, 1) is undefined in the CTA1 hints study, any expression (potentially) including <emph>Y</emph>(0, 1) is also undefined; this includes</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>δ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>ξ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> Indeed, since hint behavior is unobserved in the control group altogether, conventional approaches to mediation analysis do not apply. Further, were we to operationalize hint behavior as a continuous variable, as may seem natural, mediation analysis, in this case, would also be nearly impossible. This is because students assigned to the control group requested exactly zero hints, whereas merely 5 members of the treatment group requested no hints over the course of the study, and these students barely used the tutor at all. That being the case, the data provide little to no information about the distribution of potential outcomes when the software is used but no hints are requested.</p> <p>Dichotomizing hint requests into <emph>H</emph> provides a way forward. Since the experimental design ensures that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><mo>,</mo></math> </ephtml> we have that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>≡</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>;</mo></math> </ephtml> this potential outcome is observed for subjects in the treatment arm with <emph>H</emph> = 0. As above,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> are the observed outcomes for subjects in the control and treatment arms, respectively. Therefore, we observe each of the potential outcomes in the definitions of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> for at least some non-trivial portion of subjects in the study sample.</p> <p>Of course, <emph>H</emph> = 0 means something different in the two treatment arms: in the treatment group <emph>H</emph> = 0 typically means requesting <emph>some</emph> hints, but not many, whereas subjects in the control group request exactly zero hints. One way to make sense of mediational estimands involving <emph>H</emph> is to imagine an experiment in which students are assigned to different values of <emph>H</emph>. 70% of students in the treatment group are randomly assigned to <emph>H</emph> = 1 and 30% to <emph>H</emph> = 0. This assignment constraint</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo>,</mo></math> </ephtml> the proportion of problems on which they may request hints, so that if <emph>H<subs>i</subs></emph> = 0,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover></mrow><mi>i</mi></msub></mrow></math> </ephtml> is constrained to be less than 0.6, but if <emph>H<subs>i</subs></emph> = 1</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover></mrow><mi>i</mi></msub></mrow></math> </ephtml> must be greater than or equal to 0.6. In contrast, all students in the control group are assigned <emph>H</emph> = 0 and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo>=</mo><mn>0</mn></mrow><mo>.</mo></math> </ephtml> </p> <p>Average potential outcomes</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> are identified due to the randomization of <emph>Z</emph>, but estimating</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> requires imputing the potential outcome <emph>Y</emph>(<reflink idref="bib1" id="ref89">1</reflink>, 0) for treated subjects with <emph>H<subs>i</subs></emph> = 1. However, it turns out that the matching estimators from Section "An Observational Study within an Experiment" already solved this problem: under matched ignorability, within matched sets, <emph>Y</emph>(<reflink idref="bib1" id="ref90">1</reflink>, 0) is independent of <emph>H</emph>, implying that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>M</mi><mo>,</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">]</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>M</mi><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">]</mo></mrow><mo>.</mo></math> </ephtml> That suggests imputations</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mrow><mo stretchy="true">{</mo><mrow><mtable><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><msub><mrow><mi>m</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn></mrow></msub></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></mtd></mtr></mtable></mrow></mrow></mrow></math> </ephtml> (<reflink idref="bib4" id="ref91">4</reflink>)</p> <p>That is, when <emph>Z<subs>i</subs></emph> = 1 and <emph>H<subs>i</subs></emph> = 0, no imputation is necessary, since</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub></mrow><mo>,</mo></math> </ephtml> the observed outcome. When <emph>Z<subs>i</subs></emph> = 1 and <emph>H<subs>i</subs></emph> = 1, impute the average of the observed outcomes from the <emph>H</emph> = 0 subjects matched to <emph>i</emph> for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> Matched ignorability implies that the imputations will be unbiased.</p> <hd id="AN0149920084-19">Estimating Log Data Indirect Effects</hd> <p>Since</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> involves only <emph>Z</emph> = 1 potential outcomes, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow></math> </ephtml> is known based on the study design,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> may be estimated using data from the treatment group only.</p> <p>Using the imputations from (<reflink idref="bib4" id="ref92">4</reflink>), we estimate</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> as</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover><mo>=</mo><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>Y</mi><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover><mo>−</mo><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub></mrow></mfrac><munder><mo>∑</mo><mrow><mi>i</mi><mo>:</mo><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></munder><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub></mrow><mo>−</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub><mo>=</mo><munder><mo>∑</mo><mi>i</mi></munder><mrow><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub></mrow></mrow><mo>,</mo></math> </ephtml> the number of subjects in the treatment group. Since</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo>=</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> whenever <emph>H<subs>i</subs></emph> = 0, the summand is non-zero only when <emph>H<subs>i</subs></emph> = 1. Further, note that when multiple <emph>H</emph> = 1 subjects share a matched set, they also share imputations</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> Re-writing</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover></mrow></math> </ephtml> in terms of matched sets <emph>M</emph> =<emph> m</emph>,</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mtable><mtr><mtd><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub></mrow></mfrac><munder><mo>∑</mo><mi>m</mi></munder><mrow><munder><mo>∑</mo><mrow><mi>i</mi><mo>:</mo><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>,</mo><msub><mrow><mi>M</mi></mrow><mi>i</mi></msub><mo>=</mo><mi>m</mi></mrow></munder><mrow><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub></mrow></mrow><mo>−</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><msub><mrow><mi>m</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn></mrow></msub></mtd></mtr><mtr><mtd><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub></mrow></mfrac><munder><mo>∑</mo><mi>m</mi></munder><mrow><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow><mo stretchy="false">(</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>M</mi><mo>=</mo><msub><mrow><mi>m</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>H</mi><mo>=</mo><mn>1</mn></mrow></msub><mo>−</mo><msub><mrow><mover accent="true"><mi>Y</mi><mo>¯</mo></mover></mrow><mrow><mi>m</mi><mo>=</mo><msub><mrow><mi>m</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>H</mi><mo>=</mo><mn>0</mn></mrow></msub><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mo>=</mo><mfrac><mn>1</mn><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub></mrow></mfrac><munder><mo>∑</mo><mi>m</mi></munder><mrow><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow><msub><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow><mi>m</mi></msub><mo>=</mo><mfrac><mrow><munder><mo>∑</mo><mi>i</mi></munder><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub></mrow></mrow><mrow><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub></mrow></mfrac><mover accent="true"><mrow><mi mathvariant="italic">TOT</mi></mrow><mo stretchy="true">̂</mo></mover></mtd></mtr></mtable></math> </ephtml> (<reflink idref="bib5" id="ref93">5</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mrow><mi>m</mi></msub></mrow></math> </ephtml> were defined in Section "Estimating the Effects of H on Posttest Scores" as the number of <emph>H</emph> = 1 subjects in match <emph>m</emph> and the difference in the sample mean of <emph>Y</emph> between <emph>H</emph> = 1 and <emph>H</emph> = 0 subjects in match <emph>m</emph>, respectively, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mrow><mi mathvariant="italic">TOT</mi></mrow><mo stretchy="true">̂</mo></mover></mrow></math> </ephtml> is the TOT estimate, (<reflink idref="bib3" id="ref94">3</reflink>) with</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>w</mi></mrow><mi>m</mi></msub><mo>=</mo><msub><mrow><mi>n</mi></mrow><mrow><mn>1</mn><mi>m</mi></mrow></msub></mrow><mo>.</mo></math> </ephtml> The upshot of (<reflink idref="bib5" id="ref95">5</reflink>) is that under matched ignorability,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> can be estimated as the TOT times the proportion of the treatment group with <emph>H</emph> = 1.</p> <p>Actually, this relationship between</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and the TOT holds in the population, regardless of how the TOT is estimated. This is because when <emph>H<subs>i</subs></emph> = 0, then</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><mo>,</mo></math> </ephtml> so the individual direct effect</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0</mn></mrow><mo>,</mo></math> </ephtml> and when <emph>H<subs>i</subs></emph> = 1,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> the effect of <emph>H</emph> on <emph>Y</emph>. Therefore,</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mi mathvariant="double-struck">E</mi><mo stretchy="true">[</mo><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><mi>H</mi><mo stretchy="true">]</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="true">[</mo><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="true">]</mo><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="italic">TOTPr</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>In the hint study, the definition of <emph>H</emph> implies that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><mi>H</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><munder><mo>∑</mo><mi>i</mi></munder><mrow><msub><mrow><mi>H</mi></mrow><mi>i</mi></msub></mrow><mo>/</mo><msub><mrow><mi>n</mi></mrow><mi>T</mi></msub><mo>=</mo></mrow></math> </ephtml> 0.3. Therefore, our estimate of the average indirect effect is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">̂</mo></mover><mo>=</mo></mrow></math> </ephtml> 0.3</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mrow><mi mathvariant="italic">TOT</mi></mrow><mo stretchy="true">̂</mo></mover></mrow><mo>.</mo></math> </ephtml> In Table 2, we estimated the TOT as −0.14 with a standard error of 0.06. That implies an average indirect effect of −0.04, with a standard error of 0.02.</p> <p>It is worth repeating that these estimates assume matched ignorability—no confounding between <emph>H</emph> and <emph>Y</emph> within matched sets. With this important caveat, this analysis suggests that the wide availability of hints actually <emph>lowers</emph> the treatment effect, that CTA1 works not because of hints, but despite them.</p> <hd id="AN0149920084-20">Estimating Log Data Direct Effects</hd> <p>What would the effect be were all students to request few hints? To answer that question, we estimate average direct effects,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><mo>.</mo></math> </ephtml> [<reflink idref="bib5" id="ref96">5</reflink>] In principle, this could be estimated as the difference in sample means</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> in the treatment group and observed</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> in the control group. In general, if we let</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Y</mi><mo>˜</mo></mover><mo>≡</mo><mi>Z</mi><mover accent="true"><mi>Y</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>Z</mi><mo stretchy="false">)</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>then we can estimate</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></math> </ephtml> as the ATE of treatment assignment <emph>Z</emph> on</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>Y</mi><mo>˜</mo></mover><mo>.</mo></math> </ephtml> Just like estimates of the ATE of <emph>Z</emph> on <emph>Y</emph>, this estimate should account for the experimental design: a paired, group-randomized trial with students clustered in schools and schools randomized within pairs.</p> <p>For the sake of simplicity, we will use an OLS regression estimator, with cluster-robust standard errors (Pustejovsky & Tipton, [<reflink idref="bib49" id="ref97">49</reflink>]), clustered at the school level. (Since in Section "An Observational Study Within an Experiment" subjects were matched within schools, school-level cluster-robust standard errors also account for the dependence of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>Y</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> between matched treated students.) Specifically, we regress:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mover accent="true"><mi>Y</mi><mo>˜</mo></mover></mrow><mi>i</mi></msub><mo>=</mo><msub><mrow><mi>α</mi></mrow><mrow><mi>p</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow></msub><mo>+</mo><mover accent="true"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="true">[</mo><mi>ξ</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo stretchy="true">]</mo></mrow><mo stretchy="true">̂</mo></mover><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>+</mo><msub><mrow><mi>ϵ</mi></mrow><mi>i</mi></msub></mrow></math> </ephtml> (<reflink idref="bib6" id="ref98">6</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>α</mi></mrow><mrow><mi>p</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo></mrow></msub></mrow></math> </ephtml> is a fixed intercept for the randomization pair of subject <emph>i</emph>'s school and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>ϵ</mi></mrow><mi>i</mi></msub></mrow></math> </ephtml> is a regression error. Using the clubSandwich package in R (Pustejovsky, [<reflink idref="bib51" id="ref99">51</reflink>]) to estimate standard errors, and 95% confidence intervals, we estimated a direct effect of 0.14 ± 0.25. Adding covariates to the regression (<reflink idref="bib6" id="ref100">6</reflink>) estimates a direct effect of 0.18 ± 0.21.</p> <p>Table 3 summarizes our mediation results: assuming matched ignorability, assignment to CTA1 increases student test scores despite the availability of hints (an indirect effect of −0.04 standard deviations), and the effect due to CTA1's other mechanisms is 0.18.</p> <p>Table 3. Estimated effects in mediation analysis.</p> <p> <ephtml> <table><thead><tr><td>Effect</td><td>Estimate</td><td>CI</td></tr></thead><tbody valign="top"><tr><td>Avg. Indirect Effect (<p id="ilm0185"><graphic href="uree_a_1823538_ilm0185.gif" content-type="Graph" /><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="double-struck">E</mi><mi>δ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math></p>)</td><td char=".">−0.04</td><td>[−0.07, −0.01]</td></tr><tr><td>Avg. Direct Effect (<p id="ilm0186"><graphic href="uree_a_1823538_ilm0186.gif" content-type="Graph" /><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="double-struck">E</mi><mi>ξ</mi></mrow></math></p>)</td><td char=".">0.14</td><td>[−0.10, 0.39]</td></tr><tr><td>Avg. Direct Effect, covariate adjusted (<p id="ilm0187"><graphic href="uree_a_1823538_ilm0187.gif" content-type="Graph" /><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="double-struck">E</mi><mi>ξ</mi></mrow></math></p>)</td><td char=".">0.18</td><td>[−0.03, 0.39]</td></tr></tbody></table> </ephtml> </p> <hd id="AN0149920084-21">Principal Stratification</hd> <p>In a principal stratification (PS; Frangakis & Rubin, [<reflink idref="bib15" id="ref101">15</reflink>]) analysis, the researcher stratifies all students in the RCT—in both the treatment and control arms—based on how they <emph>would</emph> use the software were they (perhaps counterfactually) assigned to the treatment condition. The goal is to estimate the average effect of assignment to the treatment condition separately in each stratum. In our analysis, we imagine that each student has his or her own proclivity to request hints, when possible: if assigned to treatment, each student would tend to request hints at a different rate. How does the effect of assignment to the CTA1 condition vary with these proclivities? Would the benefit of being assigned to the CTA1 condition be higher or lower for students who would tend to request more hints than for those who would tend to request fewer?</p> <p>In an RCT, identification of these varying effects relies on randomization of treatment assignment, rather than on assumptions about unmeasured confounders. Unlike the observational study or mediation analysis, PS in an RCT requires no assumptions about unmeasured confounding. PS analysis requires data on outcomes (e.g., posttest scores) in both treatment arms, as well as data on implementation. In classical PS (e.g., Feller, Grindal, et al., [<reflink idref="bib14" id="ref102">14</reflink>]; Page, [<reflink idref="bib46" id="ref103">46</reflink>]), the intermediate variable, implementation, is discrete or categorical, leading to discrete strata. Other approaches (e.g., Gilbert & Hudgens, [<reflink idref="bib19" id="ref104">19</reflink>]; Jin & Rubin, [<reflink idref="bib31" id="ref105">31</reflink>]) allow the intermediate variable to be continuous. The method we use here is based on Sales and Pane ([<reflink idref="bib60" id="ref106">60</reflink>]), in which the variable defining principal strata is latent—specifically, the "ability" parameter in (<reflink idref="bib1" id="ref107">1</reflink>). For this approach, implementation is measured with a series of binary measurements for each student, with each measurement taken from a specific problem or section. Other implementation data structures may call for different measurement models. Finally, covariates may not be strictly necessary for PS, but they can be extremely helpful.</p> <p>Below, we describe a Bayesian PS model fit in one step. However, the full process of PS analysis is a much more involved procedure. After a formal introduction in Subsection "Principal Effects for EdTech Log Data," Subsection "Specifying PS Models" describes a set of sub-models for student hint requests, posttest scores, and treatment effects. Ideally, each of these models should be carefully tailored to each data application, and each must be rigorously tested. Model checking includes the examination of residual data plots, fake data simulation, and fitting a range of alternative specifications in a sensitivity analysis. More details for checking PS models can be found in Sales and Pane ([<reflink idref="bib60" id="ref108">60</reflink>]) and Sales and Pane ([<reflink idref="bib61" id="ref109">61</reflink>]), and for Bayesian models in general in Gelman et al. ([<reflink idref="bib18" id="ref110">18</reflink>]) and McElreath ([<reflink idref="bib38" id="ref111">38</reflink>]).</p> <hd id="AN0149920084-22">Principal Effects for EdTech Log Data</hd> <p>In the PS approach,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>Y</mi><mo stretchy="false">(</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> are the only potential outcomes for <emph>Y</emph>. As in mediation analysis, hint behavior has potential outcomes as well. Specifically, the rate at which as student requests hints,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo>,</mo></math> </ephtml> has potential values</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> though only</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is relevant in the CTA1 study.</p> <p>The goal of PS is to estimate "principal effects":</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="true">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi><mo stretchy="true">]</mo></mrow></math> </ephtml> </p> <p>This is the treatment effect for subjects who <emph>would</emph> request hints at level</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi></mrow><mo>,</mo></math> </ephtml> if assigned to the treatment condition. The goal of PS is to estimate the relationship between</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and treatment effects; this contrasts with the observational study and mediation analysis whose goal was the relationship between</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> and posttest scores. Notably, although <emph>Y</emph>(0) and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> and <emph>Y</emph>(0) are never simultaneously observed, the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> estimand does not depend on any strictly-counterfactual outcomes. Unlike, say,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> <emph>Y</emph>(0) and <emph>Y</emph>(<reflink idref="bib1" id="ref112">1</reflink>) are each truly potential—they would each occur under treatment assignments <emph>Z</emph> = 0 and <emph>Z</emph> = 1. That said, estimating</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> requires estimation of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mover accent="true"><mi>h</mi><mo>¯</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><mo>,</mo></math> </ephtml> the average of <emph>Y</emph>(0) over a subset of the sample that is unobserved and must be inferred. For this reason,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> is only <emph>partially</emph> identified (e.g., Mealli et al., [<reflink idref="bib41" id="ref113">41</reflink>]); even with an infinite sample, principal effect estimates will still contain uncertainty.</p> <p>In previous sections</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> was dichotomized into <emph>H</emph>; such dichotomization is not necessary here. However,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> has some disadvantages as a continuous measure of hint usage (Sales & Pane, [<reflink idref="bib60" id="ref114">60</reflink>]). For one, since it is essentially a sample mean over a subset of each student's worked problems, much of its variance is driven by the total number of problems students worked—in particular, its extreme values belong to those students who barely used the tutor at all. Further, it does not account for the varying difficulty of the tutor's problems. In Section "Dichotomizing Hint Usage," we showed that dichotomized</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mover accent="true"><mi>h</mi><mo>¯</mo></mover></math> </ephtml> largely agrees with a dichotomized version of a more sophisticated measure of hint usage. However, when modeling hint usage continuously, the correspondence may not hold (see Sales & Pane, [<reflink idref="bib60" id="ref115">60</reflink>], for a more complete discussion).</p> <p>For those reasons, we modeled hint usage at the problem level, with Equation (<reflink idref="bib1" id="ref116">1</reflink>), but with two important differences. First, a conceptual difference: the <emph>η</emph> in (<reflink idref="bib1" id="ref117">1</reflink>) was replaced by</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> indicating that it is measuring <emph>potential</emph> hint usage—the hint usage a student would exhibit were he or she assigned to the treatment condition. Whereas <emph>η</emph> is only defined for students assigned to the treatment condition,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is defined for all the students in the study. Second, a modeling difference: instead of the normal mixture model (<reflink idref="bib2" id="ref118">2</reflink>), we modeled</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> with a normal regression model. With this change, the PS estimand became</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mo stretchy="true">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi><mo stretchy="true">]</mo></mrow></math> </ephtml> </p> <p>the treatment effect for students whose potential hint usage, if assigned to the treatment condition,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi></mrow><mo>.</mo></math> </ephtml> A full treatment of this variant of PS, including a discussion of identification and estimation, may be found in Sales and Pane ([<reflink idref="bib60" id="ref119">60</reflink>]).</p> <hd id="AN0149920084-23">Specifying PS Models</hd> <p>Estimating</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> required estimating</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi><mo stretchy="false">]</mo></mrow><mo>,</mo></math> </ephtml> despite the fact that hint requests are never observed at the same time as <emph>Y</emph>(0) (in fact</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> isn't directly observed in either the treatment or control group). However, among those students with <emph>Z</emph> = 1, hint requests are observed and the conditional distribution</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><mi mathvariant="bold-italic">x</mi></mrow></math> </ephtml> is identified; because of randomization of <emph>Z</emph>, these inferences extend to the control group as well (Feller, Grindal, et al., [<reflink idref="bib14" id="ref120">14</reflink>]).</p> <p>Our approach to PS estimation is model-based and Bayesian, following Jin and Rubin ([<reflink idref="bib31" id="ref121">31</reflink>]), Schwartz et al. ([<reflink idref="bib64" id="ref122">64</reflink>]) and others. The PS model consists of three sub-models, all depending on a vector of parameters</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi></math> </ephtml> that contains regression coefficients, variance components, and treatment effects. The sub-models are:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>h</mi><mo>;</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> giving the distribution of actual hint requests as a function of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi>p</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> giving the distribution of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> conditional on covariates, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi>Z</mi><mo>,</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo></mrow></math> </ephtml> giving the conditional distribution of outcomes. With these three models in hand, and a prior distribution</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> posterior inference for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi></math> </ephtml> proceeds based on the following structure:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mtable><mtr><mtd><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo>|</mo><mi mathvariant="bold-italic">Y</mi><mo>,</mo><mi mathvariant="bold-italic">Z</mi><mo>,</mo><mi mathvariant="bold-italic">X</mi><mo>,</mo><msub><mrow><mo>{</mo><msub><mrow><mi mathvariant="bold-italic">h</mi></mrow><mi>i</mi></msub><mo>}</mo></mrow><mrow><mi>i</mi><mo>:</mo><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>∝</mo></mtd></mtr><mtr><mtd><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><munder><mo>∏</mo><mrow><mi>i</mi><mo>:</mo><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>1</mn></mrow></munder><mrow><mrow><mo>∫</mo><mi>p</mi></mrow></mrow><mo stretchy="false">(</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo>|</mo><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mi>p</mi><mo stretchy="false">(</mo><msub><mrow><mi mathvariant="bold-italic">h</mi></mrow><mi>i</mi></msub><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mi>p</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mi>d</mi><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mo>×</mo><munder><mo>∏</mo><mrow><mi>i</mi><mo>:</mo><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>=</mo><mn>0</mn></mrow></munder><mrow><mrow><mo>∫</mo><mi>p</mi></mrow></mrow><mo stretchy="false">(</mo><msub><mrow><mi>Y</mi></mrow><mi>i</mi></msub><mo>|</mo><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mi>p</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mi>d</mi><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>.</mo></mtd></mtr></mtable></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold-italic">h</mi></mrow><mi>i</mi></msub></mrow></math> </ephtml> is the vector of hint request data for subject <emph>i</emph>, ranging over all challenging problems. In other words, we estimate parameters by integrating over unknown (latent)</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> values, using the distribution of <emph>η</emph> estimated in the treatment group. This is, in essence, an infinite mixture distribution, with outcome distribution</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> and mixing proportions</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>|</mo><mo>·</mo><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> Unlike in typical PS setups,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is unobserved for both <emph>Z</emph> = 1 and <emph>Z</emph> = 0 treatment groups, but is estimated using both <bold><emph>h</emph></bold> and <bold><emph>x</emph></bold> in the treatment group, but only <bold><emph>x</emph></bold> in the control group.</p> <p>First, students' proclivity to request hints is measured by the <emph>η</emph> parameter in (<reflink idref="bib1" id="ref123">1</reflink>). In the next level,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is modeled as a function of baseline covariates <bold><emph>x</emph></bold>:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="true">|</mo><mrow><mo stretchy="true">(</mo><mrow><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi mathvariant="bold-italic">θ</mi></mrow><mo stretchy="true">)</mo></mrow><mo>∼</mo><mi mathvariant="script">N</mi><mrow><mo stretchy="true">(</mo><mrow><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi mathvariant="bold-italic">i</mi></msub><mi mathvariant="bold-italic">β</mi><mo>+</mo><msubsup><mrow><mi>ϵ</mi></mrow><mrow><mi>t</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow><mrow><mi mathvariant="italic">tch</mi></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>ϵ</mi></mrow><mrow><mi>s</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow><mrow><mi mathvariant="italic">scl</mi></mrow></msubsup><mo>,</mo><mi>σ</mi></mrow><mo stretchy="true">)</mo></mrow></mrow></math> </ephtml> (<reflink idref="bib7" id="ref124">7</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">β</mi></math> </ephtml> is a vector of coefficients. Since students were nested within teachers, who were nested within schools, we included normally-distributed school (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">scl</mi></mrow></msup></mrow></math> </ephtml> ) and teacher (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>ϵ</mi></mrow><mrow><mi mathvariant="italic">tch</mi></mrow></msup></mrow></math> </ephtml> ) random intercepts. The covariates in the model,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub></mrow><mo>,</mo></math> </ephtml> were detailed in Table 1; preliminary model checking suggested including a quadratic term for pretest, which was added as a column of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub></mrow><mo>.</mo></math> </ephtml> </p> <p>We modeled students' posttest scores <emph>Y</emph> as conditionally normal:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="true">|</mo><mrow><mo stretchy="true">(</mo><mrow><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo>,</mo><msub><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi></msub><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo stretchy="true">)</mo></mrow><mo>∼</mo><mi mathvariant="script">N</mi><mrow><mo stretchy="true">(</mo><mrow><msub><mrow><mi>γ</mi></mrow><mrow><mn>0</mn><mi>b</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow></msub><mo>+</mo><msubsup><mrow><mi mathvariant="bold-italic">x</mi></mrow><mi>i</mi><mi>T</mi></msubsup><mi mathvariant="bold-italic">γ</mi><mo>+</mo><mi>a</mi><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>+</mo><msub><mrow><mi>Z</mi></mrow><mi>i</mi></msub><mo stretchy="false">(</mo><msub><mrow><mi>b</mi></mrow><mn>0</mn></msub><mo>+</mo><msub><mrow><mi>b</mi></mrow><mn>1</mn></msub><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>+</mo><msubsup><mrow><mi>ζ</mi></mrow><mrow><mi>t</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow><mrow><mi mathvariant="italic">tch</mi></mrow></msubsup><mo>+</mo><msubsup><mrow><mi>ζ</mi></mrow><mrow><mi>s</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow><mrow><mi mathvariant="italic">scl</mi></mrow></msubsup><mo>,</mo><msub><mrow><mi>ω</mi></mrow><mrow><mi>Z</mi><mo stretchy="true">[</mo><mi>i</mi><mo stretchy="true">]</mo></mrow></msub></mrow><mo stretchy="true">)</mo></mrow></mrow></math> </ephtml> (<reflink idref="bib8" id="ref125">8</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>γ</mi></mrow><mrow><mn>0</mn><mi>b</mi><mo stretchy="false">[</mo><mi>i</mi><mo stretchy="false">]</mo></mrow></msub></mrow></math> </ephtml> is a fixed effect for <emph>i</emph>'s randomization block,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">γ</mi></math> </ephtml> are the covariate coefficients, and <emph>ζ<sups>tch</sups></emph>, and <emph>ζ<sups>scl</sups></emph> are normally-distributed teacher and school random intercepts. The residual variance <emph>ω</emph> varies with treatment assignment <emph>Z</emph>; this captures measurement error in <emph>Y</emph>, treatment effect heterogeneity that is not linearly related to</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> and other between-student variation in <emph>Y</emph> that is not predicted by the mean model.</p> <p>Model (<reflink idref="bib8" id="ref126">8</reflink>) implies that treatment effects are linear in</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> </p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mi>τ</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi>b</mi></mrow><mn>0</mn></msub><mo>+</mo><msub><mrow><mi>b</mi></mrow><mn>1</mn></msub><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> </p> <p>While more complex models for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>τ</mi><mo stretchy="false">(</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> are theoretically possible (for instance, Jin and Rubin ([<reflink idref="bib31" id="ref127">31</reflink>]) use a quadratic model), in our experience non-linear effect models do not perform as well in model checks as the linear model.</p> <p>Covariates <bold><emph>X</emph></bold> were standardized prior to fitting. Prior distributions for the block fixed effects</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>β</mi></mrow><mi>b</mi><mi>Y</mi></msubsup></mrow></math> </ephtml> and covariate coefficients</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi mathvariant="bold-italic">β</mi></mrow><mi mathvariant="bold-italic">Y</mi></msup></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi mathvariant="bold-italic">β</mi></mrow><mi mathvariant="bold-italic">M</mi></msup></mrow></math> </ephtml> were normal with mean zero and standard deviation 2; priors for treatment effects and the coefficient on</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> were standard normal. The rest of the parameters received standard reference priors. In all cases, we expected true parameter values to be much smaller in magnitude than the prior standard deviation.</p> <hd id="AN0149920084-24">Estimating Principal Effects</hd> <p>Figure 1 shows the estimated linear function</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> The left panel shows the CTA1 treatment effect,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>−</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo>=</mo><mi>τ</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> varies; the black line shows the estimate (i.e., posterior mean) and the magenta lines are random draws from the posterior distribution, showing estimation uncertainty. Seventy-five percent of posterior draws of the slope of the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> function were positive, implying a posterior probability of 0.75 that students who asked for hints with greater regularity benefited more from the CTA1 curriculum. A 95% highest-density credible interval for the slope parameter is [−0.06, 0.12]. Therefore, the data are consistent with either a slightly negative or a positive relationship between hint requests and treatment effects, but the latter is more likely.</p> <p>PHOTO (COLOR): Figure 1. Results from the PS model. On the left, the estimated CTA1 treatment effect is plotted against hint request proclivity η(<reflink idref="bib1" id="ref128">1</reflink>). The black line is the posterior mean, and the red lines are random posterior draws. The right panel plots observed posttest scores against one random draw of η(<reflink idref="bib1" id="ref129">1</reflink>), along with regression lines for the treatment and control groups.</p> <p>The right-hand panel plots one posterior draw of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> alongside observed outcomes <emph>Y</emph>. The figure also includes estimated regression lines for the two treatment groups. Although, as observed in Sections "An Observational Study Within an Experiment" and "Mediation Analysis,"</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is anticorrelated with <emph>Y</emph>, the slope is steeper in the control group than in the treatment group. The distance between the two regression lines is the treatment effect, which grows with</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> </p> <p>These results suggest that students who, under treatment, would tend to ask for hints on challenging problems, are most responsive to treatment.</p> <hd id="AN0149920084-25">Comparing Strategies</hd> <p>In the case discussed here, the observational study and mediation approaches appear to give the opposite conclusion from principal stratification—matching and mediation suggest that excessive hint requests hurt students' posttest scores, while principal stratification suggests that students who request more hints may experience higher treatment effects. This section will compare and contrast the three methods in terms of their necessary identification assumptions, their model fitting mechanics, and the interpretation of their results.</p> <p>The fundamental difference between the observational study and mediation approach, on the one hand, and principal stratification on the other, is in the role of <emph>H</emph>, the intermediate variable. An observational study or mediation analysis treats <emph>H</emph> as a causal agent, an intervention or exposure that affects <emph>Y</emph>. In mediation analysis, it is also an (intermediate) outcome, affected by <emph>Z</emph>. The approach of principal stratification is quite nearly opposite: <emph>H</emph> is not an exposure, that "happens to" subjects, but rather its potential value <emph>H</emph>(<reflink idref="bib1" id="ref130">1</reflink>) is a characteristic of the subjects. Observed <emph>H</emph> reveals what was there all along.</p> <hd id="AN0149920084-26">Causal Identification</hd> <p>Observational studies, and mediation analyses that build on them, require ignorability assumptions such as Strong Ignorability or Matched Ignorability. These may be particularly problematic in our study since students who are more likely to request hints are also more likely to struggle with the material. We addressed this concern in three ways: first, we focused our attention on the set of problems in the tutor in which students either requested a hint or made an error, second, with propensity score matching, and third with sensitivity analysis. None of these approaches is bulletproof—they each require further assumptions about how students request hints. Eliminating problems in which students showed no signs of struggling is likely to reduce the association between hint requests and underlying academic ability, but not eliminate it. For instance, stronger students may be better able to figure out a difficult problem without access to hints or error feedback than weaker students. Matching depends both on the adequacy of the observed covariates to capture baseline differences between <emph>H</emph> = 0 and <emph>H</emph> = 1 students, and the sensitivity analyses we performed assumed that any un-measured covariate would predict <emph>H</emph> and <emph>Y</emph> no better than pretest or ethnicity indicators, respectively. These assumptions are impossible to verify from the data at hand; instead, judging their plausibility requires a keen understanding of student practice within the tutor.</p> <p>Mediation analysis builds on the matching design, and requires the same ignorability assumption; it must additionally overcome the hurdle of the almost complete lack of overlap between hint request behavior in the control group (i.e., zero) and in the treatment group. Under our approach of dichotomizing hint request behavior, mediation analysis is essentially a way to interpret the results of the observational study in the context of the full RCT.</p> <p>In contrast, no ignorability assumptions are necessary for principal stratification. However, principal stratification estimators require modeling assumptions—most problematically,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> of (5.2), the model for <emph>Y</emph>(0) conditional on</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> <bold> <emph>x</emph> </bold>, and parameters</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi><mo>.</mo></math> </ephtml> While any model specification can, and should, be checked against observed data, this process can only narrow the space of acceptable models, it cannot be used to determine the correct form of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> This is because <emph>Y</emph>(0) and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> are never observed simultaneously. Therefore, it is possible for an analyst to misspecify</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> such that principal effect estimates are severely biased, without being able to detect the model misspecification with the data. That is, the data will typically be unable to distinguish between alternative</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> models that lead to qualitatively different conclusions. In practice, we make the untestable assumption that</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>0</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow></math> </ephtml> is of the same form as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>p</mi><mo stretchy="false">(</mo><mi>Y</mi><mo>|</mo><mi mathvariant="bold-italic">x</mi><mo>,</mo><mi>Z</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi mathvariant="bold-italic">θ</mi><mo>,</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> </p> <p>An additional contrast of note between matching-based studies and principal stratification relates to transparency. However complex the process of devising a match may be, the resulting design is simple and clear—subjects with <emph>H</emph> = 1 are compared against matches with <emph>H</emph> = 0. This transparency may help researchers assess the plausibility of the match, and diagnose potential problems with resulting estimates. Though principal stratification does not depend on ignorability, it does depend on a highly intricate and parametric modeling structure. If any part of the model is sufficiently misspecified, the results may be wrong. Perhaps worse, Feller, Greif, et al. ([<reflink idref="bib13" id="ref131">13</reflink>]) argued that even well-specified models can give severely biased estimates. Extensive model interrogation and checking are necessary for principal stratification analysis, but the complexity of the principal stratification model makes this process particularly difficult. Sales and Pane ([<reflink idref="bib60" id="ref132">60</reflink>]) gives some examples of potentially fruitful model checking procedures, which are also carried out in the online supplement.</p> <hd id="AN0149920084-27">Results and Interpretation</hd> <p>Our observational study suggested that hint requests have a negative effect: students who requested hints more often than the 70th percentile score lower on the posttest, after controlling for observed covariates. Taken at face value, this suggests that the optimal strategy is to request few, if any, hints. However, this conclusion assumes that there were no omitted confounders. An omitted confounder that is as important as pretest could explain the relationship, but an omitted confounder as important as ethnicity—our second most important covariate—could not (so long as we estimate an OLS-weighted average effect).</p> <p>Mediation analysis essentially interprets the observational study result in terms of the overall CTA1 treatment effect. Specifically, the analysis in 4 showed that CTA1 affected students' posttest scores in (at least) two opposite ways: by allowing hints, it lowered their posttest scores, but its other mechanisms increased their scores by an even greater amount.</p> <p>The principal stratification analysis also found a negative correlation between hint requesting and posttest scores. Those students who request more hints are those who need more help of some form and tend to score lower on the posttest. Fortunately, the principal stratification analysis also suggests (with probability 0.75) that these students may benefit more from CTA1 than their peers who request fewer hints. However, it is unclear whether their larger treatment effects are due to their higher rate of hint requests, or to some other related characteristic.</p> <p>Can these three sets of results be reconciled? Does requesting hints help or hurt? Technically, principal stratification has nothing to say about the effect of hints—</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> is modeled as a student characteristic, not an intervention (see Jin & Rubin, [<reflink idref="bib31" id="ref133">31</reflink>] for a nice discussion of this point). Still, does not the correlation between treatment effects and hint requests suggest that hints may be beneficial? Does that contradict the results in Sections "An Observational Study Within an Experiment" and "Mediation Analysis" that hints requests lower students' posttest scores? In fact, the results of the three methods can be reconciled. For instance, it may be that requesting hints indeed lowers test scores—a negative indirect effect—but that those students who are likely to request more hints also tend to experience greater direct effects. That is, it may be that the observational and mediational results are correct—that requesting hints lowers test scores—but that the principal stratification results are also correct—students who request more hints tend to experience higher treatment effects, due to other mechanisms. (Of course, the data are also consistent with the possibilities that the relationship between hint requests and treatment effects is negative, or close to zero, in which case there is less that needs to be explained.)</p> <hd id="AN0149920084-28">Choosing Between Methods</hd> <p>The previous discussion suggests three criteria for choosing between the three approaches we've discussed here. First is the question the researcher seeks to answer. In their most straightforward interpretations, principal stratification is a method for examining treatment effect heterogeneity (are students benefiting differently?), whereas mediation analysis is a method for assessing causal mechanisms (is the wide availability of hints in CTA1 beneficial to students?). That said, principal stratification may also shed light on potential causal mechanisms—if, indeed, treatment effects are higher for students who request more hints, then hints may play a role in the tutor's effectiveness.</p> <p>A second criterion is the researcher's willingness to make ignorability assumptions. A researcher's confidence in her understanding of a data generating process and the quality of observed covariates can translate into confidence about the results of an observational study or mediation analysis. Conversely, researchers unwilling to consider untestable ignorability assumptions will find principal stratification an attractive alternative.</p> <p>The final criterion is a researcher's comfort with complex statistical models. Researchers who are able and willing to experiment with a range of models for a dataset, and perhaps devise tests of model fit tailored to a specific problem, may be more confident in the fit of a principal stratification model. Researchers who prefer transparency and non-parametric or semi-parametric analysis may prefer matching studies.</p> <p>Additional philosophical concerns come in to play as well. For instance, unlike observational studies and principal stratification, mediation analysis depends on strictly counterfactual quantities, such as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mi>H</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> that can never occur. Analogous quantities in principal stratification, such as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>Y</mi><mo stretchy="false">(</mo><mn>0</mn><mo stretchy="false">)</mo><mo>|</mo><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></math> </ephtml> are unobservable, but nevertheless refer to averages of potential outcomes among actual experimental subjects. These concerns are important but beyond the scope of the current paper; for more discussion, see the citations in the introduction.</p> <p>In summary, there is much to recommend the observational study approach if risks of confounding bias appear limited (due to the availability of high-quality covariates, good reason to believe variation in implementation is random or haphazard, or other factors). A wide variety of straightforward and transparent estimation techniques are available, such as the propensity score matching illustrated here. The result—the ATE of implementation on the outcome—is an intuitive estimand.</p> <p>When employed, mediation analysis can contextualize the estimated implementation effect from the observational study in terms of the experiment's overall treatment effect, enabling implementation to be interpreted as a causal mechanism.</p> <p>However, where unobserved confounding is a serious threat, and complex statistical modeling (and model checking) is not a practical barrier, principal stratification may be a better choice because no ignorability assumptions are necessary.</p> <p>Of course, there are approaches to modeling implementation beyond the three considered here. For instance, principal stratification based on non-parametric bounding (Miratrix et al., [<reflink idref="bib43" id="ref134">43</reflink>]) or randomization inference (Nolen & Hudgens, [<reflink idref="bib45" id="ref135">45</reflink>]) would avoid the parametric assumptions of model-based principal stratification, as would a standard moderation analysis based on predicted implementation,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mover accent="true"><mi>η</mi><mo>̂</mo></mover><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> instead of partially observed</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>η</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>.</mo></math> </ephtml> We hope that future work will develop these methods, and still others.</p> <hd id="AN0149920084-29">Discussion: Causal Inference and Measurement</hd> <p>This case study has focused on three approaches to causal modeling of hint requests during an RCT of the Cognitive Tutor program. We hope we have demonstrated some of the potential and some of the challenges that new datasets logged by online learning systems bring to these old problems.</p> <p>Measurement of students' hint request rates played a central role in all three approaches we considered. The results were largely driven by three measurement decisions. First, our decision to consider only worked Cognitive Tutor problems on which students either requested a hint or made an error (or both) reduced the relationship between hint requests and student algebra I ability. Second, our decision to dichotomize hint request rates in the observational study and mediation analysis allowed us to use of a traditional matching estimator and to identify mediational estimands. Third, our decision to use a Rasch model to measure hint usage in the principal stratification analysis accounted for differences between students in both the number and difficulty of worked CT problems.</p> <p>While questions of measurement has always been important in causal inference, analysis of EdTech log data brings them to the fore. Log datasets from technology products are large, multivariate, and complex, so careful thought is necessary in order to measure implementation constructs of interest.</p> <p>Even if they are motivated by methodological concerns, decisions about measurement are inherently also decisions about causal questions—estimates using different measurements answer different questions. Just as researchers must choose a causal approach, such as matching, mediation, or principal stratification, they must choose a measurement approach as well. As our case study demonstrated, these two decisions are deeply intertwined.</p> <p>The size, dimension, and complexity of EdTech log data suggests some exciting opportunities for innovative combinations of measurement and causal approaches. We demonstrated the inclusion of an IRT model in principal stratification; this suggests the possibility of including other measurement models into causal estimators. Multivariate measurement models that incorporate not only hint requests, but other usage measures such as time spent, actions taken, and errors could yield deep insights about how students use EdTech products and how different usage patterns correspond to different effects. The development of these causal models will require simultaneous consideration of causal inference and measurement.</p> <ref id="AN0149920084-30"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref22" type="bt">1</bibl> <bibtext> This subsection draws heavily on comments from an anonymous reviewer.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref21" type="bt">2</bibl> <bibtext> Technically, if <emph>h</emph> is the event that a student requests a hint on a problem and <emph>e</emph> is the event that the student makes an error, then</bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><mi>h</mi><mo>|</mo><mi>h</mi><mi mathvariant="normal" /><mtext>or</mtext><mi mathvariant="normal" /><mi>e</mi><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>/</mo><mo>{</mo><mn>1</mn><mo>+</mo><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><mi>e</mi><mi mathvariant="normal" /><mtext>and</mtext><mi mathvariant="normal" /><mtext>not</mtext><mi mathvariant="normal" /><mi>h</mi><mo stretchy="false">)</mo><mo>/</mo><mi>P</mi><mi>r</mi><mo stretchy="false">(</mo><mi>h</mi><mo stretchy="false">)</mo><mo>}</mo></mrow><mo>.</mo></math> </ephtml> </bibtext> </blist> <blist> <bibl id="bib3" idref="ref1" type="bt">3</bibl> <bibtext> The principal stratification model was re-run without dropping sections, and after dropping sections worked by fewer than 500 students, with similar results.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref19" type="bt">4</bibl> <bibtext> Including these students in a principal stratification model is straightforward (Sales & Pane, [60]). Including subjects with missing log data in a mediation or observational study design can be more problematic (see, e.g. Li & Zhou, [37]).</bibtext> </blist> <blist> <bibl id="bib5" idref="ref69" type="bt">5</bibl> <bibtext> This is equivalent to the "controlled direct effect," <emph>CDE</emph>(0) e.g. VanderWeele ([72], p. 57).</bibtext> </blist> <blist> <bibl id="bib6" idref="ref33" type="bt">6</bibl> <bibtext> Adam C. Sales is now affiliated with Worcester Polytechnic Institute, Worcester, Massachusetts, USA.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref57" type="bt">7</bibl> <bibtext> Supplemental data for this article can be accessed at https://doi.org/10.1080/19345747.2020.1823538.</bibtext> </blist> </ref> <ref id="AN0149920084-31"> <title> References </title> <blist> <bibtext> Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26 (1), 205 – 223. https://doi.org/10.1007/s40593-015-0089-1</bibtext> </blist> <blist> <bibtext> Anderson, J. R. (2013). The architecture of cognition. Psychology Press.</bibtext> </blist> <blist> <bibtext> Anderson, J. R., Boyle, C. F., & Reiser, B. J. (1985). Intelligent tutoring systems. Science, 228 (4698), 456 – 462. https://doi.org/10.1126/science.228.4698.456</bibtext> </blist> <blist> <bibtext> Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4 (2), 167 – 207. https://doi.org/10.1207/s15327809jls0402_2</bibtext> </blist> <blist> <bibtext> Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67 (1), 1 – 48. https://doi.org/10.18637/jss.v067.i01</bibtext> </blist> <blist> <bibtext> Beck, J. E., Chang, K.-M., Mostow, J., & Corbett, A. (2008). Does help help? introducing the Bayesian evaluation and assessment methodology. In International conference on intelligent tutoring systems (pp. 383 – 394). Springer.</bibtext> </blist> <blist> <bibtext> Bowers, J., Fredrickson, M., & Hansen, B. (2017). RItools: Randomization inference tools (development version) [R package version 0.2-0]. https://github.com/markmfredrickson/RItools</bibtext> </blist> <blist> <bibl id="bib8" idref="ref73" type="bt">8</bibl> <bibtext> Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22 (1), 31 – 72.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref82" type="bt">9</bibl> <bibtext> Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27 (4), 724 – 750. https://doi.org/10.1002/pam.20375</bibtext> </blist> <blist> <bibtext> Cook, T. D., Steiner, P. M., & Pohl, S. (2009). How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: Results from two types of within-study comparisons. Multivariate Behavioral Research, 44 (6), 828 – 847. https://doi.org/10.1080/00273170903333673</bibtext> </blist> <blist> <bibtext> Escueta, M., Quan, V., Nickow, A. J., & Oreopoulos, P. (2017). Education technology: An evidence-based review. Technical report, National Bureau of Economic Research.</bibtext> </blist> <blist> <bibtext> Fancsali, S. E., Holstein, K., Sandbothe, M., Ritter, S., McLaren, B. M., & Aleven, V. (2020). Towards practical detection of unproductive struggle. Proceedings of the 21st International Conference on Artificial Intelligence in Education.</bibtext> </blist> <blist> <bibtext> Feller, A., Greif, E., Miratrix, L., & Pillai, N. (2016). Principal stratification in the twilight zone: Weakly separated components in finite mixture models. arXiv Preprint arXiv:1602.06595.</bibtext> </blist> <blist> <bibtext> Feller, A., Grindal, T., Miratrix, L., & Page, L. C. (2016). Compared to what? Variation in the impacts of early childhood education by alternative care type. The Annals of Applied Statistics, 10 (3), 1245 – 1285.</bibtext> </blist> <blist> <bibtext> Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58 (1), 21 – 29. https://doi.org/10.1111/j.0006-341x.2002.00021.x</bibtext> </blist> <blist> <bibtext> Fred, G. P., & Van Merriënboer, J. J. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology, 86 (1), 122.</bibtext> </blist> <blist> <bibtext> Gabry, J. (2016). bayesplot: Plotting for Bayesian Models [R package version 1.1.0.]. https://CRAN.R-project.org/package=bayesplot</bibtext> </blist> <blist> <bibtext> Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2014). Bayesian data analysis (Vol. 2). CRC Press.</bibtext> </blist> <blist> <bibtext> Gilbert, P. B., & Hudgens, M. G. (2008). Evaluating candidate principal surrogate endpoints. Biometrics, 64 (4), 1146 – 1154. https://doi.org/10.1111/j.1541-0420.2008.01014.x</bibtext> </blist> <blist> <bibtext> Goldin, I. M., & Koedinger, K. R. (2012). Vincent Aleven. Learner differences in hint processing. International Educational Data Mining Society.</bibtext> </blist> <blist> <bibtext> Hansen, B. B. (2004). Full matching in an observational study of coaching for the sat. Journal of the American Statistical Association, 99 (467), 609 – 618.</bibtext> </blist> <blist> <bibtext> Hansen, B. B. (2011). Propensity score matching to extract latent experiments from nonexperimental data: A case study. In N. Dorans & S. Sinharay (Eds.), Looking back: Proceedings of a conference in honor of Paul W. Holland (Chapter 9, pp. 149 – 181). Springer.</bibtext> </blist> <blist> <bibtext> Hansen, B. B., & Bowers, J. (2008). Covariate balance in simple, stratified and clustered comparative studies. Statistical Science, 23 (2), 219 – 236.</bibtext> </blist> <blist> <bibtext> Hansen, B. B., & Klopfer, S. O. (2006). Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics, 15 (3), 609 – 627.</bibtext> </blist> <blist> <bibtext> Hirano, K., & Imbens, G. W. (2004). The propensity score with continuous treatments. Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives, 226164, 73 – 84.</bibtext> </blist> <blist> <bibtext> Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15 (3), 199 – 236. <ulink href="http://gking.harvard.edu/files/matchp.pdf">http://gking.harvard.edu/files/matchp.pdf</ulink></bibtext> </blist> <blist> <bibtext> Hong, G. (2015). Causality in a social world: Moderation, mediation and spill-over. John Wiley & Sons.</bibtext> </blist> <blist> <bibtext> Hosman, C. A., Hansen, B. B., & Holland, P. W. (2010). The sensitivity of linear regression coefficients' confidence limits to the omission of a confounder. The Annals of Applied Statistics, 4 (2), 849 – 870.</bibtext> </blist> <blist> <bibtext> Imai, K., Keele, L., Tingley, D., & Yamamoto, T. (2011). Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review, 105 (4), 765 – 789. https://doi.org/10.1017/S0003055411000414</bibtext> </blist> <blist> <bibtext> Israni, A., Sales, A. C., & Pane, J. F. (2018). Mastery learning in practice: A (mostly) descriptive analysis of log data from the cognitive tutor algebra I effectiveness trial. arXiv:1802.08616.</bibtext> </blist> <blist> <bibtext> Jin, H., & Rubin, D. B. (2008). Principal stratification for causal inference with extended partial compliance. Journal of the American Statistical Association, 103 (481), 101 – 111.</bibtext> </blist> <blist> <bibtext> Jo, B. (2008). Causal inference in randomized experiments with mediational processes. Psychological Methods, 13 (4), 314 – 336. https://doi.org/10.1037/a0014207</bibtext> </blist> <blist> <bibtext> Kalton, G. (1968). Standardization: A technique to control for extraneous variables. Applied Statistics, 17 (2), 118 – 136.</bibtext> </blist> <blist> <bibtext> Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19 (3), 239 – 264.</bibtext> </blist> <blist> <bibtext> Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36 (5), 757 – 798. https://doi.org/10.1111/j.1551-6709.2012.01245.x</bibtext> </blist> <blist> <bibtext> Leon, A. C., & Hedeker, D. (2005). A mixed-effects quintile-stratified propensity adjustment for effectiveness analyses of ordered categorical doses. Statistics in Medicine, 24 (4), 647 – 658. https://doi.org/10.1002/sim.2042</bibtext> </blist> <blist> <bibtext> Li, W., & Zhou, X.-H. (2017). Identifiability and estimation of causal mediation effects with missing data. Statistics in Medicine, 36 (25), 3948 – 3965. https://doi.org/10.1002/sim.7413</bibtext> </blist> <blist> <bibtext> McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC Press.</bibtext> </blist> <blist> <bibtext> McKendree, J. (1990). Effective feedback content for tutoring complex skills. Human-Computer Interaction, 5 (4), 381 – 413.</bibtext> </blist> <blist> <bibtext> Mealli, F., & Mattei, A. (2012). A refreshing account of principal stratification. The International Journal of Biostatistics, 8 (1), 8.</bibtext> </blist> <blist> <bibtext> Mealli, F., Pacini, B., & Stanghellini, E. (2016). Identification of principal causal effects using additional outcomes in concentration graphs. Journal of Educational and Behavioral Statistics, 41 (5), 463 – 480.</bibtext> </blist> <blist> <bibtext> Michael, J. L., & Gutman, R. (2017). Estimation of causal effects with multiple treatments: A review and new ideas. Statistical Science, 32 (3), 432 – 454.</bibtext> </blist> <blist> <bibtext> Miratrix, L., Furey, J., Feller, A., Grindal, T., & Page, L. C. (2018). Bounding, an accessible method for estimating principal causal effects, examined and explained. Journal of Research on Educational Effectiveness, Pages, 11 (1), 133 – 130.</bibtext> </blist> <blist> <bibtext> Neyman, J. (1923). Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes. Roczniki Nauk Rolniczych, 10, 1 – 51.</bibtext> </blist> <blist> <bibtext> Nolen, T. L., & Hudgens, M. G. (2011). Randomization-based inference within principal strata. Journal of the American Statistical Association, 106 (494), 581 – 593. https://doi.org/10.1198/jasa.2011.tm10356</bibtext> </blist> <blist> <bibtext> Page, L. C. (2012). Principal stratification as a framework for investigating mediational processes in experimental settings. Journal of Research on Educational Effectiveness, 5 (3), 215 – 244.</bibtext> </blist> <blist> <bibtext> Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor algebra I at scale. Educational Evaluation and Policy Analysis, 36 (2), 127 – 144.</bibtext> </blist> <blist> <bibtext> Pavlik, P. I., Jr, Cen, H., & Koedinger, K. R. (2009). Performance factors analysis – A new alternative to knowledge tracing. Proceedings of the 2009 Conference on Artificial Intelligence in Education (pp. 531–538).</bibtext> </blist> <blist> <bibtext> Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business & Economic Statistics, 36 (4), 672 – 683.</bibtext> </blist> <blist> <bibtext> Pearl, J. (2011). Principal stratification-a goal or a tool? The International Journal of Biostatistics, 7 (1), 20 – 13. https://doi.org/10.2202/1557-4679.1322</bibtext> </blist> <blist> <bibtext> Pustejovsky, J. (2018). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections [R package version 0.3.2]. https://CRAN.R-project.org/package=clubSandwich.</bibtext> </blist> <blist> <bibtext> R Core Team. (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://<ulink href="http://www.R-project.org/">www.R-project.org/</ulink>.</bibtext> </blist> <blist> <bibtext> Rasch, G. (1993). Probabilistic models for some intelligence and attainment tests. ERIC.</bibtext> </blist> <blist> <bibtext> Rosenbaum, P. R. (2002). Observational studies. Springer.</bibtext> </blist> <blist> <bibtext> Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70 (1), 41 – 55.</bibtext> </blist> <blist> <bibtext> Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. The Annals of Statistics, 6 (1), 34 – 58.</bibtext> </blist> <blist> <bibtext> Rubin, D. B. (2004). Direct and indirect causal effects via potential outcomes. Scandinavian Journal of Statistics, 31 (2), 161 – 170.</bibtext> </blist> <blist> <bibtext> Sales, A., & Pane, J. (2020). The effect of teachers reassigning students to new cognitive tutor sections. Proceedings of the 13th International Conference on Educational Data Mining. International Educational Data Mining Society.</bibtext> </blist> <blist> <bibtext> Sales, A. C., Hansen, B. B., & Rowan, B. (2018). Rebar: Reinforcing a matching estimator with predictions from high-dimensional covariates. Journal of Educational and Behavioral Statistics, 43 (1), 3 – 31. https://doi.org/10.3102/1076998617731518</bibtext> </blist> <blist> <bibtext> Sales, A. C., & Pane, J. F. (2019 a). The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. The Annals of Applied Statistics, 13 (1), 420 – 443. https://doi.org/10.1214/18-AOAS1196</bibtext> </blist> <blist> <bibtext> Sales, A. C., & Pane, J. F. (2019 b). Principal stratification for intelligent tutors: A tutorial. https://github.com/adamSales/tutorial/blob/master/paper.pdf.</bibtext> </blist> <blist> <bibtext> Sales, A., Wilks, A., & Pane, J. (2016). Student usage predicts treatment effect heterogeneity in the cognitive tutor algebra I program. Proceedings of the 9th International Conference on Educational Data Mining (pp. 207–214). International Educational Data Mining Society.</bibtext> </blist> <blist> <bibtext> Schochet, P. Z. (2015). Statistical theory for the RCT-YES software: Design-based causal inference for RCTs. Mathematica Policy Research.</bibtext> </blist> <blist> <bibtext> Schwartz, S. L., Li, F., & Mealli, F. (2011). A Bayesian semiparametric approach to intermediate variables in causal inference. Journal of the American Statistical Association, 106 (496), 1331 – 1344.</bibtext> </blist> <blist> <bibtext> Shadish, W. R., & Steiner, P. M. (2010). A primer on propensity score analysis. Newborn and Infant Nursing Reviews, 10 (1), 19 – 26.</bibtext> </blist> <blist> <bibtext> Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N. T., Razzaq, L., Dailey, M. D., O'Connor, C., & Mulcahy, C. (2011). Feedback during web-based homework: the role of hints (pp. 328–336). International Conference on Artificial Intelligence in Education. Springer.</bibtext> </blist> <blist> <bibtext> Stekhoven, D. J., & Bühlmann, P. (2012). MissForest-non-parametric missing value imputation for mixed-type data. Bioinformatics, 28 (1), 112 – 118. https://doi.org/10.1093/bioinformatics/btr597</bibtext> </blist> <blist> <bibtext> Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2 (1), 59 – 89.</bibtext> </blist> <blist> <bibtext> VanderWeele, T. J. (2008). Simple relations between principal stratification and direct and indirect effects. Statistics & Probability Letters, 78 (17), 2957 – 2962.</bibtext> </blist> <blist> <bibtext> VanderWeele, T. J. (2011). Principal stratification – Uses and limitations. The International Journal of Biostatistics, 7 (1), 1 – 14.</bibtext> </blist> <blist> <bibtext> VanderWeele, T. J. (2012). Comments: Should principal stratification be used to study mediational processes? Journal of Research on Educational Effectiveness, 5 (3), 245 – 249. https://doi.org/10.1080/19345747.2012.688412</bibtext> </blist> <blist> <bibtext> VanderWeele, T. (2015). Explanation in causal inference: Methods for mediation and interaction. Oxford University Press.</bibtext> </blist> <blist> <bibtext> Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11 (10), 1 – 17. <ulink href="http://www.jstatsoft.org/v11/i10/">http://www.jstatsoft.org/v11/i10/</ulink>.</bibtext> </blist> </ref> <aug> <p>By Adam C. Sales and John F. Pane</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib47" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib11" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib62" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib60" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib58" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib54" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib27" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib29" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib72" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib15" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib40" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib50" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib57" firstref="ref14"></nolink> <nolink nlid="nl14" bibid="bib70" firstref="ref15"></nolink> <nolink nlid="nl15" bibid="bib71" firstref="ref16"></nolink> <nolink nlid="nl16" bibid="bib69" firstref="ref17"></nolink> <nolink nlid="nl17" bibid="bib32" firstref="ref18"></nolink> <nolink nlid="nl18" bibid="bib35" firstref="ref25"></nolink> <nolink nlid="nl19" bibid="bib68" firstref="ref26"></nolink> <nolink nlid="nl20" bibid="bib39" firstref="ref27"></nolink> <nolink nlid="nl21" bibid="bib34" firstref="ref28"></nolink> <nolink nlid="nl22" bibid="bib16" firstref="ref29"></nolink> <nolink nlid="nl23" bibid="bib20" firstref="ref30"></nolink> <nolink nlid="nl24" bibid="bib66" firstref="ref32"></nolink> <nolink nlid="nl25" bibid="bib53" firstref="ref39"></nolink> <nolink nlid="nl26" bibid="bib12" firstref="ref41"></nolink> <nolink nlid="nl27" bibid="bib48" firstref="ref45"></nolink> <nolink nlid="nl28" bibid="bib108" firstref="ref52"></nolink> <nolink nlid="nl29" bibid="bib33" firstref="ref54"></nolink> <nolink nlid="nl30" bibid="bib52" firstref="ref55"></nolink> <nolink nlid="nl31" bibid="bib67" firstref="ref56"></nolink> <nolink nlid="nl32" bibid="bib25" firstref="ref58"></nolink> <nolink nlid="nl33" bibid="bib36" firstref="ref59"></nolink> <nolink nlid="nl34" bibid="bib42" firstref="ref60"></nolink> <nolink nlid="nl35" bibid="bib22" firstref="ref62"></nolink> <nolink nlid="nl36" bibid="bib26" firstref="ref63"></nolink> <nolink nlid="nl37" bibid="bib44" firstref="ref64"></nolink> <nolink nlid="nl38" bibid="bib56" firstref="ref65"></nolink> <nolink nlid="nl39" bibid="bib55" firstref="ref66"></nolink> <nolink nlid="nl40" bibid="bib21" firstref="ref70"></nolink> <nolink nlid="nl41" bibid="bib24" firstref="ref71"></nolink> <nolink nlid="nl42" bibid="bib30" firstref="ref72"></nolink> <nolink nlid="nl43" bibid="bib65" firstref="ref74"></nolink> <nolink nlid="nl44" bibid="bib23" firstref="ref75"></nolink> <nolink nlid="nl45" bibid="bib59" firstref="ref76"></nolink> <nolink nlid="nl46" bibid="bib63" firstref="ref78"></nolink> <nolink nlid="nl47" bibid="bib73" firstref="ref79"></nolink> <nolink nlid="nl48" bibid="bib28" firstref="ref80"></nolink> <nolink nlid="nl49" bibid="bib10" firstref="ref83"></nolink> <nolink nlid="nl50" bibid="bib49" firstref="ref97"></nolink> <nolink nlid="nl51" bibid="bib51" firstref="ref99"></nolink> <nolink nlid="nl52" bibid="bib14" firstref="ref102"></nolink> <nolink nlid="nl53" bibid="bib46" firstref="ref103"></nolink> <nolink nlid="nl54" bibid="bib19" firstref="ref104"></nolink> <nolink nlid="nl55" bibid="bib31" firstref="ref105"></nolink> <nolink nlid="nl56" bibid="bib61" firstref="ref109"></nolink> <nolink nlid="nl57" bibid="bib18" firstref="ref110"></nolink> <nolink nlid="nl58" bibid="bib38" firstref="ref111"></nolink> <nolink nlid="nl59" bibid="bib41" firstref="ref113"></nolink> <nolink nlid="nl60" bibid="bib64" firstref="ref122"></nolink> <nolink nlid="nl61" bibid="bib13" firstref="ref131"></nolink> <nolink nlid="nl62" bibid="bib43" firstref="ref134"></nolink> <nolink nlid="nl63" bibid="bib45" firstref="ref135"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1293487
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sales%2C+Adam+C%2E%22">Sales, Adam C.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-0416-0610">0000-0003-0416-0610</externalLink>)<br /><searchLink fieldCode="AR" term="%22Pane%2C+John+F%2E%22">Pane, John F.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-5155-2436">0000-0001-5155-2436</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+on+Educational+Effectiveness%22"><i>Journal of Research on Educational Effectiveness</i></searchLink>. 2021 14(1):241-269.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 29
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 1420374
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Technology%22">Educational Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Use+Studies%22">Use Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Curriculum%22">Mathematics Curriculum</searchLink><br /><searchLink fieldCode="DE" term="%22Curriculum+Evaluation%22">Curriculum Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Algebra%22">Algebra</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligent+Tutoring+Systems%22">Intelligent Tutoring Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Cues%22">Cues</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Solving%22">Problem Solving</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Causal+Models%22">Causal Models</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2020.1823538
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1934-5747
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2021
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1293487
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1293487
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2020.1823538
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 29
        StartPage: 241
    Subjects:
      – SubjectFull: Educational Technology
        Type: general
      – SubjectFull: Use Studies
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Mathematics Curriculum
        Type: general
      – SubjectFull: Curriculum Evaluation
        Type: general
      – SubjectFull: Algebra
        Type: general
      – SubjectFull: Intelligent Tutoring Systems
        Type: general
      – SubjectFull: Cues
        Type: general
      – SubjectFull: Problem Solving
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Causal Models
        Type: general
    Titles:
      – TitleFull: Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sales, Adam C.
      – PersonEntity:
          Name:
            NameFull: Pane, John F.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-print
              Value: 1934-5747
          Numbering:
            – Type: volume
              Value: 14
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1