View in EDS HTML Full Text PDF Full Text

The Impacts of a Standards-Based Grading System Emphasizing Formative Assessment, Feedback, and Re-Assessment: A Mixed Methods, Cluster Randomized Control Trial in Ninth Grade Mathematics Classrooms

Saved in:
Bibliographic Details
Title: The Impacts of a Standards-Based Grading System Emphasizing Formative Assessment, Feedback, and Re-Assessment: A Mixed Methods, Cluster Randomized Control Trial in Ninth Grade Mathematics Classrooms
Language: English
Authors: Steven L. Kramer (ORCID 0000-0003-2900-6484), Michael A. Posner, Alexander S. Browman (ORCID 0000-0002-2957-3262), Nancy R. Lawrence, Jennifer Roem, Kathleen Krier
Source: Journal of Research on Educational Effectiveness. 2025 18(1):56-87.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 32
Publication Date: 2025
Sponsoring Agency: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
Contract Number: 0918474
Document Type: Journal Articles
Reports - Research
Education Level: Grade 9
High Schools
Junior High Schools
Middle Schools
Secondary Education
Descriptors: Grading, Formative Evaluation, Feedback (Response), Grade 9, Secondary School Mathematics, Mathematics Instruction, Algebra, Geometry, Mathematics Achievement, Student Motivation, Educational Opportunities, High School Students
DOI: 10.1080/19345747.2023.2287594
ISSN: 1934-5747
1934-5739
Abstract: We investigated the impact in ninth-grade mathematics classrooms of Proficiency-based Assessment and Reassessment of Learning Outcomes (PARLO), a standards-based grading system. Key components of PARLO are basing student final grades on the number of learning outcomes on which the student is high-performance or proficient, providing students with formative feedback, and encouraging students to reassess for full credit after further study. Our mixed-methods study employed a cluster randomized control trial with 2,736 participating ninth graders at 14 Treatment and 15 Control schools. Data included student achievement tests, interviews with 35 teachers, and student surveys. The program improved student performance on end-of-course algebra and geometry tests by a statistically significant 0.33 SD but did not impact students' value of or expectancies for success in mathematics. However, treatment effects on mathematics performance were moderated by these psychological antecedents of motivation, such that students with higher math expectancies and value benefited more from the treatment. Furthermore, teacher interviews suggested that PARLO may have also had positive effects on growth mindsets, mastery goals, autonomy, and relatedness.
Abstractor: As Provided
Notes: https://doi.org/10.35542/osf.io/pzc3f
Entry Date: 2026
Accession Number: EJ1492695
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHFOYQgjuRulX0NkYcf9AFlAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCtIuKAptnYpSoSi9AIBEICBmvQoW2HmivFJESmr5TwGVaI9fz7gE00DWB_3ycnztc4Jh8kjl3iSY43PXx9AqqBbNFQMvVASHm-jfWuI4TlrcqRJ3pTlQ6Jkxl0n-zUYtBldB6Weoi8WMZc6lQdItH7gxVtrnDaN0ZpCUzS9W4ARpphZqetbNERdc42Hrs12VUO-ideSj4Sqeixv_yGpD5GtyDUcBjURtHPFNW0=
Text:
  Availability: 1
  Value: <anid>AN0187163326;[5ew9]01jan.25;2025Aug08.02:56;v2.2.500</anid> <title id="AN0187163326-1">The Impacts of a Standards-Based Grading System Emphasizing Formative Assessment, Feedback, and Re-Assessment: A Mixed Methods, Cluster Randomized Control Trial in Ninth Grade Mathematics Classrooms </title> <p>We investigated the impact in ninth-grade mathematics classrooms of Proficiency-based Assessment and Reassessment of Learning Outcomes (PARLO), a standards-based grading system. Key components of PARLO are basing student final grades on the number of learning outcomes on which the student is high-performance or proficient, providing students with formative feedback, and encouraging students to reassess for full credit after further study. Our mixed-methods study employed a cluster randomized control trial with 2,736 participating ninth graders at 14 Treatment and 15 Control schools. Data included student achievement tests, interviews with 35 teachers, and student surveys. The program improved student performance on end-of-course algebra and geometry tests by a statistically significant 0.33 SD but did not impact students' value of or expectancies for success in mathematics. However, treatment effects on mathematics performance were moderated by these psychological antecedents of motivation, such that students with higher math expectancies and value benefited more from the treatment. Furthermore, teacher interviews suggested that PARLO may have also had positive effects on growth mindsets, mastery goals, autonomy, and relatedness.</p> <p>Keywords: Standards-based grading; high school mathematics; growth mindsets; expectancy value theory; self-determination theory</p> <hd id="AN0187163326-2">Introduction</hd> <p>Among the most central and deeply rooted features of most American classrooms is a grading system that is designed to rank, sort, and evaluate students as <emph>learners</emph>, but is not optimally designed to <emph>help students learn</emph>. In the late 19th and early 20th centuries, many educators sought to build a "meritocratic" system that would give students access to a common elementary education, with students encouraged to persist according to their interests and abilities. Specifically, the system was designed so that more naturally talented and quicker learning students would receive higher grades than those with less natural talent or who learned more slowly, and only those with high grades would be encouraged or even permitted to progress to the next level of the education system (Farrington & Small, [<reflink idref="bib12" id="ref1">12</reflink>]; Schneider & Hutt, [<reflink idref="bib28" id="ref2">28</reflink>]).</p> <p>Thus, the grading system that is still typical in U.S. classrooms today was adopted at a time when a primary goal of public education was to sort students based on their supposed inherent intellectual abilities. As a result, the grading system does not optimally support the theoretical goal of the modern education system: learning. Specifically, classroom grading is built around <emph>summative assessments</emph> that typically only provide students with a limited amount of time to learn a topic and a limited number of opportunities to demonstrate how much they have learned. After assessment, students are often not provided with additional opportunities to improve their skills or to demonstrate that they have increased their learning. Further, early assessments are typically averaged with later assessments. This can discourage persistence in the face of initial difficulty because no matter how well students eventually learn the material they will be evaluated partly based on their initial difficulties.</p> <hd id="AN0187163326-3">A Proposed Improvement: Adding Formative Assessment</hd> <p>To better support student learning, educators in the 1960s began recommending formative assessment as a supplement to summative assessment (Bloom, [<reflink idref="bib5" id="ref3">5</reflink>]). In contrast to the traditional system, where assessments are used solely to provide a final, summative judgment of student performance, formative assessment uses assessments intermittently to provide teachers and students with feedback about each student's progress. This enables teachers and students to adapt their teaching and learning strategies to help each student progress based on their individual needs. Numerous meta-analyses have shown that formative assessment has significant positive implications for student engagement, learning, and performance, especially for previously low-performing students (e.g., Black & Wiliam, [<reflink idref="bib4" id="ref4">4</reflink>]; Kingston & Nash, [<reflink idref="bib15" id="ref5">15</reflink>]).</p> <p>To date, most formative assessment programs that have been implemented, especially in mathematics, have been teacher-facing. This approach to formative assessment provides teachers with feedback about their students' current level of proficiency, allowing the teachers to adapt their teaching strategies to meet the individual needs of their students (e.g., Supovitz et al., [<reflink idref="bib31" id="ref6">31</reflink>]). However, formative assessment can also be student-facing, wherein assessments are used to provide students with feedback about their own current level of proficiency and the nature of their mistakes so they can adapt their learning and become more proficient.</p> <p>Numerous educators have called for more widespread implementation of student-facing formative assessment (e.g., Leahy et al., [<reflink idref="bib17" id="ref7">17</reflink>]; Wiliam, [<reflink idref="bib38" id="ref8">38</reflink>]). Mills and Silver ([<reflink idref="bib20" id="ref9">20</reflink>], p. 180) articulated the argument for mathematics: "Teachers on their own cannot make students learn mathematics; students must become partners in their own learning in order for effective teaching and learning to occur." Supporting student agency by incorporating student-facing formative assessment may be especially efficacious for middle and high school students, an age when students are exploring their independence and have a particular need for status and respect (Yeager, [<reflink idref="bib39" id="ref10">39</reflink>]; Yeager et al., [<reflink idref="bib40" id="ref11">40</reflink>]).</p> <p>Nonetheless, prior research on student-facing formative assessment in math class has been limited, mostly focusing on improving the feedback that students receive about their work. For example, Murphy et al. ([<reflink idref="bib21" id="ref12">21</reflink>]) reported results of a cluster randomized control trial (RCT) that evaluated "Assistments," an online program that provides students immediate feedback on the correctness of homework problems. "Assistments" increased the mathematics achievement of seventh graders by a statistically significant 0.2 standard deviations (SD). Programs like "Assistments," are philosophically compatible with and potentially complementary to, but differ from, the program we investigated, which focused on reengineering the grading system to better support student learning.</p> <hd id="AN0187163326-4">Standards-Based Grading: The PARLO Assessment System</hd> <p>The present work examines the effects of the <emph>Proficiency-based Assessment and Re-assessment of Learning Outcomes (PARLO)</emph> system. The system was developed by a partnership between Dylan Wiliam, a pioneer researcher in formative assessment, and the Math Science Partnership of Greater Philadelphia (MSP-GP), an NSF-funded project that brought together 13 institutions of higher education and 46 Pennsylvania and New Jersey school districts to improve grades 6-12 mathematics and science education. It was partly inspired by a program at the Young Women's Leadership Charter School (YWLCS), a public charter high school in Chicago that changed its grading system to allow students to reassess for full credit whenever they had mastered classroom content. The goal was to promote students' agency by fostering a sense of partnership with their teachers in ensuring that they mastered the big ideas from each course. As a result of these changes, YWLCS regularly achieved the highest graduation rate of any nonselective public school in Chicago, despite serving a similar student body as neighboring schools (mostly low-income Black and Latina students; Farrington & Small, [<reflink idref="bib12" id="ref13">12</reflink>]).</p> <p>The PARLO assessment system is designed to be implementable in a single teacher's classroom, embedded in a school that may have a traditional American grading system. Consequently, PARLO teachers do assign final summative grades. However, PARLO reengineers the classroom assessment system by better integrating summative assessment and student-facing formative assessment. Specifically, while students receive summative scores on quizzes and assignments, such scores are not recorded indelibly to be used in a weighted average that determines final grades. Rather, the teacher uses such assessment evidence both to evaluate the extent to which a student is proficient in each of the course's learning outcomes <emph>at that particular moment in time</emph>, and then to provide students with personalized feedback designed to guide further learning. Students are then given opportunities to do further work, at home or in school, and to be reassessed for full credit. In other words, summative assessments become formative tools designed to promote further learning, instead of merely yardsticks to measure the student's efficiency as a learner.</p> <p>The first step in implementing PARLO is clarifying and sharing learning intentions and success criteria (Wiliam, [<reflink idref="bib38" id="ref14">38</reflink>]). To do this, the teacher organizes their course instruction around 10-15 learning outcomes per semester, which together define the material to be mastered. These learning outcomes are then shared with students and their parents so that they can be partners in the student's educational progress. An important characteristic of PARLO that distinguishes it from similar programs such as Mastery Learning (Bloom, [<reflink idref="bib5" id="ref15">5</reflink>]) is the way it defines success criteria. In addition to establishing criteria for "proficient" performance, teachers also share criteria for "high performance." Achieving high performance on a learning outcome is intended to challenge talented students, while potentially being achievable through hard work by nearly all students. In practice, teachers in the current study used Webb's Depth of Knowledge (Webb, [<reflink idref="bib36" id="ref16">36</reflink>]) to define success criteria, with high-performance requiring students either to solve problems at a greater depth of knowledge (application or strategic thinking), or else to tutor a fellow student and successfully bring them from not-yet-proficient up to proficient.</p> <p>The second step in implementing PARLO employs the central idea of student-facing formative assessment: eliciting evidence of learner's achievement and providing feedback that moves learning forward (Wiliam, [<reflink idref="bib38" id="ref17">38</reflink>]). PARLO teachers use short quizzes, end-of-lesson written "exit tickets," notes from observing students working in groups, and other formative assessment techniques to provide students feedback about how well they are progressing toward proficient or high performance on each learning outcome. Note that other student-facing formative assessment programs like Assistments (Murphy et al., [<reflink idref="bib21" id="ref18">21</reflink>]) implement this step. In future iterations of PARLO, teachers might use programs like Assistments to support PARLO implementation.</p> <p>The remaining two steps in implementing PARLO focus on changing the way teachers assign summative grades. As will be explained in more detail below, the changes to summative grading were the key features differentiating the Treatment from the Control schools in the current study. These two steps are:</p> <p></p> <ulist> <item> Reassessment for full credit after further learning. Student grades are not averaged over the semester; instead, students are rated on each learning outcome as not-yet-proficient, proficient, or high-performance based on the best work they can show by the end of the semester. While students can reassess for full credit, before reassessment they must first engage in further learning, completing activities such as:</item> <p></p> <item> Error logs, wherein students explain what they did wrong on a problem, rework it, and describe what they would need to remember so as not to make the same mistake again.</item> <p></p> <item> Remediation plans, or contracts with students about activities to be undertaken before reassessment.</item> <p></p> <item> Flashback days, or in-class opportunities for students to work individually or together to revisit learning outcomes and learn material at proficient or high-performance level.</item> <p></p> <item> Final Grades Based on Learning Outcomes. With project help, each school developed its own algorithm to determine a letter grade based on the number of Learning Outcomes scored Proficient and the number scored High Performance. Other factors sometimes used to assign grades, such as attendance, attitude, and homework completion, are not averaged as part of the final grade. Instead, they are viewed as means to the end of understanding course content.</item> </ulist> <p>At the time that we designed the intervention, a grading system with components like the four we described above was commonly known as "proficiency-based"—hence the name "PARLO." However, terminology has since shifted, and PARLO-type grading systems have become known under the name <emph>standards-based grading</emph> (see, e.g., Marzano, [<reflink idref="bib18" id="ref19">18</reflink>]).</p> <hd id="AN0187163326-5">Theoretical Framework</hd> <p>We hypothesized that the PARLO system would positively impact student learning in two ways: by providing <emph>additional time and opportunity for students to learn</emph>; and by <emph>encouraging students' motivation/desire to learn</emph>.</p> <hd id="AN0187163326-6">Opportunity to Learn</hd> <p>As noted above, traditional summative assessment systems encourage teachers to provide a limited amount of time for students to learn a topic and a limited number of opportunities to demonstrate how much they have learned. In contrast, PARLO teachers provide students with more opportunities to learn from their mistakes, improve, and be reassessed to demonstrate their increased proficiency. Further, student persistence is encouraged by giving them full credit for material learned, regardless of initial difficulties they may have had.</p> <hd id="AN0187163326-7">Motivation</hd> <p>According to expectancy-value theory (Wigfield & Eccles, [<reflink idref="bib37" id="ref20">37</reflink>]), students' motivation and classroom engagement are largely determined by the degrees to which they both <emph>value</emph> the material being taught and <emph>expect to succeed</emph> in class. Yet, studies have found that students exposed to an assessment system designed to compare and rank students report weaker feelings of intrinsic value (enjoyment of learning for its own sake), utility value (the belief that learning is useful and will have purpose and relevance in one's life), and expectancies for success than those exposed to a system that provided students with opportunities to improve their skills or to demonstrate that they have increased their learning (Covington & Omelich, [<reflink idref="bib9" id="ref21">9</reflink>]; Haley, [<reflink idref="bib13" id="ref22">13</reflink>]; Sanchez et al., [<reflink idref="bib25" id="ref23">25</reflink>]), and that these effects may be especially pronounced among students from less advantaged backgrounds (Jury et al., [<reflink idref="bib14" id="ref24">14</reflink>]; Smeding et al., [<reflink idref="bib30" id="ref25">30</reflink>]).</p> <hd id="AN0187163326-8">Additional Motivational Antecedents Addressed</hd> <p>This study was designed using concepts from the expectancy-value theory of motivation (Wigfield & Eccles, [<reflink idref="bib37" id="ref26">37</reflink>]). Additionally, after we had collected our data and begun to analyze it, we found concepts from three additional theories of motivation to be helpful in understanding what the teachers told us in interviews: growth mindset theory (Dweck, [<reflink idref="bib10" id="ref27">10</reflink>]), self-determination theory (Ryan & Deci, [<reflink idref="bib24" id="ref28">24</reflink>]), and achievement goal theory (Senko, [<reflink idref="bib29" id="ref29">29</reflink>]). Growth mindset theory suggests that students show greater motivation, engagement, learning, and performance in school, especially in the face of difficulty, if they believe that they can grow and improve their intelligence and academic skills through hard work (a growth mindset), versus if they instead believe that their ability levels are innate and unchangeable (a fixed mindset). Self-determination theory suggests that students will be more internally motivated to engage with their schoolwork when three needs are met: <emph>competence</emph> (roughly analogous to expectancies for success, described above), <emph>autonomy</emph> (the feeling that they have opportunities to make meaningful choices about their learning), and <emph>relatedness</emph> (the feeling that they are valued and belong in class). Finally, achievement goal theory suggests that ​students experience greater academic outcomes when they adopt <emph>mastery goals</emph>, i.e., engaging in schoolwork either to learn as much as possible, or else to meet self-determined standards of success in learning. As will be discussed below, each of these themes emerged in interviews with teachers and open-ended survey responses from students who were exposed to PARLO—a program that encouraged the capacity of students to improve their mathematics abilities (i.e., growth mindsets and mastery goals), and provided students with flexibility in how they could go about acquiring and demonstrating this improvement (i.e., self-determination). Below, we refer to feelings, attitudes, beliefs, and goals that contribute to student motivation and engagement as <emph>motivational antecedents</emph>.</p> <hd id="AN0187163326-9">Conceptual Model</hd> <p>Figure 1 displays our current conceptual model regarding how the PARLO system may influence students' academic experiences and learning. Because it was developed using the still-to-be described results of the present work, we return to the complete model in the Discussion section. For now, we wish to note the most important difference between the PARLO Treatment and the Control group in the present work: the two white boxes on the bottom left of the figure. Unlike the Control teachers, PARLO teachers were trained and expected to reassess students for full credit after further learning, and to base a student's summative grade on the number of proficient and the number of high-performance learning outcomes the student had mastered. The two gray boxes on the top left—organizing instruction by learning outcomes and providing formative assessment and feedback—are also necessary parts of the PARLO system and were implemented by teachers in our study. However, these two PARLO components were popular innovations already in place in the school districts where we conducted the study. Given this environment, the project delivered professional development to teachers in the Control group to further assist them in implementing these two formative assessment-related practices. We anticipated that PARLO teachers would be more likely than Control teachers to implement the PARLO components displayed in the grayed-out boxes, but only because implementing the bottom two components encouraged them to do so. Thus, the current study tests the impact of experimentally adding the lower two components of the PARLO standards-based grading system—reassessment for full credit after further learning, and summative grades based on the number of proficient and the number of high-performance learning outcomes—to classrooms that were already endeavoring to implement the top two components.</p> <p>Graph: Figure 1. Conceptual model.</p> <p>We also note that the focus of the conceptual model, and of the current article, is on <emph>changing teacher's assessment practices</emph>. Specific inputs, such as software tools made available to teachers and specific professional development provided, evolved over the course of the project, and will likely be implemented differently in the future. Consequently these "program inputs" are not displayed in the model. Our intent is to generalize results to future implementations of standards-based grading that are similar to PARLO. See our Conclusion section for ideas about possible future implementations.</p> <hd id="AN0187163326-10">Prior Research on Standards-Based Grading and PARLO</hd> <p>While numerous researchers have written about standard-based grading, published work has focused on case studies of small-scale implementation in single classrooms or schools (e.g., Farrington & Small, [<reflink idref="bib12" id="ref30">12</reflink>]). Consequently, researchers have decried the lack of studies evaluating the practice (Marzano, [<reflink idref="bib18" id="ref31">18</reflink>]; Scarlett, [<reflink idref="bib27" id="ref32">27</reflink>]). The current study should help to fill that gap.</p> <p>There have been two pilot studies of the PARLO system. First, Clymer and Wiliam ([<reflink idref="bib8" id="ref33">8</reflink>]) implemented PARLO for a full school year in a single eighth grade science classroom. Average achievement on an end-of-class exam increased by 0.4 <emph>SD</emph> over the previous year's scores. In addition, student interviews indicated that these positive effects on performance may have emerged because students developed stronger growth mindsets. Second, Posner ([<reflink idref="bib23" id="ref34">23</reflink>]) taught two sections of an introductory undergraduate statistics class for non-majors, nonrandomly employing PARLO in one section and traditional grading in the other. He found that the PARLO group scored significantly higher than the traditional grading group on measures of perceived intrinsic value, utility value, expectations for success, and motivation in statistics.</p> <hd id="AN0187163326-11">The Current Study</hd> <p>While these initial results were promising, the studies described were only small pilots that did not provide a randomized control-style investigation of the effects of the PARLO assessment system. The present work describes the results of such an investigation. Specifically, we employed a mixed-methods, cluster randomized control approach with school as the unit of analysis, to address the following two research questions: <emph>What is the impact of the PARLO system on 9<sups>th</sups> grade students' expectancies for success, utility value, intrinsic value, long-term motivation, and academic performance in mathematics class? What are the mechanisms by which this impact occurs?</emph></p> <p>In addition, while not part of our <emph>a priori</emph> hypotheses, our examinations of the qualitative data unexpectedly revealed that the relationship between the PARLO system and student motivation might be bi-directional: not only might PARLO improve student motivation, but students with higher initial motivation might also benefit more from the PARLO system. Consequently, we used our quantitative data to investigate an additional research question: <emph>Is the impact of the PARLO system on student academic performance moderated by established antecedents of student motivation—specifically, students' value of and expectancy for success in mathematics?</emph></p> <hd id="AN0187163326-12">Method</hd> <p>This was a mixed-methods study, collecting qualitative and quantitative data throughout program implementation. Our data analysis followed a sequential explanatory design, performed in three phases. In the first phase, we conducted quantitative analyses to see whether the PARLO system had its predicted effects on motivation and on student achievement. In the second phase, we used qualitative data to look for explanations of the results we found in phase 1. Our qualitative analysis began inductively, thus allowing for the emergence of trends we did not hypothesize a priori. We summarized preliminary results as memos describing major themes observed and evidence for those themes. We then reviewed the memos in light of the quantitative findings, looking for explanations of those findings. The second phase yielded an unexpected result, indicating that student motivation might be moderating the PARLO system's effects on student achievement. Consequently, we conducted a third phase, investigating whether the quantitative data we had collected could confirm the moderating effect.</p> <hd id="AN0187163326-13">Participants</hd> <p>We recruited urban, suburban, and rural schools; public, charter, and religious schools; schools from high- and low-performing districts; and schools with a wide variety of racial make-ups. In order to ensure reasonably high fidelity of implementation, we only assigned a school to the project if both the administration and the ninth-grade mathematics teachers agreed in advance that they would participate as assigned either in the treatment or in the control condition, and would maintain participation regardless of subsequent random assignment. All ninth-grade algebra and geometry teachers at each school were asked to participate. Teachers received a stipend in exchange for their participation.</p> <p>We ultimately recruited two cohorts of schools: a cohort of 20 schools (14 public schools, 3 charter schools, and 3 Catholic all-girl schools) that participated during the 2010–11 and 2011–12 school years; and a second cohort of 15 schools (14 public schools from one large urban school district and 1 additional public school) that participated during the 2011–12 and 2012–13 school years. Cohorts participated for two years (i.e., two separate ninth grade classes) because implementing PARLO required significant changes to teachers' instructional practices that we anticipated might take more than a single year to take root.</p> <p>Several schools dropped out of the study after randomization but before data collection was completed. The final sample of schools in the quantitative study included 14 PARLO and 15 control schools across both cohorts. One additional PARLO school provided qualitative data but dropped out before quantitative data collection could be completed. In all cases, school-level attrition was caused by a change in administration, which is unlikely to be related to the treatment. The overall attrition rate was 17%, and the differential attrition between treatment and control schools was 10.4%. At the student level, the overall attrition rate was 16.9% and the differential attrition was 3.9%. These rates are within acceptable standards when the intervention is unlikely to affect attrition (USDOE, 2022). See Tables A1, A3, and A4 in the online appendix for details about attrition.</p> <p>Our Institutional Review Board (IRB) required active consent (i.e., parental opt-in) before administering our quantitative measures during the 2010–11 school year—the first implementation year for Cohort 1. However, many students failed to return a consent form, creating a risk of self-selection and bias in the data. As a result, our external reviewer and advisory board for this project determined that the quantitative data collected in 2010–11 could not be used for valid analyses. With the approval of our IRB, this issue was resolved beginning in the 2011–12 school year through the use of passive consent (i.e., parental opt-out). Thus, our quantitative analyses focus on data collected during academic years 2011–12 and 2012–13.</p> <p>Across two years of data collection, 3,273 students completed algebra or geometry pretests and had available demographic data for the analysis—1,936 in treatment schools and 1,337 in control schools. Of these, 2,736 students provided complete quantitative data, including baseline motivation and motivational antecedent scores, race, gender, and post-test scores, and thus were included in the analytic sample for our primary quantitative analysis. Of the total 2,736 students, 1,649 were from the classes of 38 teachers at the 14 PARLO treatment schools, and 1,087 were from the classes of 27 teachers at the 15 control schools. Tables A5 and A6 in the online appendix compare the demographic characteristics of treatment and control group students and teachers in the analytic sample.</p> <hd id="AN0187163326-14">Random Assignment and Conditions</hd> <p>In June 2010, all ninth-grade algebra and geometry teachers at all Cohort 1 schools who had agreed to participate attended three 6-hour days of professional development (PD) in the summer. After teachers completed this PD, project staff then randomly assigned participating schools either to the treatment or the control condition. Randomization was done in three blocks: charter vs. Catholic vs. public. The following year, we randomly assigned Cohort 2 schools to treatment or control conditions following a process that was identical but for two exceptions: the PD before random assignment lasted only two days, and blocking by school type was not necessary as all Cohort 2 schools were public. Participating teachers were aware of their condition assignment. See the online appendix for details about random assignment and for a depiction of the study timeline.</p> <hd id="AN0187163326-15">Control Condition</hd> <p>When this study was conducted, two formative assessment practices were coming into widespread use, i.e., becoming business-as-usual for many teachers in local school districts: sharing learning intentions and success criteria by organizing instruction around learning outcomes tied to state standards; and using formative assessment strategies such as providing frequent feedback. Both practices were necessary but not sufficient to implement PARLO. In order to further focus the study on the unique aspects of PARLO that were not already coming into popular use, before random assignment we provided all participating teachers professional development supporting these two widely used formative assessment practices. We had each teacher work with state standards for their course to translate them into 10 to 15 learning outcomes each semester that would be the focus of teaching in their classroom. We introduced Webb's Depth of Knowledge (Webb, [<reflink idref="bib36" id="ref35">36</reflink>]) to help them think about addressing each learning outcome at application and strategic-thinking levels. We used ideas from Wiliam ([<reflink idref="bib38" id="ref36">38</reflink>]) to teach techniques for eliciting evidence from students and providing feedback to move their learning forward. The initial professional development lasted three days for Cohort 1 and two days for Cohort 2. Because the Control teachers were encouraged to implement the two aspects of PARLO reflected in the gray boxes of Figure 1, the project often described the Control schools as a "limited treatment condition," to be contrasted with the "full treatment condition" that implemented all four aspects of PARLO.</p> <p>Control teachers did not participate in any project-related PD beyond the two or three days that they participated in before randomization, and they were not obligated to modify their instructional practices. In addition to administering the student content exam and Attitudes Toward Mathematics Inventory described below, control teachers completed questionnaire measures of the extent to which they used various practices in their teaching.</p> <hd id="AN0187163326-16">PARLO Treatment Condition</hd> <p>Teachers in treatment schools participated in three training opportunities not offered to control teachers. First, in August prior to first implementing the PARLO system in their classrooms, they attended three (for Cohort 1) or four (for Cohort 2) additional six-hour days of PD. Second, during the two years of project participation, they were given the opportunity to participate in monthly Professional Learning Community (PLC) meetings. The PLC meetings were held at five different locations and facilitated by teacher leaders who were mathematics content and PARLO experts. Third, they attended two six-hour days of follow-up PD during the summer between their two years of PARLO participation. Total participation time for treatment teachers in PD and PLC meetings amounted to about 88 hours over the two years. Seventy-nine percent of the treatment teachers attended 70% or more of the PD available to them.</p> <p>The PARLO-specific PD focused on the two distinguishing characteristics of the PARLO standards-based grading system: 1) Reassessment for full credit after further learning; and 2) Basing a student's semester grade on their number of proficient and number of high-performance learning outcomes. Treatment teachers were provided with a project-developed software tool, PARLO <emph>Tracker</emph>, and trained in how to use it. Teachers could enter into Tracker the learning outcomes they had developed and enter evidence collected, including assessments and reassessments, about each student's progress on each learning outcome. Students and parents could access Tracker online to track student progress. (Tracker was developed with user input and became available at the beginning of the second year of the project, which is the first year that quantitative data used in the current study was collected. Use was optional, and when surveyed during Year 3 of the project, 16 of 25 PARLO teachers (64%) reported using Tracker regularly.) Additional topics covered in the professional development included: creating scoring rubrics for learning outcomes and sharing them with students; helping students track their proficiency; teaching students what to do when they are not yet proficient; requiring proof of new learning before reassessment, including use of tools like error logs and remediation plans; scoring proficiency in learning outcomes based on the best evidence to date instead of traditional methods like averaging scores; and converting learning outcome scores into semester grades. PARLO teachers were responsible for developing their own learning outcomes based on state standards, developing their own methods for informing students about the PARLO system and grading scheme, establishing their own classroom procedures, and deciding on their own means of collecting assessment and/or reassessment evidence. However, through PLCs they were able to share ideas and support each other in developing routines, assessments, and so on.</p> <hd id="AN0187163326-17">Quantitative Measures</hd> <p></p> <hd id="AN0187163326-18">Implementation Measures</hd> <p>In the spring of 2012, we administered a 7-item measure to Treatment and Control teachers, investigating whether they implemented key aspects of the PARLO system. In the spring of 2013 we administered an additional survey to PARLO teachers only, investigating in more detail to what extent they had implemented the PARLO system. See the online appendix section <emph>Implementation: Additional Details</emph> for a detailed description of the measures.</p> <hd id="AN0187163326-19">Algebra and Geometry Content Exams</hd> <p>The state where we conducted our research had recently adopted content standards for both geometry and algebra based on the Common Core State Standards for Mathematics (National Governors Association Center for Best Practices & Council of Chief State School Officers, [<reflink idref="bib22" id="ref37">22</reflink>]), but had not yet implemented an end-of-course test of the material. Consequently, we created a test to mimic as closely as possible a state-designed test of its published content standards. To do so, we administered (with permission) an adapted version of the Virginia Standards of Learning multiple choice Algebra and Geometry Tests. For each course (algebra or geometry), we selected items from the Virginia test that addressed our state's standards. Then, for any standards not covered, we added items adapted from textbook sources or from released items from older assessments in our participating state. In each participating classroom, we administered the appropriate content exam as both a pretest during the first five days of the school year (Algebra: <emph>M</emph> = 34.2%, <emph>SD</emph> = 11.0%, Cronbach's</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.72; Geometry: <emph>M</emph> = 36.0%, <emph>SD</emph> = 11.3%,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.75) and, with items in a different order, during the last month of the school year (Algebra: <emph>M</emph> = 46.9%, <emph>SD</emph> = 17.5%,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.90; Geometry: <emph>M</emph> = 49.7%, <emph>SD</emph> = 14.9%,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.94). Following suggestions by May et al. ([<reflink idref="bib19" id="ref38">19</reflink>]) for combining differing tests into a single measure, we used a linear transformation to standardize the pretests within each subject area (algebra or geometry) to have a mean of 0 and a standard deviation of 1, and then combined the two sets of scores to create a single measure. We did the same for post-test scores. See the online appendix for the complete tests used and technical details about computing student scores.</p> <hd id="AN0187163326-20">Motivation and Motivational Antecedents Measures</hd> <p>We assessed concepts from the expectancy-value theory of motivation using the Attitudes Toward Mathematics Inventory (ATMI; Tapia & Marsh, [<reflink idref="bib32" id="ref39">32</reflink>]). The four ATMI subscales are described below. In each participating classroom, we collected baseline scores on the four ATMI subscales during the first five days of the school year, and we collected outcome data on the same subscales during the last month of the school year. Students responded to all of the subscales using a 5-point scale (1 = "strongly disagree"; 5 = "strongly agree"). See the online appendix for the full set of items from the ATMI and technical details about computing student scores.</p> <hd id="AN0187163326-21">Perceived Intrinsic Value of Mathematics</hd> <p>The extent to which students saw mathematics as having intrinsic value was assessed using the ATMI's 10-item "enjoyment" subscale. Items included "I have usually enjoyed studying mathematics in school" (baseline: <emph>M</emph> = 3.27; <emph>SD</emph> = 0.88,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.89; outcome measure: <emph>M</emph> = 3.08; <emph>SD</emph> = 0.86,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.89).</p> <hd id="AN0187163326-22">Perceived Utility Value of Mathematics</hd> <p>The extent to which students saw mathematics as having utility value was assessed using the ATMI's 10-item "value" subscale. Items included "I can think of many ways that I use math outside of school" (baseline: <emph>M</emph> = 3.73; <emph>SD</emph> = 0.67,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.86; outcome measure: <emph>M</emph> = 3.54; <emph>SD</emph> = 0.74,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.87).</p> <hd id="AN0187163326-23">Expectancies for Success in Mathematics</hd> <p>The strength of students' expectancies for success in mathematics was assessed using the ATMI's 15-item "self-confidence" subscale. Items included "I believe I am good at solving math problems" (baseline: <emph>M</emph> = 3.45; <emph>SD</emph> = 0.77,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.79; outcome measure: <emph>M</emph> = 3.31; <emph>SD</emph> = 0.83,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.81).</p> <hd id="AN0187163326-24">Long-Term Motivation to Engage with Mathematics</hd> <p>The ATMI's 5-item "motivation" subscale assessed the degree to which students enjoyed the challenge of mathematics and planned to pursue the subject over the long term. Items included "I plan to take as much mathematics as I can during my education" (baseline: <emph>M</emph> = 3.21; <emph>SD</emph> = 0.79,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.79; outcome measure: <emph>M</emph> = 3.10; <emph>SD</emph> = 0.85,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi>α</mi></math> </ephtml> = 0.81).</p> <hd id="AN0187163326-25">Student-Level Covariates</hd> <p>We coded each student's race and gender based on information provided by the school or, when school-provided data were not available, based on a self-report survey administered simultaneously with the ATMI. A preliminary analysis using Akaike's Information Criterion (AIC; Akaike, [<reflink idref="bib1" id="ref40">1</reflink>]) indicated that the most informative way to operationalize "race" was to create a dichotomous variable, with "1" indicating the student was identified as White (51%) or Asian (4%), and 0 indicating that the student was identified as Black (30%), Hispanic (8%) or Multiracial/other (7%).</p> <hd id="AN0187163326-26">School-Level Covariates</hd> <p>We obtained from state databases the schoolwide percent low-income students enrolled in the academic year 2010–11, the first year of PARLO implementation for the study. We also aggregated student-level covariates (race, gender, and content pretest and baseline motivation-related scores) to the school level as potential additional covariates to control for in our analysis.</p> <hd id="AN0187163326-27">Qualitative Data</hd> <p>Qualitative data were collected throughout the three operational years of the project. The qualitative data sources used for this paper included 84 PARLO teacher interviews and an open-ended survey of 678 students from a nonrandom sample of 6 of the 14 PARLO treatment schools.</p> <hd id="AN0187163326-28">Teacher Interviews</hd> <p>During the first year of participation for Cohort 1 and Cohort 2, PARLO teachers were randomly selected to be interviewed; however, if a teacher was the only participant at their school, they were automatically selected to be interviewed. During the 2011–12 school year, the 13 Cohort 1 teachers who were interviewed were the same teachers who were interviewed the previous year. During the 2012–13 school year, all 25 remaining participating teachers were interviewed. Interviews were conducted in the fall and spring of each year, and some teachers were interviewed twice during the same year. Over the three years of the study, 11 teachers were interviewed once, 8 were interviewed twice, 7 were interviewed three times, and 9 were interviewed four times. Overall, 35 teachers at 15 schools participated in 84 interviews.</p> <p>Interview data for the current article was drawn primarily from questions that addressed how PARLO affected students—for example, "Does student engagement 'look' different? By that, I mean are students interacting with math, with you, with each other differently under PARLO?" See the online appendix for detailed interview questions. Interviews were semi-structured, permitting teachers to expound upon questions that warranted further elaboration. With few exceptions, interviews were conducted following a lesson observation, either during teachers' preparatory or lunch periods, or, rarely, after school. Interviews lasted about 25 to 35 minutes and were audio recorded with the teacher's permission. Every interviewee agreed to be recorded. Each interview was transcribed, then entered into NVivo software for analysis.</p> <hd id="AN0187163326-29">Student Survey</hd> <p>During the 2011–12 school year, six of the treatment schools agreed to have their students complete an open-ended online 8-question survey (e.g., "How would you describe PARLO to next year's ninth grade students?") A total of 678 students completed the survey. See the online appendix for the full list of questions asked on the student survey.</p> <hd id="AN0187163326-30">Data Analysis</hd> <p></p> <hd id="AN0187163326-31">Quantitative Analyses</hd> <p>We conducted all statistical analyses using the <emph>lmer</emph> command from the <emph>lmerTest</emph> package of the R programming language (Kuznetsova et al., [<reflink idref="bib16" id="ref41">16</reflink>]). We used Restricted Maximum Likelihood (REML) to test all research questions, but as recommended by Zuur et al. ([<reflink idref="bib42" id="ref42">42</reflink>]), we used maximum likelihood in preliminary analyses that employed AIC (Akaike, [<reflink idref="bib1" id="ref43">1</reflink>]) to choose variance components and covariates, or that used log likelihood to evaluate heteroscedasticity. We calculated degrees of freedom for statistical tests using Satterthwaite's approximation (Satterthwaite, [<reflink idref="bib26" id="ref44">26</reflink>]).</p> <p>Our data set identified each student by course (geometry vs. algebra), teacher, school, and study year. Consequently, the data were structured as a five-level hierarchical linear model: students within course within study-year within teachers within schools. We used AIC to choose the most appropriate variance structure for our analysis, using students' mathematics content post-test as our dependent variable. This analysis indicated that after entering variance components for school and for course-within-year-within-teacher, adding additional variance components for year-within-teacher or teacher-within-school reduced the efficiency of the model. We therefore used the following three-level variance structure in our analyses: "student," within "course-within-study-year-within-teacher," within "school."</p> <p>Our first quantitative analysis investigated the main effects of PARLO using mathematics content post-test scores as the dependent variable. We used AIC to select covariates for our model, choosing the following set: school level percent low-income students in 2011; a dichotomous indicator identifying students enrolled in geometry; a dichotomous variable indicating female sex; a dichotomous variable indicating White or Asian race (vs. Black, Hispanic, or multiracial/other); students' baseline expectancy score; linear and quadratic terms for the pretest; and interactions between the geometry indicator and the linear and quadratic pretest variables. We also included indicators to ensure that students were compared to other students in the same study-year and in the same blocking group used for random assignments. The reference group was Cohort 1 public school students. Other groups were: Catholic school students; Charter School students; Cohort 2 Public school students tested in 2010–11; and Cohort 2 public school students tested in 2011–12. All non-dichotomous student level variables were centered around the student mean, and percent low-income was centered around the mean of the school means.</p> <p>Our second quantitative analysis investigated the main effects of PARLO on the four motivation-related subscales obtained by the ATMI. We used the same covariates that we used the first analysis, with the following exceptions: we used scores for all four motivation and antecedent subscales, whereas previously only expectancy scores were used. We also removed the quadratic term for the pretest and its interaction with the geometry indicator. See the online appendix for further details about how we selected covariates to use in each of our models.</p> <p>Our two main-effects analyses used two approaches to account for missing data. First, we used Full Information Maximum Likelihood (FIML; Allison, [<reflink idref="bib2" id="ref45">2</reflink>]), an approach that, like multiple imputation, minimizes bias due to missingness. However, because it is difficult to analyze interactions between treatment and student-level covariates using FIML, and because these interactions were important in addressing <emph>for whom</emph> the PARLO system might be effective, we also ran the analyses using listwise deletion. The main effects estimated by the two analyses were very similar; consequently, we report detailed results of the listwise deletion analysis here. See Tables A10 and A12 of the online appendix for details of the FIML analysis.</p> <p>Our third quantitative analysis, unlike the first two, was not pre-planned. We conducted the analysis to see whether we could corroborate an unanticipated finding from our qualitative data: According to teacher interviews, student motivation moderated PARLO's impact, such that more highly motivated students benefited more from the PARLO program (see Results for details). We therefore used interaction terms to investigate how our four motivation-related quantitative variables moderated PARLO's impact on mathematics content post-test scores.</p> <p>Note that for each of the four measures of motivation, we had two observations per student: a score from the first week of the school year, and a score from the last month. Because motivation can change throughout the school year, our preferred approach to measuring motivation across the year would be to average the baseline and end-of-year scores together. We note that end-of-year motivation scores might be endogenous, which can make interpretations of some models confusing or bias results under certain circumstances. However, there is preliminary evidence (based on Monte Carlo results) that if x is an endogenous variable and w is an exogenous variable, the coefficient of the interaction term xw (i.e. the moderator effect) will be consistently estimated as long as w is binary and x is homoscedastic conditional on the other variables in the model (Bun & Harrison, [<reflink idref="bib6" id="ref46">6</reflink>]). PARLO treatment was a binary variable, and all four motivation variables were homoscedastic, so we could safely analyze the interactions using the averaged baseline-and-end-of-year scores. These moderation analyses used the same covariates we had used when testing the main effects of PARLO on mathematics post-test scores, except that instead of using the baseline confidence score as a covariate, we included the main effect for each moderator being tested. As a sensitivity check, we also re-ran the moderation analysis using baseline scores instead of year-average scores for all four motivation-related moderators. See the online appendix for details.</p> <p>As recommended by the What Works Clearinghouse Procedures Handbook (USDOE, 2020), whenever we used multiple statistical tests to address a single issue we used the Benjamini-Hochberg (BH) correction to account for multiple comparisons. The BH procedure controls the false discovery rate for multiple comparisons, ensuring that the expected proportion of falsely identified statistically significant results equals the intended alpha, in this case 5%.</p> <hd id="AN0187163326-32">Qualitative Analyses</hd> <p>At the conclusion of teachers' participation in the study (June 2013), two qualitative researchers began coding three years' worth of interview transcripts. To enhance reproducibility, the researchers established a coding scheme using a procedure similar to the three-stage process outlined by Campbell et al. ([<reflink idref="bib7" id="ref47">7</reflink>]). In the first stage, each researcher read the same three interview transcripts, jotting notes in the margins to identify possible broad themes. Once themes were identified, the researchers met to compare notes and look for agreement or disagreement. If there was a lack of consensus, they discussed why they had coded the text under a particular theme, until agreement was reached that either only one of their themes was accurate, or that both of their themes were accurate. In the latter instances, the text was double coded. At the end of this process, 17 broad themes were identified.</p> <p>In the second stage, the remaining transcripts were divided, and each researcher coded half of the interview transcripts. In the third stage, each researcher assumed responsibility for the analyses of a defined set of themes. The researchers then prepared memos by theme and study year, with each memo summarizing major evidence for the existence of that theme in the interviews conducted that year. Initial analysis of the student survey followed a similar process, with an additional analysis that identified key words or phrases that appeared frequently in students' responses to survey prompts.</p> <p>To produce the findings described in this article, a third researcher read all three years' worth of memos to both identify any claims that addressed how or why the PARLO system affected students' motivational antecedents, motivation, or academic performance, and to summarize evidence supporting those claims. Finally, for claims that were supported by numerous teachers' interviews, the research team made a final pass through all 84 transcribed interviews to quantify how many teachers supported that particular claim.</p> <hd id="AN0187163326-33">Results</hd> <p></p> <hd id="AN0187163326-34">PARLO Implementation</hd> <p>All PARLO teachers interviewed indicated they had implemented the new system, allowing reassessment for full credit and basing final grades on the number of proficient learning outcomes and the number of high-performance learning outcomes. In practice, most teachers rated individual assignments using a "traffic light" system: Red (not yet proficient), Yellow (approaching proficient), Green (proficient), or Blue (high performance; available only for high cognitive demand tasks). In interviews, teachers confirmed that learning outcome grades were developed by teacher judgments based on cumulative evidence, with more recent evidence weighted most heavily. Most PARLO teachers also required proof of learning before reassessment. In an online questionnaire administered to the 25 PARLO teachers participating during the 2012–13 school year, 88% reported using resubmission/correction of work, error logs, remediation plans, and/or flashback days at least "fairly often."</p> <p>When surveyed in the spring of 2012, PARLO teachers were significantly more likely than Control teachers to agree that they planned to utilize "a 'high performance', 'proficient', or 'not yet proficient' assessment system" (76% vs. 22% agreement) and use "the traffic light system to monitor student progress" (50% vs. 15% agreement). Both these contrasts were statistically significant after using the BH adjustment to control for a false discovery rate of 5%.</p> <p>Two additional differences between groups were significant after controlling for a false discovery rate of 10%, but not 5%. First, PARLO teachers were somewhat more likely to implement "holding students accountable for learning outcomes" (94% vs. 74%) and somewhat more likely to implement "collecting evidence of student learning" (88% vs. 67%). Both Treatment and Control groups indicated they were highly likely to implement "using formative assessment strategies in your classrooms": 91% for PARLO teachers and 93% for Control teachers. See the online appendix for additional details about the implementation measures and their results.</p> <p>Overall, the above data indicate that the contrast between the PARLO teachers and Control teachers appears to have been as intended. A large majority of both groups implemented the PARLO characteristics depicted in the gray boxes of Figure 1, although there is evidence that the PARLO group may have implemented learning outcomes more extensively. The PARLO teachers were much more likely than Control teachers to base a student's grades on the number of proficient and number of advanced learning outcomes. While we do not have statistics on the percentage of Control teachers who gave students the opportunity to reassess for full credit after additional learning, all PARLO teachers interviewed indicated that they had adopted this practice.</p> <hd id="AN0187163326-35">PARLO's Impact on Quantitative Measures of Mathematics Performance</hd> <p></p> <hd id="AN0187163326-36">Main Effects: Did PARLO Increase Mathematics Performance?</hd> <p>Table 1 summarizes the PARLO treatment's main effects on student mathematics learning. Recall that students' mathematics content test scores were standardized, so that within each subject area (algebra or geometry), the post-tests had a mean of 0 and a standard deviation of 1. Consequently, the results reported in Table 1 can be interpreted as effect sizes in standard deviation units. After controlling for random-assignment grouping, students' pretest scores, schoolwide percent low-income, geometry vs. algebra class, race, sex, and baseline expectancy for success, students in the PARLO system scored 0.33 <emph>SD</emph> higher than students in the control group on the project-administered end-of-course tests (<emph>p</emph> = 0.014).</p> <p>Table 1. PARLO treatment main effect on mathematics achievement.</p> <p> <ephtml> <table><thead><tr><td /><td>Estimate</td><td>Standard Error</td><td>Degrees of Freedom</td><td>p-value</td></tr></thead><tbody valign="top"><tr><td><italic>School Level Fixed Effects</italic></td><td /><td /><td /><td /></tr><tr><td>Intercept</td><td char=".">−0.66</td><td>0.12</td><td char=".">20</td><td char="."><.0001</td></tr><tr><td>Catholic Girls' School</td><td char=".">0.30</td><td>0.22</td><td char=".">21</td><td char=".">.18</td></tr><tr><td>Charter School</td><td char=".">−0.05</td><td>0.29</td><td char=".">33</td><td char=".">.86</td></tr><tr><td>Cohort 2 Public School, 2011-2012 school year</td><td char=".">0.23</td><td>0.15</td><td char=".">14</td><td char=".">.14</td></tr><tr><td>Cohort 2 Public School, 2012-2013 school year</td><td char=".">0.34</td><td>0.15</td><td char=".">14</td><td char=".">.035</td></tr><tr><td>Proportion Disadvantaged</td><td char=".">−0.53</td><td>0.23</td><td char=".">12</td><td char=".">.039</td></tr><tr><td><bold>PARLO Treatment</bold></td><td char="."><bold>0.33</bold></td><td><bold>0.12</bold></td><td char="."><bold>14</bold></td><td char="."><bold>.014</bold></td></tr><tr><td><italic>Course Level Fixed Effects</italic></td><td /><td /><td /><td /></tr><tr><td>Geometry Student</td><td char=".">0.20</td><td>0.16</td><td char=".">50</td><td char=".">.23</td></tr><tr><td><italic>Student Level Fixed Effects</italic></td><td /><td /><td /><td /></tr><tr><td>Baseline Expectancy of Success</td><td char=".">0.20</td><td>0.02</td><td char=".">2,687</td><td char="."><.0001</td></tr><tr><td>White or Asian</td><td char=".">0.15</td><td>0.04</td><td char=".">2,654</td><td char="."><.0001</td></tr><tr><td>Female</td><td char=".">0.10</td><td>0.03</td><td char=".">2,672</td><td char=".">.0005</td></tr><tr><td>Pretest</td><td char=".">0.26</td><td>0.02</td><td char=".">2,710</td><td char="."><.0001</td></tr><tr><td>Pretest-squared</td><td char=".">0.04</td><td>0.01</td><td char=".">2,673</td><td char=".">.0028</td></tr><tr><td>Geometry X Pretest</td><td char=".">0.20</td><td>0.05</td><td char=".">2,720</td><td char=".">.0002</td></tr><tr><td>Geometry X Pretest-squared</td><td char=".">−0.02</td><td>0.03</td><td char=".">2,700</td><td char=".">0.54</td></tr><tr><td><italic>Random Effects</italic></td><td><italic>N</italic></td><td><italic>Variance</italic></td><td /><td /></tr><tr><td>School</td><td char=".">29</td><td>0.03</td><td /><td /></tr><tr><td>Course x Teacher x Year</td><td char=".">85</td><td>0.14</td><td /><td /></tr><tr><td>Residual</td><td char=".">2,736</td><td>0.53</td><td /><td /></tr></tbody></table> </ephtml> </p> <p>1 <emph>Notes:</emph> This analysis utilized data from 2,736 students, 65 teachers, and 29 schools. "Mathematics Achievement" was defined as the Algebra post-test z-score for algebra students and the Geometry post-test z-score for geometry students. All non-dichotomous variables are grand mean centered. The reference group for the blocking variables was Cohort 1 Public School students who participated during the 2011-12 school year. The intercept represents the post-test score for students at average values for pretest, expectancy of success, and school-level proportion disadvantaged and in the reference group for all dichotomous variables (i.e. in the Cohort 1 Public School randomization block; in a control school, taking algebra, not White/Asian, and not Female).</p> <p>Using post-test standard deviation as the denominator and combining Treatment with Control groups, Geometry students gained approximately 0.91 standard deviations and Algebra students gained approximately 0.73 standard deviations from our pretest to our post-test. Thus, an impact of 0.33 <emph>SD</emph> might be thought of as roughly equivalent to about 36% (for Geometry students) to 45% (for Algebra students) of a year's learning.</p> <hd id="AN0187163326-37">Interaction Effects: For Whom Did Academic Performance Increase?</hd> <p>We investigated interactions between the PARLO program and each of the student level covariates in our model to determine whether baseline student characteristics moderated the effects of the PARLO system on mathematics content learning. There were no statistically significant interactions. In other words, students benefited from the PARLO system regardless of their course enrollment (geometry or algebra), their content pretest scores, their baseline expectancies for success in mathematics, their race, or their gender. See the online appendix for complete details regarding these analyses.</p> <hd id="AN0187163326-38">PARLO's Impact on Quantitative Measures of Students' Mathematics Motivation</hd> <p></p> <hd id="AN0187163326-39">Main Effects: Did PARLO Impact Motivation?</hd> <p>Table 2 summarizes the PARLO treatment's main effects on students' self-reported intrinsic value, utility value, expectancies for success, and long-term motivation to engage with mathematics. (See Table A11 of the online appendix for more detailed information including covariate coefficients and variance components.<bold>)</bold> As can be seen in the table, the PARLO system had substantively small and statistically non-significant effects on post-test measures of the four constructs, with effects ranging from −0.08 to −0.02 on a 5-point scale.</p> <p>Table 2. Effects of PARLO treatment on students' motivational antecedents and long-term motivation in math class.</p> <p> <ephtml> <table><thead><tr><td>Dependent Variable (Subscale)</td><td>PARLO effect Estimate</td><td>Standard Error</td><td>df</td><td>95% conf. interval</td><td>p-value</td><td>Effect size in SD units</td></tr></thead><tbody valign="top"><tr><td>Intrinsic Value</td><td>−0.06</td><td>0.06</td><td>15</td><td>(−0.18, +0.06)</td><td>.31</td><td>−0.07</td></tr><tr><td>Utility Value</td><td>−0.02</td><td>0.05</td><td>14</td><td>(−0.12, +0.07)</td><td>.60</td><td>−0.03</td></tr><tr><td>Expectancy</td><td>−0.06</td><td>0.04</td><td>51</td><td>(−0.15, +0.02)</td><td>.14</td><td>−0.08</td></tr><tr><td>Long-term Motivation</td><td>−0.08</td><td>0.06</td><td>19</td><td>(−0.21, +0.05)</td><td>.23</td><td>−0.09</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Notes:</emph> This analysis utilized data from 2,698 students, 65 teachers, and 29 schools. Dependent variables are measured on a 1-5 Likert scale. <emph>df</emph> = degrees of freedom. Standard deviations for the dependent variables were as follows: Intrinsic Value: 0.86; Utility value: 0.74; Expectancies for success: 0.83; Long-term motivation: 0.85. Reported results controlled for the following covariates: Assignment block for randomization; School-level proportion disadvantaged; course assignment (geometry or algebra), student-level gender, race, pretest, and baseline score on all four subscales of the ATMI.</p> <hd id="AN0187163326-40">Interaction Effects: Were There Any Subgroups for Whom Average Motivation Changed?</hd> <p>While there were no main effects of treatment on any of the motivation related measures, it seemed possible that one or more subgroups of students might have been impacted. We investigated this possibility by testing whether any of our four measures were impacted by interactions between treatment condition and each of our student level covariates: gender, race, geometry vs. algebra class, and baseline expectancies, intrinsic value, utility value, long-term motivation, and mathematics content test scores. After using the Benjamini-Hochberg (BH) procedure to control for multiple comparisons, none of the interactions were significant. (See the online appendix for additional details.) Thus, we found no quantitative evidence that the PARLO system affected the intrinsic value, utility value, expectancies for success, or long-term motivation of any student subgroup.</p> <hd id="AN0187163326-41">Mechanisms Leading to PARLO's Positive Impact on Mathematics Performance</hd> <p></p> <hd id="AN0187163326-42">Opportunities to Learn</hd> <p>We had hypothesized that PARLO would improve student achievement partly by improving opportunities to learn. Teacher interviews provided detail about how increased opportunities may have operated.</p> <hd id="AN0187163326-43">Reassessment Opportunities</hd> <p>The central feature of PARLO is encouraging students to re-study and reassess to achieve proficiency or high performance on each learning outcome, and a recurring theme—emerging among 89% of teachers interviewed (31/35)—was that teachers felt this central feature of PARLO led directly to better student learning of mathematics content. Specifically, teachers reported that reassessment opportunities were useful to students of all ability levels. Regarding struggling students, teachers made comments like the following:</p> <p>...some of the students that have typically struggled in math are able to see how their hard work pays off... I think it's something that they have not had before, because if they struggled on a test, even if they came in and got help, that test grade still remained, and they weren't able to demonstrate their understanding. (Teacher #31)</p> <p>Teachers noted that high achievers also benefited, specifically from opportunities to reassess for high performance. For example:</p> <p>The kids that really, really want to learn it will do anything that they can to achieve that high performance. In the traditional class, they've got that one shot to master it, and they can't revisit it. So, they can't ever get to that achievement. They can't push themselves to reach that next level. It's either one or none. [With] the PARLO, they can go back and push themselves to get it. (Teacher # 32)</p> <hd id="AN0187163326-44">Clearer Focus on Learning</hd> <p>Sixty percent of teachers interviewed (21/35) also noted that providing feedback and reassessment opportunities for each learning outcome helped students focus their learning efforts fruitfully, asking better questions and showing better understanding of what they needed to work on—for example:</p> <p>They're coming to me with better questions. They're not just coming to me and saying, "What can I do?" They come in knowing where they need to focus, knowing what they need to work on. (Teacher #15)</p> <hd id="AN0187163326-45">Peer Interactions</hd> <p>Although the interviews did not explicitly ask about student peer support, 49% of teachers interviewed (17/35) brought up peer collaboration under PARLO as one contributor to better student learning—for example:</p> <p>They tend to actually be getting involved more with other students, because... they also recognize quickly that in teaching others, they're getting a better understanding themselves because now they can present it and defend. (Teacher #6)</p> <hd id="AN0187163326-46">Revisiting Topics</hd> <p>Thirty-one percent of teachers interviewed (11/35) noted the pedagogical value of allowing students to revisit topics over time—for example:</p> <p>One big thing is that PARLO allows me time to come back and go over things that we already covered... And by going over the learning outcomes... I see students that have gaps in their learning. Like, one student, she was truant in February and March, but she is a very high-level student, so now I see her learning all of that stuff and connecting the dots between our most recent learning outcomes and stuff that we learned three months ago. And also, just kids filling in the gaps in their own learning as we go through it. "Oh, I remember this" or "I forget how we do this" and then re-remembering it... I noticed that they're getting it very easily now the second time around. (Teacher #23)</p> <hd id="AN0187163326-47">Increased Engagement</hd> <p>While our quantitative analysis did not detect any program impact on motivation, teacher interviews indicated that student engagement, which is often associated with motivation, did increase. Eighty percent of teachers (28/35) interviewed said that under PARLO, their students were more likely to participate in discussions about content with both the teacher and their fellow students. The same percent of teachers said that students were more likely to ask questions when concepts were not well understood.</p> <p>When asked to describe "successes" under the PARLO program, 63% of teachers (22/35) reported increased student persistence in the face of initial difficulty, increased ownership of the learning process, and increased responsibility—for example:</p> <p>Students are taking ownership in what they are learning... They are learning how to organize and keep up with it, each marking period they are getting better at it. (Teacher #15)</p> <p>Earlier this year, the K-8 math coordinator visited my classroom, to follow up on some of her students from last year. After class she said to me, "Was that Rachel [pseudonym] In the front row, with her head up?" I said, "Yes." And the coordinator said she could not believe it was the same girl because last year, Rachel, when she came to class, sat in the back of the classroom with her head on her desk. This year, Rachel told me during one-on-one conference that she wants high performance on every Outcome. And she's always asking for more and more tests. She's also begging me to teach her next year as she wants to stay in a PARLO classroom. (Teacher #4)</p> <hd id="AN0187163326-48">Examining Potential Reasons behind PARLO's Non-Significant Effects on Quantitative Measures o...</hd> <p>Posner's ([<reflink idref="bib23" id="ref48">23</reflink>]) pilot study of the PARLO system found positive effects on intrinsic value, utility value, expectancy for success, and motivation in college statistics. In contrast, the current study found no effects of PARLO on the motivation-related constructs we measured. Our qualitative analysis investigated why this might have occurred.</p> <hd id="AN0187163326-49">Decreased Engagement for Some Students</hd> <p>Despite teachers' reports that under PARLO overall engagement increased, a minority of teachers expressed concern that the PARLO system might interfere with some students' engagement. They identified two potential problems, which we labeled "contentment" and "procrastination."</p> <hd id="AN0187163326-50">Contentment</hd> <p>Twenty-nine percent of teachers interviewed (10/35), indicated that PARLO might discourage some students from doing their best. Specifically, they noted that some students seemed content with proficient performance and reluctant to try high cognitive demand problems in order to achieve high performance—for example:</p> <p>What I stopped doing was marking the problems with [an] asterisk if it were blue [high performance], because students that are not that motivated would just choose not to do those...(Now) they don't know if it's green [proficient] or blue [high performance] until we get the assessments back. (Teacher #28)</p> <hd id="AN0187163326-51">Procrastination</hd> <p>Twenty-six percent of teachers interviewed (9/35) indicated that the PARLO system might encourage some students to put off studying under the assumption that if they did poorly at first, they could always reassess later—for example:</p> <p>As far as kids that don't take advantage, we have so many. They're the kids that aren't motivated... John [pseudonym] has gotten to the point where he pretty much hands in tests with nothing on them. And I think that he does subconsciously think, "Oh, I can reassess," but he doesn't follow through. (Teacher #36)</p> <hd id="AN0187163326-52">Potential Net Zero Impact on Expectancies for Success</hd> <p>Re-assessment opportunities under PARLO appeared to increase students' expectancies for success, but this positive effect was perhaps balanced by a negative effect on expectancies caused by student uncertainties about their final grades.</p> <hd id="AN0187163326-53">Positive Effects of Re-Assessment Opportunities</hd> <p>In 2011–12, 678 PARLO students completed an open-ended survey about their experiences with the new assessment system. When asked "What do you like about PARLO?" the most common word or phrase students mentioned was "retake" (<emph>N =</emph> 90), followed by "make up" (<emph>N</emph> = 52) and "another chance" (<emph>N</emph> = 48). Students made comments like, "It gives you the chance to retake tests, quizzes, or homework. You can work at your own pace without having to rush to learn something new," and "I think PARLO is very good for math students struggling."</p> <hd id="AN0187163326-54">Negative Effects of Uncertainties about Final Grades</hd> <p>When asked, "What do you not like about PARLO?" 41% of students (277/678) indicated that they felt they never knew their exact grade until report card time or the end of the year, which they found confusing. When asked, "How would you describe PARLO to next year's ninth grade students?," the single most commonly mentioned word was "confusing," mentioned by 73 of the 678 students interviewed. Even students who found the system helpful complained about uncertainty in grading, making comments like, "PARLO is very helpful. You can always bring your grade up. However, one downer is that you never know what your grade is until report cards arrive."</p> <hd id="AN0187163326-55">No Detected Impact on Intrinsic Value and Utility Value</hd> <p>As noted in our methods section, the first step in our qualitative analysis was inductive, identifying major themes that emerged from the data. Changes in students' intrinsic value for mathematics or changes in students' beliefs about mathematics utility value did not emerge as themes. Unlike student expectancies for success, which seemed to have a balance of positive and negative impacts, we found no evidence that the PARLO system had either a positive or a negative impact on students' valuing of mathematics.</p> <hd id="AN0187163326-56">Unmeasured Motivational Constructs</hd> <p>A majority of the teachers interviewed (24 out of 35, or 69%) indicated that students' experiences with PARLO enhanced their motivation. The discrepancy between this finding and our quantitative results (i.e., no program effect on motivational antecedents or long-term motivation) may have occurred because, by "motivation," teachers were referring to constructs we did not measure quantitatively.</p> <hd id="AN0187163326-57">Mastery Goals</hd> <p>Of the 35 teachers interviewed, 24 (69%) indicated that students' experiences with PARLO promoted the adoption of mastery goals. For example:</p> <p>...last year all they were concerned about were points and grades. This year they are talking about math more than my students last year. They know their content. (Teacher #26)</p> <hd id="AN0187163326-58">Growth Mindsets</hd> <p>Of the 35 teachers interviewed, 20 (57%) also reported that, under PARLO, students were more likely to adopt a growth mindset—for example:</p> <p>They are finally, really understanding that knowledge is gained and built over time, you don't just know something, you don't just get something and that's it. You have to work at it, and if you want to keep it you have to continue to work at it. (Teacher #24)</p> <p>I think their work ethic is better. They realize they need to keep on top of stuff, keep working. If you fail one thing, don't just say, 'Oh, it's over.' It's not over! You can still learn stuff. (Teacher #1)</p> <hd id="AN0187163326-59">Autonomy</hd> <p>A smaller but still substantial number of teachers (34%; 12/35) described a different mechanism by which PARLO increased motivation: that under PARLO, some students felt that they had autonomy and could control their own mathematics destinies (Ryan & Deci, [<reflink idref="bib24" id="ref49">24</reflink>])—for example:</p> <p>They have the ownership of their grade now. It's no longer "What can he give me?" So, the ball is in her court now. It's no longer, "You failed me, or you gave me this grade." They talk about getting their grade up to where they want it to be. (Teacher #16)</p> <p>It should be noted, however, that some teachers highlighted one characteristic of the PARLO assessment system that caused some students to perceive <emph>decreased</emph> autonomy: Under PARLO, a student's grades are based solely on evidence of content understanding. Compliance measures like homework completion and attendance no longer count toward grades. Some students expressed concern that they could control their compliance, but not necessarily ensure their learning.</p> <hd id="AN0187163326-60">Relatedness</hd> <p>Finally, PARLO may also have helped meet students' needs for relatedness (Ryan & Deci, [<reflink idref="bib24" id="ref50">24</reflink>]). As noted above, teachers reported that increased peer support was one of the reasons that mathematics performance improved under PARLO. This increased peer support may have been effective because it improved students' sense of relatedness. For example, a teacher described classmates' response to a PARLO student coming up to the chalkboard to demonstrate proficiency on a learning outcome:</p> <p>They were still supportive...And they're like, "Come on, Jenny (pseudonym), you can do it! You just have a few more steps to go! You're almost there!" Never did they yell out, "Hey stupid, wrong step..." They're using the language [of PARLO]. (Teacher # 4)</p> <hd id="AN0187163326-61">The Emergence of Student Motivation as a Potential Moderator of PARLO's Impact</hd> <p>We turn now to an unexpected finding that emerged from our qualitative analysis. We had theorized that the PARLO system might improve students' math motivation. In contrast, teacher interviews suggested that the converse might also be true: higher student motivation might improve the effectiveness of the PARLO system.</p> <p>Why did some students respond to the PARLO system with active and enthusiastic engagement, whereas other students responded with apathy and work avoidance? As illustrated in the quote from Teacher #36 in our discussion (above) of procrastination, teachers attributed the different student behaviors to differences in student motivation. In fact, 71% of teachers (25/35) noted that "motivation," "caring," "willingness" or a similar concept was important in determining the effectiveness of PARLO, making comments like the following:</p> <p>Motivation is huge. And if the kids don't have the motivation, then who cares if they can take a test? They don't care. (Teacher #2)</p> <p>The thing I really like about PARLO is the kids that are willing to work and seem to actually care, are the ones who really seem to benefit from it. (Teacher #9)</p> <hd id="AN0187163326-62">Quantitative Verification of Motivation as a Moderator of PARLO Impact</hd> <p>As noted in our Methods section, when we designed this study we did not plan to conduct a quantitative analysis to see whether student motivation might moderate the PARLO treatment's effect on student mathematics learning: However, after the qualitative data indicated that this might be the case, we conducted a quantitative analysis to see whether we could corroborate the unanticipated qualitative finding. If student motivation did moderate treatment effects on student achievement, we would expect to see a positive interaction between PARLO treatment and each of our three measured psychological antecedents of motivation, as well as between PARLO and our measure of long-term motivation. Table 3 summarizes the results of the analysis.</p> <p>Table 3. Interactions with treatment condition: do students' perception of intrinsic value, utility value, expectancy of success, and long-term motivation moderate PARLO's impact on mathematic achievement?</p> <p> <ephtml> <table><thead><tr><td>Interaction</td><td>n</td><td>Effect size Estimate</td><td>Standard Error</td><td>Degrees of Freedom</td><td>95% conf. interval</td><td>p-value</td></tr></thead><tbody valign="top"><tr><td>PARLO × Average Intrinsic Value</td><td>2,529</td><td>0.09</td><td>0.04</td><td>2,482</td><td>(+0.01, +0.17)</td><td char=".">0.026</td></tr><tr><td>PARLO × Average Utility Value</td><td>2,532</td><td>0.11</td><td>0.05</td><td>2,483</td><td>(+0.01, +0.21)</td><td char=".">0.028</td></tr><tr><td>PARLO × Average Expectancy</td><td>2,529</td><td>0.13</td><td>0.04</td><td>2,481</td><td>(+0.05, +0.22)</td><td char=".">0.002</td></tr><tr><td>PARLO × Average Longterm Motivation</td><td>2,533</td><td>0.07</td><td>0.04</td><td>2,485</td><td>(−0.02, +0.15)</td><td char=".">0.13</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Notes:</emph> This analysis utilized data from 65 teachers, and 29 schools. "Mathematics Achievement" was defined as the Algebra Post-test z-score for algebra students and the Geometry post-test z-score for geometry students. n = number of students with data available for the model testing each specific moderator. Because the analysis utilized average of baseline and end-of-year scores, students who were missing end-of-year scores were not included in the analysis. Reported results controlled for the following covariates: Assignment block for randomization; School-level proportion disadvantaged; course assignment (geometry or algebra), student-level gender, race, pretest, pretest-squared, main effects of the motivation subscale being studied and main effects of the PARLO treatment.</p> <p>As can be seen in the table, all four interactions were positive and three of the four were statistically significant. To interpret the table substantively, the positive impact of PARLO on the end of the year mathematics content score increased by 0.09, 0.11, 0.13, and 0.07 <emph>SD</emph> for each 1-point increase on the 5-point measure of intrinsic value, utility value, expectancies, and long-term motivation, respectively. Thus, PARLO's impact on individual students' mathematics achievement was positively affected by their levels of motivation in mathematics.</p> <p>We used the BH adjustment for multiple comparisons to account for the fact that Table 3 employs four statistical tests. The BH procedure confirmed the statistical significance of intrinsic value, utility value, and expectancies for success as moderators. Note that the analysis reported in Table 3 averaged the baseline and end-of-year scores to measure each of the four motivation-related constructs. As a sensitivity check, we also ran the moderation analysis that instead used only baseline scores for all four motivation-related moderators. The results pointed in the same direction as those reported in Table 3 but were not statistically significant. See the online appendix for details about the BH procedure and the sensitivity check.</p> <hd id="AN0187163326-63">Discussion</hd> <p>The present work investigated the impact of the PARLO standards-based grading system on ninth graders' learning of algebra and geometry, hypothesizing that the reengineered grading system would positively impact student learning both by directly providing students additional opportunities to learn, and in a mediated manner by enhancing student motivation. Partly supporting these predictions, our quantitative analysis indicated that PARLO had a positive impact on ninth graders' learning of mathematics content, with an estimated effect size of 0.33 <emph>SD</emph>. In normally distributed data, an effect size of 0.33 <emph>SD</emph> would move a student from the 50<sups>th</sups> percentile on a test up to the 63<sups>rd</sups> percentile. Comparing the PARLO impact to typical student growth from pre- to post-test in our data set, we estimate 0.33 <emph>SD</emph> to be the equivalent to about 36% to 45% of a year's learning. The program was effective regardless of students' race, gender, or prior achievement.</p> <p>As discussed, we had also hypothesized that one mechanism through which PARLO might improve achievement would be the mediating effect of motivation: the PARLO program would improve motivation, which in turn would improve student engagement and learning. Our quantitative analysis did not support this hypothesis. We found no significant effect of PARLO on any of our four motivation-related measures.</p> <p>Our qualitative analysis provided insights into the reasons behind our quantitative findings. As predicted, teachers reported a number of ways that increased opportunities to learn under PARLO improved student achievement. These opportunities included the direct effects of encouraging students to re-study and reassess, helping students to focus on the content they needed to learn, increased support from students' peers, and providing opportunities to revisit topics over time.</p> <p>Qualitative data also provided insight into why students did not report increased expectancies for success under PARLO. There appear to have been two countervailing tendencies, potentially leading to a net zero impact. As predicted, students' appreciation of the opportunities to reassess appears to have strengthened their expectancies. Balancing this, however, one aspect of our PARLO implementation may have had the unintended consequence of decreasing students' expectancies. The assessment system at YWLCS, which inspired PARLO, did not assign students letter grades, instead using long report cards that described proficiency by learning outcome. In our professional development sessions, we showed these report cards to teachers and suggested, "When all outcomes have been rated, students' overall proficiency can be converted to a letter grade if necessary." As a result, many PARLO teachers provided students with little information about their overall grade until all outcomes had been rated at the end of the semester or year. This appears to have created anxiety and confusion among some students about their grades, which may have reduced their expectancies for success. We anticipate that future implementations of the PARLO system will correct this problem, instead encouraging teachers to help students understand where they stand relative to a final mathematics grade and what they need to do if they want to improve it.</p> <p>Our quantitative analysis did not indicate any program effects on students' valuing of math class, and the qualitative data seldom mentioned the topic. While we had hypothesized that changing the assessment system to focus less on quick learning and ranking and more on mastery of course content would positively impact students' value of mathematics, this expectation may have been unreasonable. Making mathematics more enjoyable (i.e., increasing intrinsic value) or increasing the perceived utility value of mathematics may be most directly impacted by designing engaging instructional activities or by explicitly connecting curriculum to potential applications, neither of which was a focus of PARLO.</p> <p>Teachers interviewed did, however, report that the PARLO system positively impacted student engagement and motivation in ways that our quantitative analysis did not address. Motivational antecedents that may have been positively affected include growth mindsets and mastery goals, autonomy, and relatedness.</p> <p>An additional unanticipated and noteworthy finding emerged from our qualitative analysis. Teacher interviews indicated that student motivation moderated PARLO's impact—that is, that more motivated students benefited more strongly from PARLO. Once this finding emerged from our qualitative analysis, our quantitative data provided some corroboration. Students who scored higher on year-average scores for intrinsic value, utility value, or expectancies for success in mathematics benefited more from the PARLO system than did students who scored lower on those same measures.</p> <p>To unify these distinct but interconnected findings, we return to Figure 1, which depicts our current conceptual model regarding how the PARLO system may influence students' academic experiences and learning. Solid lines in the figure indicate relationships that have been supported either by prior research or that were supported both by our present quantitative <emph>and</emph> qualitative analyses. Dashed lines indicate relationships that we hypothesize based on patterns that emerged in our qualitative data, but have not received robust quantitative support in the present or past research, and the lack of a connection indicates relationships that did not emerge in the present or prior research.</p> <p>We note that this current conceptual model includes two refinements to our initial hypotheses. First, we now theorize that the PARLO system impacts motivation indirectly through improving some, but not all, of the motivational antecedents we have considered. Specifically, the current study provides evidence that the PARLO standards based grading system impacts expectancies, autonomy, relatedness, growth mindset, and mastery goals–but not perceived intrinsic value or utility value. Second, motivation is believed to moderate the relationship between PARLO-provided opportunities to learn and actual learning. Thus, supplementing the PARLO program with additional support for motivational antecedents would in theory magnify the impact of the program on learning.</p> <hd id="AN0187163326-64">Study Limitations</hd> <p>We designed our study using an expectancy-value framework (Wigfield & Eccles, [<reflink idref="bib37" id="ref51">37</reflink>]), and our quantitative measures reflect that design. Findings that emerged from our qualitative analysis of teacher interviews caused us to expand the framework to include constructs from growth mindset theory (Dweck, [<reflink idref="bib10" id="ref52">10</reflink>]), self-determination theory (Ryan & Deci, [<reflink idref="bib24" id="ref53">24</reflink>]), and achievement goal theory (Senko, [<reflink idref="bib29" id="ref54">29</reflink>]). However, because these connections emerged through inductive qualitative analyses conducted following data collection, quantitative data on growth mindsets, autonomy, relatedness, and mastery goals were not collected. This may have contributed to the discrepancy between teacher interviews, which reported increased motivation under PARLO, and quantitative analyses, which did not detect a program impact on motivation.</p> <p>The study would have been stronger had we interviewed the control teachers. Doing so might have provided important insight into the ways in which treatment and control students' experiences in the classroom differed.</p> <p>The PARLO system works by changing the grading system to provide students with more opportunities to learn. To be effective, students must understand the system. We did not provide direct assistance to help teachers explain the PARLO grading system to their students, and our data indicate that some students did not understand it. In a free response survey, 73 out of 678 students described PARLO as "confusing." Future implementations might be able to increase PARLO's effectiveness by providing materials and techniques teachers can use to explain the new system to their students.</p> <p>Informal aspects of our intervention may have influenced outcomes in important but untracked ways. For example, some but not all teachers encouraged peer-to-peer support by grading a student as high performance on a learning outcome if they helped another student attain proficiency. "Flashback days" were an innovation developed by some participating teachers and shared with all PARLO teachers through PLCs, but not implemented by all of them. Teachers also developed techniques to reduce workload requirements under PARLO. Some of these innovations may be key ingredients that greatly enhance program success, but we were not able to determine which innovations were the most important. Future research should seek to track how such elements—in addition to factors like the level of transparency with which proficiency ratings are translated into final grades, the way the program is introduced to students, the amount and types of supports provided to parents and guardians, and the number of courses or grade levels in a school that implement the PARLO program—might affect program results.</p> <p>Finally, we note that treatment and control teachers had all volunteered, if assigned to treatment, to change a central pillar of their classroom instruction: the grading system. Implementing the PARLO system required considerable effort from the teachers involved. Results may not generalize to teachers or schools who are less willing to make such a change.</p> <hd id="AN0187163326-65">Conclusion: The Future</hd> <p>Working with teachers who were already implementing formative assessment and mathematics instruction designed around clearly state learning outcomes, the PARLO standards-based grading program reinforced those teaching practices and added two new elements: reassessment for full credit after further learning; and basing a student's final grade on the number of learning outcomes the student learned at a proficient level and the number of learning outcomes the student learned at a high performance level. This treatment improved the amount of mathematics students learned by 0.33 standard deviations, which is roughly 36% to 45% of the effect an entire year's learning had for our sample.</p> <p>Moving forward, we anticipate that future work will not replicate the current project precisely, but rather build on lessons learned. For example, we recommend that future implementations correct elements of the program that may have unintentionally reduced program effectiveness. Specifically, we recommend that procedures for computing final grades be made transparent and explained clearly to students; and that the program provide materials and procedures to assist teachers in introducing the PARLO program to students. If possible, future implementations should provide optional-use assessment, reassessment, and re-teaching materials to reduce the amount of work required of PARLO teachers. Future work might also consider implementing standards-based grading in more courses or more grade levels (vs. ninth grade math only), so that the PARLO grading system does not seem so unusual (and thus potentially confusing) compared to grading in other courses.</p> <p>Our data also provides evidence that the PARLO standards-based grading system, while effective regardless of race, gender, or prior knowledge, is especially effective for students who are more highly motivated. This finding makes sense: the PARLO system provides students additional opportunities to learn, and more motivated students are more likely to take advantage of those opportunities. If one views motivation as a fixed attribute of students, then this finding could be seen as troubling. A standards-based grading system would have positive effects, but it would be especially helpful for the "haves" who are already motivated and engaged.</p> <p>Fortunately, however, research has consistently shown that motivation is malleable: the mindsets that are psychological antecedents to motivation, including beliefs in intrinsic and utility value of subject matter, expectancies for success, growth mindsets, autonomy, and relatedness are greatly impacted by teacher behaviors and classroom learning conditions (Binning & Browman, [<reflink idref="bib3" id="ref55">3</reflink>]; Farrington et al., [<reflink idref="bib11" id="ref56">11</reflink>]). Furthermore, educational psychologists have developed interventions that are designed to target the specific underlying psychological processes that can promote student motivation and engagement. Importantly, a growing body of research has found that such interventions can have strong positive impacts on student learning and well-being, but only in contexts that are fertile ground for supporting the target psychological factors (Walton & Yeager, [<reflink idref="bib35" id="ref57">35</reflink>], Yeager & Walton, [<reflink idref="bib41" id="ref58">41</reflink>]). To create such "fertile ground" researchers have suggested that it may be especially important to reform grading systems in order to make them less focused on summative assessment and ranking (Farrington et al., [<reflink idref="bib11" id="ref59">11</reflink>]). This raises an exciting avenue for future work: implementing the PARLO standards-based grading system, while simultaneously supporting PARLO with interventions that foster student motivation. Such a combined intervention could theoretically magnify the impact of PARLO by supporting student motivation, while simultaneously magnifying the impact of the motivation interventions by creating a grading environment in which they can have a larger impact.</p> <p>Finally, we note that there is no theoretical reason to expect the PARLO standards-based grading system to be less successful in other grade levels or in other disciplines than it was in ninth grade mathematics. We encourage other researchers to build on the potential identified in our work, finding effective ways to implement standards-based grading. We hope that the results will have large benefits for cultivating student learning.</p> <hd id="AN0187163326-66">Open Research Statements</hd> <p></p> <hd id="AN0187163326-67">Study and Analysis Plan Registration</hd> <p>There is no study and analysis plan registration associated with this manuscript.</p> <hd id="AN0187163326-68">Data, Code, and Materials Transparency</hd> <p>The data, code, and materials underlying the results reported in this manuscript are available on the Open Science Framework: https://osf.io/htkfs.</p> <hd id="AN0187163326-69">Design and Analysis Reporting Guidelines</hd> <p>This manuscript is accompanied by a completed JREE Randomized Trial Checklist.</p> <hd id="AN0187163326-70">Transparency Declaration</hd> <p>The lead author (the manuscript's guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.</p> <hd id="AN0187163326-71">Replication Statement</hd> <p>This manuscript reports an original study.</p> <hd id="AN0187163326-72">Acknowledgments</hd> <p>The authors thank Helen Kramer, Holly Bozeman, John Baker, Teresa Harrison, Maurice Bun and two anonymous reviewers for their feedback. This research was supported by the National Science Foundation under grant number DRL-0918474. Research involving human subjects was reviewed and approved by the Institutional Review Board (IRB) at Research for Better Schools. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.</p> <hd id="AN0187163326-73">Disclosure Statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <hd id="AN0187163326-74">Open Scholarship</hd> <p>Graph</p> <p>This article has earned the https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/ badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at http://10.35542/osf.io/pzc3f.</p> <ref id="AN0187163326-75"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref40" type="bt">1</bibl> <bibtext> Supplemental data for this article can be accessed online at https://doi.org/10.1080/19345747.2023.2287594.</bibtext> </blist> </ref> <ref id="AN0187163326-76"> <title> References </title> <blist> <bibtext> Akaike, H. (2011). Akaike's information criterion. In M. Lovric (Ed.), International encyclopedia of statistical science (pp. 25 – 25). Springer. https://doi.org/10.1007/978-3-642-04898-2_110</bibtext> </blist> <blist> <bibl id="bib2" idref="ref45" type="bt">2</bibl> <bibtext> Allison, P. D. (2012). Handling missing data by maximum likelihood. In SAS Global Forum, Paper 312-212. Retrieved September 8, 2021, from https://statisticalhorizons.com/wp-content/uploads/MissingDataByML.pdf.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref55" type="bt">3</bibl> <bibtext> Binning, K. R., & Browman, A. S. (2020). Theoretical, ethical, and policy considerations for conducting social–psychological interventions to close educational achievement gaps. Social Issues and Policy Review, 14 (1), 182 – 216. https://doi.org/10.1111/sipr.12066</bibtext> </blist> <blist> <bibl id="bib4" idref="ref4" type="bt">4</bibl> <bibtext> Black, P., & Wiliam, D. (2010). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 92 (1), 81 – 90. https://doi.org/10.1177/003172171009200119</bibtext> </blist> <blist> <bibl id="bib5" idref="ref3" type="bt">5</bibl> <bibtext> Bloom, B. S. (1968). Learning for mastery. Instruction and curriculum. Regional education laboratory for the Carolinas and Virginia, topical papers and reprints, number 1. Evaluation Comment, 1 (2), 1 – 12. https://eric.ed.gov/?id=ED053419</bibtext> </blist> <blist> <bibl id="bib6" idref="ref46" type="bt">6</bibl> <bibtext> Bun, M. J. G., & Harrison, T. D. (2018). OLS and IV estimation of regression models including endogenous interaction terms. Econometric Reviews, 38 (7), 814 – 827. https://doi.org/10.1080/07474938.2018.1427486</bibtext> </blist> <blist> <bibl id="bib7" idref="ref47" type="bt">7</bibl> <bibtext> Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research, 42 (3), 294 – 320. https://doi.org/10.1177/0049124113500475</bibtext> </blist> <blist> <bibl id="bib8" idref="ref33" type="bt">8</bibl> <bibtext> Clymer, J. B., & Wiliam, D. (2007). Improving the way we grade science. Educational Leadership, 64 (4), 36 – 42. https://eric.ed.gov/?id=EJ766296</bibtext> </blist> <blist> <bibl id="bib9" idref="ref21" type="bt">9</bibl> <bibtext> Covington, M. V., & Omelich, C. L. (1984). Task-oriented versus competitive learning structures: Motivational and performance consequences. Journal of Educational Psychology, 76 (6), 1038 – 1050. https://doi.org/10.1037/0022-0663.76.6.1038</bibtext> </blist> <blist> <bibtext> Dweck, C. S. (2007). Mindsets: The new psychology of success. Ballantine.</bibtext> </blist> <blist> <bibtext> Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners. The role of noncognitive factors in shaping school performance: A critical literature review. University of Chicago Consortium on Chicago School Research. Retrieved March 18, 2013, from <ulink href="http://ccsr.uchicago.edu/publications/teaching-adolescents-become-learners-role-noncognitive-factors-shaping-school">http://ccsr.uchicago.edu/publications/teaching-adolescents-become-learners-role-noncognitive-factors-shaping-school</ulink>.</bibtext> </blist> <blist> <bibtext> Farrington, C. A., & Small, M. (2008). A new model of student assessment for the 21st century. American Youth Policy Forum. https://<ulink href="http://www.researchgate.net/profile/Camille-Farrington/publication/253449779">www.researchgate.net/profile/Camille-Farrington/publication/253449779</ulink></bibtext> </blist> <blist> <bibtext> Haley, J. M. (2015). To curve or not to curve? The effect of college science grading policies on implicit theories of intelligence, perceived classroom goal structures and self-efficacy [Unpublished doctoral dissertation]. Boston College. <ulink href="http://hdl.handle.net/2345/bc-ir:104165">http://hdl.handle.net/2345/bc-ir:104165</ulink></bibtext> </blist> <blist> <bibtext> Jury, M., Smeding, A., & Darnon, C. (2015). First-generation students' underperformance at university: The impact of the function of selection. Educational Psychology, 6, 710. https://doi.org/10.3389/fpsyg.2015.00710</bibtext> </blist> <blist> <bibtext> Kingston, N., & Nash, B. (2011). Formative assessment: A meta-analysis and a call for research. Educational Measurement: Issues and Practice, 30 (4), 28 – 37. https://doi.org/10.1111/j.1745-3992.2011.00220.x</bibtext> </blist> <blist> <bibtext> Kuznetsova, A., Brockhoff, P., & Christensen, R. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82, 1 – 26. https://doi.org/10.18637/jss.v082.i13</bibtext> </blist> <blist> <bibtext> Leahy, S., Lyon, C., Thompson, M., & William, D. (2005). Classroom assessment: Minute by minute, day by day. Educational Leadership, 63 (3), 19 – 24. https://eric.ed.gov/?id=EJ745452</bibtext> </blist> <blist> <bibtext> Marzano, R. J. (2010). Formative assessment & standards-based grading. Marzano Research Laboratory.</bibtext> </blist> <blist> <bibtext> May, H., Perez-Johnson, I., Haimson, J., Sattar, S., & Gleason, P. (2009). Using state tests in education experiments: A discussion of the issues (NCEE 2009-013) National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.</bibtext> </blist> <blist> <bibtext> Mills, V. L., & Silver, E. A. (2018). Putting it all together and moving forward: Concluding thoughts. In E. A. Silver & V. L. Mills (Eds.), A fresh look at assessment in mathematics teaching. National Council of Teachers of Mathematics.</bibtext> </blist> <blist> <bibtext> Murphy, R., Roschelle, J., Feng, M., & Mason, C. A. (2020). Investigating efficacy, moderators and mediators for an online mathematics homework intervention. Journal of Research on Educational Effectiveness, 13 (2), 235 – 270. https://doi.org/10.1080/19345747.2019.1710885</bibtext> </blist> <blist> <bibtext> National Governors Association Center for Best Practices, Council of Chief State School Officers. (2010). Common core state standards. National Governors Association Center for Best Practices, Council of Chief State School Officers.</bibtext> </blist> <blist> <bibtext> Posner, M. A. (2011). The impact of a proficiency-based assessment and reassessment of learning outcomes system on student achievement and attitudes. Statistics Education Research Journal, 10 (1), 3 – 14. https://eric.ed.gov/?id=EJ925274 https://doi.org/10.52041/serj.v10i1.352</bibtext> </blist> <blist> <bibtext> Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860. https://doi.org/10.1016/j.cedpsych.2020.101860</bibtext> </blist> <blist> <bibtext> Sanchez, C. E., Atkinson, K. M., Koenka, A. C., Moshontz, H., & Cooper, H. (2017). Self-grading and peer-grading for formative and summative assessments in 3rd through 12th grade classrooms: A meta-analysis. Journal of Educational Psychology, 109 (8), 1049 – 1066. https://doi.org/10.1037/edu0000190</bibtext> </blist> <blist> <bibtext> Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2 (6), 110 – 114. https://doi.org/10.2307/3002019</bibtext> </blist> <blist> <bibtext> Scarlett, M. H. (2018). "Why did I get a C?": Communicating student performance using standards-based grading. InSight: A Journal of Scholarly Teaching, 13, 59 – 75. <ulink href="http://files.eric.ed.gov/fulltext/EJ1184948.pdf">http://files.eric.ed.gov/fulltext/EJ1184948.pdf</ulink> https://doi.org/10.46504/14201804sc</bibtext> </blist> <blist> <bibtext> Schneider, J., & Hutt, E. (2014). Making the grade: A history of the A–F marking scheme. Journal of Curriculum Studies, 46 (2), 201 – 224. https://doi.org/10.1080/00220272.2013.790480</bibtext> </blist> <blist> <bibtext> Senko, C. (2016). Achievement goal theory: A story of early promises, eventual discords, and future possibilities. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (2nd ed., pp. 75 – 95). Routledge. https://<ulink href="http://www.researchgate.net/profile/Corwin-Senko/publication/312591307">www.researchgate.net/profile/Corwin-Senko/publication/312591307</ulink></bibtext> </blist> <blist> <bibtext> Smeding, A., Darnon, C., Souchal, C., Toczek-Capelle, M.-C., & Butera, F. (2013). Reducing the socio-economic status achievement gap at university by promoting mastery-oriented assessment. PloS One, 8 (8), e71678. https://doi.org/10.1371/journal.pone.0071678</bibtext> </blist> <blist> <bibtext> Supovitz, J. A., Ebby, C. B., Remillard, J., & Nathenson, R. A. (2018). Experimental impacts of the ongoing assessment project on teachers and students. CPRE Research Report # RR 2018-1. Consortium for Policy Research in Education. https://eric.ed.gov/?id=ED593465</bibtext> </blist> <blist> <bibtext> Tapia, M., & Marsh, G. E. (2004). An instrument to measure mathematics attitudes. Academic Exchange Quarterly, 8, 16 – 22. Downloaded 8/30/2021 from <ulink href="http://www.rapidintellect.com/AEQweb/cho25344l.htm">http://www.rapidintellect.com/AEQweb/cho25344l.htm</ulink></bibtext> </blist> <blist> <bibtext> U.S. Department of Education. (2020). What works clearinghouse procedures handbook, v4.1. Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. Downloaded 8/30/2021 from https://ies.ed.gov/ncee/wwc/handbooks</bibtext> </blist> <blist> <bibtext> U.S. Department of Education. (2022). What works clearinghouse procedures and standards handbook, v5.0. Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, What Works Clearinghouse. Downloaded 12/15/2022 from https://ies.ed.gov/ncee/wwc/handbooks</bibtext> </blist> <blist> <bibtext> Walton, G. M., & Yeager, D. S. (2020). Seed and soil: Psychological affordances in contexts help to explain where wise interventions succeed or fail. Current Directions in Psychological Science, 29 (3), 219 – 226. https://doi.org/10.1177/0963721420904453</bibtext> </blist> <blist> <bibtext> Webb, N. L. (2002). Depth-of-knowledge levels for four content areas. Language Arts, 28, 1 – 9. <ulink href="http://facstaff.wcer.wisc.edu/normw/All">http://facstaff.wcer.wisc.edu/normw/All</ulink> content areas DOK levels 2032802.pdf.</bibtext> </blist> <blist> <bibtext> Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement and motivation. Contemporary Educational Psychology, 25 (1), 68 – 81. https://doi.org/10.1006/ceps.1999.1015</bibtext> </blist> <blist> <bibtext> Wiliam, D. (2011). Embedded formative assessment. Solution Tree.</bibtext> </blist> <blist> <bibtext> Yeager, D. (2017). Social-emotional learning programs for adolescents. The Future of Children, 27 (1), 73 – 94. https://doi.org/10.1353/foc.2017.0004</bibtext> </blist> <blist> <bibtext> Yeager, D. S., Lee, H. Y., & Dahl, R. E. (2017). Competence and motivation during adolescence. In A. J. Elliot, C. S. Dweck, & D. S. Yeager (Eds.), Handbook of competence and motivation: Theory and application (pp. 431 – 448). The Guilford Press. https://<ulink href="http://www.researchgate.net/publication/337856583">www.researchgate.net/publication/337856583</ulink></bibtext> </blist> <blist> <bibtext> Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education: They're not magic. Review of Educational Research, 81 (2), 267 – 301. https://doi.org/10.3102/0034654311405999</bibtext> </blist> <blist> <bibtext> Zuur, Alain, Ieno, E. N., Walker, N., Saveliev, A., & Smith, G. M. (2009). Mixed effects models and extensions in ecology with R. Springer. https://doi.org/10.1007/978-0-387-87458-6_1</bibtext> </blist> </ref> <aug> <p>By Steven L. Kramer; Michael A. Posner; Alexander S. Browman; Nancy R. Lawrence; Jennifer Roem and Kathleen Krier</p> <p>Reported by Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib12" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib28" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib15" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib31" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib17" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib38" firstref="ref8"></nolink> <nolink nlid="nl7" bibid="bib20" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib39" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib40" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib21" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib36" firstref="ref16"></nolink> <nolink nlid="nl12" bibid="bib18" firstref="ref19"></nolink> <nolink nlid="nl13" bibid="bib37" firstref="ref20"></nolink> <nolink nlid="nl14" bibid="bib13" firstref="ref22"></nolink> <nolink nlid="nl15" bibid="bib25" firstref="ref23"></nolink> <nolink nlid="nl16" bibid="bib14" firstref="ref24"></nolink> <nolink nlid="nl17" bibid="bib30" firstref="ref25"></nolink> <nolink nlid="nl18" bibid="bib10" firstref="ref27"></nolink> <nolink nlid="nl19" bibid="bib24" firstref="ref28"></nolink> <nolink nlid="nl20" bibid="bib29" firstref="ref29"></nolink> <nolink nlid="nl21" bibid="bib27" firstref="ref32"></nolink> <nolink nlid="nl22" bibid="bib23" firstref="ref34"></nolink> <nolink nlid="nl23" bibid="bib22" firstref="ref37"></nolink> <nolink nlid="nl24" bibid="bib19" firstref="ref38"></nolink> <nolink nlid="nl25" bibid="bib32" firstref="ref39"></nolink> <nolink nlid="nl26" bibid="bib16" firstref="ref41"></nolink> <nolink nlid="nl27" bibid="bib42" firstref="ref42"></nolink> <nolink nlid="nl28" bibid="bib26" firstref="ref44"></nolink> <nolink nlid="nl29" bibid="bib11" firstref="ref56"></nolink> <nolink nlid="nl30" bibid="bib35" firstref="ref57"></nolink> <nolink nlid="nl31" bibid="bib41" firstref="ref58"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1492695
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Impacts of a Standards-Based Grading System Emphasizing Formative Assessment, Feedback, and Re-Assessment: A Mixed Methods, Cluster Randomized Control Trial in Ninth Grade Mathematics Classrooms
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Steven+L%2E+Kramer%22">Steven L. Kramer</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2900-6484">0000-0003-2900-6484</externalLink>)<br /><searchLink fieldCode="AR" term="%22Michael+A%2E+Posner%22">Michael A. Posner</searchLink><br /><searchLink fieldCode="AR" term="%22Alexander+S%2E+Browman%22">Alexander S. Browman</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2957-3262">0000-0002-2957-3262</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nancy+R%2E+Lawrence%22">Nancy R. Lawrence</searchLink><br /><searchLink fieldCode="AR" term="%22Jennifer+Roem%22">Jennifer Roem</searchLink><br /><searchLink fieldCode="AR" term="%22Kathleen+Krier%22">Kathleen Krier</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+on+Educational+Effectiveness%22"><i>Journal of Research on Educational Effectiveness</i></searchLink>. 2025 18(1):56-87.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 32
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 0918474
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Grade+9%22">Grade 9</searchLink><br /><searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Grading%22">Grading</searchLink><br /><searchLink fieldCode="DE" term="%22Formative+Evaluation%22">Formative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+9%22">Grade 9</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Mathematics%22">Secondary School Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Instruction%22">Mathematics Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Algebra%22">Algebra</searchLink><br /><searchLink fieldCode="DE" term="%22Geometry%22">Geometry</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Motivation%22">Student Motivation</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Opportunities%22">Educational Opportunities</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2023.2287594
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1934-5747<br />1934-5739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We investigated the impact in ninth-grade mathematics classrooms of Proficiency-based Assessment and Reassessment of Learning Outcomes (PARLO), a standards-based grading system. Key components of PARLO are basing student final grades on the number of learning outcomes on which the student is high-performance or proficient, providing students with formative feedback, and encouraging students to reassess for full credit after further study. Our mixed-methods study employed a cluster randomized control trial with 2,736 participating ninth graders at 14 Treatment and 15 Control schools. Data included student achievement tests, interviews with 35 teachers, and student surveys. The program improved student performance on end-of-course algebra and geometry tests by a statistically significant 0.33 SD but did not impact students' value of or expectancies for success in mathematics. However, treatment effects on mathematics performance were moderated by these psychological antecedents of motivation, such that students with higher math expectancies and value benefited more from the treatment. Furthermore, teacher interviews suggested that PARLO may have also had positive effects on growth mindsets, mastery goals, autonomy, and relatedness.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://doi.org/10.35542/osf.io/pzc3f
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1492695
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1492695
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2023.2287594
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 56
    Subjects:
      – SubjectFull: Grading
        Type: general
      – SubjectFull: Formative Evaluation
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Grade 9
        Type: general
      – SubjectFull: Secondary School Mathematics
        Type: general
      – SubjectFull: Mathematics Instruction
        Type: general
      – SubjectFull: Algebra
        Type: general
      – SubjectFull: Geometry
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Student Motivation
        Type: general
      – SubjectFull: Educational Opportunities
        Type: general
      – SubjectFull: High School Students
        Type: general
    Titles:
      – TitleFull: The Impacts of a Standards-Based Grading System Emphasizing Formative Assessment, Feedback, and Re-Assessment: A Mixed Methods, Cluster Randomized Control Trial in Ninth Grade Mathematics Classrooms
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Steven L. Kramer
      – PersonEntity:
          Name:
            NameFull: Michael A. Posner
      – PersonEntity:
          Name:
            NameFull: Alexander S. Browman
      – PersonEntity:
          Name:
            NameFull: Nancy R. Lawrence
      – PersonEntity:
          Name:
            NameFull: Jennifer Roem
      – PersonEntity:
          Name:
            NameFull: Kathleen Krier
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1934-5747
            – Type: issn-electronic
              Value: 1934-5739
          Numbering:
            – Type: volume
              Value: 18
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1