View in EDS HTML Full Text PDF Full Text

Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices

Saved in:
Bibliographic Details
Title: Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices
Language: English
Authors: Snow, Eric, Rutstein, Daisy, Basu, Satabdi, Bienkowski, Marie, Everson, Howard T.
Source: International Journal of Testing. 2019 19(2):103-127.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 25
Publication Date: 2019
Document Type: Journal Articles
Reports - Descriptive
Education Level: Elementary Education
Secondary Education
Descriptors: Evidence Based Practice, Test Construction, Computation, Cognitive Tests, Elementary School Students, Secondary School Students, Thinking Skills, Foreign Countries, Computer Science Education
Geographic Terms: Hong Kong, United States
DOI: 10.1080/15305058.2018.1543311
ISSN: 1530-5058
Abstract: Computational thinking is a core skill in computer science that has become a focus of instruction in primary and secondary education worldwide. Since 2010, researchers have leveraged Evidence-Centered Design (ECD) methods to develop measures of students' Computational Thinking (CT) practices. This article describes how ECD was used to develop CT assessments for primary students in Hong Kong and secondary students in the United States. We demonstrate how leveraging ECD yields a principled design for developing assessments of hard-to-assess constructs and, as part of the process, creates reusable artifacts--design patterns and task templates--that inform the design of other, related assessments. Leveraging ECD, as described in this article, represents a principled approach to measuring students' computational thinking practices, and situates the approach in emerging computational thinking curricula and programs to emphasize the links between curricula and assessment design.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1217617
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwF4l3DMdTIPd5QbUx_mVRINAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLYyilXaeT_JXszikgIBEICBm-jniGJ3CRRZFqvMgvowQ5z9k4VLUddmaPDLZuMUnANMJXqgabx0F-U96cb6s6rP_MhEvyrjdAHxrooXlP7-AU8qY9mf0rp3oggWSMw5sLl2WB5LX_7l4hWFdYo4pX8hJ3NPNyJZ1bVoloY9SBePnHT0ucYRMScU5-0WRwNyHdW10gVKQWVSnFDXI5elACnEssYKTcqo7c0Et7Zx
Text:
  Availability: 1
  Value: <anid>AN0136766079;k0301apr.19;2019Jun04.02:21;v2.2.500</anid> <title id="AN0136766079-1">Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices </title> <p>Computational thinking is a core skill in computer science that has become a focus of instruction in primary and secondary education worldwide. Since 2010, researchers have leveraged Evidence-Centered Design (ECD) methods to develop measures of students' Computational Thinking (CT) practices. This article describes how ECD was used to develop CT assessments for primary students in Hong Kong and secondary students in the United States. We demonstrate how leveraging ECD yields a principled design for developing assessments of hard-to-assess constructs and, as part of the process, creates reusable artifacts—design patterns and task templates—that inform the design of other, related assessments. Leveraging ECD, as described in this article, represents a principled approach to measuring students' computational thinking practices, and situates the approach in emerging computational thinking curricula and programs to emphasize the links between curricula and assessment design.</p> <p>Keywords: Computational thinking practices; evidence-centered design; assessment</p> <hd id="AN0136766079-2">INTRODUCTION</hd> <p>Computational thinking (CT) is increasingly recognized as an essential form of literacy for all informed citizens in the modern world, not just computer scientists. As a result, many nations are introducing coding and computational thinking instruction as part of compulsory education, seeking to build skills such as problem solving and logical thinking in addition to other digital competencies (Bocconi, Chioccariello, Dettori, Ferrari, & Engelhardt, [<reflink idref="bib7" id="ref1">7</reflink>]; Yadav, Good, Voogt, & Fisser, [<reflink idref="bib40" id="ref2">40</reflink>]).</p> <p>In the United States, the <emph>Computer Science for All</emph> initiative,[<reflink idref="bib1" id="ref3">1</reflink>] the Computer Science Teachers Association (CSTA) Standards (CSTA Standards Revision Task Force, [<reflink idref="bib10" id="ref4">10</reflink>]),[<reflink idref="bib2" id="ref5">2</reflink>] and the new <emph>K-12 CS Framework</emph> (K-12 CS Framework, 2016)[<reflink idref="bib3" id="ref6">3</reflink>] are all aimed at supporting states as they adopt and adapt existing curricula, such as CS Discoveries,[<reflink idref="bib4" id="ref7">4</reflink>]<emph>Exploring Computer Science,</emph>[<reflink idref="bib5" id="ref8">5</reflink>] and <emph>AP CS Principles</emph>,[<reflink idref="bib6" id="ref9">6</reflink>] and build more robust programs for improving computational thinking outcomes in primary and secondary schools. Other countries are moving in similar directions. The United Kingdom, for example, has adopted standards that mandate computer science instruction for all students aged 5–18 years (Department for Education, [<reflink idref="bib13" id="ref10">13</reflink>]), Israel recently extended the reach of CS education to include the primary grades (grades fourth through sixth) (Bocconi et al., [<reflink idref="bib7" id="ref11">7</reflink>]; Gal-Ezer & Stephenson, [<reflink idref="bib15" id="ref12">15</reflink>]), and Hong Kong is piloting a new initiative, <emph>CoolThink@JC</emph> (EdUHK, MIT, and CityU, [<reflink idref="bib14" id="ref13">14</reflink>]), that provides instruction in programing and computational thinking to students in primary grades 4–6.</p> <p>As CT initiatives such as these continue to expand, there are increasing pressures to create assessments that provide high-quality evidence of student learning. Teachers need evidence to help them guide and refine their instruction; school leaders need evidence to help them make decisions about the efficacy of CT-focused initiatives; and state and national leaders rely on evidence to help determine when to deliver, and where to focus, future resources.</p> <p>Efforts to design and develop high-quality assessments of CT, however, have not necessarily kept pace with the evidentiary needs of corresponding initiatives. Moreover, many of the assessments that do exist have largely focused on measuring fundamental CS conceptual knowledge and programing skills rather than the <emph>application</emph> of these skills in authentic computational problem-solving contexts (i.e., the practices of computational thinking). The <emph>K-12 CS Framework</emph> defines practices as "the behaviors and ways of thinking that computationally literate students use to fully engage in today's data-rich and interconnected world." Creating assessments that measure the CT <emph>practices</emph> that students engage in when they apply their CT conceptual knowledge constitutes a high but important bar to reach for, and also reflects a broader shift in the STEM fields away from measurements of factual knowledge and toward using and applying knowledge (procedural knowledge) to understand the world and solve problems.</p> <p>Evidence-Centered Design (ECD) is a methodology for creating assessments for hard-to-assess constructs that is especially helpful when the knowledge and skills to be measured involve complex, multistep performances, such as those required by CT practices (Mislevy & Haertel, [<reflink idref="bib22" id="ref14">22</reflink>]). For example, there are many different aspects to the CT practice of testing and refining computational artifacts, and it may be difficult to determine which of these aspects should be measured and how these aspects can be included in a classroom-based assessment. The use of ECD helps to situate within the assessment design the practice of identifying and fixing errors in a computational artifact. Since 2010, researchers at SRI International have leveraged ECD methods to develop measures of CT practices in a number of different educational systems and settings.</p> <p>This article describes how ECD principles were applied to design and develop assessments that measure CT practices in two different instructional contexts: (<reflink idref="bib1" id="ref15">1</reflink>) end-of-unit assessments for an introductory CS curriculum for early secondary students in the United States, and (<reflink idref="bib2" id="ref16">2</reflink>) pre-post assessments of CT Practices for a pilot CT program for late primary students in Hong Kong. We begin by briefly describing the ECD approach generally, and then demonstrating how we applied these principled design methods to analyze and model computational thinking practices and create corresponding assessment designs. In describing our work, we highlight lessons learned about how ECD methods and artifacts (i.e., domain analyses, domain models, design patterns, and task templates) can be used to support the development of assessments that measure important CT learning outcomes. We conclude with a discussion of the value added by using ECD principles when developing assessments to support CT instruction.</p> <hd id="AN0136766079-3">BACKGROUND</hd> <p></p> <hd id="AN0136766079-4">Evidence-Centered Design</hd> <p>Evidence-Centered Design (ECD) is a systematic process that supports assessment design and improvement by explicitly linking particular features of assessment tasks, the evidence of student performances generated by those tasks, and the knowledge and skills implicated by that evidence (Mislevy & Riconscente, [<reflink idref="bib23" id="ref17">23</reflink>]; Mislevy, Steinberg, & Almond, [<reflink idref="bib25" id="ref18">25</reflink>]). ECD is especially helpful when the knowledge and skills to be measured involve complex, multistep performances, such as those required in CT practices (Mislevy & Haertel, [<reflink idref="bib22" id="ref19">22</reflink>]).</p> <p>ECD helps designers translate the broad learning goals often found in standards and curricula into statements of student ability and provides a framework that allows designers to describe and create the kinds of tasks that will elicit evidence of that ability. ECD guides designers on how the evidence can be aggregated to model what students know and can do. Our prior design experience in difficult-to-assess domains has demonstrated that the up-front design work required by ECD lays the groundwork for efficiently generating families of assessment items. For examples of domains where ECD has been applied, see DeBarger and Snow ([<reflink idref="bib12" id="ref20">12</reflink>]) for life sciences; Mislevy, Riconscente, and Rutstein ([<reflink idref="bib24" id="ref21">24</reflink>]) for model-based reasoning; and Cheng, Ructtinger, Fujii, and Mislevy ([<reflink idref="bib9" id="ref22">9</reflink>]) for systems thinking.</p> <p>ECD supports assessment design by prescribing five layers of activity (Mislevy & Haertel, [<reflink idref="bib22" id="ref23">22</reflink>]), shown in Table 1. Each of the layers indicates a grain size in which the student model (what students know and can do), the evidence model (what evidence is produced by the student and how that evidence is aggregated), and the task model (what tasks are like that elicit the evidence) are defined. The top layer (domain analysis) provides an overview of the concept or concepts to be measured (a high-level grain size), and the bottom layer (assessment delivery) defines the specifics of how the student interacts with the assessment (at a finer grain size). Although these layers suggest steps in a sequential process, design occurs in cycles of iteration and refinement, both within and across ECD layers. Work in different layers can occur simultaneously and involves experts in content, pedagogy, assessment, and psychometrics. In this article, we focus on our work in the domain analysis, domain modeling, and conceptual assessment framework layers, to demonstrate how ECD can be applied to a newly emerging domain. Other publications describe our work in assessment implementation and delivery (Bienkowski, Snow, Rutstein, & Grover, [<reflink idref="bib6" id="ref24">6</reflink>]; Snow, Tate, Rutstein, & Bienkowski, [<reflink idref="bib31" id="ref25">31</reflink>]).</p> <hd id="AN0136766079-5">Domain Analysis: Analyzing Computational Thinking Practices</hd> <p>Our work in assessment of computational thinking (CT) began at a time when the construct was just being defined (Wing, [<reflink idref="bib38" id="ref26">38</reflink>]; National Research Council, [<reflink idref="bib26" id="ref27">26</reflink>]). ECD was helpful in focusing our early work on analyzing the target domain of CT and distinguishing it from the broader discipline of computer science (CSTA Standards Task Force, [<reflink idref="bib11" id="ref28">11</reflink>]). ECD's domain analysis phase guided us to identify the broad knowledge and skills involved in the domain of computational thinking and the boundaries of the domain and subdomains. The output of domain analysis in ECD is an overall conceptual model of the domain, often initially expressed as narrative descriptions of major constructs such as "Abstraction" and "Modeling."</p> <hd id="AN0136766079-6">Identifying the Computational Thinking Domain</hd> <p>Our analysis of the CT domain drew on many sources: emerging construct definitions; curriculum frameworks and standards; curricula; and research findings in computer science education, scientific inquiry, engineering design, communication, and collaboration. We reviewed prior work in computer science assessment, such as Herman, Loui, and Zilles ([<reflink idref="bib17" id="ref29">17</reflink>]); McCracken and colleagues ([<reflink idref="bib21" id="ref30">21</reflink>]); Tew and Guzdial ([<reflink idref="bib33" id="ref31">33</reflink>]) for postsecondary, Werner, Denner, Bliesner, and Rex ([<reflink idref="bib35" id="ref32">35</reflink>]); Nicholson, Good, and Howland ([<reflink idref="bib27" id="ref33">27</reflink>]); Robertson and Howells ([<reflink idref="bib28" id="ref34">28</reflink>]); and Brennan and Resnick ([<reflink idref="bib8" id="ref35">8</reflink>]) for middle school. We paid special attention to the CSTA standards, the Next Generation Science Standards, the Exploring Computer Science (ECS) curriculum (including learning objectives), and the AP Computer Science Principles framework (including learning objectives and evidence statements).</p> <p>Computational thinking (CT) is largely recognized as a subset of the larger computational science domain. Our domain analysis indicates that CT refers to the cognitive processes involved in expressing solutions as computational steps or algorithms that can be carried out by a computer (Aho, [<reflink idref="bib1" id="ref36">1</reflink>]; Wing, [<reflink idref="bib38" id="ref37">38</reflink>]). CT is generally considered to be a problem-solving process that includes: formulating problems in a way that enables use of a computer and other tools to help solve them; logically organizing and analyzing data; representing data through abstractions such as models and simulations; automating solutions through algorithmic thinking; identifying, analyzing, and implementing possible solutions with the goal of achieving the most efficient and effective combination of steps and resources; and generalizing and transferring problem solving processes to a wide variety of problems (Barr, Harrison, & Conery, [<reflink idref="bib3" id="ref38">3</reflink>]). As can be seen in these examples, CT is a core skillset that focuses on foundational problem-solving skills in a computational context and further relies on basic understanding of concepts such as computers, computation, data, information, devices, and the like.</p> <p>Different frameworks for characterizing various aspects of CT have been defined, reflecting differing emphases in pedagogy and learning outcomes. For example, Brennan and Resnick ([<reflink idref="bib8" id="ref39">8</reflink>]), working in the limited context of <emph>Scratch</emph> programing, conceptualized a three-dimensional CT framework consisting of computational concepts, practices and perspectives. They describe computational concepts as the concepts designers engage with as they program (such as iteration and parallelism), computational practices as what designers develop as they engage with the concepts (such as debugging projects or remixing others' work), and computational perspectives as the opinions designers form about the world around them and about themselves.</p> <hd id="AN0136766079-7">Specifying Computational Thinking Practices</hd> <p>In our work, we conceptualized CT as the problem-solving skills or behaviors that computationally literate students demonstrate to fully engage in today's data-rich and interconnected world. We focused on these skills and behaviors (referred as "CT practices" henceforth) since they involve the understanding and application of computer science concepts to solve meaningful problems. Our conceptualization of CT aligns with the suggestion of focusing more on computational 'doings' and less on computational 'thinking' (Hemmendinger, [<reflink idref="bib16" id="ref40">16</reflink>]), meaning that engaging students in the process of developing abstractions and engaging in other computational representational practices is required in order to support the development of their CT skills (Basu et al., [<reflink idref="bib5" id="ref41">5</reflink>]).</p> <p>Our CT practices (and indeed, the decision to focus on practices over conceptual knowledge) were influenced by the ECS curriculum (which in turn drew from the AP Computer Science Principles course [Arpaci-Dusseau et al., [<reflink idref="bib2" id="ref42">2</reflink>]]). We focused on four major CT practices: designing and applying abstraction and models, designing and developing computational artifacts, analyzing the effects of developments in computing, and analyzing one's own computational work and the work of others (Bienkowski et al., [<reflink idref="bib6" id="ref43">6</reflink>]). More recent definitions of CT include the framing of CT in the context of the Next Generation Science Standards and science learning (Sneider, Stephenson, Schafer, & Flick, [<reflink idref="bib30" id="ref44">30</reflink>]), standards from the Computer Science Teachers Association (CSTA Standards Revision Task Force, [<reflink idref="bib10" id="ref45">10</reflink>]), and definitions in the K-12 CS Framework (K-12 CS Framework, [<reflink idref="bib18" id="ref46">18</reflink>]).</p> <hd id="AN0136766079-8">Prior Assessments of Computational Thinking Practices</hd> <p>As part of the domain analysis we reviewed the current state of assessments measuring concepts and practices in this area and related them to the skills and behaviors that students should be able to demonstrate, and actions that students should be able to do and explain their thinking about. When we reviewed existing assessments (primarily in computer programing, because the CT domain was still being defined) during our domain analysis, we saw many efforts in K–12 that analyzed students' final programing artifact(s) for a given assignment (Werner, Denner, Campe, and Kawamoto, [<reflink idref="bib36" id="ref47">36</reflink>]), across a course (Seiter & Foreman, [<reflink idref="bib29" id="ref48">29</reflink>]), or in open-ended game design tasks (Koh, Basawapatna, Bennett, & Repenning, [<reflink idref="bib19" id="ref49">19</reflink>]). Our domain analysis showed that CT practices are called forth as students approach a problem and program their computational artifacts, but the review of assessments available at the time indicated that what is often examined is the overall quality of the final artifacts themselves. In some prior work, students' use of particular programing structures is used to infer their knowledge of CT practices such as abstraction, modeling, or algorithmic thinking (Werner et al., [<reflink idref="bib36" id="ref50">36</reflink>]). In open-ended interactive media and game assignments, researchers have also looked for the presence of patterns of programing constructs (referred to as CT patterns) that reflect certain CT practices (Koh et al., [<reflink idref="bib19" id="ref51">19</reflink>]). However, the presence of a programing structure in a student's final program may not be sufficient evidence for their knowledge of a CT practice or set of practices (Kurland & Pea, [<reflink idref="bib20" id="ref52">20</reflink>]). The challenge is in the need for evidence of repeated (and correct) use, or some insight into the processes or practices subsumed when analyzing a programing artifact.</p> <p>Some researchers have designed performance tasks to assess students' programing abilities. For example, Webb ([<reflink idref="bib34" id="ref53">34</reflink>]) measured skills in an authentic performance task, asking middle-school students who had had 10 hours of instruction in game-based programing to find and fix a buggy (i.e., malfunctioning) program. Building off this work, Werner and colleagues ([<reflink idref="bib36" id="ref54">36</reflink>]) created an assessment for middle school students that also had find-fault type of tasks as well as those involving programs that required modification to meet a new specification. Brennan and Resnick ([<reflink idref="bib8" id="ref55">8</reflink>]) studied computational practices using artifact-based interviews and design scenarios. The design scenarios were similar to the fault-finding and fixing described previously, as students were given a program, and asked to explain it, extend it, fix it, and add a new feature.</p> <hd id="AN0136766079-9">Toward a Domain Model for Computational Thinking Practices</hd> <p>Our domain analysis showed that there are no comprehensive assessment frameworks for CT practices in K–12 computer science (Wilson, Sudol, Stephenson, & Stehlik, [<reflink idref="bib37" id="ref56">37</reflink>]; Yadav et al., [<reflink idref="bib39" id="ref57">39</reflink>]). While the computer science education field has found utility in standards and curriculum learning objectives, which we examined during our analysis of the CT domain, they do not provide complete specifications for developing assessments of CT practices. Further work, using other ECD layers, is needed to provide the level of detail for content and measurement that are required for the design of assessment items, tasks and scales. Without assessment frameworks, it is difficult to know how to measure what really matters and assessments can end up being too closely tied to particular courses.</p> <p>To measure CT practices effectively, we needed to identify what to measure and build empirical support and validity evidence for task models that can support inferences about student learning. In the following section, we describe how ECD was applied to model information from our analysis of the CT domain as a precursor to designing and delivering assessments.</p> <hd id="AN0136766079-10">Domain Modeling: Modeling Computational Thinking Practices</hd> <p>Domain modeling in ECD uses the information that was gathered in the domain analysis and organizes it around aspects of the assessment using templates such as a design pattern. In modeling CT practices, our design patterns, or design specifications, encompassed the core CT practices we identified (as mentioned): designing and applying abstraction and models, designing and developing computational artifacts, analyzing the effects of developments in computing, and analyzing one's own computational work and the work of others (Bienkowski et al., [<reflink idref="bib6" id="ref58">6</reflink>]). With these broad constructs identified from the domain analysis, the goal of the domain modeling work was to develop design patterns for CT practices that could be used across a wide range of assessment opportunities and that were independent of any particular CS curriculum.</p> <p>Figure 1 shows an excerpt of a design pattern for the CT practice "Design and Apply Abstractions and Models." The Overview gives a definition of the construct in terms of what students know and be able to do, and identifies the boundary of the construct relative to others.</p> <p>Graph: FIGURE 1 Design and Apply Abstractions and Models, CT practices design pattern excerpt.</p> <p>After defining an Overview for each construct, we specified design patterns (one per construct) by articulating the focal knowledge, skills and other attributes (<emph>f</emph>KSAs) underlying the construct, and additional knowledge, skills and other attributes (KSAs) involved, and then specifying variable and characteristic features of the assessment task. For a given assessment, ECD guides us to use <emph>f</emph>KSAs to represent a link between the broader core ideas and practices and the content of a specific curriculum that targets a particular grade level or age range. For our specific use of ECD in this domain of CT practices, abilities (skills) were given preference to knowledge and other attributes because the target construct was about practice, or the application of a concept using a skill.</p> <p>During domain modeling, we also write the design specifications that impose constraints and guidelines on how the <emph>f</emph>KSAs can be measured. ECD guides us to identify these constraints and guidelines in the form of characteristic features of tasks—features that any task that measures the <emph>f</emph>KSAs must have—and variable features—ways that the tasks can vary but still measure the <emph>f</emph>KSAs. For example, while we might require certain tasks to provide code to the students (a characteristic feature), the programing language used to present the code (a variable feature) may differ depending on the context of the assessment. Variable features allow assessment designers to tailor tasks to be, for example, independent of programing constructs in any language, or designed for a specific curriculum and/or particular programing language.</p> <p>The characteristic and variable features in ECD design patterns compel designers to be explicit about what a task must contain and also prepares them for ways to make new, related tasks by varying features. During our ECD domain modeling and creating CT practice design patterns, we also specified examples of potential work products, which are characterizations of the types of responses students might produce (e.g., explanations or matching responses), and potential observables, which defines the quality(ies) of their response that will be used to score the work product (e.g., the amount of detail provided, the accuracy of the response, or the degree to which their explanation supports their claim). The potential observables can then be used as the basis for rubric development. They provide a key connection between what students produce based on the task to making inferences about the <emph>f</emph>KSAs.</p> <p>In summary, ECD design patterns can be leveraged to help define and model key constructs in a domain of interest; specify focal knowledge, skills, and other attributes related to the construct; set forth characteristic and variable features of related assessment tasks; and give future assessment designers potential observations and work products to guide task development.</p> <hd id="AN0136766079-11">A Conceptual Assessment Framework for Computational Thinking Practices</hd> <p>Once design patterns are specified during the domain modeling phase (and reviewed by curriculum designers and experts in the field), we pursue work with the Conceptual Assessment Framework (CAF) layer of ECD where task templates are created. Task templates are documents that lay out general specifications using design patterns as a guide. These specifications provide details about what it is we want to measure (the student model), what evidence will be collected from the task and how that will be scored (the evidence model), and the framing of the tasks that will allow for the desired evidence to be collected (the task model). They are general in the sense that they can be used to generate multiple tasks that measure the same concepts. While they provide more information about the structure of the tasks than a design pattern, they still contain information about what can vary between multiple tasks. While developing a task template, we select the <emph>f</emph>KSAs that will be measured and provide a brief description of the type of questions that we would want to include. We also provided information about the scoring of the student responses. We use "task" to mean a scenario with a related set of questions; we chose scenario-based tasks because they allow us to present more authentic situations to students. We require these scenarios to be something that most students can relate to, and also to provide opportunities to measure the skills of interest.</p> <p>In addition to the design pattern, the work at the CAF layer starts bringing in other constraints. It is here that important factors such as testing format and test length are taken into consideration. The CAF may also contain specifications related to additional frameworks. For example, we often consider Bloom's taxonomy (Sosniak, [<reflink idref="bib32" id="ref59">32</reflink>]) when developing the test items or questions, striving to make our assessments go beyond the <emph>remembering</emph> and <emph>understanding</emph> levels of Bloom's taxonomy, and to focus on the <emph>applying</emph>, <emph>analyzing</emph>, and <emph>evaluating</emph> levels. These more advanced levels of the taxonomy align well with our focus on CT practices. We also consider any additional knowledge, skills or abilities that may be required of students when completing the task, and any variable features that are part of the design pattern in order to make decisions related to the difficulty of the task.</p> <p>Once the domain is analyzed and modeled, and task blueprints are codified in the CAF, we can implement the assessment. In the following sections, we describe two distinct cases, a high school course in the United States, and a late elementary curriculum implemented in Hong Kong, where we leveraged the CT practice design patterns to develop tasks assessing CT practices. We also describe how the <emph>f</emph>KSAs developed in the context of one curriculum were reused in another curriculum.</p> <hd id="AN0136766079-12">Case 1: Principled Assessment of Computational Thinking (PACT)</hd> <p>Beginning in 2010, our team worked on designing and developing assessments for a full-semester introductory high school computer science (CS) curriculum that focused on equity and inquiry. <emph>Exploring Computer Science</emph> (ECS) is an introductory computer science (CS) course designed for early high school students who have little to no experience with computer science. The course consists of six instructional units that focus on human computer interaction, problem solving, web design, programing, robotics, and data analysis, respectively.</p> <p>The course is designed to introduce students to different aspects of CS with a focus students engaging in computational thinking (CT) practices. We developed and validated (Snow et al., [<reflink idref="bib31" id="ref60">31</reflink>]) four interim assessments, each aligned to one of the first four ECS instructional units, and one cumulative assessment that covered the first four instructional units. The assessments were designed to measure the CT practices highlighted in each of the instructional units.</p> <p>Our first step in assessment task development and implementation for the PACT initiative was to select a set of <emph>f</emph>KSAs to be covered by each of the assessments (unit and cumulative). We reviewed the <emph>f</emph>KSAs from the design patterns alongside the learning goals of the ECS instructional units. In some cases, <emph>f</emph>KSAs from the CT design patterns could be directly applied to a given unit. For example, from the design pattern for the CT practice "Design and Implement Creative Solutions and Artifacts," one of the <emph>f</emph>KSAs "Ability to state what a program currently in design will output as a result of anticipated inputs" was kept. This <emph>f</emph>KSA aligns with the ECS learning goal "Explain how a particular program functions." In the final version of the <emph>f</emph>KSA list, the <emph>f</emph>KSA was shortened to be "Ability to state what a program will output given inputs." This was done to clarify that the inputs should be given and the students would not need to come up with them in order to meet this <emph>f</emph>KSA. This particular refinement of a <emph>f</emph>KSA reminds us that ECD is not a linear but an iterative process.</p> <p>In other cases, the <emph>f</emph>KSAs were modified to align them more directly to the curriculum learning goals. For example, the <emph>f</emph>KSA "Ability to use programming constructs to create executable solutions" was modified to align with the learning goal "Select appropriate programming structures" creating a new <emph>f</emph>KSA "Ability to evaluate the relationship between features of a programming structure and features of a problem or algorithm."</p> <p>In order to capture these changes, a new set of design patterns, one for each of the first four ECS units, were developed. The design patterns were created by selecting and modifying <emph>f</emph>KSAs from the CT design patterns in order to create a set of fKSAs for how these CT practices were represented in the ECS curriculum. Further fields of the design pattern were developed similarly to the process described above, again with aspects being drawn from the CT practices design patterns.</p> <p>We developed Item ideas as an initial step in specifying the task templates. We began by specifying the item ideas at a general level, first choosing the set of <emph>f</emph>KSAs the task would measure and then specifying the task in an outline format. For example, we designed one task to measure three <emph>f</emph>KSAs. In this task students are presented with two algorithms written in a narrative form. The student is then provided input to the two algorithms and asked to specify what the output would be (aligned to the <emph>f</emph>KSA described "Ability to state what a program will output given inputs"). As a student progresses in the task they are prompted to match a programing structure with a part of the program (aligned to the <emph>f</emph>KSA "Ability to evaluate the relationship between features of a programming structure and features of a problem or algorithm"). Finally, the student is presented with a comparative question about the two programs (aligned to the <emph>f</emph>KSA "Ability to compare the tradeoffs between different algorithms for solving the same problem"). While the format of the task is specified, decisions still need to be made before the task is finalized. In particular, the algorithms and the criteria by which these algorithms differ need to be determined. The final version of this task is shown in Figure 2.</p> <p>Graph: FIGURE 2 Gabriela's and Lucia's Algorithms, example task from the ECS Unit 4 assessment.</p> <p>One of the challenges with this question was the representation of the algorithms. For this question, we did not initially want to use code in a specific programing language to represent the algorithms because we wanted students to be able to complete the question even if they struggled with their knowledge of a particular programing language. We had originally considered using pseudocode, but we found that this was also confusing for students, and again the lack of familiarity with the format would make it difficult for some students to understand the steps in the algorithm. For example, we could have used mathematical formulas to represent the <emph>if</emph> statements but decided that this change would cause the item to rely on students' knowledge of the <emph>greater than</emph> and <emph>less than</emph> signs and this was not something we wanted to measure (i.e., construct irrelevant knowledge).</p> <p>We made a similar decision in parts c and d of the task shown in Figure 2, where we ask students what programing structure could be used to program different steps of the algorithm. For this part of the problem, we did not want the item to focus on whether the student could connect the feature to the specific Scratch Block. In the item, then, students were provided with the Scratch block, which then meant the item was focused on their understanding of the function of the block.</p> <p>Both of these decisions were related to representations used in the problem. For the last item in this task (part c in Figure 2), we faced a different type of challenge. For this item, we wanted students to compare the two algorithms. In order to do that, we designed the item to present the students with a scenario-based problem and ask them which algorithm would be better suited to the problem. After piloting the task with students, we learned that there were two ways that students tended to respond the item. One was to explain why one of the algorithms would not work, while another was explaining why one of the algorithms did work. We found that the students who responded to the question in relation to the algorithm that did not work provided richer answers that showed more insights into how well the student could follow the algorithm. Based on this pilot data, in the revisions to the task, we modified the question from having students compare the two algorithms to having them evaluate one of the algorithms. While this changed the <emph>f</emph>KSA being measured by the item, we felt that the revised question provided more information about students' overall ability related to designing and developing computational artifacts.</p> <p>In our experience, ECD provides us with the scaffolding to meet challenges such as those encountered in the context of PACT. It helps us identify the different decisions that we had to make and provides guidance and design documentation. Documentation that we had generated during the domain modeling phase gave us information on the boundaries within which the final decisions must lie and also provided a place to keep track of the decisions that were made, making it easier to address these same decisions in future assessment development.</p> <hd id="AN0136766079-13">Case 2: CoolThink@JC Assessments for CT Practices</hd> <p>In the <emph>CoolThink@JC</emph>, a project involving the development and evaluation of an intervention designed to teach students about computer science in the elementary grades in Hong Kong, the Brennan and Resnick ([<reflink idref="bib8" id="ref61">8</reflink>]) definition of CT practices informed the development of the pilot lessons. This definition of CT practices, influenced by work in a particular pedagogical setting, delineates the practices differently than the PACT design patterns.</p> <p>Brennan and Resnick ([<reflink idref="bib8" id="ref62">8</reflink>]) identified the major constructs they wanted to emphasize based on their observations of and interviews with young people working with the <emph>Scratch</emph> programing language and the supporting community and repository of Scratch programs online. Thus, they identified a smaller scope and more detailed level for CT practices than the broad domain analysis for our PACT project did: being incremental and iterative, testing and debugging, reusing and remixing, and abstracting and modularizing. They identified instances of students engaging in these behaviors in the Scratch environment and, correspondingly, their major CT practice constructs are closer to coding than the problem-solving orientation of the PACT design patterns.</p> <p>Brennan and Resnick ([<reflink idref="bib8" id="ref63">8</reflink>]) did not identify <emph>f</emph>KSAs and other parts of a design pattern as we did, but our work with the <emph>CoolThink@JC</emph> team helped us understand the details of these main constructs as they related to the pilot lessons. In the domain modeling phase for this project, we again drew from the PACT domain modeling work, as well as the definition of the constructs used in the lessons. Some CT practices, such as designing and applying abstraction, are defined similarly in both the PACT and <emph>CoolThink@JC</emph> projects. However, two major constructs from the PACT design patterns—designing computational artifacts and analyzing one's own computational work and the work of others—were reorganized into new constructs, <emph>Algorithmic Thinking</emph>, <emph>Being Incremental and Iterative</emph>, <emph>Testing and Debugging</emph>, <emph>and Reusing and Remixing</emph> for the <emph>CoolThink@JC</emph> project. <emph>Analyzing the Effects of Developments in Computing</emph> was no longer listed as a CT Practice because it was not emphasized by the <emph>CoolThink@JC</emph> lessons.</p> <p>Our goal in the <emph>CoolThink@JC</emph> project was to develop assessments targeting various aspects of CT, with one assessment specific to the sub-domain of CT practices. We developed the <emph>CoolThink@JC</emph> assessment for CT Practices guided by the ECD process, as we did for the PACT assessments described above. We first identified which <emph>f</emph>KSAs to measure, considering as before what was important for the curriculum, with an added constraint of what could be measured using multiple-choice items. Because the design patterns from PACT didn't map directly to these programing-focused practices, we selected those that did align and also reviewed more recent literature to augment the existing set of <emph>f</emph>KASs. The result was an initial set of <emph>f</emph>KSAs for each of the four target <emph>CoolThink@JC</emph> CT practices where approximately 80% were new or modified. Through conversations with the lesson developers, we narrowed down the full set of <emph>f</emph>KSAs to those deemed most critical in the curriculum. These <emph>f</emph>KSAs were used as the focus of the assessments.</p> <p>We described previously how ECD is an iterative process. In PACT, we followed a generally linear process, with some refinement of the <emph>f</emph>KSAs as we developed and tested assessment items. In this project, we could have revisited the original design patterns and inserted these programing-focused constructs as categories or groups of <emph>f</emph>KSAs under the existing constructs. Alternatively, we could have developed entirely new (but overlapping) design patterns for these constructs. In the end, we chose neither approach because of the project's timeline for testing and delivery of the assessments and because the assessment was being used in an evaluation project (single use). Instead of completing a full design pattern, an "ECD Lite" approach was taken. In this approach we used the set of <emph>f</emph>KSAs as our design pattern and identified aspects related to those <emph>f</emph>KSAs in the CAF phase of the work.</p> <p>Similar to the PACT project, we started the work at the CAF level with the development of item ideas. For the <emph>CoolThink@JC</emph> project we had additional considerations as we had to be sensitive to differences in cultural context, take into consideration the young age of the students and design the questions so they could be scored automatically. For example, in one item the context was a fundraiser at the school. We had to ensure that this was something that elementary school children were familiar with, and we needed to collect information on the type of events, and prizes that be familiar to all students. When developing the items, we made sure that their features aligned with the <emph>f</emph>KSAs. Our item development included translation into Chinese, and delivery was conducted via an online testing platform in Hong Kong. An additional step was included to ensure that the English and Chinese translations both had the same meaning and intention. The technology platform used for assessment delivery did permit drag and drop features which meant we could develop some items where students had to bring together different lines of code in order to create short code snippets without having to worry if they got the exact syntax correct.</p> <p>Even though the characteristic features were not fully specified for the <emph>CoolThink@JC</emph> project, the item development could still make sure the previous specifications were satisfied, especially for items on the <emph>CoolThink@JC</emph> CT Practices assessment that measure the same or similar <emph>f</emph>KSAs as those developed for the PACT assessments. For example, one item created for the <emph>CoolThink@JC</emph> project (the Robot question) focused on two <emph>f</emph>KSAs, "Ability to summarize the behavior of a given algorithm" and "Ability to compare multiple approaches to solving a problem." These represent two of the three <emph>f</emph>KSAs discussed in the PACT example in Case 1 above. For this item, the characteristic features were the same as those for PACT: the item had to have multiple algorithms, and students would be expected to compare the algorithms based on given criteria.</p> <p>One main difference between the tasks in the two projects that involved comparing algorithms was the complexity of the algorithms presented. Since the students using the <emph>CoolThink@JC</emph> curriculum were younger, the algorithms were less complex and required less reading than the algorithms in the PACT project. We had to consider that these students may not have had as much exposure to algorithms and abstract reasoning. For this reason, we wanted to make sure the algorithms for the <emph>CoolThink@JC</emph> item were straightforward, so that the focus was on students being able to compare the algorithms. We also wanted to target the extent to which students could recognize that there were tradeoffs between the algorithms. For this reason, we gave the students two criteria they could compare the algorithms on, time and cost (see Figure 3). The task then asks the students to examine the algorithms to determine which of them meet a specific set of constraints. In the PACT project, we had students explain their answer, but in the <emph>CoolThink@JC</emph> project we did not have students explain their answer and instead had them focus on the identification of which algorithms met the constraints.</p> <p>Graph: FIGURE 3 Robots, example task from the CoolThink@JC CT Practices assessment.</p> <hd id="AN0136766079-14">Challenges in Developing Assessments of Computational Thinking</hd> <p>In the previous cases, we highlighted the process of using ECD to develop assessments tasks measuring CT practices. While following the ECD process can help identify and align different features of assessment tasks, it does not guarantee that every task generated is a great one. In addition to the challenges described above of generating algorithms that are grade-level appropriate, in a format that students understand and that are able to be scored automatically, we faced several other challenges.</p> <p>For example, one challenge was the identification of appropriate scenarios. While we specified that we wanted scenarios that students could relate to, it was not always clear what that scenario should be. In one of our tasks for the PACT project, we asked students to generate their own algorithms. In order for students to be able to complete this as part of the assessment, the algorithm they developed could not be too complicated. In our initial version of the item, we set the scenario as organizing props for a theater production. We found that the scenario was confusing for students. Some of that confusion might have been in how the scenario was presented, but for others it may have been due to lack of familiarity with the context. The task scenario was revised to organizing food on shelves at a food bank. We provided students with a quick introduction to a food bank and gave them constraints on the type of food that needed to be organized. We found that students were better able to understand the overall purpose of the algorithm with this revised scenario.</p> <p>This question introduced an additional challenge of how to score the computational artifact, in this case the algorithm that was produced. Here the potential observations from the design pattern helped to highlight the desired features of the algorithm. In this case, we wanted to look at the <emph>completeness</emph> of the algorithm and the <emph>clarity</emph> of the algorithm. We determined that for an algorithm to be complete it had to address two parts, one was how the food was sorted, and the other was how the sorted food was organized on the shelves. The clarity came into play in that the student had to make each of these two stages clear in order to get full credit.</p> <p>Similar to the previous example, assessing CT practices often involves evaluating how well students can create a computational artifact. In the case of an algorithm, it made sense that we could have students develop an algorithm using a narrative structure. However, when it comes to designing a web page, or developing code, it was challenging to determine how this could be done in a paper/pencil environment, as this is not the typical environment students use in the classroom for these types of tasks.</p> <p>When we were developing an assessment for the Web Design unit of the ECS curriculum we wanted to see if students could develop a webpage. We thought about this in two parts, one was the designing of the webpage and the other was generating the webpage using code. Our initial item focused on the design of the webpage. For this item, we gave students a scenario and asked them to develop a list of design and content elements that the webpage had to include and then we had them sketch a design based on these elements.</p> <p>What we found after piloting was that we did not learn much from the sketches about student's ability to design a webpage above what we learned from the elements they specified. For one, it was often hard to determine if students had included the elements or not as the drawings were sometimes hard to interpret. In addition, most of the students were able to include at least one of the design and content elements that they had specified in their drawing. We also were still missing the measurement of whether or not the students were able to develop HTML code to produce the website.</p> <p>In revising this task, we broke the original item into two new items, one where we just asked students to create the design and content elements, and another where we gave a sketch of a webpage and had students generate the HTML code to produce this web site. In our design patterns, we specified that full knowledge of HTML was an additional <emph>f</emph>KSA and not something we wanted to focus on. We addressed this in two ways, one was in providing a page that listed some of the HTML commands, and the other was that in scoring we allowed for some errors in the way the code was specified. Similar to the scoring of the developed algorithm in the last example, we again looked for completeness and clarity of the generated code. For completeness, we checked to see that the student had included each of the elements that were provided to them. For clarity, the code had to be close to the correct command.</p> <p>We faced a similar challenge when measuring how well students could generate Scratch code. Here again we made similar decisions regarding how to measure if students could generate code. We followed the same pattern that we did for the HTML item, where we presented the student with a scenario and gave them specific requirements. We then provided the students with a page that gave the students the Scratch blocks and scored on completeness, but did not require students get the syntax exactly correct.</p> <p>In the <emph>CoolThink@JC</emph> assessment, one practice that we particularly struggled with was <emph>testing and debugging</emph>. As part of this practice, students should engage in multiple rounds of testing and not just wait until the end to test. This was difficult to accomplish in a static environment, where we were only looking at the end result of what a student produced. We ended up providing the students with a scenario and asking them to drag over the steps they would take to program a solution. As part of the steps where they were able to drag, there were multiple opportunities to test the program for errors and fix errors. When scoring we examined whether students did not test and fix, tested and fixed only at the end (or only once), or tested and fixed multiple times. While this is not enough evidence to determine how students would behave in actual practice, it does provide some evidence of how well students engage in the practice in a more constrained environment.</p> <hd id="AN0136766079-15">Discussion and Conclusion</hd> <p>The cases we describe in this article represent early efforts at applying a principled approach to designing and development assessments of CT practices for students in the United States and Hong Kong. Through these cases, we hope to have illustrated some of the challenges that we have ahead of us as we work to meet the evidentiary needs of these initiatives, and how leveraging ECD gave us opportunities to address these challenges.</p> <p>Developing assessments for new, multidimensional constructs like CT practices is a time-consuming, expensive and necessarily an iterative process. The CT domain itself is currently more broad than deep, and more emergent than organized, which makes locating and specifying the CT Practice constructs require ongoing guidance from experts in computer science and assessment. Construct shifts are a real concern in this domain as the main concepts and skills associated with CT in CS, as well as in other STEM subject areas, are continually emerging and being refined. Related to this is the challenge of mapping the domain and CT practices to specific curricula and their learning objectives.</p> <p>Without a clear, well-defined CT assessment framework to guide the creation of assessments, designers have to spend more time building consensus on the knowledge and skills that are important to measure in the CT domain, the kinds of tasks and situations that elicit the desired performances and products, and the features of these performances and products that convey evidence of CT practices. This work is often particularly challenging when assessments are needed in new and emerging domains like computational thinking, or when the boundaries of the constructs are broadened, as we have seen with the introduction of the latest revisions of the science standards in the United States. We believe that ECD has strong potential to be leveraged to address the challenges with designing and developing assessments of new, multidimensional constructs like CT practices.</p> <p>We point in particular to the design patterns for computational thinking practices we have developed and the cases we have presented as examples of how the design patterns can be applied in primary and secondary instructional settings. The use of design patterns can help scaffold decisions that need to be made around the development of scenarios, algorithms, item formats and scoring. While the initial creation of design patterns can seem like a daunting task, this investment can provide a framework that can be used by assessment developers, curriculum designers, and teachers to develop assessment opportunities for students. Design patterns can be used to address the challenge of how to measure the CT practices, by first providing a clear list of the specific <emph>f</emph>KSAs related to each practice, and then providing guidance on how to develop tasks that measure these <emph>f</emph>KSAs.</p> <p>One of the main challenges our development team faced was developing scenarios for the CT practice assessment tasks, both in terms of keeping the scenarios focused on the target skills, and on making sure they remained developmentally and culturally appropriate. Here again, we made sure to obtain feedback from partners about typical experiences that students in our age range might have and we had a review that focused on ensuring that the contexts provided were accessible to students. The CT practices design patterns helped our development team keep the scenarios focused on the relevant <emph>f</emph>KSAs and not on the additional KSAs, and ensured that they designed the different parts of the task scenario to elicit evidence consistent with the relevant potential observations. Another challenge with which CT practice design patterns helped the assessment development team was in tracking and managing the interplay of potential work products and potential observations as the assessment tasks were piloted and revised.</p> <p>For example, in both the ECS Unit 3 and Unit 4 assessments, we moved from more open-ended tasks to two-part tasks that better aligned with new <emph>f</emph>KSAs and additional KSAs, delineating the ability to use HTML or <emph>Scratch</emph> code from knowledge of the actual code. In order to support this change, we provided students with pages of basic HTML or <emph>Scratch</emph> code that they could use and specified the potential observations to allow for some errors in the way the code could be specified in the second part of the tasks.</p> <p>These examples also highlight another important characteristic of design patterns, namely that they should be seen as living documents that help assessment development teams keep track of the important decisions about what to measure and how to do that as new assessments are piloted and revised. For example, recently, in some CS learning environments students' log data is being collected and analyzed as students build computational artifacts with the hope that the log data can provide empirical evidence of students' CT practices. Basu, Biswas, and Kinnebrew ([<reflink idref="bib4" id="ref64">4</reflink>]) analyzed log data from a computational modeling environment for middle school science to assess students' CT practices related to abstraction, modularization, and testing and debugging.</p> <p>Computational thinking practices are essential skills in the modern world and many nations have implemented coding and computational thinking instruction as part of compulsory education, seeking to build these skills in future citizens of their nations. However, efforts at designing, piloting and validating assessments of CT practices have not kept pace with the evidentiary needs of these initiatives. While there are many interrelated facets to this challenge—logistical, methodological, financial, and political—there are also many opportunities to apply principled approaches, such as ECD, to design and develop assessments that can meet teachers' and other education stakeholders needs as the initiatives come fully to scale.</p> <p>Table 1 Five Layers of Activity in Evidence-Centered Design.</p> <p> <ephtml> <table><thead><tr><td>Layer</td><td>Activity</td></tr></thead><tbody valign="top"><tr><td>Domain Analysis</td><td>Gather substantive information about the domain of interest (e.g., computational thinking practices) that has implications for assessment; how knowledge is constructed, acquired, used, and communicated</td></tr><tr><td>Domain Modeling</td><td>Express information from domain analysis in narrative form as an assessment argument:<list list-type="Bullet"><list-item><p>knowledge and skills to be assessed</p></list-item><list-item><p>kinds of tasks/situations that elicit performances</p></list-item><list-item><p>features of performances that convey evidence</p></list-item></list></td></tr><tr><td>Conceptual Assessment Framework</td><td>Further specify assessment argument in structures and other details for tasks and tests, evaluation procedures, and measurement models</td></tr><tr><td>Assessment Implementation</td><td>Implement assessment, including presentation-ready tasks and calibrated measurement models</td></tr><tr><td>Assessment Delivery</td><td>Coordinate interactions of students and tasks: task- and test-level scoring; reporting</td></tr></tbody></table> </ephtml> </p> <p>1 Source: Adapted from Haertel et al. (2016).</p> <ref id="AN0136766079-16"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref3" type="bt">1</bibl> <bibtext> See https://<ulink href="http://www.nsf.gov/news/special%5freports/csed/csforall.jsp">www.nsf.gov/news/special%5freports/csed/csforall.jsp</ulink>.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref5" type="bt">2</bibl> <bibtext> See https://<ulink href="http://www.csteachers.org/page/standards">www.csteachers.org/page/standards</ulink>.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref6" type="bt">3</bibl> <bibtext> See https://k12cs.org/.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref7" type="bt">4</bibl> <bibtext> See https://code.org/educate/csd.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref8" type="bt">5</bibl> <bibtext> See <ulink href="http://www.exploringcs.org/">http://www.exploringcs.org/</ulink>.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref9" type="bt">6</bibl> <bibtext> See https://apstudent.collegeboard.org/apcourse/ap-computer-science-principles.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref1" type="bt">7</bibl> <bibtext> Eric Snow is now affiliated with Educational Testing Service.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref35" type="bt">8</bibl> <bibtext> Color versions of one or more figures in the article can be found online at <ulink href="http://www.tandfonline.com/hijt">www.tandfonline.com/hijt</ulink></bibtext> </blist> </ref> <ref id="AN0136766079-17"> <title> References </title> <blist> <bibtext> Aho, A. V. (2011). Ubiquity symposium: Computation and computational thinking. Ubiquity, 2011, 1. doi: 10.1145/1922681.1922682</bibtext> </blist> <blist> <bibtext> Arpaci-Dusseau, A., Astrachan, O., Barnett, D., Bauer, M., Carrell, M., Dovi, R., ... Uche, C. (2013). Computer science principles: Analysis of a proposed advanced placement course. Paper presented at the 44th ACM Technical Symposium on Computer Science Education, New York, NY : ACM. doi: 10.1145/2445196.2445273</bibtext> </blist> <blist> <bibtext> Barr, D., Harrison, J., & Conery, L. (2011). Computational thinking: A digital age skill for everyone. Learning & Leading with Technology, 38 (6), 20 – 23. Retrieved from <ulink href="http://eric.ed.gov/?id=EJ918910">http://eric.ed.gov/?id=EJ918910</ulink></bibtext> </blist> <blist> <bibtext> Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a Computational Thinking-based science learning environment. User Modeling and User-Adapted Interaction, 27 (1), 5 – 53.</bibtext> </blist> <blist> <bibtext> Basu, S., Biswas, G., Sengupta, P., Dickes, A., Kinnebrew, J. S., & Clark, D. (2016). Identifying middle school students' challenges in computational thinking-based science learning. Research and Practice in Technology Enhanced Learning, 11 (1), 1 – 35.</bibtext> </blist> <blist> <bibtext> Bienkowski, M., Snow, E., Rutstein, D., & Grover, S. (2015). Assessment design patterns for computational thinking practices: A first look. Menlo Park, CA : SRI International. Retrieved from <ulink href="http://pact.sri.com/resources.html">http://pact.sri.com/resources.html</ulink></bibtext> </blist> <blist> <bibtext> Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., & Engelhardt, K. (2016). Developing computational thinking in compulsory education– implications for policy and practice. European Commission, Joint Research Centre. EUR 28295 EN, doi: 10.2791/792158.</bibtext> </blist> <blist> <bibtext> Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Paper presented at the 2012 Annual Meeting of the American Educational Research Association, Vancouver, Canada.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref22" type="bt">9</bibl> <bibtext> Cheng, B. H., Ructtinger, L., Fujii, R., & Mislevy, R. (2010). Assessing Systems Thinking and Complexity in Science (Large-Scale Assessment Technical Report 7). Menlo Park, CA : SRI International.</bibtext> </blist> <blist> <bibtext> Computer Science Teachers Association (CSTA) Standards Revision Task Force. (2017). K-12 computer science standards, Revised 2017. New York, NY : ACM.</bibtext> </blist> <blist> <bibtext> CSTA Standards Task Force (2011). CSTA K-12 Computer Science Standards, Revised 2011. New York, NY : CSTA. Retrieved from <ulink href="http://c.ymcdn.com/sites/www.csteachers.org/resource/resmgr/Docs/Standards/CSTA%5fK-12%5fCSS.pdf">http://c.ymcdn.com/sites/www.csteachers.org/resource/resmgr/Docs/Standards/CSTA%5fK-12%5fCSS.pdf</ulink></bibtext> </blist> <blist> <bibtext> DeBarger, A. H., & Snow, A. (2010). Design pattern on model use in interdependence among living systems (Large-Scale Assessment Technical Report 13). Menlo Park, CA : SRI International.</bibtext> </blist> <blist> <bibtext> Department for Education. (2013). National curriculum in England: Computing programmes of study. Retrieved from https://<ulink href="http://www.gov.uk/government/publications/national-curriculum-in-england-computing-programmes-of-study">www.gov.uk/government/publications/national-curriculum-in-england-computing-programmes-of-study</ulink></bibtext> </blist> <blist> <bibtext> Education University of Hong Kong (EdUHK), Massachusetts Institute of Technology (MIT), and City University of Hong Kong (CityU). (2017). CoolThink@JC. Hong Kong : Hong Kong Jockey Club Charities Trust.</bibtext> </blist> <blist> <bibtext> Gal-Ezer, J., & Stephenson, C. (2014). A tale of two countries: Successes and challenges in K-12 computer science education in Israel and the United States. ACM Transactions on Computing Education, 14 (2), 1 – 8.</bibtext> </blist> <blist> <bibtext> Hemmendinger, D. (2010). A plea for modesty. ACM Inroads, 1 (2), 4 – 7.</bibtext> </blist> <blist> <bibtext> Herman, G., Loui, M., & Zilles, C. (2010). Creating the digital logic concept inventory. Paper presented at the 41 st Technical Symposium on Computer Science Education, New York, NY : ACM. doi: 10.1145/1734263.1734298</bibtext> </blist> <blist> <bibtext> K–12 Computer Science Framework (2016). Retrieved from <ulink href="http://www.k12cs.org">http://www.k12cs.org</ulink></bibtext> </blist> <blist> <bibtext> Koh, K. H., Basawapatna, A. R., Bennett, V. E., & Repenning, A. (2010). Towards the automatic recognition of computational thinking for adaptive visual language learning. Paper presented at VL/HCC '10, IEEE Computer, Madrid, Spain.</bibtext> </blist> <blist> <bibtext> Kurland, M., & Pea, R. (1985). Children's mental models of recursive logo programs. Journal of Educational Computing Research, 1 (2), 235 – 243. doi: 10.2190/JV9Y-5PD0-MX22-9J4Y</bibtext> </blist> <blist> <bibtext> McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., ... Wilusz, T. (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. ACM SIGCSE Bulletin, 33 (4), 125 – 180.</bibtext> </blist> <blist> <bibtext> Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25 (4), 6 – 20.</bibtext> </blist> <blist> <bibtext> Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design. In T. M. Haladyna and S. M. Downing (Eds.), Handbook of test development (pp. 61 – 90). New York, NY : Routledge.</bibtext> </blist> <blist> <bibtext> Mislevy, R. J., Riconscente, M. M., & Rutstein, D. W. (2009). Design patterns for assessing model-based reasoning (large-scale assessment technical report 6). Menlo Park, CA : SRI International.</bibtext> </blist> <blist> <bibtext> Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1 (1), 3 – 62.</bibtext> </blist> <blist> <bibtext> National Research Council (2010). Report of a workshop on the scope and nature of computational thinking. Washington, DC : The National Academies Press. doi: 10.17226/12840.</bibtext> </blist> <blist> <bibtext> Nicholson, K., Good, J., & Howland, K. (2009). Concrete thoughts on abstraction. Paper presented at the Psychology of Programming Interest Group (PPIG 2009), Limerick, Ireland. Retrieved from <ulink href="http://www.ppig.org/library/paper/concrete-thoughts-abstraction">http://www.ppig.org/library/paper/concrete-thoughts-abstraction</ulink></bibtext> </blist> <blist> <bibtext> Robertson, J., & Howells, C. (2008). Computer game design: Opportunities for successful learning. Computers and Education, 50 (2), 559 – 578.</bibtext> </blist> <blist> <bibtext> Seiter, L., & Foreman, B. (2013, August). Modeling the learning progressions of computational thinking of primary grade students. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research (pp. 59 – 66). New York, NY : ACM.</bibtext> </blist> <blist> <bibtext> Sneider, C., Stephenson, C., Schafer, B., & Flick, L. (2014). Exploring the science framework and NGSS: Computational thinking in the science classroom. Science Scope, 38 (3), 10 – 15.</bibtext> </blist> <blist> <bibtext> Snow, E., Tate, C., Rutstein, D., & Bienkowski, M. (2017). Assessment design patterns for computational thinking practices in exploring computer science. Menlo Park, CA : SRI International. Retrieved from <ulink href="http://pact.sri.com/resources.html">http://pact.sri.com/resources.html</ulink></bibtext> </blist> <blist> <bibtext> Sosniak, L. A. (1994). Bloom's taxonomy, L. W. Anderson (Ed.). Chicago, IL : University of Chicago Press.</bibtext> </blist> <blist> <bibtext> Tew, E. A., & Guzdial, M. (2010). The FCS1: A language independent assessment of CS1 knowledge. Paper presented at the 42nd ACM Technical Symposium on Computer Science Education. doi:10.1145/1953163.1953200</bibtext> </blist> <blist> <bibtext> Webb, D. C. (2010). Troubleshooting assessment: An authentic problem-solving activity for it education. Procedia-Social and Behavioral Sciences, 9, 903 – 907.</bibtext> </blist> <blist> <bibtext> Werner, L., Denner, J., Bliesner, M., & Rex, P. (2009). Can middle-schoolers use Storytelling Alice to make games? Results of a pilot study. Paper presented at the 4th International Conference on Foundations of Digital Games. doi: 10.1145/1536513.1536552</bibtext> </blist> <blist> <bibtext> Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). The fairy performance assessment: Measuring computational thinking in middle school. Paper presented at the 43rd ACM Technical Symposium on Computer Science Education. doi:10.1145/2157136.2157200</bibtext> </blist> <blist> <bibtext> Wilson, C., Sudol, L. A., Stephenson, C., & Stehlik, M. (2010). Running on empty: The failure to teach K-12 computer science in the digital age. New York, NY : Association for Computing Machinery/Computer Science Teachers Association.</bibtext> </blist> <blist> <bibtext> Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49 (3), 33 – 35. doi: 10.1145/1118178.1118215</bibtext> </blist> <blist> <bibtext> Yadav, A., Burkhart, D., Moix, D., Snow, E., Bandaru, P., & Clayborn, L. (2015). Sowing the seeds: A landscape study on assessment in secondary computer science education. New York, NY : Comp. Sci. Teachers Assn.</bibtext> </blist> <blist> <bibtext> Yadav, A., Good, J., Voogt, J., & Fisser, P. (2017). Computational thinking as an emerging competence domain. In Competence-based vocational and professional education (pp. 1051 – 1067). New York, NY : Springer International Publishing.</bibtext> </blist> </ref> <aug> <p>By Eric Snow; Daisy Rutstein; Satabdi Basu; Marie Bienkowski and Howard T. Everson</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib40" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib10" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib13" firstref="ref10"></nolink> <nolink nlid="nl4" bibid="bib15" firstref="ref12"></nolink> <nolink nlid="nl5" bibid="bib14" firstref="ref13"></nolink> <nolink nlid="nl6" bibid="bib22" firstref="ref14"></nolink> <nolink nlid="nl7" bibid="bib23" firstref="ref17"></nolink> <nolink nlid="nl8" bibid="bib25" firstref="ref18"></nolink> <nolink nlid="nl9" bibid="bib12" firstref="ref20"></nolink> <nolink nlid="nl10" bibid="bib24" firstref="ref21"></nolink> <nolink nlid="nl11" bibid="bib31" firstref="ref25"></nolink> <nolink nlid="nl12" bibid="bib38" firstref="ref26"></nolink> <nolink nlid="nl13" bibid="bib26" firstref="ref27"></nolink> <nolink nlid="nl14" bibid="bib11" firstref="ref28"></nolink> <nolink nlid="nl15" bibid="bib17" firstref="ref29"></nolink> <nolink nlid="nl16" bibid="bib21" firstref="ref30"></nolink> <nolink nlid="nl17" bibid="bib33" firstref="ref31"></nolink> <nolink nlid="nl18" bibid="bib35" firstref="ref32"></nolink> <nolink nlid="nl19" bibid="bib27" firstref="ref33"></nolink> <nolink nlid="nl20" bibid="bib28" firstref="ref34"></nolink> <nolink nlid="nl21" bibid="bib16" firstref="ref40"></nolink> <nolink nlid="nl22" bibid="bib30" firstref="ref44"></nolink> <nolink nlid="nl23" bibid="bib18" firstref="ref46"></nolink> <nolink nlid="nl24" bibid="bib36" firstref="ref47"></nolink> <nolink nlid="nl25" bibid="bib29" firstref="ref48"></nolink> <nolink nlid="nl26" bibid="bib19" firstref="ref49"></nolink> <nolink nlid="nl27" bibid="bib20" firstref="ref52"></nolink> <nolink nlid="nl28" bibid="bib34" firstref="ref53"></nolink> <nolink nlid="nl29" bibid="bib37" firstref="ref56"></nolink> <nolink nlid="nl30" bibid="bib39" firstref="ref57"></nolink> <nolink nlid="nl31" bibid="bib32" firstref="ref59"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1217617
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Snow%2C+Eric%22">Snow, Eric</searchLink><br /><searchLink fieldCode="AR" term="%22Rutstein%2C+Daisy%22">Rutstein, Daisy</searchLink><br /><searchLink fieldCode="AR" term="%22Basu%2C+Satabdi%22">Basu, Satabdi</searchLink><br /><searchLink fieldCode="AR" term="%22Bienkowski%2C+Marie%22">Bienkowski, Marie</searchLink><br /><searchLink fieldCode="AR" term="%22Everson%2C+Howard+T%2E%22">Everson, Howard T.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Testing%22"><i>International Journal of Testing</i></searchLink>. 2019 19(2):103-127.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 25
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Tests%22">Cognitive Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Thinking+Skills%22">Thinking Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+Education%22">Computer Science Education</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Hong+Kong%22">Hong Kong</searchLink><br /><searchLink fieldCode="DE" term="%22United+States%22">United States</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/15305058.2018.1543311
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1530-5058
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Computational thinking is a core skill in computer science that has become a focus of instruction in primary and secondary education worldwide. Since 2010, researchers have leveraged Evidence-Centered Design (ECD) methods to develop measures of students' Computational Thinking (CT) practices. This article describes how ECD was used to develop CT assessments for primary students in Hong Kong and secondary students in the United States. We demonstrate how leveraging ECD yields a principled design for developing assessments of hard-to-assess constructs and, as part of the process, creates reusable artifacts--design patterns and task templates--that inform the design of other, related assessments. Leveraging ECD, as described in this article, represents a principled approach to measuring students' computational thinking practices, and situates the approach in emerging computational thinking curricula and programs to emphasize the links between curricula and assessment design.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1217617
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1217617
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/15305058.2018.1543311
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
        StartPage: 103
    Subjects:
      – SubjectFull: Evidence Based Practice
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Cognitive Tests
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Thinking Skills
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Computer Science Education
        Type: general
      – SubjectFull: Hong Kong
        Type: general
      – SubjectFull: United States
        Type: general
    Titles:
      – TitleFull: Leveraging Evidence-Centered Design to Develop Assessments of Computational Thinking Practices
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Snow, Eric
      – PersonEntity:
          Name:
            NameFull: Rutstein, Daisy
      – PersonEntity:
          Name:
            NameFull: Basu, Satabdi
      – PersonEntity:
          Name:
            NameFull: Bienkowski, Marie
      – PersonEntity:
          Name:
            NameFull: Everson, Howard T.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2019
          Identifiers:
            – Type: issn-print
              Value: 1530-5058
          Numbering:
            – Type: volume
              Value: 19
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: International Journal of Testing
              Type: main
ResultId 1