NAEP and Policy: Chasing the Tail of the Assessment Tiger
Saved in:
| Title: | NAEP and Policy: Chasing the Tail of the Assessment Tiger |
|---|---|
| Language: | English |
| Authors: | Diket, Read M., Brewer, Thomas M. |
| Source: | Arts Education Policy Review. 2011 112(1):35-47. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 13 |
| Publication Date: | 2011 |
| Document Type: | Journal Articles Reports - Evaluative |
| Education Level: | Elementary Secondary Education |
| Descriptors: | Visual Arts, Art Education, National Competency Tests, Educational Policy, Policy Analysis, Politics of Education, Educational Assessment, Teacher Role |
| Assessment and Survey Identifiers: | National Assessment of Educational Progress |
| DOI: | 10.1080/10632913.2011.518126 |
| ISSN: | 1063-2913 |
| Abstract: | This paper grabs hold of the "assessment tiger" by considering the history of the National Assessment of Educational Progress (NAEP) "Arts Report Cards" for the visual arts, which were constructed and have been administered four times within thirty-five years. Two purposes of the NAEP have persisted since its founding: (1) measuring learning and (2) developing expectations for change and reform in education from a national platform. These purposes provided the basis for a functional analysis derived from Efland (1976) that looked closely at the NAEP's collaborative context, consensus building, and block content. Hamblen's 1995 critical theory analysis exemplar also serves as a means for reviewing the politics of assessment, influences on policy directions, and curricular emphases within a subject area. Using Persky's 2004 critical evaluation of the 1997 NAEP Arts Assessment, this article considers recommendations she derived from the 1997 NAEP "Arts Report Card" that guided the 2008 NAEP construction and administration. The 2008 NAEP "Arts Report Card," released in June 2009, informs a number of initial interpretations, recommendations, implications, and applications for those who want to "own" the tiger. |
| Abstractor: | As Provided |
| Number of References: | 31 |
| Entry Date: | 2010 |
| Accession Number: | EJ906018 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHuiTzZk8R5N18lFK-NXp0QAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDMJtmBYNaUhZgfaWUgIBEICBmjT8_kWsvlIFQ7tZS23oNpRiH_yystOT19YP1aW8ugCGsJBiffjDdWEOc3VzOdGlzSwGO_kQTXqu3pCvZv_XDw5PPJ2odPn_yLxfd9fUt6HzsY46FIF72kZ-2dhed5cULIQnNIi1xjDxsd6K3pT-yZ3Z77z2Ofl3PfMC07Tpms3rNwUORIoS-DtfrexW95RYhBXcdokLqBGJSwc= Text: Availability: 1 Value: <anid>AN0055310780;aep01jan.11;2019Feb14.14:06;v2.2.500</anid> <title id="AN0055310780-1">NAEP and Policy: Chasing the Tail of the Assessment Tiger. </title> <p>This paper grabs hold of the "assessment tiger" by considering the history of the National Assessment of Educational Progress (NAEP) Arts Report Cards for the visual arts, which were constructed and have been administered four times within thirty-five years. Two purposes of the NAEP have persisted since its founding: (<reflink idref="bib1" id="ref1">1</reflink>) measuring learning and (<reflink idref="bib2" id="ref2">2</reflink>) developing expectations for change and reform in education from a national platform. These purposes provided the basis for a functional analysis derived from Efland (1976) that looked closely at the NAEP's collaborative context, consensus building, and block content. Hamblen's 1995 critical theory analysis exemplar also serves as a means for reviewing the politics of assessment, influences on policy directions, and curricular emphases within a subject area. Using Persky's 2004 critical evaluation of the 1997 NAEP Arts Assessment, this article considers recommendations she derived from the 1997 NAEP Arts Report Card that guided the 2008 NAEP construction and administration. The 2008 NAEP Arts Report Card, released in June 2009, informs a number of initial interpretations, recommendations, implications, and applications for those who want to "own" the tiger.</p> <p>When we started preparing this article, we decided to use a combination of Efland's [<reflink idref="bib13" id="ref3">13</reflink>] functional analysis approach to school arts and Hamblen's [<reflink idref="bib15" id="ref4">15</reflink>] critical theory approach to examine the foundations of the National Assessment of Educational Progress (NAEP) <emph>Arts Report Card</emph> in visual art. We applied the principle of best fit when determining the concepts and criteria used from each writer.</p> <p>In constructing this article, the work of Diket, Sabol, and [<reflink idref="bib8" id="ref5">8</reflink>] with the 1997 secondary analysis of the NAEP by the Visual Arts Consortium of university researchers afforded a significant amount of information about the 1997 assessment and suggested several research design and policy comparisons to the first two art assessments in the 1970s (Diket et al., 2009). Laura Chapman's <emph>Instant Art, Instant Culture</emph> also served as a source of information about the 1970s assessments. The most recent iteration of the assessment, <emph>NAEP Arts 2008: Music and Visual Arts Report Card</emph>, was released in June 2009 ([<reflink idref="bib17" id="ref6">17</reflink>]). Some initial interpretations and recommendations regarding this significant national assessment can be found at the end of the article. This article illuminates central issues that may lead to further analyses of the art results released publicly in 2009 and the restricted data still to be released at the time of this publication.</p> <hd id="AN0055310780-2">MID-TWENTIETH-CENTURY FOUNDATIONS FOR NAEP ASSESSMENT AND POLICY ANALYSIS</hd> <p>In the 1960s, education scholar Ralph Tyler worked with U.S. Commissioner of Education Francis Keppel to conceptualize a meaningful national assessment. Tyler wanted to appraise and track student learning in subject areas at specific ages, rather than at grade levels, using a new type of measurement that combined multiple choice and constructed responses. The measurement Tyler envisioned was based on specific subject matter determined by "lay people working together to reach consensus" ([<reflink idref="bib18" id="ref7">18</reflink>], 27). Tyler wanted to determine what students learned over time; in contrast, Keppel was interested in extracting data consistent with legislation that had created the U.S. Office of Education. In our view, both of these purposes were perpetuated in subsequent NAEPs—that is, (<reflink idref="bib1" id="ref8">1</reflink>) measuring learning and (<reflink idref="bib2" id="ref9">2</reflink>) developing expectations for change and reform in education through a national platform.</p> <p>According to <emph>The Nation's Report Card</emph> ([<reflink idref="bib16" id="ref10">16</reflink>]), Congress began to show interest in the idea of a national assessment of education in 1963, but did not provide funding, so schools had no federal incentive to conduct such an assessment. The private sector, led by the Carnegie Corporation, provided start-up funding for conferences in 1963 and 1964 with the primary objective of determining the feasibility of a national assessment. Because of anticipated technical issues, in 1963, eight of the nineteen participants in the first conference were statistical experts; other participants represented corporations or the U.S. Office of Education. The original group also included a university president and a state department of education official. Carnegie Corporation president John Gardner urged the attendees to consider Tyler's list of points. These same points continue to concern stakeholders today and include the framing of test objectives, design issues with existing tests, specifications and procedures for a new kind of test, the reporting of features necessary for making statistical results meaningful to consumers, identification of the report's audience, choices of categories to be reported publicly, and questions of access to restricted data. In his formal summary of this first Carnegie conference, David Goslin indicated that conferees were aware that the national assessment might negatively impact school curricula, and that results might be misinterpreted by the general public. However, in the participants' view, the project had enormous potential to raise standards, foster increased interest in schooling, and encourage governmental support of America's schools.</p> <p>The next Carnegie-sponsored meeting took place in January 1964 and included elementary and secondary educators, school superintendents, and leaders from the National Education Association. New participants expressed concern about negative impacts on education if results were not favorable, as well as about the effects of another layer of testing on schools and students. At a third Carnegie-sponsored meeting in 1964, chief state school officers, along with educators and leaders with political influence, were invited to broaden the research and support base of the project ([<reflink idref="bib18" id="ref11">18</reflink>]). Later that year, the Exploratory Committee for the Assessment of Progress in Education (ECAPE) formulated a plan for testing objectives in a corpus of subject areas, including the fine arts and particularly emphasizing art and music. Music preceded visual arts at the planning table.</p> <p>After the 1966 renewal of the Elementary and Secondary Education Act, and with increased education funding, evaluation was instituted as a requirement and key component of new federal programs, and ECAPE became CAPE, the Committee for the Assessment of Progress in Education. Dissemination of results to the public following each national assessment apparently consumed a great deal of assessment officers' time, and the failure to develop a public relations approach early on—which Tyler had strongly advocated—likely affected the ability of national assessment to serve as a political instrument ([<reflink idref="bib18" id="ref12">18</reflink>]). In 1969, CAPE was reduced to an advisory body and soon ceased to exist. New governance was later established, and the project was named the National Assessment of Educational Progress (NAEP). In 1988, Congress established the National Assessment Governing Board (NAGB) and authorized federal funding for trial state assessments. In 1992, NAEP initiated the Secondary Analysis Grant Program; most recently, however, secondary analysis efforts and funding have been grouped with other programs.</p> <hd id="AN0055310780-3">FUNCTIONAL ANALYSIS I: EFLAND</hd> <p>To gain a deeper understanding of the aims and purposes of the NAEP Arts Assessment, we based the first part of our analysis on an approach devised by Efland in 1976 to examine school art. We drew criteria from the Efland functional analysis to discover agendas, ideologies, and latent functions in the NAEP Art Assessment that are not readily perceptible to the public. These criteria provided clearly defined linkages in our analysis, presentation, and recommendations stemming from the system of federal arts testing. A "hidden agenda" or, as described in Efland's work, a "hidden curriculum" may be present within phenomena. Efland also reveals the ways in which ideological issues, doctrines, opinions, or ways of thinking that an individual class or group exerts can impose themselves or prevail over other possibilities. A latent function or hidden dynamic may be involved in a situation when a policy appears to be beneficial but actually produces a problematic outcome. For example, Efland states that art education in public school has been labeled as a significant academic subject, but in reality, many schools treat art more like recess or time off for good behavior. Of course, it is reality that ultimately prevails in shaping results.</p> <hd id="AN0055310780-4">Is There a Hidden Agenda in the NAEP?</hd> <p>The content and process of the NAEP have never been hidden from public view. In fact, reports on the frameworks, exercises, procedures, personnel, and other relevant issues are available in published form and on the Web. These reports are so numerous as to be somewhat daunting. However, even though the NAEP "curriculum vita" is in plain view, the language of statistics—despite explicit discussion of research designs, item blocks, and procedures used to determine achievement in a subject area—remains mysterious and esoteric to the lay person, including those who are politically oriented ([<reflink idref="bib18" id="ref13">18</reflink>]). Today, NAEP officers attempt to make the assessment results more meaningful by releasing data from assessment blocks to the public. By emphasizing well-chosen examples of what students know and are able to do in subject areas, these NAEP reports substantiate national exemplars of reasoning for targeted age groups. Meaning is not always achieved, because the examples themselves are controversial in terms of the extent to which they can reveal student capabilities. Concerns about hidden agendas in these art assessments include a lack of clarity about the explicit opportunities students have had to study art at school; who taught what about visual art during students' development, and when; where (or whether) this art was taught in an educational setting or the community; and, in particular, how many students in the sample were currently taking an art class at school at the time of the visual arts assessment.</p> <p>The particular blocks that were used in 1997 and 2008 also need explication. For analytical purposes, we placed the item map from 1997 alongside the 2008 item map. Apparently, four of the old blocks were repeated, and it appears likely that more blocks that appeared in the restricted data in 2008 were reused but were not covered in the arts report. The Portrait block is essentially the same, the Mother/Child block has been changed in format and its language updated, and the terms used to describe procedures in the Bearden block have been slightly modified. For example, instructions for test takers changed from asking students to "describe" to asking them to "support with references." The bicycle task, released in 1997 and used again 2008, presented extremely difficult design challenges for students. Product projects may be present in middle school art classes, but formal attention to design is often not a curriculum component. Statisticians trained in the arts and familiar with the ways that art is taught in middle school need to study the restricted data to see if some of these issues can be reported in more detail and with greater clarity.</p> <p>What does this mean in the interim when restricted data from the 2008 NAEP has not been released but the public report is available? We can draw broad implications from the published report using the item map. The Mother/Child block focuses on the treatment of space in historical genres from the Renaissance to Modern periods. To make questions accessible to children that have not formally studied art, children were asked to identify the "modern" work among the Mother/Child reproductions, rather than to identify style. Interest in space treatments continued in questions about the Bearden twentieth-century collage, for which students were asked to recognize projected inner and outer spatial relations that were presented in the same compositional format. The Portrait Block and Bearden blocks were designed to be engaging, even if students did not do well on individual questions, and both had good discrimination within block items on the item map. All of the questions from the bicycle task fell in the upper range of the item map; thus, this task must have been very confusing to students unfamiliar with the design process and design components.</p> <p>Art teachers might want to know that the students who were able to explain their answers achieved higher scores in general—that is, understanding one's own work and being able to discuss it produced a much higher score overall. Good communication about the design and materials of a packaging idea was associated with a higher range of achievement overall, and the ability to explain use of elements and demonstrate understanding of formal relationships was associated with placement in the upper 25-percent level of achievement. Correct identification of examples was associated with the lower end of the upper range of achievement. In the median range (middle 50 percent), students were able to write about their own experiences, recognize degrees of realism, describe their own self-portrait in terms of what it said about the self, and recognize characteristics of media and subject matter. Even children in the lowest achievement quartile could recognize expressive qualities and describe drawing media and "unusual" features in the Bearden collage. </p> <p>None of these features, approaches, or results have been hidden from the public. However, early in the effort to associate the NAEP with art education practices in school settings, Laura Chapman, a noted visual arts education trend analyst, claimed that connections among the reporting levels were not that informative, based on her secondary study of the 1974–75 and 1978–79 NAEPs in visual art. Chapman observed that the opportunity to study visual art was often limited to seventh or eighth grade, as reflected in other national surveys during the period. Further, she noted that the visual arts (as with other core subject areas of the period) needed to put their "own house in order" ([<reflink idref="bib2" id="ref14">2</reflink>], xv) and in part placed the responsibility for constructive use of national NAEP data on the professional field of art education. She noted that art educators "have contributed to the very problems and attitudes about which we are quick to complain" ([<reflink idref="bib2" id="ref15">2</reflink>], xv), and that during the years following the first NAEPs in art, three-fourths of respondents to a public survey were of the opinion that formal study of art was not necessary for understanding or enjoying art. Chapman's [<reflink idref="bib2" id="ref16">2</reflink>] book, <emph>Instant Art, Instant Culture</emph>, was reprinted by the National Art Education Association (NAEA) around 2005. Her reissued book reached a receptive and professionally reflective audience in a field that was then only marginally involved in the policy and program ramifications of the accountability movement. Chapman argued persuasively for vigilance in national assessment matters, lest assessment become destructive rather than constructive. During well-attended sessions of NAEA conferences over the past decade, she has shared deep analyses of national legislation and extracted disturbing descriptive statistics from public reports. Given her analyses, will the field take action today regarding the NAEP and other assessments?</p> <hd id="AN0055310780-5">Contexts</hd> <p>Looking further into extant conditions and critiques, it can be argued that eighth grade provides a good cohort for the NAEP in its study of visual arts understanding, especially if only one national dipstick can be used. Eighth grade offers a developmental midpoint for looking at achievement in visual art. The original NAEP "vision" was to look at the fourth, eighth, and twelfth grades, but the expense of scoring three levels proved prohibitive ([<reflink idref="bib21" id="ref17">21</reflink>]).</p> <p>Art stakeholders can use NAEP data to follow trends in art study in the schools and associate findings with achievement on other special-population and subject-area tests, the ongoing kindergarten study, and NAEPs in reading and writing that were cross-tabulated with course-taking patterns in the High School Transcript Study (<ulink href="http://nces.ed.gov/nationsreportcard/hsts/">http://nces.ed.gov/nationsreportcard/hsts/</ulink>). The SAT also provides data about course–taking correlations (see, e.g., [<reflink idref="bib4" id="ref18">4</reflink>], 96–97), although editors from the Arts Education Partnership cautioned that "more research is needed to interpret the relationship" (<reflink idref="bib6" id="ref19">6</reflink>). All of these studies suggest potential relationships between the study of art and educational achievement in other literacy areas. However, the current advocacy emphasis on "causal" relationships has occluded the comprehensive understanding of correlations and may pose a credibility challenge for the art education field. We simply do not know enough about how artistic understanding converges with general literacy during students' development to allow us to make causal claims.</p> <p>A number of issues raised by participants in the development of the NAEP have persisted, despite the sincerity exhibited by the governing board and assessment contractors and the considerable effort expended in the public reporting process. For example, the particular content of the exercise blocks that are intended to estimate achievement remain unsatisfactory to many stakeholders in art education ([<reflink idref="bib12" id="ref20">12</reflink>]; [<reflink idref="bib28" id="ref21">28</reflink>]). Criticism comes from the technical evaluation community as well. Linn's ([<reflink idref="bib19" id="ref22">19</reflink>]) external evaluation of the NAEP protocol concluded that many recommendations of evaluators were heeded by NAEP contractors, but that some recommendations cannot be pursued because of current statistical constraints. He also noted that a consensus on the "best" statistical method for assessment was lacking. Thus, achievement scores must be interpreted two ways: (<reflink idref="bib1" id="ref23">1</reflink>) in terms of the nature of released questions (what the public release blocks tell us about what students know and can do); and (<reflink idref="bib2" id="ref24">2</reflink>) in light of the particular spiral bib method used in determining achievement scores. The spiral bib process poses a hermeneutical problem for statisticians—the method predetermines and delimits what can be explained from scores presented as percentages and standard errors. The take-home message is that NAEP assessments are not perfect but rather represent a reasoned approach to a very difficult and contentious task. Still, "the analysis of the NAEP achievement levels is one of the few areas where the conclusions of evaluators of NAEP have been uniformly and consistently negative without having any impact on the practice" ([<reflink idref="bib19" id="ref25">19</reflink>], 18). The policy ramifications of this condition are enormous.</p> <p>In 2001, George W. Bush pushed for state-level testing in reading and mathematics in grades 4 and 8, making state participation in the NAEP a requirement, so that state-specific test results could be confirmed by results on the national assessment. In effect, this shift created a high-stakes climate in two subject areas and, notably, invited comparisons among states rather than regions, as was the original NAEP emphasis. That same year, Congress enacted No Child Left Behind, supporting Bush's testing proposal with some modifications. One modification proposed by arts advocates positioned the arts in NCLB legislation, although not with the same level of visibility as existed in Goals 2000. Results for the NAEP that was administered in 2003 were reported under NCLB requirements, and subsequent NAEPs in other fields have also been reported under that legislation. In 2008, the NAEP honored a legislation modification and collected data on achievement in music and the visual arts.</p> <hd id="AN0055310780-6">Consensus Framework and Content</hd> <p>Strong parallels persist between a broad consensus regarding what students ought to know in the area of visual art and national curricular frameworks guiding NAEP test blocks. However, the NAEP is not designed to test what is <emph>taught</emph> at schools—rather, it evaluates achievement based upon a consensus regarding what students ought to know about the visual arts, as formulated in broad-based standards set by selected representatives of the field. The 2008 NAEP framework document offers considerable insight into the consensual "vision" that drives the design of assessment task blocks for eighth grade students (see [<reflink idref="bib22" id="ref26">22</reflink>], vii–viii). The emphases in the following paragraphs are ours; the italicized phrases reveal distinctions of particular import in understanding the explicit language of the 2008 NAEP Arts:</p> <p>The assessment should <emph>affirm and articulate the arts as ways of knowing and forms of knowledge with the unique capacity to integrate the intellect</emph>, the emotions, and physical skills in the construction of meaning.</p> <p>The assessment should <emph>honor discrete disciplines, but should at the same time encourage students to see the artistic experience as a unified whole</emph> and to make connections between the arts and other disciplines.</p> <p> <emph>The NAEP assessment and national standards must work hand in hand.</emph> </p> <p>The assessment <emph>should connect with students' real-life experiences so students can use their personal knowledge, ... their everyday experiences ... or their understanding of traditional regional art forms and community arts resources.</emph></p> <p>The assessment should <emph>assess students' knowledge, attitudes, and performance in the modalities and forms of expression characteristic of the arts ... as well as verbal or written linguistic modes</emph>, that is, writing or talking about the arts.</p> <p>The assessment should go beyond quantification to include critical judgment ... to <emph>ensure that reporting includes descriptive information on student performance as well as numerical data.</emph></p> <p>The assessment should use a common list of background variables to recognize differences and inequities in school resources and the conditions related to achievement, such as teacher qualifications, instructional time in the arts, school structure, cultural and social background of the school community, and incentives. This recognition must be evident in reported data. <emph>Results have meaning only in terms of the availability and continuity of arts instruction.</emph></p> <p>The assessment should <emph>address both processes and products, and should expand the public's information about the importance of each.</emph></p> <p>The assessment should be based on a comprehensive vision of arts education and should communicate that vision clearly. <emph>The assessment should focus on what ought to be in arts education rather than what is</emph>, but idealism should be tempered with reality. Hence, exercises should model multifaceted and thoughtful activities without making unreasonable demands on time, materials, and human resources.</p> <p>The assessment should <emph>reflect a pluralistic view of arts education in terms of both individual products and the cultural bases of the arts</emph>. It should be oriented toward the demonstration of student learning, be sensitive to a variety of instructional approaches, <emph>include the range of contemporary theories evident in arts education, and include examples of appropriate exercises addressing universal themes</emph>.</p> <p>The framework particularized for assessing music and visual art in the 1997 NAEP Arts Assessment was used again for the 2008 NAEP Arts Assessment. The NAEP general framework for the arts, produced in the standards sorting process of the 1990s, reflected subject-area standards developed and endorsed collaboratively by professional organizations and published as the <emph>National Voluntary K–12 Standards for the Arts</emph> (Music Educators National Conference [MENC] [<reflink idref="bib20" id="ref27">20</reflink>]). The explicit intent of the NAEP vision for the arts was that the National Standards and the NAEP Arts Assessment would reflect a common view of arts education. In making inferences about these connections, it helps to remember the distinction between broad consensus and detailed agreement. A common view will not necessarily produce the same content choices or approaches among teachers or school districts.</p> <p>A focus on arts education rather than subject areas such as visual art, music, dance, or theater may be a sticking point if the public does not understand the distinction. For example, the NAEA authored standards specifically for the visual arts, which were published along with standards in other areas of the arts (MENC [<reflink idref="bib20" id="ref28">20</reflink>]), in close proximity to the orchestration of the 1997 NAEP Arts Assessment cycle. Counterpart professional organizations played the same role in formulating the dance, music, and theater portions of the 1994 National Standards. Over the subsequent sixteen years, it is difficult to determine if art education has made significant advances in the sophistication and cohesiveness of visual arts education in the schools, or the extent to which the NAEP provides valid answers to this question that are useful in policy development, especially for art education at the local level.</p> <p>Visual art education has been more willing to listen to its own "homegrown" critics, acknowledging and reviewing influential studies that affirm a line of inquiry that insists on experimental rigor in reporting results. Of note are Winner and Hetland's [<reflink idref="bib31" id="ref29">31</reflink>] study and the large compilation of studies entitled <emph>Critical Links</emph> ([<reflink idref="bib4" id="ref30">4</reflink>]). Burton, Horowitz, and [<reflink idref="bib1" id="ref31">1</reflink>] countered these analyses with their study of "habits of mind" associated with art in school, providing a more positive outlook, concurring with arts research reported in Fiske (1999) and a magnet school NAEP replication study by Siegesmund, Diket, and McColloch (2001). While it is possible to assess what students in a specific classroom or school or district are learning about art in art courses, the national picture remains elusive.</p> <p>We are fortunate in the visual art subject area that the NAEP vision (see [<reflink idref="bib22" id="ref32">22</reflink>], 27) regarding interdisciplinary work that crosses arts subject areas or combines visual art with other core subjects in school was not explicitly made a part of the 2008 arts assessment. If this component had been formally articulated, aspects of the visual art disciplines might have been assessed alongside components from music, or through testing understanding of the importance of art in a specific historical period. The visual arts, like music as a field, need separate subject assessments, first, to establish and verify acquisition of foundational knowledge from the field, and second, to study closely the transmission of field-valued knowledge, skills, and experiences as they persist in school settings.</p> <p>Inevitably, reading and writing skill is reflected as a communication component of achievement in art subjects. NAEP blocks for these areas involve the "picturing" of art works (inventions and sculptures that require understanding artistic intent), and blocks have been released that query understanding of arts function in ancient cultures. These examples from reading and writing NAEPs at the fourth, eighth, and twelfth grades show that NAEP agents understand and are exploring the portents of multidisciplinary questioning.</p> <p>The NAEP questionnaires provide information that enables comparisons of program features. For example, the intensity of arts study (time in art class), art media availability (clay, painting, or neither), and student choices to use artistic products in other school subjects (school climate of receptiveness) represent various program features. The NAEP does not compare the philosophies and theories that may shape programs in school. A few consensual indications regarding philosophy in the area of art do appear in the texts of NAEP reports. These few instances probably represent only a consensus among participants at the planning table.</p> <hd id="AN0055310780-7">What about Ideologies, Doctrines, or Ways of Thinking from a Dominant Class or Group?</hd> <p>How do these issues relate to the initiation and development of consensus frameworks? The visual art representatives at the table during the NAEP Arts Assessment development in the 1990s included notables like Elliot Eisner (a key curricular theorist for visual art), Jerome Hausman (one of three visual art analysts for the creation of the NAEP in the 1970s), and Tom Hatfield (then executive director of the NAEA; [<reflink idref="bib29" id="ref33">29</reflink>]). Other well-known art educators and researchers contributed as advisors, planning committee members, subcontractors, and liaisons. <emph>Arts Education Policy Review</emph> recorded the mixed reactions of these insiders to the 1997 NAEP Art Assessment. Eisner remained aloof from the test findings (1999), as did Mary Ann [<reflink idref="bib28" id="ref34">28</reflink>], a consultant from the Getty Trust who was then engaged in promoting serious study of art using the controversial discipline-based art education (DBAE) model.</p> <p>In his 1999 article, Eisner contributed some key points to the dialogue in his reflection on the 1997 visual arts NAEP. First, he noted that the NAEP takes the temperature in a subject area, but measurement can only guide efforts to improve achievement—fields have a responsibility to orchestrate referendums, and, originally, the test was intended to be a low-stakes assessment. Second, NAEPs are open for inspection; however, the formal reports do not define problems or formulate approaches for improvement in the subject. Third, the basis for standardization and benchmarks are consensual among proponents but not explicitly described for the public. Fourth, the NAEP is not designed or constructed to provide predictive validity or make direct comparisons to other forms of performance. Finally, NAEP blocks are, at best, artificial situations of limited duration that tend to separate a student from personal experiences with art at his or her school; outside of school, opportunities correlate with parental education, a relationship also noted by [<reflink idref="bib28" id="ref35">28</reflink>]. In summary, detailed consensus about the NAEP is elusive, even among practitioners in visual arts. For those outside of art education, the NAEP report supposes that art can be learned in any cultural setting and from a variety of informants (e.g., artists, local arts agents, generalist educators, and librarians).</p> <hd id="AN0055310780-8">Secondary Analysis</hd> <p>The visual arts have a history of secondary analysis of the NAEP. As already noted, Chapman began such efforts in the 1970s ([<reflink idref="bib2" id="ref36">2</reflink>]). A team composed of Diket, Sabol, Burton, and McCollister traveled to the Washington area for an NAEP Arts training course in the autumn after the release of the NAEP <emph>Arts Report Card</emph> in the spring of 1999. The group worked with restricted data on site and was rewarded with important discoveries that merited further investigation ([<reflink idref="bib9" id="ref37">9</reflink>]). Eventually, this team formed the Visual Arts Consortium of University Researchers, which authored and was awarded an NAEP secondary analysis grant for study of the 1997 NAEP. The consortium's project led to a four-part statistical revaluation of NAEP findings that organized background and sorting variables around factors such as regional comparisons on critical issues, provided quartile analysis of teacher and instructional variables regarding achievement, and modeled assessment results as a "path" correlating with school and external factors (Diket, [<reflink idref="bib8" id="ref38">8</reflink>]). The resulting publication also reviewed current doctrines, options, and thinking modes regarding visual art and related NAEP findings to expectations for the field.</p> <p>In 2008, the Visual Arts Consortium of University Researchers expanded to include former NAEA Research Commission president Thomas Brewer, as well as Richard Siegesmund, NAEP replication specialist from art education. Outreach to emerging art education researchers, fostered through the Southeastern College Art Conference ([<reflink idref="bib6" id="ref39">6</reflink>]), also brought Karen Heid and Bryna Bobick to the project. Under an NAEA travel grant, the consortium primaries met in late October of 2008 to plan a secondary analysis of the 2008 NAEP visual art data. Plans also included collaboration with younger colleagues to ensure future perpetuation of expertise regarding NAEP for the art education field. The consortium sought and benefited from the sage advice of Laura Chapman.</p> <p>Secondary analyses of restricted data and block replications of released blocks afford art education researchers means to extract and focus additional data from assessments and clarify them for reporting to the field. The availability of extensive discussion regarding released blocks on the NAEP Web site enables teachers who seek guidance from the NAEP to use the blocks as models for designing their own assessments, as pre-assessment instruments for a unit or module of study, or as a means of comparing local learning data to national data.</p> <p>As the [<reflink idref="bib22" id="ref40">22</reflink>]<emph>Arts Education Assessment Framework</emph> aptly notes:</p> <p>For some, the NAEP assessment will be too soft; for others, not far enough. Such is the nature of a process that strives for consensus (agreement at certain levels of acceptance) rather than absolute agreement (a process that builds from a broad base of national input). (<reflink idref="bib6" id="ref41">6</reflink>)</p> <hd id="AN0055310780-9">Is There a Hidden Dynamic and Is Art Education Still Seen as Time Off for Good Behavior?</hd> <p>A latent function or hidden dynamic may be occurring in today's art education policy and assessment environment—that is, we may see a particular policy that appears to be beneficial but in actual practice achieves the opposite effect or no progress at all. For example, the presence of national and state standards for the visual arts and the rigor of the 1997 and 2008 NAEP Arts Assessments represent art as a significant academic subject to policymakers and the public. An unintentional hidden dynamic produces a different reality about art in school. Despite many efforts, fewer than 50 percent of the eighth graders who were administered the 2008 NAEP Art Assessment had received art instruction. Those who did may have received art instruction that was low in cognitive complexity or provided as time off for good behavior, or as free drawing opportunities. The 1997 NAEP <emph>Arts Report Card</emph> (Persky, [<reflink idref="bib24" id="ref42">24</reflink>]) exemplifies the quality of arts instruction and makes the places where students gain access to artistic ideas more explicit. In contrast, the 2008 <emph>Arts Report Card</emph> shows a continuum of opportunity to study art at school (and in the general culture) over the two tests and looks at changes in instructional content. Distinctions about the learning opportunities that students in the assessment actually experienced at school are not made at all clear in the 2008 report. We know what students taking art reported about art classes at school, but we do not know how many of those assessed were actually taking art in the assessment year. The hidden dynamic persists in the disregard for uneven opportunities available to students and the lack of time and resources devoted to rigorous curricular study of any aspect of art in many, if not most, schools. This lack of correlation between study and learning may be one reason the NAEP report for art does not arouse the same general anxieties as the NAEP report for math.</p> <hd id="AN0055310780-10">FUNCTIONAL ANALYSIS II: HAMBLEN</hd> <p>In 1995, Karen Hamblen set forth a critical theory–based policy analysis framework appropriate for examining the political history of the NAEP Arts Assessment. Hamblen notes that critical theory presupposes that all ideas and actions have political aspects based on selected values and unequal power relationships. The power of money—private or public—influence, and prestige affect all policymaking, even in the smallest arenas. Thus, it is important to determine who is involved in a policy, what policy is selected, and who does and does not benefit from the policy. It is important to understand the policymakers' backgrounds and determine whether the committee or boards are window dressing or a truly democratic mix, as well as the extent to which policymakers have content knowledge and pedagogical acumen. It is also important to recognize that research in the arts is typically under-funded, which often opens the door to the influence of foundations with specialized interests and money to spend. Hamblen points out that an understanding of what policies are (or are not) presented, discussed, and implemented can have significant importance in understanding what is happening and determining ramifications. Often, surprising final reports are produced that committee members do not recognize. Some of these final reports seem to be public relations tools, rather than agreed-upon evaluations or comprehensive policy recommendations.</p> <p>In addition, Hamblen argues that policy agendas are rarely grassroots efforts by teachers clamoring for more standards, assessments, and administrative procedures and are often dismissed by teachers as top-down and thus irrelevant. However, field practitioners ought to (<reflink idref="bib1" id="ref43">1</reflink>) be critical and constructive players in policymaking, (<reflink idref="bib2" id="ref44">2</reflink>) develop strong courses of action, and (<reflink idref="bib3" id="ref45">3</reflink>) turn agendas and policy into practice. She recommends that arts educators in a variety of roles be active policymakers and policy critics. These guidelines, among others, proved useful for our continuing analysis of NAEP that follows.</p> <hd id="AN0055310780-11">Teachers as Critical and Constructive Players</hd> <p>Critical theory presupposes that all ideas and actions have political aspects that are based on selected values and unequal power relationships. In the context of the NAEP, Diket's deep study of NAEP task blocks has observed that a priority is not placed on the directionality or "path" of creative solutions developed by students ([<reflink idref="bib5" id="ref46">5</reflink>]). Using the 1997 restricted data, Pamela Thorpe and Diket (Diket et al. 2002) conducted an AMOS path analysis of the Portrait block and found junctures at which students dropped out of the block. They suggested some ideas for the NAEP through Richard Siegesmund for replicating the Portrait block as a qualitative study. Although Thorpe and Diket knew where the breaks in the train of reasoning were happening, they could not verify from statistical analysis alone why and how this disjunct was occurring in a specific block of items in students' work. Diket served as consultant to Siegesmund's second NAEP replication, using the Bearden block in the 1997 NAEP (under a National Art Education Foundation grant; reported out by Siegesmund and Diket 2003). Two points became apparent: the NAEP rubric for the block did not give students any credit for changing the problem (i.e., creative reseating of the problem by the students), and the rubric could not credit symbolic or expressive solutions that were not based in direct observation of the given images. Siegesmund and Diket presented these findings together at the NAEA conference in 2003, suggesting expansions to rubrics that teachers might want to use in evaluating student self-portraiture. In particular, they recommended crediting solutions that were original and appropriate to portraiture as a process and devised coding and credit for symbolic or expressive solutions.</p> <p>Many different ideas stream into the NAEP "vision" from teachers, school administrators, arts organizations and agencies, field researchers, university personnel, and NAEP officials. However, ideas find their way into the assessment through the specific representatives invited by the NAEP to participate; those with the power an invitation provides can control the philosophy informing the assessment. When ideas are framed at a consensual level among those people at the table (i.e., College Board 1994), the NAEP contractor takes the overall concept and interprets its intent in a statistically defensible manner. The NAEP is piloted and then proceeds to national data collection. Upon the completion of the primary statistical analysis, an NAEP report is produced for Congress and then is released as a document to the public. With every step, the NAEP moves farther and farther away from the classroom and thus becomes more abstracted from the reality of most individual teachers. No alternative process may exist, but this reality has deep ramifications for what the results mean to art teachers in the classroom. For the NAEP to be useful, art teachers must examine their practice at its philosophic roots and recognize parallels in the examples used in the NAEP. When this reflection is multiplied in thousands of teachers, it is hard to conceive of consensus on such a grand scale. As we go forward, we must look for greater consensus about what is essential in artistic understanding. Achieving such consensus could improve the NAEP and would assist teachers.</p> <hd id="AN0055310780-12">Presupposition and Supposition</hd> <p>We take <emph>presupposition</emph> to mean that which is prior to knowledge and used for the purpose of argument and <emph>supposition</emph> as a message that expresses an opinion based on incomplete evidence. For example, although art teachers could have administered the NAEP in their schools, the NAEP contractor decided to bring in facilitators "knowledgeable about students, classrooms, and the visual arts" to administer the exercises ([<reflink idref="bib22" id="ref47">22</reflink>], 26; also see NAGB 2008). Critical theorists might interpret this choice as implying that art teachers would not be objective in administering the test or might alter the intent of the assessment toward individual expression and creative responses they preferred. Alternately, a critical theorist could argue that experimental protocol prefers facilitators who are trained, impartial, and stick to a script. We cannot clarify the matter of presupposition in NAEP documents, although we can find indicative explanations separately in the NAEP sample exercise booklet ([<reflink idref="bib30" id="ref48">30</reflink>]). This booklet leads us to assume that the decision was pragmatic, and that the contractor favored images of objectivity and continuity, as well as consistency with other NAEP subject areas. Otherwise, the implied presupposition would be that art teachers were not deemed trustworthy to administer the test. Ultimately, however, we can only surmise about the contractor's decisions. Some art teachers may have felt left out of the process of national testing, having taught at least some of the thirty or so eighth graders selected for testing at their school. At the very least, they might have wanted to know that general facilitators, rather than specialist subject-area teachers, administered NAEPs for all the subject areas. Before jumping to conclusions, critics need to understand the distinction between presupposition and supposition.</p> <p>Persky's ([<reflink idref="bib23" id="ref49">23</reflink>]) review of the NAEP provided the final points in this analysis of the presuppositions and suppositions underlying the NAEP. Her article was based on the 1997 NAEP Arts Assessment and provided recommendations for the 2008 Arts Assessment. She provides an overview of the 1997 test construction, with recommendations for further studies and actions. We now compare her recommendations with the 2008 Arts Education Assessment Framework ([<reflink idref="bib22" id="ref50">22</reflink>]) and suggest that they should have been heeded in the formulation of the 2008 NAEP.</p> <p>[<reflink idref="bib23" id="ref51">23</reflink>] writes at length about the philosophy and definitions at play in the NAGB and NAEP development: "Important outcomes that fit best with a general education model for art education are successfully making meaning and being creative as an 'end in itself.' " She supposes that if a sequential and rigorous arts education functions as a critical part of the overall curriculum, and if such an education emphasizes creating, performing, studying, and analyzing works of art, then the appropriate arts assessment should include opportunities for students to analyze, critique, formulate value judgments about works of art, and create and perform works of art. The 1997 and 2008 NAEPs presupposed that looking carefully at artwork produces exploration of meaning and leads to desired forms of expression. Thus, both presuppositions and suppositions reside in the NAEP visual arts framework. Persky indicates that the specifications for the 1997 NAEP suppose that "aesthetic, social-cultural, and historical contexts of the arts [combine] with the knowledge and skills necessary to participate in the arts, [and] shall not be considered as separable" (<reflink idref="bib610" id="ref52">610</reflink>). National assessment affords a rare opportunity to explore both the explicit and the implicit expectations for learning in the visual arts, and to determine which are presuppositions, which are suppositions, and which are empirical findings.</p> <p>The 1997 NAEP Art Assessment drop-in type exercises included a mix and a balance of creating exercises and responding exercises that were intended to engage a wide variety of knowledge and skills in studio production, art criticism, art history, and aesthetics. Students completed paper-and-pencil exercises by describing, analyzing, critiquing, and interpreting works of visual art, and then created two-dimensional works of art linked to the responding exercises. Many of the NAEP tasks were meticulously designed, supposedly to assess student awareness of contemporary art forms and approaches, and to better engage students in thoughtful creation when they were asked to draw in response.</p> <p>A major question for both assessments raised by Persky was whether the instructions were too detailed for eighth-grade students. For the 1997 NAEP Art, the arts committee members and Educational Testing Services staff chose detailed, explicit directions for both content reasons and the purpose of consistent scoring. It turns out that the scoring in both the 1997 and 2008 guides for written responses was far less complex than the scoring for evaluating art products. Persky mentioned this conundrum frequently throughout her article. One significant suggestion she made was that by intensifying the holistic adjudication approaches to tasks and scoring, a vital component of the next NAEP might be made even more successful. Persky confessed that it might be simplest to duck the challenges posed by the making and scoring of the 3-D performance; in fact, the 2008 test makers did wisely avoid the problem by asking students to design a bicycle carrier but without asking them to build it. Issues about the time given for tasks were cited several times, and, admittedly, students assessed in 1997 had a wide range of art backgrounds. Familiarity with art making would be an advantage in creating a product, while lack of experience in art making would increase the amount of time a student needed to even initiate the creation of a given product. Some students reported quite substantial arts experience, gained either in or outside of school. Persky noted that many people in the field were critical of some of the NAEP's approaches. Reactions to these insights (as devil's advocate) indicate that the NAEP considers critical analyses as starting points for future assessment designs but does not always heed them. Perhaps their presupposition is that the NAEP knows best, or, adapting to the supposition that the art assessment demands parity with other disciplines, perhaps they assume that looking like other NAEPs is more important than dealing with the unique nature of art and art study. Unfortunately, the supposition and reality are that the NAEP has the final say about its own project, even though art educators might wish it to attend more closely to Persky's observations.</p> <hd id="AN0055310780-13">Strong Course of Action</hd> <p>Even minuscule policy decisions and changes cannot escape the influence of money and prestige, and thus it is worthwhile to question who is involved, what decisions are reached, and who does and does not benefit from a policy. Another touchy subject in a field that is so under-funded for research, policy, and programming is the substantial influence of large, external grant-making foundations and government agencies. One only need look back to the development of the National Standards of Visual Arts and the many iterations of various state standards and the frameworks and construction for the 1997 and 2008 NAEP Art Assessment to see that the Getty Foundation's interest in a particular definition of the disciplined study of art exerted a major humanities-oriented impact on visual art education policy development, pre-K–12. One can also easily recognize that the National Endowment for the Arts, the Arts Education Partnership, and the Kennedy Center for the Arts have promoted certain values for the kinds of arts education centered on artists-in-schools, visiting artists, teaching artists, arts outside of school, audience development, and the use of the arts to teach other subjects. Given such high-profile influences, it is difficult for arts educators to maintain a public policy focus on teaching art rigorously and comprehensively as a curricular subject, despite a broadly recognized need to bolster and support a continuum of art instruction and services for all students throughout our nation's schools. These varied influences are another reason why it is even more important to understand policymakers' backgrounds and determine whether committee or board memberships represent current expertise and specializations within art education. The composition of the NAGB (twenty-five members) is determined by the Education Sciences Reform Act (PL 107-279). Aside from this board's array of political and administrative appointees, there are three classroom teachers, two curriculum specialists, and four representatives from the general public. Of the fifty NAGB members at the time of the 1997 and 2008 NAEP <emph>Arts Report Card</emph> releases, only one member could be identified as a visual arts researcher and advocate. Importantly, a senior-level NAEP administrator who supported the arts as a learning domain was present, but where did the content knowledge for the arts assessment come from?</p> <p>Another point worth noting is the ongoing work of thousands of professional art educators that is conducted on a much broader but less visible scale than the highly publicized projects funded by foundations and government agencies. For example, [<reflink idref="bib25" id="ref53">25</reflink>], president-elect of the NAEA, surveyed art educators about the effects they noted during the years of NCLB. What and how policies are and are not being presented, discussed, and implemented may be historical considerations, but three policy decisions seem especially relevant and ought to be prominent in our discussion of the NAEP and visual art: the content of the assessment blocks, the relationship of items on the NAEP to school curriculum, and the generalization of achievement across students who are taking or have taken art at school and students who have not taken art in school during the assessment year.</p> <hd id="AN0055310780-14">Content of Assessment Blocks</hd> <p>The NAEP policy is to try an array of assessment blocks that are intended to distinguish general abilities in the subject before selecting specific blocks for national testing. Blocks are designed to work in conjunction with student and school questionnaires and determine the presence of opportunities: "to see, hear, touch, and understand the accumulated wisdom of our artistic heritage," and "to make [the student's] own contributions through productions and performances" (NAGB 2008, 3). These descriptions coincide with the terms "responding" and "creating" in the NAEP framework document ([<reflink idref="bib29" id="ref54">29</reflink>]). In the 2008 NAEPs in visual art and music, parallel projects are bundled in each assessment. Assessment blocks enable both longitudinal trend analysis, comparing achievement on the 1997 NAEP to achievement in the same grade eleven years later, and item-analysis assessments of various models of arts learning that were expanded in 2008 to include aesthetics and exploration of social, cultural, and historical contexts of art. From the released information in the official report, however, our field cannot be sure of the extent to which these two goals have been reached. Skills not considered to be separable are reported "as a whole," rather than as isolated elements of component knowledge or technical skills. The NAEP framers note the following:</p> <p>Many art educators worry that an assessment of the arts will artificially quantify those essential aspects of the arts that seem unquantifiable—inspiration, imagination, and creativity. This framework has been designed to honor the essential aspects of the arts as much as is compatible with the constraints of funding and time available in schools for the NAEP assessment. (NAGB 2008, 4)</p> <p>The assessment blocks variously used in 1997 and 2008 address students' awareness of art history, portraiture as personal expression, product design, cultural awareness, and societal issues. Other blocks query the relationship of memory to the ability to foster imaginative solutions in art, three-dimensional transformations in clay, and spatial understanding. As previously noted, comparisons between 1997 and 2008 remain elusive, and block content remains controversial. The result is value without clarity.</p> <hd id="AN0055310780-15">Relationship of Items to School Curricula</hd> <p>The decision to assess music and visual art separately was upheld in the 2008 assessment. In 1997, dance and theater were piloted as assessment areas, and music and art were assessed with sufficient student participants. On both occasions, the test makers wanted assessment blocks in music and the visual arts to engage students, regardless of their exposure to the art form as a school subject. At the same time, a caveat was put in place: the blocks must discriminate student levels on achievement criteria. The general presentation of items appears congruent with one kind of authentic assessment in art. The blocks elicit artistic reflection by students and require scoring by an external jury or panel using specific criteria. Each assessment block consists of related items and exercises clustered around a central theme or activity. Items are presented in a variety of formats—multiple choice, constructed response, or production. When an NAEP item is revealed to be too difficult, two causes are possible: students lack fundamental understanding of art concepts, or they cannot adequately reformulate the "problem" to relate it to what they studied at school or initiated in other artwork they had produced. In summary, it could be said that items in the blocks are authentic but decontextualized from students' prior experiences by scoring procedures. This condition follows logically from a fact mentioned earlier: general consensus does not translate into identical classroom opportunity or instruction in specific content.</p> <p>The measurement of achievement in visual art combines scores for students who are currently enrolled or have taken art at school during the assessment year with the scores of those who have not taken art in school during the assessment year. There is an explicit expectation that art can be learned as a subject in formal school contexts (supposition), and that some visual art knowledge may be accrued through participation in the general culture (presupposition). NAEP student questionnaires include a general school questionnaire and a questionnaire for students taking visual arts in the assessment year. These documents provide information about aptitude and attitudes in visual art, specifics of in-school media availability, pedagogical delivery, and student perceptions of teacher attributes. Participation in out-of-school settings is queried as well.</p> <hd id="AN0055310780-16">Teachers as Active Policymakers and Critics</hd> <p></p> <hd id="AN0055310780-17">Evaluations as Public Relations Tools</hd> <p>Decisions by the art education profession are critical in the development of understanding, influence, and dissemination about the 2008 NAEP Art Assessment. The NAEA sees the NAEP Arts Assessment as an opportunity to learn as much as possible from national testing (i.e., Diket 2010; Sabol 2010), with the potential for production of secondary analyses to inform the field. In the past, art education as a field has continued on its way as the NAEP has appeared and then disappeared. Unfortunately, the field has not fully benefited from some of the keen insights that secondary analysis can extract. Given the nature of art subject matter, the extent of field consensus, current conditions, and the NAEP itself, what is a reasonable policy position toward the NAEP?</p> <p>Policy agendas are rarely grassroots efforts by teachers clamoring for more standards, assessments, and administrative procedures, and they are are often dismissed as top-down, irrelevant actions. <emph>The NAEP is what it is, and no more.</emph> As a large-scale national assessment, the NAEP can accomplish certain goals that no other assessment can accomplish. It has a special role to play in its ability to define and refine knowledge and experience in the arts extracted from a rich and diverse array of possibilities. But the NAEP cannot and should not be the sole assessment of visual art education. Nor should it be thought of as the standard-by-standard measuring instrument for either the voluntary National Standards or state replications of those standards. To achieve a state, regional, or even national picture, many of these standards need to be examined in other assessment formats, more frequently and over a longer duration than is possible with the NAEP. However, the NAEP is one significant and unique measure alongside important work being conducted in many states, universities, private organizations, local districts, and schools (NAGB 2008, 7). In other words, the NAEP is not a substitute for regular and continuous local assessment in the classroom. Instead, the NAEP ought to be one of many measures that we regularly consult while improving education in the visual arts.</p> <p>Practitioners ought to be critical and constructive players in policymaking, develop strong courses of action, turn agendas and policy into practice, and be active policymakers and policy critics in whatever capacity in which they serve. In this article, we take Hamblen's recommendations to heart and believe that the field of art education should too. We want to understand and critique the policies and foundations of the NAEP <emph>Arts Report Card</emph>, and to use that critique to participate fully in policy development and a fostering of practices that will improve art teaching and learning. By digging into and not blindly rebuking or discounting the significant efforts of the NAEP, we can become more effective art education policymakers. We must keep the NAEP in perspective and use its findings thoughtfully as we work for significant improvements in various aspects of art education based on what we have learned from theorists, practitioners, researchers, teacher educators, and teachers in the field, and, most of all, from art itself. Perhaps most importantly, we need to teach future art educators how to interpret data from various sources and use the findings to effect positive policy change so that more students have the opportunity to study art rigorously in the P–12 years.</p> <p>For example, the NAEP arts education framework published in 2008 envisions a society that believes the arts are essential to every child's complete development. Throughout students' lives, they should be able to draw from artistic experience and knowledge as a means of understanding what happens both inside and outside their own skin, in the same way that they use mathematical, scientific, historical, and other frameworks for understanding. Our society does not expect most students to become professional artists any more than it expects high school math students to become professional mathematicians. Instead, the expectation is that all students will experience enough of the discipline, challenge, and joy of creation in different art forms to intimately understand the human significance of dance, music, theater, and the visual arts (NAGB 2008).</p> <p>The Visual Arts Consortium previously mentioned currently plans to look closely at the potential impact of the NCLB assessment environment on eighth grade students' participation in visual arts coursework and their opportunity to find meaning and utility in arts contexts. What is the current impact of the NAEP on overall school culture as a climate for learning? For example, how are at-risk groups being affected by this legislation? Are they encouraged by NAEP findings to take art in schools when coursework is available? Our consortium will be looking closely at NAEP task blocks in these and other regards and working with associated colleagues who are preparing to replicate blocks and examine curricular trends in art for their states.</p> <p>Results from the U.S. Department of Education 2008 <emph>Arts Report Card</emph> stated that the availability of arts instruction has remained steady for over a decade, although significant racial/ethnic, gender, and socioeconomic gaps are evident in both music and visual arts for eighth graders. In 2008, 57 percent of eighth-graders attended schools where music instruction was offered at least three or four times a week, and 47 percent attended schools where visual arts instruction was offered at least as often. Access to both music and visual arts instruction did not differ significantly by race/ethnicity, gender, or eligibility for free or reduced-price school lunch. However, the average responding scores in music and visual arts on the 2008 assessment were twenty-two to thirty-two points higher for white and Asian/Pacific Islander students than for black and Hispanic students. Female students outperformed male students in both assessment areas, scoring ten points higher in music and eleven points higher in visual arts. Students eligible for free or reduced-price lunch scored, on average, twenty-eight points lower in music and twenty-nine points lower in visual arts than non-eligible students. These findings deserve a closer look using restricted data.</p> <hd id="AN0055310780-18">CONCLUSIONS AND POLICY RECOMMENDATIONS</hd> <p>Previous examples show that the visual arts education community can work the policy "podium" afforded by the 2008 NAEP. The 2008 <emph>Arts Report Card</emph>, the "tail of the assessment tiger" should be used judiciously to expand national awareness of artistic thinking as essential in the education of all American school children. NAEPs over the years reflect changing expectations in the visual arts community and, taken together, project a growing estimation of capabilities when students are given an opportunity to explore and study art.</p> <hd id="AN0055310780-19">Educational Synergies</hd> <p>The visual arts community works best in synergy, with stakeholders at all levels working to ensure the efficacy and availability of educational programs for America's youth. For example, the American Association of Museums includes as one of its goals the "rebuilding" of social and civic connections (George Bassi, Lauren Rogers Museum of Art director, informal conversation). Potential synergy partners include museums, government, corporate participants, non-profits, arts organizations, and schools. How does synergy work to benefit American school youth? The 2008 NAEP <emph>Arts Report Card</emph> indicates that opportunities for schools to visit museums have been seriously curtailed by economic cutbacks in education budgets. Extrapolating from the same report, the lack of school field trip funding may have been countered in part by teachers' increased Internet access to museum holdings and museum-sponsored opportunities to continue dialogues in virtual spaces. Thus, teachers may be "partnering" virtually by using museum Internet resources at school. Moreover, museums can ask patrons—individual, civic, and corporate—to include supplements to donations that support transportation costs associated with bringing students back into museums. Reciprocally, museums can lend their local prestige and resources to public and private school art programs. The effect on artistic achievement might be substantial if social, civic, and educational resources could better align to achieve common goals. Obviously, synergy makes it more difficult to untangle partners' contributions to achievement scores, but the various stakeholders must ask if such credit really matters as long as American students become more literate visually.</p> <p>As next steps on synergy, we recommend:</p> <p></p> <ulist> <item> • Art fields should exhibit sincere interest in exploring the NAEPs as a source of information about visual art within and outside of America's schools.</item> <p></p> <item> • Art educators should participate <emph>en masse</emph> in survey queries about their teaching practices and share and discuss survey findings with other stakeholders.</item> <p></p> <item> • Teachers should examine their core beliefs, values, and goals for visual art in education.</item> <p></p> <item> • Art teachers should move from a stance of "protecting their turf" to the position of being "best equipped to explore the richness and complexity of art" within an education, and not apart from general educational objectives.</item> </ulist> <hd id="AN0055310780-20">Content</hd> <p>Prior to the availability of tools and released question blocks on the NAEP Web site that made the assessment more visible to stakeholders, and prior to NCLB, which intended the NAEP to inform policy, examinations of various subject areas in the report cards focused more on demographic variances than on subject proficiency. Currently, the focus is placed on growth paths in students' ability to think in and about subject areas. The NAEP has traditionally placed emphasis on decoding and critical thinking skills and literacy in and across subject areas toward the goal of developing a literate and capable citizenry. The standards movement recalibrated the lens and ramped up the stakes for subject areas, including but not featuring visual art. In art, student capabilities studied at the national level do not presuppose particular curricular choices; rather, the NAEP illuminates students' familiarity with various approaches to critical and historical analyses and their awareness of personal decision paths in art making. In the visual arts, strategies for analysis in responding and creating rely upon students' general knowledge of composition, art media, historical styles, spatial awareness, and artistic notation. As this analysis shows, content and correlation with the study of art in schools are remaining challenges for NAEP.</p> <p>As next steps on content, we recommend:</p> <p></p> <ulist> <item> • Examine reports about what content art teachers say is customarily taught in eighth grade (as may be shown in part by fast-response surveys distributed nationally in 2009).</item> <p></p> <item> • Use restricted data from student questionnaires related to what experiences they have had with art and correlate those indicators with students' ability to answer similar items within blocks.</item> <p></p> <item> • Examine the base understanding that students need to have in order to answer more difficult, constructed response questions.</item> <p></p> <item> • Implement replications of NAEP art blocks in schools and closely examine the effect of context on student performance.</item> <p></p> <item> • Examine state frameworks in the visual arts and determine the overall relationship to the NAEP content.</item> <p></p> <item> • Compare state findings for visual arts testing to national reports—how achievement in visual arts is described and computed in various testing situations.</item> </ulist> <hd id="AN0055310780-21">Using the Report and Taking Another Look</hd> <p>The 2008 NAEP Arts Assessment stands as an important but incomplete "report card" on the transmission of artistic culture. The fact of its national administration makes a positive pronouncement about the worth of artistic understanding to America's youth. However, statistical presentations of results always require careful and thoughtful scrutiny. This is why in-depth policy analysis of the NAEP must continue.</p> <p>As specific steps in using the 2008 NAEP <emph>Arts Report Card</emph> and taking another look at both the assessment and its results, we recommend:</p> <p></p> <ulist> <item> • Teachers and arts administrators should study the 1997 and 2008 NAEP <emph>Arts Report Card</emph> for explicit content.</item> <p></p> <item> • Replication studies funded in 2009 by the National Art Education Foundation should be reported in a timely fashion in peer-reviewed journals.</item> <p></p> <item> • Arts education statisticians with the university consortium should tender proposals for secondary analysis of the 2008 NAEP Arts Assessment's restricted data when the data are made available, and report embedded information back to the field in a timely fashion.</item> </ulist> <p>Given the current reflective protocol of NAGB officers and the transparency of the NAEP in secondary analysis, proponents of the arts in education have the opportunity to inform test makers and modify the decision processes by which youth in America's schools and communities are taught. We need to accomplish these tasks on many levels—at the classroom level and in the community, in state departments of education and arts agencies, and at the national level with governmental agencies and arts organizations. Working together, we can accumulate a body of knowledge that is sufficient for teaching the arts to subsequent generations. However, we need to think beyond sufficiency to efficacy. Artists have shown that they can represent ideas as recognizable and physical forms, and can treat the symptoms of culture with discerning prognoses. The supposition is that non-artists can also recognize and contribute to the dialogue on culture if they have certain basic understandings and insights about art. With the NAEP, we have moved beyond the realm of presupposition to that of supposition. Now we have a chance to move toward substantiated knowledge about what the inclusion of arts in an education contributes to humankind.</p> <hd id="AN0055310780-22">ACKNOWLEDGMENTS</hd> <p>We thank Anne Grey and Dawn Mauldin at UCF for their assistance in the preparation of this manuscript. We would also like to acknowledge the 2008 National Art Education Association NAEP Consortium members who will be involved with the upcoming Secondary Arts Analysis: Read Diket, William Carey University (1997); David Burton, Virginia Commonwealth University (1997); Robert Sabol, Purdue University (1997); Richard Siegesmund, University of Georgia (new); Thomas Brewer, University of Central Florida (new); Laura Chapman, Independent Scholar, (new, consultant); and Deborah Reeve, NAEA Executive Director, ex-officio.</p> <ref id="AN0055310780-23"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Burton, J., Horowitz, R. and Abeles, H.2000. Learning in and through the arts: The question of transfer. Studies in Art Education, 41(3): 228–57.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Chapman, L.1982. Instant art, instant culture: The unspoken policy for American schools, Reston, VA: National Art Education Association.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref45" type="bt">3</bibl> <bibtext> College Board. 1994. Arts education assessment and exercise specifications: Excerpts for the visual arts, Reston, VA: National Art Education Association.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref18" type="bt">4</bibl> <bibtext> Deasy, R.J.2002. Critical links: Learning in the arts and student academic and social development, Washington, DC: Arts Education Partnership.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref46" type="bt">5</bibl> <bibtext> Diket, R.M.2001. A factor analytic model of eighth-grade art learning: Secondary analysis of NAEP arts data. Studies in Art Education, 43(1): 5–17.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref19" type="bt">6</bibl> <bibtext> Diket, R.M.Two sides of a coin. Invited presentation in assessment super session, National Art Education Association Annual Conference. Baltimore, MD. March</bibtext> </blist> <blist> <bibl id="bib7" type="bt">7</bibl> <bibtext> Diket, R. and Brewer, T.NAEP Arts 2008 and what it means to visual art education. Paper presented at the SECAC Conference. New Orleans, LA. September 24–27</bibtext> </blist> <blist> <bibl id="bib8" idref="ref5" type="bt">8</bibl> <bibtext> Diket, R.M., Sabol, F.R. and Burton, D.2001. Implications of the 1997 NAEP visual arts data for policies concerning artistic development in America's schools and communities, CFDA Number 84.902B/Award Number 902B000006Hattiesburg, MS: William Carey University.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref37" type="bt">9</bibl> <bibtext> Diket, R.M., Burton, D., McCollister, S. and Sabol, F.R.2000. Taking another look: Secondary analysis of the NAEP Report Card in the visual arts. Studies in Art Education, 41(3): 202–07.</bibtext> </blist> <blist> <bibtext> Diket, R.M., Burton, R., Sabol, R. and Thorpe, P.The 1997 NAEP Arts considered with new directions for learning assessments. Panel discussion, American Educational Research Association annual meeting. New Orleans. April</bibtext> </blist> <blist> <bibtext> Diket, R.M., Brewer, T.M., Sabol, R., Burton, D. and Seigesmund, R.NAEP Arts 2008: Information for the next decade of visual arts education. Paper presented at the National Art Education Association Conference. Minneapolis, MN. April 17–21</bibtext> </blist> <blist> <bibtext> Eisner, E.W.1999. The national assessment in the visual arts. Arts Education Policy Review, 100(6): 16–20.</bibtext> </blist> <blist> <bibtext> Efland, A.1976. The school art style: A functional analysis. Studies in Art Education, 17(2): 37–44.</bibtext> </blist> <blist> <bibtext> Fiske, E.B.1999. Champions of change: The impact of the arts on learning, Washington, DC: President's Committee on the Arts and Humanities.</bibtext> </blist> <blist> <bibtext> Hamblen, K.1995. Beyond the public face of policymaking. Arts Education Policy Review, 96(3): 28–33.</bibtext> </blist> <blist> <bibtext> Jones, LyleV. and Olkin, I.2004. The nation's report card: Evolution and perspectives, Bloomington, IN: Phi Delta Kappa Educational Foundation.</bibtext> </blist> <blist> <bibtext> Keiper, S., Sandene, B.A., Persky, H.R. and Kuang, M.2009. The nation's report card: Arts 2008 music and visual arts, Washington, DC: National Center for Education Statistics, Institute of Education Sciences. NCES 2009–488. <ulink href="http://nationsreportcard.gov/arts%5f2008/">http://nationsreportcard.gov/arts%5f2008/</ulink></bibtext> </blist> <blist> <bibtext> Lehmann, I.J.2004. "The genesis of NAEP". In The nation's report card: Evolution and perspectives, Edited by: Jones, LyleV. and Olkin, Ingram. 25–92. Bloomington, IN: Phi Delta Kappa Educational Foundation.</bibtext> </blist> <blist> <bibtext> Linn, R.2001. The influence of external evaluations on the National Assessment of Educational Progress, CSE Technical Report. Doc. no. ED457243.</bibtext> </blist> <blist> <bibtext> Music Educators National Conference (MENC). 1994. National standards for arts education: What every young American should know and be able to do in the arts, Reston, VA: Music Educators National Conference.</bibtext> </blist> <blist> <bibtext> National Assessment Governing Board (NAGB). 1994. Arts education assessment framework, Washington, DC: National Assessment Governing Board. Doc. no. ED426957</bibtext> </blist> <blist> <bibtext> National Assessment Governing Board (NAGB). 2008. 2008 arts education assessment framework, Washington, DC: National Assessment Governing Board. Doc. no. ED502927</bibtext> </blist> <blist> <bibtext> Persky, H.2004. "The NAEP arts assessment: Pushing the boundaries of large-scale performance assessment". In Handbook of research and policy in art education, Edited by: Eisner, E. and Day, M.607–35. Mahwah, NJ: Erlbaum.</bibtext> </blist> <blist> <bibtext> Persky, H.R., Sandene, B.A and Askew, J.M.1998. The NAEP 1997 arts report card: Eighth-grade findings from the National Assessment of Educational Progress, Washington, DC: National Center for Education Statistics.</bibtext> </blist> <blist> <bibtext> Sabol, F.R.2010. No Child Left Behind: A study of its impact on art education, Reston, VA: National Art Education Association. <ulink href="http://www.arteducators.org/research/NCLB%5fSummary%5fof%5fFindings%5f2&amp;#8211;10%5f3.pdf">http://www.arteducators.org/research/NCLB%5fSummary%5fof%5fFindings%5f2&amp;#8211;10%5f3.pdf</ulink></bibtext> </blist> <blist> <bibtext> Siegesmund, R. and Diket, R.M.Using NAEP in the classroom. Presented at the National Art Education Association annual conference. Minneapolis, MN. April</bibtext> </blist> <blist> <bibtext> Siegesmund, R., Diket, R. and McColloch, S.2001. Re-visioning NAEP: Amending a performance assessment for middle school students. Studies in Art Education, 43(1): 45–56.</bibtext> </blist> <blist> <bibtext> Stankiewicz, M.A.1999. Spinning the arts NAEP. Arts Education Policy Review, 100(1): 29–32.</bibtext> </blist> <blist> <bibtext> U.S. Department of Education. 2008a. NAEP arts education framework projectAvailable at <ulink href="http://www.nagb.org/publications/framework/arts-framework08.pdf">http://www.nagb.org/publications/framework/arts-framework08.pdf</ulink></bibtext> </blist> <blist> <bibtext> U.S. Department of Education. 2008b. The NAEP 2008 arts assessment: Music and visual arts bookletAvailable at <ulink href="http://nces.ed.gov/nationsreportcard/pdf/demo%5fbooklet/08-SQ-Arts.pdf">http://nces.ed.gov/nationsreportcard/pdf/demo%5fbooklet/08-SQ-Arts.pdf</ulink></bibtext> </blist> <blist> <bibtext> Winner, E. and Hetland, L.2001. The arts and academic improvement: What the evidence shows. Executive summary. Translations, 10(1)</bibtext> </blist> </ref> <aug> <p>By ReadM. Diket and ThomasM. Brewer</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib13" firstref="ref3"></nolink> <nolink nlid="nl2" bibid="bib15" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib17" firstref="ref6"></nolink> <nolink nlid="nl4" bibid="bib18" firstref="ref7"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref10"></nolink> <nolink nlid="nl6" bibid="bib21" firstref="ref17"></nolink> <nolink nlid="nl7" bibid="bib12" firstref="ref20"></nolink> <nolink nlid="nl8" bibid="bib28" firstref="ref21"></nolink> <nolink nlid="nl9" bibid="bib19" firstref="ref22"></nolink> <nolink nlid="nl10" bibid="bib22" firstref="ref26"></nolink> <nolink nlid="nl11" bibid="bib20" firstref="ref27"></nolink> <nolink nlid="nl12" bibid="bib31" firstref="ref29"></nolink> <nolink nlid="nl13" bibid="bib29" firstref="ref33"></nolink> <nolink nlid="nl14" bibid="bib24" firstref="ref42"></nolink> <nolink nlid="nl15" bibid="bib30" firstref="ref48"></nolink> <nolink nlid="nl16" bibid="bib23" firstref="ref49"></nolink> <nolink nlid="nl17" bibid="bib610" firstref="ref52"></nolink> <nolink nlid="nl18" bibid="bib25" firstref="ref53"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ906018 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: NAEP and Policy: Chasing the Tail of the Assessment Tiger – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Diket%2C+Read+M%2E%22">Diket, Read M.</searchLink><br /><searchLink fieldCode="AR" term="%22Brewer%2C+Thomas+M%2E%22">Brewer, Thomas M.</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Arts+Education+Policy+Review%22"><i>Arts Education Policy Review</i></searchLink>. 2011 112(1):35-47. – Name: Avail Label: Availability Group: Avail Data: Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 13 – Name: DatePubCY Label: Publication Date Group: Date Data: 2011 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Evaluative – Name: Audience Label: Education Level Group: Audnce Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink> – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Visual+Arts%22">Visual Arts</searchLink><br /><searchLink fieldCode="DE" term="%22Art+Education%22">Art Education</searchLink><br /><searchLink fieldCode="DE" term="%22National+Competency+Tests%22">National Competency Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Policy%22">Educational Policy</searchLink><br /><searchLink fieldCode="DE" term="%22Policy+Analysis%22">Policy Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Politics+of+Education%22">Politics of Education</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Role%22">Teacher Role</searchLink> – Name: SubjectThesaurus Label: Assessment and Survey Identifiers Group: Su Data: <searchLink fieldCode="SU" term="%22National+Assessment+of+Educational+Progress%22">National Assessment of Educational Progress</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1080/10632913.2011.518126 – Name: ISSN Label: ISSN Group: ISSN Data: 1063-2913 – Name: Abstract Label: Abstract Group: Ab Data: This paper grabs hold of the "assessment tiger" by considering the history of the National Assessment of Educational Progress (NAEP) "Arts Report Cards" for the visual arts, which were constructed and have been administered four times within thirty-five years. Two purposes of the NAEP have persisted since its founding: (1) measuring learning and (2) developing expectations for change and reform in education from a national platform. These purposes provided the basis for a functional analysis derived from Efland (1976) that looked closely at the NAEP's collaborative context, consensus building, and block content. Hamblen's 1995 critical theory analysis exemplar also serves as a means for reviewing the politics of assessment, influences on policy directions, and curricular emphases within a subject area. Using Persky's 2004 critical evaluation of the 1997 NAEP Arts Assessment, this article considers recommendations she derived from the 1997 NAEP "Arts Report Card" that guided the 2008 NAEP construction and administration. The 2008 NAEP "Arts Report Card," released in June 2009, informs a number of initial interpretations, recommendations, implications, and applications for those who want to "own" the tiger. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: Ref Label: Number of References Group: RefInfo Data: 31 – Name: DateEntry Label: Entry Date Group: Date Data: 2010 – Name: AN Label: Accession Number Group: ID Data: EJ906018 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ906018 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1080/10632913.2011.518126 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 13 StartPage: 35 Subjects: – SubjectFull: Visual Arts Type: general – SubjectFull: Art Education Type: general – SubjectFull: National Competency Tests Type: general – SubjectFull: Educational Policy Type: general – SubjectFull: Policy Analysis Type: general – SubjectFull: Politics of Education Type: general – SubjectFull: Educational Assessment Type: general – SubjectFull: Teacher Role Type: general – SubjectFull: National Assessment of Educational Progress Type: general Titles: – TitleFull: NAEP and Policy: Chasing the Tail of the Assessment Tiger Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Diket, Read M. – PersonEntity: Name: NameFull: Brewer, Thomas M. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2011 Identifiers: – Type: issn-print Value: 1063-2913 Numbering: – Type: volume Value: 112 – Type: issue Value: 1 Titles: – TitleFull: Arts Education Policy Review Type: main |
| ResultId | 1 |
