View in EDS HTML Full Text PDF Full Text

Evolving Educational Testing to Meet Students' Needs: Design-in-Real-Time Assessment

Saved in:
Bibliographic Details
Title: Evolving Educational Testing to Meet Students' Needs: Design-in-Real-Time Assessment
Language: English
Authors: Stephen G. Sireci, Javier Suárez-Álvarez, April L. Zenisky, Maria Elena Oliveri
Source: Educational Measurement: Issues and Practice. 2024 43(4):112-118.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 7
Publication Date: 2024
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305N210031
Document Type: Journal Articles
Reports - Descriptive
Education Level: Adult Education
Descriptors: Educational Assessment, Student Needs, Test Format, Test Construction, Evaluation Methods, Adaptive Testing, Culture Fair Tests, Educational Innovation, Preferences, Standards, Student Characteristics, Test Selection, Computer Assisted Testing, Adult Students, Academic Aspiration, Occupational Aspiration, Literacy, Numeracy, Delivery Systems, Individual Needs
DOI: 10.1111/emip.12653
ISSN: 0731-1745
1745-3992
Abstract: The goal in personalized assessment is to best fit the needs of each individual test taker, given the assessment purposes. Design-in-Real-Time (DIRTy) assessment reflects the progressive evolution in testing from a single test, to an adaptive test, to an adaptive assessment "system." In this article, we lay the foundation for DIRTy assessment and illustrate how it meets the complex needs of each individual learner. The assessment framework incorporates culturally responsive assessment principles, thus making it innovative with respect to both technology and equity. Key aspects are (a) assessment building blocks called "assessment task modules" (ATMs) linked to multiple content standards and skill domains, (b) gathering information on test takers' characteristics and preferences and using this information to improve their testing experience, and (c) selecting, modifying, and compiling ATMs to create a personalized test that best meets the needs of the testing purpose and individual test taker.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2025
Accession Number: EJ1456429
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHDEUP8UM2Q3OOi4YqJRsVqAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDME7G9JDkQfvTRYRXgIBEICBm4t22tjz2mMximjB-DbSddsTRKy9C4q56B3SWd7E3NIADPk8jxYAT3s1zDKsB-1XBH0etpqSk7JY3aZgiYCpUbyUb4tFIXpfcqETiUiaH_0VIgYCizIYkZ5J-yEWCfqO6bY9836Nq6GJY0uxk1doHodCzaBsQrqTVJey9qEobahRtRTSI-jiPMIvJFa1It6xRH3ZhmrVMX7KvKXb
Text:
  Availability: 1
  Value: <anid>AN0182049108;ems01dec.24;2025Jan07.03:56;v2.2.500</anid> <title id="AN0182049108-1">Evolving Educational Testing to Meet Students' Needs: Design‐in‐Real‐Time Assessment </title> <p>The goal in personalized assessment is to best fit the needs of each individual test taker, given the assessment purposes. Design‐In‐Real‐Time (DIRTy) assessment reflects the progressive evolution in testing from a single test, to an adaptive test, to an adaptive assessment system. In this article, we lay the foundation for DIRTy assessment and illustrate how it meets the complex needs of each individual learner. The assessment framework incorporates culturally responsive assessment principles, thus making it innovative with respect to both technology and equity. Key aspects are (a) assessment building blocks called "assessment task modules" (ATMs) linked to multiple content standards and skill domains, (b) gathering information on test takers' characteristics and preferences and using this information to improve their testing experience, and (c) selecting, modifying, and compiling ATMs to create a personalized test that best meets the needs of the testing purpose and individual test taker.</p> <p>Keywords: adaptive testing; assessment technology; culturally responsive assessment; personalized assessment; test development</p> <p>The shift[<reflink idref="bib1" id="ref1">1</reflink>] from traditional paper‐based tests to computerized‐adaptive testing (CAT) represented a significant innovation in educational assessment during the 20th century (Zenisky & Sireci, [<reflink idref="bib48" id="ref2">48</reflink>]). CATs revolutionized assessment practices with their efficiency (reduced testing time) and improved accuracy. In this 21st century, personalized assessments may represent the next great opportunity for innovation. Personalized assessments seek to integrate "personal characteristics in the assessment design and administration, measurement conditions, scoring procedures, and interpretation and use of test scores" (Buzick et al., [<reflink idref="bib7" id="ref3">7</reflink>], p. 1). In this article, we outline an operationalization of personalized assessment called design‐in‐real‐time (DIRTy) assessment, which will be used with the Adult Skills Assessment Program (ASAP).</p> <hd id="AN0182049108-2">The Changing Educational Assessment Landscape</hd> <p>In the first two decades of this 21st century, we have seen more technology‐based assessment innovation than in the previous two centuries combined (International Test Commission & Association of Test Publishers, [<reflink idref="bib18" id="ref4">18</reflink>]; von Davier et al., [<reflink idref="bib40" id="ref5">40</reflink>]). Test content generation is increasingly automatized, enhancing test development processes that have been traditionally manual and costly (Attali et al., [<reflink idref="bib4" id="ref6">4</reflink>]; Gierl & Haladyna, [<reflink idref="bib13" id="ref7">13</reflink>]). Automated scoring of constructed responses that once required human scoring is increasingly common (von Davier et al., [<reflink idref="bib41" id="ref8">41</reflink>]; Yamamoto et al., [<reflink idref="bib45" id="ref9">45</reflink>]). In addition, digital assessments can provide log (process) data that can be used to better understand test takers' cognitive processes (Araneda et al., [<reflink idref="bib3" id="ref10">3</reflink>]; He et al., [<reflink idref="bib16" id="ref11">16</reflink>]; Ulitzsch et al., [<reflink idref="bib38" id="ref12">38</reflink>]). Process data can also be used to measure the degree to which test takers engage with tasks and can be used to inform estimation of their proficiencies (Pohl et al., [<reflink idref="bib27" id="ref13">27</reflink>]; Wise et al., [<reflink idref="bib44" id="ref14">44</reflink>]).</p> <p>Another hallmark of 21st‐century educational assessment is an awareness of the great individual and cultural diversity within the populations of test takers we test. Recent U.S. Census data highlight a shift from a relatively homogeneous white majority in the 20th century to a racially mixed majority in the 21st century (Vespa et al., [<reflink idref="bib39" id="ref15">39</reflink>]). Testing agencies, governments, and research organizations increasingly recognize the necessity of developing assessments that are more socioculturally responsive and better represent traditionally minoritized populations (Dixon‐Román, [<reflink idref="bib9" id="ref16">9</reflink>]; Lyons et al., [<reflink idref="bib22" id="ref17">22</reflink>]; Mislevy, [<reflink idref="bib23" id="ref18">23</reflink>]; Randall, [<reflink idref="bib29" id="ref19">29</reflink>]; Randall et al., [<reflink idref="bib30" id="ref20">30</reflink>]). In response to these changes, new assessments strive to avoid white‐centric development and administration orientations to represent better the increasingly diverse population of students.</p> <p>Current technology makes it possible to not just adapt assessments to students in terms of item difficulty, as is common in computerized‐adaptive testing (Wainer, [<reflink idref="bib42" id="ref21">42</reflink>]), but also to <emph>personalize</emph> assessments to best meet the needs of <emph>each individual student</emph> and <emph>each individual testing purpose</emph> (Buzick et al., [<reflink idref="bib7" id="ref22">7</reflink>]; Doran et al., [<reflink idref="bib10" id="ref23">10</reflink>]; Sireci, [<reflink idref="bib32" id="ref24">32</reflink>]). Such personalization allows for culturally responsive assessment in the context of an assessment <emph>system</emph> that develops and delivers the most optimal assessment possible to meet the needs of a specific testing situation. Personalized assessment means there is no test form and no pre‐assembled test. Rather, each assessment is <emph>designed‐in‐real time</emph>, and so the adaptation of a test is extended from merely adapting based on item difficulty to adapting based on test takers' personal choices and characteristics, subject to the constraints needed to fulfill specific testing purposes.</p> <p>Rather than developing a test for a single purpose, a DIRTy assessment system draws from banks (or "warehouses") of assessment task modules to assemble and deliver distinctive assessments to support different assessment purposes. The example we use to introduce the concept is the Adult Skills Assessment Program (ASAP), which is a project funded by the Institute of Education Sciences (IES) to develop literacy and numeracy assessments for adult learners in both academic (e.g., adult basic education) and occupational (e.g., workforce development) settings. As subsequently described, the vision of the ASAP includes providing test takers with choices and scaffolds, and incorporates culturally responsive assessment principles via real‐time test assembly.</p> <hd id="AN0182049108-3">DIRTy Assessment: An Overview</hd> <p>CAT is now commonplace in many testing programs, with multistage‐adaptive testing (MST) being one of its most popular versions (Han & Guo, [<reflink idref="bib15" id="ref25">15</reflink>]; Yan et al., [<reflink idref="bib46" id="ref26">46</reflink>]). MST adapts the difficulty of an assessment at the item set (module) level rather than at the item level (Luecht & Sireci, [<reflink idref="bib21" id="ref27">21</reflink>]). MST adaptively routes respondents to one or several preassembled modules based on their performance on previously administered modules (Zenisky & Hambleton, [<reflink idref="bib47" id="ref28">47</reflink>]; Zenisky et al., [<reflink idref="bib49" id="ref29">49</reflink>]). DIRTy assessment incorporates adaptive testing, but adds a test configuration layer that selects modules to best conform to the identified needs of a learner, given a specific assessment purpose. Moreover, the modules are not necessarily static, but rather can be templates that are prompted to generate specific item variations in real time (Foster, [<reflink idref="bib12" id="ref30">12</reflink>]; Luecht, [<reflink idref="bib20" id="ref31">20</reflink>]). In a typical CAT or MST environment, content constraints, item difficulty, and other constraints (item exposure, pool usage, etc.) dictate what items get selected for a test taker as the test progresses. DIRTy assessments generalize the adaptivity to make the assessment adaptive with respect to</p> <p></p> <ulist> <item> testing purpose,</item> <p></p> <item> the sociocultural characteristics of the test taker,</item> <p></p> <item> the test taker's (known) funds of knowledge, and</item> <p></p> <item> the decisions <emph>made by the test taker</emph> before and during the assessment.</item> </ulist> <p>In the next sections, we describe the proposed architecture for the ASAP to illustrate how DIRTy assessment can promote personalized assessment that helps serve learners.</p> <hd id="AN0182049108-4">The Adult Skills Assessment Program</hd> <p>The ASAP is a comprehensive assessment system, which is currently under development, that is designed to support the academic and occupational goals of adult learners. The system involves focusing on adult learners' needs. The theory of action for the ASAP is that by providing information on adult learners' literacy and numeracy skills, we can help guide them to the proper education and training they need to accomplish their academic and career goals. Given that adult learners have both academic and occupational goals, the ASAP bridges adult education and the workplace by linking college and career readiness standards for adult education (Pimentel, [<reflink idref="bib26" id="ref32">26</reflink>]) to job requirements in the O*NET system (Tippins & Hilton, [<reflink idref="bib37" id="ref33">37</reflink>]).</p> <p>The theory of action for the ASAP is depicted in Figure 1. All actions begin with the adult learner (student or employee), who may have been directed to the ASAP by an adult education teacher, a career counselor, or an employer. If sent to the system by a teacher, counselor, or employer, a predefined purpose has been documented. Examples of such purposes include (a) assessing to what extent the student has mastered material recently taught, currently taught, or about to be taught (classroom or diagnostic mode); (b) determining where to best place a learner in an adult education or workplace training program (placement mode), or (c) determining to what extent an employee has the prerequisite skills for a particular job or training program (employment or certification mode). These purposes are not explicitly labeled in Figure 1, but are invoked when the learner experiences the ASAP assessment. That experience includes several layers of choices (e.g., Would you like to practice? Continue? Exit and restart later? Retry the assessment with scaffolds? Choose the assessment context, etc.). Choices presented to test takers are subject to the constraints associated with the testing purpose. For example, restarting an exam with scaffolds may not be allowable for some certification purposes.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/EMS/01dec24/emip12653-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="emip12653-fig-0001.jpg" title="1 ASAP Theory of Action" /> </p> <p></p> <p>Regardless of the choices made by test takers, the goal of DIRTy assessments is to assemble and deliver the best assessment for each individual learner, given the constraints of the testing purpose. Thus, it is <emph>the interaction of the unique test‐taker and the testing purpose that sets the parameters for the test to be assembled in real time</emph>. The conclusion of the assessment is information on the constructs measured by the ASAP—literacy and numeracy. The information is explicitly intended to initiate <emph>action</emph>. For some test takers it may require further interaction with the system until sufficient measurement opportunities warrant a more conclusive action such as a placement, certification, or diagnosis. The information informs the learner and others, depending on the purpose (e.g., teacher, counselor, employer, etc.). The information empowers both the learner and others to initiate action for further learning and employment opportunities.</p> <hd id="AN0182049108-6">ASAP's DIRTy Architecture</hd> <p>Before describing the ASAP architecture, we must point out the ASAP has not yet been built. Thus, what we describe here is our design and roadmap for implementing the ASAP. Perhaps the most succinct way to describe the ASAP is as a combination of (a) a digital warehouse, or repository, of assessment task modules and task templates from which assessments are built; and (b) an "assessment compiler" that selects and delivers these modules to personalize the assessment and fulfill the testing purposes. The modules are referred to as <emph>assessment task modules</emph> (ATMs) that comprise several individual items or item templates.</p> <p>The smallest component of the ASAP is an individual item/item template. An ATM comprises two to seven[<reflink idref="bib2" id="ref34">2</reflink>] items and represents the smallest unit to be delivered to a test taker in the ASAP system. Typically, an assessment will comprise several ATMs before its purpose is fulfilled (i.e., the assessment terminates). The nomenclature of the components in the ASAP architecture is presented in Table 1. The grain size of the system starts with an individual item/item template, then proceeds to the ATM that targets a skill cluster. The skill cluster essentially represents the test specifications for the ATM, <emph>but an additional layer of test specifications is needed for the assessment itself</emph>. The specifications for a specific assessment leverage the information in the warehouse of ATMs to represent the intended content and provide results of sufficient reliability for the assessment purpose. Thus, a specific ASAP assessment involves one or more ATMs, as dictated by the assessment purpose.</p> <p>1 Table Components of the ASAP Architecture</p> <p> <ephtml> <table><thead><tr><th align="left">Component</th><th align="center">Description</th></tr></thead><tbody><tr><td align="left">Item/item template</td><td align="left">Smallest unit in ASAP system—associated with a curriculum standard or other specific knowledge or skill associated with a detailed work activity. Items may be static or dynamic, with the latter being in the form of templates that render in different ways based on examinee characteristics and choices.</td></tr><tr><td align="left">Skill cluster</td><td align="left">A set of skills needed to successfully complete a task that assesses one or more standards or similar knowledge, skills, or abilities. The skill cluster defines the specifications for an assessment task module.</td></tr><tr><td align="left">Assessment task module</td><td align="left">A set of items measuring a skill cluster. All assessment task modules are targeted to skill clusters and so involve at least two items (2–7 items).</td></tr><tr><td align="left">Test specifications</td><td align="left">Test specifications generated from the testing purpose, allowable test taker choices, and information related to personal characteristics of a specific test taker.</td></tr><tr><td align="left">ASAP assessment</td><td align="left">A compilation of assessment task modules assembled to fit testing purpose, learner characteristics, and learner goals.</td></tr></tbody></table> </ephtml> </p> <hd id="AN0182049108-7">Real‐time Test Assembly and Delivery</hd> <p>To accomplish the goals of learner‐centered and culturally responsive assessment, in addition to meeting the psychometric specifications for the purpose of the assessment, ASAP assessments must factor in the unique characteristics and interests of the learner. Characteristics can include funds of knowledge such as multilingualism, cultural capital (music, cuisine, stories, history, norms), familiar digital supports, and interests. For example, test takers can choose to take the assessment within a workplace context (e.g., passages and other stimuli are within an occupational context), a leisure context (e.g., pleasure reading), or an academic context. To achieve such flexibility, the ASAP assessment system will leverage artificial intelligence (AI) to alter the context, given constraints such as linguistic complexity and measurement of specific standards or skills.</p> <p>Incorporating learner characteristics into the ASAP or any other assessment is complex and requires methods for understanding the most important characteristics of interest. Some characteristics will be input into the system as part of the rostering process (e.g., demographic data, workplace information), whereas other information will be input by the learner via a goals, interests, and preferences survey or an "empathy interview" (Nelsestuen & Smith, [<reflink idref="bib25" id="ref35">25</reflink>]). These surveys or interviews would gather important information to improve the assessment such as the test taker's experience with digital technology, proficiency in English and other languages, interests, reasons for interacting with the ASAP, and their goals.</p> <p>The incorporation of test taker characteristics and preferences cannot, of course, be unlimited. The limitations are a function of (a) testing purpose, (b) ATM warehouse depth (i.e., limited availability of contexts), and (c) test specifications related to measurement precision, content representation, security, and other factors. Thus, just like a CAT provides the most optimal test, subject to the constraints such as item bank size, item exposure, and content; DIRTy assessment will provide the most optimal test given the opportunities for personalization that are available in the system. Thus, what DIRTy assessment seeks to optimize extends beyond testing time and measurement precision to optimization of the testing experience (Araneda, [<reflink idref="bib2" id="ref36">2</reflink>]), supporting learning (Gordon, [<reflink idref="bib14" id="ref37">14</reflink>]), and promoting more valid assessment for each individual learner (Buzick et al., [<reflink idref="bib7" id="ref38">7</reflink>]; Sireci, [<reflink idref="bib32" id="ref39">32</reflink>]). The last benefit is due to the design goals focusing on producing the most optimal test for each individual test taker, rather than a specific test form or MST route that is optimal for most test takers.</p> <hd id="AN0182049108-8">The ASAP Delivery System</hd> <p>The components of the ASAP architecture described in Table 1 reflect both measurement units and measurement specifications. There are two levels of measurement unit—items and ATMs. There are also two levels of measurement specifications–skill clusters and DIRTy test specifications. The skill cluster specifies how ATMs are built, while the test specifications dictate the requirements for selecting and configuring the ATMs into a unique assessment experience for each test taker.</p> <p>The coordination of the two levels of measurement units and measurement specifications is handled by an <emph>assessment compiler and delivery system</emph>. This system is illustrated in Figure 2, which illustrates the three inputs that dictate the test to be produced in real time: testing purpose, the test specifications associated with that purpose, and any personalization options. In this example the testing purpose is within the workplace mode and is to certify "reading and understanding safety guides and user manuals." That purpose requires a secure test administration, a test containing 20 items (based on test information/reliability targets), four ATMs associated with that testing purpose, and scaffold level 2, which dictates the types of allowable supports (e.g., online calculator). The personalization options indicate the workplace context is confectionary food preparation in the state of Arkansas. Thus, reading text presented will be framed within that context. These inputs are fed into the assessment compiler, which searches the ATM warehouse to identify and select the ATMs most appropriate for the test taker.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/EMS/01dec24/emip12653-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="emip12653-fig-0002.jpg" title="2 Assessment Compiler and Delivery System" /> </p> <p></p> <p>As can be seen in Figure 2, the rules specified by the testing purpose determine the level of personalization allowable. For most education and training purposes, such as to provide diagnostic information to classroom teachers; all scaffolds, feedback, and learner choices could be activated. For other purposes, some or all of those personalizations could be deactivated.</p> <hd id="AN0182049108-10">DIRTy Assessment Summary</hd> <p>Design‐in‐real‐time assessment involves creating an assessment at the latest possible moment when all conditions of the testing situation are present. Of course, there is a testing purpose that places a test taker in the assessment situation. At that point, any relevant information we know about the test taker, including information they may input into the system at that time, is used to tailor the assessment to the individual. What is new in DIRTy assessment is tailoring the test to multiple, personal factors, and delaying test specifications until the interaction of a test taker and a testing purpose occurs. DIRTy assessment, as we envision it, requires sets of items aligned with both content standards and job tasks that represent the building blocks of a unique instantiation of an assessment, and a system to search and assemble those sets of items (ATMs in our vernacular) to meet the additional configuration goals of personalized assessment.</p> <hd id="AN0182049108-11">Additional Benefits of DIRTy Assessment</hd> <p>Designing tests in real time supports the exciting developments in educational testing that involve the use of tests to help students learn (Bennett, [<reflink idref="bib5" id="ref40">5</reflink>]; Clark & Karvonen, [<reflink idref="bib8" id="ref41">8</reflink>]; Gordon, [<reflink idref="bib14" id="ref42">14</reflink>]). One development is using scaffolds to design the assessment as a learning experience. Another is culturally responsive assessment, in which the content and contexts of items and ATMs can be altered to suit test takers' experiences and choices, or to expose them to a diversity of cultures and contexts outside of their normal realm of experiences.</p> <hd id="AN0182049108-12">Scaffolding to Assessment for Learning</hd> <p>Scaffolds, which are supports and tools test takers can use to solve items, can be incorporated into a DIRTy assessment. For the ASAP, based on the assessment purposes and test taker choices, scaffolds will include accessing different language versions of items, directions, and other assessment material; use of calculators, glossaries, spreadsheets, and other tools; and even search engines and video tutorials. For some purposes, some or all of these scaffolds may be turned off. For diagnostic purposes, after a test taker completes the assessment, they can retake it with the scaffolds turned on. In this way, the ASAP can be used as instruction, in addition to an assessment; and the construct measured may change from literacy or numeracy to <emph>problem solving</emph> (i.e., a learner can now solve a previously unsolved problem with available tools). By using the assessments to enable test takers to solve problems, the assessment becomes a vehicle en route to solving the problem, and hence promotes learning (Clark & Karvonen, [<reflink idref="bib8" id="ref43">8</reflink>]; Gordon, [<reflink idref="bib14" id="ref44">14</reflink>]).</p> <hd id="AN0182049108-13">Culturally Responsive Assessment</hd> <p>Culturally responsive assessment (Hood, [<reflink idref="bib17" id="ref45">17</reflink>]; Montenegro & Jankowski, [<reflink idref="bib24" id="ref46">24</reflink>]) encourages students to draw from their cultural experiences while interacting with an assessment. Walker et al. ([<reflink idref="bib43" id="ref47">43</reflink>]) described culturally responsive assessment as "assessments that take into account the background characteristics of the students; their beliefs, values, and ethics; their lived experiences; and everything that affects how they learn and behave and communicate" (p. 1). By incorporating these personal characteristics into assessment delivery, DIRTy assessment can provide assessments that better match these characteristics. For example, reading passages can reflect the communities in which test takers live, and the test delivery system can allow test takers to access translations of test material while taking the assessment (Questionmark, [<reflink idref="bib28" id="ref48">28</reflink>]). Randall ([<reflink idref="bib29" id="ref49">29</reflink>]) pointed out culturally responsive assessment is inclusive. That is, rather than screening out test material specific to a particular culture, culturally responsive assessment explicitly includes it to conspicuously value that culture. The extension of culturally responsive assessment to anti‐racist assessment (i.e., incorporation of test content that acknowledges racism in our society; Randall, [<reflink idref="bib29" id="ref50">29</reflink>]) can also easily be accomplished within the DIRTy assessment approach.</p> <hd id="AN0182049108-14">Discussion</hd> <p>Over 50 years ago, Frederic Lord laid the foundations for computerized‐adaptive testing (e.g., Lord, [<reflink idref="bib19" id="ref51">19</reflink>]). This revolutionary idea that we could give different assessments to individuals in real time, and score the individuals on the same scale, represented an important innovation. However, it took over a decade for computers to have sufficient processing power to implement the idea on a large‐sale operational program. We are at a similar point in time with respect to the next innovation in assessment—design in real time (DIRTy), personalized assessment. The ideas laid out in this article essentially dictate a research and development agenda. Current technology makes this agenda and the vision underlying it achievable, and work in generalizing computerized‐adaptive testing to adapt based on multiple criteria is already being operationalized (Doran et al., [<reflink idref="bib10" id="ref52">10</reflink>]).</p> <p>Clearly however, obstacles to overcome remain. The system required to support DIRTy assessment has components that all currently exist, but are not yet integrated into a cohesive system. ASAP has the potential to leverage all modern psychometrics and AI has to offer, such as items generated in real time, calibrated independently of examinees, and modules that match test takers' characteristics and assessment purposes. It will require comprehensive item authoring, test delivery, results reporting, and information sharing systems that will need to communicate with each other. It will also require delivery on a variety of devices—and it will have to minimize the digital literacy demands, due to the relatively low digital skills of many adult learners.</p> <p>The field of educational measurement is being challenged to embrace AI‐technological developments while regaining public trust by delivering tests that serve multiple and culturally diverse users (International Test Commission & Association of Test Publishers, [<reflink idref="bib18" id="ref53">18</reflink>]; Sireci, [<reflink idref="bib33" id="ref54">33</reflink>]; Sireci & Randall, [<reflink idref="bib35" id="ref55">35</reflink>]; von Davier et al., [<reflink idref="bib40" id="ref56">40</reflink>]). The ASAP system accepts and responds to the challenge by striving to develop a warehouse of assessment task modules assembled in real‐time for adult learners for specific purposes. DIRTy uniqueness will reside in the interaction between routing users based on statistical information (e.g., estimates of learners' scores), and users' preferences such as personal goals and interests.</p> <p>ASAP will also connect adult education content standards (CCRSAE) to occupations and detailed work activities (ONET) by combining AI techniques with human validation. This connection will allow the ASAP to bridge academic and workplace settings for adult learners to thrive outside the classroom, for teachers to get support in their daily work, and for employers to identify in‐demand skills. Previous research has successfully created crosswalks between the European Skills, Competencies, Qualifications, and Occupations and ONET using AI techniques and human validation (European Commission, [<reflink idref="bib11" id="ref57">11</reflink>]). Butterfuss and Doran ([<reflink idref="bib6" id="ref58">6</reflink>]) have also illustrated the successful use of AI to align content standards across different systems. We expect this use to increase, which will allow items within ATMs to provide information regarding both workplace and academic skills.</p> <p>It is also important to note the validation requirements for DIRTy assessments are the same as those dictated by the <emph>Standards for Educational and Psychological Testing</emph> (American Educational Research Association et al., [<reflink idref="bib1" id="ref59">1</reflink>]) for all educational assessments. Validity evidence based on test content, response processes, internal structure, relations to other variables, and consequences of testing should be synthesized into a comprehensive validity argument (Sireci & Benitez, [<reflink idref="bib34" id="ref60">34</reflink>]). In many cases, such as the ASAP, validating the theory of action will also be important. Validating the theory of action for an assessment system requires both a validity argument and evaluation of the goals of the assessment program (Sireci, [<reflink idref="bib31" id="ref61">31</reflink>]).</p> <p>In closing, we would like to emphasize the role of technology in improving educational assessments. The use of AI is one innovation in 21st‐century assessment. Personalizing assessments to fulfill assessment purposes while simultaneously making the assessment better fit the test taker via DIRTy assessment is another. We hope the vision described in this article not only explains how the ASAP will work; we hope it also inspires others to use DIRTy design to improve assessments for all test takers.</p> <ref id="AN0182049108-15"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant number 305N210031 to the University of Massachusetts Amherst. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref34" type="bt">2</bibl> <bibtext> The maximum of seven items criterion is based on research suggesting longer item sets lead to redundancy in measurement (Sireci, Thissen, & Wainer, [36]).</bibtext> </blist> </ref> <ref id="AN0182049108-16"> <title> References </title> <blist> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.</bibtext> </blist> <blist> <bibtext> Araneda, S. (2023). An experiential approach to test design and validation. Unpublished doctoral dissertation, University of Massachusetts Amherst. https://scholarworks.umass.edu/dissertations_2/2725/</bibtext> </blist> <blist> <bibl id="bib3" idref="ref10" type="bt">3</bibl> <bibtext> Araneda, S., Lee, D., Lewis, J., Sireci, S. G., Moon, J. A., Lehman, B., Arslan, B., & Keehner, M. (2022). Exploring relationships among test takers' behaviors and performance using response process data. Education Sciences, 12, p. 104. https://doi.org/10.3390/educsci12020104.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref6" type="bt">4</bibl> <bibtext> Attali, Y., Runge, A., LaFlair, G. T., Yancey, K., Goodwin, S., Park, Y., & von Davier, A. A. (2022). The interactive reading task: Transformer‐based automatic item generation. Frontiers in Artificial Intelligence, 5. https://doi.org/10.3389/frai.2022.903077.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref40" type="bt">5</bibl> <bibtext> Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and formative assessment. Measurement, 8, pp. 70 – 91.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref58" type="bt">6</bibl> <bibtext> Butterfuss, R., & Doran, H. (2024, February). An application of text embeddings to support alignment of educational content standards. Paper presented Generative Artificial Intelligence for Measurement and Education Meeting. unpublished paper?.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref3" type="bt">7</bibl> <bibtext> Buzick, H. M., Casabianca, J., & Gholson, M. L. (2023). Personalizing large‐scale assessment in practice. Educational Measurement: Issues and Practice, 42, 5–11. https://doi.org/10.1111/emip.12551.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref41" type="bt">8</bibl> <bibtext> Clark, A. K., & Karvonen, M. (2021, October). Instructionally embedded assessment: Theory of action for an innovative system. Frontiers in Education, 6, p. 724938. https://<ulink href="http://www.frontiersin.org/articles/10.3389/feduc.2021.724938/full">www.frontiersin.org/articles/10.3389/feduc.2021.724938/full</ulink>.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref16" type="bt">9</bibl> <bibtext> Dixon‐Román, E. (2020). A haunting logic of psychometrics: Toward the speculative and indeterminacy of blackness in measurement. Educational Measurement: Issues and Practice, 39 (3), pp. 94 – 96. https://doi.org/10.1111/emip.12375.</bibtext> </blist> <blist> <bibtext> Doran, H., Yamada, T., Diaz, T., Gonulates, E., & Culver, V. (2024, January). A generalized objective function for computer adaptive item selection. Journal of Educational Measurement (early view published online). https://doi.org/10.1111/jedm.12405</bibtext> </blist> <blist> <bibtext> European Commission. (2022). The crosswalk between ESCO and O*NET. Technical Report.</bibtext> </blist> <blist> <bibtext> Foster, D. (2020). SmartItems. Midvale, Utah : Caveon.</bibtext> </blist> <blist> <bibtext> Gierl, M. J., & Haladyna, T. M. (2012). Automatic item generation. Routledge. https://doi.org/10.4324/9780203803912.</bibtext> </blist> <blist> <bibtext> Gordon, E. W. (2020). Toward assessment in the service of learning. Educational Measurement: Issues and Practice, 39 (3), pp. 72 – 78. https://doi.org/10.1111/emip.12370.</bibtext> </blist> <blist> <bibtext> Han, K. C. T., & Guo, F. (2014). Multistage testing by shaping modules on the fly. In A. Yan, A.A. von Davier, & C. Lewis (Eds.), Computerized multistage testing: Theory and applications (pp. 119 – 133). Routledge.</bibtext> </blist> <blist> <bibtext> He, Q., Borgonovi, F., & Suárez‐Álvarez, J. (2022). Clustering sequential navigation patterns in multiple‐source reading tasks with dynamic time warping method. Journal of Computer Assisted Learning, 39, 719–736. https://doi.org/10.1111/jcal.12748.</bibtext> </blist> <blist> <bibtext> Hood, S. (1998). Culturally responsive performance‐based assessment: Conceptual and psychometric considerations. Journal of Negro Education, 67 (3), pp. 187 – 196.</bibtext> </blist> <blist> <bibtext> International Test Commission and Association of Test Publishers. (2022). Guidelines for technology‐based assessment. Retrieved from https://<ulink href="http://www.intestcom.org/upload/media‐library/guidelines‐for‐technology‐based‐assessment‐v20221108‐16684036687NAG8.pdf">www.intestcom.org/upload/media‐library/guidelines‐for‐technology‐based‐assessment‐v20221108‐16684036687NAG8.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1970). Some test theory for tailored testing. In W. H. Holtzman (Ed.), Computer‐assisted instruction, testing, and guidance (pp. 139 – 183). New York : Harper and Row.</bibtext> </blist> <blist> <bibtext> Luecht, R. (2013). Assessment engineering task model maps, task models and templates as a new way to develop and implement test specifications. Journal of Applied Testing Technology, 14. https://<ulink href="http://www.testpublishers.org/assets/documents/test%20specifications%20jatt%20special%20issue%2013.pdf">www.testpublishers.org/assets/documents/test%20specifications%20jatt%20special%20issue%2013.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Luecht, R. L., & Sireci (2011). A review of models for computer‐based testing. Research report 2011–2012. New York : The College Board.</bibtext> </blist> <blist> <bibtext> Lyons, S., Johnson, M., & Hinds, B. F. (2021). Confronting inequity in assessment. Wayland, MA : Lyons Assessment Consulting.</bibtext> </blist> <blist> <bibtext> Mislevy, R. J. (2018). Sociocognitive foundations of educational measurement. New York, NY : Routledge.</bibtext> </blist> <blist> <bibtext> Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper No. 29). Urbana, IL : University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).</bibtext> </blist> <blist> <bibtext> Nelsestuen, K., & Smith, J. (2020). Empathy interviews. The Learning Professional, 41 (5), pp. 59 – 59.</bibtext> </blist> <blist> <bibtext> Pimentel, S. (2013). College and career readiness standards for adult education. Washington DC : Department of Education (ED), Office of Career, Technical, and Adult Education (OCTAE); MPR Associates, Inc.</bibtext> </blist> <blist> <bibtext> Pohl, S., Ulitzsch, E., & von Davier, M. (2021). Reframing rankings in educational assessments. Science, 372 (6540), pp. 338 – 340. https://doi.org/10.1126/science.abd3300.</bibtext> </blist> <blist> <bibtext> Questionmark (2023). Instant translate: Questionmark's ITCC's award winning translation tool. Retrieved from https://<ulink href="http://www.questionmark.com/app/uploads/2023/05/Instant‐Translate‐Fact‐Sheet.pdf">www.questionmark.com/app/uploads/2023/05/Instant‐Translate‐Fact‐Sheet.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Randall, J. (2021). " Color‐neutral" is not a thing: Redefining construct definition and representation through a justice‐oriented critical antiracist lens. Educational Measurement: Issues and Practice, 40 (4), pp. 82 – 90. https://doi.org/10.1111/EMIP.12429.</bibtext> </blist> <blist> <bibtext> Randall, J., Slomp, D., Poe, M., & Oliveri, M. E. (2022). Disrupting white supremacy in Assessment: Toward a justice‐oriented, antiracist validity framework. Educational Assessment, 27, 170–178. https://doi.org/10.1080/10627197.2022.2042682.</bibtext> </blist> <blist> <bibtext> Sireci, S. G. (2015). A theory of action for validation. In H. Jiao & R. Lissitz (Eds.). The next generation of testing: Common core standards, Smarter‐Balanced, PARCC, and the nationwide testing movement (pp. 251 – 269). Charlotte : Information Age Publishing Inc.</bibtext> </blist> <blist> <bibtext> Sireci, S. G. (2020). Standardization and UNDERSTANDardization in educational assessment. Educational Measurement: Issues and Practice, 39 (3), pp. 100 – 105.</bibtext> </blist> <blist> <bibtext> Sireci, S. G. (2021). Valuing educational measurement. Educational Measurement: Issues and Practice, 40 (1), pp. 7 – 16. 10.1111/emip.12415.</bibtext> </blist> <blist> <bibtext> Sireci, S. G., & Benitez, I., (2023). Evidence for test validation: A guide for practitioners. Psicothema, 35 (3), pp. 217 – 226.</bibtext> </blist> <blist> <bibtext> Sireci, S. G., & Randall, J. (2021). Evolving notions of fairness in testing in the United States. In B. E. Clauser & M. B. Bunch (Eds.), The history of educational measurement: Key advancements in theory, policy, and practice (pp. 111 – 135). Routledge.</bibtext> </blist> <blist> <bibtext> Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testletbased tests. Journal of Educational Measurement, 28, p. 237247.</bibtext> </blist> <blist> <bibtext> Tippins, N., & Hilton, M. (Eds.). (2010). A database for a changing economy: Review of the Occupational Information Network (O*NET). Washington, DC : The National Academies Press.</bibtext> </blist> <blist> <bibtext> Ulitzsch, E., Shin, H. J., & Lüdtke, O. (2023). Accounting for careless and insufficient effort responding in large‐scale survey data—Development, evaluation, and application of a screen‐time‐based weighting procedure. Behavior Research Methods, 56 (2), 804 – 825. https://doi.org/10.3758/s13428‐022‐02053‐6.</bibtext> </blist> <blist> <bibtext> Vespa, J., Medina, L., & Armstrong, D. M. (2020). Demographic turning points for the United States: Population projections for 2020 to 2060 population estimates and projections current population reports. Retrieved from https://<ulink href="http://www.census.gov/programs‐surveys/popproj">www.census.gov/programs‐surveys/popproj</ulink>.</bibtext> </blist> <blist> <bibtext> von Davier, A. A., Mislevy, R. J., & Hao, J. (2021). Computational psychometrics: New methodologies for a new generation of digital learning and assessment (Springer Cham:). Springer International Publishing. https://doi.org/10.1007/978‐3‐030‐74394‐9.</bibtext> </blist> <blist> <bibtext> von Davier, M., Tyack, L., & Khorramdel, L. (2022). Scoring graphical responses in TIMSS 2019 using artificial neural networks. Educational and Psychological Measurement, 83 (3), 556 – 585. https://doi.org/10.1177/00131644221098021.</bibtext> </blist> <blist> <bibtext> Wainer, H. (Ed.). (2000). Computerized adaptive testing: A primer (2nd edition). Hillsdale, NJ : Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> Walker, M. E., Olivera‐Aguilar, M., Lehman, B., Laitusis, C., Guzman‐Orth, D., & Gholson, M. (2023). Culturally responsive assessment: Provisional principles (Research Report No. RR‐23‐11). ETS. https://doi.org/10.1002/ets2.12374</bibtext> </blist> <blist> <bibtext> Wise, S. L., Im, S., & Lee, J. (2021). The impact of disengaged test taking on a state's accountability test results. Educational Assessment, 26 (3), pp. 163 – 174. https://doi.org/10.1080/10627197.2021.1956897.</bibtext> </blist> <blist> <bibtext> Yamamoto, K., Shin, H. J., & Khorramdel, L. (2019). Introduction of multistage adaptive testing design in PISA 2018. OECD Education Working Papers, No. 209, OECD Publishing, Paris, https://doi.org/10.1787/b9435d4b‐en.</bibtext> </blist> <blist> <bibtext> Yan, D., von Davier, A. A., & Lewis, C. (Eds.) (2014). Computerized multistage testing: Theory and applications. Chapman and Hall/CRC. https://doi.org/10.1201/b16858.</bibtext> </blist> <blist> <bibtext> Zenisky, A., & Hambleton, R. K. (2014). Multistage test designs: Moving research results into practice. In Yan D., von Davier A. A., & Lewis C. (Eds.), Computerized multistage testing: Theory and applications (pp. 21 – 36). New York, NY : CRC Press.</bibtext> </blist> <blist> <bibtext> Zenisky, A. L., & Sireci, S. G. (2002). Technological innovations in large‐scale assessment. Applied Measurement in Education, 15 (4), pp. 337 – 362. https://doi.org/10.1207/S15324818AME1504_02</bibtext> </blist> <blist> <bibtext> Zenisky, A., Sireci, S. G., Lewis, J., Lim, H., O'Donnell, F., Wells, C. S., Padellaro, H., Jung, H., Banda, E., Pham, D., Hong, S., Park, Y., Botha, S., Lee, M., & Garcia, A. (2018). Massachusetts Adult Proficiency Tests for College and Career Readiness: Technical Manual. Center for Educational Assessment research report Version 2 No. 990. Amherst, MA : Center for Educational Assessment.</bibtext> </blist> </ref> <aug> <p>By Stephen G. Sireci; Javier Suárez‐Álvarez; April L. Zenisky and Maria Elena Oliveri</p> <p>Reported by Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib48" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib18" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib40" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib13" firstref="ref7"></nolink> <nolink nlid="nl5" bibid="bib41" firstref="ref8"></nolink> <nolink nlid="nl6" bibid="bib45" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib16" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib38" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib27" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib44" firstref="ref14"></nolink> <nolink nlid="nl11" bibid="bib39" firstref="ref15"></nolink> <nolink nlid="nl12" bibid="bib22" firstref="ref17"></nolink> <nolink nlid="nl13" bibid="bib23" firstref="ref18"></nolink> <nolink nlid="nl14" bibid="bib29" firstref="ref19"></nolink> <nolink nlid="nl15" bibid="bib30" firstref="ref20"></nolink> <nolink nlid="nl16" bibid="bib42" firstref="ref21"></nolink> <nolink nlid="nl17" bibid="bib10" firstref="ref23"></nolink> <nolink nlid="nl18" bibid="bib32" firstref="ref24"></nolink> <nolink nlid="nl19" bibid="bib15" firstref="ref25"></nolink> <nolink nlid="nl20" bibid="bib46" firstref="ref26"></nolink> <nolink nlid="nl21" bibid="bib21" firstref="ref27"></nolink> <nolink nlid="nl22" bibid="bib47" firstref="ref28"></nolink> <nolink nlid="nl23" bibid="bib49" firstref="ref29"></nolink> <nolink nlid="nl24" bibid="bib12" firstref="ref30"></nolink> <nolink nlid="nl25" bibid="bib20" firstref="ref31"></nolink> <nolink nlid="nl26" bibid="bib26" firstref="ref32"></nolink> <nolink nlid="nl27" bibid="bib37" firstref="ref33"></nolink> <nolink nlid="nl28" bibid="bib25" firstref="ref35"></nolink> <nolink nlid="nl29" bibid="bib14" firstref="ref37"></nolink> <nolink nlid="nl30" bibid="bib17" firstref="ref45"></nolink> <nolink nlid="nl31" bibid="bib24" firstref="ref46"></nolink> <nolink nlid="nl32" bibid="bib43" firstref="ref47"></nolink> <nolink nlid="nl33" bibid="bib28" firstref="ref48"></nolink> <nolink nlid="nl34" bibid="bib19" firstref="ref51"></nolink> <nolink nlid="nl35" bibid="bib33" firstref="ref54"></nolink> <nolink nlid="nl36" bibid="bib35" firstref="ref55"></nolink> <nolink nlid="nl37" bibid="bib11" firstref="ref57"></nolink> <nolink nlid="nl38" bibid="bib34" firstref="ref60"></nolink> <nolink nlid="nl39" bibid="bib31" firstref="ref61"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1456429
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evolving Educational Testing to Meet Students' Needs: Design-in-Real-Time Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Stephen+G%2E+Sireci%22">Stephen G. Sireci</searchLink><br /><searchLink fieldCode="AR" term="%22Javier+Suárez-Álvarez%22">Javier Suárez-Álvarez</searchLink><br /><searchLink fieldCode="AR" term="%22April+L%2E+Zenisky%22">April L. Zenisky</searchLink><br /><searchLink fieldCode="AR" term="%22Maria+Elena+Oliveri%22">Maria Elena Oliveri</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Measurement%3A+Issues+and+Practice%22"><i>Educational Measurement: Issues and Practice</i></searchLink>. 2024 43(4):112-118.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305N210031
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Adult+Education%22">Adult Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Needs%22">Student Needs</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Format%22">Test Format</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Adaptive+Testing%22">Adaptive Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Culture+Fair+Tests%22">Culture Fair Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Innovation%22">Educational Innovation</searchLink><br /><searchLink fieldCode="DE" term="%22Preferences%22">Preferences</searchLink><br /><searchLink fieldCode="DE" term="%22Standards%22">Standards</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Selection%22">Test Selection</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Adult+Students%22">Adult Students</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Aspiration%22">Academic Aspiration</searchLink><br /><searchLink fieldCode="DE" term="%22Occupational+Aspiration%22">Occupational Aspiration</searchLink><br /><searchLink fieldCode="DE" term="%22Literacy%22">Literacy</searchLink><br /><searchLink fieldCode="DE" term="%22Numeracy%22">Numeracy</searchLink><br /><searchLink fieldCode="DE" term="%22Delivery+Systems%22">Delivery Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Individual+Needs%22">Individual Needs</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/emip.12653
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0731-1745<br />1745-3992
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The goal in personalized assessment is to best fit the needs of each individual test taker, given the assessment purposes. Design-in-Real-Time (DIRTy) assessment reflects the progressive evolution in testing from a single test, to an adaptive test, to an adaptive assessment "system." In this article, we lay the foundation for DIRTy assessment and illustrate how it meets the complex needs of each individual learner. The assessment framework incorporates culturally responsive assessment principles, thus making it innovative with respect to both technology and equity. Key aspects are (a) assessment building blocks called "assessment task modules" (ATMs) linked to multiple content standards and skill domains, (b) gathering information on test takers' characteristics and preferences and using this information to improve their testing experience, and (c) selecting, modifying, and compiling ATMs to create a personalized test that best meets the needs of the testing purpose and individual test taker.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1456429
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1456429
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.12653
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
        StartPage: 112
    Subjects:
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Student Needs
        Type: general
      – SubjectFull: Test Format
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Adaptive Testing
        Type: general
      – SubjectFull: Culture Fair Tests
        Type: general
      – SubjectFull: Educational Innovation
        Type: general
      – SubjectFull: Preferences
        Type: general
      – SubjectFull: Standards
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Test Selection
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Adult Students
        Type: general
      – SubjectFull: Academic Aspiration
        Type: general
      – SubjectFull: Occupational Aspiration
        Type: general
      – SubjectFull: Literacy
        Type: general
      – SubjectFull: Numeracy
        Type: general
      – SubjectFull: Delivery Systems
        Type: general
      – SubjectFull: Individual Needs
        Type: general
    Titles:
      – TitleFull: Evolving Educational Testing to Meet Students' Needs: Design-in-Real-Time Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Stephen G. Sireci
      – PersonEntity:
          Name:
            NameFull: Javier Suárez-Álvarez
      – PersonEntity:
          Name:
            NameFull: April L. Zenisky
      – PersonEntity:
          Name:
            NameFull: Maria Elena Oliveri
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0731-1745
            – Type: issn-electronic
              Value: 1745-3992
          Numbering:
            – Type: volume
              Value: 43
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Educational Measurement: Issues and Practice
              Type: main
ResultId 1