View in EDS HTML Full Text PDF Full Text

Utilizing Response Time for Item Selection in On-the-Fly Multistage Adaptive Testing for PISA Assessment

Saved in:
Bibliographic Details
Title: Utilizing Response Time for Item Selection in On-the-Fly Multistage Adaptive Testing for PISA Assessment
Language: English
Authors: Xiuxiu Tang (ORCID 0000-0002-5774-4292), Yi Zheng (ORCID 0000-0003-2671-0820), Tong Wu, Kit-Tai Hau, Hua-Hua Chang
Source: Journal of Educational Measurement. 2025 62(3):468-495.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 28
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Reaction Time, Test Items, Achievement Tests, Foreign Countries, Secondary School Students, International Assessment, Adaptive Testing, Reading Tests, Accuracy, Test Length, Item Banks, Computer Assisted Testing
Assessment and Survey Identifiers: Program for International Student Assessment
DOI: 10.1111/jedm.12403
ISSN: 0022-0655
1745-3984
Abstract: Multistage adaptive testing (MST) has been recently adopted for international large-scale assessments such as Programme for International Student Assessment (PISA). MST offers improved measurement efficiency over traditional nonadaptive tests and improved practical convenience over single-item-adaptive computerized adaptive testing (CAT). As a third alternative adaptive test design to MST and CAT, Zheng and Chang proposed the "on-the-fly multistage adaptive testing" (OMST), which combines the benefits of MST and CAT and offsets their limitations. In this study, we adopted the OMST design while also incorporating response time (RT) in item selection. Via simulations emulating the PISA 2018 reading test, including using the real item attributes and replicating PISA 2018 reading test's MST design, we compared the performance of our OMST designs against the simulated MST design in (1) measurement accuracy of test takers' ability, (2) test time efficiency and consistency, and (3) expected gains in precision by design. We also investigated the performance of OMST in item bank usage and constraints management. Results show great potential for the proposed RT-incorporated OMST designs to be used for PISA and potentially other international large-scale assessments.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1487175
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFslUwgPHK6CJaHkyntJkWsAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOJsJC4LmiX3ua8RBwIBEICBm1eUcUN7Juo9MEEASrrt8j7P7skPcQNDdp-wlJ-D4XdbOu1_krBTh7PFvsDRGqfUux3xhicFb8vPHV5ooCPYgIp6ypZB1YWJ64q7-L0AF_LPulK9nXnVrH16tmapcl4Bsnn7vL-EEdlthZVVOh4O5ksfAnxIYKhlH1GD_L_YDV4J2CKcbqDKmLi2B7u7XQLQzVBnnUo0FH0zVJqG
Text:
  Availability: 1
  Value: <anid>AN0188804204;mea01sep.25;2025Oct24.05:55;v2.2.500</anid> <title id="AN0188804204-1">Utilizing Response Time for Item Selection in On‐the‐Fly Multistage Adaptive Testing for PISA Assessment </title> <sbt id="AN0188804204-2">Introduction</sbt> <p>Multistage adaptive testing (MST) has been recently adopted for international large‐scale assessments such as Programme for International Student Assessment (PISA). MST offers improved measurement efficiency over traditional nonadaptive tests and improved practical convenience over single‐item‐adaptive computerized adaptive testing (CAT). As a third alternative adaptive test design to MST and CAT, Zheng and Chang proposed the "on‐the‐fly multistage adaptive testing" (OMST), which combines the benefits of MST and CAT and offsets their limitations. In this study, we adopted the OMST design while also incorporating response time (RT) in item selection. Via simulations emulating the PISA 2018 reading test, including using the real item attributes and replicating PISA 2018 reading test's MST design, we compared the performance of our OMST designs against the simulated MST design in (<reflink idref="bib1" id="ref1">1</reflink>) measurement accuracy of test takers' ability, (<reflink idref="bib2" id="ref2">2</reflink>) test time efficiency and consistency, and (<reflink idref="bib3" id="ref3">3</reflink>) expected gains in precision by design. We also investigated the performance of OMST in item bank usage and constraints management. Results show great potential for the proposed RT‐incorporated OMST designs to be used for PISA and potentially other international large‐scale assessments.</p> <p>Given recent technological advancements and the broadening availability of computers, more and more international large‐scale assessments (ILSAs) have gradually transitioned from traditional paper‐and‐pencil tests to <emph>computer‐based tests</emph>, which deliver the tests via computers. Moreover, ILSAs have shown an increasing interest in using computer‐based <emph>adaptive</emph> testing modes, such as <emph>multistage adaptive testing</emph> (MST) and <emph>computerized adaptive testing</emph> (CAT). For example, the Programme for International Student Assessment (PISA) switched to the MST mode for its reading assessment in 2018 (OECD, [<reflink idref="bib26" id="ref4">26</reflink>]).</p> <p>One main reason for ILSAs to adopt <emph>adaptive</emph> test designs is to reduce measurement error, in other words, improve measurement accuracy for each test taker, especially those at the two ends of the proficiency scale (i.e., test takers with very high or very low abilities). This improvement is especially beneficial to ILSAs because ILSA test takers, who come from a wide range of countries/regions, typically exhibit substantially varied performance levels in those assessments. The widely spread proficiency scales of the ILSAs are associated with two main causes: the heterogeneous populations and the large sample sizes (Rutkowski et al., [<reflink idref="bib29" id="ref5">29</reflink>]). The heterogeneous populations across countries/regions are caused by factors like the different levels of economic development, different education systems, and different types of instruction, different cultural attitudes toward education, and different levels of access to educational resources. Different areas within a large country often contribute to the heterogeneity of the population as well. Given the heterogeneous populations, the large sample sizes of the ILSAs subsequently result in substantial, nonignorable volumes of test takers at extreme ability levels in the test taker samples.</p> <p>Because ILSA test takers' ability levels vary widely across countries/regions, it is very hard to make a one‐size‐fits‐all linear (i.e., nonadaptive) test for test takers from all countries/regions, unless the items in the linear test cover the entire scale, ranging from the easiest item to the hardest, which will be impractically long and not efficient for measuring anyone. That is why more and more ILSAs are interested in adaptive test designs, which tailor the test to different ability levels and offer targeted, efficient measurement.</p> <p>Both CAT and MST belong to the family of <emph>computer‐based adaptive testing designs</emph> and are widely used in large‐scale assessments. Compared to the traditional linear test design, both CAT and MST provide more efficient measurement of test takers across the entire ability scale (Lord, [<reflink idref="bib22" id="ref6">22</reflink>]). One significant difference between MST and CAT is that MST is a <emph>group sequential design</emph>, which selects groups of items sequentially for a test taker, while CAT is a <emph>fully sequential design</emph>, which selects a single item at a time for a test taker. In general, CAT can provide a higher measurement efficiency than MST.</p> <p>However, MST is more widely used in ILSAs in practice. One reason is that the single‐item‐adaptive CAT design based on three/two‐parameter model (3PL/2PL) model and the maximum Fisher information item selection method had a serious defect that led to the severe overestimation or underestimation of test takers' ability levels (Chang & Ying, [<reflink idref="bib8" id="ref7">8</reflink>]; Rulison & Loken, [<reflink idref="bib28" id="ref8">28</reflink>]). So, MST was proposed as one of the solutions to this problem, plus it offers many other benefits, such as allowing test takers to review and revise their answers within a stage, and allowing test developers to fully review the assembled test forms before test administration. It is meant to "[strike] a balance among adaptability, practicality, measurement accuracy, and control over test forms" (Zenisky et al., [<reflink idref="bib42" id="ref9">42</reflink>]).</p> <p>However, is MST good enough for ILSAs? We will discuss this in detail in the next section and describe an alternative test design for ILSAs, named <emph>on‐the‐fly multistage adaptive testing</emph> (OMST). Then, we will present a simulation study that compares OMST with MST using PISA data under settings of the PISA 2018 reading test.</p> <p>In this study, response time will also be utilized in the OMST design as additional information to improve the measurement efficiency. Traditionally, measurement efficiency is solely assessed by the number of administered items required for a given measurement accuracy. However, the amount of time examinees spends to complete the test should also be counted in assessing measurement efficiency. The response time models and methods used in our simulation study are described in Methods section.</p> <hd id="AN0188804204-3">Background Regarding MST, Single‐Item CAT and OMST</hd> <p></p> <hd id="AN0188804204-4">Measurement Accuracy</hd> <p>Some articles point out that the main goal of ILSAs is to assess and describe the group‐level performance of each country/region instead of individual test takers' performances, so ILSAs should adopt a test design that optimizes measurement accuracy at the group level (e.g., von Davier et al., [<reflink idref="bib36" id="ref10">36</reflink>]). Nevertheless, we argue that the measurement accuracy of each individual test taker still matters. As this paper's simulation results show, when we improve the measurement accuracy for each test taker, the whole group's measurement accuracy also increases.</p> <p>As we described earlier, ILSA test takers typically have a wide range of ability levels, and ILSAs need to provide a similar level of measurement accuracy to all ability levels. In general, MST is more accurate than traditional linear testing, but its accuracy level is not as high as single‐item‐adaptive CAT (Hendrickson, [<reflink idref="bib19" id="ref11">19</reflink>]; Lord, [<reflink idref="bib22" id="ref12">22</reflink>]). There are several reasons. First, the variety of module difficulties is limited in MST. In MST, all modules (i.e., a group of items) are preassembled before the test administration, and the difficulties of those modules are fixed and limited to a handful of distinctive levels. On the contrary, in single‐item‐adaptive CAT, all items are selected on the fly during test administration, and the difficulties of those selected items can be fine‐tuned to the specific ability level of each test taker. Second, the number of adaptive points in MST is less than single‐item‐adaptive CAT. MST only adapts between stages (i.e., a block of items), and a typical MST only contains several stages, which means that there are only a few adaptive points in MST. In contrast, single‐item‐adaptive CAT adapts while selecting each item, so it has many more adaptive points than MST given the same test length. Therefore, single‐item‐adaptive CAT is a more fine‐tuned design to measure each person's ability, and its measurement accuracy is generally higher than MST.</p> <p>In general, more stages, a greater variety of module difficulties, and a longer test length are helpful to improve the measurement accuracy of MST. However, adding more stages and increasing the variety of module difficulties will make the construction of the test much more complex but without much gain in measurement accuracy. In the scenario of ILSAs, MST may not be the best design to produce the most accurate measurement for the highly heterogeneous populations and the large number of test takers with extreme abilities. Instead, other adaptive test designs that select items on the fly may be a better solution for improving measurement accuracy and reducing measurement errors for ILSAs.</p> <hd id="AN0188804204-5">Test Assembly  MST</hd> <p>The general structure of an MST test includes several stages, and each stage contains several modules (i.e., a group of items) anchored at different difficulties. Those modules are assembled into a panel, and multiple parallel panels are often assembled before the test administration.</p> <p>Because MSTs are preassembled before test administration, one advantage of MST is that test developers have more control over the assembled tests. However, this also comes with an overwhelming complexity in the design and assembly of an MST test. As Zheng et al. ([<reflink idref="bib46" id="ref13">46</reflink>]) described:</p> <p>... MST assembly consists of grouping items into modules and modules into panels optimally according to three goals: (<reflink idref="bib1" id="ref14">1</reflink>) to make information curves of modules in a stage sufficiently distinct to provide adaptivity between stages, (<reflink idref="bib2" id="ref15">2</reflink>) to make information curves of corresponding pathways across panels sufficiently similar to achieve parallel panels, and (<reflink idref="bib3" id="ref16">3</reflink>) to meet all nonstatistical constraints [such as content coverage, enemy items, exposure control], for every pathway in each panel. Due to the large number of pathways (forms), meeting all three goals becomes highly demanding, especially when the item bank is limited. (Zheng et al., [<reflink idref="bib46" id="ref17">46</reflink>]; p. 88)</p> <p>While ATA algorithms, such as the shadow‐test approach (STA), offer a solution for MST assembly, test developers may also need to review all assembled panels to ensure their quality meet the requirements (Van Der Linden, [<reflink idref="bib32" id="ref18">32</reflink>]; Van Der Linden & Reese, [<reflink idref="bib34" id="ref19">34</reflink>]; Zheng et al. [<reflink idref="bib44" id="ref20">44</reflink>]; Zheng et al., [<reflink idref="bib45" id="ref21">45</reflink>]). On the one hand, it gives test developers more control over test quality. On the other hand, it could be a burden for test developers because it is time‐consuming when there are a great number of test forms. For example, in PISA 2018 reading assessment, there are 64 possible routing paths in its design A and another 64 paths in its design B, which means that 128 test formats need to be reviewed before the test administration. As a result, computers, instead of humans, will be utilized to complete the review of test quality check, and this can be achieved in the same way in CAT.</p> <hd id="AN0188804204-6">An Alternative Testing Design for ILSA: On‐the‐Fly Multistage Adaptive Testing</hd> <p>Given the limitations of MST described in the previous sections, an alternative test design is needed to better serve the measurement needs of ILSAs. On‐the‐fly multistage adaptive testing (OMST; Zheng & Chang, [<reflink idref="bib43" id="ref22">43</reflink>]) could be a potential solution. Like MST, OMST contains several stages and only adapts between stages. At the completion of each stage, the provisional ability estimate is computed based on the test taker's performance on all previously administered items. Between stages, a new group of items is assembled for the next stage to match the provisional ability estimate. The number of stages in a test and the number of items in each stage may vary in OMST. Therefore, OMST can be viewed as a variation of single‐item‐adaptive CAT, or as MST but utilizing single‐item‐adaptive CAT's item selection algorithms. As a result, the design of OMST combines the merits of both single‐item‐adaptive CAT and MST, and offsets their limitations.</p> <p>The main advantage of single‐item‐adaptive CAT over MST is the relatively higher measurement efficiency. However, a major drawback of the single‐item‐adaptive CAT design is that its item selection method aimed to maximize the Fisher information always tends to select highly discriminating items first. Chang and Ying ([<reflink idref="bib8" id="ref23">8</reflink>]) demonstrate that using highly discriminating items in the early stage of the test is likely to cause big step sizes in estimating the ability. If the high‐ability test takers accidently answer the first several items incorrectly, or the low‐ability examinees accidently answer the first several items correctly, the basic single‐item‐adaptive CAT design may not be able to recover the ability estimation for those test takers to their true abilities; severe underestimation and overestimation will occur. For example, in early 2000s, the GRE CAT and GMAT CAT systems did not produce reliable scores for thousands of test takers (Carlson, [<reflink idref="bib3" id="ref24">3</reflink>]; Merritt, [<reflink idref="bib25" id="ref25">25</reflink>]).</p> <p>As Chang ([<reflink idref="bib4" id="ref26">4</reflink>]) pointed out, however, the excessive measurement error issue of early single‐item‐adaptive CAT designs can be addressed by redesigning item selection algorithms. There are two ways to modify the algorithms. First, we can keep the general framework of single‐item‐adaptive CAT to select items one by one on the fly but modify the selection criteria. For example, we can stratify the item bank based on <emph>a‐</emph>parameter value; then low‐discriminating items (i.e., items with low <emph>a</emph> values) are administrated at the early stage of the test, and high‐discriminating items are administrated at the later stage of the test. Examples are the <emph>a</emph>‐stratified item selection method (Chang & Ying, [<reflink idref="bib7" id="ref27">7</reflink>]) and the <emph>a</emph>‐stratified design with <emph>b</emph>‐blocking method (Chang et al., [<reflink idref="bib5" id="ref28">5</reflink>]). Second, since the estimation of the person ability parameter <emph>θ</emph> could be incorrect at the early stage of the test, we can update the ability estimate after the completion of a group of items, instead of single items in the single‐item‐adaptive CAT design. A typical example is the MST design.</p> <p>Similar to MST, OMST is a group sequential design and only updates the ability estimate at the end of each stage, which stabilizes the initial estimation. Moreover, by administering items in groups, both MST and OMST allow test takers to freely review and revise their answers within a stage, which provides a natural and friendly testing experience and reduces the anxiety of the test takers (Stocking, [<reflink idref="bib30" id="ref29">30</reflink>]; Vispoel et al., [<reflink idref="bib35" id="ref30">35</reflink>]; Wise, [<reflink idref="bib40" id="ref31">40</reflink>]).</p> <p>OMST has several advantages over MST. First, OMST has higher measurement efficiency. In OMST, each stage of items is assembled on the fly to match each individual test taker's provisional ability estimate, instead of preassembling all modules and panels before test administration. Therefore, OMST is more finely adaptive and more efficient than MST. It can provide more accurate ability estimation for test takers in ILSAs, especially for those in the two extremes of the proficiency scale.</p> <p>Second, the design of OMST can be highly flexible. Many factors can be altered and implemented easily without a substantial redesign of the test assembly algorithms, for example, the number of stages, the number of items within each stage, and the degree of adaptation. For example, the <emph>hybrid adaptive design</emph> proposed by S. Wang et al. ([<reflink idref="bib37" id="ref32">37</reflink>]) can be viewed as an extension of OMST, which shrinks the stage length gradually as the test taker moves to the later stage of the test. With this design, the test transitions from either an MST step or an OMST step to a single‐item‐adaptive CAT step. As indicated in Chang and Ying ([<reflink idref="bib6" id="ref33">6</reflink>]), at the beginning of the test, longer stage lengths are helpful to produce a more precise ability estimate when we do not have much information about the test taker; as the test proceeds, shorter stage lengths allow for more adaptations to adjust the ability estimate to this test taker's true ability value. Similarly, Choi et al. ([<reflink idref="bib11" id="ref34">11</reflink>]) proposed a strategy of requiring intervals between reassemblies of shadow tests in CAT, which yielded reductions in the number of reassemblies without compromising the measurement accuracy.</p> <p>Third, OMST reduces the complexity of test assembly. Since the modules for each stage in OMST are assembled on the fly, tailored to the specific test taker, there is no need to preassemble modules and panels to satisfy the three conflicting goals mentioned earlier in this paper (Zheng et al., [<reflink idref="bib46" id="ref35">46</reflink>], p. 88). This greatly simplifies the process of test assembly.</p> <p>In sum, OMST not only possesses the strengths of single‐item‐adaptive CAT and MST, but also mitigates the weaknesses of the two designs. Like MST, it is able to avoid the severe estimation problem of the single‐item‐adaptive CAT design and allows test takers to freely review and revise their answers within a stage. At the same time, by selecting items on the fly to match individual test takers' ability levels, it can produce a comparable estimation accuracy to single‐item‐adaptive CAT (Zheng & Chang, [<reflink idref="bib43" id="ref36">43</reflink>]). It also makes test assembly simpler than MST. Therefore, ILSAs, like PISA, may benefit from considering the OMST design.</p> <hd id="AN0188804204-7">Methods</hd> <p>In this paper, we report two simulations to compare the proposed OMST designs against the simulated MST test design as described in the PISA technical report (OECD, [<reflink idref="bib26" id="ref37">26</reflink>]) for PISA 2018 Reading. Note that although we strived to match the original MST design for PISA 2018 Reading, the simulated MST design is not exactly the same as the original MST design. The performance of test designs was assessed in five aspects: estimation accuracy, item bank usage, constraints management, test efficiency and stability (mean and standard deviation of test time), as well as expected gains in precision by design.</p> <hd id="AN0188804204-8">Emulating PISA Item Bank</hd> <p>The test scenario we emulated is the 2018 PISA reading assessment. The item bank data used in this study were downloaded from PISA's website. In the original MST design, the total number of items in the bank was 245, and they were grouped into 45 units. In the original item bank, the maximum number of items per unit was 8, the minimum number of items per unit was 3, and the average number of items per unit was 5. In the study, we only used auto‐scored items in the item bank because the scores of the manually scored items are not available for real‐time item selection. As a result, the final item bank for our simulation contains 163 items, and each item belonged to one of 44 distinct units. After removing human‐coded items from the bank, the maximum number of items per unit was 6, the minimum number of items per unit was 0 (only one unit did not contain any items), and the average number of items per unit was 4 in the final item bank. Each item was coded in one of 7 cognitive process categories.</p> <p>All items' discrimination and difficulty parameters (<emph>a, b</emph>) were obtained directly from the downloaded item bank spreadsheet. With test takers' responses and response time data, all items' time discrimination and time intensity parameters ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>α</mi><annotation encoding="application/x-tex">$\alpha $</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>β</mi><annotation encoding="application/x-tex">$\beta $</annotation></semantics></math> </ephtml> ) were estimated using a modified version of van der Linden's ([<reflink idref="bib33" id="ref38">33</reflink>]) Markov chain Monte Carlo (MCMC) routine (Choe et al., [<reflink idref="bib10" id="ref39">10</reflink>]), fixing the 2‐parameter logistic (2PL) model item parameters to the precalibrated values provided in the item bank spreadsheet and setting the mean of test takers' latent speed parameter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>τ</mi><annotation encoding="application/x-tex">$\tau $</annotation></semantics></math> </ephtml> to 0. The chain length for the MCMC calibration was 50,000, and the burn‐in size was half of the chain length (<reflink idref="bib25" id="ref40">25</reflink>,000). The trace plots suggested that all parameters appeared to converge.</p> <hd id="AN0188804204-9">Response Time Model</hd> <p>Various approaches have been proposed to model response time (RT) in testing, among which the more popular one is the hierarchical framework for modeling speed and accuracy (van der Linden, [<reflink idref="bib33" id="ref41">33</reflink>]). It has two levels of models. On the first level, the responses and response times are modeled (e.g., via the 3‐parameter logistic model and the lognormal model, respectively). On the second level, two other models specify the relations between the parameters in the first‐level models: one is the population model that describes the joint distribution of the person parameters in a population from which the test takers can be assumed to be sampled (e.g., a multivariate normal distribution for the latent ability <emph>θ</emph> and speed <emph>τ</emph>); the other is the item‐domain model that captures the relations between the item parameters by specifying a joint distribution for the item parameters in the domain of items that the test represents (e.g., a multivariate normal distribution for parameters of items).</p> <p>With a "plug‐and‐play" approach, this two‐level hierarchical framework for modeling RT allows other models to replace either the item response theory (IRT) model for item responses or the lognormal model for RT. Alternative RT models include the Box‐Cox model (Entink et al., [<reflink idref="bib16" id="ref42">16</reflink>]), the Cox proportional hazards model (Wang, Fan, et al., [<reflink idref="bib39" id="ref43">39</reflink>]), and the linear transformation model (Wang, Chang, et al., [<reflink idref="bib38" id="ref44">38</reflink>]).</p> <p>In this study, on the first level of the hierarchical framework, we used the lognormal model proposed by van der Linden (2007) to model RT. Given a test taker's latent speed parameter ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>τ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\tau }_i}$</annotation></semantics></math> </ephtml> ), the density function of RT for test taker <emph>i</emph> on item <emph>j</emph> ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><annotation encoding="application/x-tex">${{t}_{ij}}$</annotation></semantics></math> </ephtml> ) is written as 1 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>|</mo><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><mfrac><msub><mi>α</mi><mi>j</mi></msub><mrow><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><msqrt><mrow><mn>2</mn><mi>π</mi></mrow></msqrt></mrow></mfrac><msup><mi>e</mi><mrow><mo>−</mo><mfrac><msup><mfenced separators="" open="[" close="]"><mrow><msub><mi>α</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><mrow><mi>log</mi><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><msub><mi>β</mi><mi>j</mi></msub><mo>+</mo><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced></mrow></mfenced><mn>2</mn></msup><mn>2</mn></mfrac></mrow></msup><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}f\left({{{t}_{ij}}{\mathrm{|}}{{\tau }_i}} \right) = \frac{{{{\alpha }_j}}}{{{{t}_{ij}}\sqrt {2\pi } }}{{e}^{ - \frac{{{{{\left[ {{{\alpha }_j}\left({{\mathrm{log}}{{t}_{ij}} - {{\beta }_j} + {{\tau }_i}} \right)} \right]}}^2}}}{2}}},\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>α</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{\alpha }_j}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>β</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{\beta }_j}$</annotation></semantics></math> </ephtml> are the time discrimination and time intensity parameters for item <emph>j</emph>, respectively, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>β</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{\beta }_j}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>τ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\tau }_i}$</annotation></semantics></math> </ephtml> are fixed to be on the same scale. The expected RT for test taker <emph>i</emph> on item <emph>j</emph> is 2 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>T</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>|</mo><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><msup><mi>e</mi><mrow><msub><mi>β</mi><mi>j</mi></msub><mo>−</mo><msub><mi>τ</mi><mi>i</mi></msub><mo>+</mo><mfrac><mn>1</mn><mrow><mn>2</mn><msup><msub><mi>α</mi><mi>j</mi></msub><mn>2</mn></msup></mrow></mfrac></mrow></msup><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}E\left({{{T}_{ij}}{\mathrm{|}}{{\tau }_i}} \right) = {{e}^{{{\beta }_j} - {{\tau }_i} + \frac{1}{{2{{\alpha }_j}^2}}}}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>Based on the <emph>k</emph> items that have been administered to test taker <emph>i</emph>, the maximum likelihood estimation (MLE) of latent speed <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>τ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\tau }_i}$</annotation></semantics></math> </ephtml> can be computed by 3 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo linebreak="badbreak">=</mo><mfrac><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msup><msub><mi>α</mi><mi>j</mi></msub><mn>2</mn></msup><mfenced separators="" open="(" close=")"><mrow><msub><mi>β</mi><mi>j</mi></msub><mo>−</mo><mi>log</mi><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mfenced></mrow><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msup><msub><mi>α</mi><mi>j</mi></msub><mn>2</mn></msup></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{\hat{\tau }}_i} = \frac{{\mathop \sum \nolimits_{j = 1}^k {{\alpha }_j}^2\left({{{\beta }_j} - {\mathrm{log}}{{t}_{ij}}} \right)}}{{\mathop \sum \nolimits_{j = 1}^k {{\alpha }_j}^2}}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>As the response model in the study, the 2PL model was adopted to specify the probability of a test taker of latent trait <emph>θ</emph> giving a correct response to item <emph>j</emph>, denoted by X<subs>ij</subs>, as follows: 4 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>|</mo><mi>θ</mi></mrow></mfenced><mo linebreak="badbreak">=</mo><msub><mi>P</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><msub><mi>θ</mi><mi>i</mi></msub></mfenced><mo linebreak="goodbreak">=</mo><mfrac><msup><mi>e</mi><mrow><msub><mi>a</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><mrow><msub><mi>θ</mi><mi>i</mi></msub><mo>−</mo><msub><mi>b</mi><mi>j</mi></msub></mrow></mfenced></mrow></msup><mrow><mn>1</mn><mo>+</mo><msup><mi>e</mi><mrow><msub><mi>a</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><mrow><msub><mi>θ</mi><mi>i</mi></msub><mo>−</mo><msub><mi>b</mi><mi>j</mi></msub></mrow></mfenced></mrow></msup></mrow></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}P\left({{{X}_{ij}} = 1{\mathrm{|}}\theta } \right) = {{P}_j}\left({{{\theta }_i}} \right) = \frac{{{{e}^{{{a}_j}\left({{{\theta }_i} - {{b}_j}} \right)}}}}{{1 + {{e}^{{{a}_j}\left({{{\theta }_i} - {{b}_j}} \right)}}}},\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">${{X}_{ij}} = 1$</annotation></semantics></math> </ephtml> represents the correct answer for test taker <emph>i</emph> to item <emph>j</emph>, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">${{X}_{ij}} = 0$</annotation></semantics></math> </ephtml> represents the incorrect answer; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{a}_j}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{b}_j}$</annotation></semantics></math> </ephtml> are the discrimination and difficulty parameters associated with item <emph>j</emph>, respectively.</p> <p>The likelihood for test taker <emph>i</emph>'s responses is written by 5 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>|</mo><msub><mi>θ</mi><mi>i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><munderover><mo>∏</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></munderover><msub><mi>P</mi><mi>j</mi></msub><msup><mfenced separators="" open="(" close=")"><msub><mi>θ</mi><mi>i</mi></msub></mfenced><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></msup><msup><mrow><mo>(</mo><mn>1</mn><mo>−</mo><msub><mi>P</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><msub><mi>θ</mi><mi>i</mi></msub></mfenced><mo>)</mo></mrow><mrow><mn>1</mn><mo>−</mo><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></msup><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}f\left({{{x}_{ij}}{\mathrm{|}}{{\theta }_i}} \right) = \mathop \prod \limits_{j = 1}^J {{P}_j}{{\left({{{\theta }_i}} \right)}^{{{X}_{ij}}}}{{(1 - {{P}_j}\left({{{\theta }_i}} \right))}^{1 - {{X}_{ij}}}}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>Because of the conditional independence of response <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><annotation encoding="application/x-tex">${{X}_{ij}}$</annotation></semantics></math> </ephtml> and response time <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>T</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><annotation encoding="application/x-tex">${{T}_{ij}}$</annotation></semantics></math> </ephtml> given <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>(</mo><mrow><mi>θ</mi><mo>,</mo><mi>τ</mi></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$({\theta ,\tau })$</annotation></semantics></math> </ephtml> , the joint likelihood for both item and test taker parameters is 6 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">x</mi><mi mathvariant="bold-italic">i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">t</mi><mi>i</mi></msub><mrow><mo>|</mo></mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi>i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><munderover><mo>∏</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></munderover><mi>f</mi><mrow><mo>(</mo><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>|</mo><msub><mi>θ</mi><mi>i</mi></msub><mo>,</mo><msub><mi>a</mi><mi>j</mi></msub><mo>,</mo><msub><mi>b</mi><mi>j</mi></msub><mo>)</mo></mrow><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mrow><mo>|</mo></mrow><msub><mi>τ</mi><mi>i</mi></msub><mo>,</mo><msub><mi>α</mi><mi>j</mi></msub><mo>,</mo><msub><mi>β</mi><mi>j</mi></msub></mrow></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}f\left({{{{\bm{x}}}_{\bm{i}}},{{{\bm{t}}}_i}|{{{\bm{\xi }}}_i},{{{\bm{\psi }}}_j}} \right) = \mathop \prod \limits_{j = 1}^J f({{x}_{ij}}|{{\theta }_i},{{a}_j},{{b}_j})f\left({{{t}_{ij}}|{{\tau }_i},{{\alpha }_j},{{\beta }_j}} \right),\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi>i</mi></msub><mo>=</mo><mrow><mo>(</mo><mrow><msub><mi>θ</mi><mi>i</mi></msub><mo>,</mo><msub><mi>τ</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${{{\bm{\xi }}}_i} = ({{{\theta }_i},{{\tau }_i}})$</annotation></semantics></math> </ephtml> is the parameters vector for test taker <emph>i</emph>, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mo>=</mo><mrow><mo>(</mo><mrow><msub><mi>a</mi><mi>j</mi></msub><mo>,</mo><msub><mi>b</mi><mi>j</mi></msub><mo>,</mo><msub><mi>α</mi><mi>j</mi></msub><mo>,</mo><msub><mi>β</mi><mi>j</mi></msub></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${{{\bm{\psi }}}_j} = ({{{a}_j},{{b}_j},{{\alpha }_j},{{\beta }_j}})$</annotation></semantics></math> </ephtml> is the parameters vector for item <emph>j</emph>.</p> <p>On the second level, both parameter vectors <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">ξ</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{\xi }}}_{\bm{i}}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi mathvariant="bold-italic">ψ</mi><mi mathvariant="bold-italic">j</mi></msub><annotation encoding="application/x-tex">${{{\bm{\psi }}}_{\bm{j}}}$</annotation></semantics></math> </ephtml> are assumed to be randomly drawn from multivariate normal distributions.</p> <p></p> <ulist> <item> (<reflink idref="bib1" id="ref45">1</reflink>) The population model: 7 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0028" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi mathvariant="bold-italic">i</mi></msub><mo linebreak="badbreak">=</mo><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi mathvariant="bold-italic">i</mi></msub><mrow><mo>|</mo></mrow><msub><mi mathvariant="bold-italic">μ</mi><mi>p</mi></msub><mo>,</mo><msub><mi mathvariant="bold">Σ</mi><mi>p</mi></msub></mrow></mfenced><mo linebreak="goodbreak">=</mo><mfrac><mrow><mrow><mo>|</mo></mrow><msubsup><mo>∑</mo><mi>p</mi><mrow><mo>−</mo><mn>1</mn></mrow></msubsup><msup><mo>|</mo><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msup></mrow><mrow><mn>2</mn><mi>π</mi></mrow></mfrac><mi>exp</mi><mfenced separators="" open="[" close="]"><mrow><mo linebreak="badbreak">−</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><msup><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi>i</mi></msub><mo>−</mo><msub><mi mathvariant="bold-italic">μ</mi><mi mathvariant="bold-italic">p</mi></msub></mrow></mfenced><mi>T</mi></msup><munderover><mo>∑</mo><mi>p</mi><mrow><mo>−</mo><mn>1</mn></mrow></munderover><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi>i</mi></msub><mo>−</mo><msub><mi mathvariant="bold-italic">μ</mi><mi>p</mi></msub></mrow></mfenced></mrow></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{\bm{\xi }}}_{\bm{i}}} = f\left({{{{\bm{\xi }}}_{\bm{i}}}|{{{\bm{\mu }}}_p},{{{{\bm \Sigma }}}_p}} \right) = \frac{{|\mathop \sum \nolimits_p^{ - 1} {{|}^{1/2}}}}{{2\pi }}{\mathrm{exp}}\left[ { - \frac{1}{2}{{{\left({{{{\bm{\xi }}}_i} - {{{\bm{\mu }}}_{\bm{p}}}} \right)}}^T}\mathop \sum \limits_p^{ - 1} \left({{{{\bm{\xi }}}_i} - {{{\bm{\mu }}}_p}} \right)} \right]\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <p>with mean vector 8 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0029" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">μ</mi><mi>p</mi></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="(" close=")"><mrow><msub><mi>μ</mi><mi>θ</mi></msub><mo>,</mo><mspace width="0.33em" /><msub><mi>μ</mi><mi>τ</mi></msub></mrow></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{\bm{\mu }}}_p} = \left({{{\mu }_\theta },\ {{\mu }_\tau }} \right)\end{equation}$$</annotation></semantics></math> </ephtml> and covariance matrix 9 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0030" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">Σ</mi><mi>p</mi></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="(" close=")">σθ2σθτσθτστ2</mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{{\bm \Sigma }}}_p} = \left({ \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} {\sigma _\theta ^2}&{{{\sigma }_{\theta \tau }}}\\ {{{\sigma }_{\theta \tau }}}&{\sigma _\tau ^2} \end{array} } \right).\end{equation}$$</annotation></semantics></math> </ephtml></p> <p></p> <ulist> <item> (<reflink idref="bib2" id="ref46">2</reflink>) The item‐domain model: 10 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0031" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mo linebreak="badbreak">=</mo><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mrow><mo>|</mo></mrow><msub><mi mathvariant="bold-italic">μ</mi><mi>I</mi></msub><mo>,</mo><msub><mi mathvariant="bold">Σ</mi><mi>I</mi></msub></mrow></mfenced><mo linebreak="goodbreak">=</mo><mfrac><mrow><mrow><mo>|</mo></mrow><msubsup><mo>∑</mo><mi>I</mi><mrow><mo>−</mo><mn>1</mn></mrow></msubsup><msup><mo>|</mo><mrow><mn>1</mn><mo>/</mo><mn>2</mn></mrow></msup></mrow><msup><mfenced separators="" open="(" close=")"><mrow><mn>2</mn><mi>π</mi></mrow></mfenced><mrow><mn>5</mn><mo>/</mo><mn>2</mn></mrow></msup></mfrac><mi>exp</mi><mfenced separators="" open="[" close="]"><mrow><mo linebreak="badbreak">−</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><msup><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mo>−</mo><msub><mi mathvariant="bold-italic">μ</mi><mi>I</mi></msub></mrow></mfenced><mi>T</mi></msup><munderover><mo>∑</mo><mi>I</mi><mrow><mo>−</mo><mn>1</mn></mrow></munderover><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mo>−</mo><msub><mi mathvariant="bold-italic">μ</mi><mi>I</mi></msub></mrow></mfenced></mrow></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{\bm{\psi }}}_j} = f\left({{{{\bm{\psi }}}_j}|{{{\bm{\mu }}}_I},{{{{\bm \Sigma }}}_I}} \right) = \frac{{|\mathop \sum \nolimits_I^{ - 1} {{|}^{1/2}}}}{{{{{\left({2\pi } \right)}}^{5/2}}}}{\mathrm{exp}}\left[ { - \frac{1}{2}{{{\left({{{{\bm{\psi }}}_j} - {{{\bm{\mu }}}_I}} \right)}}^T}\mathop \sum \limits_I^{ - 1} \left({{{{\bm{\psi }}}_j} - {{{\bm{\mu }}}_I}} \right)} \right]\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <p>with mean vector 11 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0032" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold-italic">μ</mi><mi>I</mi></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="(" close=")"><mrow><msub><mi>μ</mi><mi>a</mi></msub><mo>,</mo><msub><mi>μ</mi><mi>b</mi></msub><mo>,</mo><msub><mi>μ</mi><mi>α</mi></msub><mo>,</mo><msub><mi>μ</mi><mi>β</mi></msub></mrow></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{\bm{\mu }}}_I} = \left({{{\mu }_a},{{\mu }_b},{{\mu }_\alpha },{{\mu }_\beta }} \right)\end{equation}$$</annotation></semantics></math> </ephtml> and covariance matrix 12 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0033" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="bold">Σ</mi><mi>I</mi></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="(" close=")">σa2σabσaασaβσbaσb2σbασbβσαaσαbσα2σαβσβaσβbσβασβ2</mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{{{\bm \Sigma }}}_I} = \left({ \def\eqcellsep{&}\begin{array}{@{}*{4}{c}@{}} {\sigma _a^2}&{{{\sigma }_{ab}}}&{{{\sigma }_{a\alpha }}}&{{{\sigma }_{a\beta }}}\\ {{{\sigma }_{ba}}}&{\sigma _b^2}&{{{\sigma }_{b\alpha }}}&{{{\sigma }_{b\beta }}}\\ {{{\sigma }_{\alpha a}}}&{{{\sigma }_{\alpha b}}}&{\sigma _\alpha ^2}&{{{\sigma }_{\alpha \beta }}}\\ {{{\sigma }_{\beta a}}}&{{{\sigma }_{\beta b}}}&{{{\sigma }_{\beta \alpha }}}&{\sigma _\beta ^2} \end{array} } \right).\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>For the full model, the joint sampling distribution in Equation 6 for all parameters is extended as 13 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0034" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><mi mathvariant="bold-italic">u</mi><mo>,</mo><mi mathvariant="bold-italic">t</mi><mo>|</mo><mrow><mi mathvariant="bold-italic">ξ</mi></mrow><mo>,</mo><mrow><mi mathvariant="bold-italic">ψ</mi></mrow></mrow></mfenced><mo linebreak="badbreak">=</mo><munderover><mo>∏</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>I</mi></munderover><munderover><mo>∏</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></munderover><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">u</mi><mi mathvariant="bold-italic">i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">t</mi><mi mathvariant="bold-italic">i</mi></msub><mo>|</mo><msub><mi mathvariant="bold-italic">ξ</mi><mi>i</mi></msub><mo>,</mo><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub></mrow></mfenced><mi>f</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi mathvariant="bold-italic">ξ</mi><mi mathvariant="bold-italic">i</mi></msub><mrow><mo>|</mo></mrow><msub><mi mathvariant="bold-italic">μ</mi><mi>p</mi></msub><mo>,</mo><msub><mi mathvariant="bold">Σ</mi><mi>p</mi></msub></mrow></mfenced><mi>f</mi><mrow><mo>(</mo><msub><mi mathvariant="bold-italic">ψ</mi><mi>j</mi></msub><mo>|</mo><msub><mi mathvariant="bold-italic">μ</mi><mi>I</mi></msub><mo>,</mo><msub><mi mathvariant="bold">Σ</mi><mi>I</mi></msub><mo>)</mo></mrow><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}f\left({{\bm{u}},{\bm{t}}|{\bm{\xi }},{\bm{\psi }}} \right) = \mathop \prod \limits_{i = 1}^I \mathop \prod \limits_{j = 1}^J f\left({{{{\bm{u}}}_{\bm{i}}},{{{\bm{t}}}_{\bm{i}}}{\mathrm{|}}{{{\bm{\xi }}}_i},{{{\bm{\psi }}}_j}} \right)f\left({{{{\bm{\xi }}}_{\bm{i}}}|{{{\bm{\mu }}}_p},{{{{\bm \Sigma }}}_p}} \right)f({{{\bm{\psi }}}_j}|{{{\bm{\mu }}}_I},{{{{\bm \Sigma }}}_I}).\end{equation}$$</annotation></semantics></math> </ephtml></p> <hd id="AN0188804204-10">Item Selection Methods</hd> <p></p> <hd id="AN0188804204-11">MST design</hd> <p>In the MST design, we emulate the design of PISA 2018 Reading MST as described in the technical report (OECD, [<reflink idref="bib26" id="ref47">26</reflink>]). In this test, items were bundled into item units, and an item unit was considered the unit for adaptive item selection. The MST design had three stages. The first stage (the Core stage) contains 2 item units, the second stage contains 3 item units, and the third stage contains 2 item units. All the routing paths (64 paths) in the MST were constructed by following the instructions given in Figure 2.7 of the technical report (p. 18). All testlets (bundles of items prestructured by test developers) were constructed following Tables 2.4–2.6 of the report (pp. 12, 14, and 16), which document the unit IDs included in each testlet. While PISA 2018 Reading MST used number‐correct (NC) score to route test takers between stages, in this simulation study, we used the MFI method for routing, which selects items with the maximum Fisher information at the current ability estimate. The reason for using MFI instead of NC is because IRT‐based routing methods generally tended to outperform the NC methods in terms of item recovery (Svetina et al., [<reflink idref="bib31" id="ref48">31</reflink>]).</p> <hd id="AN0188804204-12">OMST designs</hd> <p>To make our OMST results more comparable to the original MST design, we implemented the same multistage structure as the original MST design and set OMST to have three stages, each containing 2 units, 3 units, and 2 units, respectively. In terms of content domain coverage, Table 2.8 in the technical report (OECD, [<reflink idref="bib26" id="ref49">26</reflink>], p. 26; also see Table A.4 in Appendix) shows that there are 7 content domains, and the recommended percentages of each domain in a test are 15%, 15%, 15%, 20%, 10%, 15%, and 10%. The <emph>maximum priority index</emph> (MPI) (Cheng & Chang, [<reflink idref="bib9" id="ref50">9</reflink>]) was used as the item selection method to control content coverage in our OMST designs.</p> <p>The MPI method is a heuristic method, which does not guarantee that all constraints are satisfied as 0‐1 programming does. However, it is easy to compute and hence practical for large‐scale implementation, and it also do not face the infeasibility issue—if the item pool is not sufficient to meet the constraints, 0‐1 programming fails to return a solution. The MPI method proposed by Cheng and Chang ([<reflink idref="bib9" id="ref51">9</reflink>]) was therefore chosen as the constraint‐controlled item selection method in this study. It is able to accommodate various nonstatistical constraints simultaneously, such as content coverage, item exposure, answer key balancing, and so on.</p> <p>We adapted the MPI method to select a unit at a time: 14 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0035" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><msub><mi>I</mi><mi>j</mi></msub><mo linebreak="badbreak">=</mo><msub><mi>I</mi><mi>j</mi></msub><mfenced separators="" open="(" close=")"><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub></mfenced><munderover><mo>∏</mo><mrow><mi>k</mi><mspace width="0.33em" /><mo>=</mo><mspace width="0.33em" /><mn>1</mn></mrow><mi>K</mi></munderover><msup><mfenced separators="" open="(" close=")"><mrow><msub><mi>w</mi><mi>k</mi></msub><msub><mi>f</mi><mi>k</mi></msub></mrow></mfenced><msub><mi>c</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub></msup><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}P{{I}_j} = {{I}_j}\left({{{{\hat{\theta }}}_i}} \right)\mathop \prod \limits_{k\ = \ 1}^K {{\left({{{w}_k}{{f}_k}} \right)}^{{{c}_{jk}}}},\end{equation}$$</annotation></semantics></math> </ephtml> 15 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0036" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>k</mi></msub><mo linebreak="badbreak">=</mo><mfrac><mrow><msub><mi>X</mi><mi>k</mi></msub><mo>−</mo><msub><mi>y</mi><mi>k</mi></msub></mrow><msub><mi>X</mi><mi>k</mi></msub></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{f}_k} = \frac{{{{X}_k} - {{y}_k}}}{{{{X}_k}}},\end{equation}$$</annotation></semantics></math> </ephtml> 16 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0037" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><msub><mi>I</mi><mi>u</mi></msub><mo linebreak="badbreak">=</mo><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>M</mi></munderover><mi>P</mi><msub><mi>I</mi><mi>j</mi></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}P{{I}_u} = \mathop \sum \limits_{j = 1}^M P{{I}_j},\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0038" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><msub><mi>I</mi><mi>j</mi></msub></mrow><annotation encoding="application/x-tex">$P{{I}_j}$</annotation></semantics></math> </ephtml> represents each item's priority index, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0039" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><msub><mi>I</mi><mi>u</mi></msub></mrow><annotation encoding="application/x-tex">$P{{I}_u}$</annotation></semantics></math> </ephtml> represents each unit <emph>u</emph>'s priority index which is the summation of all items' <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0040" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><msub><mi>I</mi><mi>j</mi></msub></mrow><annotation encoding="application/x-tex">$P{{I}_j}$</annotation></semantics></math> </ephtml> within the unit. Here, <emph>M</emph> is used to denote the total number of items in unit <emph>u</emph>. <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0041" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>I</mi><mi>j</mi></msub><annotation encoding="application/x-tex">${{I}_j}$</annotation></semantics></math> </ephtml> represents the Fisher information of item <emph>j</emph> evaluated at the current <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0042" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> , <emph>K</emph> is used to denote constraints. The exponent in Equation 14, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0043" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">${{c}_{jk}}$</annotation></semantics></math> </ephtml> , indicates the relevance of each item to each constraint: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0044" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">${{c}_{jk}} = 1$</annotation></semantics></math> </ephtml> means that constraint <emph>k</emph> is relevant to item <emph>j</emph> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0045" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mrow><mi>j</mi><mi>k</mi></mrow></msub><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">${{c}_{jk}} = 0$</annotation></semantics></math> </ephtml> means that constraint <emph>k</emph> is not relevant to item <emph>j</emph>. The letter <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0046" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>f</mi><mi>k</mi></msub><annotation encoding="application/x-tex">${{f}_k}$</annotation></semantics></math> </ephtml> denotes the degree to which item <emph>j</emph> helps to meet constraint <emph>k</emph>, and it can be computed using Equation 15: for content constraint k, the test needs to have <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0047" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>X</mi><mi>k</mi></msub><annotation encoding="application/x-tex">${{X}_k}$</annotation></semantics></math> </ephtml> items from a certain content area, and so far <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0048" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>y</mi><mi>k</mi></msub><annotation encoding="application/x-tex">${{y}_k}$</annotation></semantics></math> </ephtml> such items have been selected.</p> <p>We also included two RT‐incorporated item selection methods in our OMST designs. The first method is named <emph>Maximum Priority Index per Time Unit</emph> (MPIT), which combines the MIT criterion (Fan et al., [<reflink idref="bib17" id="ref52">17</reflink>]) with the MPI method to select the next item unit that (<reflink idref="bib1" id="ref53">1</reflink>) maximizes the information per expected response time unit and (<reflink idref="bib2" id="ref54">2</reflink>) best satisfies the content constraints: 17 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0049" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mi>I</mi><msub><mi>T</mi><mi>u</mi></msub><mfenced separators="" open="(" close=")"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>,</mo><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><mfrac><mrow><mi>P</mi><msub><mi>I</mi><mi>u</mi></msub></mrow><mrow><mi>E</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>T</mi><mrow><mi>i</mi><mi>u</mi></mrow></msub><mo>|</mo><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></mfenced></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}PI{{T}_u}\left({{{{\hat{\theta }}}_i},{{{\hat{\tau }}}_i}} \right) = \frac{{P{{I}_u}}}{{E\left({{{T}_{iu}}{\mathrm{|}}{{{\hat{\tau }}}_i}} \right)}}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>The second method is named <emph>Maximum Generalized Priority Index per Time Unit</emph> (MGPIT), which adapts RT to the MPI method with the GMIT criterion (Choe et al., [<reflink idref="bib10" id="ref55">10</reflink>]). GMIT was developed based on MIT, and two more variables (the centering parameter <emph>v</emph> and the weighting exponent <emph>w</emph>) were added to adjust the influence of the expected RT on selecting items. MGPIT selects the next unit that maximize 18 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0050" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi>P</mi><mi>I</mi><msub><mi>T</mi><mi>u</mi></msub><mfenced separators="" open="(" close=")"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>,</mo><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><mfrac><mrow><mi>P</mi><msub><mi>I</mi><mi>u</mi></msub></mrow><msup><mfenced separators="" open="|" close="|"><mrow><mi>E</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>T</mi><mrow><mi>i</mi><mi>u</mi></mrow></msub><mo>|</mo><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub></mrow></mfenced><mo>−</mo><mi>v</mi></mrow></mfenced><mi>w</mi></msup></mfrac><mo>,</mo><mspace width="0.33em" /><mfenced separators="" open="{" close="}"><mrow><mi>v</mi><mo>,</mo><mi>w</mi></mrow></mfenced><mo>∈</mo><msubsup><mi mathvariant="double-struck">R</mi><mrow><mo>≥</mo><mn>0</mn></mrow><mn>2</mn></msubsup><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}GPI{{T}_u}\left({{{{\hat{\theta }}}_i},{{{\hat{\tau }}}_i}} \right) = \frac{{P{{I}_u}}}{{{{{\left| {E\left({{{T}_{iu}}{\mathrm{|}}{{{\hat{\tau }}}_i}} \right) - v} \right|}}^w}}},\ \left\{ {v,w} \right\} \in \mathbb{R}_{ \ge 0}^2.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>In this study, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0051" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi><mspace width="0.33em" /><mo>=</mo><mspace width="0.33em" /><mi>med</mi><mo>(</mo><mrow><mi>E</mi><mo>[</mo><mrow><msub><mi>T</mi><mi>j</mi></msub><mrow><mo>|</mo><mi>med</mi></mrow><mrow><mo>(</mo><mi>τ</mi><mo>)</mo></mrow></mrow><mo>]</mo></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$v\ = \ {\mathrm{med}}({E[ {{{T}_j}{\mathrm{|med}}(\tau)} ]})$</annotation></semantics></math> </ephtml> , the median of the expected RT at the median of <emph>τ</emph>, because it was the best performer of GMIT in Choe et al. ([<reflink idref="bib10" id="ref56">10</reflink>]). Their study also shows that the choice of <emph>w</emph> was less important and based on the minimum accuracy or maximum average rate of test overlap deemed acceptable, so we set <emph>w</emph> to 1 in our study for simplicity.</p> <p>To balance item exposure rates across different units, a <emph>randomesque</emph> (Kingsbury & Zara, [<reflink idref="bib21" id="ref57">21</reflink>]) component was added to the OMST item selection procedure. With the randomesque component, the next unit was randomly chosen from the top <emph>R</emph> units with the largest PIT or GPIT values. We simulated the test with for <emph>R</emph> = 1, 3, 6, and 10, and found that the design with <emph>R</emph> = 10 gave the best overall performance (see Table 1).</p> <p>1 Table Comparisons of OMST Designs Using the MPIT Method with Different Random Values</p> <p> <ephtml> <table><thead><tr><th align="left" /><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0052" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\theta }})$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0053" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\tau }})$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0054" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0055" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mover accent="true"><mi>V</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\bar{V}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0056" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mover><mrow><mi>t</mi><mi>t</mi></mrow><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline {tt} $</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0057" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi>s</mi><mrow><mi>t</mi><mi>t</mi></mrow></msub><annotation encoding="application/x-tex">${{s}_{tt}}$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td align="left">OMST without randomesque</td><td align="left">.285</td><td align="left">.190</td><td align="left">76.229</td><td align="left">1.391</td><td align="left">1,467,526</td><td align="left">786,477</td></tr><tr><td align="left">OMST with randomesque, R = 3</td><td align="left">.270</td><td align="left">.184</td><td align="left">54.593</td><td align="left">1.539</td><td align="left">1,433,534</td><td align="left">774,528</td></tr><tr><td align="left">OMST with randomesque, R = 6</td><td align="left">.283</td><td align="left">.186</td><td align="left">29.958</td><td align="left">1.564</td><td align="left">1,433,893</td><td align="left">784,253</td></tr><tr><td align="left">OMST with randomesque, R = 10</td><td align="left">.290</td><td align="left">.191</td><td align="left">15.754</td><td align="left">1.544</td><td align="left">1,494,651</td><td align="left">832,575</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0058" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\theta }})$</annotation></semantics></math> </ephtml> is the root mean squared error of test takers' latent ability estimate; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0059" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\tau }})$</annotation></semantics></math> </ephtml> is the root mean squared error of test takers' latent speed estimate; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0060" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> is the measure for item exposure rate; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0061" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>V</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\bar{V}$</annotation></semantics></math> </ephtml> is the average number of violated constraints; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0062" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover><mrow><mi>t</mi><mi>t</mi></mrow><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline {tt} $</annotation></semantics></math> </ephtml> is the mean of test time; and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0063" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>s</mi><mrow><mi>t</mi><mi>t</mi></mrow></msub><annotation encoding="application/x-tex">${{s}_{tt}}$</annotation></semantics></math> </ephtml> is the standard deviation of test time.</p> <hd id="AN0188804204-13">Simulation design</hd> <p>Four types of test designs were simulated: (<reflink idref="bib1" id="ref58">1</reflink>) the MST design emulating PISA 2018 reading test, (<reflink idref="bib2" id="ref59">2</reflink>) the OMST design with a randomesque procedure (<emph>R</emph> = 10), (<reflink idref="bib3" id="ref60">3</reflink>) the OMST design using MPIT method with a randomesque procedure (<emph>R</emph> = 10), and (<reflink idref="bib4" id="ref61">4</reflink>) the OMST design using MGPIT method with a randomesque procedure (<emph>R</emph> = 10).</p> <p>The first and second test designs (MST, OMST, respectively) did not incorporate RT when selecting items, while the third and fourth did. Comparing the basic OMST against the OMST using RT in item selection via the MPIT or MGPIT method, we can evaluate the additional effect of RT in OMST designs for improving measurement efficiency.</p> <p>We conducted two simulations. The first simulation aims to characterize the performance of the above four designs conditional on five different ability groups. Each group consisted of 1,000 test takers and was simulated from the bivariate normal distribution of ability parameters <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0064" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>θ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\theta }_i}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0065" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>τ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\tau }_i}$</annotation></semantics></math> </ephtml> . Groups 1 to 5 were ordered from the low ability level to the high ability level. <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0066" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfenced separators="" open="(" close=")"><mrow><msub><mi>θ</mi><mi>i</mi></msub><mo>,</mo><mspace width="0.33em" /><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced><mo>∼</mo><msub><mi>N</mi><mn>2</mn></msub><mfenced separators="" open="(" close=")"><mrow><msub><mi>μ</mi><mi>g</mi></msub><mo>,</mo><msub><mi mathvariant="normal">Σ</mi><mi>g</mi></msub></mrow></mfenced><mo>,</mo><mspace width="0.33em" /><mi>g</mi><mo>∈</mo><mfenced separators="" open="{" close="}"><mrow><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mtext>...</mtext><mo>,</mo><mn>5</mn></mrow></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*}\left({{{\theta }_i},\ {{\tau }_i}} \right)\sim {{N}_2}\left({{{\mu }_g},{{{{\Sigma}}}_g}} \right),\ g \in \left\{ {1,2, \ldots ,5} \right\},\end{equation*}$$</annotation></semantics></math> </ephtml><ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0067" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>μ</mi><mn>1</mn></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="[" close="]">−20</mfenced><mo>,</mo><mspace width="0.33em" /><msub><mi>μ</mi><mn>2</mn></msub><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">−10</mfenced><mo>,</mo><mspace width="0.33em" /><msub><mi>μ</mi><mn>3</mn></msub><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">00</mfenced><mo>,</mo><mspace width="0.33em" /><msub><mi>μ</mi><mn>4</mn></msub><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">10</mfenced><mo>,</mo><mspace width="0.33em" /><msub><mi>μ</mi><mn>5</mn></msub><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">20</mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*}{{\mu }_1} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} { - 2}\\ 0 \end{array} } \right],\ {{\mu }_2} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} { - 1}\\ 0 \end{array} } \right],\ {{\mu }_3} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} 0\\ 0 \end{array} } \right],\ {{\mu }_4} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} 1\\ 0 \end{array} } \right],\ {{\mu }_5} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} 2\\ 0 \end{array} } \right],\end{equation*}$$</annotation></semantics></math> </ephtml><ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0068" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">Σ</mi><mi>g</mi></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="[" close="]">0.250.1250.1250.25</mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*}{{{{\Sigma}}}_g} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} {0.25}&{0.125}\\ {0.125}&{0.25} \end{array} } \right].\end{equation*}$$</annotation></semantics></math> </ephtml></p> <p>The second simulation was to compare the four designs for the overall test takers. 1,000 test takers were simulated from the bivariate normal distribution of ability parameters <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0069" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>θ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\theta }_i}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0070" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>τ</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{\tau }_i}$</annotation></semantics></math> </ephtml> . <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0071" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfenced separators="" open="(" close=")"><mrow><msub><mi>θ</mi><mi>i</mi></msub><mo>,</mo><mspace width="0.33em" /><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced><mo>∼</mo><msub><mi>N</mi><mn>2</mn></msub><mfenced separators="" open="(" close=")"><mrow><mi>μ</mi><mo>,</mo><mi mathvariant="normal">Σ</mi></mrow></mfenced><mo>,</mo><mspace width="0.33em" /><mi>μ</mi><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">00</mfenced><mo>,</mo><mspace width="0.33em" /><mi mathvariant="normal">Σ</mi><mo linebreak="goodbreak">=</mo><mfenced separators="" open="[" close="]">10.250.250.25</mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*}\left({{{\theta }_i},\ {{\tau }_i}} \right)\sim {{N}_2}\left({\mu ,{{\Sigma}}} \right),\ \mu = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} 0\\ 0 \end{array} } \right],\ {{\Sigma}} = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} 1&{0.25}\\ {0.25}&{0.25} \end{array} } \right].\end{equation*}$$</annotation></semantics></math> </ephtml></p> <p>All simulations were conducted on RStudio (Posit team, [<reflink idref="bib27" id="ref62">27</reflink>]). The design of MST was simulated using functions from R package mstR (Magis et al., [<reflink idref="bib24" id="ref63">24</reflink>]), and designs of OMST were simulated using functions written by the authors following the aforementioned equations.</p> <hd id="AN0188804204-14">Evaluation Criteria</hd> <p>The performance of test designs is assessed in five aspects: estimation accuracy, item bank usage, constraint management, test efficiency and stability, and expected gains in precision by design. For each aspect, the following measures were adopted respectively, and each criterion outcome was averaged over the 200 replications.</p> <p>Note that the proposed three OMST designs are not compared to the simulated MST design in terms of item bank usage and constraint management. This is because the study does not utilize the full item bank for the PISA 2018 reading assessment, which may impact the MST's performance in item exposure and constraint control. For example, even when the original PISA 2018 MST tests fully satisfy all content constraints, our simulated MST may show violations because the human‐scored items were not used in our simulation, which does not represent the actual PISA 2018 MST. Therefore, we solely evaluate the performance of the three OMST designs based on the criteria of item bank usage and constraint management. All other criteria are compared across all four test designs.</p> <p></p> <ulist> <item> Estimation accuracy measures: root mean squared error (RMSE) of test takers' latent ability estimate (EAP) and latent speed estimate (MLE) 19 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0072" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mfenced separators="" open="(" close=")"><mover accent="true"><mi>θ</mi><mo>̂</mo></mover></mfenced><mo linebreak="badbreak">=</mo><msqrt><mrow><mfrac><mn>1</mn><mi>n</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msup><mfenced separators="" open="(" close=")"><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>−</mo><msub><mi>θ</mi><mi>i</mi></msub></mrow></mfenced><mn>2</mn></msup></mrow></msqrt><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}RMSE\left({\hat{\theta }} \right) = \sqrt {\frac{1}{n}\mathop \sum \limits_{i = 1}^n {{{\left({{{{\hat{\theta }}}_i} - {{\theta }_i}} \right)}}^2}} ,\end{equation}$$</annotation></semantics></math> </ephtml> 20 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0073" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mfenced separators="" open="(" close=")"><mover accent="true"><mi>τ</mi><mo>̂</mo></mover></mfenced><mo linebreak="badbreak">=</mo><msqrt><mrow><mfrac><mn>1</mn><mi>n</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msup><mfenced separators="" open="(" close=")"><mrow><msub><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mi>i</mi></msub><mo>−</mo><msub><mi>τ</mi><mi>i</mi></msub></mrow></mfenced><mn>2</mn></msup></mrow></msqrt><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}RMSE\left({\hat{\tau }} \right) = \sqrt {\frac{1}{n}\mathop \sum \limits_{i = 1}^n {{{\left({{{{\hat{\tau }}}_i} - {{\tau }_i}} \right)}}^2}}.\end{equation}$$</annotation></semantics></math> </ephtml></item> <p></p> <item> Item bank usage measure: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0074" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> of exposure rates (Chang & Ying, [<reflink idref="bib7" id="ref64">7</reflink>]) 21 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0075" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mo linebreak="badbreak">=</mo><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mfrac><msup><mfenced separators="" open="(" close=")"><mrow><mi>e</mi><msub><mi>r</mi><mi>j</mi></msub><mo>−</mo><msub><mover><mrow><mi>e</mi><mi>r</mi></mrow><mo>¯</mo></mover><mi>j</mi></msub></mrow></mfenced><mn>2</mn></msup><msub><mover><mrow><mi>e</mi><mi>r</mi></mrow><mo>¯</mo></mover><mi>j</mi></msub></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{\chi }^2} = \mathop \sum \limits_{j = 1}^N \frac{{{{{\left({e{{r}_j} - {{{\overline {er} }}_j}} \right)}}^2}}}{{{{{\overline {er} }}_j}}},\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <p>where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0076" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi><msub><mi>r</mi><mi>j</mi></msub><mo>=</mo><mspace width="0.33em" /><mfrac><mrow><mrow><mi>number</mi><mspace width="0.33em" /><mi>of</mi><mspace width="0.33em" /><mi>times</mi><mspace width="0.33em" /><mi>the</mi><mspace width="0.33em" /></mrow><mi>j</mi><mrow><mi>th</mi><mspace width="0.33em" /><mi>item</mi><mspace width="0.33em" /><mi>is</mi><mspace width="0.33em" /><mi>used</mi></mrow><mspace width="0.33em" /></mrow><mi>N</mi></mfrac></mrow><annotation encoding="application/x-tex">$e{{r}_j} = \ \frac{{{\mathrm{number\ of\ times\ the\ }}j{\mathrm{th\ item\ is\ used}}\ }}{N}$</annotation></semantics></math> </ephtml> represents the observed exposure rate for the <emph>j</emph>th item. <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0077" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover><mrow><mi>e</mi><mi>r</mi></mrow><mo>¯</mo></mover><mi>j</mi></msub><annotation encoding="application/x-tex">${{\overline {er} }_j}$</annotation></semantics></math> </ephtml> represents the mean of the observed exposure rates for the <emph>j</emph>th item in all test takers. <emph>N</emph> is the number of test takers in the test, and <emph>j</emph> is used to denote item <emph>j</emph>. The descriptive statistics of exposure rates were also reported for the two simulation designs, which includes the mean and standard deviation (<emph>SD</emph>) of exposure rates, and the proportion of items never exposed.</p> <p></p> <ulist> <item> (c) Constraints management measure: the average number of violated constraints ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0078" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>V</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\bar{V}$</annotation></semantics></math> </ephtml> ) 22 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0079" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>V</mi><mo>¯</mo></mover><mo linebreak="badbreak">=</mo><mfrac><mrow><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></msubsup><msub><mi>V</mi><mi>i</mi></msub></mrow><mi>N</mi></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}\bar{V} = \frac{{\mathop \sum \nolimits_{i = 1}^N {{V}_i}}}{N},\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <p>where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0080" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>V</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{V}_i}$</annotation></semantics></math> </ephtml> denotes the number of constraint violations in the <emph>i</emph>th person's test. For each content domain, the minimum number of items were calculated based on the recommended percentages of each domain in the technical report (see OMST Designs section). For a test given to a person, if the number of items selected from a content domain is less than the minimum number required, that will be counted as one violation. Violating multiple domains' constraints in a single test will be recorded by the specific number of constraints violated. This number is averaged across all examinees to constitute the average number of violated constraints.</p> <p></p> <ulist> <item> (d) Test efficiency and stability measures: mean and standard deviation of test time 23 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0081" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover><mrow><mi>t</mi><mi>t</mi></mrow><mo>¯</mo></mover><mo linebreak="badbreak">=</mo><mfrac><mn>1</mn><mi>n</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mspace width="0.33em" /><mo>=</mo><mspace width="0.33em" /><mn>1</mn></mrow><mi>n</mi></munderover><mi>t</mi><msub><mi>t</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><mfrac><mn>1</mn><mi>n</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mspace width="0.33em" /><mo>=</mo><mspace width="0.33em" /><mn>1</mn></mrow><mi>n</mi></munderover><munder><mo>∑</mo><mrow><mi>j</mi><mo>∈</mo><msub><mi>R</mi><mi>i</mi></msub></mrow></munder><msub><mi>t</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}\overline {tt} = \frac{1}{n}\mathop \sum \limits_{i\ = \ 1}^n t{{t}_i} = \frac{1}{n}\mathop \sum \limits_{i\ = \ 1}^n \mathop \sum \limits_{j \in {{R}_i}} {{t}_{ij}},\end{equation}$$</annotation></semantics></math> </ephtml> 24 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0082" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>t</mi><mi>t</mi></mrow></msub><mo linebreak="badbreak">=</mo><msqrt><mrow><mfrac><mn>1</mn><mrow><mi>n</mi><mo>−</mo><mn>1</mn></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msup><mfenced separators="" open="(" close=")"><mrow><mi>t</mi><msub><mi>t</mi><mi>i</mi></msub><mo>−</mo><mover><mrow><mi>t</mi><mi>t</mi></mrow><mo>¯</mo></mover></mrow></mfenced><mn>2</mn></msup></mrow></msqrt><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{{s}_{tt}} = \sqrt {\frac{1}{{n - 1}}\mathop \sum \limits_{i = 1}^n {{{\left({t{{t}_i} - \overline {tt} } \right)}}^2}} ,\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <p>where R<subs>j</subs> is the set of all items administered to test taker <emph>i</emph>.</p> <p></p> <ulist> <item> (e) The expected gains in measurement precision by design (Yamamoto et al., [<reflink idref="bib41" id="ref65">41</reflink>]): the below equation was used to quantify the expected gains in precision by design with 60 quadrature points (<emph>Q</emph> = 60) between −3 and 3, where q indicates each quadrature point. <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0083" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>P</mi><mi>q</mi></msub><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${{P}_q}(\theta)$</annotation></semantics></math> </ephtml> indicates the empirical density (the density of simulated students with ability <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0084" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> ranging from −3 to 3) of students at each quadrature point, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0085" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>E</mi><msub><mrow><mo>(</mo><mrow><mi>O</mi><mi>M</mi><mi>S</mi><mi>T</mi></mrow><mo>)</mo></mrow><mi>q</mi></msub></mrow><annotation encoding="application/x-tex">$SE{{({OMST})}_q}$</annotation></semantics></math> </ephtml> stands for the standard error at each quadrature point for a given OMST design, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0086" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>E</mi><msub><mrow><mo>(</mo><mrow><mi>M</mi><mi>S</mi><mi>T</mi></mrow><mo>)</mo></mrow><mi>q</mi></msub></mrow><annotation encoding="application/x-tex">$SE{{({MST})}_q}$</annotation></semantics></math> </ephtml> represents standard error at each quadrature point under the baseline MST design: 25 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0087" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mfenced separators="" open="(" close=")"><mrow><mi>G</mi><mi>a</mi><mi>i</mi><msub><mi>n</mi><mrow><mi>D</mi><mi>e</mi><mi>s</mi><mi>i</mi><mi>g</mi><mi>n</mi></mrow></msub></mrow></mfenced><mo linebreak="badbreak">=</mo><munderover><mo>∑</mo><mrow><mi>q</mi><mo>=</mo><mn>1</mn></mrow><mi>Q</mi></munderover><mfrac><mrow><mi>S</mi><mi>E</mi><msub><mfenced separators="" open="(" close=")"><mrow><mi>M</mi><mi>S</mi><mi>T</mi></mrow></mfenced><mi>q</mi></msub><mo>−</mo><mi>S</mi><mi>E</mi><msub><mfenced separators="" open="(" close=")"><mrow><mi>O</mi><mi>M</mi><mi>S</mi><mi>T</mi></mrow></mfenced><mi>q</mi></msub></mrow><mrow><mi>S</mi><mi>E</mi><msub><mfenced separators="" open="(" close=")"><mrow><mi>M</mi><mi>S</mi><mi>T</mi></mrow></mfenced><mi>q</mi></msub></mrow></mfrac><msub><mi>P</mi><mi>q</mi></msub><mfenced open="(" close=")"><mi>θ</mi></mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}E\left({Gai{{n}_{Design}}} \right) = \mathop \sum \limits_{q = 1}^Q \frac{{SE{{{\left({MST} \right)}}_q} - SE{{{\left({OMST} \right)}}_q}}}{{SE{{{\left({MST} \right)}}_q}}}{{P}_q}\left(\theta \right).\end{equation}$$</annotation></semantics></math> </ephtml></item> </ulist> <hd id="AN0188804204-15">Results</hd> <p></p> <hd id="AN0188804204-16">Choosing the R Value for Randomesque</hd> <p>Table 1 shows that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0088" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> value was reduced greatly from 76.229 without the randomesque component to 15.754 when <emph>R</emph> = 10, which means that the item pool was best used when <emph>R</emph> = 10 among all the four simulated designs. Other evaluation criteria, including estimation accuracy, violated constraints, and test time, did not differ much among all designs; in fact, the four designs performed quite closely in terms of those criteria. Hence, we decided to set <emph>R</emph> = 10 for all OMST designs in the study.</p> <hd id="AN0188804204-17">Simulation 1 Results</hd> <p>Figures 1–5 summarize the conditional performances of four designs (i.e., MST, OMST, OMST using MPIT, and OMST using MGPIT) in five different ability groups.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0001.jpg" title="1 RMSE of estimated ability." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0002.jpg" title="2 RMSE of estimated speed." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0003.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0003.jpg" title="3 Chi‐square of item exposure rates." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0004.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0004.jpg" title="4 Mean of item exposure rates." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0005.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0005.jpg" title="5 Standard deviation of item exposure rates." /> </p> <p></p> <hd id="AN0188804204-23">Estimation accuracy</hd> <p>As we can see from Figure 1, as expected, all four designs are most accurate for test takers with the average abilities, since they have lower RMSE values. For test takers who are at the two ends of ability range, the estimation accuracy is lower, leading to higher RMSE values. However, the RMSE values of the simulated MST design are always significantly higher than those of three OMST designs, regardless of the ability group or whether response time is incorporated in item selection. The three OMST designs demonstrate comparable levels of estimation accuracy across different ability groups. Among them, the two OMST designs that integrate response time exhibit slightly lower accuracy than the OMST design that does not incorporate response time, but the differences are nominal. In terms of estimating examinees' latent speeds, Figure 2 shows that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0089" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\tau }})$</annotation></semantics></math> </ephtml> values in all designs are almost the same (around .2), which is consistent with findings in other articles (Choe et al., [<reflink idref="bib10" id="ref66">10</reflink>]; Du et al., [<reflink idref="bib15" id="ref67">15</reflink>]).</p> <hd id="AN0188804204-24">Item bank usage</hd> <p>Figure 3 presents the results of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0090" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> of item exposure rates for the three OMST designs. Note that a lower value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0091" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> means a more even usage of the item bank. We can see that the OMST design using MGPIT method has the lowest <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0092" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> values across all ability groups. It seems that the proposed MGPIT method is able to better control the item pool usage than all the other test designs across different ability groups. Figures 4–6 present the means and <emph>SD</emph>s of the item exposure rates as well as the proportion of items never exposed in the three OMST designs. Compared to the basic OMST design, RT‐incorporated OMSTs have lower means of item exposure rates, around .16, across all ability groups; OMST using MGPIT also has lower <emph>SD</emph>s of item exposure rates across all ability groups, and OMST using MPIT has comparable <emph>SD</emph>s of item exposure rates to OMST for the low‐ and high‐ability groups (Group 1 and Group 5) but lower <emph>SD</emph>s for the moderate ability groups (Groups 2, 3, and 4). In terms of items never exposed, OMST using MGPIT has the lowest proportion of items never exposed (close to 0) than the other two OMST designs. Based on the above results, it appears that OMST using MGPIT resulted in the best item bank usage, because it has the lowest <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0093" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> values, means and <emph>SD</emph>s of item exposure rates, and proportion of items never exposed across all ability groups.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0006.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0006.jpg" title="6 Proportion of items never exposed." /> </p> <p></p> <hd id="AN0188804204-26">Constraints management</hd> <p>Figure 7 presents the average number of violated constraints in the three OMST designs. From this figure, we can see that the basic OMST without RT has the lowest average number of violated constraints across all groups, ranging from .67 to 1.08. The two RT‐incorporated OMSTs have similar average numbers of violated constraints in different ability groups.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0007.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0007.jpg" title="7 Average number of violated constraints." /> </p> <p></p> <p>All OMST designs resulted in some violations. This was caused by the unit‐level item selection. Once a unit is chosen, all items in the unit are selected. Although overall this unit is optimal for the item selection criterion, it may include one or two items from the content domains whose quotas have already been filled. When the total test length is limited, having extra items from one content domain takes up the space for another content domain, eventually leading to a shortage in some content domains. Note that Zheng and Chang's ([<reflink idref="bib43" id="ref68">43</reflink>]) original OMST design included an <emph>item replacement step</emph> that ensures all constraints are met. This algorithm takes place after all items are selected for the next stage using the MPI method, and before the items are administered to the test taker. The algorithm compares the current selection of items against all constraints and identifies violations. If a violation occurs, the algorithm swaps an item from the current selection with the next best item in the item bank to meet the constraint. This algorithm guarantees all constraints are met as long as the item bank has sufficient supply. However, this algorithm will not help clear out all violations when items are bundled in units, due to the reasons given earlier in this paragraph.</p> <hd id="AN0188804204-28">Test efficiency and stability</hd> <p>Figure 8 illustrates the mean test times for each design, while Figure 9 displays the standard deviations (<emph>SD</emph>s) of test times. It is evident from Figure 8 that our proposed RT‐incorporated OMST designs yielded shorter test times. Conversely, the OMST design without RT performed similarly to the simulated MST design. The results indicate a 25% reduction in test time when RT is incorporated into OMST, compared to both the simulated MST and standalone OMST designs. Notably, recall that all OMST designs demonstrate significant improvement in estimation accuracy. Therefore, it appears that our RT‐incorporated OMST designs achieved better estimation accuracy and shorter test times simultaneously. In contrast, the basic OMST design without RT consumes the same amount of test time as the simulated MST design. Additionally, Figure 9 reveals that both the simulated MST and basic OMST designs exhibit similar levels of <emph>SD</emph>s in test times, which are higher compared to the RT‐incorporated OMST designs. The reduced <emph>SD</emph>s in test times in the RT‐incorporated OMST designs imply greater similarity in the time taken by different test takers, particularly in designs using the MPIT and MGPIT methods. Both RT‐incorporated OMST designs effectively reduce test time; however, the OMST employing the MGPIT design demonstrates the smallest <emph>SD</emph>s in test times across all ability groups. It appears that the OMST utilizing the MGPIT method outperforms others in terms of test efficiency and stability.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0008.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0008.jpg" title="8 Mean test time." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01sep25/jedm12403-fig-0009.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12403-fig-0009.jpg" title="9 Standard deviation of test time." /> </p> <p></p> <hd id="AN0188804204-31">Summary of Simulation 1 results</hd> <p>In sum, compared to MST, all three OMST designs resulted in higher accuracy in estimating test takers' abilities across different ability groups. By incorporating RT in the item selection process, OMST was able to not only maintain the improved measurement accuracy in the basic OMST design, but also greatly reduce mean test time and make all test takers spend the similar amount of time on the test across all ability groups. It suggests that the two RT‐incorporated item selection methods, MPIT and MGPIT, could help OMST designs to further improve the measurement efficiency for PISA across all ability groups.</p> <hd id="AN0188804204-32">Simulation 2 Results</hd> <p>Simulation 2 aims to evaluate the <emph>overall</emph> performance of the four test designs, as opposed to <emph>conditional</emph> performance for specific ability groups. The results are summarized in Table 2. We can see that <emph>RMSE</emph>( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0094" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> ) of MST is much larger than that of the three OMST designs, which is consistent with our findings in Simulation 1. <emph>RMSE</emph>( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0095" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> ) of MST is approximately .5, while <emph>RMSE</emph>( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0096" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> )s of the three OMSTs are approximately .3. In general, using OMST instead of MST improves measurement accuracy by 40%. <emph>RMSE</emph>( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0097" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\tau }$</annotation></semantics></math> </ephtml> ) is similar for all designs, which is consistent with other researchers' studies on RT (Choe et al., [<reflink idref="bib10" id="ref69">10</reflink>]).</p> <p>2 Table The Overall Results of All Designs</p> <p> <ephtml> <table><thead><tr valign="bottom"><th align="left" /><th align="center">MST</th><th align="center">OMST</th><th align="center">OMST Using MPIT</th><th align="center">OMST Using MGPIT</th></tr><tr valign="bottom"><th align="left" /><th align="center">Mean (<italic>SD</italic>)</th><th align="center">Mean (<italic>SD</italic>)</th><th align="center">Mean (<italic>SD</italic>)</th><th align="center">Mean (<italic>SD</italic>)</th></tr></thead><tbody><tr><td align="left">RMSE(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0098" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><mo>̂</mo><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math></p>)</td><td>.498</td><td>.280</td><td>.294</td><td>.302</td></tr><tr><td align="center">(.000)</td><td>(.004)</td><td>(.008)</td><td>(.007)</td></tr><tr><td align="left">RMSE(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0099" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>τ</mi><mo>̂</mo><annotation encoding="application/x-tex">$\hat{\tau }$</annotation></semantics></math></p>)</td><td align="center">.188</td><td>.198</td><td>.191</td><td>.198</td></tr><tr><td align="center">(.000)</td><td>(.001)</td><td>(.001)</td><td>(.001)</td></tr><tr><td align="left"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0100" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>χ</mi>2<annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math></p></td><td align="center">‐</td><td>18.241</td><td>16.112</td><td>13.478</td></tr><tr><td>(.468)</td><td>(.384)</td><td>(.356)</td></tr><tr><td align="left">Mean of exposure rates</td><td align="center">‐</td><td>.186</td><td>.157</td><td>.162</td></tr><tr><td>(.000)</td><td>(.001)</td><td>(.000)</td></tr><tr><td align="left">SD of exposure rates</td><td align="center">‐</td><td>.144</td><td>.126</td><td>.115</td></tr><tr><td>(.002)</td><td>(.001)</td><td>(.001)</td></tr><tr><td align="left">Never exposed</td><td align="center">‐</td><td>.010</td><td>.003</td><td>.000</td></tr><tr><td>(.013)</td><td>(.008)</td><td>(.000)</td></tr><tr><td align="left"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0101" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>V</mi><mo>¯</mo><annotation encoding="application/x-tex">$\bar{V}$</annotation></semantics></math></p></td><td align="center">‐</td><td>.866</td><td>1.568</td><td>1.469</td></tr><tr><td>(.022)</td><td>(.027)</td><td>(.026)</td></tr><tr><td align="left"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0102" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>t</mi><mi>t</mi><mo>¯</mo><annotation encoding="application/x-tex">$\overline {tt} $</annotation></semantics></math></p></td><td>1,987,225</td><td>2,021,459</td><td>1,555,318</td><td>1,565,059</td></tr><tr><td>(41)</td><td>(4,636)</td><td>(7,423)</td><td>(5,913)</td></tr><tr><td align="left"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0103" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>s</mi><mi>t</mi><mi>t</mi><annotation encoding="application/x-tex">${{s}_{tt}}$</annotation></semantics></math></p></td><td>1,083,705</td><td>1,102,496</td><td>864,990</td><td>748,765</td></tr><tr><td>(40)</td><td>(14,512)</td><td>(15,478)</td><td>(16,742)</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note</emph>: MST is the multistage adaptive testing design; OMST is the on‐the‐fly multistage adaptive testing design with a randomesque procedure (<emph>R</emph> = 10); OMST using MPIT is the on‐the‐fly multistage adaptive testing design using the proposed <emph>Maximum Priority Index per Time Unit</emph> (MGPIT) method with a randomesque procedure (<emph>R</emph> = 10); OMST using MGPIT is the on‐the‐fly multistage adaptive testing design using the proposed <emph>Maximum Generalized Priority Index per Time Unit</emph> (MGPIT) method with a randomesque procedure (<emph>R</emph> = 10); <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0104" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>M</mi><mi>S</mi><mi>E</mi><mo>(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>)</mo></mrow><annotation encoding="application/x-tex">$RMSE({\hat{\theta }})$</annotation></semantics></math> </ephtml> is the root mean squared error of test takers' latent ability estimate; <emph>RMSE</emph>( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0105" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>τ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\tau }$</annotation></semantics></math> </ephtml> ) is the root mean squared error of test takers' latent speed estimate; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0106" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> is the measure for item exposure rates; mean of exposure rates is the mean of item exposure rates; <emph>SD</emph> of exposure rates is the standard deviation of item exposure rates; never exposed indicates the proportion of items that are never exposed; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0107" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>V</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\bar{V}$</annotation></semantics></math> </ephtml> is the constraints management measure, which is the average number of violated constraints; <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0108" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover><mrow><mi>t</mi><mi>t</mi></mrow><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline {tt} $</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0109" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>s</mi><mrow><mi>t</mi><mi>t</mi></mrow></msub><annotation encoding="application/x-tex">${{s}_{tt}}$</annotation></semantics></math> </ephtml> are test efficiency and stability measures, and they are mean and standard deviation of test time, respectively; mean and <emph>SD</emph> are the mean and standard deviation of replications results.</p> <p>In terms of item exposure, the basic OMST without RT exhibits a relatively larger <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0110" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> value compared to the two OMSTs incorporating RT. Furthermore, the OMST utilizing MGPIT demonstrates even smaller <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0111" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> values when compared to the OMST employing MPIT. This observation aligns with the findings presented in Choe et al. ([<reflink idref="bib10" id="ref70">10</reflink>]). Introducing the centering variable <emph>v</emph> in MGPIT is anticipated to enhance item pool utilization, thereby leading to a reduction in the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0112" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">${{\chi }^2}$</annotation></semantics></math> </ephtml> value.</p> <p>Similar trends are observed concerning the mean and standard deviation (<emph>SD</emph>) of item exposure rates, as well as the proportion of items never exposed: the basic OMST design yields larger values in these criteria compared to the two RT‐incorporated OMST designs. However, the differences among the three designs are relatively small in terms of exposure rates. The means of exposure rates for the three OMST designs range from .157 to .186, with <emph>SD</emph>s ranging from .115 to .144. There are almost no items that remain unexposed in any of the three OMST designs. Thus, this suggests that the proposed three OMST designs, whether incorporating RT or not, demonstrate relatively balanced usage of the item pool.</p> <p>In terms of constraints control, after incorporating RT into item selection, both OMST designs using MPIT or MGPIT produced a higher average number of violated constraints compared to the basic OMST, though the differences are small (around .7). Regarding test time, the two RT‐incorporated OMST designs displayed lower mean test times and smaller standard deviations of test time compared to both MST and the basic OMST. These findings are consistent with our previous observations conditional on test taker ability.</p> <hd id="AN0188804204-33">Expected gains in measurement precision by design</hd> <p>Using Equation 25, the expected gains of OMST over MST in measurement precision is 3.25, and the expected gains of OMST incorporating RT over MST in measurement precision is 2.84. This means that, compared to the design of MST, it is expected to obtain additional gains in measurement precision by using the proposed OMST designs (with or without incorporated RT). All OMST designs contribute to the accuracy of the person ability estimator across the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12403:jedm12403-math-0113" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> proficiency scale between −3 and 3 with around 10%∼50% higher accuracy (i.e., 1 – <emph>SE</emph>(<emph>OMST</emph>)/<emph>SE</emph>(<emph>MST</emph>)). The basic OMST had larger expected gains than RT‐incorporated OMSTs. However, the differences between them are small across the proficiency scale.</p> <hd id="AN0188804204-34">Conclusions and Discussion</hd> <p>This study proposed three OMST designs for the settings of international large‐scale assessments such as PISA. All three designs utilize Cheng and Chang's ([<reflink idref="bib9" id="ref71">9</reflink>]) MPI approach for simultaneously managing various test assembly demands and constraints. Two of them also incorporate response time in item selection, using the MPIT (Fan et al., [<reflink idref="bib17" id="ref72">17</reflink>]) or MGPIT (Choe et al., [<reflink idref="bib10" id="ref73">10</reflink>]) method. Hence, the RT‐incorporated designs not only maximize the information per expected response time unit but also address content constraints. Two simulation studies were conducted to compare the performance of the OMST designs with an MST design under settings that emulate the PISA 2018 reading test using the test's item bank. Results suggest that the proposed OMST designs resulted in (<reflink idref="bib1" id="ref74">1</reflink>) more accurate estimation of test takers' abilities than the simulated MST; (<reflink idref="bib2" id="ref75">2</reflink>) shorter test times as well as more similar test times across test takers than the simulated MST; (<reflink idref="bib3" id="ref76">3</reflink>) large gains in measurement precision than the simulated MST; (<reflink idref="bib4" id="ref77">4</reflink>) a balanced item exposure; and (<reflink idref="bib5" id="ref78">5</reflink>) a minor extent of violations of the content constraints because items are bundled in units. In short, the proposed OMST designs using RT‐incorporated item selection methods show promise for application to PISA and other ILSAs to improve measurement efficiency, because they are able to improve measurement accuracy while simultaneously reducing mean test time.</p> <p>In the context of ILSAs, the ability range of their test takers is notably broad, and providing accurate measurement results for each population or subpopulation is important, especially for people with extreme abilities. In the meantime, if we could use less items to achieve the same level of measurement accuracy, it would be beneficial for both test takers and test administers since it would save their time, energy and other expenses related to test taking. Also, improving each individual's measurement accuracy would contribute to the accuracy of the overall population. Simulations in the study show that the proposed OMST design is about 40% more accurate than the simulated MST design. Therefore, improving measurement accuracy is important in ILSAs.</p> <p>MST is widely used in ILSAs and national large‐scale assessments, but its improvement in measurement accuracy is limited, because there are a limited number of module difficulty variations, and those modules are preassembled before the test administration. In contrast, the proposed OMST design provide more accurate measurement than MST, whether for the whole population or for subpopulations with different average abilities. Compared to MST, OMST has finer‐grained adaptation, because items are selected on the fly to match each individual's ability. By doing so, the items provide more information about the precise location of the test taker's ability parameter. Therefore, from the perspective of measurement accuracy, our study shows that OMST can produce more accurate results for test takers in the whole proficiency scale, which may help address the measurement issue resulting from the heterogeneous populations and the large number of people who are at two ends of the ability range in ILSAs.</p> <p>Given a certain measurement accuracy, shortening test time is also beneficial for ILSAs. Results from the simulation show a 25% reduction in test time when RT is incorporated into OMST, compared to both the simulated MST and standalone OMST designs. Meanwhile, using OMST, with or without incorporating RT, improves measurement accuracy by 40% compared to MST. Therefore, using RT in selecting items could help OMST, and potentially other adaptive testing designs, to further improve their measurement efficiency in ILSAs.</p> <p>Based on the simulation results, the two proposed RT‐incorporated item selection methods, MPIT and MGPIT, have very similar performances in many evaluation criteria. The main reason for the negligible difference between the two methods is that the item bank used for the study is small, only containing 163 items. Moreover, the selection of items for the test taker was on the unit level for the sake of emulating PISA 2018 reading assessment, but there were only 44 units of items in the bank, which means only 44 options were available for the two item selection methods. Therefore, the limited number of items in the bank may have led to the similar performance between the proposed two item selection methods. When we have a larger item bank, according to Choe et al. ([<reflink idref="bib10" id="ref79">10</reflink>]), the MGPIT method is likely to reduce the item exposure rate and thereby further optimize item pool usage compared to the MPIT method.</p> <p>Compared to RT‐involved item selection methods in previous CAT/MST studies, our proposed methods can not only take into account test time when selecting items, but also control the test assembly constraints. A test's blueprint usually has requirements for the distribution of items in different content areas. Simulation results suggest that our proposed item selection methods are able to satisfy such content constraints with minimal violations. There were still a few instances of violations because the items were selected unit by unit instead of item by item. The proposed item selection methods are able to produce no or few violations of constraints when items are selected one by one, or when items in each unit align with the constraint distributions stated in the blueprint. Apart from that, mixed‐integer linear programming may be used to solve the constraint violations caused by the unit‐level item selection.</p> <p>Moreover, our study demonstrates that OMST is a highly flexible testing design, which can accommodate a variety of testing needs in ILSAs. The OMST item selection algorithm can accommodate multiple statistical and nonstatistical constraints. In our study, seven content constraints were included based on PISA 2018 reading assessment's blueprint, and a randomesque component was added to the item selection algorithm to help balance item exposure rates. Various other constraints, such as answer key balance, testing time control, or preventing enemy items from appearing in the same test, also can be easily programmed in the OMST item selection algorithm. Furthermore, if new constraints are added, or existing constraints need to change, the OMST item selection algorithm only needs to be minimally modified. In contrast, MST may need to be reassembled to make sure all test forms meet the new requirements.</p> <p>In the study, a randomesque component was added to the OMST item selection procedure to balance item exposure rates across different units, helping OMST use the item pool evenly. Without the randomesque procedure, larger units (i.e., containing more items within a unit) or small units containing highly discriminant items tend to be more frequently selected due to their large unit‐level PI values, while some units may never be selected. This a typical issue in CAT item selection criteria that optimize test information. Hence, we recommend that an item exposure control algorithm, randomesque or other methods, be used with the proposed OMST designs to even out item bank usage.</p> <p>One limitation of the simulation study was that the item bank used in the study differed from the real item bank used in the PISA 2018 reading test. Because human‐scored items cannot be scored immediately in simulation studies, they were excluded from the item bank of this study, and only machine‐scored items were included. As a result, the results from the simulation study cannot be directly transferred to PISA. Nevertheless, the exact same item bank was used for the MST and the OMSTs in our simulation study. Also note that PISA 2018 MST only used machine‐scored items to route students during test administration, which means that the human‐scored items did not play a role in selecting items for each student in the PISA 2018 MST in the first place. Hence our comparison of MST and OMST has a reasonable basis. In the future, with the affordances of generative AI, those human‐scored items in PISA may be automatically scored by computers instead. At that time, those items may be utilized in adaptive item selection as well.</p> <p>Another limitation of the study was that there was no comparison with other approaches such as shadow‐test approach (Diao & Van Der Linden, [<reflink idref="bib14" id="ref80">14</reflink>]) or a more basic stepwise assembly approach (Ali et al., [<reflink idref="bib1" id="ref81">1</reflink>]; Bock & Zimowski, [<reflink idref="bib2" id="ref82">2</reflink>]). In the future, it would be interesting to compare the performances of different test assembly methods for OMST. Note that in real application, the test assembly (i.e., item selection) algorithm of OMST, whether it is a heuristic approach (e.g., MPI) or a 0‐1 programming approach (e.g., shadow test), needs to be implemented in the test delivery system, which may be more computationally intense than the administration of MST.</p> <p>This simulation study was also limited by the test design parameters set by the PISA 2018 reading test because we tried to emulate that particular test design to provide a fair comparison between the MST design used in that test with our proposed designs. For example, the item bank used for the study is small, only containing 163 items, and items are grouped into 44 units. When we have a larger item bank, the OMST design is expected to further improve measurement efficiency. In addition, the number of stages in OMST was fixed at three in this study, but allowing more stages could also result in even more efficient estimation. A further study may be conducted to investigate the performance of OMST in other reasonable scenarios of international large‐scale assessments.</p> <p>Additionally, the proposed OMST designs do not consider all practical testing restrictions, such as link items and the proportion of human‐coded to machine‐coded items. The sample size is not typical for PISA, neither for calibrations nor for reporting. Furthermore, omitted responses, not‐reached items, item position effects, and item‐by‐country interactions often exist in PISA and other ILSAs (Debeer et al., [<reflink idref="bib13" id="ref83">13</reflink>]; Debeer & Janssen, [<reflink idref="bib12" id="ref84">12</reflink>]; Guo et al., [<reflink idref="bib18" id="ref85">18</reflink>]; Lu & Wang, [<reflink idref="bib23" id="ref86">23</reflink>]). These factors limit the generalization of this study's findings to operational PISA settings and should be investigated in future OMST research for ILSAs.</p> <p>For future research, we may replace Fisher information with Kullback‐Leibler (KL) information in the proposed item selection methods to investigate its effect on measurement efficiency. While methods in this study were based on Fisher information to select items for each test taker, KL information‐based item selection chooses items that effectively discriminate between different levels of ability. Another possible research direction is to explore the use of the joint expected a posteriori estimator (J‐EAP) to estimate test takers latent abilities and speed traits. Research demonstrates that the J‐EAP estimation method for ability and speededness surpasses the standard MLE regarding correlation, root mean square error, and bias (Kern & Choe, [<reflink idref="bib20" id="ref87">20</reflink>]). It is also shown that employing the J‐EAP under the maximum information per time unit item selection method (MIT) leads to additional reductions in average test taker time and variability in test times among test takers beyond those achieved by the MIT with the MLE, while maintaining estimation efficiency.</p> <ref id="AN0188804204-35"> <title> References </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Ali, U. S., Van Rijn, P. W., & Robin, F. (2023). Optimizing multistage adaptive testing designs for large‐scale survey assessments. In M. Wiberg, D. Molenaar, J. González, J.‐S. Kim, & H. Hwang (Eds.), Quantitative psychology (Vol. 422, pp. 335 – 345). Springer Nature Switzerland. https://doi.org/10.1007/978‐3‐031‐27781‐8_29</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Bock, R. D., & Zimowski, M. F. (1997). Multiple group IRT. In W. J. Van Der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 433 – 448). New York : Springer. https://doi.org/10.1007/978‐1‐4757‐2691‐6_25</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Carlson, S. (2000). ETS finds flaws in the way online GRE rates some students. The Chronicle of Higher Education, 47 (8), A47.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref26" type="bt">4</bibl> <bibtext> Chang, H.‐H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80 (1), 1 – 20. https://doi.org/10.1007/s11336‐014‐9401‐5</bibtext> </blist> <blist> <bibl id="bib5" idref="ref28" type="bt">5</bibl> <bibtext> Chang, H.‐H., Qian, J., & Ying, Z. (2001). A‐stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25 (4), 333 – 341. https://doi.org/10.1177/01466210122032181</bibtext> </blist> <blist> <bibl id="bib6" idref="ref33" type="bt">6</bibl> <bibtext> Chang, H.‐H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20 (3), 213 – 229. https://doi.org/10.1177/014662169602000303</bibtext> </blist> <blist> <bibl id="bib7" idref="ref27" type="bt">7</bibl> <bibtext> Chang, H.‐H., & Ying, Z. (1999). A‐stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23 (3), 211 – 222. https://doi.org/10.1177/01466219922031338</bibtext> </blist> <blist> <bibl id="bib8" idref="ref7" type="bt">8</bibl> <bibtext> Chang, H.‐H., & Ying, Z. (2008). To Weight or Not to Weight? Balancing Influence of Initial Items in Adaptive Testing. Psychometrika, 73 (3), 441 – 450. https://doi.org/10.1007/s11336‐007‐9047‐7</bibtext> </blist> <blist> <bibl id="bib9" idref="ref50" type="bt">9</bibl> <bibtext> Cheng, Y., & Chang, H.‐H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62 (2), 369 – 383. https://doi.org/10.1348/000711008X304376</bibtext> </blist> <blist> <bibtext> Choe, E. M., Kern, J. L., & Chang, H.‐H. (2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43 (2), 135 – 158. https://doi.org/10.3102/1076998617723642</bibtext> </blist> <blist> <bibtext> Choi, S. W., Moellering, K. T., Li, J., & van der Linden, W. J. (2016). Optimal reassembly of shadow tests in CAT. Applied psychological measurement, 40 (7), 469 – 485. https://doi.org/10.1177/0146621616654597</bibtext> </blist> <blist> <bibtext> Debeer, D., & Janssen, R. (2013). Modeling Item‐position effects within an IRT framework. Journal of Educational Measurement, 50 (2), 164 – 185. https://doi.org/10.1111/jedm.12009</bibtext> </blist> <blist> <bibtext> Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling skipped and not‐reached items using IRTrees. Journal of Educational Measurement, 54 (3), 333 – 363. https://doi.org/10.1111/jedm.12147</bibtext> </blist> <blist> <bibtext> Diao, Q., & Van Der Linden, W. J. (2011). Automated test assembly using lp_Solve Version 5.5 in R. Applied Psychological Measurement, 35 (5), 398 – 409. https://doi.org/10.1177/0146621610392211</bibtext> </blist> <blist> <bibtext> Du, Y., Li, A., & Chang, H.‐H. (2019). Utilizing response time in on‐the‐fly multistage adaptive testing. In M. Wiberg, S. Culpepper, R. Janssen, J. González, & D. Molenaar (Eds.), Quantitative psychology (Vol. 265, pp. 107 – 117). Springer International Publishing. https://doi.org/10.1007/978‐3‐030‐01310‐3_10</bibtext> </blist> <blist> <bibtext> Entink, R. H. K., Linden, W. J., & Fox, J.‐P. (2009). A Box‐Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62 (3), 621 – 640. https://doi.org/10.1348/000711008X374126</bibtext> </blist> <blist> <bibtext> Fan, Z., Wang, C., Chang, H.‐H., & Douglas, J. (2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37 (5), 655 – 670. https://doi.org/10.3102/1076998611422912</bibtext> </blist> <blist> <bibtext> Guo, J., Xu, X., & Xin, T. (2023). A note on latent traits estimates under IRT models with missingness. Journal of Educational Measurement, 60 (4), 575 – 625. https://doi.org/10.1111/jedm.12365</bibtext> </blist> <blist> <bibtext> Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26 (2), 44 – 52. https://doi.org/10.1111/j.1745‐3992.2007.00093.x</bibtext> </blist> <blist> <bibtext> Kern, J. L., & Choe, E. (2021). Using a response time–based expected a posteriori estimator to control for differential speededness in computerized adaptive test. Applied Psychological Measurement, 45 (5), 361 – 385. https://doi.org/10.1177/01466216211014601</bibtext> </blist> <blist> <bibtext> Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2 (4), 359 – 375. https://doi.org/10.1207/s15324818ame0204_6</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1980). Applications of item response theory to practical testing problems. L. Erlbaum Associates. <ulink href="http://www.123library.org/book%5fdetails/?id=72568">http://www.123library.org/book%5fdetails/?id=72568</ulink></bibtext> </blist> <blist> <bibtext> Lu, J., & Wang, C. (2020). A response time process model for not‐reached and omitted items. Journal of Educational Measurement, 57 (4), 584 – 620. https://doi.org/10.1111/jedm.12270</bibtext> </blist> <blist> <bibtext> Magis, D., von Davier, A. A., & Yan, D. (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR (1st ed. 2017). Springer International Publishing : Imprint: Springer. https://doi.org/10.1007/978‐3‐319‐69218‐0</bibtext> </blist> <blist> <bibtext> Merritt, J. (2003). Why the folks at ETS flunked the course—A tech‐savvy service will soon be giving B‐school applicants their GMATs. Business Week, Dec. 29.</bibtext> </blist> <blist> <bibtext> OECD. (2018). PISA 2018 Technical Report. Retrieved from https://<ulink href="http://www.oecd.org/pisa/data/pisa2018technicalreport/">www.oecd.org/pisa/data/pisa2018technicalreport/</ulink></bibtext> </blist> <blist> <bibtext> Posit Team. (2024). RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. Retrieved from <ulink href="http://www.posit.co/">http://www.posit.co/</ulink>.</bibtext> </blist> <blist> <bibtext> Rulison, K. L., & Loken, E. (2009). I've fallen and i can't get up: Can high‐ability students recover from early mistakes in CAT? Applied Psychological Measurement, 33 (2), 83 – 101. https://doi.org/10.1177/0146621608324023</bibtext> </blist> <blist> <bibtext> Rutkowski, D., Rutkowski, L., & Liaw, Y. L. (2018). Measuring widening proficiency differences in international assessments: Are current approaches enough? Educational Measurement: Issues and Practice, 37 (4), 40 – 48.</bibtext> </blist> <blist> <bibtext> Stocking, M. L. (1997). revising item responses in computerized adaptive tests: A comparison of three models. Applied Psychological Measurement, 21 (2), 129 – 142. https://doi.org/10.1177/01466216970212003</bibtext> </blist> <blist> <bibtext> Svetina, D., Liaw, Y., Rutkowski, L., & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in international large‐scale assessments. Journal of Educational Measurement, 56 (1), 192 – 213. https://doi.org/10.1111/jedm.12206</bibtext> </blist> <blist> <bibtext> Van Der Linden, W. J. (2005). Linear models for optimal test design. New York : Springer. https://doi.org/10.1007/0‐387‐29054‐0</bibtext> </blist> <blist> <bibtext> van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72 (3), 287 – 308. https://doi.org/10.1007/s11336‐006‐1478‐z</bibtext> </blist> <blist> <bibtext> Van Der Linden, W. J., & Reese, L. M. (1998). A model for optimal constrained adaptive testing. Applied Psychological Measurement, 22 (3), 259 – 270. https://doi.org/10.1177/01466216980223006</bibtext> </blist> <blist> <bibtext> Vispoel, W. P., Hendrickson, A. B., & Bleiler, T. (2000). Limiting answer review and change on computerized adaptive vocabulary tests: Psychometric and attitudinal results. Journal of Educational Measurement, 37 (1), 21 – 38. https://doi.org/10.1111/j.1745‐3984.2000.tb01074.x</bibtext> </blist> <blist> <bibtext> von Davier, M., Sinharay, S., Oranje, A., & Beaton, A. (2006). 32 The statistical procedures used in national assessment of educational progress: Recent developments and future directions. In Handbook of statistics (Vol. 26, pp. 1039 – 1055). Elsevier. https://doi.org/10.1016/S0169‐7161(06)26032‐2</bibtext> </blist> <blist> <bibtext> Wang, S., Lin, H., Chang, H.‐H., & Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design: Hybrid computerized adaptive testing. Journal of Educational Measurement, 53 (1), 45 – 62. https://doi.org/10.1111/jedm.12100</bibtext> </blist> <blist> <bibtext> Wang, C., Chang, H.‐H., & Douglas, J. A. (2013). The linear transformation model with frailties for the analysis of item response times: Linear Transformation Model for RTs. British Journal of Mathematical and Statistical Psychology, 66 (1), 144 – 168. https://doi.org/10.1111/j.2044‐8317.2012.02045.x</bibtext> </blist> <blist> <bibtext> Wang, C., Fan, Z., Chang, H.‐H., & Douglas, J. A. (2013). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38 (4), 381 – 417. https://doi.org/10.3102/1076998612461831</bibtext> </blist> <blist> <bibtext> Wise, S. L. (1996). A critical analysis of the arguments for and against item review in computerized adaptive testing. In Annual Meeting of the National Council on Measurement in Education (NCME), volume 1996.</bibtext> </blist> <blist> <bibtext> Yamamoto, K., Shin, H. J., & Khorramdel, L. (2018). Multistage adaptive testing design in international large‐scale assessments. Educational Measurement: Issues and Practice, 37 (4), 16 – 27. https://doi.org/10.1111/emip.12226</bibtext> </blist> <blist> <bibtext> Zenisky, A., Hambleton, R. K., & Luecht, R. M. (2009). Multistage testing: Issues, designs, and research. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of adaptive testing (pp. 355 – 372). New York : Springer. https://doi.org/10.1007/978‐0‐387‐85461‐8_18</bibtext> </blist> <blist> <bibtext> Zheng, Y., & Chang, H.‐H. (2015). On‐the‐fly assembled multistage adaptive testing. Applied Psychological Measurement, 39 (2), 104 – 118. https://doi.org/10.1177/0146621614544519</bibtext> </blist> <blist> <bibtext> Zheng, Y., Nozawa, Y., Gao, X., & Chang, H.‐H. (2012). Multistage adaptive testing for a large‐scale classification test: Design, heuristic assembly, and comparison with other testing modes. ACT Research Report Series, 2012 (6). ACT, Inc.</bibtext> </blist> <blist> <bibtext> Zheng, Y., Nozawa, Y., Zhu, R., & Gao, X. (2016). Automated top‐down heuristic assembly of a classification multistage test. International Journal of Quantitative Research in Education, 3 (4), 242 – 265.</bibtext> </blist> <blist> <bibtext> Zheng, Y., Wang, C., Culbertson, M. J., & Chang, H. H. (2014). Overview of test assembly methods in multistage testing. In D. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing: Theory and applications (pp. 87 – 99). CRC Press. https://doi.org/10.1201/b16858‐16</bibtext> </blist> </ref> <aug> <p>By Xiuxiu Tang; Yi Zheng; Tong Wu; Kit‐Tai Hau and Hua‐Hua Chang</p> <p>Reported by Author; Author; Author; Author; Author</p> <p></p> <p>Xiuxiu Tang is a Postdoctoral Researcher, University of Notre Dame. Her primary research interests include computerized adaptive testing, process data modeling and cognitive diagnostic modeling.</p> <p>Yi Zheng is an Associate Professor, Arizona State University. Her primary research interests include both the broader topics on measurement and assessment and the more specific topics on advancing methods and technology of test design, delivery, scoring, and analysis.</p> <p>Tong Wu is a Data Scientist, Amazon Web Services. His primary research interests include computerized adaptive testing, response time modeling, online calibration and cognitive diagnostic modeling.</p> <p>Kit‐Tai Hau is a Professor, Department of Educational Psychology, The Chinese University of Hong Kong. His primary research interests include motivation, psychometrics, research methodology, large scale educational monitoring.</p> <p>Hua‐Hua Chang is the Charles R. Hicks Chair Professor, Department of Educational Studies, Purdue University. His research interests include computerized adaptive testing, statistically detecting biased items, cognitive diagnosis, and asymptotic properties in item response theory.</p> </aug> <nolink nlid="nl1" bibid="bib26" firstref="ref4"></nolink> <nolink nlid="nl2" bibid="bib29" firstref="ref5"></nolink> <nolink nlid="nl3" bibid="bib22" firstref="ref6"></nolink> <nolink nlid="nl4" bibid="bib28" firstref="ref8"></nolink> <nolink nlid="nl5" bibid="bib42" firstref="ref9"></nolink> <nolink nlid="nl6" bibid="bib36" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib19" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib46" firstref="ref13"></nolink> <nolink nlid="nl9" bibid="bib32" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib34" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib44" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib45" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib43" firstref="ref22"></nolink> <nolink nlid="nl14" bibid="bib25" firstref="ref25"></nolink> <nolink nlid="nl15" bibid="bib30" firstref="ref29"></nolink> <nolink nlid="nl16" bibid="bib35" firstref="ref30"></nolink> <nolink nlid="nl17" bibid="bib40" firstref="ref31"></nolink> <nolink nlid="nl18" bibid="bib37" firstref="ref32"></nolink> <nolink nlid="nl19" bibid="bib11" firstref="ref34"></nolink> <nolink nlid="nl20" bibid="bib33" firstref="ref38"></nolink> <nolink nlid="nl21" bibid="bib10" firstref="ref39"></nolink> <nolink nlid="nl22" bibid="bib16" firstref="ref42"></nolink> <nolink nlid="nl23" bibid="bib39" firstref="ref43"></nolink> <nolink nlid="nl24" bibid="bib38" firstref="ref44"></nolink> <nolink nlid="nl25" bibid="bib31" firstref="ref48"></nolink> <nolink nlid="nl26" bibid="bib17" firstref="ref52"></nolink> <nolink nlid="nl27" bibid="bib21" firstref="ref57"></nolink> <nolink nlid="nl28" bibid="bib27" firstref="ref62"></nolink> <nolink nlid="nl29" bibid="bib24" firstref="ref63"></nolink> <nolink nlid="nl30" bibid="bib41" firstref="ref65"></nolink> <nolink nlid="nl31" bibid="bib15" firstref="ref67"></nolink> <nolink nlid="nl32" bibid="bib14" firstref="ref80"></nolink> <nolink nlid="nl33" bibid="bib13" firstref="ref83"></nolink> <nolink nlid="nl34" bibid="bib12" firstref="ref84"></nolink> <nolink nlid="nl35" bibid="bib18" firstref="ref85"></nolink> <nolink nlid="nl36" bibid="bib23" firstref="ref86"></nolink> <nolink nlid="nl37" bibid="bib20" firstref="ref87"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1487175
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Utilizing Response Time for Item Selection in On-the-Fly Multistage Adaptive Testing for PISA Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Xiuxiu+Tang%22">Xiuxiu Tang</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-5774-4292">0000-0002-5774-4292</externalLink>)<br /><searchLink fieldCode="AR" term="%22Yi+Zheng%22">Yi Zheng</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2671-0820">0000-0003-2671-0820</externalLink>)<br /><searchLink fieldCode="AR" term="%22Tong+Wu%22">Tong Wu</searchLink><br /><searchLink fieldCode="AR" term="%22Kit-Tai+Hau%22">Kit-Tai Hau</searchLink><br /><searchLink fieldCode="AR" term="%22Hua-Hua+Chang%22">Hua-Hua Chang</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2025 62(3):468-495.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Reaction+Time%22">Reaction Time</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Adaptive+Testing%22">Adaptive Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Tests%22">Reading Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Length%22">Test Length</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Banks%22">Item Banks</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.12403
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Multistage adaptive testing (MST) has been recently adopted for international large-scale assessments such as Programme for International Student Assessment (PISA). MST offers improved measurement efficiency over traditional nonadaptive tests and improved practical convenience over single-item-adaptive computerized adaptive testing (CAT). As a third alternative adaptive test design to MST and CAT, Zheng and Chang proposed the "on-the-fly multistage adaptive testing" (OMST), which combines the benefits of MST and CAT and offsets their limitations. In this study, we adopted the OMST design while also incorporating response time (RT) in item selection. Via simulations emulating the PISA 2018 reading test, including using the real item attributes and replicating PISA 2018 reading test's MST design, we compared the performance of our OMST designs against the simulated MST design in (1) measurement accuracy of test takers' ability, (2) test time efficiency and consistency, and (3) expected gains in precision by design. We also investigated the performance of OMST in item bank usage and constraints management. Results show great potential for the proposed RT-incorporated OMST designs to be used for PISA and potentially other international large-scale assessments.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1487175
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1487175
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.12403
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
        StartPage: 468
    Subjects:
      – SubjectFull: Reaction Time
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Adaptive Testing
        Type: general
      – SubjectFull: Reading Tests
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Test Length
        Type: general
      – SubjectFull: Item Banks
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: Utilizing Response Time for Item Selection in On-the-Fly Multistage Adaptive Testing for PISA Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Xiuxiu Tang
      – PersonEntity:
          Name:
            NameFull: Yi Zheng
      – PersonEntity:
          Name:
            NameFull: Tong Wu
      – PersonEntity:
          Name:
            NameFull: Kit-Tai Hau
      – PersonEntity:
          Name:
            NameFull: Hua-Hua Chang
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 62
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1