View in EDS HTML Full Text PDF Full Text

Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing

Saved in:
Bibliographic Details
Title: Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing
Language: English
Authors: Kylie Gorney (ORCID 0000-0002-8924-0726), Mark D. Reckase
Source: Journal of Educational Measurement. 2025 62(2):360-379.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 20
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Computer Assisted Testing, Adaptive Testing, Test Items, Item Banks, Item Response Theory, Evaluation Methods, Test Use, Measurement, Accuracy
DOI: 10.1111/jedm.12436
ISSN: 0022-0655
1745-3984
Abstract: In computerized adaptive testing, item exposure control methods are often used to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta and Ponsoda), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we develop four extensions of the restricted method that involve the use of multiple maximum exposure rates. A detailed simulation study reveals that (a) all four of the new methods improve item pool utilization and (b) three of the new methods also improve measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both types of outcomes simultaneously.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1475940
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwE8-1BTj5nEJytvVlQogt6ZAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDImK_Mbk-J7Oz4_GegIBEICBm84nZJy181V2-3YeOphrRBY4hVVZnH4Y3b9Y6jW8fRfhUfuXJm4G-Jw8dviK3SSmS5Oi7g04it6XW0oq14nbDSIOwhe8_bcNLnEPgbN9SN5oZRwCmRW_c7rO0rz4tjPexxgWQrwQGvN5cKuKBIgg6l3Nhvx-jzfgndwhWVZdRE2Beub3rHXvz83AHAVisvVYCd_STipn_p6DRlgH
Text:
  Availability: 1
  Value: <anid>AN0186313290;mea01jun.25;2025Jul03.03:58;v2.2.500</anid> <title id="AN0186313290-1">Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing </title> <p>In computerized adaptive testing, item exposure control methods are often used to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta and Ponsoda), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we develop four extensions of the restricted method that involve the use of multiple maximum exposure rates. A detailed simulation study reveals that (a) all four of the new methods improve item pool utilization and (b) three of the new methods also improve measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both types of outcomes simultaneously.</p> <p>In recent decades, computerized adaptive testing (CAT) has become increasingly popular in educational testing (Chang, [<reflink idref="bib2" id="ref1">2</reflink>]). Unlike linear tests, where the same set of items is administered to all examinees, CATs select items adaptively based on an examinee's current ability estimate. There are at least two benefits to creating a test tailored to each examinee. First, measurement efficiency is generally improved since fewer items are needed to achieve the same level of measurement accuracy. Second, test security is improved since different sets of items are administered to different examinees. Importantly, examinees cannot share or disclose information about items they have not seen.</p> <p>Two of the features that distinguish various CAT algorithms are (a) the item selection method and (b) the item exposure control method that are used. The item selection method determines which items are administered to each examinee. Depending on the chosen method, certain items may be administered more frequently than others, resulting in a skewed distribution of item exposure rates. A distribution that is slightly skewed may not be an issue; however, a distribution that is highly skewed is undesirable for at least two reasons. First, it implies that some items are rarely being administered, resulting in an underutilization of costly resources. Second, it implies that other items are being administered quite frequently, potentially posing a threat to the security of the test. For these reasons, item selection methods are often paired with item exposure control methods to provide a more balanced usage of the item pool.</p> <p>Several item exposure control methods have been suggested in the literature, many of which use maximum exposure rates to limit the proportion of times that each item is administered. For example, van der Linden and Veldkamp ([<reflink idref="bib15" id="ref2">15</reflink>]) proposed the item ineligibility method, where the eligibility probability, or the probability that an item is eligible to be selected, decreases if the maximum exposure rate is exceeded. Barrada et al. ([<reflink idref="bib1" id="ref3">1</reflink>]) extended the item ineligibility method by using multiple maximum exposure rates that vary throughout the course of the test. The idea behind their method is to limit the proportion of times that each item is administered in each position of the test so as to provide a more balanced usage of the item pool.</p> <p>However, regardless of whether a single maximum exposure rate is used or multiple maximum exposure rates are used, the item eligibility method is limited in that it does not provide strict control. In other words, it is possible for the observed exposure rates to exceed the maximum exposure rates. In practice, this property seems undesirable, given that the maximum exposure rates are typically thought of as upper limits that we do not wish to exceed. Revuelta and Ponsoda ([<reflink idref="bib12" id="ref4">12</reflink>]) suggested a potential solution to this problem by proposing the restricted method, in which an item cannot be selected if the maximum exposure rate is reached. The restricted method is useful in that it provides strict control over the maximum exposure rate. However, the method has not yet been applied using multiple maximum exposure rates, which presumably would lead to an even more balanced usage of the item pool.</p> <p>The remainder of this paper is organized as follows. In the "Background" section, we review three existing methods for item exposure control. In the "Method" section, we introduce four new methods that (a) are extensions of the restricted method and (b) involve the use of multiple maximum exposure rates. In the "Simulation Study" section, we compare the performance of the new and existing methods with respect to item pool utilization and measurement accuracy. Finally, in the "Discussion" section, we conclude with a brief summary of our findings and identify potential directions for future research.</p> <hd id="AN0186313290-2">Background</hd> <p>Let <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>⋯</mi><mo>,</mo><mi>I</mi></mrow><annotation encoding="application/x-tex">$i=1, \dots, I$</annotation></semantics></math> </ephtml> denote the items in an item pool, and let <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>⋯</mi><mo>,</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">$k=1, \dots, K$</annotation></semantics></math> </ephtml> denote the positions in a test in which the items can be administered. Item selection in CAT typically involves maximizing or minimizing some objective function, where the objective function is determined by both the item selection method and the item exposure control method that are used. For example, if the maximum Fisher information method is used for item selection (Lord, [<reflink idref="bib11" id="ref5">11</reflink>]), maximization of the objective function can be expressed as 1 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><munder><mi>max</mi><mrow><mi>i</mi><mo>∉</mo><mi mathvariant="script">A</mi></mrow></munder><mfenced separators="" open="{" close="}"><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><msub><mi>F</mi><mi>i</mi></msub><mrow><mo>(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>)</mo></mrow></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \max _{i \notin \mathcal {A}} {\left\lbrace w_{ik} F_i(\hat{\theta }) \right\rbrace}, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi mathvariant="script">A</mi><annotation encoding="application/x-tex">$\mathcal {A}$</annotation></semantics></math> </ephtml> is the set of items that have already been administered to the examinee, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">$w_{ik}$</annotation></semantics></math> </ephtml> is a weight given to item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> at position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>F</mi><mi>i</mi></msub><mrow><mo>(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$F_i(\hat{\theta })$</annotation></semantics></math> </ephtml> is the Fisher information of item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> at the current or interim ability estimate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> .</p> <p>The weight <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">$w_{ik}$</annotation></semantics></math> </ephtml> is determined by the item exposure control method that is used. Note that when no item exposure control method is used, it can be assumed that all items are weighted equally at all positions. For example, one can set 2 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mn>1</mn><mspace width="0.33em" /><mtext>for all</mtext><mspace width="0.33em" /><mi>i</mi><mo>,</mo><mi>k</mi><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}=1 \text{ for all } i,k. \end{equation}$$</annotation></semantics></math> </ephtml> In what follows, we will review the weights for three existing item exposure control methods before introducing the weights for our four new methods.</p> <hd id="AN0186313290-3">Using a Single Maximum Exposure Rate</hd> <p>The restricted method (REST; Revuelta & Ponsoda, [<reflink idref="bib12" id="ref6">12</reflink>]) for item exposure control only allows an item to be administered if its observed exposure rate is less than the maximum exposure rate. Thus, the weight for REST is given by 3 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifriobs<rmax,0otherwise,</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if } r_i^{\text{obs}}<r^{\text{max}}, \\ 0 &\text{otherwise,} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_i^{\text{obs}}$</annotation></semantics></math> </ephtml> is the observed exposure rate of item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> (i.e., the proportion of times that item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> has been administered), and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> is the maximum exposure rate.</p> <p>The item eligibility method (ELIG; van der Linden & Veldkamp, [<reflink idref="bib15" id="ref7">15</reflink>]) is similar to REST in that it also involves the use of a single maximum exposure rate. However, whereas REST is a deterministic method for controlling the item exposure rates, ELIG is probabilistic, meaning that a random component is also involved.</p> <p>To implement ELIG, one must first identify which items are eligible to be selected. The probability that item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> is eligible to be selected for examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> is given by 4 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>ε</mi><mi>i</mi><mrow><mo>(</mo><mi>j</mi><mo>)</mo></mrow></msubsup><mo linebreak="badbreak">=</mo><mi>min</mi><mfenced separators="" open="{" close="}"><mfrac><msup><mi>r</mi><mi>max</mi></msup><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup></mfrac><msubsup><mi>ε</mi><mi>i</mi><mrow><mo>(</mo><mi>j</mi><mo>−</mo><mn>1</mn><mo>)</mo></mrow></msubsup><mo>,</mo><mn>1</mn></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \epsilon _i^{(j)}=\min {\left\lbrace \frac{r^{\text{max}}}{r_i^{\text{obs}}} \epsilon _i^{(j-1)}, 1 \right\rbrace}, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>ε</mi><mi>i</mi><mrow><mo>(</mo><mi>j</mi><mo>−</mo><mn>1</mn><mo>)</mo></mrow></msubsup><annotation encoding="application/x-tex">$\epsilon _i^{(j-1)}$</annotation></semantics></math> </ephtml> is the eligibility probability for the previous examinee who took the test. To initialize the procedure, van der Linden and Veldkamp ([<reflink idref="bib15" id="ref8">15</reflink>]) recommended setting <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>ε</mi><mi>i</mi><mrow><mo>(</mo><mi>j</mi><mo>)</mo></mrow></msubsup><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$\epsilon _i^{(j)}=1$</annotation></semantics></math> </ephtml> and keeping it at this value until item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> is administered for the first time (i.e., until <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><mo>></mo><mn>0</mn></mrow><annotation encoding="application/x-tex">$r_i^{\text{obs}}>0$</annotation></semantics></math> </ephtml> ). Once the item has been administered, the eligibility probability can be updated for each subsequent examinee who takes the test. As shown in Equation 4, the eligibility probability typically increases if the observed exposure rate is less than the maximum exposure rate, decreases if the observed exposure rate is greater than the maximum exposure rate, or stays the same if the observed exposure rate is equal to the maximum exposure rate.</p> <p>After all eligibility probabilities have been computed for an examinee, a random number <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>U</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$U_i$</annotation></semantics></math> </ephtml> is drawn from a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="script">U</mi><mo>(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathcal {U}(0,1)$</annotation></semantics></math> </ephtml> distribution for each item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> . An item can only be administered if <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0028" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>U</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$U_i$</annotation></semantics></math> </ephtml> is less than the corresponding eligibility probability. Thus, the weight for ELIG is given by 5 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0029" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifUi<εi(j),0otherwise.</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if } U_i<\epsilon _i^{(j)}, \\ 0 &\text{otherwise.} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> It is important to note that because <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0030" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> must always be greater than 0, the eligibility probabilities will also always be greater than 0 (see Equation 4). As a result, ELIG does not provide strict control over the maximum exposure rate since all items—including those with observed exposure rates that are greater than or equal to the maximum exposure rate—have a chance of being eligible and therefore of being administered.</p> <hd id="AN0186313290-4">Using Multiple Maximum Exposure Rates</hd> <p>Barrada et al. ([<reflink idref="bib1" id="ref9">1</reflink>]) extended ELIG by using multiple maximum exposure rates that vary by item and position (ELIG‐IP). The idea behind their method is to limit the proportion of times that each item is administered in each position of the test so as to provide a more balanced usage of the item pool. They defined the maximum proportion of times that item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0031" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> should be administered in position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0032" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> as 6 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0033" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>r</mi><mrow><mi>i</mi><mi>k</mi></mrow><mi>max</mi></msubsup><mo linebreak="badbreak">=</mo><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><mo linebreak="goodbreak">−</mo><mi>min</mi><mrow><mo>{</mo><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mi>max</mi></msubsup><mo>,</mo><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mtext>obs</mtext></msubsup><mo>}</mo></mrow><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} r_{ik}^{\text{max}}=r_{1 \dots k}^{\text{max}}-\min \lbrace r_{1 \dots k-1}^{\text{max}}, r_{i,1 \dots k-1}^{\text{obs}}\rbrace, \end{equation}$$</annotation></semantics></math> </ephtml> where, for example, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0034" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k-1}^{\text{max}}$</annotation></semantics></math> </ephtml> is the maximum proportion of times that an item should be administered in positions 1 to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0035" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$k-1$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0036" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_{i,1 \dots k-1}^{\text{obs}}$</annotation></semantics></math> </ephtml> is the observed proportion of times that item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0037" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> has been administered in positions 1 to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0038" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$k-1$</annotation></semantics></math> </ephtml> .</p> <p>To implement ELIG‐IP, one must choose the values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0039" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0040" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>⋯</mi><mo>,</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">$k=1, \dots, K$</annotation></semantics></math> </ephtml> . Barrada et al. ([<reflink idref="bib1" id="ref10">1</reflink>]) provided several recommendations as to how this can be done. They stated that the random selection of items at the beginning of the test should provide a more balanced usage of the item pool while having a limited impact on measurement accuracy. Therefore, they recommended that the first value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0041" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> be chosen to reflect random item selection. Barrada et al. also recommended that the last value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0042" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> be chosen to reflect the security requirements of the testing program. Therefore, they suggested that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0043" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>K</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots K}^{\text{max}}$</annotation></semantics></math> </ephtml> be set equal to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0044" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> , which is the value that would have been chosen had a single maximum exposure rate been applied. Following these recommendations, we set 7 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0045" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><mo linebreak="badbreak">=</mo><mfrac><mn>1</mn><mi>I</mi></mfrac><mo linebreak="goodbreak">+</mo><mfrac><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mrow><mi>K</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mfenced separators="" open="(" close=")"><msup><mi>r</mi><mi>max</mi></msup><mo>−</mo><mfrac><mn>1</mn><mi>I</mi></mfrac></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} r_{1 \dots k}^{\text{max}}=\frac{1}{I} + \frac{k-1}{K-1} {\left(r^{\text{max}}-\frac{1}{I} \right)} \end{equation}$$</annotation></semantics></math> </ephtml> so that as the test progresses and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0046" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> increases, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0047" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> increases linearly from <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0048" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mfrac><mn>1</mn><mi>I</mi></mfrac><annotation encoding="application/x-tex">$\frac{1}{I}$</annotation></semantics></math> </ephtml> (which represents random item selection) to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0049" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> (which represents the security requirements of the testing program).</p> <p>After the maximum exposure rates have been determined, the eligibility probabilities can be computed. The probability that item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0050" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> is eligible to be selected in position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0051" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> for examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0052" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> is given by 8 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0053" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>ε</mi><mrow><mi>i</mi><mi>k</mi></mrow><mrow><mo>(</mo><mi>j</mi><mo>)</mo></mrow></msubsup><mo linebreak="badbreak">=</mo><mi>min</mi><mfenced separators="" open="{" close="}"><mfrac><msubsup><mi>r</mi><mrow><mi>i</mi><mi>k</mi></mrow><mi>max</mi></msubsup><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mtext>obs</mtext></msubsup></mfrac><msubsup><mi>ε</mi><mrow><mi>i</mi><mi>k</mi></mrow><mrow><mo>(</mo><mi>j</mi><mo>−</mo><mn>1</mn><mo>)</mo></mrow></msubsup><mo>,</mo><mn>1</mn></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \epsilon _{ik}^{(j)}=\min {\left\lbrace \frac{r_{ik}^{\text{max}}}{r_{i,1 \dots k}^{\text{obs}}} \epsilon _{ik}^{(j-1)}, 1 \right\rbrace}, \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0054" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>ε</mi><mrow><mi>i</mi><mi>k</mi></mrow><mrow><mo>(</mo><mi>j</mi><mo>−</mo><mn>1</mn><mo>)</mo></mrow></msubsup><annotation encoding="application/x-tex">$\epsilon _{ik}^{(j-1)}$</annotation></semantics></math> </ephtml> is the eligibility probability for the previous examinee who took the test. To initialize the procedure, one can set <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0055" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>ε</mi><mrow><mi>i</mi><mi>k</mi></mrow><mrow><mo>(</mo><mi>j</mi><mo>)</mo></mrow></msubsup><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">$\epsilon _{ik}^{(j)}=1$</annotation></semantics></math> </ephtml> and keep it at this value until item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0056" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> is administered in positions 1 to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0057" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> for the first time (i.e., until <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0058" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mtext>obs</mtext></msubsup><mo>></mo><mn>0</mn></mrow><annotation encoding="application/x-tex">$r_{i,1 \dots k}^{\text{obs}}>0$</annotation></semantics></math> </ephtml> ). Once this occurs, the eligibility probability can be updated for each subsequent examinee who takes the test.</p> <p>After all eligibility probabilities have been computed for an examinee, a random number <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0059" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>U</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">$U_{ik}$</annotation></semantics></math> </ephtml> is drawn from a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0060" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="script">U</mi><mo>(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathcal {U}(0,1)$</annotation></semantics></math> </ephtml> distribution for each combination of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0061" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0062" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> . An item can only be administered in a given position if <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0063" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>U</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">$U_{ik}$</annotation></semantics></math> </ephtml> is less than the corresponding eligibility probability. Thus, the weight for ELIG‐IP is given by 9 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0064" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifUik<εik(j),0otherwise,</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if}\ U_{ik}<\epsilon _{ik}^{(j)}, \\ 0 &\text{otherwise,} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> which parallels the weight for ELIG that is given by Equation 5.</p> <hd id="AN0186313290-5">Method</hd> <p>In the previous section, we reviewed three existing item exposure control methods: the restricted method with a single maximum exposure rate (REST), the item eligibility method with a single maximum exposure rate (ELIG), and the item eligibility method with multiple maximum exposure rates that vary by item and position (ELIG‐IP). In this section, we propose four new item exposure control methods, all of which are extensions of the restricted method that involve the use of multiple maximum exposure rates. The first three methods use techniques similar to that of Barrada et al. ([<reflink idref="bib1" id="ref11">1</reflink>]), while the fourth method uses an even simpler approach.</p> <hd id="AN0186313290-6">Using Multiple Maximum Exposure Rates That Vary by Item and Position</hd> <p>When a single maximum exposure rate is used with either the restricted method or the item eligibility method, the following quantities are required: <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0065" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_i^{\text{obs}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0066" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> . In REST, the two quantities are directly compared to determine whether an item is eligible to be selected (Equation 3). In ELIG, a ratio of the two quantities is used to compute the eligibility probability (Equation 4).</p> <p>Barrada et al. ([<reflink idref="bib1" id="ref12">1</reflink>]) extended the item eligibility method by using multiple maximum exposure rates that vary by item and position. In their method (ELIG‐IP), a ratio of the following quantities is used to compute the eligibility probability (Equation 8): <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0067" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_{i,1 \dots k}^{\text{obs}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0068" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{ik}^{\text{max}}$</annotation></semantics></math> </ephtml> . Therefore, one way to extend the restricted method is by directly comparing these quantities instead. We refer to this extension as <emph>the restricted method with multiple maximum exposure rates that vary by item and position</emph> (REST‐IP). The corresponding weight is given by 10 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0069" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifri,1⋯kobs<rikmax,0otherwise.</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if}\ r_{i,1 \dots k}^{\text{obs}}< r_{ik}^{\text{max}}, \\ 0 &\text{otherwise.} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>However, in looking at Equation 10, a reasonable question to ask is whether it is fair to compare <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0070" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_{i,1 \dots k}^{\text{obs}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0071" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{ik}^{\text{max}}$</annotation></semantics></math> </ephtml> . Note that the first quantity, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0072" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mo>,</mo><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mtext>obs</mtext></msubsup><annotation encoding="application/x-tex">$r_{i,1 \dots k}^{\text{obs}}$</annotation></semantics></math> </ephtml> , is computed across positions 1 to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0073" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> , while the second quantity, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0074" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mi>i</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{ik}^{\text{max}}$</annotation></semantics></math> </ephtml> , is defined for position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0075" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> only. As a result, the use of this weight is expected to limit the exposure of items more often than necessary, potentially resulting in an unnecessary sacrifice in measurement accuracy.</p> <p>As two possible alternatives, we consider additional extensions of the restricted method that involve the use of fairer comparisons. We refer to these extensions as REST‐IP2 and REST‐IP3. The corresponding weights are given by 11 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0076" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifrikobs<rikmax,0otherwise,</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if}\ r_{ik}^{\text{obs}}< r_{ik}^{\text{max}}, \\ 0 &\text{otherwise,} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> and 12 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0077" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifri,1⋯kobs<r1⋯kmax,0otherwise,</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if}\ r_{i,1 \dots k}^{\text{obs}}< r_{1 \dots k}^{\text{max}}, \\ 0 &\text{otherwise,} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> respectively.</p> <hd id="AN0186313290-7">Using Multiple Maximum Exposure Rates That Vary by Position Only</hd> <p>Thus far, we have considered four item exposure control methods that use multiple maximum exposure rates: ELIG‐IP, REST‐IP, REST‐IP2, and REST‐IP3. All four methods attempt to provide a more balanced usage of the item pool by limiting the proportion of times that each item is administered in each position of the test. However, it seems reasonable to ask whether this precise level of control is actually necessary. From a test security standpoint, it matters little whether an item is administered in the first position or last position of the test—at the end of the day, the item has still been exposed.</p> <p>Given the argument above, it seems justified to limit the proportion of times that each item is administered across <emph>all</emph> positions of the test rather than the proportion of times that each item is administered in <emph>each</emph> position of the test. This is, in fact, the approach that is taken by methods that use a single maximum exposure rate, such as REST and ELIG. However, if the same maximum exposure rate is applied across all positions, then an additional aspect to consider is that the popular items—when eligible—will likely be administered in earlier positions of the test. For example, if the maximum Fisher information method is used for item selection, items with large discrimination parameters will tend to be administered first. Yet, Chang and Ying ([<reflink idref="bib5" id="ref13">5</reflink>]) have shown that from a measurement perspective, it is actually advantageous to administer less discriminating items at the beginning of the test (when less is known about an examinee's ability) and to reserve more discriminating items for the end of the test (when more is known about an examinee's ability). Researchers such as Chang et al. ([<reflink idref="bib3" id="ref14">3</reflink>]), Chang and Ying ([<reflink idref="bib4" id="ref15">4</reflink>]), and Revuelta and Ponsoda ([<reflink idref="bib12" id="ref16">12</reflink>]) have proposed item selection methods that achieve this desired ordering of items. In what follows, we show that this ordering can also be achieved through the use of multiple maximum exposure rates.</p> <p>The aims of the current approach are as follows: (a) to limit the proportion of times that each item is administered across all positions of the test, (b) to promote the selection of less popular items at the beginning of the test, and (c) to reserve the selection of more popular items for the end of the test. To satisfy all three conditions, we propose an item exposure control method in which a different maximum exposure rate is applied at each position of the test. Then, an item is only allowed to be administered in a given position if its observed exposure rate is less than the maximum exposure rate that is applied at that position. The weight for the proposed method is given by 13 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0078" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mo linebreak="badbreak">=</mo><mfenced separators="" open="{" close="">1ifriobs<rkmax,0otherwise,</mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} w_{ik}={\begin{cases} 1 &\text{if}\ r_i^{\text{obs}}< r_k^{\text{max}}, \\ 0 &\text{otherwise,} \end{cases}} \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0079" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> is the maximum exposure rate that is applied at position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0080" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> . Because a different maximum exposure rate is specified for each position of the test (and not for each combination of item and position, as is the case for REST‐IP, REST‐IP2, and REST‐IP3), we refer to this method as <emph>the restricted method with multiple maximum exposure rates that vary by position only</emph> (REST‐P).</p> <p>To implement REST‐P, one must choose the values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0081" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0082" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>=</mo><mn>1</mn><mo>,</mo><mi>⋯</mi><mo>,</mo><mi>K</mi></mrow><annotation encoding="application/x-tex">$k=1, \dots, K$</annotation></semantics></math> </ephtml> . Following Barrada et al. ([<reflink idref="bib1" id="ref17">1</reflink>]), we set the first value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0083" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> to reflect random item selection, set the last value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0084" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> to reflect the security requirements of the testing program, and set the values in between to increase accordingly. Thus, the maximum exposure rate to be applied at position <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0085" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> is defined as 14 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0086" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><mo linebreak="badbreak">=</mo><mfrac><mi>K</mi><mi>I</mi></mfrac><mo linebreak="goodbreak">+</mo><mfrac><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mrow><mi>K</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mfenced separators="" open="(" close=")"><msup><mi>r</mi><mi>max</mi></msup><mo>−</mo><mfrac><mi>K</mi><mi>I</mi></mfrac></mfenced><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} r_k^{\text{max}}=\frac{K}{I}+\frac{k-1}{K-1}{\left(r^{\text{max}}-\frac{K}{I} \right)}. \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>To examine the impact of using Equation 14 to define the maximum exposure rates, we computed the values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0087" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> for an item pool of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0088" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>=</mo><mn>400</mn></mrow><annotation encoding="application/x-tex">$I=400$</annotation></semantics></math> </ephtml> items, a test length of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0089" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>40</mn></mrow><annotation encoding="application/x-tex">$K=40$</annotation></semantics></math> </ephtml> items, and an <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0090" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> equal to .20. We then computed the corresponding weights (using Equation 13) for items with different observed exposure rates at different positions of the test. The weights are shown in the left‐hand plot of Figure 1, where a weight of 1 indicates that an item is eligible to be administered in a given position, and a weight of 0 indicates that an item is not eligible to be administered. The plot reveals that most of the weights at the beginning of the test are equal to 0, meaning that most items are not eligible to be administered in these positions. The only items that are eligible are those with small observed exposure rates. In this way, the method ensures that less popular items will be administered at the beginning of the test. The plot also reveals that as the test progresses, more weights become equal to 1, meaning that more items are eligible to be administered. Note, however, that items with larger observed exposure rates are still not eligible to be administered until much later on. In this way, the method ensures that popular items are reserved for the end of the test, as is desired.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0001.jpg" title="1 Weights for REST‐P and REST (I=400$I=400$, K=40$K=40$, rmax=.20$r^{\text{max}}=.20$)." /> </p> <p></p> <p>For comparison, the weights for the original method, REST, are shown in the right‐hand plot of Figure 1. The plot reveals that the same weights are applied across all positions of the test. As a result, popular items—which tend to be administered as soon as they are available—will typically be administered earlier in the test, which is the exact opposite of what is desired.</p> <hd id="AN0186313290-9">Simulation Study</hd> <p></p> <hd id="AN0186313290-10">Design and Analysis</hd> <p>We conducted a detailed simulation study to compare the performance of the new and existing item exposure control methods. To mimic realistic CAT conditions, the size of the item pool was fixed at <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0094" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>=</mo><mn>400</mn></mrow><annotation encoding="application/x-tex">$I=400$</annotation></semantics></math> </ephtml> items, and the following factors were manipulated: test length (<reflink idref="bib20" id="ref18">20</reflink>, 30, 40), <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0095" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> (.20, .25, .30), and item exposure control method (NONE, REST, REST‐IP, REST‐IP2, REST‐IP3, REST‐P). The weights for each of the item exposure control methods are given by Equations 2, 3, 10, 11, 12, and 13, respectively.</p> <p>The three factors were fully crossed, resulting in a total of 54 ( <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0096" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>3</mn><mo>×</mo><mn>3</mn><mo>×</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">$3 \times 3 \times 6$</annotation></semantics></math> </ephtml> ) conditions. For each condition, the following steps were repeated 100 times:</p> <p></p> <ulist> <item> 1. Generate person and item parameters for the three‐parameter logistic (3PL) model. Under the 3PL model, the probability that examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0097" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> will answer item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0098" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> correctly is assumed to be <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0099" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mrow><mo>(</mo><msub><mi>X</mi><mrow><mi>j</mi><mi>i</mi></mrow></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>|</mo><msub><mi>θ</mi><mi>j</mi></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><msub><mi>c</mi><mi>i</mi></msub><mo linebreak="goodbreak">+</mo><mrow><mo>(</mo><mn>1</mn><mo linebreak="goodbreak">−</mo><msub><mi>c</mi><mi>i</mi></msub><mo>)</mo></mrow><mfrac><mrow><mi>exp</mi><mo>[</mo><msub><mi>a</mi><mi>i</mi></msub><mrow><mo>(</mo><msub><mi>θ</mi><mi>j</mi></msub><mo>−</mo><msub><mi>b</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>]</mo></mrow><mrow><mn>1</mn><mo>+</mo><mi>exp</mi><mo>[</mo><msub><mi>a</mi><mi>i</mi></msub><mrow><mo>(</mo><msub><mi>θ</mi><mi>j</mi></msub><mo>−</mo><msub><mi>b</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>]</mo></mrow></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} P(X_{ji}=1|\theta _j)=c_i+(1-c_i) \frac{\exp [a_i(\theta _j-b_i)]}{1+\exp [a_i(\theta _j-b_i)]}, \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0100" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>θ</mi><mi>j</mi></msub><annotation encoding="application/x-tex">$\theta _j$</annotation></semantics></math> </ephtml> is the ability parameter of examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0101" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0102" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$a_i$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0103" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$b_i$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0104" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$c_i$</annotation></semantics></math> </ephtml> are the discrimination, difficulty, and pseudo‐guessing parameters, respectively, of item <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0105" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> . Sample <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0106" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo>=</mo><mn>10</mn><mo>,</mo><mn>000</mn></mrow><annotation encoding="application/x-tex">$J=10,000$</annotation></semantics></math> </ephtml> person parameters such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0107" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>θ</mi><mi>j</mi></msub><mo>∼</mo><mi mathvariant="script">N</mi><mrow><mo>(</mo><mn>0</mn><mo>,</mo><mn>1</mn><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\theta _j \sim \mathcal {N}(0,1)$</annotation></semantics></math> </ephtml> . Sample the item discrimination and difficulty parameters such that <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0108" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfenced open="[" close="]">ai∗bi</mfenced><mo>∼</mo><mi mathvariant="script">N</mi><mfenced separators="" open="(" close=")"><mfenced open="[" close="]">−.20.00</mfenced><mo>,</mo><mfenced open="[" close="]">.10.10.101.00</mfenced></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \def\eqcellsep{&}\begin{bmatrix} a_i^* \\ b_i \end{bmatrix} \sim \mathcal {N}{\left(\def\eqcellsep{&}\begin{bmatrix} -.20 \\.00 \end{bmatrix}, \def\eqcellsep{&}\begin{bmatrix}.10 &.10 \\.10 & 1.00 \end{bmatrix} \right)}, \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0109" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>a</mi><mi>i</mi><mo>∗</mo></msubsup><mo>=</mo><mi>log</mi><mrow><mo>(</mo><msub><mi>a</mi><mi>i</mi></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$a_i^*=\log (a_i)$</annotation></semantics></math> </ephtml> . Sample the item pseudo‐guessing parameters such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0110" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>c</mi><mi>i</mi></msub><mo>∼</mo><mi mathvariant="script">U</mi><mrow><mo>(</mo><mo>.</mo><mn>05</mn><mo>,</mo><mspace width="0.16em" /><mo>.</mo><mn>30</mn><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$c_i \sim \mathcal {U}(.05,\,.30)$</annotation></semantics></math> </ephtml> . Note that the item parameter distributions were chosen to be similar to those of the operational item pool in Table 1 of Han ([<reflink idref="bib8" id="ref19">8</reflink>]).</item> <p></p> <item> 2. Administer the CAT by selecting the items according to Equation 1. For each examinee, select the first item using an initial ability estimate of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0111" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">$\hat{\theta }=0$</annotation></semantics></math> </ephtml> . Select the remaining items using the interim ability estimates that are computed using maximum likelihood estimation and are bounded between <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0112" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>−</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">$-4$</annotation></semantics></math> </ephtml> and 4.</item> <p></p> <item> 3. Compute the following measures of item pool utilization:</item> <p></p> <item> (a) The proportion of overexposed items, given by <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0113" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>OVER</mi><mo linebreak="badbreak">=</mo><mfrac><mn>1</mn><mi>I</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>I</mi></munderover><mrow><mn mathvariant="bold">1</mn></mrow><mrow><mo>(</mo><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><mo linebreak="goodbreak">></mo><msup><mi>r</mi><mi>max</mi></msup><mo>)</mo></mrow><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \textit{OVER} = \frac{1}{I} \sum _{i=1}^I \bm{1}(r_i^{\text{obs}} > r^{\text{max}}), \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0114" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mn mathvariant="bold">1</mn></mrow><mo>(</mo><mo>·</mo><mo>)</mo></mrow><annotation encoding="application/x-tex">$\bm{1}(\cdot)$</annotation></semantics></math> </ephtml> is an indicator function.</item> <p></p> <item> (b) The maximum of the observed exposure rates, given by <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0115" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>MAX</mi><mo linebreak="badbreak">=</mo><munder><mi>max</mi><mrow><mn>1</mn><mo linebreak="badbreak">≤</mo><mi>i</mi><mo linebreak="goodbreak">≤</mo><mi>I</mi></mrow></munder><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \textit{MAX} = \underset{1 \le i \le I}{\max } r_i^{\text{obs}}. \end{equation*}$$</annotation></semantics></math> </ephtml></item> <p></p> <item> (c) The balance of the observed exposure rates, given by the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0116" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> statistic (Chang & Ying, [<reflink idref="bib4" id="ref20">4</reflink>]) as <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0117" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>χ</mi><mn>2</mn></msup><mo linebreak="badbreak">=</mo><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>I</mi></munderover><mfrac><msup><mrow><mo>(</mo><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><mo>−</mo><mover accent="true"><mi>r</mi><mo>¯</mo></mover><mo>)</mo></mrow><mn>2</mn></msup><mover accent="true"><mi>r</mi><mo>¯</mo></mover></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \chi ^2=\sum _{i=1}^I \frac{(r_i^{\text{obs}} - \bar{r})^2}{\bar{r}}, \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0118" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>r</mi><mo>¯</mo></mover><mo>=</mo><mfrac><mi>K</mi><mi>I</mi></mfrac></mrow><annotation encoding="application/x-tex">$\bar{r}=\frac{K}{I}$</annotation></semantics></math> </ephtml> is the average observed exposure rate. Note that smaller values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0119" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> indicate better balance.</item> <p></p> <item> (d) The average test overlap rate between two examinees (Chen et al., [<reflink idref="bib6" id="ref21">6</reflink>]), given by <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0120" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover><mi>TOR</mi><mo>¯</mo></mover><mo linebreak="badbreak">=</mo><msup><mfenced separators="" open="(" close=")"><mfrac linethickness="0pt"><mi>J</mi><mn>2</mn></mfrac></mfenced><mrow><mo>−</mo><mn>1</mn></mrow></msup><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>J</mi><mo>−</mo><mn>1</mn></mrow></munderover><munderover><mo>∑</mo><mrow><msup><mi>j</mi><mo>′</mo></msup><mo>=</mo><mi>j</mi><mo>+</mo><mn>1</mn></mrow><mi>J</mi></munderover><msub><mi>TOR</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow></msub><mo linebreak="goodbreak">=</mo><mfrac><mrow><mi>J</mi><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>I</mi></msubsup><msup><mrow><mo>(</mo><msubsup><mi>r</mi><mi>i</mi><mtext>obs</mtext></msubsup><mo>)</mo></mrow><mn>2</mn></msup></mrow><mrow><mi>K</mi><mo>(</mo><mi>J</mi><mo>−</mo><mn>1</mn><mo>)</mo></mrow></mfrac><mo linebreak="goodbreak">−</mo><mfrac><mn>1</mn><mrow><mi>J</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \overline{\textit{TOR}} = {J \atopwithdelims ()2}^{-1} \sum _{j=1}^{J-1} \sum _{j^\prime =j+1}^J \textit{TOR}_{jj^\prime } = \frac{J \sum _{i=1}^I (r_i^{\text{obs}})^2}{K(J-1)}-\frac{1}{J-1}, \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0121" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>TOR</mi><mrow><mi>j</mi><msup><mi>j</mi><mo>′</mo></msup></mrow></msub><annotation encoding="application/x-tex">$\textit{TOR}_{jj^\prime }$</annotation></semantics></math> </ephtml> is the test overlap rate between examinees <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0122" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0123" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>j</mi><mo>′</mo></msup><annotation encoding="application/x-tex">$j^\prime$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0124" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>J</mi><annotation encoding="application/x-tex">$J$</annotation></semantics></math> </ephtml> is the total number of examinees.</item> <p></p> <item> 4. Compute the following measures of measurement accuracy:</item> <p></p> <item> (a) The bias of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0125" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> , given by <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0126" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>BIAS</mi><mo linebreak="badbreak">=</mo><mfrac><mn>1</mn><mi>J</mi></mfrac><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></munderover><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>j</mi></msub><mo linebreak="goodbreak">−</mo><msub><mi>θ</mi><mi>j</mi></msub><mo>)</mo></mrow><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \textit{BIAS}=\frac{1}{J} \sum _{j=1}^J (\hat{\theta }_j - \theta _j), \end{equation*}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0127" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>j</mi></msub><annotation encoding="application/x-tex">$\hat{\theta }_j$</annotation></semantics></math> </ephtml> is the final ability estimate of examinee <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0128" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>j</mi><annotation encoding="application/x-tex">$j$</annotation></semantics></math> </ephtml> .</item> <p></p> <item> (b) The root mean squared error (RMSE) of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0129" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> , given by <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0130" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>RMSE</mi><mo linebreak="badbreak">=</mo><msqrt><mrow><mfrac><mn>1</mn><mi>J</mi></mfrac><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>J</mi></msubsup><msup><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>j</mi></msub><mo>−</mo><msub><mi>θ</mi><mi>j</mi></msub><mo>)</mo></mrow><mn>2</mn></msup></mrow></msqrt><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} \textit{RMSE}=\sqrt {\frac{1}{J} \sum _{j=1}^J (\hat{\theta }_j - \theta _j)^2}. \end{equation*}$$</annotation></semantics></math> </ephtml></item> <p></p> <item> (c) The correlation between <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0131" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0132" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> .</item> </ulist> <p>1 Table Measures of Item Pool Utilization and Measurement Accuracy</p> <p> <ephtml> <table><thead><tr><th /><th align="center">Item Pool Utilization</th><th align="center">Measurement Accuracy</th></tr><tr><th>Method</th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0133" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>OVER</mi><annotation encoding="application/x-tex">$\textit{OVER}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0134" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>MAX</mi><annotation encoding="application/x-tex">$\textit{MAX}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0135" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0136" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mover><mi>TOR</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline{\textit{TOR}}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0137" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>BIAS</mi><annotation encoding="application/x-tex">$\textit{BIAS}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0138" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>RMSE</mi><annotation encoding="application/x-tex">$\textit{RMSE}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0139" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>CORR</mi><annotation encoding="application/x-tex">$\textit{CORR}$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>NONE</td><td>.124</td><td>1.000</td><td>162.0</td><td>.480</td><td>.001</td><td>.392</td><td>.932</td></tr><tr><td /><td>(.005)</td><td>(.000)</td><td>(5.0)</td><td>(.013)</td><td>(.005)</td><td>(.011)</td><td>(.004)</td></tr><tr><td>REST</td><td>.000</td><td>.250</td><td>59.1</td><td>.223</td><td>.002</td><td>.451</td><td>.913</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(.5)</td><td>(.001)</td><td>(.005)</td><td>(.009)</td><td>(.003)</td></tr><tr><td>REST‐IP</td><td>.003</td><td>.202</td><td>19.6</td><td>.124</td><td>.000</td><td>.503</td><td>.896</td></tr><tr><td /><td>(.001)</td><td>(.015)</td><td>(.4)</td><td>(.001)</td><td>(.006)</td><td>(.009)</td><td>(.004)</td></tr><tr><td>REST‐IP2</td><td>.170</td><td>.252</td><td>53.8</td><td>.209</td><td>.002</td><td>.447</td><td>.914</td></tr><tr><td /><td>(.009)</td><td>(.001)</td><td>(.5)</td><td>(.001)</td><td>(.005)</td><td>(.009)</td><td>(.004)</td></tr><tr><td>REST‐IP3</td><td>.146</td><td>.251</td><td>56.3</td><td>.216</td><td>.001</td><td>.442</td><td>.916</td></tr><tr><td /><td>(.014)</td><td>(.001)</td><td>(.5)</td><td>(.001)</td><td>(.005)</td><td>(.010)</td><td>(.003)</td></tr><tr><td>REST‐P</td><td>.000</td><td>.250</td><td>32.7</td><td>.157</td><td>.003</td><td>.443</td><td>.915</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(.1)</td><td>(.000)</td><td>(.005)</td><td>(.010)</td><td>(.004)</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>. <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0140" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>OVER</mi><annotation encoding="application/x-tex">$\textit{OVER}$</annotation></semantics></math> </ephtml>  = proportion of overexposed items, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0141" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>MAX</mi><annotation encoding="application/x-tex">$\textit{MAX}$</annotation></semantics></math> </ephtml>  = maximum of the observed exposure rates, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0142" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml>  = balance of the observed exposure rates, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0143" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover><mi>TOR</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline{\textit{TOR}}$</annotation></semantics></math> </ephtml>  = average test overlap rate</p> <p>After all replications were completed, the mean and standard deviation of each of the outcome measures were computed to obtain the final results. Code for replicating the analysis is available on the Open Science Framework at https://osf.io/58fce.</p> <hd id="AN0186313290-11">Results</hd> <p>Table 1 displays the mean (and standard deviation) of each of the outcome measures, averaged across test length and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0144" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> .[<reflink idref="bib1" id="ref22">1</reflink>] Each pair of rows corresponds to a different item exposure control method, and each column corresponds to a different outcome measure.</p> <hd id="AN0186313290-12">Item pool utilization</hd> <p>Table 1 reveals that of all the item exposure control methods, only REST and REST‐P have zero overexposed items and are therefore able to provide strict control over <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0145" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> . Initially, this result may seem surprising, given that one might have expected REST‐IP, REST‐IP2, and REST‐IP3 to also provide strict control. However, one problem with these methods is that they each involve computing the observed proportion of times that an item has been administered <emph>up to the current position</emph>. As a result, the proportion of times that an item has been administered in any future positions is not considered. To help understand why this could be an issue, consider the case where an item has only been administered in the last position of the test. Suppose the item has been administered so many times that it has reached the maximum exposure rate, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0146" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> . Clearly, for the next examinee, the item should not be administered in the last position of the test. In fact, it should not be administered at all. However, according to REST‐IP, REST‐IP2, and REST‐IP3, the item can still be administered, as long as it is administered in an earlier position of the test. In this way, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0147" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> can easily be exceeded if any one of these methods is used.</p> <p>In addition to providing information on the proportion of overexposed items, Table 1 also displays the maximum of the observed exposure rates. By examining these results, we can gain a sense of how much—if at all—the observed exposure rates exceed the maximum exposure rate. Note that because Table 1 contains <emph>average</emph> results, it is desirable for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0148" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>MAX</mi><annotation encoding="application/x-tex">$\textit{MAX}$</annotation></semantics></math> </ephtml> to be less than or equal to .25, which is the average value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0149" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> across all the conditions. The table reveals that of the six methods, only REST, REST‐IP, and REST‐P achieve this goal. In contrast, REST‐IP2 and REST‐IP3 are shown to produce observed exposure rates that slightly exceed the average maximum exposure rate of .25, while NONE is shown to produce observed exposure rates that greatly exceed .25 and in fact are equal to 1.</p> <p>Table 1 also displays the results for two additional measures of item pool utilization: the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0150" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math> </ephtml> statistic and the average test overlap rate. The table reveals that REST‐IP produces the smallest values and therefore performs best with respect to both measures, a result that is not surprising if we recall that this method involves the use of an unfair comparison that limits the exposure of items more often than is necessary (see Equation 10). After REST‐IP, it can be seen that REST‐P performs best with respect to both measures, followed by REST‐IP2, REST‐IP3, REST, and then NONE. The fact that REST‐P outperforms both REST‐IP2 and REST‐IP3 indicates that it is not necessary to limit the proportion of times that each item is administered in <emph>each</emph> position of the test in order to improve test security. Instead, it is sufficient to limit the proportion of times that each item is administered across <emph>all</emph> positions of the test while also using multiple maximum exposure rates.</p> <hd id="AN0186313290-13">Measurement accuracy</hd> <p>The last three columns of Table 1 display the results for the three measures of measurement accuracy. The table reveals that the bias for all six methods seems reasonable; hence, we focus our discussion on the RMSE and the correlation. As expected, NONE produces the smallest value of the former and the largest value of the latter and therefore performs best with respect to both measures. After NONE, it can be seen that REST‐P, REST‐IP2, and REST‐IP3 perform similarly to one another, followed by REST and then REST‐IP. Taken together, these results show that the use of multiple maximum exposure rates rather than a single maximum exposure rate typically improves measurement accuracy. The only exception occurs when REST‐IP is used, a result that is not surprising given how severely REST‐IP limits the exposure of items.</p> <p>To examine how the methods perform for examinees of different ability levels, Figure 2 separates the bias and RMSE results by quintile. Quintiles were formed by placing examinees with the smallest <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0151" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi><mi mathvariant="normal">s</mi></mrow><annotation encoding="application/x-tex">$\theta{\rm s}$</annotation></semantics></math> </ephtml> in Quintile 1, placing examinees with the largest <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0152" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi><mi mathvariant="normal">s</mi></mrow><annotation encoding="application/x-tex">$\theta{\rm s}$</annotation></semantics></math> </ephtml> in Quintile 5, and arranging all other examinees accordingly. The figure reveals that all six methods tend to produce final ability estimates that are negatively biased for low‐ability examinees and positively biased for high‐ability examinees. In addition, all six methods tend to produce RMSE values that are larger for low‐ability examinees and smaller for high‐ability examinees. This result is not surprising given that the item discrimination and item difficulty parameters were simulated to be positively correlated.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0002.jpg" title="2 Measures of measurement accuracy by quintile." /> </p> <p></p> <hd id="AN0186313290-15">Item ordering</hd> <p>Lastly, given Chang and Ying's ([<reflink idref="bib5" id="ref23">5</reflink>]) statement that it is advantageous to administer less discriminating items at the beginning of the test (when less is known about an examinee's ability) and to reserve more discriminating items for the end of the test (when more is known about an examinee's ability), we computed the average discrimination parameter for the items that were administered at each position of the test. Figure 3 displays the results for an item pool of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0153" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>=</mo><mn>400</mn></mrow><annotation encoding="application/x-tex">$I=400$</annotation></semantics></math> </ephtml> items, a test length of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0154" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>40</mn></mrow><annotation encoding="application/x-tex">$K=40$</annotation></semantics></math> </ephtml> items, and an <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0155" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> equal to .20. The results for the other test lengths and other values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0156" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> were found to be similar and are not shown here.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0003.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0003.jpg" title="3 Average discrimination parameter by position (I=400$I=400$, K=40$K=40$, rmax=.20$r^{\text{max}}=.20$)." /> </p> <p></p> <p>Figure 3 reveals that when REST‐P is used, the average discrimination parameter increases as the test progresses. In other words, less discriminating items tend to be administered earlier in the test, while more discriminating items tend to be administered later in the test, as is desired. In contrast, when NONE or REST is used, the average discrimination parameter decreases as the test progresses. In other words, more discriminating items tend to be administered earlier in the test, while less discriminating items tend to be administered later in the test, which is the exact opposite of what is desired. Finally, when REST‐IP, REST‐IP2, or REST‐IP3 is used, the average discrimination parameter stays consistent as the test progresses. Taken together, these results show how the choice of item exposure control method influences the order in which items are administered.</p> <hd id="AN0186313290-17">Summary</hd> <p>From the simulation results, it can be seen that each item exposure control method has its own benefits and drawbacks. Although the use of no item exposure control method (NONE) ensures the best performance in terms of measurement accuracy, it also produces extremely skewed distributions of item exposure rates, yields unacceptably large average test overlap rates, and permits the overexposure of items. The use of any of the restricted methods (REST, REST‐IP, REST‐IP2, REST‐IP3, REST‐P) improves item pool utilization; however, these improvements differ in magnitude and their effect on measurement accuracy.</p> <p>Compared to the original restricted method (REST), all four of the new methods (REST‐IP, REST‐IP2, REST‐IP3, REST‐P) provide a more balanced usage of the item pool. However, only three of the methods (REST‐IP2, REST‐IP3, REST‐P) also improve measurement accuracy. Of these three methods, REST‐P produces the least skewed distributions of item exposure rates, yields the smallest average test overlap rates, and is the only method to prevent the overexposure of items. Thus, REST‐P seems to produce the most favorable results overall.</p> <hd id="AN0186313290-18">Discussion</hd> <p>Item exposure control methods are often used in CAT to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta & Ponsoda, [<reflink idref="bib12" id="ref24">12</reflink>]), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. ([<reflink idref="bib1" id="ref25">1</reflink>]) showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we developed four extensions of the restricted method that involve the use of multiple maximum exposure rates. The first three methods (REST‐IP, REST‐IP2, REST‐IP3) require specifying a different maximum exposure rate for each combination of item and position. The fourth method (REST‐P) uses an even simpler approach that only requires specifying a different maximum exposure rate for each position.</p> <p>To compare the performance of the four new methods (REST‐IP, REST‐IP2, REST‐IP3, REST‐P) with two existing methods (NONE, REST), we conducted a detailed simulation study. Performance was evaluated with respect to item pool utilization and measurement accuracy. As expected, all four of the new methods (REST‐IP, REST‐IP2, REST‐IP3, REST‐P) were found to provide a more balanced usage of the item pool than the original restricted method (REST). In addition, three of the new methods (REST‐IP2, REST‐IP3, REST‐P) also improved measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both item pool utilization and measurement accuracy simultaneously.</p> <p>Of the three methods (REST‐IP2, REST‐IP3, REST‐P) that were found to improve both item pool utilization and measurement accuracy, REST‐P appeared to produce the most favorable results overall. Not only did REST‐P produce the least skewed distributions of item exposure rates and yield the smallest average test overlap rates, but it was also the only method (of the three) to completely prevent the overexposure of items.</p> <p>An additional finding from the simulation study was that REST‐P is the only item exposure control method to follow Chang and Ying's ([<reflink idref="bib5" id="ref26">5</reflink>]) recommendation of administering less discriminating items at the beginning of the test and reserving more discriminating items for the end of the test. Researchers such as Chang et al. ([<reflink idref="bib3" id="ref27">3</reflink>]), Chang and Ying ([<reflink idref="bib4" id="ref28">4</reflink>]), and Revuelta and Ponsoda ([<reflink idref="bib12" id="ref29">12</reflink>]) have proposed item selection methods that also achieve this desired ordering of items. To compare the performance of our method to theirs, we conducted a small simulation study. Six methods were included in the comparison: the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0160" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified method (AS; Chang & Ying, [<reflink idref="bib4" id="ref30">4</reflink>]), the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0161" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0162" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>b</mi><annotation encoding="application/x-tex">$b$</annotation></semantics></math> </ephtml> ‐blocking method (ASB; Chang et al., [<reflink idref="bib3" id="ref31">3</reflink>]), the progressive method (PROG; Revuelta & Ponsoda, [<reflink idref="bib12" id="ref32">12</reflink>]), and three versions of REST‐P (applied using different values of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0163" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> ). As shown in Appendix B, REST‐P appears to be a promising alternative to AS, ASB, and PROG.</p> <p>There are several limitations to this paper, providing many opportunities for future research. First, to keep the size and scope of our study manageable, we only examined a limited number of simulation conditions. In the future, it would be interesting to examine additional conditions, including those where the new item exposure control methods are compared to other popular item exposure control methods such as the Sympson‐Hetter method (Sympson & Hetter, [<reflink idref="bib14" id="ref33">14</reflink>]), the randomesque method (Kingsbury & Zara, [<reflink idref="bib10" id="ref34">10</reflink>]), or the fade‐away method (Han, [<reflink idref="bib7" id="ref35">7</reflink>]), other item selection methods such as the continuous <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0164" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified method (Huebner et al., [<reflink idref="bib9" id="ref36">9</reflink>]), or other combinations of item selection and item exposure control methods such as the restrictive progressive method or the restrictive threshold method (Wang et al., [<reflink idref="bib17" id="ref37">17</reflink>]). It would also be interesting to study how the methods perform when used with content balancing.</p> <p>Second, it is possible to develop modifications of the proposed methods. For example, REST‐P could be modified by using a different function (i.e., a function other than Equation 14) to define <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0165" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> . Similarly, REST‐IP, REST‐IP2, and REST‐IP3 could be modified by using a different function (i.e., a function other than Equation 7) to define <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0166" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> . One possibility is to use a function that includes an acceleration parameter, as in Barrada et al. ([<reflink idref="bib1" id="ref38">1</reflink>]). The acceleration parameter would control the rate at which <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0167" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mi>k</mi><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_k^{\text{max}}$</annotation></semantics></math> </ephtml> (or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0168" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msubsup><mi>r</mi><mrow><mn>1</mn><mi>⋯</mi><mi>k</mi></mrow><mi>max</mi></msubsup><annotation encoding="application/x-tex">$r_{1 \dots k}^{\text{max}}$</annotation></semantics></math> </ephtml> ) increases as the test progresses.</p> <p>Finally, it is possible to develop extensions of the proposed methods. For example, one challenge in CAT is that examinees with similar ability levels tend to have higher test overlap rates with one another than with other examinees. The proposed methods could be extended so that the item exposure rates are controlled within each ability group rather than just at the aggregate level (e.g., Stocking & Lewis, [<reflink idref="bib13" id="ref39">13</reflink>]; van der Linden & Veldkamp, [<reflink idref="bib16" id="ref40">16</reflink>]). It is also possible to extend the proposed methods to different types of tests. All analyses in this paper were performed on fixed‐length CATs, where all examinees are administered the same number of items. It would be interesting to see if extensions could be developed that are effective for variable‐length CATs, where different examinees are administered different numbers of items.</p> <hd id="AN0186313290-19">A Appendix</hd> <p>Table 1 displays the results for each of the outcome measures <emph>averaged</emph> across test length and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> . However, it is also important to separate the results by these factors to study how the methods perform for different types of tests.</p> <p>Tables A1 and A2 and Figures A1 and A2 display the four measures of item pool utilization and the three measures of measurement accuracy for nine combinations of test length and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> . The tables and figures reveal that the relative performance of the methods is mostly consistent across conditions, suggesting that the conclusions that were drawn at the aggregate level hold regardless of test length or the choice of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> .</p> <p>A1 Table Proportion of Overexposed Items</p> <p> <ephtml> <table><thead><tr><th /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>20</mn></mrow><annotation encoding="application/x-tex">$K=20$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>30</mn></mrow><annotation encoding="application/x-tex">$K=30$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>40</mn></mrow><annotation encoding="application/x-tex">$K=40$</annotation></semantics></math></p></th></tr><tr><th>Method</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th></tr></thead><tbody><tr><td>NONE</td><td>.095</td><td>.083</td><td>.071</td><td>.140</td><td>.123</td><td>.110</td><td>.182</td><td>.163</td><td>.146</td></tr><tr><td>REST</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td></tr><tr><td>REST‐IP</td><td>.000</td><td>.000</td><td>.000</td><td>.002</td><td>.000</td><td>.000</td><td>.020</td><td>.001</td><td>.000</td></tr><tr><td>REST‐IP2</td><td>.157</td><td>.092</td><td>.078</td><td>.224</td><td>.146</td><td>.129</td><td>.320</td><td>.206</td><td>.175</td></tr><tr><td>REST‐IP3</td><td>.094</td><td>.065</td><td>.050</td><td>.193</td><td>.136</td><td>.097</td><td>.318</td><td>.212</td><td>.150</td></tr><tr><td>REST‐P</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td><td>.000</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note</emph>. The column headings .20, .25, and .30 represent different values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml></p> <p>A2 Table Maximum of the Observed Exposure Rates</p> <p> <ephtml> <table><thead><tr><th /><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>20</mn></mrow><annotation encoding="application/x-tex">$K=20$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>30</mn></mrow><annotation encoding="application/x-tex">$K=30$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>K</mi><mo>=</mo><mn>40</mn></mrow><annotation encoding="application/x-tex">$K=40$</annotation></semantics></math></p></th></tr><tr><th>Method</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th><th align="center">.20</th><th align="center">.25</th><th align="center">.30</th></tr></thead><tbody><tr><td>NONE</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td><td>1.000</td></tr><tr><td>REST</td><td>.200</td><td>.250</td><td>.300</td><td>.200</td><td>.250</td><td>.300</td><td>.200</td><td>.250</td><td>.300</td></tr><tr><td>REST‐IP</td><td>.154</td><td>.158</td><td>.169</td><td>.200</td><td>.197</td><td>.197</td><td>.258</td><td>.245</td><td>.237</td></tr><tr><td>REST‐IP2</td><td>.202</td><td>.251</td><td>.301</td><td>.202</td><td>.252</td><td>.302</td><td>.204</td><td>.253</td><td>.302</td></tr><tr><td>REST‐IP3</td><td>.201</td><td>.251</td><td>.301</td><td>.202</td><td>.251</td><td>.301</td><td>.202</td><td>.251</td><td>.301</td></tr><tr><td>REST‐P</td><td>.200</td><td>.250</td><td>.300</td><td>.200</td><td>.250</td><td>.300</td><td>.200</td><td>.250</td><td>.300</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note</emph>. The column headings .20, .25, and .30 represent different values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0004.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0004.jpg" title="A1 Measures of item pool utilization." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0005.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0005.jpg" title="A2 Measures of measurement accuracy." /> </p> <p></p> <hd id="AN0186313290-22">B Appendix</hd> <p>Chang and Ying ([<reflink idref="bib5" id="ref41">5</reflink>]) showed that it is advantageous to administer less discriminating items at the beginning of the test and to reserve more discriminating items for the end of the test. While REST‐P uses multiple maximum exposure rates to achieve this desired ordering of items, researchers such as Chang et al. ([<reflink idref="bib3" id="ref42">3</reflink>]), Chang and Ying ([<reflink idref="bib4" id="ref43">4</reflink>]), and Revuelta and Ponsoda ([<reflink idref="bib12" id="ref44">12</reflink>]) have shown that this ordering can also be achieved through the use of certain item selection methods.</p> <p>Chang and Ying ([<reflink idref="bib4" id="ref45">4</reflink>]) proposed the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified method (AS) for item selection. The method first involves partitioning the item pool into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> strata, where items with the smallest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> parameters are placed in Stratum 1, items with the largest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> parameters are placed in Stratum <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> , and items in between are arranged accordingly. Next, the test is partitioned into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> stages, where each stage corresponds to a different stratum. Thus, at Stage <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>s</mi><annotation encoding="application/x-tex">$s$</annotation></semantics></math> </ephtml> , only the items in Stratum <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>s</mi><annotation encoding="application/x-tex">$s$</annotation></semantics></math> </ephtml> can be selected. In this way, the method ensures that the least discriminating items are administered at the beginning of the test, while the most discriminating items are reserved for the end of the test. Finally, to select the items within each stage, Chang and Ying suggested minimizing the distance between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$b_i$</annotation></semantics></math> </ephtml> . Thus, maximization of the objective function for AS can be expressed as</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><munder><mi>max</mi><mrow><mi>i</mi><mo>∉</mo><mi mathvariant="script">A</mi></mrow></munder><mfenced separators="" open="{" close="}"><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mfenced separators="" open="(" close=")"><msub><mi>v</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mrow><mrow><mo stretchy="false">|</mo></mrow><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>−</mo><msub><mi>b</mi><mi>i</mi></msub><mrow><mo stretchy="false">|</mo></mrow></mrow></mfenced></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \max _{i \notin \mathcal {A}} {\left\lbrace w_{ik} {\left(\frac{v_{ik}}{|\hat{\theta }-b_i|} \right)} \right\rbrace}, \end{equation}$$</annotation></semantics></math> </ephtml> </p> <p>where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>v</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><annotation encoding="application/x-tex">$v_{ik}$</annotation></semantics></math> </ephtml> is an indicator variable that is equal to 1 if the stratum of item <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>i</mi><annotation encoding="application/x-tex">$i$</annotation></semantics></math> </ephtml> matches the stage of position <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> and 0 otherwise.</p> <p>Chang et al. ([<reflink idref="bib3" id="ref46">3</reflink>]) refined the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified method by proposing the <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> ‐stratified with <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>b</mi><annotation encoding="application/x-tex">$b$</annotation></semantics></math> </ephtml> ‐blocking method (ASB). The method first involves dividing the item pool into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>B</mi><annotation encoding="application/x-tex">$B$</annotation></semantics></math> </ephtml> blocks, where items with the smallest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>b</mi><annotation encoding="application/x-tex">$b$</annotation></semantics></math> </ephtml> parameters are placed in Block 1, items with the largest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>b</mi><annotation encoding="application/x-tex">$b$</annotation></semantics></math> </ephtml> parameters are placed in Block <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>B</mi><annotation encoding="application/x-tex">$B$</annotation></semantics></math> </ephtml> , and items in between are arranged accordingly. Second, each block is partitioned into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> strata, where items with the smallest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> parameters are placed in Stratum 1, items with the largest <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>a</mi><annotation encoding="application/x-tex">$a$</annotation></semantics></math> </ephtml> parameters are placed in Stratum <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> , and items in between are arranged accordingly. Third, the strata are combined across blocks. Fourth, the test is partitioned into <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>S</mi><annotation encoding="application/x-tex">$S$</annotation></semantics></math> </ephtml> stages, where each stage corresponds to a different stratum. Finally, within each stage, items are selected by minimizing the distance between <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">$b_i$</annotation></semantics></math> </ephtml> . Thus, ASB can also be expressed using Equation 15.</p> <p>Revuelta and Ponsoda ([<reflink idref="bib12" id="ref47">12</reflink>]) proposed the progressive method (PROG) for item selection. The method selects the item that maximizes a weighted average of the Fisher information <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>F</mi><mi>i</mi></msub><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow></mrow><annotation encoding="application/x-tex">$F_i(\hat{\theta })$</annotation></semantics></math> </ephtml> and a random component <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>R</mi><mi>i</mi></msub><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow></mrow><annotation encoding="application/x-tex">$R_i(\hat{\theta })$</annotation></semantics></math> </ephtml> , where the random component is drawn from a uniform distribution that is bounded by 0 and the maximum Fisher information of the items that have not been administered to the examinee. Maximization of the objective function for PROG can be expressed as</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><munder><mi>max</mi><mrow><mi>i</mi><mo>∉</mo><mi mathvariant="script">A</mi></mrow></munder><mfenced separators="" open="{" close="}"><msub><mi>w</mi><mrow><mi>i</mi><mi>k</mi></mrow></msub><mfenced separators="" open="(" close=")"><mrow><mo stretchy="false">(</mo><mn>1</mn><mo linebreak="goodbreak">−</mo><msub><mi>s</mi><mi>k</mi></msub><mo stretchy="false">)</mo></mrow><msub><mi>R</mi><mi>i</mi></msub><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow><mo>+</mo><msub><mi>s</mi><mi>k</mi></msub><msub><mi>F</mi><mi>i</mi></msub><mrow><mo stretchy="false">(</mo><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo stretchy="false">)</mo></mrow></mfenced></mfenced><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \max _{i \notin \mathcal {A}} {\left\lbrace w_{ik} {\left((1-s_k) R_i(\hat{\theta })+s_k F_i(\hat{\theta }) \right)} \right\rbrace}, \end{equation}$$</annotation></semantics></math> </ephtml> </p> <p>where <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>s</mi><mi>k</mi></msub><annotation encoding="application/x-tex">$s_k$</annotation></semantics></math> </ephtml> is a weight that determines how much emphasis should be placed on each component at position <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> . Revuelta and Ponsoda set</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>k</mi></msub><mo linebreak="badbreak">=</mo><mrow><mi>k</mi><mo>−</mo><mn>1</mn></mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">$$\begin{equation} s_k=\frac{k-1}{K} \end{equation}$$</annotation></semantics></math> </ephtml> </p> <p>so that as the test progresses and <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>k</mi><annotation encoding="application/x-tex">$k$</annotation></semantics></math> </ephtml> increases, <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>s</mi><mi>k</mi></msub><annotation encoding="application/x-tex">$s_k$</annotation></semantics></math> </ephtml> increases linearly from 0 to <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>−</mo><mn>1</mn></mrow><mi>K</mi><annotation encoding="application/x-tex">$\frac{K-1}{K}$</annotation></semantics></math> </ephtml> , reflecting an increased emphasis on the Fisher information component (and a decreased emphasis on the random component) as the test progresses.</p> <p>We conducted a small simulation study to compare the performance of AS, ASB, PROG, and REST‐P. To mimic realistic CAT conditions, the size of the item pool was fixed at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mo>=</mo><mn>400</mn></mrow><annotation encoding="application/x-tex">$I=400$</annotation></semantics></math> </ephtml> items and the test length was fixed at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>=</mo><mn>40</mn></mrow><annotation encoding="application/x-tex">$K=40$</annotation></semantics></math> </ephtml> items. The only factors that were manipulated were the methods used for item selection and item exposure control. For PROG, no item exposure control method was used, as in Revuelta and Ponsoda ([<reflink idref="bib12" id="ref48">12</reflink>]). For AS, no item exposure control method was used and the number of strata was fixed at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">$S=4$</annotation></semantics></math> </ephtml> , as in Chang and Ying ([<reflink idref="bib4" id="ref49">4</reflink>]). For ASB, no item exposure control method was used, the number of strata was fixed at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">$S=4$</annotation></semantics></math> </ephtml> , and the number of blocks was fixed at <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>B</mi><mo>=</mo><mn>100</mn></mrow><annotation encoding="application/x-tex">$B=100$</annotation></semantics></math> </ephtml> , similar to Chang et al. ([<reflink idref="bib3" id="ref50">3</reflink>]). For REST‐P, the maximum Fisher information method was used for item selection and three different values of <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> were considered (.20, .25, .30), as in the main study. For each condition, 100 replications were conducted.</p> <p>Table B1 displays the mean (and standard deviation) of each of the outcome measures. Figure B1 displays the average discrimination parameter for the items that were administered at each position of the test. The table and figure reveal the following:</p> <p></p> <ulist> <item> Compared to AS, all three versions of REST‐P provide a similar or more balanced usage of the item pool while also improving measurement accuracy.</item> <p></p> <item> Compared to ASB, two of the three versions of REST‐P ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>20</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.20$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>25</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.25$</annotation></semantics></math> </ephtml> ) provide a similar or more balanced usage of the item pool while also improving measurement accuracy. The third version of REST‐P ( <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>30</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.30$</annotation></semantics></math> </ephtml> ) improves measurement accuracy but provides a less balanced usage of the item pool.</item> <p></p> <item> Compared to PROG, all three versions of REST‐P provide a more balanced usage of the item pool but reduce measurement accuracy.</item> <p></p> <item> All methods except PROG tend to administer less discriminating items earlier in the test and more discriminating items later in the test, as is desired.</item> </ulist> <p>In summary, REST‐P appears to be a promising alternative to AS, ASB, and PROG.</p> <p>B1 Table Measures of Item Pool Utilization and Measurement Accuracy</p> <p> <ephtml> <table><thead><tr><th /><th align="center">Item Pool Utilization</th><th align="center">Measurement Accuracy</th></tr><tr><th>Method</th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>OVER</mi><annotation encoding="application/x-tex">$\textit{OVER}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>MAX</mi><annotation encoding="application/x-tex">$\textit{MAX}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msup><mi>χ</mi><mn>2</mn></msup><annotation encoding="application/x-tex">$\chi ^2$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mover><mi>TOR</mi><mo>¯</mo></mover><annotation encoding="application/x-tex">$\overline{\textit{TOR}}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>BIAS</mi><annotation encoding="application/x-tex">$\textit{BIAS}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>RMSE</mi><annotation encoding="application/x-tex">$\textit{RMSE}$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>CORR</mi><annotation encoding="application/x-tex">$\textit{CORR}$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>AS</td><td>.000</td><td>1.000</td><td>37.4</td><td>.193</td><td>.018</td><td>.452</td><td>.911</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(2.5)</td><td>(.006)</td><td>(.006)</td><td>(.008)</td><td>(.003)</td></tr><tr><td>ASB</td><td>.000</td><td>1.000</td><td>28.0</td><td>.170</td><td>−.005</td><td>.458</td><td>.909</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(1.6)</td><td>(.004)</td><td>(.007)</td><td>(.009)</td><td>(.003)</td></tr><tr><td>PROG</td><td>.000</td><td>.822</td><td>136.0</td><td>.441</td><td>−.000</td><td>.341</td><td>.948</td></tr><tr><td /><td>(.000)</td><td>(.023)</td><td>(4.7)</td><td>(.012)</td><td>(.004)</td><td>(.008)</td><td>(.002)</td></tr><tr><td>REST‐P (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>20</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.20$</annotation></semantics></math></p>)</td><td>.000</td><td>.200</td><td>19.1</td><td>.148</td><td>.001</td><td>.396</td><td>.930</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(.0)</td><td>(.000)</td><td>(.004)</td><td>(.008)</td><td>(.003)</td></tr><tr><td>REST‐P (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>25</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.25$</annotation></semantics></math></p>)</td><td>.000</td><td>.250</td><td>28.5</td><td>.171</td><td>.002</td><td>.383</td><td>.934</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(.1)</td><td>(.000)</td><td>(.004)</td><td>(.008)</td><td>(.003)</td></tr><tr><td>REST‐P (<p><math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msup><mi>r</mi><mi>max</mi></msup><mo>=</mo><mo>.</mo><mn>30</mn></mrow><annotation encoding="application/x-tex">$r^{\text{max}}=.30$</annotation></semantics></math></p>)</td><td>.000</td><td>.300</td><td>37.6</td><td>.194</td><td>.001</td><td>.374</td><td>.937</td></tr><tr><td /><td>(.000)</td><td>(.000)</td><td>(.2)</td><td>(.000)</td><td>(.004)</td><td>(.007)</td><td>(.002)</td></tr></tbody></table> </ephtml> </p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01jun25/jedm12436-fig-0006.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm12436-fig-0006.jpg" title="B1 Average discrimination parameter by position." /> </p> <p></p> <ref id="AN0186313290-24"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref3" type="bt">1</bibl> <bibtext> Appendix A separates the results for nine combinations of test length and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm12436:jedm12436-math-0236" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msup><mi>r</mi><mi>max</mi></msup><annotation encoding="application/x-tex">$r^{\text{max}}$</annotation></semantics></math> </ephtml> .</bibtext> </blist> </ref> <ref id="AN0186313290-25"> <title> References </title> <blist> <bibtext> Barrada, J. R., Veldkamp, B. P., & Olea, J. (2009). Multiple maximum exposure rates in computerized adaptive testing. Applied Psychological Measurement, 33 (1), 58 – 73. https://doi.org/10.1177/0146621608315329</bibtext> </blist> <blist> <bibl id="bib2" idref="ref1" type="bt">2</bibl> <bibtext> Chang, H.‐H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80 (1), 1 – 20. https://doi.org/10.1007/S11336‐014‐9401‐5</bibtext> </blist> <blist> <bibl id="bib3" idref="ref14" type="bt">3</bibl> <bibtext> Chang, H.‐H., Qian, J., & Ying, Z. (2001). a ‐stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25 (4), 333 – 341. https://doi.org/10.1177/01466210122032181</bibtext> </blist> <blist> <bibl id="bib4" idref="ref15" type="bt">4</bibl> <bibtext> Chang, H.‐H., & Ying, Z. (1999). a ‐stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23 (3), 211 – 222. https://doi.org/10.1177/01466219922031338</bibtext> </blist> <blist> <bibl id="bib5" idref="ref13" type="bt">5</bibl> <bibtext> Chang, H.‐H., & Ying, Z. (2008). To weight or not to weight? Balancing influence of initial items in adaptive testing. Psychometrika, 73 (3), 441 – 450. https://doi.org/10.1007/s11336‐007‐9047‐7</bibtext> </blist> <blist> <bibl id="bib6" idref="ref21" type="bt">6</bibl> <bibtext> Chen, S.‐Y., Ankenmann, R. D., & Spray, J. A. (2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40 (2), 129 – 145. https://doi.org/10.1111/j.1745‐3984.2003.tb01100.x</bibtext> </blist> <blist> <bibl id="bib7" idref="ref35" type="bt">7</bibl> <bibtext> Han, K. T. (2009). A gradual maximum information ratio approach to item selection in computerized adaptive testing (Research Report RR‐09‐07). Graduate Management Admission Council.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref19" type="bt">8</bibl> <bibtext> Han, K. T. (2012). An efficiency balanced information criterion for item selection in computerized adaptive testing. Journal of Educational Measurement, 49 (3), 225 – 246. https://doi.org/10.1111/j.1745‐3984.2012.00173.x</bibtext> </blist> <blist> <bibl id="bib9" idref="ref36" type="bt">9</bibl> <bibtext> Huebner, A., Wang, C., Daly, B., & Pinkelman, C. (2018). A continuous a ‐stratification index for item exposure control in computerized adaptive testing. Applied Psychological Measurement, 42 (7), 523 – 537. https://doi.org/10.1177/0146621618758289</bibtext> </blist> <blist> <bibtext> Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2 (4), 359 – 375. https://doi.org/10.1207/s15324818ame0204_6</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35 (4), 311 – 327. https://doi.org/10.1111/j.1745‐3984.1998.tb00541.x</bibtext> </blist> <blist> <bibtext> Stocking, M. L., & Lewis, C. (1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 23 (1), 57 – 75. https://doi.org/10.3102/10769986023001057</bibtext> </blist> <blist> <bibtext> Sympson, J. B., & Hetter, R. D. (1985). Controlling item‐exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association (pp. 973 – 977). Navy Personnel Research and Development Center.</bibtext> </blist> <blist> <bibtext> van der Linden, W. J., & Veldkamp, B. P. (2004). Constraining item exposure in computerized adaptive testing with shadow tests. Journal of Educational and Behavioral Statistics, 29 (3), 273 – 291. https://doi.org/10.3102/10769986029003273</bibtext> </blist> <blist> <bibtext> van der Linden, W. J., & Veldkamp, B. P. (2007). Conditional item‐exposure control in adaptive testing using item‐ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32 (4), 398 – 418. https://doi.org/10.3102/1076998606298044</bibtext> </blist> <blist> <bibtext> Wang, C., Chang, H.‐H., & Huebner, A. (2011). Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 48 (3), 255 – 273. https://doi.org/10.1111/j.1745‐3984.2011.00145.x</bibtext> </blist> </ref> <aug> <p>By Kylie Gorney and Mark D. Reckase</p> <p>Reported by Author; Author</p> <p></p> <p>KYLIE GORNEY is Assistant Professor of Measurement and Quantitative Methods at Michigan State University, 460 Erickson Hall, 620 Farm Lane, East Lansing, MI 48824; kgorney@msu.edu. Her primary research interests include test security, item response theory, and computerized adaptive testing.</p> <p>MARK D. RECKASE is University Distinguished Professor Emeritus of Measurement and Quantitative Methods at Michigan State University; reckase@msu.edu. His primary research interests include computerized adaptive testing, standard setting, and multidimensional item response theory.</p> </aug> <nolink nlid="nl1" bibid="bib15" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib12" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib11" firstref="ref5"></nolink> <nolink nlid="nl4" bibid="bib20" firstref="ref18"></nolink> <nolink nlid="nl5" bibid="bib14" firstref="ref33"></nolink> <nolink nlid="nl6" bibid="bib10" firstref="ref34"></nolink> <nolink nlid="nl7" bibid="bib17" firstref="ref37"></nolink> <nolink nlid="nl8" bibid="bib13" firstref="ref39"></nolink> <nolink nlid="nl9" bibid="bib16" firstref="ref40"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1475940
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Kylie+Gorney%22">Kylie Gorney</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-8924-0726">0000-0002-8924-0726</externalLink>)<br /><searchLink fieldCode="AR" term="%22Mark+D%2E+Reckase%22">Mark D. Reckase</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2025 62(2):360-379.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 20
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Adaptive+Testing%22">Adaptive Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Banks%22">Item Banks</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Use%22">Test Use</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.12436
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In computerized adaptive testing, item exposure control methods are often used to provide a more balanced usage of the item pool. Many of the most popular methods, including the restricted method (Revuelta and Ponsoda), use a single maximum exposure rate to limit the proportion of times that each item is administered. However, Barrada et al. showed that by using multiple maximum exposure rates, it is possible to obtain an even more balanced usage of the item pool. Therefore, in this paper, we develop four extensions of the restricted method that involve the use of multiple maximum exposure rates. A detailed simulation study reveals that (a) all four of the new methods improve item pool utilization and (b) three of the new methods also improve measurement accuracy. Taken together, these results are highly encouraging, as they reveal that it is possible to improve both types of outcomes simultaneously.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1475940
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1475940
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.12436
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
        StartPage: 360
    Subjects:
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Adaptive Testing
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Item Banks
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Test Use
        Type: general
      – SubjectFull: Measurement
        Type: general
      – SubjectFull: Accuracy
        Type: general
    Titles:
      – TitleFull: Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kylie Gorney
      – PersonEntity:
          Name:
            NameFull: Mark D. Reckase
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 62
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1