View in EDS HTML Full Text PDF Full Text

IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices

Saved in:
Bibliographic Details
Title: IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices
Language: English
Authors: Tim Moses, YoungKoung Kim
Source: Journal of Educational Measurement. 2025 62(4):718-739.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 22
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Item Response Theory, Scoring, Reliability, Accuracy, Probability, Statistical Distributions, Test Length
DOI: 10.1111/jedm.70008
ISSN: 0022-0655
1745-3984
Abstract: This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT-based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability estimates for 1-, 2-, and 3-parameter logistic IRT models (1PL, 2PL, and 3PL) for tests of dichotomously scored items, using IRT calibrations from two datasets. The generalized recursion procedure is shown to produce conditional probability distributions for the considered IRT estimators that can be used in the estimation of marginal reliabilities and conditional accuracies (biases and CSEMs). These reliabilities and conditional accuracies are shown to have less extreme and more plausible values compared to theoretical approaches based on test information. The proposed recursion procedure for the estimation of reliability and other accuracy measures are demonstrated for testing situations involving different test lengths, IRT models, and different types of IRT parameter inaccuracies.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1491360
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEq44Lsbn0t-6mgRLv8F6XGAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJ8olBxDd5y7TsPrAgIBEICBmwuscqRDPlbVSETeCaaGVlArLXlYOa81KO0LLbEm8_iUvfHsLP8ocJzAtuRcHpzqMEGj1O5eVETe5NHXaqccam079ZlrUo3CSHrnqvw4xSYN-psBpmDGtzzFw9khqO6Gbk45AkbMGwTYi6NlpJpS2GDnEcBWs-3wW76wwDysL8y0ihufm4GqHWQLBXJ8HGEIyWBlE6BTbLW5M52b
Text:
  Availability: 1
  Value: <anid>AN0190280393;mea01dec.25;2025Dec18.05:08;v2.2.500</anid> <title id="AN0190280393-1">IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices </title> <sbt id="AN0190280393-2">Introduction</sbt> <p>This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT‐based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability estimates for 1‐, 2‐, and 3‐parameter logistic IRT models (1PL, 2PL, and 3PL) for tests of dichotomously scored items, using IRT calibrations from two datasets. The generalized recursion procedure is shown to produce conditional probability distributions for the considered IRT estimators that can be used in the estimation of marginal reliabilities and conditional accuracies (biases and CSEMs). These reliabilities and conditional accuracies are shown to have less extreme and more plausible values compared to theoretical approaches based on test information. The proposed recursion procedure for the estimation of reliability and other accuracy measures are demonstrated for testing situations involving different test lengths, IRT models, and different types of IRT parameter inaccuracies.</p> <p>Obtaining test scores and their accuracy indices based on Item Response Theory (IRT) can be challenging in practical applications. Samples of examinees and items obtained in practical testing situations can be small, resulting in IRT‐based estimates that do not satisfy asymptotic conditions, and may not resemble their true values. Ad hoc practices might be sought to replace estimates of extreme values with values that are more reasonable.</p> <p>When maximum likelihood estimation (MLE) is used for IRT‐based scoring, examinees' abilities are estimated as those that reflect maximum test information (Lord, [<reflink idref="bib18" id="ref1">18</reflink>]). At extremely high or low scores, MLE ability estimates can be infinite or undefined (Kim & Moses, [<reflink idref="bib10" id="ref2">10</reflink>]; Kolen, [<reflink idref="bib11" id="ref3">11</reflink>]; Yen & Fitzpatrick, [<reflink idref="bib28" id="ref4">28</reflink>]). These extreme MLE scores and score information values might be replaced with arbitrary values in practice (e.g., Smarter Balanced, [<reflink idref="bib26" id="ref5">26</reflink>]). The resulting indices derived from score information that reflects some arbitrary values can be arbitrarily high or low (i.e., estimated test reliability). In addition, MLE estimates are biased versions of their true values, such that estimates are systematically too low for low ability examinees and systematically high for high ability examinees (Lord, [<reflink idref="bib19" id="ref6">19</reflink>]; Warm, [<reflink idref="bib27" id="ref7">27</reflink>]). Bayesian scoring approaches tend to produce estimates with bias patterns that are opposite to, and of larger magnitudes than those of MLEs for high or low ability examinees (Kolen, [<reflink idref="bib11" id="ref8">11</reflink>]; Lord, [<reflink idref="bib20" id="ref9">20</reflink>]). Finally, test characteristic curve (TCC) estimates are noted to exhibit less bias than MLE scores (Lord, [<reflink idref="bib19" id="ref10">19</reflink>]), but are less efficient and reliable, except for long tests (Yen & Fitzpatrick, [<reflink idref="bib28" id="ref11">28</reflink>]).</p> <p>Developments on applications and interpretations of IRT scores include derivations of accuracy, for specific, conditional examinee abilities and also for overall groups. Studies evaluating conditional accuracies have focused on conditional biases derived based on test information (Lord, [<reflink idref="bib18" id="ref12">18</reflink>]; Lord, [<reflink idref="bib19" id="ref13">19</reflink>]; Warm, [<reflink idref="bib27" id="ref14">27</reflink>]). Other developments describe the use of test information and bias functions in the estimation of marginal reliability (Samajima, [<reflink idref="bib25" id="ref15">25</reflink>]). Discussions of IRT‐based marginal reliability estimates have considered parallel forms and squared correlation approaches for different IRT ability estimates (Kim, [<reflink idref="bib7" id="ref16">7</reflink>]), the relationship of these to classical test theory reliabilities (Kim & Feldt, [<reflink idref="bib9" id="ref17">9</reflink>]), and the derivation of confidence intervals (Andersson & Xin, [<reflink idref="bib1" id="ref18">1</reflink>]). Uses of IRT‐based probability tables such as produced by the Lord‐Wingersky recursion algorithm for summed test scores (Lord & Wingersky, [<reflink idref="bib21" id="ref19">21</reflink>]) have been described for reliability, such as for total score variances (Andersson & Xin, [<reflink idref="bib1" id="ref20">1</reflink>]; Kim & Feldt, [<reflink idref="bib9" id="ref21">9</reflink>]), and also generalized to test scores based on nominal, real‐number item scores and test score reliabilities (Kim, [<reflink idref="bib8" id="ref22">8</reflink>]). There are opportunities to expand on these developments, such as by exploiting weighted item scores considered by Kim ([<reflink idref="bib8" id="ref23">8</reflink>]), their connections to different IRT scores, and the discrete probability tables produced by the recursion algorithm. These may reduce the chance that reliabilities and other accuracy estimates of IRT scores assume extreme or arbitrary values.</p> <p>The current study builds on previous works, especially Kim's ([<reflink idref="bib8" id="ref24">8</reflink>]) consideration of a generalized recursion procedure for test scores based on differently weighted item scores. The next section summarizes MLE, TCC, and EAP abilities, estimation approaches and statistical properties. Then we show how a generalized recursion procedure can be used to consider probability distributions of these estimators, from which marginal reliabilities and conditional accuracies can be estimated. We consider reliability estimates for 1‐, 2‐, and 3‐parameter logistic IRT models (1PL, 2PL, and 3PL) for tests with dichotomously scored items, using IRT calibrations from two datasets. Implications of test length, IRT model, and other issues for reliabilities and conditional accuracies are described.</p> <hd id="AN0190280393-3">IRT Scoring Methods</hd> <p>Many IRT scoring methods can be described in terms of a test of <emph>I</emph> dichotomously scored items, where correct responses to each item are models of logistic form with known item discrimination, difficulty, and guessing parameters, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0001" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0002" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0003" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> , a latent examinee ability parameter, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0004" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> , and a scaling constant often defined to approximate the normal ogive model, <emph>D =</emph> 1.702 (Baker & Kim, [<reflink idref="bib2" id="ref25">2</reflink>]; Lord, [<reflink idref="bib18" id="ref26">18</reflink>]), 1 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0005" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mfenced separators="" open="(" close=")"><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced><mo linebreak="badbreak">=</mo><msub><mi>c</mi><mi>i</mi></msub><mo linebreak="goodbreak">+</mo><mfenced separators="" open="(" close=")"><mrow><mn>1</mn><mo>−</mo><msub><mi>c</mi><mi>i</mi></msub></mrow></mfenced><mfrac><mn>1</mn><mrow><mn>1</mn><mo>+</mo><mrow><mi>exp</mi><mspace width="0.33em" /></mrow><mfenced separators="" open="[" close="]"><mrow><mo>−</mo><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub><mfenced separators="" open="(" close=")"><mrow><mi>θ</mi><mo>−</mo><msub><mi>b</mi><mi>i</mi></msub></mrow></mfenced></mrow></mfenced></mrow></mfrac><mrow><mi>for</mi><mspace width="0.33em" /><mi>item</mi><mspace width="0.33em" /></mrow><mi>i</mi><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}{\mathrm{Pr}}\left({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right) = {{c}_i} + \left({1 - {{c}_i}} \right)\frac{1}{{1 + {\mathrm{exp\ }}\left[ { - D{{a}_i}\left({\theta - {{b}_i}} \right)} \right]}}{\mathrm{for\ item\ }}i.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>For fixed and available <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0006" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0007" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0008" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values, scoring can be implemented by estimating <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0009" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0010" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> , and also by estimating IRT‐based true test scores at <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0011" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> (described below). In this study, MLE, TCC, and Bayesian Expected a Posteriori (EAP) scoring approaches are considered.</p> <hd id="AN0190280393-4">Scoring with maximum likelihood estimation (MLE)</hd> <p>MLE scores reflect assumptions of random examinee samples, and item responses that reflect Equation 1's IRT model and are conditionally independent given <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0012" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> (Baker & Kim, [<reflink idref="bib2" id="ref27">2</reflink>]; Lord, [<reflink idref="bib18" id="ref28">18</reflink>]). The MLE is an estimated examinee ability, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0013" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mo>,</mo></mrow><annotation encoding="application/x-tex">$\hat{\theta },$</annotation></semantics></math> </ephtml> that maximizes, 2 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0014" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>P</mi><mi>r</mi><mfenced separators="" open="(" close=")"><mrow><mi>U</mi><mo>|</mo><mi>θ</mi></mrow></mfenced><mo linebreak="badbreak">=</mo><munder><mo>∏</mo><mi>i</mi></munder><mi>Pr</mi><msup><mfenced separators="" open="(" close=")"><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced><msub><mi>u</mi><mi>i</mi></msub></msup><msup><mfenced separators="" open="[" close="]"><mrow><mn>1</mn><mo>−</mo><mi>Pr</mi><mfenced separators="" open="(" close=")"><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced></mrow></mfenced><mfenced separators="" open="(" close=")"><mrow><mn>1</mn><mo>−</mo><msub><mi>u</mi><mi>i</mi></msub></mrow></mfenced></msup></mrow><annotation encoding="application/x-tex">$$\begin{equation}Pr\left({U{\mathrm{|}}\theta } \right) = \mathop \prod \limits_i {\mathrm{Pr}}{{\left({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right)}^{{{u}_i}}}{{\left[ {1 - {\mathrm{Pr}}\left({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right)} \right]}^{\left({1 - {{u}_i}} \right)}}\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0015" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>U</mi><mo>=</mo><mo>(</mo><mrow><msub><mi>u</mi><mn>1</mn></msub><mo>,</mo><msub><mi>u</mi><mn>2</mn></msub><mo>,</mo><mtext>...</mtext><mo>.</mo><mo>.</mo><mo>,</mo><msub><mi>u</mi><mi>I</mi></msub></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$U = ({{{u}_1},{{u}_2}, \ldots..,{{u}_I}})$</annotation></semantics></math> </ephtml> is the vector of an examinee's observed correct/incorrect responses to all test items. Closed form solutions to maximizing Equation 2 do not exist, making iterative solutions necessary (Baker & Kim, [<reflink idref="bib2" id="ref29">2</reflink>]). One solution of particular interest in this study involves taking the log of Equation 2 and setting it to zero (Lord, [<reflink idref="bib18" id="ref30">18</reflink>]), resulting in 3 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0016" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mi>Pr</mi><mfenced separators="" open="(" close=")"><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}\sum_i {{w}_{i,MLE}}{{u}_i} = \sum_i {{w}_{i,MLE}}{\mathrm{Pr}}\left({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right)\end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0017" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> is a weighted total test score and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0018" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mo>=</mo><mfrac><mrow><mi>∂</mi><mi>Pr</mi><mrow><mo>(</mo><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><mo>/</mo><mi>∂</mi><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow><mrow><mi>Pr</mi><mrow><mo>(</mo><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><mrow><mo>[</mo><mrow><mn>1</mn><mo>−</mo><mi>Pr</mi><mo>(</mo><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><mo>]</mo></mrow></mrow></mfrac></mrow><annotation encoding="application/x-tex">${{w}_{i,MLE}} = \frac{{\partial {\mathrm{Pr}}({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })/\partial (\theta)}}{{{\mathrm{Pr}}({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })[ {1 - {\mathrm{Pr}}({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })} ]}}$</annotation></semantics></math> </ephtml> denotes item weights that optimize test information for examinees based on the IRT model with fixed <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0019" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0020" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> , and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0021" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values (Lord, [<reflink idref="bib18" id="ref31">18</reflink>]; Yen & Fitzpatrick, [<reflink idref="bib28" id="ref32">28</reflink>]). For the Rasch (all <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0022" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> = <emph>D</emph> = 1) and other 1PL IRT models, all items' <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0023" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> values are equal and all <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0024" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values are zero, and the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0025" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><annotation encoding="application/x-tex">${{w}_{i,MLE}}$</annotation></semantics></math> </ephtml> s equal a constant, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0026" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$D{{a}_i}$</annotation></semantics></math> </ephtml> , such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0027" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> is a summed score of equally weighted item scores. For the 2PL model (different <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0028" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> values and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0029" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values that equal zero), the item weights equal scaled item discriminations, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0030" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$D{{a}_i}$</annotation></semantics></math> </ephtml> , such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0031" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> corresponds to a pattern score. For the 3PL model, the item weights are a complex function of IRT parameters and examinee ability, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0032" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mrow><mi>Pr</mi><mrow><mo>(</mo><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><mo>−</mo><msub><mi>c</mi><mi>i</mi></msub></mrow><mrow><mn>1</mn><mo>−</mo><msub><mi>c</mi><mi>i</mi></msub></mrow></mfrac><mfrac><mrow><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub></mrow><mrow><mi>Pr</mi><mo>(</mo><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">$\frac{{{\mathrm{Pr}}({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta }) - {{c}_i}}}{{1 - {{c}_i}}}\frac{{D{{a}_i}}}{{{\mathrm{Pr}}({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })}}$</annotation></semantics></math> </ephtml> , such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0033" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> corresponds to a pattern score. For long tests, MLEs theoretically converge to their true values.</p> <p>In practice, MLEs reflect variability estimated as the conditional standard error of measurement (CSEM) based on test information (Lord, [<reflink idref="bib18" id="ref33">18</reflink>]), 4 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0034" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><msqrt><mrow><mi>I</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></msqrt></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}\frac{1}{{\sqrt {I\left(\theta \right)} }}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>Biases have also been derived (Lord, [<reflink idref="bib19" id="ref34">19</reflink>]),5 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0035" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mi>D</mi><mrow><msup><mi>I</mi><mn>2</mn></msup><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></mfrac><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>a</mi><mi>i</mi></msub><msub><mi>I</mi><mi>i</mi></msub><mfenced open="(" close=")"><mi>θ</mi></mfenced><mfenced separators="" open="[" close="]"><mfrac><mn>1</mn><mrow><mn>1</mn><mo>+</mo><mi>exp</mi><mfenced separators="" open="[" close="]"><mo>−</mo><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub><mfenced separators="" open="(" close=")"><mi>θ</mi><mo>−</mo><msub><mi>b</mi><mi>i</mi></msub></mfenced></mfenced></mrow></mfrac><mo>−</mo><mfrac><mn>1</mn><mn>2</mn></mfrac></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} \frac{D}{{I}^{2}\left(\theta \right)}\sum _{i}{a}_{i}{I}_{i}\left(\theta \right)\left[\frac{1}{1+\exp \left[-D{a}_{i}\left(\theta -{b}_{i}\right)\right]}-\frac{1}{2}\right] \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0036" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>I</mi><mspace width="0.33em" /><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><mo>=</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>I</mi><mi>i</mi></msub><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><mo>=</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>MLE</mi></mrow></msub><mi>∂</mi><mi>Pr</mi><mrow><mo>(</mo><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi><mo>)</mo></mrow><mo>/</mo><mi>∂</mi><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$I\ (\theta)={\sum}_{i}{I}_{i}(\theta)={\sum}_{i}{w}_{i,\textit{MLE}}\partial \mathrm{Pr}({u}_{i}=1|{a}_{i},{b}_{i},{c}_{i},\theta)/\partial (\theta)$</annotation></semantics></math> </ephtml> . Lord ([<reflink idref="bib19" id="ref35">19</reflink>]) showed that the bias of the MLE is positively correlated with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0037" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> (i.e., systematically underestimates <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0038" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> below the mean, and systematically overestimates <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0039" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> above the mean). The theoretically derived biases and CSEMs depend on test information. In practice, the biases, standard errors and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0040" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}}$</annotation></semantics></math> </ephtml> values can assume implausible values for extreme abilities. Figure 1 illustrates these issues, showing that biases (Figure 1a) and CSEMs (Figure 1b) produced with Equations 4 and 5 can implausibly exceed the typical scale of abilities (–5 to 5) for very low abilities (i.e., biases below –5, and CSEMs approaching 5).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01dec25/jedm70008-fig-0001.jpg?ephost1=dGJyMNLe80Sepq84zdnyOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm70008-fig-0001.jpg" title="1 Conditional biases (a) and CSEMs of scores (b) for the 25‐item test of the second study dataset using asymptotic methods and test information with the 3PL IRT model (Equations 4 and 5)." /> </p> <p></p> <hd id="AN0190280393-6">Scoring with test characteristic curve (TCC) estimation</hd> <p>Test scores can be used to implement IRT‐based scoring and ability estimation. The resulting scores are obtained as simple versions of MLE estimation procedures (Lord, [<reflink idref="bib18" id="ref36">18</reflink>]). These test characteristic curve (TCC) approaches are based on iteratively solving for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0041" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> values that preserve, 6 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0042" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>TCC</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>i</mi></munder><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo linebreak="goodbreak">=</mo><munder><mo>∑</mo><mi>i</mi></munder><mi>Pr</mi><mfenced separators="" open="(" close=")"><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mrow><mo linebreak="badbreak">=</mo><mn>1</mn><mo>|</mo></mrow><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} \sum _{i}{w}_{i,\textit{TCC}}{u}_{i}=\sum _{i}\ {u}_{i}=\sum _{i}\mathrm{Pr}\left(\ {u}_{i}=1|{a}_{i},{b}_{i},{c}_{i},\theta \right) \end{equation}$$</annotation></semantics></math> </ephtml> where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0043" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{u}_i}$</annotation></semantics></math> </ephtml> is a summed score, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0044" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">${{w}_{i,TCC}} = 1$</annotation></semantics></math> </ephtml> for all <emph>i</emph>. Solutions to Equation 6 utilize iterative methods. Similar to Equation 3, Equation 6 reflects assumptions that item parameter estimates and examinees' total scores are fixed (i.e., measurement error is ignored). The resulting TCC estimates, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0045" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><annotation encoding="application/x-tex">${{\hat{\theta }}_{TCC}}$</annotation></semantics></math> </ephtml> , reflect measurement error. Studies have shown that associated true scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0046" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><mi>Pr</mi><mrow><mo>(</mo><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sum_i {\mathrm{Pr}}({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},{{{\hat{\theta }}}_{TCC}}})$</annotation></semantics></math> </ephtml> have biases that are relatively small and variabilities that are relatively large (Lord, [<reflink idref="bib19" id="ref37">19</reflink>]).</p> <p>Equations 3 and 6 reflect a useful relationship in estimates of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0047" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><annotation encoding="application/x-tex">${{\hat{\theta }}_{TCC}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0048" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}}$</annotation></semantics></math> </ephtml> (Lord, [<reflink idref="bib18" id="ref38">18</reflink>]). Equation 3 is a more general version of Equation 6. For Rasch models, MLE and TCC ability estimates are equivalent. For 2PL and 3PL models, Equation 3 reflects a pattern score, such that <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0049" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> is a summed pattern score that corresponds to an item response pattern <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0050" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>U</mi><mo>=</mo><mo>(</mo><mrow><msub><mi>u</mi><mn>1</mn></msub><mo>,</mo><msub><mi>u</mi><mn>2</mn></msub><mo>,</mo><mtext>...</mtext><mo>.</mo><mo>.</mo><mo>,</mo><msub><mi>u</mi><mi>I</mi></msub></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$U = ({{{u}_1},{{u}_2}, \ldots..,{{u}_I}})$</annotation></semantics></math> </ephtml> , whereas Equation 6 reflects a summed score of equally weighted item scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0051" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i \ {{u}_i} = \sum_i {{w}_{i,TCC}}{{u}_i}$</annotation></semantics></math> </ephtml> .</p> <hd id="AN0190280393-7">Scoring with Expected a Posteriori (EAP) estimation</hd> <p>Bayesian IRT scoring approaches have orientations that are reversed from those of the MLE and TCC approaches. The true <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0052" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> is modeled as a random variable with a prior distribution, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0053" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">${\mathrm{Pr}}(\theta)$</annotation></semantics></math> </ephtml> , multiplied by <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0054" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}(U|\theta)$</annotation></semantics></math> </ephtml> , resulting in a conditional posterior distribution of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0055" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> on <emph>U</emph>,7 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0056" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mfenced separators="" open="(" close=")"><mi>θ</mi><mo>|</mo><mi>U</mi></mfenced><mo linebreak="badbreak">=</mo><mfrac><mrow><mi>Pr</mi><mrow><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>U</mi></mfenced></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \mathrm{Pr}\left(\theta |U\right)=\frac{\mathrm{Pr}(U|\theta)\mathrm{Pr}\left(\theta \right)}{\mathrm{Pr}\left(U\right)}. \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>The Expected a Posteriori (EAP) approach involves computing the expected value of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0057" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> based on Equation 7, 8 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0058" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>EAP</mi><mo linebreak="badbreak">=</mo><mi>E</mi><mfenced separators="" open="(" close=")"><mi>θ</mi><mo>|</mo><mi>U</mi></mfenced><mo linebreak="goodbreak">=</mo><mfrac><mrow><msub><mo>∫</mo><mi>θ</mi></msub><mi>θ</mi><mi>Pr</mi><mrow><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced><mi>d</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><mrow><msub><mo>∫</mo><mi>θ</mi></msub><mi>Pr</mi><mrow><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced><mi>d</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} \textit{EAP}=E\left(\theta |U\right)=\frac{\int_{\theta}\theta \mathrm{Pr}(U|\theta)\mathrm{Pr}\left(\theta \right)d\left(\theta \right)}{\int_{\theta}\mathrm{Pr}(U|\theta)\mathrm{Pr}\left(\theta \right)d\left(\theta \right)}. \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>Another EAP approach is based on summed scores (similar to the TCC approach), <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0059" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{u}_i}$</annotation></semantics></math> </ephtml> , 9 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0060" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow /><mi>s</mi></msub><mi>EAP</mi><mo linebreak="badbreak">=</mo><mi>E</mi><mspace width="0.33em" /><mfenced separators="" open="(" close=")"><mrow><mi>θ</mi><mo>|</mo></mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub></mfenced><mo linebreak="goodbreak">=</mo><mfrac><mrow><msub><mo>∫</mo><mi>θ</mi></msub><mi>θ</mi><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced><mi>d</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><mrow><msub><mo>∫</mo><mi>θ</mi></msub><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced><mi>d</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} {}_{s}\textit{EAP}=E\ \left(\theta |{\sum}_{i}{u}_{i}\right)=\frac{\int_{\theta}\theta \mathrm{Pr}({\sum}_{i}{u}_{i}|\theta)\mathrm{Pr}\left(\theta \right)d\left(\theta \right)}{\int_{\theta}\mathrm{Pr}({\sum}_{i}{u}_{i}|\theta)\mathrm{Pr}\left(\theta \right)d\left(\theta \right)}. \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>In practice, solutions to Equations 8 and 9 involve approximating the mathematical integrations using sums over discrete distributions with a quadratured version of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0061" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">${\mathrm{Pr}}(\theta)$</annotation></semantics></math> </ephtml> that is usually based on a standard normal distribution. The conditional biases of EAP estimates at specific <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0062" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> values have been found to be negatively correlated with <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0063" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> (systematically overestimate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0064" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> below the mean, and systematically underestimate <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0065" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> above the mean, Lord, [<reflink idref="bib20" id="ref39">20</reflink>]). Compared to MLE, EAP estimates are expected to exhibit larger absolute conditional biases and smaller total error (Kolen, [<reflink idref="bib11" id="ref40">11</reflink>]; Lord, [<reflink idref="bib18" id="ref41">18</reflink>]; Lord, [<reflink idref="bib19" id="ref42">19</reflink>]).</p> <hd id="AN0190280393-8">Generalized Recursion and IRT Scoring Methods</hd> <p>Lord and Wingersky ([<reflink idref="bib21" id="ref43">21</reflink>]) introduced the recursion procedure as a feasible approach to estimating distributions of test scores computed from equally weighted item scores using a <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0066" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> distribution and the IRT parameter estimates for a test's items. This section summarizes the Lord and Wingersky procedure and the generalization introduced by Kim ([<reflink idref="bib8" id="ref44">8</reflink>]) to nominally weighted item scores that is of interest in the current study. To obtain an estimated conditional distribution of the summed test scores on <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0067" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> , computations proceed iteratively through the items of a test, <emph>i</emph> = 1 to <emph>I</emph>. 10 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0068" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>For</mi><mspace width="0.28em" /><mi>i</mi><mo linebreak="badbreak">=</mo><mn>1</mn><mo>,</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation} {\mathrm{For}}\;i = 1,\end{equation}$$</annotation></semantics></math> </ephtml><ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0069" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mspace width="0.33em" /><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mn>1</mn><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub><mspace width="0.33em" /></mrow></msub><mo linebreak="badbreak">=</mo><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$$\begin{equation*}\ Scor{{e}_{1,{{u}_i}\ }} = {{u}_i}\end{equation*}$$</annotation></semantics></math> </ephtml><ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0070" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mrow><mo>(</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mn>1</mn><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub><mspace width="0.33em" /></mrow></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><mi>Pr</mi><msup><mfenced separators="" open="(" close=")"><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced><msub><mi>u</mi><mi>i</mi></msub></msup><mspace width="0.33em" /><msup><mfenced separators="" open="[" close="]"><mrow><mn>1</mn><mo>−</mo><mi>Pr</mi><mfenced separators="" open="(" close=")"><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow></mfenced></mrow></mfenced><mfenced separators="" open="(" close=")"><mrow><mn>1</mn><mo>−</mo><msub><mi>u</mi><mi>i</mi></msub></mrow></mfenced></msup><mo>:</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*}{\mathrm{Pr}}(Scor{{e}_{1,{{u}_i}\ }}|\theta) = {\mathrm{Pr}}{{\left({{{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right)}^{{{u}_i}}}\ {{\left[ {1 - {\mathrm{Pr}}\left({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta } \right)} \right]}^{\left({1 - {{u}_i}} \right)}}:\end{equation*}$$</annotation></semantics></math> </ephtml></p> <p> <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0071" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>For</mi><mspace width="0.28em" /><mi>i</mi><mo linebreak="badbreak">></mo><mn>1</mn><mo>:</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} {\mathrm{For}}\;i > 1: \end{equation*}$$</annotation></semantics></math> </ephtml> <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0072" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub><mspace width="0.33em" /></mrow></msub><mo linebreak="badbreak">=</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>−</mo><mn>1</mn></mrow></msub><mo linebreak="goodbreak">+</mo><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$$\begin{equation*} Scor{{e}_{i,{{u}_i}\ }} = Scor{{e}_{i - 1}} + {{u}_i}\end{equation*}$$</annotation></semantics></math> </ephtml> <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0073" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mrow><mo>(</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub></mrow></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mspace width="-0.16em" /><mo linebreak="badbreak">=</mo><mspace width="-0.16em" /><mi>Pr</mi><mrow><mo>(</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><msup><mrow><mo>(</mo><mrow><msub><mi>u</mi><mi>i</mi></msub><mspace width="-0.16em" /><mo>=</mo><mspace width="-0.16em" /><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><msub><mi>u</mi><mi>i</mi></msub></msup><msup><mrow><mo>[</mo><mrow><mn>1</mn><mspace width="-0.16em" /><mo>−</mo><mspace width="-0.16em" /><mi>Pr</mi><mo>(</mo><mrow><msub><mi>u</mi><mi>i</mi></msub><mspace width="-0.16em" /><mo>=</mo><mspace width="-0.16em" /><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow><mo>]</mo></mrow><mrow><mo>(</mo><mrow><mn>1</mn><mspace width="-0.16em" /><mo>−</mo><mspace width="-0.16em" /><msub><mi>u</mi><mi>i</mi></msub></mrow><mo>)</mo></mrow></msup><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation*} {\mathrm{Pr}}(Scor{{e}_{i,{{u}_i} }}|\theta)\! =\! {\mathrm{Pr}}(Scor{{e}_{i - 1}}|\theta){\mathrm{Pr}}{{({ {{u}_i} \!=\! 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })}^{{{u}_i}}}{{[ {1\! -\! {\mathrm{Pr}}({ {{u}_i} \!=\! 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })}]}^{({1 \!-\! {{u}_i}})}}.\end{equation*}$$</annotation></semantics></math> </ephtml> </p> <p>At each of the <emph>I</emph> steps, vectors containing all unique scores are formed, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0074" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$Scor{{e}_i}$</annotation></semantics></math> </ephtml> , and other vectors of score probabilities are formed by summing the probabilities at each unique score, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0075" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">${\mathrm{Pr}}(Scor{{e}_i}|\theta)$</annotation></semantics></math> </ephtml> . As the recursion proceeds, the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0076" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$Scor{{e}_i}$</annotation></semantics></math> </ephtml> s and associated probabilities, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0077" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo form="prefix">Pr</mo><mo>(</mo><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$\Pr ({Scor{{e}_i}{\mathrm{|}}\theta })$</annotation></semantics></math> </ephtml> , at steps <emph>i</emph> > 1 to <emph>I</emph> denote increasingly long vectors that reflect more of the possible set of test scores. When the recursion is completed for all items (<emph>i</emph> = <emph>I</emph>), conditional distributions of probabilities at each unique score, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0078" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mspace width="0.33em" /><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mo>=</mo><mi>Pr</mi><mrow><mo>(</mo><mi>Scor</mi><msub><mi>e</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}\ ({\sum}_{i}{u}_{i}|\theta)=\mathrm{Pr}(\textit{Scor}{e}_{i}|\theta)$</annotation></semantics></math> </ephtml> , can be obtained for particular <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0079" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> values. When these conditional distributions are averaged over a specified discrete distribution of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0080" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> , an estimated observed summed score distribution is produced, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0081" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>=</mo><msub><mo>∑</mo><mi>θ</mi></msub><mi>Pr</mi><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><mspace width="0.33em" /><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{u}_{i})={\sum}_{\theta}\mathrm{Pr}(\theta)\ \mathrm{Pr}({\sum}_{i}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> . Several variations of the Lord and Wingersky recursion procedure have been developed (Kolen & Brennan, [<reflink idref="bib12" id="ref45">12</reflink>]). Kim ([<reflink idref="bib8" id="ref46">8</reflink>]) developed a version for estimating conditional distributions of real‐number, noninteger scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0082" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><msub><mi>u</mi><mi>i</mi></msub></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr} ({\sum}_{i}{w}_{{u}_{i}}|\theta)$</annotation></semantics></math> </ephtml> , using item score weights that are nominally defined. Kim's ([<reflink idref="bib8" id="ref47">8</reflink>]) generalized recursion procedure involves modifying the score calculations in Equation 10 to reflect nominally weighted item scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0083" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Scor</mi><msub><mi>e</mi><mrow><mn>1</mn><mo>,</mo><msub><mi>w</mi><msub><mi>u</mi><mi>i</mi></msub></msub></mrow></msub><mo>=</mo><msub><mi>w</mi><msub><mi>u</mi><mi>i</mi></msub></msub></mrow><annotation encoding="application/x-tex">$\textit{Scor}{e}_{1,{w}_{{u}_{i}}}={w}_{{u}_{i}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0084" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Scor</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>></mo><mn>1</mn><mo>,</mo><msub><mi>w</mi><msub><mi>u</mi><mi>i</mi></msub></msub></mrow></msub><mo>=</mo><mi>Scor</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mi>w</mi><msub><mi>u</mi><mi>i</mi></msub></msub></mrow><annotation encoding="application/x-tex">$\textit{Scor}{e}_{i>1,{w}_{{u}_{i}}}=\textit{Scor}{e}_{i-1}+{w}_{{u}_{i}}$</annotation></semantics></math> </ephtml> , and then aggregating unique scores and probabilities at those unique scores resulting from summing the nominally weighted item scores.</p> <hd id="AN0190280393-9">Integrating Generalized Recursion and IRT Scoring for the Estimation of Marginal Reliabilitie...</hd> <p>In this study an integration of Kim's ([<reflink idref="bib8" id="ref48">8</reflink>]) generalized recursion procedure is considered using MLE item weights to produce <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0085" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>MLE</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{w}_{i,\textit{MLE}}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> . This procedure modifies the scores in Equation 10 to reflect the MLE item weights, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0086" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><mi>e</mi></mrow><mrow><mn>1</mn><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub></mrow></msub><mo>=</mo><msub><mi>u</mi><mi>i</mi></msub><msub><mi>w</mi><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>,</mo><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></mrow></msub></mrow><annotation encoding="application/x-tex">${Score}_{1,{u}_{i}}={u}_{i}{w}_{{u}_{i},{MLE}}$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0087" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>></mo><mn>1</mn><mo>,</mo><msub><mi>u</mi><mi>i</mi></msub><mspace width="0.33em" /></mrow></msub><mo>=</mo><mi>S</mi><mi>c</mi><mi>o</mi><mi>r</mi><msub><mi>e</mi><mrow><mi>i</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mi>u</mi><mi>i</mi></msub><mspace width="0.33em" /><msub><mi>w</mi><mrow><msub><mi>u</mi><mi>i</mi></msub><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub></mrow><annotation encoding="application/x-tex">$Scor{{e}_{i > 1,{{u}_i}\ }} = Scor{{e}_{i - 1}} + {{u}_i}\ {{w}_{{{u}_i},MLE}}$</annotation></semantics></math> </ephtml> . Aggregating the probabilities at unique weighted scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0088" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> , produces a conditional distribution, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0089" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>MLE</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{w}_{i,\textit{MLE}}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> . This conditional distribution can be used with the previously described IRT scoring procedures to obtain distributions of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0090" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0091" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><annotation encoding="application/x-tex">$\hat{\theta }$</annotation></semantics></math> </ephtml> , including <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0092" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>MLE</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\widehat{\theta}}_{\textit{MLE}}|\theta)$</annotation></semantics></math> </ephtml> from <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0093" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$(\sum_i {{w}_{i,MLE}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)}}$</annotation></semantics></math> </ephtml> and Equation 3, and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0094" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>TCC</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\widehat{\theta}}_{\textit{TCC}}|\theta)$</annotation></semantics></math> </ephtml> distributions from <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0095" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>TCC</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{w}_{i,\textit{TCC}}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> and Equation 6. Distributions of EAP estimates can also be obtained. Because <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0096" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> corresponds to <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0097" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>U</mi><mo>=</mo><mo>(</mo><mrow><msub><mi>u</mi><mn>1</mn></msub><mo>,</mo><msub><mi>u</mi><mn>2</mn></msub><mo>,</mo><mtext>...</mtext><mo>.</mo><mo>.</mo><mo>,</mo><msub><mi>u</mi><mi>I</mi></msub></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$U = ({{{u}_1},{{u}_2}, \ldots..,{{u}_I}})$</annotation></semantics></math> </ephtml> , the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0098" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>MLE</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{w}_{i,\textit{MLE}}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> distribution can be used as the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0099" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}(U|\theta)$</annotation></semantics></math> </ephtml> distribution for obtaining the posterior distribution <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0100" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>|</mo><mi>U</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}(\theta |U)$</annotation></semantics></math> </ephtml> (Equation 7) and EAP estimates (Equation 8). Similarly, the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0101" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>TCC</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{w}_{i,\textit{TCC}}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> distribution can be used to obtain <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0102" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\mathrm{Pr}({\sum}_{i}{u}_{i}|\theta)$</annotation></semantics></math> </ephtml> and <subs>s</subs>EAP (Equation 9). The integration of generalized recursion procedures and IRT scoring methods allows for joint "true, observed" probability tables of IRT‐based abilities and/or test scores. These tables have several potential uses, including the estimation of marginal reliability, true score variance and error variance, conditional biases and CSEMs.</p> <p>Consider the estimation of marginal reliability, where the total variance for an estimator is computed as an average or expectation of squared deviations from a mean. Total variance is assumed to reflect a Within Groups and Between Groups ANOVA decomposition of error and true score variances (Kim, [<reflink idref="bib7" id="ref49">7</reflink>]; Kim & Feldt, [<reflink idref="bib9" id="ref50">9</reflink>]),11 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0103" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>σTotal2Estimator=EEstimator−EEstimator2=σError2Estimator+σTrue2Estimator.<annotation encoding="application/x-tex">$$\begin{equation} \def\eqcellsep{&}\begin{array}{c}{{\sigma}_{\textit{Total}}^{2}\left(\textit{Esti}\textit{mator}\right)=E{\left[\textit{Esti}\textit{mator}-E\left(\textit{Esti}\textit{mator}\right)\right]}^{2}}\\ {={\sigma}_{\textit{Error}}^{2}\left(\textit{Esti}\textit{mator}\right)+{\sigma}_{\textit{True}}^{2}\left(\textit{Esti}\textit{mator}\right).}\end{array} \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>The estimation of reliability through an "observed score variance approach" (p. 181, Kim & Feldt, [<reflink idref="bib9" id="ref51">9</reflink>]) involves estimating the total and error variances and assuming the additive property of true score and error variances shown in Equation 11. The reliability of an estimator can be obtained from this estimation approach and its additivity assumption, expressed as the ratio of true and total variances, 12 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0104" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mi>e</mi><mi>l</mi><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow></mfenced><mo linebreak="badbreak">=</mo><mfrac><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>r</mi><mi>u</mi><mi>e</mi></mrow><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow></mfenced></mrow><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow></mfenced></mrow></mfrac><mo linebreak="goodbreak">=</mo><mn>1</mn><mo linebreak="goodbreak">−</mo><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow></mfenced><mfrac><mn>1</mn><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow></mfenced></mrow></mfrac><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}Rel\left({Estimator} \right) = \frac{{\sigma _{True}^2\left({Estimator} \right)}}{{\sigma _{Total}^2\left({Estimator} \right)}} = 1 - \sigma _{Error}^2\left({Estimator} \right)\frac{1}{{\sigma _{Total}^2\left({Estimator} \right)}}.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>Marginal reliabilities can be equivalently obtained as squared correlations of the true <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0105" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> s those from an estimator (i.e., the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0106" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mfrac><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>r</mi><mi>u</mi><mi>e</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow><mo>)</mo></mrow></mrow><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow><mo>)</mo></mrow></mrow></mfrac><annotation encoding="application/x-tex">$\frac{{\sigma _{True}^2({Estimator})}}{{\sigma _{Total}^2({Estimator})}}$</annotation></semantics></math> </ephtml> in Equation 12 reflects <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0107" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi><mi>o</mi><mi>r</mi><mi>r</mi><msup><mrow><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi><mo>,</mo><mspace width="0.33em" /><mi>T</mi><mi>r</mi><mi>u</mi><mi>e</mi></mrow><mo>)</mo></mrow><mn>2</mn></msup></mrow><annotation encoding="application/x-tex">$Corr{{({Estimator,\ True})}^2}$</annotation></semantics></math> </ephtml> ). The error variances are obtained as an average, or expectation of conditional error variances of an estimator with respect to the distribution of a conditioning variable (Cond. Var.), 13 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0108" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>σError2Estimator=Eσ2Estimator|Cond.Var.=∫Cond.Var.σ2Estimator|Cond.Var.PrCond.Var.dCond.Var..<annotation encoding="application/x-tex">$$\begin{eqnarray} \sigma _{Error}^2&&\left({Estimator} \right) = E\left[ {{{\sigma }^2}\left({Estimator{\mathrm{|}}Cond.Var.} \right)} \right]\nonumber\\ && = \int\limits_{{Cond.Var.}}{{{{\sigma }^2}\left({Estimator{\mathrm{|}}Cond.Var.} \right)\Pr \left({Cond.Var.} \right)d\left({Cond.Var.} \right)}}. \end{eqnarray}$$</annotation></semantics></math> </ephtml></p> <p>Some challenges of reliability estimation for IRT estimators are that total and error variances might be estimated with different procedures, such as error variances computed from sums of item‐level error variances and total variances computed from IRT recursion procedures (Andersson & Xin, [<reflink idref="bib1" id="ref52">1</reflink>]; Kim & Feldt, [<reflink idref="bib9" id="ref53">9</reflink>]). Another issue is that for approaches where error variances for MLE are estimated through IRT information (i.e., Equation 4), the previously described uses of arbitrary values for undefined conditional errors could create situations where the additive property assumed in Equation 11 does not hold. Obtaining the error and total variances described in Equations 11 and 13 from a single recursion table means less reliance on arbitrarily defined values from asymptotic methods. Computing error and total variances from a single probability table produced from recursion also assures that total variance is always a sum of true and error variances. The next sections describe reliability estimates for the non‐Bayesian and Bayesian estimators using probability tables from IRT recursion (summarized in Table 1). These are followed by descriptions of conditional accuracy measures (CSEMs and biases).</p> <p>1 Table Summary of the IRT‐Based Estimators and Reliabilities</p> <p> <ephtml> <table><thead><tr><th>Estimator</th><th align="center">Non‐Bayesian/Bayesian</th><th align="center">Conditioning Variable</th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0109" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mspace width="0.33em" /><mi>Rel</mi><mo>(</mo><mi>Esti</mi><mi>mator</mi><mo>)</mo><mo>=</mo></mrow><annotation encoding="application/x-tex">$\ \textit{Rel}(\textit{Esti}\textit{mator})=$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0110" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mn>1</mn><mo>−</mo></mrow><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0111" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Error}^2({Estimator})$</annotation></semantics></math></p></th><th align="center"><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0112" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mfrac><mn>1</mn><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi></mrow><mo>)</mo></mrow></mrow></mfrac><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2({Estimator})}}$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0113" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi><mi>u</mi><mi>i</mi><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math></p></td><td>Non‐Bayesian</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0114" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0115" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mspace width="0.33em" /><mi>Rel</mi><mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>MLE</mi><mi>u</mi><mi>i</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\ \textit{Rel}({\sum}_{i}{w}_{i,\textit{MLE}}{u}_{i})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0116" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0117" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mi>σ</mi>2<mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi><mi>u</mi><mi>i</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\sum_\theta {{\sigma }^2}(\sum_i {{w}_{i,MLE}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)Pr}}(\theta)$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0118" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi><mi>u</mi><mi>i</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2(\sum_i {{w}_{i,MLE}}{{u}_i})}}$</annotation></semantics></math></p></td></tr><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0119" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><mo>̂</mo><mi>M</mi><mi>L</mi><mi>E</mi><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}}$</annotation></semantics></math></p></td><td>Non‐Bayesian</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0120" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0121" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>Rel</mi><mo>(</mo><mi>θ</mi><mo>̂</mo><mi>MLE</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\textit{Rel}({\widehat{\theta}}_{\textit{MLE}})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0122" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0123" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mi>σ</mi>2<mo>(</mo><mi>θ</mi><mo>̂</mo><mi>M</mi><mi>L</mi><mi>E</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\sum_\theta {{\sigma }^2}({{\hat{\theta }}_{MLE}}|\theta){\mathrm{Pr}}(\theta)$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0124" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mi>θ</mi><mo>̂</mo><mi>M</mi><mi>L</mi><mi>E</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2({{{{\hat{\theta }}}_{MLE}}})}}$</annotation></semantics></math></p></td></tr><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0125" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi><mi>u</mi><mi>i</mi><annotation encoding="application/x-tex">$\sum_i {{w}_{i,TCC}}{{u}_i}$</annotation></semantics></math></p></td><td>Non‐Bayesian</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0126" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0127" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mspace width="0.33em" /><mi>Rel</mi><mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>TCC</mi><mi>u</mi><mi>i</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\ \textit{Rel}({\sum}_{i}{w}_{i,\textit{TCC}}{u}_{i})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0128" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0129" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mi>σ</mi>2<mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi><mi>u</mi><mi>i</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\sum_\theta {{\sigma }^2}(\sum_i {{w}_{i,TCC}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)Pr}}(\theta)$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0130" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi><mi>u</mi><mi>i</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2(\sum_i {{w}_{i,TCC}}{{u}_i})}}$</annotation></semantics></math></p></td></tr><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0131" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><mo>̂</mo><mi>T</mi><mi>C</mi><mi>C</mi><annotation encoding="application/x-tex">${{\hat{\theta }}_{TCC}}$</annotation></semantics></math></p></td><td>Non‐Bayesian</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0132" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0133" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>Rel</mi><mo>(</mo><mi>θ</mi><mo>̂</mo><mi>TCC</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\textit{Rel}({\widehat{\theta}}_{\textit{TCC}})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0134" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0135" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mi>σ</mi>2<mo>(</mo><mi>θ</mi><mo>̂</mo><mi>T</mi><mi>C</mi><mi>C</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\sum_\theta {{\sigma }^2}({{\hat{\theta }}_{TCC}}|\theta){\mathrm{Pr}}(\theta)$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0136" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mi>θ</mi><mo>̂</mo><mi>T</mi><mi>C</mi><mi>C</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2({{{{\hat{\theta }}}_{TCC}}})}}$</annotation></semantics></math></p></td></tr><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0137" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>E</mi><mi>A</mi><mi>P</mi><annotation encoding="application/x-tex">$EAP$</annotation></semantics></math></p></td><td>Bayesian</td><td>U</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0138" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>Rel</mi><mo>(</mo><mi>EAP</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\textit{Rel}(\textit{EAP})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0139" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0140" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mo>(</mo><mi>θ</mi><mo>−</mo><mi>E</mi><mi>A</mi><mi>P</mi><mo>)</mo>2<mi>Pr</mi><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><mo>∑</mo><mi>θ</mi><mi>Pr</mi><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{{\sum_\theta {{{({\theta - EAP})}}^2}{\mathrm{Pr}}(U|\theta){\mathrm{Pr}}(\theta)}}{{\sum_\theta {\mathrm{Pr}}(U|\theta){\mathrm{Pr}}(\theta)}}$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0141" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2(\theta)}}$</annotation></semantics></math></p></td></tr><tr><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0142" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>s</mi><mi>E</mi><mi>A</mi><mi>P</mi><annotation encoding="application/x-tex">$_sEAP$</annotation></semantics></math></p></td><td>Bayesian</td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0143" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>i</mi><mi>u</mi><mi>i</mi><annotation encoding="application/x-tex">$\sum_i {{u}_i}$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0144" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>Rel</mi><mo>(</mo><mi>s</mi><mi>EAP</mi><mo>)</mo><mo>=</mo><annotation encoding="application/x-tex">$\textit{Rel}{(}_{s}\textit{EAP})=$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0145" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mo>−</mo><annotation encoding="application/x-tex">$1 - $</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0146" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mo>∑</mo><mi>θ</mi><mo>(</mo><mi>θ</mi><mo>−</mo><mi>s</mi><mi>E</mi><mi>A</mi><mi>P</mi><mo>)</mo>2<mi>Pr</mi><mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi><mi>u</mi><mi>i</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><mo>∑</mo><mi>θ</mi><mi>Pr</mi><mo>(</mo><mo>∑</mo><mi>i</mi><mi>w</mi><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi><mi>u</mi><mi>i</mi><mo>|</mo><mi>θ</mi><mo>)</mo><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{{\sum_\theta {{{({\theta {{ - }_s}EAP})}}^2}{\mathrm{Pr}}(\sum_i {{w}_{i,TCC}}{{u}_i}|\theta){\mathrm{Pr}}(\theta)}}{{\sum_\theta {\mathrm{Pr}}(\sum_i {{w}_{i,TCC}}{{u}_i}|\theta){\mathrm{Pr}}(\theta)}}$</annotation></semantics></math></p></td><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0147" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns="">1<mi>σ</mi><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi>2<mo>(</mo><mi>θ</mi><mo>)</mo><annotation encoding="application/x-tex">$\frac{1}{{\sigma _{Total}^2(\theta)}}$</annotation></semantics></math></p></td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>. The biases of the estimators can be estimated as the difference of the expected value of the estimator conditioned on <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0148" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> , <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0149" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mo>(</mo><mrow><mi>E</mi><mi>s</mi><mi>t</mi><mi>i</mi><mi>m</mi><mi>a</mi><mi>t</mi><mi>o</mi><mi>r</mi><mo>|</mo><mi>θ</mi></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$E({Estimator{\mathrm{|}}\theta })$</annotation></semantics></math> </ephtml> , from the corresponding true value of the score or ability. CSEMs are obtained as, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0150" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>Error</mi></msub><mrow><mo>(</mo><mi>Esti</mi><mi>mator</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${\sigma}_{\textit{Error}}(\textit{Esti}\textit{mator}|\theta)$</annotation></semantics></math> </ephtml> .</p> <hd id="AN0190280393-10">Marginal variances and reliabilities for non‐Bayesian estimators</hd> <p>For non‐Bayesian approaches, the estimate of interest is an IRT‐based test score or ability conditioned on <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0151" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> . The error variances of non‐Bayesian estimated scores and abilities are obtained in versions of Equation 13 that involve weighted averages and summations with the discrete, joint distributions of estimators and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0152" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> s produced with the previously described generalized recursion and IRT scoring procedures: 14 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0153" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>θ</mi></munder><msup><mi>σ</mi><mn>2</mn></msup><mrow><mspace width="0.33em" /><mo>(</mo></mrow><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mrow><mo>)</mo><mi>Pr</mi></mrow><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}\sigma _{Error}^2(\sum_i {{w}_{i,MLE}}{{u}_i}) = \sum_\theta {{\sigma }^2}\ (\sum_i {{w}_{i,MLE}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)Pr}}\left(\theta \right)\end{equation}$$</annotation></semantics></math> </ephtml> 15 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0154" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mi>Error</mi><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>MLE</mi></msub></mfenced><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>θ</mi></munder><msup><mi>σ</mi><mn>2</mn></msup><mspace width="0.33em" /><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>MLE</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} {\sigma}_{\textit{Error}}^{2}\left({\widehat{\theta}}_{\textit{MLE}}\right)=\sum _{\theta}{\sigma}^{2}\ ({\widehat{\theta}}_{\textit{MLE}}|\theta)\mathrm{Pr}\left(\theta \right) \end{equation}$$</annotation></semantics></math> </ephtml> 16 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0155" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>)</mo></mrow><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>θ</mi></munder><msup><mi>σ</mi><mn>2</mn></msup><mrow><mspace width="0.33em" /><mo>(</mo></mrow><munder><mo>∑</mo><mi>i</mi></munder><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mrow><mo>)</mo><mi>Pr</mi></mrow><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation}\sigma _{Error}^2(\sum_i {{w}_{i,TCC}}{{u}_i}) = \sum_\theta {{\sigma }^2}\ (\sum_i {{w}_{i,TCC}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)Pr}}\left(\theta \right)\end{equation}$$</annotation></semantics></math> </ephtml> 17 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0156" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mi>Error</mi><mn>2</mn></msubsup><mfenced separators="" open="(" close=")"><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>TCC</mi></msub></mfenced><mo linebreak="badbreak">=</mo><munder><mo>∑</mo><mi>θ</mi></munder><msup><mi>σ</mi><mn>2</mn></msup><mspace width="0.33em" /><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mi>TCC</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><annotation encoding="application/x-tex">$$\begin{equation} {\sigma}_{\textit{Error}}^{2}\left({\widehat{\theta}}_{\textit{TCC}}\right)=\sum _{\theta}{\sigma}^{2}\ ({\widehat{\theta}}_{\textit{TCC}}|\theta)\mathrm{Pr}\left(\theta \right) \end{equation}$$</annotation></semantics></math> </ephtml></p> <p>The total variance of interest is of the dependent variable of the estimation approach (Kim, [<reflink idref="bib7" id="ref54">7</reflink>]). For non‐Bayesian estimators, the total variance is of the estimated ability, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0157" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Total}^2({{{{\hat{\theta }}}_{MLE}}})$</annotation></semantics></math> </ephtml> or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0158" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Total}^2({{{{\hat{\theta }}}_{TCC}}})$</annotation></semantics></math> </ephtml> , or of the test score, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0159" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Total}^2(\sum_i {{w}_{i,MLE}}{{u}_i})$</annotation></semantics></math> </ephtml> or <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0160" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Total}^2(\sum_i {{w}_{i,TCC}}{{u}_i})$</annotation></semantics></math> </ephtml> . For total and error variances computed from the same probability table, the total variances of the four non‐Bayesian estimators can be shown to reflect the sum of true score and error variances shown in Equation 11. For example, 18 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0161" xmlns="http://www.w3.org/1998/Math/MathML"><semantics>σTotal2θ̂MLE=Eθ̂MLE−Eθ̂MLE2=∑θ∑θ̂MLE|θPrθPrθ̂MLE|θθ̂MLE|θ−Eθ̂MLE|θ+Eθ̂MLE|θ−Eθ̂MLE2=∑θσ2θ̂MLE|θPrθ+∑θEθ̂MLE|θ−Eθ̂MLE2Prθ=σError2θ̂MLE+σTrue2θ̂MLE<annotation encoding="application/x-tex">$$\begin{eqnarray} {{\sigma}_{\textit{Total}}^{2}\left({\widehat{\theta}}_{\textit{MLE}}\right)}& =& E{\left[{\widehat{\theta}}_{\textit{MLE}}-E\left({\widehat{\theta}}_{\textit{MLE}}\right)\right]}^{2}\nonumber\\ &=&\sum _{\theta}\sum _{{\widehat{\theta}}_{\textit{MLE}}|\theta}{\mathrm{Pr}}\left(\theta \right){\mathrm{Pr}}\left({\widehat{\theta}}_{\textit{MLE}}|\theta \right)\nonumber\\ &&{\left[\left\{{\widehat{\theta}}_{\textit{MLE}}|\theta -E\left({\widehat{\theta}}_{\textit{MLE}}|\theta \right)\right\}+\left\{E\left({\widehat{\theta}}_{\textit{MLE}}|\theta \right)-E\left({\widehat{\theta}}_{\textit{MLE}}\right)\right\}\right]}^{2}\nonumber\\ && =\sum _{\theta}{\sigma}^{2}\left({\widehat{\theta}}_{\textit{MLE}}|\theta \right)\operatorname{\mathrm{Pr}}\left(\theta \right)+\sum _{\theta}{\left[E\left({\widehat{\theta}}_{\textit{MLE}}|\theta \right)-E\left({\widehat{\theta}}_{\textit{MLE}}\right)\right]}^{2}{\mathrm{Pr}}\left(\theta \right)\nonumber\\ && ={\mathrm{\sigma}}_{\mathrm{Error}}^{2}\left({\widehat{\theta}}_{\textit{MLE}}\right)+{\mathrm{\sigma}}_{\mathrm{True}}^{2}\left({\widehat{\theta}}_{\textit{MLE}}\right) \end{eqnarray}$$</annotation></semantics></math> </ephtml></p> <p>Marginal reliabilities can be computed as ratios of the total and error variances (Table 1).</p> <hd id="AN0190280393-11">Marginal variances and reliabilities for Bayesian estimators</hd> <p>For Bayesian approaches, the estimation is of an ability conditioned on the observed item response pattern, <emph>U</emph>, or on the summed test score. Error variances can be computed as posterior variances from probability tables produced from recursion procedures, 19 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0162" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><mspace width="0.33em" /><mfenced separators="" open="(" close=")"><mrow><mi>E</mi><mi>A</mi><mi>P</mi></mrow></mfenced><mo linebreak="badbreak">=</mo><mfrac><mrow><msub><mo>∑</mo><mi>θ</mi></msub><msup><mfenced separators="" open="(" close=")"><mrow><mi>θ</mi><mo>−</mo><mi>E</mi><mi>A</mi><mi>P</mi></mrow></mfenced><mn>2</mn></msup><mi>Pr</mi><mrow><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><mrow><msub><mo>∑</mo><mi>θ</mi></msub><mi>Pr</mi><mrow><mo>(</mo><mi>U</mi><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></mfrac><mspace width="0.33em" /></mrow><annotation encoding="application/x-tex">$$\begin{equation}\sigma _{Error}^2\ \left({EAP} \right) = \frac{{\sum_\theta {{{\left({\theta - EAP} \right)}}^2}{\mathrm{Pr}}(U|\theta){\mathrm{Pr}}\left(\theta \right)}}{{\sum_\theta {\mathrm{Pr}}(U|\theta){\mathrm{Pr}}\left(\theta \right)}}\ \end{equation}$$</annotation></semantics></math> </ephtml> 20 <ephtml> <math display="block" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0163" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>E</mi><mi>r</mi><mi>r</mi><mi>o</mi><mi>r</mi></mrow><mn>2</mn></msubsup><msub><mo>(</mo><mi>s</mi></msub><mrow><mi>E</mi><mi>A</mi><mi>P</mi><mo>)</mo><mo>=</mo></mrow><mfrac><mrow><msub><mo>∑</mo><mi>θ</mi></msub><msup><mfenced separators="" open="(" close=")"><mrow><mi>θ</mi><msub><mo>−</mo><mi>s</mi></msub><mi>E</mi><mi>A</mi><mi>P</mi></mrow></mfenced><mn>2</mn></msup><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow><mrow><msub><mo>∑</mo><mi>θ</mi></msub><mi>Pr</mi><mrow><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><mi>Pr</mi><mfenced open="(" close=")"><mi>θ</mi></mfenced></mrow></mfrac><mspace width="0.33em" /><mo>.</mo></mrow><annotation encoding="application/x-tex">$$\begin{equation}\sigma _{Error}^2{{(}_s}EAP) = \frac{{\sum_\theta {{{\left({\theta {{ - }_s}EAP} \right)}}^2}{\mathrm{Pr}}(\sum_i {{w}_{i,TCC}}{{u}_i}|\theta){\mathrm{Pr}}\left(\theta \right)}}{{\sum_\theta {\mathrm{Pr}}(\sum_i {{w}_{i,TCC}}{{u}_i}|\theta){\mathrm{Pr}}\left(\theta \right)}}\.\end{equation}$$</annotation></semantics></math> </ephtml></p> <p>For Bayesian estimators, the total variance is of the true ability, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0164" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>σ</mi><mrow><mi>T</mi><mi>o</mi><mi>t</mi><mi>a</mi><mi>l</mi></mrow><mn>2</mn></msubsup><mrow><mo>(</mo><mi>θ</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sigma _{Total}^2(\theta)$</annotation></semantics></math> </ephtml> (Kim, [<reflink idref="bib7" id="ref55">7</reflink>]). When computed from a single probability table, the total, error and true score variances of the EAP and <subs>s</subs>EAP estimators and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0165" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> reflect the additive relationships described in Equations 11 and 18. Marginal reliabilities can be computed as ratios of the total and error variances (Table 1).</p> <hd id="AN0190280393-12">Conditional standard errors of measurement and biases</hd> <p>Conditional accuracies across the range of <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0166" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> represent additional interests in evaluations of IRT‐based ability estimates and test scores. Conditional accuracies provide more detail about accuracy than marginal reliabilities. Conditional standard errors of measurement (CSEMs) at specific abilities are an ongoing focus in the field (Kolen et al., [<reflink idref="bib13" id="ref56">13</reflink>]; Lee et al., [<reflink idref="bib15" id="ref57">15</reflink>]; Moses & Kim, [<reflink idref="bib22" id="ref58">22</reflink>]). CSEMs can be obtained as the square root of the conditional variances used in the computation of the overall error variances (e.g., the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0167" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">$\sigma (\sum_i {{w}_{i,MLE}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)}}$</annotation></semantics></math> </ephtml> in Equation 14). Prior works have indicated that CSEMs for scores are larger than CSEMs for non‐Bayesian ability estimates, which are larger than CSEMs for Bayesian ability estimates (Kolen et al., 2010; Lord, [<reflink idref="bib19" id="ref59">19</reflink>]). The shapes of CSEMs for scores are inverse U‐shaped, whereas the shapes of CSEMs for abilities are U‐shaped. Additional interest is in evaluating CSEMs as produced from the generalized recursion procedure, and comparing to those based on asymptotic information functions (Equation 4).</p> <p>Conditional biases are also of interest, especially as IRT estimators are described as exhibiting estimation biases not reflected in test scores (Kim, [<reflink idref="bib7" id="ref60">7</reflink>]; Lord, [<reflink idref="bib19" id="ref61">19</reflink>]; Lord, [<reflink idref="bib20" id="ref62">20</reflink>]). Biases for the IRT‐based abilities and scores can be obtained as deviations of the conditional means of each estimator from its true quantity. For example, the biases of the MLE summed pattern scores, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0168" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i}$</annotation></semantics></math> </ephtml> , and abilities, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0169" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}}$</annotation></semantics></math> </ephtml> , can be estimated from the probability tables produced from the recursion as, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0170" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mo>(</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mo>|</mo><mi>θ</mi><mo>)</mo><mo>−</mo><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mi>Pr</mi><mrow><mo>(</mo><mrow><mspace width="0.33em" /><msub><mi>u</mi><mi>i</mi></msub><mo>=</mo><mn>1</mn><mo>|</mo><msub><mi>a</mi><mi>i</mi></msub><mo>,</mo><msub><mi>b</mi><mi>i</mi></msub><mo>,</mo><msub><mi>c</mi><mi>i</mi></msub><mo>,</mo><mi>θ</mi></mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$E(\sum_i {{w}_{i,MLE}}{{u}_i}{\mathrm{|}}\theta {\mathrm{)}} - \sum_i {{w}_{i,MLE}}{\mathrm{Pr}}({\ {{u}_i} = 1{\mathrm{|}}{{a}_i},{{b}_i},{{c}_i},\theta })$</annotation></semantics></math> </ephtml> and <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0171" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>E</mi><mo>(</mo><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mo>|</mo><mi>θ</mi></mrow><mo>)</mo><mo>−</mo><mi>θ</mi></mrow><annotation encoding="application/x-tex">$E({{{{\hat{\theta }}}_{MLE}}{\mathrm{|}}\theta }) - \theta $</annotation></semantics></math> </ephtml> , respectively. Biases for other IRT‐based scores and ability estimators can be estimated similarly. Prior discussions indicate that scores exhibit relatively small biases, that MLE estimators exhibit conditional biases that are positively correlated with ability, and that EAP estimators exhibit relatively large biases that are negatively correlated with ability (Kolen, [<reflink idref="bib11" id="ref63">11</reflink>]; Lord, [<reflink idref="bib19" id="ref64">19</reflink>]; Lord, [<reflink idref="bib20" id="ref65">20</reflink>]). Another interest is comparing conditional biases produced from the generalized recursion table to those based on asymptotic information functions (Equation 5), summarized in the Discussion section.</p> <hd id="AN0190280393-13">Scope of the Current Study</hd> <p>The current study considers the generalized recursion procedure for reliability estimation with several IRT‐based ability and score estimators, including those based on MLE, TCC, and EAP. The proposed method exploits a relationship of weighted test scores and IRT‐based abilities (Equations 3 and 6), a relationship that has not been extensively discussed in the literature. The accuracies of the IRT‐based estimators and our estimation approach for reliability are considered in terms of marginal reliability, and conditional biases and CSEMs. Estimation accuracies are evaluated for different test datasets, test lengths, IRT models, and for different conditions of IRT parameter inaccuracy. General implications of the procedures for IRT‐based estimators, associated reliabilities and accuracies are considered in the Discussion section.</p> <hd id="AN0190280393-14">Method</hd> <p></p> <hd id="AN0190280393-15">Datasets, IRT Models, and Questions</hd> <p>Analyses for this study were based on results obtained from two different datasets that allowed for considering the generality of the results, and also for addressing different questions. The first dataset is one where Rasch and 3PL IRT parameter estimates were obtained for a 36‐item test administered to an undefined sample of 1,638 examinees (Form Y in Kolen & Brennan, [<reflink idref="bib12" id="ref66">12</reflink>]), using BILOG‐MG and its default calibration settings. Distributions of the IRT parameters are summarized in Table 2. Reliability estimates were produced for the Rasch and 3PL IRT models for the considered IRT‐based estimators described in Table 1, using a quadratured approximation of a standard normal distribution for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0172" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">${\mathrm{Pr}}(\theta)$</annotation></semantics></math> </ephtml> , where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0173" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> ranged from –5 to 5 in 0.2 units. One interest was in how the reliability estimates compared to those based on the observed examinee data, such as the coefficient alpha reliabilities of the summed scores (0.860), and of the summed pattern scores based on 3PL IRT parameters and MLE ability estimates (0.944). Another question with this dataset was about the sensitivity of the reliability estimates to manipulated IRT item characteristics and ability distributions (described in the Evaluations section).</p> <p>2 Table Means and Standard Deviations (SDs) of the IRT Parameters of the Study Datasets</p> <p> <ephtml> <table><thead><tr><th>Dataset/IRT Model</th><th><italic>N</italic> Items</th><th>Mean <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0174" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">a</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{a}}}_{\bm{i}}}$</annotation></semantics></math></p> (<italic>SD</italic><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0175" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">a</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{a}}}_{\bm{i}}}$</annotation></semantics></math></p>)</th><th>Mean <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0176" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">b</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{b}}}_{\bm{i}}}$</annotation></semantics></math></p> (<italic>SD</italic><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0177" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">b</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{b}}}_{\bm{i}}}$</annotation></semantics></math></p>)</th><th>Mean <p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0178" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{c}}}_{\bm{i}}}$</annotation></semantics></math></p> (<italic>SD</italic><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0179" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><msub><mi mathvariant="bold-italic">c</mi><mi mathvariant="bold-italic">i</mi></msub><annotation encoding="application/x-tex">${{{\bm{c}}}_{\bm{i}}}$</annotation></semantics></math></p>)</th></tr></thead><tbody><tr><td>First Dataset/Rasch</td><td>36</td><td>1.00 (0.00)</td><td>−0.10 (0.99)</td><td>0.00 (0.00)</td></tr><tr><td>First Dataset/3PL</td><td>36</td><td>0.86 (0.26)</td><td>0.31 (1.03)</td><td>0.16 (0.07)</td></tr><tr><td>Second Dataset/1PL</td><td>100</td><td>0.54 (0.00)</td><td>−0.70 (1.75)</td><td>0.00 (0.00)</td></tr><tr><td>Second Dataset/2PL</td><td>100</td><td>0.58 (0.28)</td><td>−0.56 (1.83)</td><td>0.00 (0.00)</td></tr><tr><td>Second Dataset/3PL</td><td>100</td><td>0.93 (0.57)</td><td>0.21 (1.75)</td><td>0.21 (0.08)</td></tr></tbody></table> </ephtml> </p> <p>For the second dataset, 1PL, 2PL, and 3PL IRT parameter estimates were obtained from a medical education exam used in first‐ and second‐year medical courses. IRTPRO 6.0 (Cai et al., [<reflink idref="bib3" id="ref67">3</reflink>]) was used for IRT calibrations, based on the administration data of 100‐item test forms with 1,316 examinees. For the marginal maximum likelihood estimation, the minimum and maximum abilities were set at –5 and 5. Distributions of the IRT parameters are summarized in Table 2. The original, unscaled item parameter estimates were on scales corresponding to a standard normal ability distribution. Reliability estimates were produced for the 1PL, 2PL, and 3PL IRT models for the considered IRT‐based estimators described in Table 1, using a quadratured approximation of a standard normal distribution for <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0180" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Pr</mi><mo>(</mo><mi>θ</mi><mo>)</mo></mrow><annotation encoding="application/x-tex">${\mathrm{Pr}}(\theta)$</annotation></semantics></math> </ephtml> , where <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0181" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> ranged from –5 to 5 in 0.2 units. Because test length was an anticipated issue in the reliability estimates, results were produced to address the question of how reliability varied for tests of different lengths, for one 100‐item test form, and for shortened forms of randomly selected sets of 25 and 50 items. An additional question with this dataset addressed conditional biases and CSEMs of the estimators (described below).</p> <hd id="AN0190280393-16">Evaluations</hd> <p>A major interest of this study was considering the recursion and IRT‐based reliability estimates of the MLE, TCC, and EAP estimators across the tests, datasets, IRT models, and test lengths. Results were also considered with respect to different manipulations of IRT parameter accuracy, for the purpose of gauging their implications for practice. Reliabilities obtained for each dataset were also produced by simulating 5,000 sets of observed item scores from the IRT models, applying traditional scoring procedures (i.e., MLE, TCC, and EAP procedures applied to the observed simulated item scores), and then estimating reliabilities. As described in the results, these simulations revealed some limitations for particular estimators (i.e., the MLE and EAP estimators for the 3PL model).</p> <p>A specific question considered with the first dataset was the sensitivity of the reliability estimates to inaccuracies in the IRT model parameters. This issue was addressed by re‐computing reliabilities after introducing manipulations to the original parameters. The considered manipulations were systematic changes of about two standard deviations in the item characteristics, chosen to provide insight about the impact of relatively large variations in IRT parameters on the reliability estimates:</p> <p></p> <ulist> <item> Values of 2.0 were added to the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0182" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> estimates to consider the impact of a more difficult test on estimated reliability (Rasch and 3PL IRT models).</item> <p></p> <item> Values of 0.50 were subtracted from the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0183" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> estimates to consider the impact of a less discriminating test on estimated reliability (3PL IRT model).</item> <p></p> <item> Values of 0.15 were added and subtracted from the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0184" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> estimates to consider the impact of more and less guessing on estimated reliability (3PL IRT model).</item> </ulist> <p>Another considered manipulation introduced normally distributed random error of mean = 0.0 and standard deviation = 2.0, N(0.0, 2.0), to the ability values (Rasch and 3PL IRT models).</p> <p>For the second dataset, in addition to manipulating test length, conditional biases and CSEMs were also computed and considered. These enabled some discussion of conditional biases of the respective estimators in comparison with each other (Bayesian vs. non‐Bayesian, scores vs. <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0185" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math> </ephtml> estimates), and also in comparison with asymptotically derived biases and CSEMs.</p> <hd id="AN0190280393-17">Results</hd> <p></p> <hd id="AN0190280393-18">First Dataset: Marginal Reliabilities</hd> <p>The reliabilities of the recursion and IRT‐based estimators with and without manipulation are shown in Table 3, based on the Rasch and 3PL IRT models for the first dataset. The reliabilities based on the Rasch model are lower than those based on the 3PL model, and are similar for the score, EAP estimators and the coefficient alpha of summed scores in the observed data (0.868, 0.864 and 0.860). The reliabilities of the MLE and TCC ability estimators are slightly lower (0.859). For the 3PL model, the reliabilities of the MLE pattern scores and ability and the EAP are similarly high (0.936 or higher), and are similar to the coefficient alpha of MLE‐based weighted item scores in the observed data (0.944). The reliabilities of the TCC scores and abilities based on the 3PL model are lower (0.858 and 0.768).</p> <p>3 Table Recursion‐ and IRT‐Based Reliabilities for the First Dataset: Rasch and 3PL IRT Models; with and without Manipulations of the IRT Item and Ability Parameters</p> <p> <ephtml> <table><thead><tr><th>IRT Model</th><th>Manipulation</th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0186" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i})$</annotation></semantics></math></p></th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0187" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}})$</annotation></semantics></math></p></th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0188" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,TCC}}{{u}_i})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0189" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><mo>)</mo></mrow><annotation encoding="application/x-tex">$Rel({{{{\hat{\theta }}}_{TCC}}})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0190" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><mo>(</mo><mrow><mi>E</mi><mi>A</mi><mi>P</mi></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$Rel({EAP})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0191" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><msub><mo>(</mo><mi>s</mi></msub><mrow><mi>E</mi><mi>A</mi><mi>P</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$Rel{{(}_s}EAP)$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>Rasch</td><td>None</td><td>0.868</td><td>0.859</td><td>0.868</td><td>0.859</td><td>0.864</td><td>0.864</td></tr><tr><td /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0192" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p> + N(0.0, 2.0)</td><td>0.622</td><td>0.665</td><td>0.641</td><td>0.616</td><td>0.715</td><td>0.677</td></tr><tr><td align="left" /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0193" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>b</mi><mi>i</mi><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math></p> + 2.0</td><td>0.828</td><td>0.687</td><td>0.828</td><td>0.687</td><td>0.791</td><td>0.791</td></tr><tr><td>3PL</td><td>None</td><td>0.948</td><td>0.936</td><td>0.858</td><td>0.768</td><td>0.941</td><td>0.839</td></tr><tr><td /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0194" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>θ</mi><annotation encoding="application/x-tex">$\theta $</annotation></semantics></math></p> + N(0.0, 2.0)</td><td>0.811</td><td>0.794</td><td>0.622</td><td>0.665</td><td>0.828</td><td>0.715</td></tr><tr><td align="left" /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0195" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>b</mi><mi>i</mi><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math></p> + 2.0</td><td>0.913</td><td>0.808</td><td>0.600</td><td>0.341</td><td>0.858</td><td>0.503</td></tr><tr><td align="left" /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0196" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>a</mi><mi>i</mi><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math></p> – 0.5</td><td>0.856</td><td>0.843</td><td>0.608</td><td>0.553</td><td>0.849</td><td>0.600</td></tr><tr><td align="left" /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0197" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>c</mi><mi>i</mi><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math></p> – 0.15</td><td>0.929</td><td>0.903</td><td>0.902</td><td>0.861</td><td>0.918</td><td>0.887</td></tr><tr><td align="left" /><td><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0198" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mi>c</mi><mi>i</mi><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math></p> + 0.15</td><td>0.963</td><td>0.956</td><td>0.792</td><td>0.680</td><td>0.959</td><td>0.775</td></tr></tbody></table> </ephtml> </p> <p>2 <emph>Note</emph>. Italicized and underlined reliabilities based on pattern scores and ability estimates from pattern scores were shown to be unrealistically high in data simulated to approximate practical scoring situations where true abilities were unknown.</p> <p>Table 3 shows the implications of manipulations of abilities and IRT item parameters on the reliability estimates. Most of the manipulations considered resulted in reduced reliabilities, which were expected results given that most of the manipulations made the tests more different from the test takers. The introduction of random error to true abilities resulted in greater reliability reductions than increased <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0199" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> values for the Rasch model, and resulted in some of the largest reliability reductions for the 3PL model. Manipulations to increase the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0200" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>b</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{b}_i}$</annotation></semantics></math> </ephtml> values had the greatest impact on the ability estimates for the Rasch model results, and for the 3PL model had the greatest impact on estimators based on summed scores (TCC and <subs>s</subs>EAP estimators). Manipulations to reduce the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0201" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>a</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{a}_i}$</annotation></semantics></math> </ephtml> values for the 3PL model resulted in reduced reliabilities, particularly for the estimators based on summed scores. Results for manipulating the <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0202" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values are noteworthy in that they indicate different effects on reliabilities of summed scores versus summed pattern scores. That is, decreases in <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0203" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values resulted in increases in reliabilities of estimators based on summed scores, whereas increases in <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0204" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><msub><mi>c</mi><mi>i</mi></msub><annotation encoding="application/x-tex">${{c}_i}$</annotation></semantics></math> </ephtml> values resulted in increased reliabilities of estimators based on pattern scores.</p> <p>The use of simulated item response data to evaluate the IRT scoring approaches and estimators showed that reliabilities obtained from simulated scores were usually similar to those shown in Table 3. One exception was the summed pattern scores and associated ability estimators for the 3PL model, which had considerably reduced reliabilities in the simulated data versus the reliabilities reported in the table (more than 0.1 lower). This issue is shown in the italicized and underlined reliability values in Table 3 and is described in detail in the Discussion section.</p> <hd id="AN0190280393-19">Second Dataset: Marginal Reliabilities and Conditional Accuracies</hd> <p></p> <hd id="AN0190280393-20">Marginal reliabilities</hd> <p>The reliabilities of the recursion and IRT‐based estimators for tests of different lengths are shown in Table 4, based on 1PL, 2PL, and 3PL IRT models for the second dataset. The major distinction in results is for reliabilities of summed scores versus pattern scores, especially for the 3PL IRT model. For estimators based on summed scores (TCC and <subs>s</subs>EAP estimators), reliabilities are very similar for the 1PL and 2PL models, and are usually reduced for the 3PL model. For estimators based on pattern scores (MLE and EAP estimators), reliabilities increase from the 1PL to the 3PL model. The reliabilities are smallest for the shorter test length of 25 items, while for the test length of 100 items are, as expected, higher. Similar to the first dataset, the use of simulated response data to evaluate the IRT scoring approaches and estimators for the second dataset indicated that reliability estimates based on 3PL pattern scores were implausibly high, more so for tests of 25 items than for the tests of 100 items. This issue is shown in the italicized and underlined reliability values in Table 4 and is described in detail in the Discussion section.</p> <p>4 Table Recursion‐ and IRT‐Based Reliabilities for the Second Dataset: 1PL, 2PL, and 3PL IRT Models; 25‐, 50‐, and 100‐Item Tests</p> <p> <ephtml> <table><thead><tr><th>IRT Model</th><th>Test Length</th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0205" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,MLE}}{{u}_i})$</annotation></semantics></math></p></th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0206" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>M</mi><mi>L</mi><mi>E</mi></mrow></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">${{\hat{\theta }}_{MLE}})$</annotation></semantics></math></p></th><th><italic>Rel</italic>(<p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0207" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><msub><mo>∑</mo><mi>i</mi></msub><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><msub><mi>u</mi><mi>i</mi></msub><mrow><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$\sum_i {{w}_{i,TCC}}{{u}_i})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0208" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><mo>(</mo><msub><mover accent="true"><mi>θ</mi><mo>̂</mo></mover><mrow><mi>T</mi><mi>C</mi><mi>C</mi></mrow></msub><mo>)</mo></mrow><annotation encoding="application/x-tex">$Rel({{{{\hat{\theta }}}_{TCC}}})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0209" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><mo>(</mo><mrow><mi>E</mi><mi>A</mi><mi>P</mi></mrow><mo>)</mo></mrow><annotation encoding="application/x-tex">$Rel({EAP})$</annotation></semantics></math></p></th><th><p><math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0210" xmlns="http://www.w3.org/1998/Math/MathML"><semantics xmlns=""><mrow><mi>R</mi><mi>e</mi><mi>l</mi><msub><mo>(</mo><mi>s</mi></msub><mrow><mi>E</mi><mi>A</mi><mi>P</mi><mo>)</mo></mrow></mrow><annotation encoding="application/x-tex">$Rel{{(}_s}EAP)$</annotation></semantics></math></p></th></tr></thead><tbody><tr><td>1PL</td><td>25 Items</td><td>0.598</td><td>0.595</td><td>0.598</td><td>0.595</td><td>0.598</td><td>0.598</td></tr><tr><td>2PL</td><td align="left" /><td>0.708</td><td>0.695</td><td>0.674</td><td>0.662</td><td>0.704</td><td>0.671</td></tr><tr><td>3PL</td><td align="left" /><td>0.902</td><td>0.882</td><td>0.679</td><td>0.610</td><td>0.888</td><td>0.664</td></tr><tr><td>1PL</td><td>50 Items</td><td>0.744</td><td>0.742</td><td>0.744</td><td>0.742</td><td>0.743</td><td>0.743</td></tr><tr><td>2PL</td><td align="left" /><td>0.801</td><td>0.793</td><td>0.770</td><td>0.761</td><td>0.798</td><td>0.767</td></tr><tr><td>3PL</td><td align="left" /><td>0.926</td><td>0.918</td><td>0.770</td><td>0.743</td><td>0.920</td><td>0.762</td></tr><tr><td>1PL</td><td>100 Items</td><td>0.854</td><td>0.852</td><td>0.854</td><td>0.852</td><td>0.853</td><td>0.853</td></tr><tr><td>2PL</td><td align="left" /><td>0.879</td><td>0.874</td><td>0.856</td><td>0.850</td><td>0.876</td><td>0.853</td></tr><tr><td>3PL</td><td align="left" /><td>0.965</td><td>0.960</td><td>0.854</td><td>0.833</td><td>0.961</td><td>0.845</td></tr></tbody></table> </ephtml> </p> <p>3 <emph>Note</emph>. Italicized and underlined reliabilities based on pattern scores and ability estimates from pattern scores were shown to be unrealistically high in data simulated to approximate practical scoring situations where true abilities were unknown.</p> <hd id="AN0190280393-21">Conditional accuracies</hd> <p>To gain more insight about the tendencies of the IRT ability estimators, the estimators' conditional biases and CSEMs were computed across the ability scale. Figures 2–4 present the conditional biases (Figure 2a‐c), CSEMs for IRT‐based scores (Figure 3a‐c) and CSEMs for ability estimators (Figure 4a‐c) for the 25‐item test using the 1PL, 2PL, and 3PL IRT models.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01dec25/jedm70008-fig-0002.jpg?ephost1=dGJyMNLe80Sepq84zdnyOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm70008-fig-0002.jpg" title="2 Conditional biases based on generalized recursion for 1PL (a), 2PL (b), and 3PL (c) IRT model score and ability estimates and a 25‐item test." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01dec25/jedm70008-fig-0003.jpg?ephost1=dGJyMNLe80Sepq84zdnyOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm70008-fig-0003.jpg" title="3 CSEMs based on generalized recursion for 1PL (a), 2PL (b), and 3PL (c) IRT model score estimates and a 25‐item test." /> </p> <p></p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/MEA/01dec25/jedm70008-fig-0004.jpg?ephost1=dGJyMNLe80Sepq84zdnyOLCmsE6epq5Srqa4SK6WxWXS" alt="jedm70008-fig-0004.jpg" title="4 CSEMs based on generalized recursion for 1PL (a), 2PL (b), and 3PL (c) IRT model ability estimates and a 25‐item test." /> </p> <p></p> <p>The conditional biases of the estimators shown in Figure 2 reflect and expand on previously noted results (Lord, [<reflink idref="bib19" id="ref68">19</reflink>]; Lord, [<reflink idref="bib20" id="ref69">20</reflink>]). The conditional biases of MLE are positively related to ability (usually negative for lower abilities and positive for higher abilities), whereas the conditional biases of the EAP estimators are negatively related to ability. The conditional biases of TCC are often similar to those of the MLE, though they are larger and somewhat different for the 3PL model where guessing parameters appear to affect TCC ability estimation. The conditional biases of the summed pattern scores and the summed scores are nearly zero.</p> <p>The CSEMs shown in Figures 3 and 4 reflect and expand on previously noted results (Kolen, [<reflink idref="bib11" id="ref70">11</reflink>]; Kolen et al., [<reflink idref="bib13" id="ref71">13</reflink>]; Lee et al., [<reflink idref="bib15" id="ref72">15</reflink>]). The shapes of CSEMs for summed pattern scores and summed scores are concave and reflective of larger test score scales than CSEMs of abilities. For the 1PL model, the CSEMs of the summed pattern scores and summed scores differ by a constant reflective of the discrimination value for all items, <ephtml> <math display="inline" altimg="urn:x-wiley:00220655:media:jedm70008:jedm70008-math-0211" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><msub><mi>a</mi><mi>i</mi></msub><mo>≈</mo><mn>0.54</mn></mrow><annotation encoding="application/x-tex">$D{{a}_i} \approx 0.54$</annotation></semantics></math> </ephtml> (Figure 3a). The CSEMs of the summed scores based on the 3PL model are relatively large for lower abilities where guessing parameters affect estimation (Figure 3c). The CSEMs of MLE and TCC abilities are lower and similar for the 1PL and 2PL IRT models (Figure 4a and b), but the CSEMs of the TCC are larger than those of the MLE for the 3PL IRT model (Figure 4c). The EAP estimators usually have the smallest CSEMs for the three IRT models.</p> <hd id="AN0190280393-25">Discussion</hd> <p>IRT scoring procedures are used in several educational, psychological and certification tests, and the estimation of the accuracy and statistical characteristics of these scores has continual developments. The current study builds on several recent studies of reliability and conditional accuracy for IRT estimators. For IRT‐based reliabilities, the current study expands on previous considerations of recursion procedures for total score variances (Andersson & Xin, [<reflink idref="bib1" id="ref73">1</reflink>]; Kim & Feldt, [<reflink idref="bib9" id="ref74">9</reflink>]), and generalized recursion procedures for nominal, noninteger scores (Kim, [<reflink idref="bib8" id="ref75">8</reflink>]) to consider weighted test scores corresponding to MLE and TCC estimators, and also EAP estimators. This means that reliabilities can be estimated from error variances and total variances calculated from a single table of joint probabilities for observed and true IRT results. This approach has some practical advantages, in that it avoids some of the challenges with asymptotic approaches to biases and error variances (Equations 4 and 5) that can result in implausibly large conditional biases and errors (Figure 1) and lead practitioners to consider using arbitrary values (Smarter Balanced, [<reflink idref="bib26" id="ref76">26</reflink>]). The estimation of error and total variances from a single probability table also ensures that total variances are always sums of error and true score variances. Although computations reflect a recursion procedure that requires more time than for information‐based estimation, computational time increases can be minimized through efficient programming algorithms, such as those described in Kolen and Brennan ([<reflink idref="bib12" id="ref77">12</reflink>]).</p> <p>The consideration of a wide range of IRT estimators, including IRT‐based abilities and test scores, allowed for broad comparisons of the reliabilities, biases, and CSEMs of each estimator. The reliabilities of all estimators reflected some of the expected trends, such as increases with increased test length (Table 4), and values that were usually higher when the IRT parameters were accurate (Table 3). Results also showed distinctions among the estimators, such as that reliabilities of estimators based on summed scores (TCC and <subs>s</subs>EAP estimators) were less influenced by the use of 1PL, 2PL, and 3PL IRT models than the estimators based on pattern scores (MLE and EAP estimators). In addition, the manipulations to increase and decrease item guessing parameters for the 3PL model had different influences on reliabilities of the summed scores versus the pattern scores. The finding that reliabilities for estimators based on summed scores increase with reductions of item guessing parameters whereas reliabilities for estimators based on pattern scores increase with increases in item guessing parameters appears to reflect properties of summed scores versus pattern scores. That is, the results appear to indicate that summed scores reflect more information and reliability when there is less guessing and more observable low scores to model, whereas pattern scores and associated ability estimates exhibit more information and reliability with increased guessing where correct item responses from low scoring examinees are downweighted (Lord, [<reflink idref="bib18" id="ref78">18</reflink>]).</p> <p>The conditional accuracy results in Figures 2–4 add more detail about differences between scores and ability estimators. The figures indicate that scores exhibit relatively small biases by nature of the estimation (Equations 3 and 6), with inverse U‐shaped CSEMs, whereas ability estimators reflect unique bias patterns and U‐shaped CSEMs (Lord, [<reflink idref="bib19" id="ref79">19</reflink>]; Lord, [<reflink idref="bib20" id="ref80">20</reflink>], Kolen et al., [<reflink idref="bib13" id="ref81">13</reflink>]; Lee et al., [<reflink idref="bib15" id="ref82">15</reflink>]). These conditional accuracies suggest that summed pattern scores may be especially beneficial, in that these sums of differentially weighted item scores avoid some of the estimation difficulties of the summed scores, while also exhibiting higher reliability and less bias than IRT ability estimators. Summed pattern scores have received less research attention than summed scores and IRT ability estimates. One implication of this study is that more research is warranted for the viability of summed IRT‐based pattern scores.</p> <p>The procedures and results summarized in this study reflect a general limitation of most IRT‐based procedures in practice, in that IRT terms are often regarded as parameters that are fixed and accurate when used in future test administrations. These are relatively strong assumptions that are especially strong for the more complex IRT models. The assumptions of fixed and accurate parameters are made for item difficulties with the 1PL and Rasch IRT models, for item difficulties and discriminations with the 2PL IRT model, and for item difficulties, discriminations and guessing with the 3PL model.</p> <p>A particularly strong set of assumptions is made with IRT estimators based on pattern scoring for the 3PL model, where the assumptions of accurate parameters are made not only for the three parameters per item, but also for the ability estimate that contributes to the weighting of correct item responses in the pattern scoring. Simulations summarized in this study indicated that the reliabilities for pattern scores based on the 3PL model are implausibly high based on the proposed approach, and may perhaps be regarded as an upper bound that is realized only when abilities are known and tests are long and highly reliable. Particularly for shorter, less reliable tests, the recursion approach is discouraged for reliability estimation of pattern scores based on the 3PL model as it does not reflect the scoring inaccuracies of the pattern scores and of the weights of correct item responses that are likely to occur in practice. These findings suggest that more realistic reliability estimates for pattern scores based on the 3PL model can be obtained by introducing errors to abilities to reflect the scoring errors likely to occur in practice with these methods (Table 3).</p> <p>The generalized recursion procedure for IRT‐based abilities and test scores may have usefulness for applications other than marginal reliabilities and conditional accuracies. Another application of interest for K‐12 testing and for certification and licensure testing is accuracy evaluations at specific cut scores (i.e., classification accuracy and consistency). Approaches to classification have focused most extensively on scores and strong true score models (Hanson & Brennan, 1990; Livingston & Lewis, [<reflink idref="bib17" id="ref83">17</reflink>]). Some approaches have been considered that make use of IRT models, but mostly for estimating distributions and classification results of summed scores rather than proficiencies (Lee et al., [<reflink idref="bib16" id="ref84">16</reflink>]). Some classification procedures have been considered for IRT ability estimators, but in limited ways, such as ability estimates for the Rasch model approximated with summed score distributions (Hunynh, [<reflink idref="bib6" id="ref85">6</reflink>]), and procedures for classification accuracy (not consistency) based on test information functions for MLE estimators (Guo, [<reflink idref="bib4" id="ref86">4</reflink>]; Rudner, [[<reflink idref="bib23" id="ref87">23</reflink>]]). The generalized recursion approach considered in the current study may have potential for developing classification accuracy and consistency procedures for a more expansive set of IRT‐based score and ability estimators. Such procedures would not rely on test information and the previously noted difficulties of test information in practice. In addition, procedures based on generalized recursion have potential for an integrated set of accuracy and classification indices, such that marginal reliabilities, SEMs, CSEMs, and classification consistency and accuracy could be produced for a particular score or ability estimator and chosen IRT model. More research about these possibilities is encouraged.</p> <ref id="AN0190280393-26"> <title> References </title> <blist> <bibl id="bib1" idref="ref18" type="bt">1</bibl> <bibtext> Andersson, B., & Xin, T. (2018). Large sample confidence intervals for item response theory reliability coefficients. Educational and Psychological Measurement, 78, 32 – 45.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref25" type="bt">2</bibl> <bibtext> Baker, F. B., & Kim, S. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY : Marcel Dekker.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref67" type="bt">3</bibl> <bibtext> Cai, L., Thissen, D., & du Toit, S. H. C. (2022). IRTPRO 6.0: Flexible, multidimensional, multiple categorical IRT modeling [Computer software]. Chapel Hill, NC : Vector Psychometric Group.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref86" type="bt">4</bibl> <bibtext> Guo, F. (2006). Expected classification accuracy using the latent distribution. Practical Assessment Research & Evaluation, 11.</bibtext> </blist> <blist> <bibl id="bib5" type="bt">5</bibl> <bibtext> Hanson, B.A., & Brennan, R.L. (1990). An investigation of classification consistency indexes estimated under alternative strong true score models. Journal of Educational Measurement, 27, 345 – 359.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref85" type="bt">6</bibl> <bibtext> Hunynh, H. (1990). Error rates in competency testing when test retaking is permitted. Journal of Educational Statistics, 15, 39 – 52.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref16" type="bt">7</bibl> <bibtext> Kim, S. (2012). A note on the reliability coefficients for item response model‐based ability estimates. Psychometrika, 77, 153 – 162.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref22" type="bt">8</bibl> <bibtext> Kim, S. (2013). Generalization of the Lord‐Wingersky algorithm to computing the distribution of summed test scores based on real‐number item scores. Journal of Educational Measurement, 50, 381 – 389.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref17" type="bt">9</bibl> <bibtext> Kim, S., & Feldt, L. S. (2010). The estimation for the IRT reliability coefficient and its lower and upper bounds, with comparisons to CTT reliability statistics. Asia Pacific Education Review, 11, 179 – 188.</bibtext> </blist> <blist> <bibtext> Kim, S., & Moses, T. (2024). Item response theory proficiency estimation methods under multistage testing. In D. Yan, A. A. von Davier, & D. J. Weiss (Ed.), Research for practical issues and solutions in computerized multistage testing (Vol. 1, pp. 185 – 199). Routledge.</bibtext> </blist> <blist> <bibtext> Kolen, M. H. (2006). Scaling and Norming. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT : Praeger.</bibtext> </blist> <blist> <bibtext> Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking. New York, NY : Springer.</bibtext> </blist> <blist> <bibtext> Kolen, M. J., Hanson, B. A., & Brennan, R. L. (1992). Conditional standard errors of measurement for scale scores. Journal of Educational Measurement 29, 285 – 307.</bibtext> </blist> <blist> <bibtext> Kolen, M. J., Tong, Y, & Brennan, R. L. (2010). Scoring and scaling educational tests. In A. A. von Davier (Eds.) Statistical models for test equating, scaling, and linking (pp. 43 – 58). Springer.</bibtext> </blist> <blist> <bibtext> Lee, W., Brennan, R. L., & Kolen, M. J. (2000). Estimators of conditional scale‐score standard errors of measurement: A simulation study. Journal of Educational Measurement, 37, 1 – 20.</bibtext> </blist> <blist> <bibtext> Lee, W., Kim, S. Y., Choi, J., & Kang, Y. (2020). IRT approaches to modeling scores on mixed‐format tests. Journal of Educational Measurement, 57, 230 – 254.</bibtext> </blist> <blist> <bibtext> Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179 – 197.</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ : Lawrence Erlbaum Associates.</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1983). Unbiased estimators of ability parameters, of their variance, and of other parallel‐forms reliability. Psychometrika, 48, 233 – 245.</bibtext> </blist> <blist> <bibtext> Lord, F. M. (1986). Maximum likelihood and Bayesian parameter estimation in item response theory. Journal of Educational Measurement, 23, 157 – 162.</bibtext> </blist> <blist> <bibtext> Lord, F. J., & Wingersky, M. S. (1984). Comparison of IRT true‐score and equipercentile observed‐score "equatings." Applied Measurement in Education, 8, 452 – 461.</bibtext> </blist> <blist> <bibtext> Moses, T., & Kim, Y. (2017). Stabilizing conditional standard errors of measurement in scale score transformations. Journal of Educational Measurement, 54, 184 – 199.</bibtext> </blist> <blist> <bibtext> Rudner, L. J. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment Research & Evaluation, 7.</bibtext> </blist> <blist> <bibtext> Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment Research & Evaluation, 10.</bibtext> </blist> <blist> <bibtext> Samajima, F. (1994). Estimation of reliability coefficients using the test information function and its modifications. Applied Psychological Measurement, 18, 229 – 244.</bibtext> </blist> <blist> <bibtext> Smarter Balanced. (2016). Smarter Balanced Assessment Consortium: 2013–14 Technical Report. Retrieved from https://portal.smarterbalanced.org/library/en/2013‐14‐technical‐report.pdf/.</bibtext> </blist> <blist> <bibtext> Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427 – 450.</bibtext> </blist> <blist> <bibtext> Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT : American Council on Education/Praeger.</bibtext> </blist> </ref> <aug> <p>By Tim Moses and YoungKoung Kim</p> <p>Reported by Author; Author</p> <p></p> <p>TIM MOSES is Research Professor, Buros Center for Testing, University of Nebraska‐Lincoln, 21 Teachers College Hall, Lincoln, NE 68588; tmoses2@unl.edu. His primary research interests include psychometric methods and statistical applications to testing.</p> <p>YOUNGKOUNG KIM is Senior Psychometrician, College Board, 225 Liberty Street, New York, NY, 10281; ykim@collegeboard.org. Her primary research interests include automatic essay scoring, applications of AI‐powered scoring to large‐scale assessments, and general IRT modeling, particularly IRT equating and scaling.</p> </aug> <nolink nlid="nl1" bibid="bib18" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib10" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib11" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib28" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib26" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib19" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib27" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib25" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib21" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref45"></nolink> <nolink nlid="nl12" bibid="bib13" firstref="ref56"></nolink> <nolink nlid="nl13" bibid="bib15" firstref="ref57"></nolink> <nolink nlid="nl14" bibid="bib22" firstref="ref58"></nolink> <nolink nlid="nl15" bibid="bib17" firstref="ref83"></nolink> <nolink nlid="nl16" bibid="bib16" firstref="ref84"></nolink> <nolink nlid="nl17" bibid="bib23" firstref="ref87"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1491360
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Tim+Moses%22">Tim Moses</searchLink><br /><searchLink fieldCode="AR" term="%22YoungKoung+Kim%22">YoungKoung Kim</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Measurement%22"><i>Journal of Educational Measurement</i></searchLink>. 2025 62(4):718-739.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Probability%22">Probability</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Distributions%22">Statistical Distributions</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Length%22">Test Length</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jedm.70008
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-0655<br />1745-3984
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study considers the estimation of marginal reliability and conditional accuracy measures using a generalized recursion procedure with several IRT-based ability and score estimators. The estimators include MLE, TCC, and EAP abilities, and corresponding test scores obtained with different weightings of the item scores. We consider reliability estimates for 1-, 2-, and 3-parameter logistic IRT models (1PL, 2PL, and 3PL) for tests of dichotomously scored items, using IRT calibrations from two datasets. The generalized recursion procedure is shown to produce conditional probability distributions for the considered IRT estimators that can be used in the estimation of marginal reliabilities and conditional accuracies (biases and CSEMs). These reliabilities and conditional accuracies are shown to have less extreme and more plausible values compared to theoretical approaches based on test information. The proposed recursion procedure for the estimation of reliability and other accuracy measures are demonstrated for testing situations involving different test lengths, IRT models, and different types of IRT parameter inaccuracies.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1491360
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1491360
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jedm.70008
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 718
    Subjects:
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Probability
        Type: general
      – SubjectFull: Statistical Distributions
        Type: general
      – SubjectFull: Test Length
        Type: general
    Titles:
      – TitleFull: IRT Scoring and Recursion for Estimating Reliability and Other Accuracy Indices
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Tim Moses
      – PersonEntity:
          Name:
            NameFull: YoungKoung Kim
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0022-0655
            – Type: issn-electronic
              Value: 1745-3984
          Numbering:
            – Type: volume
              Value: 62
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Educational Measurement
              Type: main
ResultId 1