View in EDS HTML Full Text PDF Full Text

Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM

Saved in:
Bibliographic Details
Title: Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM
Language: English
Authors: Sarah Depaoli (ORCID 0000-0002-1277-0462), Sonja D. Winter (ORCID 0000-0002-2203-002X), Haiyan Liu
Source: Structural Equation Modeling: A Multidisciplinary Journal. 2024 31(4):604-625.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 22
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Descriptors: Structural Equation Models, Bayesian Statistics, Comparative Testing, Evaluation Utilization, Test Selection, Robustness (Statistics), Goodness of Fit
DOI: 10.1080/10705511.2023.2280952
ISSN: 1070-5511
1532-8007
Abstract: We extended current knowledge by examining the performance of several Bayesian model fit and comparison indices through a simulation study using the confirmatory factor analysis. Our goal was to determine whether commonly implemented Bayesian indices can detect specification errors. Specifically, we wanted to uncover any differences in detecting under-fitting or over-fitting a model. We examined a conventional Bayesian fit index (the posterior predictive p-value), approximate Bayesian fit indices (Bayesian RMSEA, CFI, and TLI), and model comparison indices (BIC and DIC). We varied the type and severity of model mis-specification, sample size, and priors. We focused on the ability of these indices to detect model under- or over-fitting. We provide practical advice for applied researchers regarding how to assess and compare models using these common indices implemented in the Bayesian framework.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1431135
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGTJvf4Drqb-nCVjMh2JPWLAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDFQJHb722UlOmXjBKAIBEICBm_6Bsg8OJkF-1XKRVAl_xnAXuzYbo2BPs9dG9Z5HSI2NeNDdJJ9h7D9n5oxGNgZNe4gsiNvpI5NHEEd7TPrlz48KzXY5XjzxlflQ5Bvr-ZHKf5wFG6m6eDX_y-xkjDP9sqC28ng3Yhzf2fXDenBUHWY8CSlVAW4K7bMm_08-DFUXRT7eWboIcH6Ne2Lqg41UczaRQ8Vpdjfc5Vpw
Text:
  Availability: 1
  Value: <anid>AN0178359442;7mz01jul.24;2024Jul12.05:53;v2.2.500</anid> <title id="AN0178359442-1">Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM </title> <p>We extended current knowledge by examining the performance of several Bayesian model fit and comparison indices through a simulation study using the confirmatory factor analysis. Our goal was to determine whether commonly implemented Bayesian indices can detect specification errors. Specifically, we wanted to uncover any differences in detecting under-fitting or over-fitting a model. We examined a conventional Bayesian fit index (the posterior predictive p-value), approximate Bayesian fit indices (Bayesian RMSEA, CFI, and TLI), and model comparison indices (BIC and DIC). We varied the type and severity of model mis-specification, sample size, and priors. We focused on the ability of these indices to detect model under- or over-fitting. We provide practical advice for applied researchers regarding how to assess and compare models using these common indices implemented in the Bayesian framework.</p> <p>Keywords: Approximate model fit; Bayesian model comparison; BIC; DIC</p> <p>The Bayesian methodology has gained popularity in a variety of modeling situations, and many methodological advancements have been made specifically within the context of Bayesian structural equation modeling (BSEM). For example, the Bayesian estimation framework can improve the estimation accuracy for latent variable and mixture models (e.g., Depaoli, [<reflink idref="bib5" id="ref1">5</reflink>]; Liu et al., [<reflink idref="bib18" id="ref2">18</reflink>]), add flexibility to the conventional measurement invariance testing process (van de Schoot et al., [<reflink idref="bib32" id="ref3">32</reflink>]), and circumvent convergence problems that can arise in traditional frequentist estimation (e.g., Smid et al., [<reflink idref="bib30" id="ref4">30</reflink>]; Yuan et al., [<reflink idref="bib40" id="ref5">40</reflink>]). In addition to accuracy and estimation improvements, there have also been practical advancements making Bayesian estimation a viable alternative. Specifically, computational and software advances have made Bayesian estimation a useful strategy for SEMs. With latent variable software (e.g., M<emph>plus</emph>; L. K. Muthén & Muthén, [<reflink idref="bib25" id="ref6">25</reflink>]) and many different packages in R (Merkle & Rosseel, [<reflink idref="bib22" id="ref7">22</reflink>], e.g., blavaan), applied researchers have a variety of tools for implementing these methods with relative ease.</p> <p>With the continued rise in the use of Bayesian estimation within SEM (Van de Schoot et al., [<reflink idref="bib33" id="ref8">33</reflink>]), it is important to gain a full understanding of the various components important in assessing model performance. Previous literature (see e.g., Kaplan, [<reflink idref="bib14" id="ref9">14</reflink>]) has overwhelmingly concluded that specification errors can influence model performance and the accuracy of results and conclusions drawn. In the current investigation, we focus on detecting model mis-specification within BSEM as a means to improve the accuracy and conclusions drawn in applied BSEM contexts.</p> <p>Applied researchers implementing SEMs are careful to examine model results for indications of model mis-specification. There has been an abundance of methodological research conducted in the frequentist SEM setting, pointing toward strategies and tools that can be used to best detect specification error and improve the accuracy of the modeling results being narrated. Within the Bayesian estimation framework, there have been some recent advances regarding tools for detecting specification errors (e.g., Garnier-Villarreal & Jorgensen, [<reflink idref="bib7" id="ref10">7</reflink>]), but there is a lack of evidence pointing toward how these tools perform under different types of mis-specification.</p> <p>The current investigation examines how well a variety of the most common Bayesian model fit and comparison tools can detect the two main forms of model mis-specification. Specifically, we focus on under-fitting a model by not modeling parameters that indeed exist at the population level, as well as over-fitting a model by including "extra" model parameters according to the population model. We examine specification errors closely within the Bayesian framework by assessing the ability of fit and comparison measures to detect under- and over-fitted models. These results will be important to increase our understanding of detecting model mis-specification within BSEM, and thus improve recommendations that the methodological literature can provide to applied researchers implementing these tools in their own work.</p> <hd id="AN0178359442-2">Goals and Structure of the Current Paper</hd> <p>The current manuscript presents a simulation study to investigate whether under- and over-fitting a model can be detected by commonly implemented Bayesian model fit and comparison indices. Specifically, our aim is to examine how well these indices can detect model mis-specification under varying levels of model complexity, sample sizes, and prior specifications. The goal is to provide applied researchers with a guide for when certain indices can properly detect under- or over-fitting a model. We also further explore the ability for (approximate) model fit measures to be used in a model comparison context.</p> <p>The manuscript is structured as follows. In the next section, we describe the role of model fit and comparison measures for detecting specification errors within the frequentist framework. Conventionally, the SEM literature has focused on frequentist estimation methods, and the general knowledge that the field has about the detection of mis-specification is within that context. We then present an introduction to these specification and model assessment issues within the Bayesian framework. We first present notable aspects of the Bayesian estimation of SEMs, and then we describe the current state of knowledge surrounding model assessment and detection of misfits within the estimation framework. Next, we present the details of the model used in the current investigation (a confirmatory factor analysis model), as well as the prior distributions implemented here. This is followed by a brief description of ways to under- and over-fit the model. Then we detail the various Bayesian model fit and comparison indices examined here. Our simulation study examines various model forms, and we have separated the Results into examining the performance (approximate) model fit (Section I) and model comparison (Section II). We conclude with a discussion of the implications of these findings.</p> <hd id="AN0178359442-3">Detecting Model Mis-Specification in SEM Implementing Frequentist Estimation</hd> <p>Model fit assessment is a crucial step in SEM and has traditionally been studied within the frequentist framework (Heene et al., [<reflink idref="bib8" id="ref11">8</reflink>]; Marsh et al., [<reflink idref="bib20" id="ref12">20</reflink>]; Rigdon, [<reflink idref="bib26" id="ref13">26</reflink>]). The chi-square test and several comparative fit indices are commonly employed to evaluate the agreement between the model-implied covariance matrix and the observed covariance matrix (Maydeu-Olivares, [<reflink idref="bib21" id="ref14">21</reflink>]). If the chi-square test yields a <emph>p</emph>-value below a predetermined significance level (e.g., 0.05), it suggests that the observed data significantly deviate from the model, indicating a poor fit. Conversely, if the chi-square test is not significant, it implies that the observed data do not significantly differ from the model, indicating a relatively good fit. However, it is important to note that the chi-square test has limitations. It is sensitive to sample size, meaning that even minor deviations from the model can yield statistically significant results in larger samples (e.g., Shi et al., [<reflink idref="bib29" id="ref15">29</reflink>]). Therefore, it is common practice to consider additional fit indices, such as the comparative fit index (CFI), Tucker-Lewis index (TLI), the root mean square error of approximation (RMSEA), and the standardized root mean square residual (SRMR), in conjunction with the chi-square test to obtain a more comprehensive evaluation of model fit in SEM (Hoyle, [<reflink idref="bib10" id="ref16">10</reflink>]; Rigdon, [<reflink idref="bib26" id="ref17">26</reflink>]).</p> <p>The performance of the fit indices in detecting model mis-specification relies on the properly predetermined thresholds, providing a context for the interpretation of the index value. Mulaik et al. ([<reflink idref="bib23" id="ref18">23</reflink>]) conducted a thorough analysis of fit indices and proposed threshold values for interpretation. They suggested a threshold of 0.90 or higher for incremental fit indices (CFI and TLI), a threshold of 0.06 or lower for RMSEA, and a threshold of 0.08 or lower for SRMR to indicate a good fit. In a subsequent investigation, Hu and Bentler ([<reflink idref="bib12" id="ref19">12</reflink>]) assessed the sensitivity of the fit indices to the under-parameterized model mis-specification, and they suggested a cutoff value close to.95 for TLI and CFI, a cutoff value close to.08 for SRMR, and a cutoff value close to.06 for RMSEA are needed before we can conclude that there is a relatively good fit between the hypothesized model and the observed data. They further proposed a 2-index presentation strategy using the ML-based SRMR and supplemented it with other fit indices such as TLI, CFI, or RMSEA.</p> <p>Model mis-specification in SEM can result in either under-fitting or over-fitting (Liu et al., [<reflink idref="bib17" id="ref20">17</reflink>]). Under-fitting occurs when the model is too simplistic and lacks the necessary parameters to adequately capture the relationships among variables in the data (Hu & Bentler, [<reflink idref="bib12" id="ref21">12</reflink>]). This means that the model does not include certain parameters that could account for the observed correlations or covariation in the data. Under-fitted models are often overly simplistic and fail to adequately represent the underlying relationships among variables. On the other hand, over-fitting in SEM occurs when the model includes parameters that do not truly exist in the population. These additional parameters result in an excessively close fit to the observed data. Over-fitting is challenging to detect in the frequentist framework because the estimated parameters have zero true values in the population. The excessive complexity of the model can make it difficult to identify over-fitting since the model appears to fit the observed data well (Hu & Bentler, [<reflink idref="bib13" id="ref22">13</reflink>]).</p> <p>Model mis-specification in SEM can occur in the measurement part and the structural part of the model, and this has implications for the performance of fit indices. In the measurement part of the model, mis-specification refers to inaccurately representing the relationships between latent constructs and their indicators. This result can happen due to various factors, such as inappropriate indicator selection, failure to consider relevant dimensions of the construct or neglect of correlated residuals. Heene et al. ([<reflink idref="bib8" id="ref23">8</reflink>]) examined the sensitivity of fit indices in SEM when violations of the assumption of uncorrelated errors occur. The study revealed that violations of this assumption had a significant impact on the performance of fit indices.</p> <p>In the structural part of the model, mis-specification includes the wrong covariance or pattern coefficients. Hsu et al. ([<reflink idref="bib11" id="ref24">11</reflink>]) investigated the impact of mis-specification in the structural part of SEMs on parameter estimates and model fit indices in a multilevel SEM. They found that CFI, TLI, and RMSEA were more sensitive to mis-specification in pattern coefficients, while SRMR was more sensitive to mis-specification in the factor covariance. Overall, these studies highlight the importance of considering and addressing model mis-specification in the measurement and structural part of SEMs, as it can significantly influence the performance of fit indices and the interpretation of model fit.</p> <hd id="AN0178359442-4">Shifting to Bayesian Estimation in SEM</hd> <p>The Bayesian framework provides increased flexibility of models that can be estimated within SEM (see e.g., Lee, [<reflink idref="bib15" id="ref25">15</reflink>]; Muthén & Asparouhov, [<reflink idref="bib24" id="ref26">24</reflink>]), as well as improved accuracy and convergence (Depaoli, [<reflink idref="bib5" id="ref27">5</reflink>]). These benefits have helped propel Bayesian estimation as a viable and preferred tool within SEM. With the increased implementation of Bayesian methodology in the applied SEM literature, it is important to have a complete understanding of the model assessment tools available for use.</p> <hd id="AN0178359442-5">Mis-Specification in Bayesian SEM</hd> <p>Model fit evaluation of SEMs estimated through Bayesian methods traditionally relied on posterior predictive <emph>p</emph>-values (PPP-values), which are based on a comparison between the observed data to replicated data generated under the posterior model estimates. As with the frequentist chi-square test of model fit, with large sample sizes, the PPP-value becomes increasingly sensitive to substantively irrelevant mis-specifications (e.g., Asparouhov & Muthén, [<reflink idref="bib1" id="ref28">1</reflink>]; Cain & Zhang, [<reflink idref="bib3" id="ref29">3</reflink>]; Rindskopf, [<reflink idref="bib27" id="ref30">27</reflink>]). Due to this limitation of the PPP-value, researchers have turned to alternative indices, such as information criteria, which are used to compare several competing models. Examples of information criteria are the Bayesian information criterion (BIC; Schwarz, [<reflink idref="bib28" id="ref31">28</reflink>]), Deviance information criterion (DIC; Spiegelhalter et al., [<reflink idref="bib31" id="ref32">31</reflink>]), the widely available information criterion (WAIC; Watanabe, [<reflink idref="bib37" id="ref33">37</reflink>]), and leave-one-out information criterion (LOOIC; Vehtari et al., [<reflink idref="bib34" id="ref34">34</reflink>]). In addition, researchers recently introduced Bayesian versions of frequentist approximate fit indices, such as the RMSEA, CFI, and TLI (Asparouhov & Muthén, [<reflink idref="bib2" id="ref35">2</reflink>]; Garnier-Villarreal & Jorgensen, [<reflink idref="bib7" id="ref36">7</reflink>]; Hoofs et al., [<reflink idref="bib9" id="ref37">9</reflink>]). One helpful aspect of these Bayesian approximate fit indices is that the index estimate is captured by a posterior distribution, and 90% credible intervals (CIs) can be computed. These CIs are useful in capturing the spread of the posterior for the approximate fit index.</p> <p>Previous research on the ability of these indices to detect model mis-specification has focused on detecting model under-fitting. In the context of under-fitting, findings across studies indicate that it may be inappropriate to generalize fixed cutoff values based on frequentist approximate fit indices to their Bayesian counterparts (Edwards & Konold, [<reflink idref="bib6" id="ref38">6</reflink>]; Garnier-Villarreal & Jorgensen, [<reflink idref="bib7" id="ref39">7</reflink>]; Winter & Depaoli, [<reflink idref="bib38" id="ref40">38</reflink>], [<reflink idref="bib39" id="ref41">39</reflink>]). We agree with this assessment because the Bayesian versions were not developed in the context of applying strict cutoff values, and the indices are not directly comparable across the two estimation frameworks. However, we recognize that cutoff values can provide some degree of information if used with flexibility in mind. There are approaches that might be more appropriate for assessing models via these indices that examine results based on CIs as an attempt to capture the bulk of the posterior. Specifically, as sample sizes increase, Bayesian approximate fit indices and their 90% CIs suggest that a model fits better than the population-level fit, resulting in lower model rejection rates of mis-specified models when using traditional frequentist cutoff values (e.g., CFI >.95 Hu & Bentler, [<reflink idref="bib13" id="ref42">13</reflink>]). In light of this finding, Winter & Depaoli, [<reflink idref="bib38" id="ref43">38</reflink>], [<reflink idref="bib39" id="ref44">39</reflink>] noted that the approximate fit indices performed better when used to select the best-fitting model, outperforming the PPP-value, BIC, and DIC with small sample sizes. So far, the use of the Bayesian approximate fit indices for model selection has not received further attention. However, there may be a case for them to be used in a broader context. Specifically, their frequentist counterparts have been recommended as model selection indices, particularly in the context of measurement invariance assessment (Cheung & Rensvold, [<reflink idref="bib4" id="ref45">4</reflink>]). In part, we focus on the ability for the Bayesian indices to be used in that was as well.</p> <p>A second focus of previous research on Bayesian model fit evaluation has been the impact of the priors (e.g., Cain & Zhang, [<reflink idref="bib3" id="ref46">3</reflink>]; Edwards & Konold, [<reflink idref="bib6" id="ref47">6</reflink>];; Liu et al., [<reflink idref="bib17" id="ref48">17</reflink>]; Winter & Depaoli, [<reflink idref="bib38" id="ref49">38</reflink>], [<reflink idref="bib39" id="ref50">39</reflink>]). Two aspects of priors have been examined: their level of informativeness and the extent to which priors diverge from the data likelihood (i.e., prior-data disagreement). Informative priors that align with the data can improve the model selection accuracy of the DIC (Liang & Luo, [<reflink idref="bib16" id="ref51">16</reflink>]; Ward, [<reflink idref="bib36" id="ref52">36</reflink>]). In contrast, the PPP-value and approximate fit indices are less affected by informative, aligned priors (Liang & Luo, [<reflink idref="bib16" id="ref53">16</reflink>]; Winter & Depaoli, [<reflink idref="bib38" id="ref54">38</reflink>], [<reflink idref="bib39" id="ref55">39</reflink>]). However, priors that diverge from the data likelihood can negatively affect model fit and selection indices, particularly when sample sizes are small. Specifically, informative, diverging priors can make a correctly specified model look like a poor-fitting model when evaluated with the PPP-value (Liang & Luo, [<reflink idref="bib16" id="ref56">16</reflink>]) and the approximate fit indices (e.g., Edwards & Konold, [<reflink idref="bib6" id="ref57">6</reflink>]; Winter & Depaoli, [<reflink idref="bib38" id="ref58">38</reflink>], [<reflink idref="bib39" id="ref59">39</reflink>]. Similarly, model selection accuracy of the DIC is negatively affected by increasingly divergent priors (e.g., Cain & Zhang, [<reflink idref="bib3" id="ref60">3</reflink>]). Given that the use of informative priors is often recommended to researchers who work with limited sample sizes, understanding their potentially detrimental impact on model fit evaluation is of the essence.</p> <p>These initial findings highlight the importance of fully understanding when and how Bayesian model fit indices best help researchers assess model fit. However, existing work is limited in that only specific sub-types of model mis-specification have been included. Within studies examining under-fitting, model mis-specification is often examined by comparing a correctly specified model to a limited number (typically one or two) of mis-specified models. For example, Winter & Depaoli, [<reflink idref="bib38" id="ref61">38</reflink>], [<reflink idref="bib39" id="ref62">39</reflink>] compared a three-factor CFA model with two cross-loadings to two models in which one or both of those cross-loadings were omitted. Bayesian fit indices may be differently affected by mis-specification in different parts of the model, such as factor covariances, factor loadings, or residual (co)variances.</p> <p>Questions remain around Bayesian fit indices' ability to detect model over-fitting. As discussed above, including superfluous parameters in a complex model such as an SEM can cause issues with model estimation and parameter estimate efficiency. For that reason, we need to more fully understand whether Bayesian model fit indices are sensitive to model over-fitting. Many information criteria include a term to penalize for over-fitting. Further, the Bayesian RMSEA and TLI both include a similar term based on the difference between the sample moments and estimated parameters (Garnier-Villarreal & Jorgensen, [<reflink idref="bib7" id="ref63">7</reflink>]). Thus, we would expect that these indices are sensitive to model over-fitting. In contrast, the PPP-value and Bayesian CFI do not include a penalty term and may be less sensitive or insensitive to over-fitting.</p> <p>The current study aims to contribute to the existing literature on Bayesian model fit evaluation by investigating whether different types of under- and over-fitting of a model can be detected by commonly implemented Bayesian model fit and comparison indices. We will present a simulation study that examines how well these indices can detect mis-specification under varying levels of model complexity, sample size, and prior specification. Next, the model and priors used in this simulation are detailed.</p> <hd id="AN0178359442-6">The Model and Priors</hd> <p>In the current investigation, we are limiting our examination to the confirmatory factor analysis (CFA) model because it is the most common measurement model used within SEM. In addition, the CFA model lends itself nicely to an inquiry about under- and over-fitting, and contains a measurement and covariance structure. These features make the CFA model a good choice for further inquiry because results are more likely to be generalizable to the broader class of models within SEM.</p> <hd id="AN0178359442-7">Confirmatory Factor Analysis Model</hd> <p>The basic CFA model can be written as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold-italic">x</mi><mo>=</mo><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub><mi mathvariant="bold-italic">ξ</mi><mo>+</mo><mi mathvariant="bold-italic">δ</mi><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib1" id="ref64">1</reflink>)</p> <p>where the <emph>x</emph>'s represent the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>q</mi><mo>=</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mo>...</mo><mo>,</mo><mi>Q</mi></mrow></math> </ephtml> observed items, and these are linked to the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">ξ</mi></math> </ephtml> latent factors through factor loadings contained in</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub></mrow><mo>.</mo></math> </ephtml> Each item contains a random disturbance term</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">δ</mi><mo>.</mo></math> </ephtml> The E(</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold-italic">δ</mi><mo>=</mo><mn>0</mn></mrow></math> </ephtml> ) and disturbances are assumed to be uncorrelated with</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">ξ</mi><mo>.</mo></math> </ephtml> </p> <p>The factor loading matrix,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub></mrow><mo>,</mo></math> </ephtml> contains fixed and freed parameters. For example, the base population model used for this investigation specifies</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub></mrow></math> </ephtml> as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub><mo>=</mo><mrow><mo stretchy="true">[</mo><mrow><mtable><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>1</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>2</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>2</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>3</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>3</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>4</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>4</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>5</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>5</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>6</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>6</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>7</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>7</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mn>1</mn></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>8</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>8</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>9</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>9</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>10</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>10</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>11</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>11</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>12</mn><mo>,</mo><mn>1</mn></mrow></msub><mo>=</mo><mn>0</mn></mrow></mtd><mtd columnalign="left"><mrow><msub><mrow><mi>λ</mi></mrow><mrow><mn>12</mn><mo>,</mo><mn>2</mn></mrow></msub><mo>=</mo><mo>?</mo></mrow></mtd></mtr></mtable></mrow><mo stretchy="true">]</mo></mrow><mo>.</mo></mrow></math> </ephtml> (<reflink idref="bib2" id="ref65">2</reflink>)</p> <p>Specifically, there are eight items and two latent factors (Factor 1 contains Items 1-6; Factor 2 contains Items 7-12). The loading for the first item on each factor is fixed to 1.0 to set the metric of the latent factor. The remaining items linked to that factor are allowed to be freely estimated with a ? placeholder. Finally, items not loading onto a factor have loadings fixed to zero.</p> <p>The observed item covariance matrix can be detailed as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">Σ</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub><msub><mrow><mi mathvariant="bold">Φ</mi></mrow><mi>ξ</mi></msub><msubsup><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi><mo>′</mo></msubsup><mo>+</mo><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib3" id="ref66">3</reflink>)</p> <p>where the covariance matrix of observed items <bold><emph>x</emph></bold> is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="bold">Σ</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi></mrow></math> </ephtml> ), the factor loading matrix is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Λ</mi></mrow><mi>x</mi></msub></mrow><mo>,</mo></math> </ephtml> the latent factor covariance matrix is</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Φ</mi></mrow><mi>ξ</mi></msub></mrow><mo>,</mo></math> </ephtml> and disturbance terms are allowed to covary through</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub></mrow><mo>.</mo></math> </ephtml> </p> <hd id="AN0178359442-8">Model Priors</hd> <p>Each model parameter receives a prior distribution. In this section, we detail priors for all parts of the CFA model as implemented here.[<reflink idref="bib1" id="ref67">1</reflink>]</p> <p>Priors for factor loadings are typically specified as the (semi-)conjugate normal (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="script">N</mi></math> </ephtml> ) distribution written as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>λ</mi></mrow><mi>x</mi></msub><mo>∼</mo><mi mathvariant="script">N</mi><mo stretchy="true">[</mo><msub><mrow><mi>μ</mi></mrow><mrow><msub><mrow><mi>λ</mi></mrow><mi>x</mi></msub></mrow></msub><mo>,</mo><msubsup><mrow><mi>σ</mi></mrow><mrow><msub><mrow><mi>λ</mi></mrow><mi>x</mi></msub></mrow><mn>2</mn></msubsup><mo stretchy="true">]</mo><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib4" id="ref68">4</reflink>)</p> <p>with this distribution defined through a mean</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>μ</mi></mrow><mrow><msub><mrow><mi>λ</mi></mrow><mi>x</mi></msub></mrow></msub></mrow></math> </ephtml> (controlling the location of the prior) and a variance hyperparameter</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>σ</mi></mrow><mrow><msub><mrow><mi>λ</mi></mrow><mi>x</mi></msub></mrow><mn>2</mn></msubsup></mrow></math> </ephtml> (controlling the spread, or degree of informativeness).</p> <p>The next prior is for the disturbance variances (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>σ</mi></mrow><mi>δ</mi><mn>2</mn></msubsup></mrow></math> </ephtml> ). The variances of the disturbance terms can be represented in a couple of different ways. If disturbance terms are allowed to covary, then a multivariate prior placed directly onto the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub></mrow></math> </ephtml> covariance matrix is appropriate. In this case, an inverse Wishart (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">W</mi></mrow></math> </ephtml> ) prior can be used, which allows for variances (diagonal elements) and covariances (off-diagonal elements):</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub><mo>∼</mo><mi mathvariant="script">I</mi><mi mathvariant="script">W</mi><mo stretchy="true">[</mo><mi mathvariant="bold">Ψ</mi><mo>,</mo><mi>ν</mi><mo stretchy="true">]</mo><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib5" id="ref69">5</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold">Ψ</mi></math> </ephtml> is a positive definite matrix of size <emph>p</emph>, and <emph>ν</emph> is the degrees of freedom, which controls the informativeness of the prior. If individual disturbances are assumed independent (i.e., non-correlated), then priors can be placed on the individual disturbance variances as opposed to the entire covariance matrix. An individual (diagonal) element within</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub></mrow></math> </ephtml> can be denoted as</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub></mrow><mo>,</mo></math> </ephtml> where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>σ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow><mn>2</mn></msubsup><mo>=</mo><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub></mrow></math> </ephtml> for diagonal elements in the <emph>Q</emph> ×<emph> Q</emph> matrix</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Θ</mi></mrow><mi>δ</mi></msub></mrow><mo>.</mo></math> </ephtml> For residual variances</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub></mrow><mo>,</mo></math> </ephtml> the inverse gamma (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">G</mi></mrow></math> </ephtml> ) prior can be used as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub><mo>∼</mo><mi mathvariant="script">I</mi><mi mathvariant="script">W</mi><mo stretchy="true">[</mo><msub><mrow><mi>a</mi></mrow><mrow><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub></mrow></msub><mo>,</mo><msub><mrow><mi>b</mi></mrow><mrow><msub><mrow><mi>θ</mi></mrow><mrow><msub><mrow><mi>δ</mi></mrow><mrow><mi>q</mi><mi>q</mi></mrow></msub></mrow></msub></mrow></msub><mo stretchy="true">]</mo><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib6" id="ref70">6</reflink>)</p> <p>where <emph>a</emph> and <emph>b</emph> are the shape and scale hyperparameters, respectively. Adjusting these hyperparameters will alter the level of informativeness of the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">G</mi></mrow></math> </ephtml> prior varies.</p> <p>Finally, the latent factor covariance matrix</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Φ</mi></mrow><mi>ξ</mi></msub></mrow></math> </ephtml> can receive an</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">W</mi></mrow></math> </ephtml> prior as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Φ</mi></mrow><mi>ξ</mi></msub><mo>∼</mo><mi>I</mi><mi>W</mi><mo stretchy="true">[</mo><mi mathvariant="bold">Ψ</mi><mo>,</mo><mi>ν</mi><mo stretchy="true">]</mo><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib7" id="ref71">7</reflink>)</p> <p>with the same definitions of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold">Ψ</mi></math> </ephtml> and <emph>ν</emph> from Equation (<reflink idref="bib5" id="ref72">5</reflink>).</p> <hd id="AN0178359442-9">Ways to Mis-Specify the CFA Model</hd> <p>Model mis-specification can be conceptualized as formulating an analysis model that is not consistent with the population model. Of course, applied SEM research is inherently "mis-specified" because models are a simplification of complex phenomena (variable relationships, etc.) being tapped into through data analysis. It would be reasonable to expect an applied research model to contain some degree of specification error due to the complexity of the true variable relationships. It is important to have an understanding of the impact of the discrepancy between a model and the "truth," as well as the ability to detect it.</p> <p>As described earlier, even if all variables have been appropriately included in a model, there are two main ways of mis-specifying that model. First, the model can be too simple, as compared to the population model. In this case, the model either fixes or does not contain a model parameter that exists in the population model. That case represents under-fitting, where model parameters in the population model have not been included in the analysis model. The second form of specification error is the case where the analysis model was specified to be more complex, and contain more model parameters, than the population model. That form of model mis-specification is considered over-fitting, where "extra" model parameters are included. Within SEMs, these two forms of model mis-specification can occur in the measurement part of the model, the structural part of the model, or both parts.</p> <p>Regarding the measurement part of the model, under-fitting could be ignoring a cross-loading that is truly non-zero, and over-fitting could be adding a loading that is truly zero. For the structural part of the model, under-fitting could be ignoring factor covariances that are non-zero, and over-fitting could be modeling residual covariances that are truly zero. There are a variety of ways to define these specification errors within each part of the model, and we explore some of the more common forms of specification error in the current investigation.</p> <hd id="AN0178359442-10">Relevant Bayesian Model Fit and Assessment Measures</hd> <p>In the current simulation, we investigate the ability of several common Bayesian model fit and comparison indices to detect model mis-specification. To assess model (mis)fit for a single model, we examined the PPP, as well as the Bayesian versions of the RMSEA, CFI, and TLI. Regarding model selection, where competing models are compared with one another, we focused on the BIC, DIC, and the Bayesian approximate fit measures (RMSEA, CFI, and TLI). These fit indices are available in accessible Bayesian SEM software such as M<emph>plus</emph> (Muthén & Muthén, [<reflink idref="bib25" id="ref73">25</reflink>]) and blavaan (Merkle & Rosseel, [<reflink idref="bib22" id="ref74">22</reflink>]).<sups>2</sups>[<reflink idref="bib2" id="ref75">2</reflink>]</p> <hd id="AN0178359442-11">Posterior Predictive p-Value</hd> <p>The posterior predictive model checking procedure has been widely used within the Bayesian estimation framework. Its formulation requires more space than we have allotted here, but we will provide a brief description to give context for this procedure. To start, each iteration of the Markov chain is used to produce a simulated dataset of the same size of the observed dataset. After many samples are derived, a discrepancy function (typically one based on the likelihood ratio test) is computed for the replicated and observed data. The discrepancy function for the observed data is expressed as:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>D</mi></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup><mo>,</mo><msub><mrow><mi>μ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup><mo stretchy="false">)</mo><mo>,</mo><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup><mo stretchy="false">)</mo><mo>,</mo><msub><mrow><mi>μ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub><mo>,</mo><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">)</mo><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>μ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup><mo stretchy="false">)</mo></mrow></math> </ephtml> refer to a random draw of the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="script">M</mi></mrow><mn>1</mn></msub></mrow></math> </ephtml> parameter estimates for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>obs</mtext></mrow></msup></mrow></math> </ephtml> at iteration <emph>s</emph>; and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>μ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub></mrow></math> </ephtml> refer to the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="script">M</mi></mrow><mn>0</mn></msub></mrow></math> </ephtml> -implied mean and covariance matrix obtained from the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="script">M</mi></mrow><mn>0</mn></msub></mrow></math> </ephtml> at iteration <emph>s</emph>. Likewise, the discrepancy function for the replicated data can be written as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>rep</mtext></mrow></msubsup><mo>=</mo><msubsup><mrow><mi>D</mi></mrow><mi>s</mi><mrow><mtext>rep</mtext></mrow></msubsup><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup><mo>,</mo><msub><mrow><mi>μ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup><mo stretchy="false">)</mo><mo>,</mo><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup><mo stretchy="false">)</mo><mo>,</mo><msub><mrow><mi>μ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub><mo>,</mo><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>0</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">)</mo><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>μ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup><mo stretchy="false">)</mo></mrow></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="bold">Σ</mi></mrow><mrow><mn>1</mn><mo>,</mo><mi>s</mi></mrow></msub><mo stretchy="false">(</mo><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup><mo stretchy="false">)</mo></mrow></math> </ephtml> indicate a random draw of the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi mathvariant="script">M</mi></mrow><mn>1</mn></msub></mrow></math> </ephtml> parameter estimates for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi mathvariant="bold-italic">y</mi></mrow><mrow><mtext>rep</mtext></mrow></msup></mrow></math> </ephtml> at iteration <emph>s</emph>. These discrepancy functions are used to compute the posterior predictive <emph>p</emph>-value (PPP) as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtext>PPP-value</mtext><mo>=</mo><mi>p</mi><mo stretchy="false">(</mo><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>rep</mtext></mrow></msubsup><mo>></mo><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup><mo>|</mo><mi mathvariant="bold-italic">y</mi><mo stretchy="false">)</mo><mo>≈</mo><mfrac><mn>1</mn><mi>S</mi></mfrac><munderover><mo>∑</mo><mrow><mi>s</mi><mo>=</mo><mn>1</mn></mrow><mi>S</mi></munderover><mrow><msub><mrow><mi>δ</mi></mrow><mi>s</mi></msub></mrow><mo>,</mo></mrow></math> </ephtml> (<reflink idref="bib8" id="ref76">8</reflink>)</p> <p>where there are S iterations in the chain, and <emph>δ<subs>s</subs></emph> is defined based on:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>δ</mi></mrow><mi>s</mi></msub><mo>=</mo><mrow><mo stretchy="true">{</mo><mrow><mtable><mtr columnalign="left"><mtd columnalign="left"><mn>1</mn></mtd><mtd columnalign="left"><mrow><mtext>if</mtext><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>rep</mtext></mrow></msubsup><mo>></mo><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup><mo>,</mo></mrow></mtd></mtr><mtr columnalign="left"><mtd columnalign="left"><mn>0</mn></mtd><mtd columnalign="left"><mrow><mtext>otherwise</mtext><mo>.</mo></mrow></mtd></mtr></mtable></mrow></mrow></mrow></math> </ephtml> </p> <p>The PPP is interpreted such that values of.5 indicate good model fit (i.e., 50% of the replicated datasets have discrepancy statistics greater than those linked to the observed data), and a PPP close to 0 indicates model mis-specification.</p> <hd id="AN0178359442-12">Bayesian Approximate Fit Indices</hd> <p>The Bayesian RMSEA is calculated as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mrow><mtext>BRMSEA</mtext></mrow></mrow><mi>s</mi></msub><mo>=</mo><msqrt><mrow><mi mathvariant="normal">max</mi><mo stretchy="true">(</mo><mn>0</mn><mo>,</mo><mfrac><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup></mrow><mrow><mo stretchy="false">(</mo><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mi>D</mi></msub><mo stretchy="false">)</mo><mi>n</mi></mrow></mfrac><mo stretchy="true">)</mo></mrow></msqrt><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mi>s</mi><mrow><mtext>obs</mtext></mrow></msubsup></mrow></math> </ephtml> is the discrepancy function for the observed data at iteration <emph>s</emph>, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup></mrow></math> </ephtml> is the number of nonredundant sample moments.</p> <p>The Bayesian CFI is computed as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mrow><mtext>BCFI</mtext></mrow></mrow><mi>s</mi></msub><mo>=</mo><mn>1</mn><mo>−</mo><mfrac><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>T</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup></mrow><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>B</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup></mrow></mfrac><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>T</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup></mrow></math> </ephtml> is the target model discrepancy function (for the observed data) at iteration <emph>s</emph>, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>B</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup></mrow></math> </ephtml> is linked to the baseline model discrepancy function.</p> <p>The Bayesian TLI is computed as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mrow><mtext>BTLI</mtext></mrow></mrow><mi>s</mi></msub><mo>=</mo><mfrac><mrow><mfrac><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>B</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>B</mi></msub></mrow></msub></mrow><mrow><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>B</mi></msub></mrow></msub></mrow></mfrac><mo>−</mo><mfrac><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>T</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>T</mi></msub></mrow></msub></mrow><mrow><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>T</mi></msub></mrow></msub></mrow></mfrac></mrow><mrow><mfrac><mrow><mi>D</mi><msubsup><mrow><mrow><mo stretchy="false">(</mo><mo>·</mo><mo stretchy="false">)</mo></mrow></mrow><mrow><mi>B</mi><mo>,</mo><mi>s</mi></mrow><mrow><mtext>obs</mtext></mrow></msubsup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>B</mi></msub></mrow></msub></mrow><mrow><msup><mrow><mi>p</mi></mrow><mo>*</mo></msup><mo>−</mo><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>B</mi></msub></mrow></msub></mrow></mfrac><mo>−</mo><mn>1</mn></mrow></mfrac><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>B</mi></msub></mrow></msub></mrow></math> </ephtml> is the model complexity term for the baseline model, and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>p</mi></mrow><mrow><msub><mrow><mi>D</mi></mrow><mi>T</mi></msub></mrow></msub></mrow></math> </ephtml> is the model complexity term for the target model.</p> <hd id="AN0178359442-13">Bayesian Information Criterion</hd> <p>With</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi></math> </ephtml> denoting a vector of parameters, and <bold><emph>y</emph></bold> representing a vector of data, the BIC is formulated as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtext>BIC</mtext><mo>=</mo><mo>−</mo><mn>2</mn><mo /><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>ML</mtext></mrow></msub><mo stretchy="false">)</mo><mo>}</mo><mo>+</mo><mi>k</mi><mo /><mi mathvariant="normal">log</mi><mo /><mi>n</mi><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>ML</mtext></mrow></msub></mrow></math> </ephtml> is the maximum likelihood estimate of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi><mo>;</mo></math> </ephtml> <emph>k</emph> is the number of parameters in the model; <emph>n</emph> is the sample size;</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo /><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>ML</mtext></mrow></msub><mo stretchy="false">)</mo><mo>}</mo></mrow></math> </ephtml> is the log likelihood based on the maximum likelihood estimates of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi><mo>;</mo></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>k</mi><mo /><mi mathvariant="normal">log</mi><mo /><mi>n</mi></mrow></math> </ephtml> is the model complexity term.</p> <hd id="AN0178359442-14">Deviance Information Criterion</hd> <p>The DIC is defined as:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mtext>DIC</mtext><mo>=</mo><mo>−</mo><mn>2</mn><mo /><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>EAP</mtext></mrow></msub><mo stretchy="false">)</mo><mo>}</mo><mo>+</mo><mn>2</mn><msub><mrow><mi>p</mi></mrow><mi>D</mi></msub><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>EAP</mtext></mrow></msub></mrow></math> </ephtml> is the posterior mean of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi><mo>,</mo></math> </ephtml> used to define the term</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo /><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>EAP</mtext></mrow></msub><mo stretchy="false">)</mo><mo>}</mo></mrow><mo>,</mo></math> </ephtml> and <emph>p<subs>D</subs></emph> is the model complexity term that is computed as:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>p</mi></mrow><mi>D</mi></msub><mo>=</mo><mn>2</mn><mo stretchy="false">(</mo><mo /><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><msub><mrow><mrow><mrow><mrow><mover accent="true"><mi mathvariant="bold-italic">θ</mi><mo stretchy="true">̂</mo></mover></mrow></mrow></mrow></mrow><mrow><mtext>EAP</mtext></mrow></msub><mo stretchy="false">)</mo><mo>}</mo><mo stretchy="false">)</mo><mo>−</mo><msub><mrow><mi>E</mi></mrow><mrow><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo>|</mo><mi mathvariant="bold-italic">y</mi><mo stretchy="false">)</mo></mrow></msub><mo stretchy="true">[</mo><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mo>}</mo><mo stretchy="true">]</mo><mo>,</mo></mrow></math> </ephtml> </p> <p>where</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>E</mi></mrow><mrow><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">θ</mi><mo>|</mo><mi mathvariant="bold-italic">y</mi><mo stretchy="false">)</mo></mrow></msub><mo stretchy="false">[</mo><mi mathvariant="normal">log</mi><mo /><mo>{</mo><mi>p</mi><mo stretchy="false">(</mo><mi mathvariant="bold-italic">y</mi><mo>|</mo><mi mathvariant="bold-italic">θ</mi><mo stretchy="false">)</mo><mo>}</mo><mo stretchy="false">]</mo></mrow></math> </ephtml> is the average log likelihood over the posterior distribution of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="bold-italic">θ</mi><mo>.</mo></math> </ephtml> </p> <p>The following section details the simulation design, and then the results are presented in terms of (I) model fit and (II) model comparison.</p> <hd id="AN0178359442-15">Simulation Design</hd> <p>The current simulation study examines the ability of model fit and selection to detect model mis-specification in a variety of modeling settings. For this investigation, we implemented a CFA model, largely because it is the most common measurement model within SEM-based research, making the results generalizable to a large body of models implemented. We include the exact specifications of the CFA model configurations in the subsections below. The design is structured such that model mis-specification is defined in terms of under-fitting and over-fitting. We will detail the design details specific to these two types of model mis-specification below.</p> <p>In all simulation cells included here, we implemented the Bayesian estimation framework via M<emph>plus</emph> version 8.10 using the default Gibbs sampler to sample from the posterior. We requested 1,000 replications within each cell of the simulation design. For each replication, we specified two Markov chains, each with 50,000 samples. The first half of the chains were discarded as the burn-in phase, and the remaining 25,000 iterations comprised the posterior estimate for each chain.</p> <hd id="AN0178359442-16">Under-Fitting</hd> <p></p> <hd id="AN0178359442-17">Population model</hd> <p>In this part of the simulation, we were interested in the influence of parameter omission on detecting model mis-specification in the Bayesian framework. We specified a single population model, which can be seen in Figure 1, panel (a). Specifically, the population model contains two latent factors, each with primary factor loadings for six continuous items (Factor 1 primary items: Items 1-6; Factor 2 primary items: Items 7–12). The two latent factors were correlated in the population model, and there was a non-zero cross-loading from Item 4 to Factor 2. Finally, Items 1 and 2 had correlated residual terms, whereas all other item residuals were left uncorrelated. All population values are in Figure 1.</p> <p>Graph: Figure 1. Population models for under-fitted conditions.</p> <hd id="AN0178359442-18">Design Factors and Priors</hd> <p>This under-fitted portion of the simulation study has design factors of sample size (5 levels) and model specification (7 levels). Prior distributions were held consistent across these conditions and were specified as the M<emph>plus</emph> default diffuse settings. Specifically, loadings received priors of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><msup><mrow><mrow><mn>10</mn></mrow></mrow><mrow><mn>10</mn></mrow></msup><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> residual variances received priors of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">G</mi><mo stretchy="false">(</mo><mo>−</mo><mn>1</mn><mo>,</mo><mn>0</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> and latent factor covariances received priors of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">I</mi><mi mathvariant="script">W</mi><mo stretchy="false">(</mo><mn>0</mn><mo>,</mo><mo>−</mo><mi>p</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><mo>,</mo></math> </ephtml> where <emph>p</emph> is the number of latent factors specified in the model.</p> <hd id="AN0178359442-19">Sample Size</hd> <p>In order to examine sample sizes representing a common range found in the applied literature, we specified five levels:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>=</mo></mrow></math> </ephtml> 50, 100, 200, 500, and 1,000.</p> <hd id="AN0178359442-20">Model Specification</hd> <p>As noted, the population model can be found in Figure 1 (top panel), and for these simulation conditions, we call this "Model 1 u" (i.e., the correct model specification for the under-specified models). In addition to this properly specified model, we have constructed six versions of under-fitted specification errors. These six analysis models are respectively called Model 2 u - Model 7 u, and they each contain a parameter omission either in the measurement or structural portions of the model.</p> <p>First, we describe the measurement model mis-specifications, see Figure 2(f) for all under-fitted conditions; note that the numbering of the models is not chronological here, but it is consistent with the presentation of the results, which showcases the different levels of severity across the specification errors. Model 4 u ignores the presence of the cross-loading leading from Factor 2 to Item 4. Model 6 u represents a measurement model mis-specification where a primary loading is removed. Specifically, the primary loading of Factor 1 regressed onto Item 4 is fixed to zero in this analysis model. This leaves Item 4 only loading onto Factor 2 (which is classified as a cross-loading in the population model). Model 7 u mis-specifies the measurement model by ignoring the presence of Factor 2 altogether. In this model, the factor structure is collapsed, and the items are all defined through Factor 1.</p> <p>Graph: Figure 2. Under-fitted mis-specification conditions.</p> <p>Next, we describe the structural model mis-specifications. Model 2 u removes the factor correlation, which is non-zero in the population model. Model 3 u ignores the correlation between the residuals for Items 1 and 2. Model 5 u constrains the residuals to be equal for the items within each factor. Specifically, residuals for Factor 1's Items 1–6 are constrained to be equal, and residuals for Factor 2's Items 7–12 are constrained equal to each other.</p> <hd id="AN0178359442-21">Over-Fitting</hd> <p></p> <hd id="AN0178359442-22">Population model</hd> <p>In this part of the investigation, the population model was defined as follows (see Figure 3 for a visual depiction). The CFA contains two uncorrelated factors, with six items per factor and no cross-loadings.</p> <p>Graph: Figure 3. Population model for over-fitted conditions.</p> <hd id="AN0178359442-23">Design Factors and Priors</hd> <p>This over-fitted portion of the study has design factors of sample size (5 levels) and model specification (5 levels). For the majority of the cells, prior distributions were held consistent across these conditions and were specified as the M<emph>plus</emph> default diffuse settings as detailed above. However, there was one alteration of the prior settings for Model 4o, which will be described below.</p> <hd id="AN0178359442-24">Sample Size</hd> <p>The same five sample size conditions of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>=</mo></mrow></math> </ephtml> 50, 100, 200, 500, and 1,000 were implemented in this portion of the simulation study.</p> <hd id="AN0178359442-25">Model Specification</hd> <p>The population model for the over-fitted portion of the design can be found in Figure 3, and for these simulation conditions, we call this "Model 1o" (i.e., the correct model specification for the over-specified models). There are four mis-specified analysis models examined here, see Figure 4(d). These analysis models are respectively called Model 2o - Model 5o, and they each contain a mis-specification by over-specifying the model (i.e., adding a free parameter in for the analysis model that was not in the population model). Model 2o adds a factor covariance to the analysis model. Model 3o adds a residual correlation parameter for Items 1 and 2. Model 4o adds a cross-loading that is not present in the population such that Item 4 is now also loading onto Factor 2. Finally, Model 5o is over-specified by adding an additional latent factor (so three factors are included here). Specifically, Items 10-12 form Factor 3, and all factors are uncorrelated.</p> <p>Graph: Figure 4. Over-fitted mis-specification conditions.</p> <hd id="AN0178359442-26">Convergence</hd> <p>Within any simulation study conducted in the Bayesian estimation framework, there are two forms of convergence that must be monitored. The first type of convergence to monitor is the convergence of the simulation replications. Within this simulation design, we requested 1000 replications within each cell. According to M<emph>plus</emph>, all 1,000 replications converged for all cells in the study.</p> <p>The second form of convergence that was examined was convergence for the individual Markov chains that are used to sample from the posterior of each model parameter. Chain convergence was monitored via the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><mover accent="true"><mi>R</mi><mo>̂</mo></mover></mrow></mrow><mo>,</mo></math> </ephtml> or potential scale reduction factor (Vehtari et al., [<reflink idref="bib35" id="ref77">35</reflink>]), as implemented in M<emph>plus</emph>. The default M<emph>plus</emph> setting of 1.05 was implemented for this index, where values exceeding this threshold represented chains that had not converged for a given model parameter. Each parameter was estimated using two Markov chains, and the</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><mover accent="true"><mi>R</mi><mo>̂</mo></mover></mrow></mrow></math> </ephtml> convergence criterion indicated that there was no sign of non-convergence in the post-burn-in portions of the chains for any of the model parameters. In other words, each chain for each model parameter, across all replications and cells of the design, produced</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><mover accent="true"><mi>R</mi><mo>̂</mo></mover></mrow></mrow></math> </ephtml> within in the acceptable range of 1.0-1.05. Given these convergence findings, the results presented below represent stable and converged estimates and can be fully narrated and interpreted.</p> <p>In the following Results sections, we highlight performance of (approximate) fit measures to detect misfit (Simulation Results I), which is followed by an assessment of model selection indices (Simulation Results II). Within each of these sections, under- and over-fit will be examined.</p> <hd id="AN0178359442-27">Simulation Results I: Model Fit Assessment Indices</hd> <p>In this section, we will assess how well the (approximate) model fit measures were able to detect model mis-specification. Under- and over-fit were assessed separately, as to gain a clearer understanding of index performance in each situation of misfit. We conducted ANOVAs to determine to what extent the model fit indices were affected by the simulation conditions using partial eta-squared (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>η</mi></mrow><mi>p</mi><mn>2</mn></msubsup></mrow></math> </ephtml> ) to measure effect size (Table 1).</p> <p>Table 1 Partial eta-squared results of simulation conditions on fit index values.</p> <p> <ephtml> <table><thead><tr><td /><td>PPP</td><td>RMSEA</td><td>CFI</td><td>TLI</td></tr></thead><tbody valign="top"><tr><td>Under-Fitting</td><td /><td /><td /><td /></tr><tr><td> Specification (S)</td><td char=".">0.690</td><td char=".">0.871</td><td char=".">0.867</td><td char=".">0.873</td></tr><tr><td> Sample size (N)</td><td char=".">0.104</td><td char=".">0.194</td><td char=".">0.177</td><td char=".">0.151</td></tr><tr><td> S × N</td><td char=".">0.131</td><td char=".">0.101</td><td char=".">0.003</td><td char=".">0.008</td></tr><tr><td>Under-Fitting: Secondary</td><td /><td /><td /><td /></tr><tr><td> Specification (S)</td><td char=".">0.638</td><td char=".">0.683</td><td char=".">0.520</td><td char=".">0.525</td></tr><tr><td> Sample size (N)</td><td char=".">0.083</td><td char=".">0.310</td><td char=".">0.332</td><td char=".">0.313</td></tr><tr><td> S × N</td><td char=".">0.096</td><td char=".">0.105</td><td char=".">0.001</td><td char=".">0.003</td></tr><tr><td>Over-Fitting</td><td /><td /><td /><td /></tr><tr><td> Specification (S)</td><td char=".">0.479</td><td char=".">0.849</td><td char=".">0.925</td><td char=".">0.926</td></tr><tr><td> Sample size (N)</td><td char=".">0.005</td><td char=".">0.413</td><td char=".">0.372</td><td char=".">0.357</td></tr><tr><td> S × N</td><td char=".">0.002</td><td char=".">0.091</td><td char=".">0.006</td><td char=".">0.015</td></tr><tr><td>Prior on Cross-Loading (Over-Fitting)</td><td /><td /><td /><td /></tr><tr><td> Specification (S)</td><td char=".">0.064</td><td char=".">0.072</td><td char=".">0.053</td><td char=".">0.055</td></tr><tr><td> Sample size (N)</td><td char=".">0.003</td><td char=".">0.544</td><td char=".">0.534</td><td char=".">0.530</td></tr><tr><td> Prior (P)</td><td char=".">0.382</td><td char=".">0.402</td><td char=".">0.336</td><td char=".">0.336</td></tr><tr><td> N × S</td><td char=".">0.001</td><td char=".">0.006</td><td char=".">0.019</td><td char=".">0.019</td></tr><tr><td> N × P</td><td char=".">0.045</td><td char=".">0.051</td><td char=".">0.112</td><td char=".">0.107</td></tr></tbody></table> </ephtml> </p> <hd id="AN0178359442-28">Under-Fitting</hd> <p>ANOVA results (Table 1) showed that the distributions of all fit index values were primarily affected by the model specification, with a smaller effect of sample size. In addition, for the PPP and RMSEA values, there was a small interaction effect of model specification and sample size. In Figure 5, we include box plots of Bayesian RMSEA, CFI, TLI, and PPP under the true model, as well as the models with various types of under-fit. These box plots represent the empirical distributions for each index, which provides information about the distributional results for the index estimates across the simulation replications.</p> <p>Graph: Figure 5. Box plots of the empirical distribution for the PPP, Bayesian RMSEA, CFI, and TLI for the true model (model 1) and the models with under-fit (models 2-7). Note that dashed lines should not be viewed as strict cutoff values for the index. Instead, they should be viewed as a way to help visually interpret results.</p> <p>The figure is structured as follows. The columns represent the five different sample size conditions, with the smallest on the left side. The rows represent the different indices as follows: PPP, Bayesian RMSEA, Bayesian CFI, and Bayesian TLI on the bottom. The <emph>y</emph>-axis for each subplot is the index estimate value. Within the boxes, the <emph>x</emph>-axis represents the model specification condition. There are seven models represented by boxes within each panel. Model 1 u is the correctly specified model, and the other six models represent an under-fit mis-specification.</p> <p>In each of the subplots, a horizontal dotted line is embedded, which represents the conventional cutoff value that is used for interpreting index results. We note here that we are not advocating for a strict cutoff value to be implemented with these indices. Indeed, we believe that strict cutoff values can be deceiving and inappropriately link the Bayesian and frequentist versions of the indices as being comparable, when they reflect distinct concepts. However, adding this reference point can be helpful when describing patterns of the simulation results. We used a value of 0.05 for the PPP, where values closer to 0 represent poor fit. For Bayesian CFI and TLI, we placed a reference point at 0.95, where values exceeding this cutoff are typically considered a good fit. Finally, for the Bayesian RMSEA, we use a reference point of 0.06, where values lower than this point are typically considered a good fit.</p> <p>The PPP results show a clear pattern in the top row. The correct model is always pinpointed as well-fitting, with PPP values centered at about 0.5. All mis-specified models are detected as such for the higher sample sizes, especially for <emph>n</emph> = 500 and 1,000. The smallest sample size (left column) showed more variability in results. Specifically, the PPP did not detect misfit regarding many of the mis-specification conditions in a majority of the replications.</p> <p>The Bayesian RMSEA results show that the correct model is consistently meeting the conventional threshold for good fit. The factor covariance model (M 2 u) is at the threshold but shifted toward misfit, and all other empirical distributions show clear misfit according to the Bayesian RMSEA. The sample size effect is that empirical distributions have less variability as sample size increases, but the same basic pattern exists across the five sample size conditions.</p> <p>The two lower rows appear very similar, indicating that results were comparable across the Bayesian CFI and TLI. Looking across the sample size panels, the main patterns are comparable, with the difference across sample size being that there was more spread in the empirical distributions for <emph>n</emph> = 50 and 100, as opposed to the larger sample sizes. The correct model shows the empirical distribution for all sample sizes met the conventional cutoff value, indicating good fit, for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>></mo></mrow></math> </ephtml> 50. In addition, the factor covariance model (M 2 u) largely met the cutoff across sample size conditions, indicating that misfit was not detected here according to these indices. Two other models had empirical distributions covering the threshold of 0.95, indicating inconclusive results according to the cutoff. These were the the residual correlation model (M 3 u) and the cross loading model (M 4 u). The models clearly identified as misfit were the equal residuals model (M 5 u), the primary loading model (M 6 u), and the one factor model (M 7 u)–all three of these represented the most egregious specification errors for the under-fit conditions, and these two indices were able to detect that.</p> <p>Figure 6 shows the "conclusiveness of results" for the Bayesian RMSEA, CFI, and TLI based on the 90% CI for the indices. For example, a conventional threshold of 0.06 is used for the RMSEA. If the CI is contained in the region 0 < CI < 0.06, then this would be conclusive evidence for a good fit. If the CI is beyond the 0.06 cutoff (CI > 0.06), then this is conclusive evidence of poor fit. If the CI contains the 0.06 cutoff, then this is inconclusive evidence. The CIs for TLI and CFI were similarly interpreted here, except using a 0.95 cutoff. We want to reiterate that these are not strict interpretation rules. Instead, we use the CIs as another piece of evidence to detail index performance in detecting misfit. Results could be interpreted in a more flexible manner if the cutoff value was shifted.</p> <p>Graph: Figure 6. Classification of the fit of the model based on the Bayesian RMSEA, CFI, and TLI for models with under-fit.</p> <p>Figure 6 is structured such that the <emph>y</emph>-axis represents the proportion of simulation replications falling within each of the three following categories: conclusive good fit (lightest grey), inconclusive results (medium grey), and conclusive poor fit (black). The different model conditions are listed along the <emph>x</emph>-axis, the columns represent sample size, and the rows are the indices.</p> <p>Figure 7 contains rejection rates for the (approximate) fit indices in under-fitted modeling situations across all sample sizes. The correct model and six under-fitted models are presented. The <emph>y</emph>-axis represents the proportion of models out of 1,000 simulation replications that were rejected based on the conventional cutoff for each of the (approximate) fit indices represented in the figure legend. Results in Figure 7 show that the correct model was deemed to fit at a high rate for all indices; the largest rejection rate (indicting misfit) for the correct model was for the Bayesian RMSEA for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>≥</mo></mrow></math> </ephtml> 100, but all other results indicated good fit. The under-fitted models showed varied results, but these results appear relatively consistent across sample size conditions. Overall, the under-fitted factor covariance model (M 2 u) had the lowest rejection rates out of the mis-specified models. Consistently at the highest rejection rates were the models with mis-specifications related to fixing residuals equal in the analysis model (M 5 u), a primary factor loading (M 6 u), and reducing the number of factors to one (M 7 u). These three models represented under-fit mis-specifications that the indices were most accurate at detecting.</p> <p>Graph: Figure 7. Rejection rates of models using the PPP, the Bayesian RMSEA, CFI, and TLI for the true model and the under-fitted model.</p> <p>Overall, it is visually apparent that the Bayesian RMSEA has more conclusive poor fit for the mis-specified models (the black bars are consistently higher for Models 2-7 for this index). All three indices did well to detect the correct model, but there were higher rates of inconclusive CI results for the lowest sample size; the correct model was conclusively deemed good fit under the largest sample sizes of <emph>n</emph> = 500 and 1,000. Across the indices, the Bayesian CFI and TLI had a harder time detecting the factor covariance mis-specification (M 5 u) as compared to the RMSEA. Overall, as sample size increased, the ability for the CI to conclusively detect misfit improved (with the exception of "M 2 u" as stated).</p> <hd id="AN0178359442-29">Under-Fitting: A Subsequent Simulation Inquiry</hd> <p>As a secondary simulation design, we wanted to explore the potential impact that model complexity would have on the notion of under-fitting. There are a variety of ways that model complexity can be defined, but we selected the most applicable definition as it relates to the CFA model. Specifically, data that are used to estimate CFAs often (in practice) do not contain clean, simple structures where all cross-loadings are zero. That is why we initially designed the simulation to include a non-zero cross-loading. However, after examining the results, we determined that a secondary inquiry worth examining was the case where additional cross-loadings are present. This situation is one that mimics many applied data situations, namely, where the loading patterns are more complicated than the basic, restricted version of the CFA.</p> <p>In this secondary simulation design, we extended the modeling situation for Model 1 u to include a second cross-loading, as seen in panel (b) of Figure 1. This additional cross-loading represents X5 on Factor 2, and the loading strength is the same as the first cross-loading. The same five sample sizes were examined, and three models were estimated: a correct model, a model omitting the cross-loading from X5 on Factor 2 but correctly estimating the other cross-loading, and a model omitting both cross-loadings. Contrary to the ANOVA results discussed above (Table 1), sample size had a more substantial effect on the RMSEA, CFI, and TLI distributions. The boxplots for each of the indices (rows) are presented in Figure 8. The distributions of RMSEA, CFI, and TLI values narrowed across all three model specifications as the sample size increased. Further, an interesting pattern emerged that was consistent across sample sizes and indices. Specifically, the drop in fit from the correct model to the model with one cross-loading omitted was noticeable and severe. However, there was virtually no distinction in fit between the model with one omitted cross-loading and the model that omitted both cross-loadings.</p> <p>Graph: Figure 8. Box plots of the empirical distribution for the Bayesian RMSEA, CFI, TLI, and PPP for the secondary simulation study examining two cross-loadings; the correct model is compared to a model with one cross-loading ignored and a model with both cross-loadings ignored. Note that dashed lines should not be viewed as strict cutoff values for the index. Instead, they should be viewed as a way to help visually interpret results.</p> <p>Figure 9 contains information about the approximate fit credible intervals. The Bayesian RMSEA detected the omitted cross-loadings as conclusively poorly fitted models at a much higher rate than the Bayesian CFI and TLI, and the ability of the Bayesian RMSEA to properly detect misfit improved as sample sizes increased.</p> <p>Graph: Figure 9. Classification of the fit of the model based on the Bayesian RMSEA, CFI, and TLI for the secondary simulation study examining two cross-loadings; the correct model is compared to a model with one cross-loading ignored and a model with both cross-loadings ignored.</p> <p>Finally, Figure 10 contains the rejection rates for these three models across sample size (in the panels) and index (lines within the panels). Overall, the PPP was most consistent in detecting the correct model across sample sizes, but once sample sizes were</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>≥</mo><mn>200</mn></mrow><mo>,</mo></math> </ephtml> all indices detected the correct model and the RMSEA and PPP were able to accurately detect the misfit. It is notable that the Bayesian CFI and TLI had relatively much smaller rejection rates for the mis-specified models, even when sample sizes were relatively larger.</p> <p>Graph: Figure 10. Rejection rates of models using the PPP, the Bayesian RMSEA, CFI, and TLI for the secondary simulation study examining two cross-loadings; the correct model is compared to a model with one cross-loading ignored and a model with both cross-loadings ignored. Note that dashed lines should not be viewed as strict cutoff values for the index. Instead, they should be viewed as a way to help visually interpret results.</p> <hd id="AN0178359442-30">Over-Fitting</hd> <p>Next, we shift our attention to these (approximate) fit measures being used to detect over-fitting, or mis-specification via the inclusion of "extra" model parameters.</p> <p>ANOVA results (Table 1), indicated that the distributions of all fit index values were primarily affected by the model specification. In addition, the RMSEA, CFI, and TLI distributions also differed substantially across sample sizes. To understand the distributions of the fit indices across different types of model mis-specification, we included box plots for 1,000 replicates of the PPP, Bayesian RMSEA, CFI, and TLI in Figure 11. This figure is structured exactly as described for Figure 5 above. The results across the four indices are strikingly similar. In all cases, across all sample sizes, only one model formulation was flagged as a poor fit, and that was the model that extracted three factors (M 5o). All other models were deemed as a good fit. Aside from a narrowing of the empirical distributions, the sample size effect was negligible in that all patterns remained across the sample size conditions.</p> <p>Graph: Figure 11. Box plots of the empirical distribution for the Bayesian RMSEA, CFI, TLI, and PPP for the true model (model 1) and the models with over-fit (models 2-5). Note that dashed lines should not be viewed as strict cutoff values for the index. Instead, they should be viewed as a way to help visually interpret results.</p> <p>Likewise, Figure 12 can be interpreted as comparable to Figure 6. Results show clearly that the three-factor model (M 5o) is conclusively marked as poor fitting, regardless of sample size. This was the only model for <emph>n</emph> > 50 conclusively flagged as being misfit according to the index 90% CI. The inconclusive results present in <emph>n</emph> = 50 and 100 disappear with larger sample sizes, indicating that all models (except M 5o) are conclusively classified as good fit under the larger sample sizes.</p> <p>Graph: Figure 12. Classification of the fit of the model based on the Bayesian RMSEA, CFI, and TLI for models with over-fit.</p> <p>Figure 13 contains rejection rates for the (approximate) fit indices in over-fitted modeling situations. Specifically, the figure displays results for the correct model and four over-fitted models. Results are comparable across all three sample size conditions. The model showing the highest rejection rate corresponds to the over-specification of the number of factors (M 5o). This severe mis-specification is almost always deemed a model misfit by the indices. In contrast, the other forms of over-specification have much lower rejection rates. When sample sizes are larger, the indices show the same rate at pointing toward proper fit for the correct model as for the remaining three mis-specified models (Models 2o-4o). These results show that over-specifying the factor covariance (M 2o), the residual correlation (M 3o), and the cross-loading (M 4o) fit just as well as the correct model according to these indices.</p> <p>Graph: Figure 13. Rejection rates of models using the PPP, the Bayesian RMSEA, CFI, and TLI for the true model and the over-fitted model.</p> <hd id="AN0178359442-31">Prior Sensitivity Analysis</hd> <p>In this next section, we investigated the influence of prior distribution settings on the performance of these (approximate) fit indices. Specifically, we extended the over-fitted model with an added cross-loading to include different prior forms on that extra cross-loading. Our hypothesis was that if a researcher were to include an extra parameter via a cross-loading that the results would be dependent, at least in part, on the prior placed on that cross-loading. The following results detail our findings for this inquiry.</p> <hd id="AN0178359442-32">The Impact of Priors on Cross-Loadings (over-Fit)</hd> <p>We included results for five different models in Figure 14: the correct model (without any cross-loading), a mis-specified model with a cross-loading added but with default diffuse priors placed on that loading parameter, and a mis-specified model with a cross-loading containing a divergent prior that was either weak, moderate, or strong/informative with respect to its strength. In the case of the diverging prior, we were specifically interested in assessing whether model fit assessment would be influenced with the inclusion of a mis-specification that was tied to a strong opinion via the prior distribution. The settings for these three divergent priors were as follows:</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><mn>0.5</mn><mo>,</mo><mn>1.0</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> (weak),</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><mn>0.5</mn><mo>,</mo><mn>0.01</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> (moderate),</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi mathvariant="script">N</mi><mo stretchy="false">(</mo><mn>0.5</mn><mo>,</mo><mn>0.001</mn><mo stretchy="false">)</mo></mrow></math> </ephtml> (strong).</p> <p>Graph: Figure 14. Box plots of the empirical distribution of RMSEA, CFI, TLI, and PPP for the true model and the models with an extra cross-loading under different priors. Note that dashed lines should not be viewed as strict cutoff values for the index. Instead, they should be viewed as a way to help visually interpret results.</p> <p>ANOVA results (Table 1), indicated that the distributions of all fit index values were affected by the prior specification. In addition, the RMSEA, CFI, and TLI distributions also differed substantially across sample sizes. In contrast, the PPP distribution was less affected by the sample size. Figure 14 contains box plots representing the empirical distributions for the indices. Each row within the figure represents a different (approximate) fit index. Just as before, the dotted line in each plot can be used as a guide for conventional cutoff values implemented in the literature for each index. The empirical distribution, as seen in the box plots, shows reduced variability as sample sizes increase. However, the main pattern is that Model 2d is the only one classified as misfit by the PPP according to the conventional cutoff values. As sample sizes increase, the approximate Bayesian fit indices do not identify any of the models as clear cases of misfit.</p> <p>Figure 15 showcases these models along the <emph>x</emph>-axis, and the <emph>y</emph>-axis represents the rejection rate proportion across the 1,000 simulation replications. The (approximate) fit measures make up the plot lines, and each panel represents a different sample size.</p> <p>Graph: Figure 15. Rejection rates of models of the correct model and over-fit models with an extra cross loading.</p> <p>Some interesting patterns emerged across the sample size conditions and various indices. For the smallest sample size of <emph>n</emph> = 50, the Bayesian approximate fit indices all had higher rejection rates, even for the correct model. That pattern continued for <emph>n</emph> = 100, but the rejection rates were much lower than with <emph>n</emph> = 50. As sample sizes continued to increase (especially</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mo>≥</mo><mn>500</mn></mrow></math> </ephtml> ), the PPP was able to pinpoint the divergent prior with the strongest precision (denoted as Model 2d in the figure) as misfit, while the other indices did not reject the mis-specified models. These findings suggest that the <emph>degree</emph> to which the prior diverges from the population is related to the ability for these indices to properly detect model mis-specification. A strong, diverging prior is linked to better detection of model misfit.</p> <hd id="AN0178359442-33">Simulation Results II: Model Selection from Competing Models</hd> <p>In this second part of the Results, we investigated the performance of Bayesian model comparison tools. We focus here on two separate groups of model comparison tools.</p> <p>The first group of model comparison tools that we focus on are not conventional <emph>model comparison</emph> indices. Instead, these are the approximate fit measures described in the last section of the Results. In this section, we explore the possibility of the Bayesian RMSEA, CFI, and TLI to be used in a relative sense when comparing multiple competing models with one another. Although these indices are not conventionally described as model comparison tools, we aim to explore whether they could be used in this sense to detect degrees (or severity) of model mis-specification. Our interpretation was such that a model was the selected model if its RMSEA was relatively smaller, CFI was larger, or TLI was larger than a competing model.</p> <p>The second group of indices represents conventional information criteria, specifically the DIC and the BIC. These indices were developed to select the optimal model out of a set of competing models, and the index values are only interpretable in a model comparison situation (i.e., the DIC of a single model is not interpretable until it is compared to the DIC for a competing model).</p> <hd id="AN0178359442-34">Under-Fitting</hd> <p>When default priors were used, the Bayesian approximate fit indices (Bayesian CFI, TLI, and RMSEA) and the conventional Bayesian model comparison indices (DIC and BIC) selected the correct model with nearly perfect accuracy for</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>n</mi><mo>≥</mo><mn>100</mn></mrow><mo>.</mo></math> </ephtml> The selection rates for all indices were close to 1.0 for detecting the correct model as compared to an under-fitted mis-specification; results are narrated as opposed to displayed in a table because they were nearly all perfect, with the exception for <emph>n</emph> = 50, where the BIC only selected the correct model in 31% of the replications. These results suggest that the Bayesian SEM approximate fit indices and the model comparison indices worked well in detecting under-fitting within the model.</p> <hd id="AN0178359442-35">Over-Fitting</hd> <p></p> <hd id="AN0178359442-36">Bayesian SEM Fit Indices</hd> <p>Results for the conventional SEM approximate fit indices (Bayesian RMSEA, CFI, and TLI) are presented in Table 2 in columns 2–4. This table contains the proportion of 1,000 replications of a model selected over the competing models. Proportions within a given column (for a given sample size) will sum to 1. The model with the highest proportion was selected at a greater rate across the competing models.</p> <p>Table 2 Over-fitting: proportion of a model being the selected model with a relatively smaller RMSEA, larger CFI or TLI, smaller DIC or BIC.</p> <p> <ephtml> <table><thead><tr><td>N</td><td>Model</td><td>RMSEA</td><td>CFI</td><td>TLI</td><td>DIC</td><td>BIC</td></tr></thead><tbody valign="top"><tr><td /><td>Correct</td><td char=".">0.520</td><td char=".">0.481</td><td char=".">0.525</td><td char=".">0.599</td><td char=".">0.875</td></tr><tr><td /><td>Three Factors</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td></tr><tr><td>50</td><td>Cross loading</td><td char=".">0.143</td><td char=".">0.158</td><td char=".">0.140</td><td char=".">0.103</td><td char=".">0.039</td></tr><tr><td /><td>Factor Covariance</td><td char=".">0.169</td><td char=".">0.180</td><td char=".">0.167</td><td char=".">0.138</td><td char=".">0.045</td></tr><tr><td /><td>Residual Correlation</td><td char=".">0.168</td><td char=".">0.181</td><td char=".">.0168</td><td char=".">0.160</td><td char=".">0.041</td></tr><tr><td /><td>Correct</td><td char=".">0.564</td><td char=".">0.536</td><td char=".">0.562</td><td char=".">0.574</td><td char=".">0.897</td></tr><tr><td /><td>Three Factors</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td></tr><tr><td>100</td><td>Cross loading</td><td char=".">0.127</td><td char=".">0.140</td><td char=".">0.127</td><td char=".">0.120</td><td char=".">0.033</td></tr><tr><td /><td>Factor Covariance</td><td char=".">0.152</td><td char=".">0.159</td><td char=".">0.153</td><td char=".">0.143</td><td char=".">0.040</td></tr><tr><td /><td>Residual Correlation</td><td char=".">0.157</td><td char=".">0.165</td><td char=".">0.158</td><td char=".">0.163</td><td char=".">0.030</td></tr><tr><td /><td>Correct</td><td char=".">0.612</td><td char=".">0.596</td><td char=".">0.611</td><td char=".">0.587</td><td char=".">0.925</td></tr><tr><td /><td>Three Factors</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td></tr><tr><td>200</td><td>Cross loading</td><td char=".">0.125</td><td char=".">0.129</td><td char=".">0.124</td><td char=".">0.133</td><td char=".">0.025</td></tr><tr><td /><td>Factor Covariance</td><td char=".">0.128</td><td char=".">0.132</td><td char=".">0.129</td><td char=".">0.137</td><td char=".">0.021</td></tr><tr><td /><td>Residual Correlation</td><td char=".">0.135</td><td char=".">0.143</td><td char=".">0.136</td><td char=".">0.143</td><td char=".">0.029</td></tr><tr><td /><td>Correct</td><td char=".">0.636</td><td char=".">0.607</td><td char=".">0.638</td><td char=".">0.569</td><td char=".">0.958</td></tr><tr><td /><td>Three Factors</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td></tr><tr><td>500</td><td>Cross loading</td><td char=".">0.126</td><td char=".">0.133</td><td char=".">0.125</td><td char=".">0.136</td><td char=".">0.017</td></tr><tr><td /><td>Factor Covariance</td><td char=".">0.133</td><td char=".">0.145</td><td char=".">0.133</td><td char=".">0.155</td><td char=".">0.01</td></tr><tr><td /><td>Residual Correlation</td><td char=".">0.105</td><td char=".">0.115</td><td char=".">0.104</td><td char=".">0.140</td><td char=".">0.015</td></tr><tr><td /><td>Correct</td><td char=".">0.660</td><td char=".">0.635</td><td char=".">0.658</td><td char=".">0.569</td><td char=".">0.975</td></tr><tr><td /><td>Three Factors</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td><td char=".">0</td></tr><tr><td>1000</td><td>Cross loading</td><td char=".">0.115</td><td char=".">0.126</td><td char=".">0.117</td><td char=".">0.137</td><td char=".">0.009</td></tr><tr><td /><td>Factor Covariance</td><td char=".">0.113</td><td char=".">0.121</td><td char=".">0.112</td><td char=".">0.149</td><td char=".">0.011</td></tr><tr><td /><td>Residual Correlation</td><td char=".">0.112</td><td char=".">0.118</td><td char=".">0.113</td><td char=".">0.145</td><td char=".">0.005</td></tr></tbody></table> </ephtml> </p> <p>Overall, results across the Bayesian RMSEA, CFI, and TLI were very similar to each other with respect to selection rates. None of the indices selected the mis-specified three-factor model. All indices favored the correct model most, with approximately the same selection rate within sample size (e.g, selection rates across these indices ranged from 53.6-56.4% for <emph>n</emph> = 100). The selection of the models with a mis-specified cross-loading, factor covariance, or residual correlation were comparably selected in relation to each other (all of these over-fitted models were selected between approximately 10-18% of the replications). At the largest sample size of <emph>n</emph> = 1000, the indices ranged from 63.5-66.0% of proper selection, with no notable difference between the index selection rates.</p> <hd id="AN0178359442-37">DIC And BIC</hd> <p>The DIC and BIC are presented in the last two columns of Table 2. Overall, the BIC selected the correct model at a higher rate than did the DIC, and correct selection rates increased with larger sample sizes. The BIC's correct selection rate ranged from</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>87.5</mn><mo>−</mo><mn>97.5</mn><mi>%</mi></mrow><mo>,</mo></math> </ephtml> as compared to the DIC's rates of</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>56.9</mn><mo>−</mo><mn>59.9</mn><mi>%</mi></mrow><mo>.</mo></math> </ephtml> Just as described with the approximate fit indices, neither of these information criteria selected the most severe mis-specification of a three factor solution in any of the replications. The selection of the models with a mis-specified cross-loading, factor covariance, or residual correlation was comparably selected in relation to each other (e.g., the BIC favored these mis-specified models between</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>1</mn><mo>−</mo><mn>4</mn><mi>%</mi></mrow></math> </ephtml> depending on sample size; the DIC favored them between approximately</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mn>10</mn><mo>−</mo><mn>16</mn><mi>%</mi></mrow></math> </ephtml> ).</p> <hd id="AN0178359442-38">Discussion</hd> <p>The current investigation provided insight as to the accuracy of common model fit and comparison tools for detecting model misfit. Misfit was categorized into two main types: under-fit (ignoring non-zero model parameters in the analysis model) and over-fit (freeing model parameters that are truly zero in the population model). Our investigation examined two aspects of model assessment within these mis-specification conditions. The first part of this study included an examination of (approximate) model fit for a given model. In this case, we sought to uncover how well the (approximate) model fit indices could detect misfit when a single model was being examined according to nominal cutoff values for the indices (Bayesian RMSEA, CFI, TLI, and PPP). The second part of this investigation surrounded model comparison. In other words, we examined how accurate select indices (Bayesian RMSEA, CFI, TLI, BIC, and DIC) are at identifying a correct model as opposed to one that includes misfit (either under- or over-fit).</p> <hd id="AN0178359442-39">Model Fit</hd> <p>We approach the idea of model fit with caution for a few reasons, but the biggest has to do with how model fit is typically implemented in the applied literature. Cutoff values are commonly implemented in the frequentist SEM literature (see e.g., Hu & Bentler, [<reflink idref="bib13" id="ref78">13</reflink>]), and our experience is that applied researchers tend to rely heavily on these cutoff values for whether a model is "viable" or not. We mentioned above our reluctance to implement any sort of cutoff value for the indices explored here. However, we opted to provide context to the values presented in the Results section by mapping them onto conventional interpretations for the frequentist versions of the RMSEA, CFI, and TLI. Even with this context of the cutoff values provided, we caution the reader to not map conventional cutoff values to the Bayesian versions of these indices in a strict way. Largely, we do not yet understand whether such cutoff values are useful or relevant to the Bayesian literature. In addition, we are proponents of interpreting the entire posterior as opposed to a central tendency estimate for the index.</p> <p>If cutoff values are to be used, then we would advocate them to be used in conjunction with understanding the CIs akin to how we interpreted results in Figures 6 and 12. According to these results, we can see conclusive poor model fit was obtained for many of the under-fitted models (Figure 6), but only for the most severely mis-specified over-fitted model (with three factors; see Figure 12). The degree to which the correct model was conclusively selected according to the CI improved with larger sample sizes for all indices examined.</p> <p>Overall, the PPP remains a good candidate for properly capturing model fit (and model misfit). The Bayesian versions of the RMSEA, CFI, and TLI had more variable results, but still remain possibilities for the assessment of model fit within the BSEM context.</p> <hd id="AN0178359442-40">Model Comparison</hd> <p>It is common to see indices such as the BIC and DIC being used for model comparison purposes. However, we also wanted to explore in more detail the potential capabilities of the Bayesian approximate fit measures to be used in this manner. Model selection is a key component of applied SEM research, and understanding the role that these indices can play is vital for substantive researchers looking to implement a model selection process within BSEM.</p> <p>With the correct model present, using the approximate model fit and model comparison indices can pinpoint the correct model almost perfectly over the under-fitted model. These indices were almost exact in their accuracy to detect model misfit when the analysis model was under-fit. This seems to us to be the most egregious type of specification error because ignoring a component of a model can influence other model parameters, as well as the overall structure of the model. Our results indicated that applied researchers can have confidence in all of the indices to select the correct model over an under-fitted comparison model.</p> <p>Using the approximate fit and comparison indices to detect over-specification of the model led to a more convoluted picture of results. Overall, intuitive results were obtained in that the more severe specification errors with respect to over-fitting (e.g., extracting an extra factor in the analysis model) were more likely to be flagged as misfit as compared to less severe errors (e.g., an extra cross-loading). In the current study, we considered sample sizes ranging between as small as 50 and as large as 1,000, which covers a typical sample size used for analyzing an SEM model. The ability of the BSEM fit indices to select the true model increases gradually as the sample size becomes larger. We are confident this trend is foreseeable with sample sizes larger than 1000. However, it is worth noting that the rate of such an increase slows down. For instance, the selection rate of RMSEA increases by around 6% when the sample size increases from 100 to 200, by around 2.4% when the sample size increases from 200 to 500, and by around 2% when the sample size increases from 500 to 1,000. The rate of change slows down overall for all fit indices included. This slowdown appears particularly problematic for the DIC and approximate fit indices. It appears that one would require an impractically large sample size for these fit indices to reliably select the correctly specified model. Only the BIC meaningfully outperformed the DIC and the approximate fit indices when properly selecting the correct model compared to over-fitted models. This result is surprising given that previous research comparing the DIC and BIC in the context of under-fitting has tended to support the use of the DIC over the BIC (Cain & Zhang, [<reflink idref="bib3" id="ref79">3</reflink>]; Winter & Depaoli, [<reflink idref="bib38" id="ref80">38</reflink>], [<reflink idref="bib39" id="ref81">39</reflink>]), perhaps giving some the impression that the DIC should always be preferred. Our study's findings have important substantive implications because they suggest that when there is a discrepancy across the indices in applied research settings, a preference toward the BIC may be warranted to avoid over-fitting (see also Lu et al., [<reflink idref="bib19" id="ref82">19</reflink>]). It is important to note that the BIC has a larger penalty term than the DIC, which acts as an advantage for the BIC for less complex models.</p> <hd id="AN0178359442-41">Future Methodological Areas of Study</hd> <p>There are many different aspects that are still in need of development in the Bayesian SEM framework, and this work exposed a couple of areas that would benefit from additional research. First, more sensitive (approximate) model fit measures would benefit researchers who are attempting to compare models with potential specification errors embedded. The most beneficial index is one that can decipher between levels of mis-specification, perhaps highlighting the model with the less severe specification error. In some of our conditions, the indices could not distinguish between the degree of mis-specification, and the ability to do this would benefit applied researchers using these indices for model selection purposes. In addition, we included specification errors that were present in the measurement and structural parts of the model. It is important to understand how errors in a specific part of the model might influence results obtained throughout the remaining parts of the model. Our investigation surrounded a common measurement model (namely, the CFA), and we included some elements of covariance. However, an investigation including a larger and more complex structural model would help shed light on the potential influence structural mis-specifications have on the performance of these model assessment indices. In addition, priors mapping onto structural model parameters have been found to have varied degrees of impact on model results. Studying this aspect in greater depth would help provide a more complete assessment of the ability of these indices to detect specification errors in a variety of modeling contexts within SEM.</p> <hd id="AN0178359442-42">Concluding Remarks</hd> <p>The indices examined here will continue to play a vital role in model assessment within the Bayesian estimation framework, especially as Bayesian methods continue to gain popularity for SEMs. Overall, our results showed a clear pattern of how different forms of model mis-specification can be best detected within the context of BSEM. These results provide an important guide for how the approximate fit and model comparison indices perform in cases of under- and over-fitting a model. When indices provide discrepant results in an applied setting, then researchers should take particular care when selecting a model. The current investigation can act as a guide for which indices to lean on more heavily when examining BSEMs. Future work should consider extending this investigation to include additional Bayesian indices (e.g., WAIC, LOOIC) to showcase overall performance for a wide selection of indices within the Bayesian estimation framework.</p> <hd id="AN0178359442-43">Disclosure statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <ref id="AN0178359442-44"> <title> References </title> <blist> <bibl id="bib1" idref="ref28" type="bt">1</bibl> <bibtext> Asparouhov, T., & Muthén, B. (2010). Bayesian analysis of latent variable models using mplus. Retrieved, 17, 2014.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref35" type="bt">2</bibl> <bibtext> Asparouhov, T., & Muthén, B. (2021). Advances in bayesian model fit evaluation for structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 28, 1 – 14. https://doi.org/10.1080/10705511.2020.1764360</bibtext> </blist> <blist> <bibl id="bib3" idref="ref29" type="bt">3</bibl> <bibtext> Cain, M. K., & Zhang, Z. (2019). Fit for a Bayesian: An evaluation of PPP and DIC for structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 26, 39 – 50. https://doi.org/10.1080/10705511.2018.1490648</bibtext> </blist> <blist> <bibl id="bib4" idref="ref45" type="bt">4</bibl> <bibtext> Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233 – 255. https://doi.org/10.1207/S15328007SEM0902_5</bibtext> </blist> <blist> <bibl id="bib5" idref="ref1" type="bt">5</bibl> <bibtext> Depaoli, S. (2013). Mixture class recovery in GMM under varying degrees of class separation: Frequentist versus Bayesian estimation. Psychological Methods, 18, 186 – 219. https://doi.org/10.1037/a0031609</bibtext> </blist> <blist> <bibl id="bib6" idref="ref38" type="bt">6</bibl> <bibtext> Edwards, K. D., & Konold, T. R. (2023). Impact of informative priors on model fit indices in Bayesian confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 30, 272 – 283. https://doi.org/10.1080/10705511.2022.2126359</bibtext> </blist> <blist> <bibl id="bib7" idref="ref10" type="bt">7</bibl> <bibtext> Garnier-Villarreal, M., & Jorgensen, T. D. (2020). Adapting fit indices for Bayesian structural equation modeling: Comparison to maximum likelihoods. Psychological Methods, 25, 46 – 70. https://doi.org/10.1037/met0000224</bibtext> </blist> <blist> <bibl id="bib8" idref="ref11" type="bt">8</bibl> <bibtext> Heene, M., Hilbert, S., Freudenthaler, H. H., & Bühner, M. (2012). Sensitivity of SEM fit indexes with respect to violations of uncorrelated errors. Structural Equation Modeling: A Multidisciplinary Journal, 19, 36 – 50. https://doi.org/10.1080/10705511.2012.634710</bibtext> </blist> <blist> <bibl id="bib9" idref="ref37" type="bt">9</bibl> <bibtext> Hoofs, H., van de Schoot, R., Jansen, N. W. H., & Kant, I. (2018). Evaluating model fit in Bayesian confirmatory factor analysis with large samples: Simulation study introducing the brmsea. Educational and Psychological Measurement, 78, 537 – 568. https://doi.org/10.1177/0013164417709314</bibtext> </blist> <blist> <bibtext> Hoyle, R. H. (2012). Handbook of structural equation modeling. Guilford press.</bibtext> </blist> <blist> <bibtext> Hsu, H.-Y., Man Kwok, O., Lin, J. H., & Acosta, S. (2015). Detecting misspecified multilevel structural equation models with common fit indices: A Monte Carlo study. Multivariate Behavioral Research, 50, 197 – 215. https://doi.org/10.1080/00273171.2014.977429</bibtext> </blist> <blist> <bibtext> Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424 – 453. https://doi.org/10.1037/1082-989X.3.4.424</bibtext> </blist> <blist> <bibtext> Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1 – 55. https://doi.org/10.1080/10705519909540118</bibtext> </blist> <blist> <bibtext> Kaplan, D. (1988). The impact of specification error on the estimation, testing, and improvement of structural equation models. Multivariate Behavioral Research, 23, 69 – 86. https://doi.org/10.1207/s15327906mbr2301_4</bibtext> </blist> <blist> <bibtext> Lee, S.-Y. (2007). Structural equation modeling: A Bayesian approach. John Wiley & Sons.</bibtext> </blist> <blist> <bibtext> Liang, X., & Luo, Y. (2020). A comprehensive comparison of model selection methods for testing factorial invariance. Structural Equation Modeling: A Multidisciplinary Journal, 27, 380 – 395. https://doi.org/10.1080/10705511.2019.1649983</bibtext> </blist> <blist> <bibtext> Liu, H., Depaoli, S., & Marvin, L. (2022). Understanding the deviance information criterion for SEM: Cautions in prior specification. Structural Equation Modeling: A Multidisciplinary Journal, 29, 278 – 294. https://doi.org/10.1080/10705511.2021.1994407</bibtext> </blist> <blist> <bibtext> Liu, H., Zhang, Z., & Grimm, K. J. (2016). Comparison of inverse Wishart and separation-strategy priors for Bayesian estimation of covariance parameter matrix in growth curve analysis. Structural Equation Modeling: A Multidisciplinary Journal, 23, 354 – 367. https://doi.org/10.1080/10705511.2015.1057285</bibtext> </blist> <blist> <bibtext> Lu, Z.-H., Chow, S.-M., & Loken, E. (2017). A comparison of Bayesian and frequentist model selection methods for factor analysis models. Psychological Methods, 22, 361 – 381. https://doi.org/10.1037/met0000145</bibtext> </blist> <blist> <bibtext> Marsh, H. W., Hau, K.-T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11, 320 – 341. Retrieved from https://doi.org/10.1207/s15328007sem1103_2</bibtext> </blist> <blist> <bibtext> Maydeu-Olivares, A. (2017). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Structural Equation Modeling: A Multidisciplinary Journal, 24, 383 – 394. https://doi.org/10.1080/10705511.2016.1269606</bibtext> </blist> <blist> <bibtext> Merkle, E. C., & Rosseel, Y. (2015). blavaan: Bayesian structural equation models via parameter expansion. arXiv preprint arXiv:1511.05604</bibtext> </blist> <blist> <bibtext> Mulaik, S., James, L., Alstine, J., Bennett, N., Lind, S., & Stilwell, C. (1989). Evaluation of goodness-of-fit indices for structural equation models. Psychological Bulletin, 105, 430 – 445. https://doi.org/10.1037/0033-2909.105.3.430</bibtext> </blist> <blist> <bibtext> Muthén, B., & Asparouhov, T. (2012). Bayesian structural equation modeling: A more flexible representation of substantive theory. Psychological Methods, 17, 313 – 335. https://doi.org/10.1037/a0026802</bibtext> </blist> <blist> <bibtext> Muthén, L. K., & Muthén, B. (2017). Mplus user's guide: Statistical analysis with latent variables, user's guide. Muthén & Muthén.</bibtext> </blist> <blist> <bibtext> Rigdon, E. E. (1996). CFI versus RMSEA: A comparison of two fit indexes for structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 3, 369 – 379. https://doi.org/10.1080/10705519609540052</bibtext> </blist> <blist> <bibtext> Rindskopf, D. (2012). Next steps in Bayesian structural equation models: Comments on, variations of, and extensions to Muthén and Asparouhov (2012). Psychological Methods, 17, 336 – 339. https://doi.org/10.1037/a0027130</bibtext> </blist> <blist> <bibtext> Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461 – 464. https://doi.org/10.1214/aos/1176344136</bibtext> </blist> <blist> <bibtext> Shi, D., Lee, T., & Maydeu-Olivares, A. (2019). Understanding the model size effect on SEM fit indices. Educational and Psychological Measurement, 79, 310 – 334. https://doi.org/10.1177/0013164418783530</bibtext> </blist> <blist> <bibtext> Smid, S. C., McNeish, D., Miočević, M., & van de Schoot, R. (2020). Bayesian versus frequentist estimation for structural equation models in small sample contexts: A systematic review. Structural Equation Modeling: A Multidisciplinary Journal, 27, 131 – 161. https://doi.org/10.1080/10705511.2019.1577140</bibtext> </blist> <blist> <bibtext> Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society Series B: Statistical Methodology, 64, 583 – 639. https://doi.org/10.1111/1467-9868.00353</bibtext> </blist> <blist> <bibtext> van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthén, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 4, 770. https://doi.org/10.3389/fpsyg.2013.00770</bibtext> </blist> <blist> <bibtext> Van de Schoot, R., Winter, S. D., Ryan, O., Zondervan-Zwijnenburg, M., & Depaoli, S. (2017). A systematic review of Bayesian articles in psychology: The last 25 years. Psychological Methods, 22, 217 – 239. https://doi.org/10.1037/met0000100</bibtext> </blist> <blist> <bibtext> Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413 – 1432. https://doi.org/10.1007/s11222-016-9696-4</bibtext> </blist> <blist> <bibtext> Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2019). Rank-normalization, folding, and localization: An improved R ̂ for assessing convergence of MCMC. Retrieved from https://arxiv.org/pdf/1903.08008.pdf</bibtext> </blist> <blist> <bibtext> Ward, E. J. (2008). A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools. Ecological Modelling, 211, 1 – 10. https://doi.org/10.1016/j.ecolmodel.2007.10.030</bibtext> </blist> <blist> <bibtext> Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571 – 3594.</bibtext> </blist> <blist> <bibtext> Winter, S. D., & Depaoli, S. (2022a). Performance of model fit and selection indices for Bayesian structural equation modeling with missing data. Structural Equation Modeling: A Multidisciplinary Journal, 29, 531 – 549. https://doi.org/10.1080/10705511.2021.2018656</bibtext> </blist> <blist> <bibtext> Winter, S. D., & Depaoli, S. (2022b). Sensitivity of Bayesian model fit indices to the prior specification of latent growth models. Structural Equation Modeling: A Multidisciplinary Journal, 29, 667 – 686. https://doi.org/10.1080/10705511.2022.2032078</bibtext> </blist> <blist> <bibtext> Yuan, K.-H., Wu, R., & Bentler, P. M. (2011). Ridge structure equation modeling with correlation matrices for ordinal and continuous data. The British Journal of Mathematical and Statistical Psychology, 64, 107 – 133. https://doi.org/10.1348/000711010X497442</bibtext> </blist> </ref> <ref id="AN0178359442-45"> <title> Footnotes </title> <blist> <bibtext> We note that other prior distribution forms could have been implemented instead of those presented here. We narrowed our investigation to these priors since they are the most commonly specified in the literature. Future investigations may examine the influence of other prior forms.</bibtext> </blist> <blist> <bibtext> For more detail about these indices, including the mathematical specification for them, please see Garnier-Villarreal and Jorgensen ([7]) and Asparouhov and Muthén ([2]). However, some details are presented here in the text.</bibtext> </blist> </ref> <aug> <p>By Sarah Depaoli; Sonja D. Winter and Haiyan Liu</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib18" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib32" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib30" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib40" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib25" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib22" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib33" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib14" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib20" firstref="ref12"></nolink> <nolink nlid="nl10" bibid="bib26" firstref="ref13"></nolink> <nolink nlid="nl11" bibid="bib21" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib29" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib10" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib23" firstref="ref18"></nolink> <nolink nlid="nl15" bibid="bib12" firstref="ref19"></nolink> <nolink nlid="nl16" bibid="bib17" firstref="ref20"></nolink> <nolink nlid="nl17" bibid="bib13" firstref="ref22"></nolink> <nolink nlid="nl18" bibid="bib11" firstref="ref24"></nolink> <nolink nlid="nl19" bibid="bib15" firstref="ref25"></nolink> <nolink nlid="nl20" bibid="bib24" firstref="ref26"></nolink> <nolink nlid="nl21" bibid="bib27" firstref="ref30"></nolink> <nolink nlid="nl22" bibid="bib28" firstref="ref31"></nolink> <nolink nlid="nl23" bibid="bib31" firstref="ref32"></nolink> <nolink nlid="nl24" bibid="bib37" firstref="ref33"></nolink> <nolink nlid="nl25" bibid="bib34" firstref="ref34"></nolink> <nolink nlid="nl26" bibid="bib38" firstref="ref40"></nolink> <nolink nlid="nl27" bibid="bib39" firstref="ref41"></nolink> <nolink nlid="nl28" bibid="bib16" firstref="ref51"></nolink> <nolink nlid="nl29" bibid="bib36" firstref="ref52"></nolink> <nolink nlid="nl30" bibid="bib35" firstref="ref77"></nolink> <nolink nlid="nl31" bibid="bib19" firstref="ref82"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1431135
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sarah+Depaoli%22">Sarah Depaoli</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-1277-0462">0000-0002-1277-0462</externalLink>)<br /><searchLink fieldCode="AR" term="%22Sonja+D%2E+Winter%22">Sonja D. Winter</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2203-002X">0000-0002-2203-002X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Haiyan+Liu%22">Haiyan Liu</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Structural+Equation+Modeling%3A+A+Multidisciplinary+Journal%22"><i>Structural Equation Modeling: A Multidisciplinary Journal</i></searchLink>. 2024 31(4):604-625.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Structural+Equation+Models%22">Structural Equation Models</searchLink><br /><searchLink fieldCode="DE" term="%22Bayesian+Statistics%22">Bayesian Statistics</searchLink><br /><searchLink fieldCode="DE" term="%22Comparative+Testing%22">Comparative Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Utilization%22">Evaluation Utilization</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Selection%22">Test Selection</searchLink><br /><searchLink fieldCode="DE" term="%22Robustness+%28Statistics%29%22">Robustness (Statistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Goodness+of+Fit%22">Goodness of Fit</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/10705511.2023.2280952
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1070-5511<br />1532-8007
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We extended current knowledge by examining the performance of several Bayesian model fit and comparison indices through a simulation study using the confirmatory factor analysis. Our goal was to determine whether commonly implemented Bayesian indices can detect specification errors. Specifically, we wanted to uncover any differences in detecting under-fitting or over-fitting a model. We examined a conventional Bayesian fit index (the posterior predictive p-value), approximate Bayesian fit indices (Bayesian RMSEA, CFI, and TLI), and model comparison indices (BIC and DIC). We varied the type and severity of model mis-specification, sample size, and priors. We focused on the ability of these indices to detect model under- or over-fitting. We provide practical advice for applied researchers regarding how to assess and compare models using these common indices implemented in the Bayesian framework.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1431135
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1431135
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/10705511.2023.2280952
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 604
    Subjects:
      – SubjectFull: Structural Equation Models
        Type: general
      – SubjectFull: Bayesian Statistics
        Type: general
      – SubjectFull: Comparative Testing
        Type: general
      – SubjectFull: Evaluation Utilization
        Type: general
      – SubjectFull: Test Selection
        Type: general
      – SubjectFull: Robustness (Statistics)
        Type: general
      – SubjectFull: Goodness of Fit
        Type: general
    Titles:
      – TitleFull: Under-Fitting and Over-Fitting: The Performance of Bayesian Model Selection and Fit Indices in SEM
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sarah Depaoli
      – PersonEntity:
          Name:
            NameFull: Sonja D. Winter
      – PersonEntity:
          Name:
            NameFull: Haiyan Liu
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1070-5511
            – Type: issn-electronic
              Value: 1532-8007
          Numbering:
            – Type: volume
              Value: 31
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Structural Equation Modeling: A Multidisciplinary Journal
              Type: main
ResultId 1