View in EDS HTML Full Text PDF Full Text

How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study.

Saved in:
Bibliographic Details
Title: How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study.
Authors: Ryan, Michael S., Gielissen, Katherine A., Shin, Dongho, Perera, Robert A., Gusic, Maryellen, Ferenchick, Gary, Ownby, Allison, Cutrer, William B., Obeso, Vivian, Santen, Sally A.
Source: Medical Education. Jul2024, Vol. 58 Issue 7, p825-837. 13p.
Subjects: Educational test & measurement standards, Research funding, Work environment, Evaluation of human services programs, Research evaluation, Descriptive statistics
Abstract: Background: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs. Methods: The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. Results: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. Conclusion: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions. Through examination of 53,000 workplace‐based assessments at 7 schools, no factors were found to be consistently associated with reliability, leading to the conclusion that these assessments are best used for formative feedback only. [ABSTRACT FROM AUTHOR]
Copyright of Medical Education is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Psychology and Behavioral Sciences Collection
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: pbh
DbLabel: Psychology and Behavioral Sciences Collection
An: 177626278
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Ryan%2C+Michael+S%2E%22&quot;&gt;Ryan, Michael S.&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Gielissen%2C+Katherine+A%2E%22&quot;&gt;Gielissen, Katherine A.&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Shin%2C+Dongho%22&quot;&gt;Shin, Dongho&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Perera%2C+Robert+A%2E%22&quot;&gt;Perera, Robert A.&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Gusic%2C+Maryellen%22&quot;&gt;Gusic, Maryellen&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Ferenchick%2C+Gary%22&quot;&gt;Ferenchick, Gary&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Ownby%2C+Allison%22&quot;&gt;Ownby, Allison&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Cutrer%2C+William+B%2E%22&quot;&gt;Cutrer, William B.&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Obeso%2C+Vivian%22&quot;&gt;Obeso, Vivian&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Santen%2C+Sally+A%2E%22&quot;&gt;Santen, Sally A.&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%22Medical+Education%22&quot;&gt;Medical Education&lt;/searchLink&gt;. Jul2024, Vol. 58 Issue 7, p825-837. 13p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Educational+test+%26+measurement+standards%22&quot;&gt;Educational test &amp; measurement standards&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Research+funding%22&quot;&gt;Research funding&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Work+environment%22&quot;&gt;Work environment&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Evaluation+of+human+services+programs%22&quot;&gt;Evaluation of human services programs&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Research+evaluation%22&quot;&gt;Research evaluation&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Descriptive+statistics%22&quot;&gt;Descriptive statistics&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace‐based assessments (WBAs). Single‐institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi‐institutional generalisability study to determine the influence of specific factors on reliability of WBAs. Methods: The authors analysed WBA data obtained for clerkship‐level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross‐classified random‐effects model was used to decompose variance components. A phi coefficient of &gt;0.7 was used as threshold for acceptable reliability. Results: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (&lt;10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to &gt;560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. Conclusion: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high‐stakes summative entrustment decisions. Through examination of 53,000 workplace‐based assessments at 7 schools, no factors were found to be consistently associated with reliability, leading to the conclusion that these assessments are best used for formative feedback only. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: &lt;i&gt;Copyright of Medical Education is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder&#39;s express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.&lt;/i&gt; (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=pbh&AN=177626278
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/medu.15291
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 825
    Subjects:
      – SubjectFull: Educational test & measurement standards
        Type: general
      – SubjectFull: Research funding
        Type: general
      – SubjectFull: Work environment
        Type: general
      – SubjectFull: Evaluation of human services programs
        Type: general
      – SubjectFull: Research evaluation
        Type: general
      – SubjectFull: Descriptive statistics
        Type: general
    Titles:
      – TitleFull: How well do workplace‐based assessments support summative entrustment decisions? A multi‐institutional generalisability study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ryan, Michael S.
      – PersonEntity:
          Name:
            NameFull: Gielissen, Katherine A.
      – PersonEntity:
          Name:
            NameFull: Shin, Dongho
      – PersonEntity:
          Name:
            NameFull: Perera, Robert A.
      – PersonEntity:
          Name:
            NameFull: Gusic, Maryellen
      – PersonEntity:
          Name:
            NameFull: Ferenchick, Gary
      – PersonEntity:
          Name:
            NameFull: Ownby, Allison
      – PersonEntity:
          Name:
            NameFull: Cutrer, William B.
      – PersonEntity:
          Name:
            NameFull: Obeso, Vivian
      – PersonEntity:
          Name:
            NameFull: Santen, Sally A.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Text: Jul2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 03080110
          Numbering:
            – Type: volume
              Value: 58
            – Type: issue
              Value: 7
          Titles:
            – TitleFull: Medical Education
              Type: main
ResultId 1