View in EDS HTML Full Text PDF Full Text

Validating the Effectiveness of a Locally Developed Assurance of Learning Assessment

Saved in:
Bibliographic Details
Title: Validating the Effectiveness of a Locally Developed Assurance of Learning Assessment
Language: English
Authors: Wray, Barry, Glew, David, Burrus, Robert, Hill, Stephen, Schuhmann, Peter
Source: Journal of Education for Business. 2020 95(4):216-226.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 11
Publication Date: 2020
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Business Schools, College Outcomes Assessment, Core Curriculum, Student Evaluation, Test Validity, Grades (Scholastic), Capstone Experiences, College Students, Predictive Validity
DOI: 10.1080/08832323.2019.1635976
ISSN: 0883-2323
Abstract: Standards for accreditation by the Association to Advance Collegiate Schools of Business emphasize the assessment of student learning as an avenue to continuously improve business school curricula. The study reported here, which took place in an Association to Advance Collegiate Schools of Business-accredited business school in a mid-sized regional university in the southeastern United States, is an in-depth review of an internally created assurance of learning assessment instrument designed to directly measure student learning across a set of core courses. Results of the study indicate that facets of the assessment are significantly related to students' grades in their previous core courses as well as to their performance in a later capstone course. This suggests that carefully developed, internally crafted assessments may be valid means to assess student learning.
Abstractor: As Provided
Entry Date: 2020
Accession Number: EJ1249901
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFRry_4jqDudvDUW9MMx61qAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDNYcTJz3UAKyntdALgIBEICBm_yeQWR8pexi_q0y23cbex44UMaZT7oPaiBlqhv3kMHPpO4QLd6RQbvFMC0h3iupMPaRaNApq2uubxz6WUFMuxZutZL7BctHlSYMyhpvIAXpc0d6DSkAqSIgVU_F_QxX7D519SuuPsjSgqO9IUUbU-r6OagLxi7hzgwwwIjeKh7pv1tXc_B6MlDL2w53lhscC7s3WXG8IRpfOHre
Text:
  Availability: 1
  Value: <anid>AN0142554751;jeb01may.20;2020Apr06.02:40;v2.2.500</anid> <title id="AN0142554751-1">Validating the effectiveness of a locally developed assurance of learning assessment </title> <p>Standards for accreditation by the Association to Advance Collegiate Schools of Business emphasize the assessment of student learning as an avenue to continuously improve business school curricula. The study reported here, which took place in an Association to Advance Collegiate Schools of Business–accredited business school in a mid-sized regional university in the southeastern United States, is an in-depth review of an internally created assurance of learning assessment instrument designed to directly measure student learning across a set of core courses. Results of the study indicate that facets of the assessment are significantly related to students' grades in their previous core courses as well as to their performance in a later capstone course. This suggests that carefully developed, internally crafted assessments may be valid means to assess student learning.</p> <p>Keywords: Accreditation; assessment; assurance of learning; capstone course</p> <p>Historically, institutions of higher education in the United States have considerable autonomy with regard to numerous aspects of their academic programs, including curriculum design, granting of academic credit, faculty credentials, course delivery, and assessment (U.S. Department of Education, [<reflink idref="bib16" id="ref1">16</reflink>]). The resulting variation in quality created a need for objective measures that could help students, employers, and lenders (including the federal government) assess the value of academic programs. Private accrediting agencies have filled this need for accountability by developing criteria to evaluate academic program quality and conducting evaluations to assess whether those criteria have been met. In addition to quality assurance, accreditation agencies serve to establish criteria for certifications and licensures, guide continuous improvement through the raising of standards, and integrate faculty and staff into institutional evaluation and planning (U.S. Department of Education, [<reflink idref="bib16" id="ref2">16</reflink>]).</p> <p>The Association to Advance Collegiate Schools of Business (AACSB) is an international accrediting agency that encourages business schools worldwide to pursue excellence and continuous improvement in their programs. An accreditation review is based, in part, on established standards related to learning and teaching. For example, AACSB's current Standard 8 addresses the school's assurance of learning (AoL) plan: "The school uses well-documented, systematic processes for determining and revising degree program learning goals; designing, delivering, and improving degree program curricula to achieve learning goals; and demonstrating that degree program learning goals have been met" (Association to Advance Collegiate Schools of Business, Business Accreditation Standards Worksheet, [<reflink idref="bib1" id="ref3">1</reflink>]).</p> <p>AACSB's learning standards have evolved over time. Zocco ([<reflink idref="bib17" id="ref4">17</reflink>]) summarized major revisions to the standards by explaining that prior to 1991, AACSB learning standards were based on a "Common Body of Knowledge." This body of knowledge was assessed largely through a review of a school's curriculum, discipline-specific contact hours, and allocation of resources. In 1991, revisions were adopted to more clearly link the school's learning goals with its mission and intended student outcomes. Acceptable assessment techniques included indirect measures of learning by surveying students, alumni, and employers. In 2003, the learning standards were further revised to emphasize the direct measurement of learning achievement rather than simply the measurement of the school's intent to provide learning, as reflected in the earlier standards.</p> <p>In 2013, AACSB's Standard 8, titled "Curricula Management and Assurance of Learning" was established to strengthen the link between a school's efforts to deliver its curriculum and to implement its AoL plan. The school is required to provide evidence of graduates' skills and content knowledge as well as the school's efforts to close any gaps between its learning goals and actual student performance. In addition, widespread faculty involvement in the process must be demonstrated (Abdelsamad, Farmer, McNeil, & Stevens, [<reflink idref="bib4" id="ref5">4</reflink>]).</p> <p>Although Standard 8 is more specific in its requirements than previous iterations of AACSB's learning goals, individual institutions retain the flexibility to determine the means by which skills and content knowledge are measured and assessed. Attaway, Chandra, Dos Santos, Thatcher, and Wright ([<reflink idref="bib5" id="ref6">5</reflink>]) noted that schools are encouraged to adopt learning measures that fit with their program goals, pedagogies, and circumstances. For instance, a school's faculty can create their own "local" tools for assessment or can rely on commercial assessment instruments provided by private sector assessment companies such as the Educational Testing Service. Standardized exams, such as the Educational Testing Service Major Field Test in Business, that are comprised of multiple choice questions are relatively easy to implement and allow for comparison of scores for recruiting purposes (Smith, Clements, & Olson, [<reflink idref="bib14" id="ref7">14</reflink>]). Similar to other standardized tests, the Major Field Test in Business demonstrates high reliability and is widely accepted as a legitimate measurement instrument (Frankforter & Guidry, [<reflink idref="bib7" id="ref8">7</reflink>]). However, such exams can be costly to implement and may not accurately reflect the curriculum at individual schools. More specifically, comparisons of a business school's own students with a national norm group of unknown characteristics may be problematic (Hogan, Maskulka, Kaminski, & Riley, [<reflink idref="bib9" id="ref9">9</reflink>]). Further, because they are not involved in question design, faculty may feel disconnected from standardized assessments and may find it difficult to interpret and use the results (Smith et al., [<reflink idref="bib14" id="ref10">14</reflink>]).</p> <p>In contrast, locally designed assessments can be linked directly to a school's mission and program goals (Gardiner, Corbitt, & Adams [<reflink idref="bib8" id="ref11">8</reflink>]), can be modified as program goals change, and because they require direct faculty involvement, may engender a sense of faculty ownership and commitment to the assessment process (Smith et al., [<reflink idref="bib14" id="ref12">14</reflink>]). Perhaps most importantly, locally constructed tests can potentially provide more valid measurement of progress toward learning goals because they are designed to assess more precisely the skills and content delivered to students by the school's own faculty.</p> <p>The purpose of the present study is to investigate the validity of a locally developed, proprietary AoL assessment instrument intended to measure student learning and guide curriculum development in the school's core business courses. Retention and application of the content delivered in functional business courses is a key issue in the assessment of student learning (Callahan, Strandholm, & Dziekan, [<reflink idref="bib6" id="ref13">6</reflink>]; Payne, Flynn, & Whitfield, [<reflink idref="bib11" id="ref14">11</reflink>]). To this end, we examine the tool we use to measure our student learning goal of mastery of discipline-specific knowledge.</p> <hd id="AN0142554751-2">Development of an AoL assessment instrument</hd> <p>When AACSB revised its accreditation standards to require the use of direct measures of student learning, our school's Assurance of Learning Committee discussed the possibility of adopting a commercially developed assessment tool to supplement our indirect measures of student outcomes. However, due to the anticipated downsides of standardized exams noted above (foremost among them being cost) the committee opted to develop an in-house instrument to measure students' mastery of discipline-specific content knowledge. To accomplish this, sets of multiple-choice questions covering basic concepts in each of the school's core business courses were created. Neither the individual sets of questions nor the assessment instrument overall were intended to be a comprehensive review of the school's core curriculum. Rather, the committee's recommendation was that the questions cover a sample of fundamental concepts that any student, having taken the course, would reasonably be expected to remember and apply.</p> <p>The school's core curriculum, which is completed by all students regardless of their selected concentration, consists of 12 courses (e.g., microeconomics and macroeconomics, financial and managerial accounting, principles of marketing, etc.). Through an iterative process of review and revision subcommittees of faculty representing each core course identified main concepts and prepared corresponding questions they felt should be part of the assessment. For example, the faculty who taught the principles of management course independently created sample questions that, when combined, resulted in a pool of about 30 items. A review of this pool revealed agreement that concepts of planning, organizing, leading, controlling, globalization, and communication were at the heart of the course. Questions reflecting these concepts were then refined by the faculty until they were ready for inclusion on the assessment. Ultimately, 100 four-item multiple-choice questions were selected to be part of the assessment instrument. Of these 100 questions, 72 were discipline-specific and 28 questions covered more general business concepts such as ethics, diversity, and social responsibility. The instrument is referred to locally as the AoL Content Knowledge Assessment, or more simply, the AoL assessment. We note that no pretest of the assessment is given, creating a situation where a student could enter a course having already acquired foundation knowledge. Thus, while we refer to the assessment as pertaining to AoL, the assessment can accurately be labeled an assessment of knowledge.</p> <p>The AoL assessment is administered online each semester using the school's learning management system (LMS). Students who have taken the business school's core courses and are registered to take the business capstone course in the following semester are required to take the assessment. The students may take the assessment at any time and from anywhere they choose, but several steps are taken to limit possible cheating. First, the assessment has a time limit of 90 min and must be completed in one sitting. No expectation is set that students will take the assessment completely closed-book, but given the breadth of concepts covered (i.e., 12 core courses), it would be difficult for a student to research each answer in the time provided. Second, three versions of each of the 100 questions were created. The LMS test feature randomly draws one of these versions for each question each time a student takes the assessment, and then sorts them in a random order. Because of this, the odds of two students sitting side by side on computers to take the same assessment are extremely low. Third, although students receive their score on the assessment immediately upon completion, they are not permitted to review the questions, thus reducing, but not eliminating, the risk that the questions are copied and shared with future students. Although students are required to take the assessment, their score is not included in any course grade. To incentivize completion of the assessment and students' good-faith effort to perform well, those who fail to take the assessment or who fail to earn a minimum score risk losing their registered seat in the capstone course the following semester. This incentive appears to be working well as response rates regularly approach 90%—a more than adequate sample for AoL purposes.</p> <hd id="AN0142554751-3">Study design</hd> <p>To examine the validity of the AoL assessment, we examine two relationships (see Figure 1). The first is the postdictive relationship between students' performance in discipline-specific courses and their score on the AoL assessment for discipline-specific questions. We anticipate when AoL assessment questions on a specific business core concept are designed well, they will reflect the level of students' performance in the corresponding business core course. For instance, if a student earned a strong grade in microeconomics, it would be logical to expect that they would perform well on assessment questions related to microeconomics. In addition, students tend to perform similarly in various contexts, so a student's performance in their core business classes is also expected to be reflected in their overall score on the AoL assessment.</p> <p>Graph: Figure 1. Two Dimensions of AoL assessment validity.</p> <p>Thus, our hypotheses:</p> <p>Hypothesis 1a (H1a): Students' AoL assessment subscores would be positively associated with their prior grades in corresponding core business courses.</p> <p>H1b: Students' overall AoL assessment scores would be positively associated with their prior grades in core business courses.</p> <p>The second association we examine is the predictive relationship between students' score on the AoL assessment and their performance in the business school capstone course. The purpose of the capstone course is to draw on the skills and knowledge students are expected to learn in the core business curriculum, and to help students gain an integrative perspective on how companies operate strategically across functional areas. As the AoL assessment is administered before the capstone course, it does not attempt to measure students' ability to integrate across functional areas. However, students must have the foundation of discipline-specific knowledge to succeed in the course. Therefore, we anticipate that the AoL assessment—intended to measure discipline-specific knowledge—will be a valid predictor of students' performance in the capstone course. Thus, our hypotheses:</p> <p>H2a: Students' AoL assessment subscores would be positively associated with their grades in the capstone business course.</p> <p>H2b: Students' overall AoL assessment scores would be positively associated with their grades in the capstone business course.</p> <hd id="AN0142554751-4">Data collection</hd> <p>The data used for the study consist of students' grades in their 12 core business courses, scores on the AoL assessment, and grades in their capstone course. Students' grades were provided by the University Registrar. All grades were measured on an ordinal scale: A, A–, B+, B, B–, etc. As this study focuses primarily on the assessment of discipline-specific learning, the 28 AoL assessment questions on the general topics of ethics, diversity, and social responsibility were omitted from further review. Using the remaining 72 questions, we calculated an overall AoL assessment score for each student as well as subscores reflecting their performance on the sets of questions from each of the 12 core courses (see Table 1 for course titles and abbreviations). In sum, for each student we had a preassessment course grade, a postassessment course grade, and 13 assessment scores (one overall score and 12 course-specific subscores).</p> <p>Table 1. Grade distribution in core business courses.</p> <p> <ephtml> <table><thead><tr><td>Core Course/Grade</td><td>A</td><td>A–</td><td>B+</td><td>B</td><td>B–</td><td>C+</td><td>C</td><td>C–</td><td>D+</td><td>D</td><td>D–</td><td>F</td><td>Other<xref ref-type="fn" rid="fn2" /></td></tr></thead><tbody valign="top"><tr><td>Course ACG-F <italic>Financial Accounting</italic></td><td char=".">16%</td><td char=".">4%</td><td char=".">5%</td><td char=".">20%</td><td char=".">6%</td><td char=".">6%</td><td char=".">11%</td><td char=".">3%</td><td char=".">1%</td><td char=".">2%</td><td char=".">0%</td><td char=".">1%</td><td char=".">24%</td></tr><tr><td>Course ACG-M <italic>Managerial Accounting</italic></td><td char=".">24%</td><td char=".">7%</td><td char=".">10%</td><td char=".">19%</td><td char=".">6%</td><td char=".">4%</td><td char=".">7%</td><td char=".">2%</td><td char=".">1%</td><td char=".">1%</td><td char=".">0%</td><td char=".">1%</td><td char=".">20%</td></tr><tr><td>Course BLA <italic>Business Law</italic></td><td char=".">33%</td><td char=".">9%</td><td char=".">12%</td><td char=".">26%</td><td char=".">5%</td><td char=".">3%</td><td char=".">7%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">5%</td></tr><tr><td>Course ECN-MI <italic>Microeconomics</italic></td><td char=".">18%</td><td char=".">9%</td><td char=".">12%</td><td char=".">13%</td><td char=".">7%</td><td char=".">5%</td><td char=".">5%</td><td char=".">4%</td><td char=".">2%</td><td char=".">1%</td><td char=".">1%</td><td char=".">1%</td><td char=".">24%</td></tr><tr><td>Course ECN-MA <italic>Macroeconomics</italic></td><td char=".">17%</td><td char=".">8%</td><td char=".">9%</td><td char=".">17%</td><td char=".">6%</td><td char=".">6%</td><td char=".">10%</td><td char=".">2%</td><td char=".">0%</td><td char=".">2%</td><td char=".">0%</td><td char=".">0%</td><td char=".">24%</td></tr><tr><td>Course FIN <italic>Finance</italic></td><td char=".">25%</td><td char=".">9%</td><td char=".">12%</td><td char=".">20%</td><td char=".">9%</td><td char=".">7%</td><td char=".">8%</td><td char=".">1%</td><td char=".">0%</td><td char=".">2%</td><td char=".">0%</td><td char=".">1%</td><td char=".">7%</td></tr><tr><td>Course INB <italic>International Business</italic></td><td char=".">29%</td><td char=".">13%</td><td char=".">16%</td><td char=".">20%</td><td char=".">7%</td><td char=".">4%</td><td char=".">2%</td><td char=".">0%</td><td char=".">0%</td><td char=".">1%</td><td char=".">0%</td><td char=".">0%</td><td char=".">7%</td></tr><tr><td>Course MGT <italic>Management</italic></td><td char=".">26%</td><td char=".">10%</td><td char=".">11%</td><td char=".">30%</td><td char=".">5%</td><td char=".">3%</td><td char=".">4%</td><td char=".">1%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">9%</td></tr><tr><td>Course MIS <italic>Management Information Systems</italic></td><td char=".">45%</td><td char=".">13%</td><td char=".">11%</td><td char=".">15%</td><td char=".">5%</td><td char=".">1%</td><td char=".">2%</td><td char=".">1%</td><td char=".">1%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">7%</td></tr><tr><td>Course MKT <italic>Marketing</italic></td><td char=".">30%</td><td char=".">17%</td><td char=".">13%</td><td char=".">20%</td><td char=".">5%</td><td char=".">3%</td><td char=".">3%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">9%</td></tr><tr><td>Course OPS <italic>Operations Management</italic></td><td char=".">43%</td><td char=".">10%</td><td char=".">14%</td><td char=".">16%</td><td char=".">6%</td><td char=".">3%</td><td char=".">3%</td><td char=".">1%</td><td char=".">0%</td><td char=".">1%</td><td char=".">0%</td><td char=".">0%</td><td char=".">4%</td></tr><tr><td>Course QMM <italic>Quantitative Methods Management</italic></td><td char=".">20%</td><td char=".">7%</td><td char=".">9%</td><td char=".">12%</td><td char=".">5%</td><td char=".">5%</td><td char=".">7%</td><td char=".">3%</td><td char=".">1%</td><td char=".">1%</td><td char=".">0%</td><td char=".">1%</td><td char=".">29%</td></tr><tr><td>Course CAP <italic>Management Capstone</italic></td><td char=".">19%</td><td char=".">24%</td><td char=".">13%</td><td char=".">27%</td><td char=".">8%</td><td char=".">3%</td><td char=".">5%</td><td char=".">1%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td><td char=".">0%</td></tr></tbody></table> </ephtml> </p> <p>Data were gathered from five semesters of AoL assessments resulting in observations for 1,025 students. After excluding those observations with missing variables, a total of 775 usable records remained for analysis. A close inspection of the overall AoL assessment score revealed a bimodal distribution, with scores falling into one of two groups: 0–40% correct or 40–100% correct (Figure 2). Although both sets of scores are approximately normally distributed, we focused our analysis on scores above 40 and omitted the other 28 observations under the assumption that these especially low-scoring students lacked the motivation to be fully engaged in the assessment process. The resulting distribution of the final sample of 747 observations is shown in Figure 3.</p> <p>PHOTO (COLOR): Figure 2. Distribution of all AoL assessment scores (n = 775).</p> <p>PHOTO (COLOR): Figure 3. Distribution of AoL assessment scores above 40.</p> <p>Descriptive statistics for the variables used in this study are shown in Tables 1–2 (the names of the 12 core courses are listed in Table 1). In addition to the measured variables, overall student score on the AoL assessment was divided into four categories to facilitate some of the statistical analyses. This classification is shown in Figure 4.</p> <p>PHOTO (COLOR): Figure 4. Distribution of overall AoL assessment score classification.</p> <p>Table 2. Descriptive statistics for overall AoL assessment and subscores.</p> <p> <ephtml> <table><thead><tr><td>Description</td><td>Min (%)</td><td>Max (%)</td><td>Mean (%)</td></tr></thead><tbody valign="top"><tr><td>AoL assessment (overall score)</td><td char=".">43</td><td char=".">92</td><td char=".">69.9</td></tr><tr><td>AoL ACG-F (financial)</td><td char=".">0</td><td char=".">100</td><td char=".">73.3</td></tr><tr><td>AoL ACG-M (managerial)</td><td char=".">17</td><td char=".">100</td><td char=".">85.4</td></tr><tr><td>AoL BLA</td><td char=".">0</td><td char=".">100</td><td char=".">69.4</td></tr><tr><td>AoL ECN-MI (micro)</td><td char=".">0</td><td char=".">100</td><td char=".">58.5</td></tr><tr><td>AoL ECN-MA (macro)</td><td char=".">0</td><td char=".">100</td><td char=".">69.2</td></tr><tr><td>AoL FIN</td><td char=".">0</td><td char=".">100</td><td char=".">62.6</td></tr><tr><td>AoL INB</td><td char=".">17</td><td char=".">100</td><td char=".">76.8</td></tr><tr><td>AoL MGT</td><td char=".">17</td><td char=".">100</td><td char=".">78.7</td></tr><tr><td>AoL MIS</td><td char=".">0</td><td char=".">100</td><td char=".">66.8</td></tr><tr><td>AoL MKT</td><td char=".">17</td><td char=".">100</td><td char=".">70.9</td></tr><tr><td>AoL OPS</td><td char=".">0</td><td char=".">100</td><td char=".">56.5</td></tr><tr><td>AoL QMM</td><td char=".">17</td><td char=".">100</td><td char=".">69.9</td></tr></tbody></table> </ephtml> </p> <hd id="AN0142554751-5">Method</hd> <p>A first step in our analysis was to address the possibility of multicollinearity among the AoL assessment subscores. For instance, it would not be surprising if a student's subscore on questions related to microeconomics (AoL ECN-MI) were correlated with his or her subscore on questions related to macroeconomics (AoL ECN-MA). A multivariate correlation analysis on the AoL assessment subscores (performed using JMP Pro 12 [SAS Institute, Cary, NC]) revealed that most of the AoL assessment subscores are highly positively correlated with each other (<emph>p</emph> <.05). The results are not surprising because individual students often perform similarly across core courses. To investigate the potential for multicollinearity, variance inflation factors were generated using an ordinary least squares regression model with the overall AoL assessment score as the response variable and all subscores as covariates. As shown in Table 3, all variance inflation factors are less than 2.0, which suggests that multicollinearity is not expected to influence the results of the analyses performed in this study.</p> <p>Table 3. Collinearity VIF and parameter estimates among AoL assessment subscores.</p> <p> <ephtml> <table><thead><tr><td>AoL subscore category</td><td>VIF</td></tr></thead><tbody valign="top"><tr><td>AoL ACG-F</td><td char=".">1.11</td></tr><tr><td>AoL ACG-M</td><td char=".">1.08</td></tr><tr><td>AoL BLA</td><td char=".">1.07</td></tr><tr><td>AoL ECN-MI</td><td char=".">1.13</td></tr><tr><td>AoL ECN-MA</td><td char=".">1.09</td></tr><tr><td>AoL FIN</td><td char=".">1.04</td></tr><tr><td>AoL INB</td><td char=".">1.06</td></tr><tr><td>AoL MGT</td><td char=".">1.10</td></tr><tr><td>AoL MIS</td><td char=".">1.05</td></tr><tr><td>AoL MKT</td><td char=".">1.12</td></tr><tr><td>AoL OPS</td><td char=".">1.18</td></tr><tr><td>AoL QMM</td><td char=".">1.05</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note.</emph> VIF = variance inflation factor.</p> <p>We also expect that student performance in the 12 core business courses would be highly related. To test these relationships, two-way likelihood ratio tests were performed on all possible pairs of business core classes. All pairwise tests were significant at α =.05 with the exception of the relationship between the international business (INB) course and the quantitative methods management (QMM) course (<emph>p</emph> =.16). These results suggest that grades in business school core courses are strongly related. In general, good students perform well and poor students perform poorly in their core business school courses.</p> <hd id="AN0142554751-6">Postdictive validity analyses</hd> <p>Three methods were used to examine the relationship between students' score on the AoL assessment and their performance in their corresponding core business courses. First, 12 linear regression models were developed to test the hypothesis (H1a). The independent variable in each of the 12 models is the grade for one of the 12 core business courses and the dependent variable is the subscore from the AoL assessment for the corresponding core business topic. For each of the 12 models the predictor variable "grade" is a multilevel ordinal variable. Throughout this research, effect coding is used for all categorical predictor variables. The JMP Pro "Effect Summary" table is utilized for test results for whole model effects for predictor variables. For more information on how JMP Pro handles coding for categorical variables, see SAS Institute ([<reflink idref="bib13" id="ref15">13</reflink>].).</p> <p>To demonstrate this, consider a model with a student's grade in their microeconomics course as the predictor variable and the student's microeconomics subscore on the AoL test as the dependent variable. The analysis of variance table for this model is given in Table 4 and the effects summary table is given in Table 5. The overall <emph>F</emph> test shows a significant result. The effects summary also shows the same significant relationship. The parameter estimates in Table 6 show a student's subscore on the AoL microeconomics questions is significantly impacted if a student received a grade of D– in their microeconomics course. This can be seen in the prediction expression in Figure 5 by examining all the coefficients for the different grade levels. The value of –30.36 for a grade of D– will result is a much lower predicted subscore than the prediction for any other grade. For instance, if a student received a C in their micro course the prediction for their subscore on the AoL microeconomics set of questions would be 63.615—9.59 = 54.025; if they received a D–, the prediction would be 63.615—30.36 = 33.255. Recall that the test for this effect is significant in the effects summary in Table 5. Therefore, it can be concluded there is a relationship between a student's performance (grade) in their microeconomics course and their subscore on the AoL instrument questions pertaining to microeconomics.</p> <p>Graph: Figure 5. Estimated regression model for AoL ECN-MI regressed on course ECN-MI.</p> <p>Table 4. Analysis of variance for AoL ECN-MI regressed on course ECN-MI.</p> <p> <ephtml> <table><thead><tr><td>Source</td><td><italic>df</italic></td><td>Sum of squares</td><td>Mean square</td><td><italic>F</italic> ratio</td></tr></thead><tbody valign="top"><tr><td>Model</td><td char=".">12</td><td char=".">10663.74</td><td char=".">888.645</td><td>2.2937</td></tr><tr><td>Error</td><td char=".">734</td><td char=".">284374.14</td><td char=".">387.431</td><td>Prob > <italic>F</italic></td></tr><tr><td>Total</td><td char=".">746</td><td char=".">295037.88</td><td /><td>.00722</td></tr></tbody></table> </ephtml> </p> <p>Table 5. Effect summary for AoL ECN-MI regressed on course ECN-MI.</p> <p> <ephtml> <table><thead><tr><td>Source</td><td>LogWorth</td><td><italic>p</italic> Value</td></tr></thead><tbody valign="top"><tr><td>Course ECN-MI</td><td char=".">2.141</td><td char=".">.00722</td></tr></tbody></table> </ephtml> </p> <p>Table 6. Parameter estimates for AoL ECN-MI Regressed on Course ECN-MI.</p> <p> <ephtml> <table><thead><tr><td>Term</td><td>Estimate</td><td><italic>SE</italic></td><td><italic>t</italic> ratio</td><td>Prob > |<italic>t</italic>|</td></tr></thead><tbody valign="top"><tr><td>Intercept</td><td char=".">63.615</td><td char=".">1.694</td><td char=".">37.55</td><td char="."><.0001</td></tr><tr><td>Course ECN-MI[A– to A]</td><td char=".">−2.572</td><td char=".">2.927</td><td char=".">−0.88</td><td char=".">.380</td></tr><tr><td>Course ECN-MI[B + to A–]</td><td char=".">−1.251</td><td char=".">3.186</td><td char=".">−0.39</td><td char=".">.695</td></tr><tr><td>Course ECN-MI[B to B+]</td><td char=".">−2.096</td><td char=".">2.893</td><td char=".">−0.72</td><td char=".">.469</td></tr><tr><td>Course ECN-MI[B– to B]</td><td char=".">−3.357</td><td char=".">3.415</td><td char=".">−0.98</td><td char=".">.326</td></tr><tr><td>Course ECN-MI[C + to B–]</td><td char=".">−1.899</td><td char=".">4.375</td><td char=".">−0.43</td><td char=".">.664</td></tr><tr><td>Course ECN-MI[C to C+]</td><td char=".">1.583</td><td char=".">4.566</td><td char=".">0.35</td><td char=".">.729</td></tr><tr><td>Course ECN-MI[C– to C]</td><td char=".">3.383</td><td char=".">4.878</td><td char=".">0.69</td><td char=".">.488</td></tr><tr><td>Course ECN-MI[D + to C–]</td><td char=".">−2.330</td><td char=".">6.645</td><td char=".">−0.35</td><td char=".">.726</td></tr><tr><td>Course ECN-MI[D to D+]</td><td char=".">7.923</td><td char=".">8.535</td><td char=".">0.93</td><td char=".">.354</td></tr><tr><td>Course ECN-MI[D– to D]</td><td char=".">−29.75</td><td char=".">11.828</td><td char=".">−2.52</td><td char=".">.012</td></tr><tr><td>Course ECN-MI[F to D–]</td><td char=".">33.35</td><td char=".">13.204</td><td char=".">2.53</td><td char=".">.012</td></tr><tr><td>Course ECN-MI[Other-F]</td><td char=".">−9.417</td><td char=".">8.927</td><td char=".">−1.05</td><td char=".">.292</td></tr></tbody></table> </ephtml> </p> <p>A similar process is used to build models for each core business topic. That is, the analysis summarized above for microeconomics was conducted for all 12 core business course topics.[<reflink idref="bib1" id="ref16">1</reflink>] The measures of overall significance for the 12 <emph>F</emph> tests for the regression models are shown in Table 7. Nine of the 12 models indicate a students' grade in a prior core business course is significantly associated with their score on the corresponding subset of core business topic questions on the AoL assessment test, providing partial support for H1a.</p> <p>Table 7. Simple ordinary least squares regression of business core course grades regressed on AoL assessment subscores.</p> <p> <ephtml> <table><thead><tr><td>AoL assessment subscore category</td><td><italic>p</italic> value</td></tr></thead><tbody valign="top"><tr><td>ACG-F</td><td char="."><.0001</td></tr><tr><td>ACG-M</td><td char=".">.0471</td></tr><tr><td>BLA</td><td char=".">.2184</td></tr><tr><td>ECN-MI</td><td char=".">.0072</td></tr><tr><td>ECN-MA</td><td char="."><.0001</td></tr><tr><td>FIN</td><td char=".">.0475</td></tr><tr><td>INB</td><td char=".">.0005</td></tr><tr><td>MGT</td><td char=".">.0294</td></tr><tr><td>MIS</td><td char=".">.1177</td></tr><tr><td>MKT</td><td char=".">.1083</td></tr><tr><td>OPS</td><td char=".">.0050</td></tr><tr><td>QMM</td><td char=".">.0133</td></tr></tbody></table> </ephtml> </p> <p>Twelve ordinary least squares regression models were also developed to test the second hypothesis regarding the relationship between students' grades in their prior core business courses and their overall AoL assessment score (H1b). Each model utilized a student's core course grade as the independent variable and the overall student score on their AoL assessment test as the dependent variable. The significance levels for the 12 <emph>F</emph> tests are shown in Table 8. All relationships except for the management information systems (MIS) course (<emph>p</emph> =.2129) and the QMM course (<emph>p</emph> =.0629) are significant at the α =.05 level, providing robust support for H1b.</p> <p>Table 8. Ordinary least squares model of AoL assessment test overall score regressed on core business course grade.</p> <p> <ephtml> <table><thead><tr><td>Business core course</td><td><italic>p</italic> value</td></tr></thead><tbody valign="top"><tr><td>ACG-F</td><td char="."><.0001</td></tr><tr><td>ACG-M</td><td char="."><.0001</td></tr><tr><td>BLA</td><td char="."><.0001</td></tr><tr><td>ECN-MI</td><td char=".">.0009</td></tr><tr><td>ECN-MA</td><td char="."><.0001</td></tr><tr><td>FIN</td><td char="."><.0001</td></tr><tr><td>INB</td><td char="."><.0001</td></tr><tr><td>MGT</td><td char=".">.0072</td></tr><tr><td>MIS</td><td char=".">.2129</td></tr><tr><td>MKT</td><td char=".">.0002</td></tr><tr><td>OPS</td><td char=".">.0002</td></tr><tr><td>QMM</td><td char=".">.0629</td></tr></tbody></table> </ephtml> </p> <p>To further investigate the relationship between prior core course grades and overall score on the AoL assessment, a single multivariate regression model was developed. The independent variables consisted of students' grades from all 12 core business courses and the dependent variable consisted of their AoL assessment test score. The results of this analysis are shown in Table 9. The overall model <emph>F</emph> test resulted in a <emph>p</emph> value <.0001 in support of H1b.</p> <p>Table 9. Model fit results for AoL assessment test score regressed on core business course grades.</p> <p> <ephtml> <table><thead><tr><td>Source</td><td><italic>df</italic></td><td>Sum of squares</td><td>Mean square</td><td><italic>F</italic> ratio</td></tr></thead><tbody valign="top"><tr><td>Model</td><td char=".">132</td><td char=".">14491.356</td><td char=".">109.783</td><td char=".">1.8687</td></tr><tr><td>Error</td><td char=".">614</td><td char=".">36071.787</td><td char=".">58.749</td><td>Prob > <italic>F</italic></td></tr><tr><td>Total</td><td char=".">746</td><td char=".">50563.143</td><td /><td char="."><.0001</td></tr></tbody></table> </ephtml> </p> <p>Notably, the effect summary in Table 10 shows <emph>p</emph> values for the significant independent variables. The core course subgrades for finance, ECN-MA, and business law were important predictors of overall AoL assessment score. However, given the high level of interrelationships among student performance in the core business courses, the significance of other course subgrades on overall AoL score cannot be discarded. The highly significant <emph>p</emph> value for the overall <emph>F</emph> test again supports H1b.</p> <p>Table 10. Effect summary for AoL assessment test overall score regressed on core business course grades.</p> <p> <ephtml> <table><thead><tr><td>Source</td><td>Log worth</td><td><italic>p</italic></td></tr></thead><tbody valign="top"><tr><td>Course FIN</td><td char=".">1.711</td><td char=".">.019</td></tr><tr><td>Course ECN-MA</td><td char=".">1.259</td><td char=".">.055</td></tr><tr><td>Course BLA</td><td char=".">1.179</td><td char=".">.066</td></tr><tr><td>Course QMM</td><td char=".">0.629</td><td char=".">.235</td></tr><tr><td>Course ACG-M</td><td char=".">0.302</td><td char=".">.499</td></tr><tr><td>Course INB</td><td char=".">0.296</td><td char=".">.505</td></tr><tr><td>Course MGT</td><td char=".">0.185</td><td char=".">.654</td></tr><tr><td>Course MKT</td><td char=".">0.128</td><td char=".">.744</td></tr><tr><td>Course MIS</td><td char=".">0.124</td><td char=".">.751</td></tr><tr><td>Course ECN-MI</td><td char=".">0.109</td><td char=".">.777</td></tr><tr><td>Course ACG-F</td><td char=".">0.071</td><td char=".">.850</td></tr><tr><td>Course OPS</td><td char=".">0.008</td><td char=".">.982</td></tr></tbody></table> </ephtml> </p> <hd id="AN0142554751-7">Predictive validity analyses</hd> <p>Having provided strong evidence of the postdictive validity of the AoL assessment, we now turn to examining its ability to predict students' performance in their capstone course. Three methods are used to examine the relationship between students' scores on the AoL assessment and their performance in the capstone business course. First, to test the hypothesis (H2a) 12 ordinal logistic regression models are developed. The independent variable for each model is an AoL assessment subscore related to a core business topic, and the dependent variable for each model is the capstone course grade. The results for the 12 whole model <emph>F</emph> tests are given in Table 11. All <emph>p</emph> values less than.05 for nine of the 12 models provide strong evidence of a relationship between students' AoL assessment subscores and their grades in the capstone course, thus providing support for H2a. Assessment subscores for the MIS course are marginally significant, while those for QMM and INB are insignificant.</p> <p>Table 11. Summary of ordinal logistic regression models of capstone course grades regressed on AoL assessment subscores.</p> <p> <ephtml> <table><thead><tr><td>AoL assessment subscore category</td><td><italic>p</italic> value</td></tr></thead><tbody valign="top"><tr><td>ACG-F</td><td char=".">.0023</td></tr><tr><td>ACG-M</td><td char=".">.0035</td></tr><tr><td>BLA</td><td char=".">.0019</td></tr><tr><td>ECN-MI</td><td char="."><.0001</td></tr><tr><td>ECN-MA</td><td char=".">.0021</td></tr><tr><td>FIN</td><td char=".">.0002</td></tr><tr><td>INB</td><td char=".">.3126</td></tr><tr><td>MGT</td><td char=".">.0005</td></tr><tr><td>MIS</td><td char=".">.0967</td></tr><tr><td>MKT</td><td char="."><.0001</td></tr><tr><td>OPS</td><td char="."><.0001</td></tr><tr><td>QMM</td><td char=".">.9073</td></tr></tbody></table> </ephtml> </p> <p>To further investigate the relationship between AoL assessment subscores on capstone course grades, a forward stepwise multivariate ordinal logistic regression model was developed, again using a <emph>p</emph>-value criteria (.05) for variable inclusion. The independent variables consisted of students' AoL assessment subscores from all 12 core business areas and the dependent variable consisted of their capstone course grade. The results of this analysis are given in Table 12. The overall model <emph>F</emph> test resulted in a highly significant <emph>p</emph> value <.0001 providing strong support for H2a.</p> <p>Table 12. Model fit summary for stepwise multivariate ordinal logistic regression, capstone course grades regressed on AoL assessment subscores.</p> <p> <ephtml> <table><thead><tr><td>Model</td><td>–Log likelihood</td><td><italic>df</italic></td><td>χ<sup>2</sup></td><td>Prob > χ<sup>2</sup></td></tr></thead><tbody valign="top"><tr><td>Difference</td><td char=".">26.3339</td><td char=".">5</td><td char=".">52.66771</td><td char="."><.0001</td></tr><tr><td>Full</td><td char=".">1233.6943</td><td /><td /><td /></tr><tr><td>Reduced</td><td char=".">1260.0282</td><td /><td /><td /></tr></tbody></table> </ephtml> </p> <p>The effect summary in Table 13 shows the selected independent variables from the stepwise approach. The AoL subscores for finance, marketing, ECN-MI, management, and operations management were selected as important predictors of capstone course grade. However, given the high level of multicollinearity of the predictive abilities of other independent AoL core topic subscores cannot be discarded. The highly significant <emph>p</emph> value for the overall <emph>F</emph> test again supports H2a.</p> <p>Table 13. Effect summary for ordinal logistic regression, capstone course grade regressed on AoL assessment subscores.</p> <p> <ephtml> <table><thead><tr><td>Source</td><td>Log worth</td><td><italic>p</italic> value</td></tr></thead><tbody valign="top"><tr><td>AoL FIN</td><td char=".">2.751</td><td char=".">.002</td></tr><tr><td>AoL MKT</td><td char=".">2.507</td><td char=".">.003</td></tr><tr><td>AoL ECN-MI</td><td char=".">1.902</td><td char=".">.013</td></tr><tr><td>AoL MGT</td><td char=".">1.506</td><td char=".">.031</td></tr><tr><td>AoL OPS</td><td char=".">1.347</td><td char=".">.045</td></tr></tbody></table> </ephtml> </p> <p>Next, we test hypothesis H2b, investigating whether students' overall score on the AoL assessment is associated with their performance in their capstone business course. A simple overview of this relationship is provided by calculating the average overall AoL assessment score for the top five assigned grade categories in the course. Figure 6 shows a visualization that provides rudimentary support for H2b: Students with higher overall AoL assessment scores tend to receive higher capstone course grades.</p> <p>PHOTO (COLOR): Figure 6. Box-and-whisker plots of overall AoL assessment scores by capstone course grade.</p> <p>We also test H2b, pertaining to the relationship between a student's overall AoL assessment score and their capstone course grade, using an ordinal logistic model. Model fit results are shown in Table 14. The <emph>p</emph> value <.0001 shows strong support for H2b. Students' overall score on the AoL assessment serves as a significant predictor of future performance in the business capstone course.</p> <p>Table 14. Model fit results for ordinal logistic regression, capstone course grade regressed on overall AoL assessment score.</p> <p> <ephtml> <table><thead><tr><td>Model</td><td>–Log likelihood</td><td><italic>df</italic></td><td>χ<sup>2</sup></td><td>Prob > χ<sup>2</sup></td></tr></thead><tbody valign="top"><tr><td>Difference</td><td char=".">23.424</td><td char=".">1</td><td char=".">46.848</td><td char="."><.0001</td></tr><tr><td>Full</td><td char=".">1236.604</td><td /><td /><td /></tr><tr><td>Reduced</td><td char=".">1260.028</td><td /><td /><td /></tr></tbody></table> </ephtml> </p> <hd id="AN0142554751-8">Discussion</hd> <p>Six statistical methods were used to validate a local assessment for student leaning of business school core concepts for AoL purposes. Empirical examinations of the relationship between students' past performance in individual business core courses and their subscores on an AoL assessment suggest that student performance on questions related to certain core concepts on the AoL assessment was significantly related to their performance in the associated core course. We also find strong evidence that student performance in business school core courses is related to their overall score on the AoL assessment.</p> <p>Regarding the relationship between students' subscores on the AoL assessment and their performance in the business school capstone course, results suggest a statistically meaningful relationship between student subscores on the core concepts in the AoL assessment and performance in the capstone business course. We also found a strong relationship between the overall composite AoL test score and student performance in the capstone course.</p> <p>All six statistical methodologies provide support for the validity of the proprietary AoL assessment. Further, valuable insight can be found in the few insignificant relationships that were discovered. For example, in the context of H1a, there are several possible explanations why student grades in the business law, MIS, and marketing courses were not significant predictors (α =.05) of student score on their respective AoL core concepts test questions. Each set of questions for each core concept in the AoL test was developed by faculty teaching the core topic and each student received a sample of 6 questions from a pool of 18 for each of the 12 core concepts. The insignificant results for these three courses could be due to the nature of how the AoL questions were developed. If the AoL questions are too easy, then students who performed poorly in the core course may still do well on the AoL questions. Likewise, if the AoL questions are too difficult, then students who performed well in the core course may still not do well on the harder questions. It is also possible that questions developed for the AoL test were not closely aligned to the concepts taught in those core courses. A more thorough investigation of these results is left for future research.</p> <p>Results for H1b suggest a strong positive relationship between students' performance in their core courses (with the exception of insignificant result for MIS) and their overall score on the AoL test. This suggests the AoL test is capturing a similar type of student learning as in the core courses because students who performed well (poorly) in their core courses also performed well (poorly) on the AoL assessment. This provides evidence of the validity of the AoL test's ability to capture student learning. With regard to insignificant results, while there are several possible reasons the MIS course may be a poor predictor of score on the AoL test, the most obvious of which pertains to course content. The MIS course is a highly technical course concerned with technology and computer skills with minimal focus on traditional business areas such as general management, marketing and accounting. As well, MIS is taken early in the business curriculum; thus, there is more time to "lose" the material covered in the course.</p> <p>The primary conclusion that can be drawn from the tests of H2a is that students' subscore on the AoL questions (on most core topics) provides significant predictive ability for capstone course grades. However, the insignificant results for AoL questions related to INB, MIS, and QMM suggests that student scores on the AoL questions for these subjects are not related to student performance in the capstone management strategy class. Again, the insignificant results for these three disciplines could be due to the nature of how the AoL questions were developed. It is also possible that questions developed for the AoL test for some core topics were not closely aligned to the concepts taught in the capstone course.</p> <p>It is worth noting that the insignificance of the AoL questions related to QMM suggests that a student's subscore on the quantitative questions pertaining to statistics on the AoL assessment test is not indicative of his or her performance in the more qualitative management capstone class. A possible explanation is the drastic difference in cognitive styles of learning and the mathematical rigor for the two courses. While much has been written on this topic (e.g., Loo, [<reflink idref="bib10" id="ref17">10</reflink>]; Strang, [<reflink idref="bib15" id="ref18">15</reflink>]), analysis of the implications of differences in cognitive learning styles for assessment in business curricula, and the relationship between performance in technical or quantitative courses and overall score on AoL assessments, appears to be an important area for future research.</p> <hd id="AN0142554751-9">Conclusions</hd> <p>The research outlined in this article provides evidence of validity for a faculty-created multiple-choice AoL assessment that has been applied at a mid-sized regional university in the southeastern United States for approximately eight years. It extends the work of others, such as Callahan et al. ([<reflink idref="bib6" id="ref19">6</reflink>]) and Phelps and Spangler ([<reflink idref="bib12" id="ref20">12</reflink>]), who created and measured the efficacy of locally developed assessments. Specifically, based on a variety of empirical approaches we find compelling evidence for both a postdictive association between student performance on the AoL assessment and their performance in business school core classes, and a predictive association between performance on the AoL assessment and student outcomes in the business capstone course. These findings lend support to the notion that business school faculty can develop effective in-house assessments for the purposes of accreditation thus avoiding the problems and expense associated with standardized tests. However, the relationship between course performance and overall content assessment may not apply to some business core courses and some assessment topic areas, particularly those in more technical or quantitative disciplines (particularly when these courses are taken early in the business curricula). In addition, content question sets that are too easy or too difficult erode the ability to assess the efficacy of true learning in a particular course. These exceptions suggest a need for future research as well as continuous improvement in the AoL assessment process.</p> <p>Consistent with AACSB's objective in instituting AoL as part of the accreditation process, results from the AoL assessment have driven tangible changes in the curriculum. As noted earlier, an internally developed instrument offers the advantage of a precise connection between what is taught and what is measured. In the present school of business, faculty are provided with summary results from the assessment as well as detailed results for each individual question. Where a lack of understanding is suspected (i.e., a low percentage of students answering a question correctly) additional instruction in that area can be provided. For example, results from the assessment recently suggested the need to expand the discussion of price ceilings and floors in the microeconomics course and the faculty made plans to improve coverage of these areas. The data analyzed here, especially those that failed to support the hypothesized relationships, suggest that other changes to the curriculum may be beneficial.</p> <p>To be sure, the AoL assessment instrument used in the present research is somewhat different than the one originally developed by the business school's faculty eight years ago. In addition to the assessment results driving curriculum changes, the assessment instrument itself has undergone numerous changes. Some of these changes have been simple refinements to the questions (e.g., clarifying the question stem, revising the distractors) to improve the measurement of student learning. Other changes have been to occasionally introduce new questions to reflect externally driven changes to the curriculum. For instance, cybersecurity was not nearly as central to an information systems course a decade ago as it is today, so the assessment has been altered to capture this content. Although the assessment used over the period of the research presented here was essentially static, the ability to make these sorts of improvements to the instrument is another key advantage of developing and maintaining a homegrown assessment.</p> <p>We end this article with a brief comment on assessment itself. Retention and application of course content is a key issue in the assessment of student learning. However, it only represents half of an actual learning process. In accordance with AACSB standards (Association to Advance Collegiate Schools of Business, [<reflink idref="bib3" id="ref21">3</reflink>]), our assessment process, including our assessment instrument, measures only whether students have achieved learning expectations. Although we believe that students acquired this knowledge as a result of our business programs, the data presented here cannot empirically support this claim. AACSB recognizes a pretest-posttest research design can be a valuable part of an AoL system, however there is no requirement for multiple assessments of the same content at different points in a program (Association to Advance Collegiate Schools of Business, [<reflink idref="bib2" id="ref22">2</reflink>]). In sum, our internally developed assessment instrument offers many advantages, but it certainly should not distract from the true objective of enhancing student learning.</p> <p>We hope that other business schools will benefit from our experience as they consider their own methods of meeting accreditation standards and assessing student learning. Clearly, our process shows that faculty generally agree on important learning outcomes and can agree on a limited number of questions to assess those outcomes for a particular core class. As long as faculty view the process as a means of improving student learning (and not just an exercise to appease accrediting bodies or an exercise to "expose" faulty teaching in a particular department), questions that are too easy will be avoided. Faculty should also strive to be reasonable in what they logically believe that students must retain from a core class that may not be instrumental in their concentration areas—avoiding questions that are too difficult. Any business school considering the development of an assessment tool should also provide ample opportunities for the faculty to adjust or remove questions to better support changing learning needs.</p> <ref id="AN0142554751-10"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref3" type="bt">1</bibl> <bibtext> Results available from the authors upon request.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref22" type="bt">2</bibl> <bibtext> Other is a designation for anything other than an A–F grade (i.e., Incomplete, Transfer, Withdrew, etc.).</bibtext> </blist> <blist> <bibl id="bib3" idref="ref21" type="bt">3</bibl> <bibtext> Color versions of one or more of the figures in the article can be found online at <ulink href="http://www.tandfonline.com/vjeb">www.tandfonline.com/vjeb</ulink>.</bibtext> </blist> </ref> <ref id="AN0142554751-11"> <title> References </title> <blist> <bibtext> AACSB, Business Accreditation Standards Worksheet. (n.d.). Retrieved from https://<ulink href="http://www.aacsb.edu/accreditation/journey/business/initial">www.aacsb.edu/accreditation/journey/business/initial</ulink></bibtext> </blist> <blist> <bibtext> Association to Advance Collegiate Schools of Business. (2017). AACSB Assurance of Learning Standards: An Interpretation. An AACSB White Paper issued by: AACSB International Accreditation Coordinating Committee AACSB International Accreditation Quality Committee. Retrieved from https://naspaaaccreditation.files.wordpress.com/2014/04/aacsb.pdf</bibtext> </blist> <blist> <bibtext> Association to Advance Collegiate Schools of Business. (2019). Accreditation Standard 8 (2013 Business Standards): Curricula Management and Assurance of Learning: An Interpretation. Retrieved from https://<ulink href="http://www.aacsb.edu/-/media/aacsb/publications/white-papers/aol%5fwhite%5fpaper%5fstandard%5f8.ashx?la=en&hash=9DF96D5B2A1E0B02E19ACF6984FFFE07D23D45CA">www.aacsb.edu/-/media/aacsb/publications/white-papers/aol%5fwhite%5fpaper%5fstandard%5f8.ashx?la=en&hash=9DF96D5B2A1E0B02E19ACF6984FFFE07D23D45CA</ulink></bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> Abdelsamad, M. H., Farmer, B. A., McNeil, R., & Stevens, G. E. (2015). Major changes in AACSB standards (2003 compared to 2013). Advanced Management Journal, 80 (3), 4 – 11.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref6" type="bt">5</bibl> <bibtext> Attaway, A. N., Chandra, S., Dos Santos, B. L., Thatcher, M. E., & Wright, A. L. (2011). An approach to meeting AACSB assurance of learning standards in an IS core course. Journal of Information Systems Education, 22 (4), 355 – 366.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref13" type="bt">6</bibl> <bibtext> Callahan, T. J., Strandholm, K., & Dziekan, J. (2009). Developing an undergraduate assessment test: A mechanism for faculty feedback about retention. Journal of Education for Business, 85 (1), 45 – 49. doi: 10.1080/08832320903217705</bibtext> </blist> <blist> <bibl id="bib7" idref="ref8" type="bt">7</bibl> <bibtext> Frankforter, S. A., & Guidry, T. L. (2015). Who Needs ETS? How to create a psychometric assessment test. American Journal of Management, 15 (1), 73 – 80.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref11" type="bt">8</bibl> <bibtext> Gardiner, L. R., Corbitt, G., & Adams, S. J. (2009). Program assessment: Getting to a practical how-to model. Journal of Education for Business, 85 (3), 139 – 144. doi: 10.1080/08832320903258576</bibtext> </blist> <blist> <bibl id="bib9" idref="ref9" type="bt">9</bibl> <bibtext> Hogan, E. A., Maskulka, T. A., Kaminski, K. A., & Riley, S. M. (2017). Drilling down: A process to enhance faculty engagement in effective assessment. The Journal of Research in Business Education, 58 (1), 16.</bibtext> </blist> <blist> <bibtext> Loo, R. (2002). A Meta-Analytic examination of Kolb's learning style preferences among business majors. Journal of Education for Business, 77 (5), 252 – 256. doi: 10.1080/08832320209599673</bibtext> </blist> <blist> <bibtext> Payne, S. L., Flynn, J., & Whitfield, J. M. (2008). Capstone business course assessment: Exploring student readiness perspectives. Journal of Education for Business, 83 (3), 141 – 146. doi: 10.3200/JOEB.83.3.141-146</bibtext> </blist> <blist> <bibtext> Phelps, A. L., & Spangler, W. E. (2013). Customized versus standardized exams for learning outcomes assessment in an undergraduate business program. American Journal of Business Education, 6 (5), 521 – 530. doi: 10.19030/ajbe.v6i5.8042</bibtext> </blist> <blist> <bibtext> SAS Institute. (n.d.). Multiple linear regression with categorical predictors. Retrieved from https://<ulink href="http://www.jmp.com/en%5fus/statistics-knowledge-portal/what-is-multiple-regression/mlr-with-categorical-predictors.html">www.jmp.com/en%5fus/statistics-knowledge-portal/what-is-multiple-regression/mlr-with-categorical-predictors.html</ulink></bibtext> </blist> <blist> <bibtext> Smith, L. J., Clements, C., & Olson, J. E. (2010). Local versus standardized content assessment: some management implications, or why bother? Journal of Education for Business, 85 (5), 249 – 257. doi: 10.1080/08832320903449485</bibtext> </blist> <blist> <bibtext> Strang, K. D. (2014). Cognitive learning strategy as a partial effect on major field test in business results. Journal of Education for Business, 89 (3), 142 – 148. doi: 10.1080/08832323.2013.781988</bibtext> </blist> <blist> <bibtext> U.S. Department of Education. (2018). Accreditation in the United States. Retrieved from https://www2.ed.gov/admins/finaid/accred/index.html</bibtext> </blist> <blist> <bibtext> Zocco, D. (2011). A recursive process model for AACSB assurance of learning. Academy of Educational Leadership Journal, 15 (4), 67 – 91.</bibtext> </blist> </ref> <aug> <p>By Barry Wray; David Glew; Robert Burrus; Stephen Hill and Peter Schuhmann</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib16" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib17" firstref="ref4"></nolink> <nolink nlid="nl3" bibid="bib14" firstref="ref7"></nolink> <nolink nlid="nl4" bibid="bib11" firstref="ref14"></nolink> <nolink nlid="nl5" bibid="bib13" firstref="ref15"></nolink> <nolink nlid="nl6" bibid="bib10" firstref="ref17"></nolink> <nolink nlid="nl7" bibid="bib15" firstref="ref18"></nolink> <nolink nlid="nl8" bibid="bib12" firstref="ref20"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1249901
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Validating the Effectiveness of a Locally Developed Assurance of Learning Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wray%2C+Barry%22">Wray, Barry</searchLink><br /><searchLink fieldCode="AR" term="%22Glew%2C+David%22">Glew, David</searchLink><br /><searchLink fieldCode="AR" term="%22Burrus%2C+Robert%22">Burrus, Robert</searchLink><br /><searchLink fieldCode="AR" term="%22Hill%2C+Stephen%22">Hill, Stephen</searchLink><br /><searchLink fieldCode="AR" term="%22Schuhmann%2C+Peter%22">Schuhmann, Peter</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Education+for+Business%22"><i>Journal of Education for Business</i></searchLink>. 2020 95(4):216-226.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 11
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Business+Schools%22">Business Schools</searchLink><br /><searchLink fieldCode="DE" term="%22College+Outcomes+Assessment%22">College Outcomes Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Core+Curriculum%22">Core Curriculum</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Grades+%28Scholastic%29%22">Grades (Scholastic)</searchLink><br /><searchLink fieldCode="DE" term="%22Capstone+Experiences%22">Capstone Experiences</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Predictive+Validity%22">Predictive Validity</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/08832323.2019.1635976
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0883-2323
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Standards for accreditation by the Association to Advance Collegiate Schools of Business emphasize the assessment of student learning as an avenue to continuously improve business school curricula. The study reported here, which took place in an Association to Advance Collegiate Schools of Business-accredited business school in a mid-sized regional university in the southeastern United States, is an in-depth review of an internally created assurance of learning assessment instrument designed to directly measure student learning across a set of core courses. Results of the study indicate that facets of the assessment are significantly related to students' grades in their previous core courses as well as to their performance in a later capstone course. This suggests that carefully developed, internally crafted assessments may be valid means to assess student learning.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1249901
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1249901
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/08832323.2019.1635976
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 216
    Subjects:
      – SubjectFull: Business Schools
        Type: general
      – SubjectFull: College Outcomes Assessment
        Type: general
      – SubjectFull: Core Curriculum
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Grades (Scholastic)
        Type: general
      – SubjectFull: Capstone Experiences
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Predictive Validity
        Type: general
    Titles:
      – TitleFull: Validating the Effectiveness of a Locally Developed Assurance of Learning Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wray, Barry
      – PersonEntity:
          Name:
            NameFull: Glew, David
      – PersonEntity:
          Name:
            NameFull: Burrus, Robert
      – PersonEntity:
          Name:
            NameFull: Hill, Stephen
      – PersonEntity:
          Name:
            NameFull: Schuhmann, Peter
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 0883-2323
          Numbering:
            – Type: volume
              Value: 95
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Education for Business
              Type: main
ResultId 1