View in EDS HTML Full Text PDF Full Text

Performance Assessment for Mathematics Tutoring Centres

Saved in:
Bibliographic Details
Title: Performance Assessment for Mathematics Tutoring Centres
Language: English
Authors: Johns, Carolyn, Byerley, Cameron, Moore-Russo, Deborah, Rickard, Brian, Oien, Janet, Burks, Linda, James, Carolyn, Mills, Melissa, Heasom, William, Ferreira, Melissa, Mammo, Behailu
Source: Teaching Mathematics and Its Applications. Mar 2023 42(1):1-29.
Availability: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://teamat.oxfordjournals.org/
Peer Reviewed: Y
Page Count: 29
Publication Date: 2023
Document Type: Journal Articles
Reports - Evaluative
Education Level: Higher Education
Postsecondary Education
Descriptors: Performance Based Assessment, Mathematics Instruction, Tutoring, Undergraduate Students, Nonprofit Organizations, Evaluation Methods, Tutorial Programs, Organizational Change, Organizational Climate
DOI: 10.1093/teamat/hrab032
ISSN: 0268-3679
1471-6976
Abstract: Evaluation has become a common, and even an expected, practice across undergraduate mathematics tutoring centres in the USA, UK and other countries. However, these evaluation efforts could benefit greatly from leveraging organizational theory research. In this study, we situate mathematics tutoring centres as non-profit organizations (NPOs) to consider how an organization performance assessment framework can be adapted to study mathematics tutoring centre data and characteristics. We use qualitative and quantitative data, collected from six mathematics tutoring centres and adapt Lee & Nowell's (2015, "Am. J. Eval.," 36, 299-319) NPO performance framework to situate our study. Using thematic analysis, the research team underwent iterative cycles of data collection and analysis to code for alignment with Lee and Nowell's framework. By adapting Lee and Nowell's framework to mathematics centres, each of the six centres was given a more relevant lens to consider its performance. Regardless of the university setting, previous evaluations for centres have focused primarily on outputs (e.g., number of visits), behavioural change outcomes (e.g., correlating visits to grades) and client satisfaction outcomes (e.g., student surveys) that ignore the particular context of a centre. However, Lee and Nowell's framework takes into account additional performance indicators that provide a more nuanced understanding of a centre's performance by bringing to light the interplay among its various dimensions. Lee and Nowell's framework allows centres to look beyond outputs and outcomes to understand why these outputs and outcomes come to be. The use of this adapted performance framework, for the six mathematics centres in this study, allows an interpretation on a variety of dimensions using relevant data while indicating possible areas for change for each centre.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1372619
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHAZfIojijnjH9z1gLShjuHAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDNZmGoueh0vB7VhrUwIBEICBmwXIoSx0EIb1wRUqCgmpMsZUW8ozTxu65cqXB7gmu6X787UjBlJ15jYvPT2RrinBD_LBG2D4yFvBtONgG6iVnM70AtO4OWC92nu5ereB5yeE-51DxUoD4ezQgzE9Jsn0bOlZOlNkFvN1kw32HuAEos7TNTpyQg0Pq7UOGBbTvs-fltZ3x1kP_rVh3VbWDvq1mXcNcdMxms4aGf6O
Text:
  Availability: 1
  Value: <anid>AN0162272513;n6201mar.23;2023Mar09.05:42;v2.2.500</anid> <title id="AN0162272513-1">Performance assessment for mathematics tutoring centres </title> <sbt id="AN0162272513-2">1 Introduction</sbt> <p>Evaluation has become a common, and even an expected, practice across undergraduate mathematics tutoring centres in the USA, UK and other countries. However, these evaluation efforts could benefit greatly from leveraging organizational theory research. In this study, we situate mathematics tutoring centres as non-profit organizations (NPOs) to consider how an organization performance assessment framework can be adapted to study mathematics tutoring centre data and characteristics. We use qualitative and quantitative data, collected from six mathematics tutoring centres and adapt Lee & Nowell's (2015 , Am. J. Eval. , 36 , 299–319) NPO performance framework to situate our study. Using thematic analysis, the research team underwent iterative cycles of data collection and analysis to code for alignment with Lee and Nowell's framework. By adapting Lee and Nowell's framework to mathematics centres, each of the six centres was given a more relevant lens to consider its performance. Regardless of the university setting, previous evaluations for centres have focused primarily on outputs (e.g. number of visits), behavioural change outcomes (e.g. correlating visits to grades) and client satisfaction outcomes (e.g. student surveys) that ignore the particular context of a centre. However, Lee and Nowell's framework takes into account additional performance indicators that provide a more nuanced understanding of a centre's performance by bringing to light the interplay among its various dimensions. Lee and Nowell's framework allows centres to look beyond outputs and outcomes to understand why these outputs and outcomes come to be. The use of this adapted performance framework, for the six mathematics centres in this study, allows an interpretation on a variety of dimensions using relevant data while indicating possible areas for change for each centre.</p> <p>Mathematics tutoring centres, known as mathematics support centres in the UK and hereafter referred to as mathematics centres or centres, have become a fixture in USA and UK universities ([<reflink idref="bib43" id="ref1">43</reflink>]; [<reflink idref="bib9" id="ref2">9</reflink>]). Mathematics centres have justified their funding through student feedback, usage data and regression models indicating a correlation between mathematics centre visits and mathematics grades (e.g. [<reflink idref="bib67" id="ref3">67</reflink>]; [<reflink idref="bib13" id="ref4">13</reflink>]; [<reflink idref="bib50" id="ref5">50</reflink>]). However, the structure of mathematics centres varies across institutions ([<reflink idref="bib10" id="ref6">10</reflink>]; [<reflink idref="bib38" id="ref7">38</reflink>]). Which factors may contribute to mathematics centre effectiveness and how effectiveness should be defined both remain open questions. The overarching aim of our research program is to form a framework that can be used in the evaluation of mathematics centres.</p> <p>Our research program began as a quantitative study with the intent to measure the effectiveness of mathematics centres by examining the predicted increase in a student's mathematics grade per visit to the centre, accounting for student's incoming ability and other factors ([<reflink idref="bib13" id="ref8">13</reflink>]; [<reflink idref="bib50" id="ref9">50</reflink>]). Upon recognizing the variety of centres, we sought to define the characteristics of centre organizational structures in order to understand their similarities and differences ([<reflink idref="bib10" id="ref10">10</reflink>]). Next, we used quantitative and qualitative measures to compare effectiveness across multiple centres ([<reflink idref="bib11" id="ref11">11</reflink>]) and then used these measures to hypothesize on the organizational features of centres that might lead to increased or decreased effectiveness (Byerley <emph>et al.</emph>, 2021).</p> <p>We recently turned to literature on organization theory to frame our research. In doing so, we recognize that organizations within the same industry can have varying but successful organizational structures based primarily on features of organization size, environment and technology ([<reflink idref="bib19" id="ref12">19</reflink>]; [<reflink idref="bib24" id="ref13">24</reflink>]). We also came to understand there are several different ways in which organizations evaluate their performance ([<reflink idref="bib1" id="ref14">1</reflink>]; [<reflink idref="bib39" id="ref15">39</reflink>]). We needed to adjust our original research aim. Rather than using the same definition of effectiveness across various mathematics centres, we needed to tailor the definition of effectiveness to the features of each particular centre.</p> <p>In this paper, we address the following research question: how can we adapt and utilize [<reflink idref="bib39" id="ref16">39</reflink>] framework to study mathematics centre data and characteristics? Specifically, we</p> <p></p> <p>• (<reflink idref="bib1" id="ref17">1</reflink>)</p> <p></p> <ulist> <item> adapt [<reflink idref="bib39" id="ref18">39</reflink>] non-profit organization (NPO) performance framework to mathematics centres and subsequently</item> <p></p> </ulist> <p>• (<reflink idref="bib2" id="ref19">2</reflink>)</p> <p></p> <ulist> <item> apply this adapted performance framework by detailing</item> <p></p> <item> how this framework can be used to interpret mathematics centre performance on a variety of dimensions and</item> <p></p> <item> how performance on various dimensions is connected, which can be used to explain mathematics centre performance and indicate areas for change.</item> </ulist> <p>To answer our research question, we begin by situating mathematics centres as NPOs and examine the prior work and literature which inform our study. Next, we describe [<reflink idref="bib39" id="ref20">39</reflink>] work on performance assessment as it applies to mathematics centres. In our methods section, we describe our mixed-methods case study design. We then discuss our cross-case analysis of six diverse mathematics centres in the USA, using Lee and Nowell's framework to describe and interpret each centre's performance. We examine the interactions of various measures to explore reasons for performance and avenues for change.</p> <hd id="AN0162272513-3">2 Literature Review</hd> <p></p> <hd id="AN0162272513-4">2.1 Organization theory</hd> <p>[<reflink idref="bib19" id="ref21">19</reflink>] defined a work organization as an organization in which 'work is carried out on a regular basis by paid employees, which have been deliberately established for explicit purposes' (p. 2). Mathematics centres are organizations established for carrying out the work of tutoring students by paid tutors. More specifically, mathematics centres can be classified as NPOs as centres do not exist for the purpose of making a financial profit.</p> <p>NPOs differ from for-profit and government agencies in multiple ways ([<reflink idref="bib51" id="ref22">51</reflink>]; [<reflink idref="bib62" id="ref23">62</reflink>]; [<reflink idref="bib1" id="ref24">1</reflink>]; [<reflink idref="bib65" id="ref25">65</reflink>]). Although NPOs are diverse, they have similarities in terms of their structures and functions. NPOs are established to achieve social objectives and lack a goal of profit maximization ([<reflink idref="bib62" id="ref26">62</reflink>]; [<reflink idref="bib1" id="ref27">1</reflink>]; [<reflink idref="bib65" id="ref28">65</reflink>]). In the broader university context, mathematics centres may be used to increase profit through student retention, but as a distinct unit, there is no profit mechanism such as student payment per visit. In addition, NPOs have broad and vague missions and complex goals. For example, mathematics centre missions often include phrases such as 'increase student success' and provide an 'active learning atmosphere'. Further, like NPOs, mathematics centres serve a special function of providing services (tutoring) and advocating for a target population (traditionally, at-risk students).</p> <p>2.1.1 Organizational structure</p> <p>In our earlier work ([<reflink idref="bib10" id="ref29">10</reflink>], [<reflink idref="bib11" id="ref30">11</reflink>], [<reflink idref="bib12" id="ref31">12</reflink>]), we hypothesized that we could identify structures of tutor centres, which lead to effectiveness. However, upon further review of the organizational theory literature, we found contemporary views of organization theory hold that there is no one best organization structure ([<reflink idref="bib45" id="ref32">45</reflink>]; [<reflink idref="bib24" id="ref33">24</reflink>]). Rather, the effectiveness of an organization is impacted by the fit between the organization's environmental factors and the organization's characteristics ([<reflink idref="bib62" id="ref34">62</reflink>]; [<reflink idref="bib1" id="ref35">1</reflink>]). [<reflink idref="bib7" id="ref36">7</reflink>]) argues, 'what works in one setting, or at one point in time, may not work in another and that efficiency related to ongoing alignment of various [factors]' (p. 62). In other words, there is no one best common structure for all mathematics centres because each centre exists in a different environment. Therefore, rather than using the same definition of effectiveness across various mathematics centres, it is necessary to tailor the definition of effectiveness to the features of each particular centre.</p> <p>2.1.2 Performance assessment</p> <p>Due to the key structural and functional differences, performance assessment of NPOs is distinct from performance assessment in for-profit or government agencies. [<reflink idref="bib65" id="ref37">65</reflink>] assert, the '"public good" nature of NPOs products and services does not reflect the true market value or price; thus, competition and price cannot be used as a measure of performance' (p. 681). NPOs lack a universal measure of performance such as bottom line used in for-profit organization ([<reflink idref="bib32" id="ref38">32</reflink>]; [<reflink idref="bib1" id="ref39">1</reflink>]). Rather, performance assessment of NPOs utilizes independent, multi-dimensional criteria that include both financial and non-financial measures ([<reflink idref="bib51" id="ref40">51</reflink>]; [<reflink idref="bib32" id="ref41">32</reflink>]; [<reflink idref="bib65" id="ref42">65</reflink>]; [<reflink idref="bib39" id="ref43">39</reflink>]). Like organizational structure, selection of a performance assessment system is contingent on contextual and organizational features such as size, structure and environment ([<reflink idref="bib65" id="ref44">65</reflink>]; [<reflink idref="bib39" id="ref45">39</reflink>]). Building off of Wadongo and Abdel-Kader, [<reflink idref="bib39" id="ref46">39</reflink>]) proposed a seven-dimensional performance assessment framework consisting of (<reflink idref="bib1" id="ref47">1</reflink>) inputs, (<reflink idref="bib2" id="ref48">2</reflink>) organizational capacity, (<reflink idref="bib3" id="ref49">3</reflink>) outputs, (<reflink idref="bib4" id="ref50">4</reflink>) behavioural and environmental change outcomes, (<reflink idref="bib5" id="ref51">5</reflink>) client satisfaction outcomes, (<reflink idref="bib6" id="ref52">6</reflink>) public value accomplishment and (<reflink idref="bib7" id="ref53">7</reflink>) network/institutional legitimacy. Lee and Nowell's framework is discussed in-depth as the analytical framework later in this article.</p> <hd id="AN0162272513-5">2.2 Evaluation of mathematics tutoring centres</hd> <p>Historically, much of the research on tutoring has focused on the practices of tutors and tutor-student interactions (e.g. [<reflink idref="bib2" id="ref54">2</reflink>]; [<reflink idref="bib31" id="ref55">31</reflink>]; [<reflink idref="bib18" id="ref56">18</reflink>]; [<reflink idref="bib40" id="ref57">40</reflink>]). Only recently has research examined mathematics centres themselves ([<reflink idref="bib43" id="ref58">43</reflink>]; [<reflink idref="bib44" id="ref59">44</reflink>]). A portion of this research involves descriptive reports on the state of mathematics centres at institutions with no evaluation of the centres. Similar to [<reflink idref="bib43" id="ref60">43</reflink>]), we focus on mathematics centre research that includes elements of evaluation. In particular, we concentrate on the methods of centre evaluation, as well as gaps in the body of existing research. For more extensive reviews of the literature on mathematics centres, see [<reflink idref="bib43" id="ref61">43</reflink>], [<reflink idref="bib44" id="ref62">44</reflink>] and [<reflink idref="bib38" id="ref63">38</reflink>].</p> <p>Literature on mathematics centre evaluation has primarily focused on evaluation as a means to ensure centre funding. [<reflink idref="bib42" id="ref64">42</reflink>] argue that while evidence shows a need for mathematics centres, funding for centres is often not secure. Techniques of evaluation generally fall into four categories: centre usage data, links between student grades (or retention) and centre visits, links between student confidence and centre visits and student feedback ([<reflink idref="bib43" id="ref65">43</reflink>]).</p> <p>Centre usage data is the most commonly used evaluation method reported in a survey of US mathematics centres ([<reflink idref="bib44" id="ref66">44</reflink>]) and can be found in a number of papers (e.g. [<reflink idref="bib59" id="ref67">59</reflink>]; [<reflink idref="bib25" id="ref68">25</reflink>]; [<reflink idref="bib28" id="ref69">28</reflink>]). Usage data, such as number of student visits or number of visits per student, can show student demand for and value of services ([<reflink idref="bib21" id="ref70">21</reflink>]; [<reflink idref="bib42" id="ref71">42</reflink>]).</p> <p>Studies examining the correlation between student grades or retention and centre visits are another common means of demonstrating centre value. Several studies at various universities have employed regression models to help account for the biassing self-selection of centre attendance (e.g. [<reflink idref="bib67" id="ref72">67</reflink>]; [<reflink idref="bib5" id="ref73">5</reflink>]; [<reflink idref="bib13" id="ref74">13</reflink>]; [<reflink idref="bib50" id="ref75">50</reflink>]; [<reflink idref="bib53" id="ref76">53</reflink>]; [<reflink idref="bib33" id="ref77">33</reflink>]). For example, [<reflink idref="bib67" id="ref78">67</reflink>] performed a regression analysis and found that visiting the centre correlated with higher final exam scores in College Algebra when controlling for well-known predictive variables (i.e. gender, SAT, mathematics placement and high school grade point average or, GPA).</p> <p>Student feedback on satisfaction and confidence is another means of demonstrating the value of centres ([<reflink idref="bib43" id="ref79">43</reflink>]). For example, self-report surveys and interviews have provided evidence that tutoring can lead to improved students' confidence ([<reflink idref="bib22" id="ref80">22</reflink>]; [<reflink idref="bib29" id="ref81">29</reflink>]; [<reflink idref="bib49" id="ref82">49</reflink>]; [<reflink idref="bib15" id="ref83">15</reflink>]; [<reflink idref="bib66" id="ref84">66</reflink>]; [<reflink idref="bib26" id="ref85">26</reflink>]). Centres also seek feedback to understand why at-risk students do not attend tutoring. However, researchers warn that student feedback may be of limited utility due to the overly positive nature of student responses ([<reflink idref="bib6" id="ref86">6</reflink>]; [<reflink idref="bib21" id="ref87">21</reflink>]; [<reflink idref="bib37" id="ref88">37</reflink>]).</p> <p>These four types of evaluation are generally only descriptive or comparative between tutoring and no tutoring. While these evaluation methods may demonstrate the impact of mathematics centres in order to obtain or maintain funding, researchers have begun to call for studies which demonstrate strategies which lead to efficiency and effectiveness ([<reflink idref="bib47" id="ref89">47</reflink>]; [<reflink idref="bib36" id="ref90">36</reflink>]). This type of evaluation can be comparative to an ideal standard, longitudinally within the same organization or to similar organizations ([<reflink idref="bib32" id="ref91">32</reflink>]). Currently, there is no established ideal of centre performance. [<reflink idref="bib43" id="ref92">43</reflink>]) point out that longitudinal studies can be problematic due to the wide range of other university factors that change year to year. In addition, they argue comparing across institutions is difficult due to reluctance to share data. Only a small portion of the literature has reported on data from multiple institutions (e.g. [<reflink idref="bib29" id="ref93">29</reflink>]; [<reflink idref="bib41" id="ref94">41</reflink>]; [<reflink idref="bib48" id="ref95">48</reflink>]; [<reflink idref="bib10" id="ref96">10</reflink>]; [<reflink idref="bib11" id="ref97">11</reflink>]; [<reflink idref="bib44" id="ref98">44</reflink>]).</p> <p>Furthermore, evaluation of mathematics centres has not taken into consideration literature on organizations. As we have argued, mathematics centres are organizations which may be able to leverage the vast body of research on organizations, specifically NPOs. Centre evaluation has elements of organizational performance evaluation practices but has not drawn from that literature. For example, [<reflink idref="bib47" id="ref99">47</reflink>] argues for a multi-faceted evaluation approach that includes correlation of final grades to centre visits, feedback from students and anecdotal information from tutors. [<reflink idref="bib29" id="ref100">29</reflink>] suggest that to show effectiveness, mathematics centres need to demonstrate their impact on the different stakeholders (i.e. the students, the centre and the institution). [<reflink idref="bib43" id="ref101">43</reflink>] also argue the need to align evaluation with the mission of centres. These assertions align with aspects of organizational theory, but no study has utilized or applied organizational theory to mathematics centre evaluation.</p> <hd id="AN0162272513-6">3 Analytical Framework</hd> <p>In order to make sense of how mathematics centres might measure their performance, we draw on [<reflink idref="bib39" id="ref102">39</reflink>] framework of NPO performance assessment. Lee and Nowell created an integrated framework utilizing a variety of perspectives from performance assessment literature. They outline seven performance dimensions found in the literature on nonprofit performance assessment: inputs, organizational capacity, outputs, behavioural and environmental outcomes, client satisfaction outcomes, public value accomplishment and network/institutional legitimacy (Table 1). The relationships between these dimensions are illustrated in Fig. 1. Drawing on contingency theory ([<reflink idref="bib45" id="ref103">45</reflink>]; [<reflink idref="bib24" id="ref104">24</reflink>]), Lee and Nowell argue contextual factors would lead an NPO to focus on certain performance dimensions. In this section, we provide an overview of Lee and Nowell's framework, highlighting its applicability to mathematics centres (Table 1). We focus on aspects particularly salient to the mathematics centre context and consistent with Lee and Nowell's contingencies for choosing performance assessment practices (Fig. 1).</p> <p>Table 1 <emph>Performance measures for centres (adapted from</emph>[<reflink idref="bib39" id="ref105">39</reflink>]<emph>, p. 302)</emph></p> <p> <ephtml> <table><thead><tr><th>Performance dimension. </th><th>Definition. </th><th>Examples of performance measures in mathematics centres. </th></tr></thead><tbody><tr><td>Inputs </td><td>Acquirement and use of resources </td><td> • Funding for tutors • Physical space (location, square footage) • Ability to acquire and manage tutors • Contracted hours for non-salaried staff • Relationship with funders </td></tr><tr><td>Organizational capacity </td><td>Effectiveness of processes to use resources efficiently; capacity to innovate </td><td> • Employee satisfaction, motivation, retention and capabilities • Employee training • Critical operating processes such as marketing the centre or the check-in process • Capacity of the centre to adapt to changing needs </td></tr><tr><td>Outputs </td><td>Countable goods and services </td><td>• Number of students served • Number of courses served • Number of hours open • Number of services provided (e.g. tutoring, review sessions, calculator and textbook loans) • Physical space (e.g. layout, décor, signage) </td></tr><tr><td>Outcomes: behavioural and environmental changes </td><td>Intended change in target population </td><td>Impact on mathematics course grades • Impact on students' confidence or feelings of belongingness • Impact on students' use of self-regulatory processes or other study resources </td></tr><tr><td>Outcomes: client satisfaction </td><td>Satisfaction of customers; meeting needs of customers </td><td>• Percent of eligible students using services • Average visits per student attending mathematics centre • Student satisfaction surveys </td></tr><tr><td>Public value accomplishment </td><td>Impact on community/society </td><td>Community's perceived value of the centre's impact </td></tr><tr><td>Network/institutional legitimacy </td><td>Relationships with stakeholders and industry community; adherence to mission </td><td>• Satisfaction of administrators and instructors • Partnership/collaboration with internal and external tutor centres • Coherence of activities with stated mission </td></tr></tbody></table> </ephtml> </p> <p>Table 1 <emph>Performance measures for centres (adapted from</emph>[<reflink idref="bib39" id="ref106">39</reflink>]<emph>, p. 302)</emph></p> <p> <ephtml> <table><thead><tr><th>Performance dimension. </th><th>Definition. </th><th>Examples of performance measures in mathematics centres. </th></tr></thead><tbody><tr><td>Inputs </td><td>Acquirement and use of resources </td><td> • Funding for tutors • Physical space (location, square footage) • Ability to acquire and manage tutors • Contracted hours for non-salaried staff • Relationship with funders </td></tr><tr><td>Organizational capacity </td><td>Effectiveness of processes to use resources efficiently; capacity to innovate </td><td> • Employee satisfaction, motivation, retention and capabilities • Employee training • Critical operating processes such as marketing the centre or the check-in process • Capacity of the centre to adapt to changing needs </td></tr><tr><td>Outputs </td><td>Countable goods and services </td><td>• Number of students served • Number of courses served • Number of hours open • Number of services provided (e.g. tutoring, review sessions, calculator and textbook loans) • Physical space (e.g. layout, décor, signage) </td></tr><tr><td>Outcomes: behavioural and environmental changes </td><td>Intended change in target population </td><td>Impact on mathematics course grades • Impact on students' confidence or feelings of belongingness • Impact on students' use of self-regulatory processes or other study resources </td></tr><tr><td>Outcomes: client satisfaction </td><td>Satisfaction of customers; meeting needs of customers </td><td>• Percent of eligible students using services • Average visits per student attending mathematics centre • Student satisfaction surveys </td></tr><tr><td>Public value accomplishment </td><td>Impact on community/society </td><td>Community's perceived value of the centre's impact </td></tr><tr><td>Network/institutional legitimacy </td><td>Relationships with stakeholders and industry community; adherence to mission </td><td>• Satisfaction of administrators and instructors • Partnership/collaboration with internal and external tutor centres • Coherence of activities with stated mission </td></tr></tbody></table> </ephtml> </p> <p>Graph: Fig. 1 Relationship between performance dimensions for mathematics centres. Each performance dimension should be considered as it pertains to the advancement of the centre's mission. Adapted from [<reflink idref="bib39" id="ref107">39</reflink>], p. 305).</p> <hd id="AN0162272513-7">3.1 Performance dimensions</hd> <p>[<reflink idref="bib39" id="ref108">39</reflink>] argue NPOs work under the constraint of budget and resources. For example, mathematics centres are unable to make money to hire additional tutors or improve their physical facilities. <emph>Input</emph> performance examines how resources, such as funding, facilities and equipment, are acquired and used. Common mathematics centre resources include funding for tutors' salaries, the physical space provided by the university to the centre and the number of hours that staff (or faculty) serve as centre leaders. In the mathematics centre context, the strength of the relationship between the centre and its resource providers (e.g. department chairs or upper administration) is a measure of resource acquisition.</p> <p> <emph>Organization capacity</emph> measures the efficiency and effectiveness of the organization's processes towards furthering their mission, as well as the organization's ability to innovate and adapt ([<reflink idref="bib39" id="ref109">39</reflink>]). The organizational capacity includes centre processes such as how students find information on tutoring; how students register for services; and how tutors are hired, trained, motivated and retained. Centres need to be able to innovate by altering tutor operations to meet changing needs or streamlining processes to lower costs. The time available to centre leaders to effectively select and implement changes is one way to measure capacity for innovation.</p> <p> <emph>Outputs</emph> are 'countable goods and services obtained by means of nonprofit activities and direct products of activities for achieving the mission' ([<reflink idref="bib39" id="ref110">39</reflink>], p. 306). For example, outputs include the number of student visits, the number of hours the centre is open, the number of courses tutored and the services the centre offers. Outputs can be compared with inputs to examine centre efficiency. Comparative ratios of centre measures such as tutor cost per student visit (i.e. total tutor wages divided by number of student visits in a year) and utilization of facility space (i.e. total centre square feet times number of hours open divided by number of student visits in a year) assess input to output efficiency.</p> <p> <emph>Outcomes</emph> are separated into two dimensions: behavioural and environmental changes, as well as client satisfaction ([<reflink idref="bib39" id="ref111">39</reflink>]). <emph>Behavioural and environmental changes</emph> examine the 'state of the target population or social condition the program is supposed to have changed' ([<reflink idref="bib52" id="ref112">52</reflink>], p. 204). Lee and Nowell argue this dimension extends beyond outputs to consider the impact on the target population. For mathematics centres, this includes examining the increased skills, knowledge or learning gained by the students from visiting the mathematics centre. Assessments of the extent to which students' confidence or feelings of belongingness have been altered, or the changes in students' use of self-regulatory processes and other study resources are also measures of behavioural and environmental changes. The second outcome dimension, <emph>client satisfaction</emph>, measures the 'extent to which the organization satisfied and met the needs of the [target] population' ([<reflink idref="bib39" id="ref113">39</reflink>], p. 303). In mathematics centres, this is often measured by the percent of eligible students using their services, the number of visits per student using the centre and student satisfaction surveys.</p> <p> <emph>Public value accomplishment</emph> is the 'ultimate value/impact the organization hopes to create for the community/society' ([<reflink idref="bib39" id="ref114">39</reflink>], p. 303). In other words, it is the extent to which the organization is achieving the social ambition outlined in its mission statement according to outsiders' evaluations and opinions.</p> <p> <emph>Network/institutional legitimacy</emph> is 'how an organization has managed its relations with other stakeholders and established a reputation for trustworthiness and excellence within this broader network' ([<reflink idref="bib39" id="ref115">39</reflink>], p. 308). For example, it involves assessing how well the mathematics centre works with other on-campus and off-campus tutor centre organizations with similar goals. It also includes the relationships that math centre staff have with individuals who can supply other resources the centre might use. The satisfaction of the funders and stakeholders, typically department faculty and administration, is also part of network. Institutional legitimacy includes the extent to which the centre's activities match the stated mission.</p> <hd id="AN0162272513-8">3.2 Contingencies</hd> <p>While all of the performance dimensions (Table 1) impact each other, organizations should focus on particular dimensions based on environmental contingencies ([<reflink idref="bib39" id="ref116">39</reflink>]). Lee and Nowell suggested three contingency factors: (<reflink idref="bib1" id="ref117">1</reflink>) funding type, (<reflink idref="bib2" id="ref118">2</reflink>) the extent to which the work is programmable and observable and (<reflink idref="bib3" id="ref119">3</reflink>) the stability of the organization's environment (Fig. 2). Lee and Nowell discuss each of these contingencies individually and do not comment on how different combinations of contingencies interact. In addition, Lee and Nowell only comment on how instability impacts performance practices but do not elaborate on stability. In our discussion and analysis, we detail how these contingencies inform performance assessment practices. At present, we discuss how these contingencies can be applied to mathematics centres.</p> <p>Graph: Fig. 2 [<reflink idref="bib39" id="ref120">39</reflink>] three, independent environmental contingency factors.</p> <p>Due to the data available, we do not consider the funding type contingency in this study. With respect to programmability, the work of tutoring is non-programmable as it cannot be carried out via predetermined procedures and protocols ([<reflink idref="bib27" id="ref121">27</reflink>]). So, only non-programmability needs to be considered in light of the degree of observability ([<reflink idref="bib39" id="ref122">39</reflink>]). Mathematics centres have high observability when centre leaders are able to directly observe tutoring and low observability when constraints on centre leader time, the physical space of the centre or the private nature of one-on-one appointments impede a centre leader's attention to the service the centre provides.</p> <p>[<reflink idref="bib39" id="ref123">39</reflink>] consider the stability of the environment as the third contingency factor. Stability is impacted by operating decisions at both the department and centre levels; instability in mathematics centres is caused by frequent employee turnover or factors that impact a tutor's ability to be prepared mathematically. Centres in departments where courses are uniform due to coordination of topic coverage, scheduling and assessments have more stable environments. In contrast, centre tutors working for departments without course coordination might experience a wide variety in topics covered, textbooks used, assessments given and schedules for the same course taught by different instructors. Tutors who specialize in tutoring only one or a few related courses have a more stable environment than tutors in centres who serve as generalists working with students in a wide variety of courses. Lack of course coordination and specialized tutoring provide tutors a less stable environment due to constant changes in the mathematics content they must address in their tutoring.</p> <hd id="AN0162272513-9">4 Method</hd> <p>This study was conducted as a secondary analysis of previously collected data ([<reflink idref="bib58" id="ref124">58</reflink>]) with some additional data collected specifically for this study. We employed a mixed-methods, multiple case study design ([<reflink idref="bib68" id="ref125">68</reflink>]). Each unit of study was an organization, an individual mathematics centre. Our cases were designed to be explanatory, heuristic case studies ([<reflink idref="bib46" id="ref126">46</reflink>]), in order to build theory on how [<reflink idref="bib39" id="ref127">39</reflink>]) performance assessment framework can be adapted for mathematics centres and used to interpret centre performance. We examine how performance dimensions impact each other, which can be used to explain centre performance and indicate areas for change. We utilize both qualitative and quantitative data to account for each centre's performance on various measures from Table 1.</p> <hd id="AN0162272513-10">4.1 Case selection</hd> <p>This study was conducted as a secondary data analysis of the data presented in [<reflink idref="bib10" id="ref128">10</reflink>], [<reflink idref="bib11" id="ref129">11</reflink>], [<reflink idref="bib12" id="ref130">12</reflink>] work. The 10 cases in the original data were collected as a convenience sample ([<reflink idref="bib11" id="ref131">11</reflink>]). They represented centres whose leaders were involved in the Research in Undergraduate Mathematics Education mathematics centre community, interested in participating in the study and able to collect the necessary quantitative data.</p> <p>From the centres used in [<reflink idref="bib11" id="ref132">11</reflink>] study, we chose various centre qualities to form groups ([<reflink idref="bib30" id="ref133">30</reflink>]) to be studied. As environmental factors impact both appropriate organizational structures ([<reflink idref="bib24" id="ref134">24</reflink>]) and performance assessment ([<reflink idref="bib39" id="ref135">39</reflink>]), we grouped centres by size, observability and stability.</p> <p>We operationalize size of the centre as the number of students eligible to use the services and split the size of the institutions into two even groups: small and large. Observability was operationalized by whether centre leadership was present with the tutors at least 50% of the time and split the institutions into two even groups with high or low observability. While there are other contributing environmental factors, we operationalized stability as the presence of course coordination. Centres where less than 50% of the students being tutored came from coordinated courses were considered to have low stability.</p> <p>These three factors produced a possible eight groups of the various combinations of the three contingency factors. However, the 10 centres represented only 6 of the possible groupings (Table 2). In particular, no large schools had environmental instability.</p> <p>Table 2 Mathematics centre groupings for case selection</p> <p> <ephtml> <table><thead><tr><th>Grouping*Selected case. </th><th>Mathematics centre. </th><th>Size(# of eligible students). </th><th>Observability (supervisor or lead tutor present). </th><th>Stability(course coordination). </th></tr></thead><tbody><tr><td>Group A </td><td>Cat </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group A* </td><td>Gorilla </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group A </td><td>Whale </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group B* </td><td>Dog </td><td>Small </td><td>High </td><td>High </td></tr><tr><td>Group C* </td><td>Hamster </td><td>Large </td><td>Low </td><td>High </td></tr><tr><td>Group C </td><td>Fish </td><td>Large </td><td>Low </td><td>High </td></tr><tr><td>Group D* </td><td>Bird </td><td>Small </td><td>Low </td><td>High </td></tr><tr><td>Group E* </td><td>Horse </td><td>Small </td><td>High </td><td>Low </td></tr><tr><td>Group F </td><td>Goat </td><td>Small </td><td>Low </td><td>Low </td></tr><tr><td>Group F* </td><td>Dolphin </td><td>Small </td><td>Low </td><td>Low </td></tr></tbody></table> </ephtml> </p> <p>Table 2 Mathematics centre groupings for case selection</p> <p> <ephtml> <table><thead><tr><th>Grouping*Selected case. </th><th>Mathematics centre. </th><th>Size(# of eligible students). </th><th>Observability (supervisor or lead tutor present). </th><th>Stability(course coordination). </th></tr></thead><tbody><tr><td>Group A </td><td>Cat </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group A* </td><td>Gorilla </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group A </td><td>Whale </td><td>Large </td><td>High </td><td>High </td></tr><tr><td>Group B* </td><td>Dog </td><td>Small </td><td>High </td><td>High </td></tr><tr><td>Group C* </td><td>Hamster </td><td>Large </td><td>Low </td><td>High </td></tr><tr><td>Group C </td><td>Fish </td><td>Large </td><td>Low </td><td>High </td></tr><tr><td>Group D* </td><td>Bird </td><td>Small </td><td>Low </td><td>High </td></tr><tr><td>Group E* </td><td>Horse </td><td>Small </td><td>High </td><td>Low </td></tr><tr><td>Group F </td><td>Goat </td><td>Small </td><td>Low </td><td>Low </td></tr><tr><td>Group F* </td><td>Dolphin </td><td>Small </td><td>Low </td><td>Low </td></tr></tbody></table> </ephtml> </p> <p>Based on our groups, we selected mathematics centres as typical cases ([<reflink idref="bib60" id="ref136">60</reflink>]) in order to maximize differences across the centres by size, observability and stability ([<reflink idref="bib30" id="ref137">30</reflink>]). Thus, although the original 10 cases were selected based on convenience, the case studies presented here were chosen purposefully based on their ability to develop theory. When a group contained more than one centre, the centre with the more extensive qualitative data from the original data set was selected in order to include as many points of data as possible. From the six groups, we selected the Gorilla, Dog, Hamster, Bird, Horse and Dolphin mathematics centres for study.</p> <hd id="AN0162272513-11">4.2 Data collection</hd> <p>During the original collection of the secondary data, all centres were asked to fill out an open-ended survey based on [<reflink idref="bib10" id="ref138">10</reflink>] characteristics of tutor organizations and to submit quantitative data on their centre. The data included the results of a multiple regression model to capture the correlation of centre visits to final course grade (see [<reflink idref="bib11" id="ref139">11</reflink>], [<reflink idref="bib12" id="ref140">12</reflink>]). Qualitative descriptions include the relationship of the centre to mathematics department faculty, the percentage of time the centre leader was expected to spend operating the centre, the tutor training, the operation hours, the services provided and the physical space. Quantitative data included the number of students eligible to use the centre (counted as number of students enrolled in courses served by the centre), the number of student visits, the square footage of the centre and the tutor hours (taken as the sum of all hours worked by all tutors in a given semester). Quantitative metrics were calculated from these data to determine the percentage of those students who visited the centre, the average number of visits to the centre per visiting student, the tutor hours per student visit and the square footage per visiting student.</p> <p>For the regression analysis, it was deemed desirable to have a standardized analysis across centres in order to compare results. Based on data available to participating centres, four factors were included in the regression analysis: (<reflink idref="bib1" id="ref141">1</reflink>) student visits to the mathematics centre, (<reflink idref="bib2" id="ref142">2</reflink>) high school grade point average, (<reflink idref="bib3" id="ref143">3</reflink>) standardized entrance test scores and (<reflink idref="bib4" id="ref144">4</reflink>) course letter grades converted to grade points.</p> <p>Depending on the year data were available, quantitative data from each institution were collected for the fall semester of 2017 and/or 2018. Only students enrolled in mathematics courses the centre purposefully served were included in data analysis. Students with missing data and students who withdrew from the course were removed from the regression analyses but were included in the quantitative metrics such as eligible and visiting students. Students enrolled in multiple mathematics courses were treated as separate data points, with the number of visits to the centre split equally between enrolled courses.</p> <p>For this study, the researchers collected additional qualitative data from the six selected universities. Centre leaders were emailed a qualitative survey of open-ended questions based on [<reflink idref="bib39" id="ref145">39</reflink>] framework to fill gaps in the initial data. The individual who collected data from the primary study emailed these questions to participants, blinded the results and sent them to the first author of this study. After a second wave of coding, answers to remaining follow-up questions were collected.</p> <hd id="AN0162272513-12">4.3 Data analysis</hd> <p>In the primary study, multiple regression analyses were conducted on the data from each mathematics centre using course grades as the dependent variable and student visits to the centre, high school grade point average and standardized test scores as the independent variables. Analyses combined students from all centre-eligible courses within each respective university.</p> <p>For this study, analysis of the data was an iterative process (Fig. 3) using constant comparative methods to look for patterns and themes ([<reflink idref="bib17" id="ref146">17</reflink>]). We used deductive coding ([<reflink idref="bib8" id="ref147">8</reflink>]) on each centre's qualitative survey and quantitative data, using Lee and Nowell's eight performance dimensions as codes. We then wrote explanatory case studies ([<reflink idref="bib68" id="ref148">68</reflink>]) regarding each centre's performance on measures from Table 1. For analysis, we grouped the six cases into two larger grouping of low and high observability, based on [<reflink idref="bib39" id="ref149">39</reflink>]) contingencies. We compared cases within and across these two groupings. With each wave of data collection and analysis, the theory was refined. Respondent validation was then employed ([<reflink idref="bib35" id="ref150">35</reflink>]) to ensure validity of the analysis.</p> <p>Graph: Fig. 3 Iterative data collection and analysis.</p> <hd id="AN0162272513-13">5 Results and Discussion</hd> <p>To interpret each centre's performance, we align ourselves with [<reflink idref="bib32" id="ref151">32</reflink>] assertion that NPO effectiveness is comparative. Organizations may be compared with themselves at an earlier point in time, to similar organizations or to an ideal. Within our analysis, we compare across centres and a hypothetical ideal. Herman and Renz also argue 'there is no commonly agreed basis for judging [nonprofit organisation] effectiveness' (p. 404). They assert effectiveness is a social construct as the measures mean nothing until someone interprets them. We acknowledge the results may be interpreted differently by others; however, as practitioners with many years of experience working in centres, we feel we have the knowledge to provide plausible interpretations of the measures.</p> <p>To examine centres' performance on [<reflink idref="bib39" id="ref152">39</reflink>] performance dimensions (Table 3), we examine the centres in two groups based on their observability contingency. We first examine centres with high observability where a supervisor is physically present and observing the tutors at least 50% of the time (Gorilla, Dog, Horse). For these centres, Lee and Nowell propose focusing on end-to-end process of inputs, capacity and outputs, as well as their behavioural outcomes. We then examine centres with low observability (Bird, Dolphin, Hamster), focusing on their organizational capacity and client service outcomes. We also note that Horse's and Dolphin's instability, resulting from less than 50% of courses served having coordination, can lead to a performance assessment that focuses on network and institutional legitimacy. Within these discussions, we also explore how each centre's relationships with its funders and the perceived value of the centres may influence centres' abilities to increase or maintain funding. While we do not separate centres by Lee and Nowell's funding contingency, for any funding model, the relationships with funders or the public's perceived value of the centre could be considered.</p> <p>Table 3 Focal performance dimensions based on contingencies</p> <p> <ephtml> <table><thead><tr><th>Contingency. </th><th>Focal performance dimensions. </th><th>Centres. </th></tr></thead><tbody><tr><td>High observability </td><td> • End-to-end process of inputs, capacity and outputs • Behavioural outcomes </td><td>Gorilla, Dog, Horse </td></tr><tr><td>Low observability </td><td> • Capacity • Client satisfaction outcomes </td><td>Bird, Dolphin, Hamster </td></tr><tr><td>Instability </td><td> • Network and institutional legitimacy </td><td>Horse, Dolphin </td></tr></tbody></table> </ephtml> </p> <p>Table 3 Focal performance dimensions based on contingencies</p> <p> <ephtml> <table><thead><tr><th>Contingency. </th><th>Focal performance dimensions. </th><th>Centres. </th></tr></thead><tbody><tr><td>High observability </td><td> • End-to-end process of inputs, capacity and outputs • Behavioural outcomes </td><td>Gorilla, Dog, Horse </td></tr><tr><td>Low observability </td><td> • Capacity • Client satisfaction outcomes </td><td>Bird, Dolphin, Hamster </td></tr><tr><td>Instability </td><td> • Network and institutional legitimacy </td><td>Horse, Dolphin </td></tr></tbody></table> </ephtml> </p> <hd id="AN0162272513-14">5.1 Background and terminology</hd> <p>To help orient the reader, we introduce common terminology we will use throughout the discussion and results (Table 4).</p> <p>Table 4 Definitions for mathematics centre terminology</p> <p> <ephtml> <table><thead><tr><th>Terminology. </th><th>Definition. </th></tr></thead><tbody><tr><td>Mathematics centre or centre </td><td>Centre that exists for the purpose of helping students in mathematics (and often statistics) </td></tr><tr><td /><td> primarily through tutoring </td></tr><tr><td>Tutoring centre </td><td>Other centres on campus which tutor non-mathematics courses </td></tr><tr><td>Academic support group </td><td>Centralized group of multiple tutor centres on campus </td></tr><tr><td>Centre leaders or leadership </td><td>All mathematics centre employees with a rank above tutor </td></tr><tr><td>Director </td><td>Full-time employee who oversees the mathematics centre </td></tr><tr><td>Director's supervisor </td><td>Individual to whom the director reports </td></tr><tr><td>Assistant director </td><td>Full-time employee who assists the director in management of mathematics centre </td></tr><tr><td>GA </td><td>Graduate students who tutor, as well as instructors (with graduate degrees or who are graduate </td></tr><tr><td /><td> students) who both tutor and teach </td></tr><tr><td>Student managers </td><td>Student employees who assist in management of mathematics centre and who rank above </td></tr><tr><td /><td> the tutors </td></tr><tr><td>Course coordinator </td><td>Individual tasked with creating common syllabus, calendar, homework and exams across all </td></tr><tr><td /><td> sections of a course </td></tr></tbody></table> </ephtml> </p> <p>Table 4 Definitions for mathematics centre terminology</p> <p> <ephtml> <table><thead><tr><th>Terminology. </th><th>Definition. </th></tr></thead><tbody><tr><td>Mathematics centre or centre </td><td>Centre that exists for the purpose of helping students in mathematics (and often statistics) </td></tr><tr><td /><td> primarily through tutoring </td></tr><tr><td>Tutoring centre </td><td>Other centres on campus which tutor non-mathematics courses </td></tr><tr><td>Academic support group </td><td>Centralized group of multiple tutor centres on campus </td></tr><tr><td>Centre leaders or leadership </td><td>All mathematics centre employees with a rank above tutor </td></tr><tr><td>Director </td><td>Full-time employee who oversees the mathematics centre </td></tr><tr><td>Director's supervisor </td><td>Individual to whom the director reports </td></tr><tr><td>Assistant director </td><td>Full-time employee who assists the director in management of mathematics centre </td></tr><tr><td>GA </td><td>Graduate students who tutor, as well as instructors (with graduate degrees or who are graduate </td></tr><tr><td /><td> students) who both tutor and teach </td></tr><tr><td>Student managers </td><td>Student employees who assist in management of mathematics centre and who rank above </td></tr><tr><td /><td> the tutors </td></tr><tr><td>Course coordinator </td><td>Individual tasked with creating common syllabus, calendar, homework and exams across all </td></tr><tr><td /><td> sections of a course </td></tr></tbody></table> </ephtml> </p> <p>The six centres studied differed in other regards besides the contingencies. Table 5 provides additional background information for each centre.</p> <p>Table 5 Background information on mathematics centres</p> <p> <ephtml> <table><thead><tr><th>Institution. </th><th># of courses tutored. </th><th>Type of tutors. </th><th>Centre leadership. </th><th>Director's supervisor. </th><th>Housing unit. </th></tr></thead><tbody><tr><td>Bird </td><td>11 </td><td>Undergraduate </td><td>Director </td><td>Director of academic support group </td><td>Academic support group </td></tr><tr><td>Dog </td><td>5 </td><td>Undergraduate; GAs </td><td>Director, two assistant directors </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Dolphin </td><td>7 </td><td>Undergraduate </td><td>Director </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Gorilla </td><td>12 </td><td>Undergraduate, Optional for GAs </td><td>Director, student managers </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Hamster </td><td>10 </td><td>GAs </td><td>Two co-directors </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Horse </td><td>13, any math question </td><td>Undergraduate, GAs </td><td>Director </td><td>Associate vice provost </td><td>Academic support group </td></tr></tbody></table> </ephtml> </p> <p>Table 5 Background information on mathematics centres</p> <p> <ephtml> <table><thead><tr><th>Institution. </th><th># of courses tutored. </th><th>Type of tutors. </th><th>Centre leadership. </th><th>Director's supervisor. </th><th>Housing unit. </th></tr></thead><tbody><tr><td>Bird </td><td>11 </td><td>Undergraduate </td><td>Director </td><td>Director of academic support group </td><td>Academic support group </td></tr><tr><td>Dog </td><td>5 </td><td>Undergraduate; GAs </td><td>Director, two assistant directors </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Dolphin </td><td>7 </td><td>Undergraduate </td><td>Director </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Gorilla </td><td>12 </td><td>Undergraduate, Optional for GAs </td><td>Director, student managers </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Hamster </td><td>10 </td><td>GAs </td><td>Two co-directors </td><td>Chair of Mathematics Department </td><td>Mathematics Department </td></tr><tr><td>Horse </td><td>13, any math question </td><td>Undergraduate, GAs </td><td>Director </td><td>Associate vice provost </td><td>Academic support group </td></tr></tbody></table> </ephtml> </p> <hd id="AN0162272513-15">5.2 Centres with high observability</hd> <p>As centres with high observability of tutors (i.e. a supervisor is physically present and observing the tutors at least 50% of the time), Gorilla, Dog and Horse should focus on the end-to-end process of inputs, capacity, outputs and behavioural outcomes ([<reflink idref="bib39" id="ref153">39</reflink>]) when compared with professional standards (Table 3) ([<reflink idref="bib39" id="ref154">39</reflink>]). As there are no established professional standards for mathematics centres, we compare across institutions to examine centre performance relative to other centres. With respect to professional standards of tutors and tutoring, we rely on literature on effective tutoring and teaching.</p> <p>As can be noted in Fig. 1, there is a flow of impact from inputs to capacity to outputs to behaviour. We begin our discussion at the start of this flow by examining centres' performance on inputs to capacity. We then explore performance in the behavioural outcome dimension, noting behavioural outcome measures that align with centres' mission statements. Within this discussion, we examine performance on the inputs, capacity and outputs chain that flows into behavioural outcomes. We also examine Horse's network and institutional legitimacy dimension given Horse's instability (Table 3). Finally, we illustrate how each centre's aforementioned performance, relationship with funders and public value might be leveraged for additional funding. In Table 6, we offer a summary of each centre's mission to frame the objective to which the centre should be assessed.</p> <p>Table 6 Centre mission or Director's interpretation of mission when no official statement exists</p> <p> <ephtml> <table><thead><tr><th>Centre. </th><th>Mission statement. </th></tr></thead><tbody><tr><td>Bird </td><td>To provide support for active learning. </td></tr><tr><td>Dog </td><td>The mission of the centre is to improve student learning and academic performances for all students in our calculus courses. We support instruction that carefully considers student thinking and engages students as active participants in their learning process. </td></tr><tr><td>Dolphin </td><td>The centre is a community of mathematical thinkers who connect with each other in order to learn together and achieve their goals. Students learn through individual tutoring and study groups. Tutors and students also have some opportunities to grow and work together during special events throughout the quarter. </td></tr><tr><td>Gorilla </td><td>We provide a place for students to work on their homework and receive help from tutors if needed. We hope to create a place for students to belong and to be productive on their mathematics homework and to assist them in succeeding in their mathematics courses. </td></tr><tr><td>Hamster </td><td>The mission of the centre is to emphasize relevance, cultivate understanding and promote utilization of mathematics and statistics by providing an engaging and collaborative educational environment for our diverse community of learners. </td></tr><tr><td>Horse </td><td>We create engaging and effective learning experiences for students studying mathematics and statistics. Engaging means: friendly and approachable tutors, interactive sessions and user-friendly technology. Effective means: building skills and confidence, real-time learning feedback and progress in the classroom. </td></tr></tbody></table> </ephtml> </p> <p>Table 6 Centre mission or Director's interpretation of mission when no official statement exists</p> <p> <ephtml> <table><thead><tr><th>Centre. </th><th>Mission statement. </th></tr></thead><tbody><tr><td>Bird </td><td>To provide support for active learning. </td></tr><tr><td>Dog </td><td>The mission of the centre is to improve student learning and academic performances for all students in our calculus courses. We support instruction that carefully considers student thinking and engages students as active participants in their learning process. </td></tr><tr><td>Dolphin </td><td>The centre is a community of mathematical thinkers who connect with each other in order to learn together and achieve their goals. Students learn through individual tutoring and study groups. Tutors and students also have some opportunities to grow and work together during special events throughout the quarter. </td></tr><tr><td>Gorilla </td><td>We provide a place for students to work on their homework and receive help from tutors if needed. We hope to create a place for students to belong and to be productive on their mathematics homework and to assist them in succeeding in their mathematics courses. </td></tr><tr><td>Hamster </td><td>The mission of the centre is to emphasize relevance, cultivate understanding and promote utilization of mathematics and statistics by providing an engaging and collaborative educational environment for our diverse community of learners. </td></tr><tr><td>Horse </td><td>We create engaging and effective learning experiences for students studying mathematics and statistics. Engaging means: friendly and approachable tutors, interactive sessions and user-friendly technology. Effective means: building skills and confidence, real-time learning feedback and progress in the classroom. </td></tr></tbody></table> </ephtml> </p> <p>5.2.1 Inputs to organizational capacity</p> <p>In this section, we examine each centre's inputs, organizational capacity (processes) and the relationship between the two. For inputs, we focus on the amount of time centre leadership has for managing the centre. We also include collaborative network ties, which can help reduce the leadership's workload, allowing leadership to 'off load' some of the intellectual work by pulling ideas from similar organizations and asking others to collaborate. We examine how inputs of centre leadership time and ability to hire/manage tutors can impact processes of employee hiring, training and evaluation.</p> <p> <emph>Inputs</emph>. The input of centre leadership time varied at each institution. Horse and Gorilla each had a single director who spent 67–75% of their time on tutoring-related activities, while Dog employed a director and two assistant directors. Gorilla and Horse both utilized their tutors to extend oversight of the centre through the use of student managers. Horse's student managers oversaw progress of tutoring sessions in-the-moment, while Gorilla's student managers were more involved. Each student manager in Gorilla had a small group of tutors they were responsible for managing, observing and providing feedback. Gorilla's director held weekly meetings with the student managers to discuss any issues that came up in the week and plan future training or fun activities. The Gorilla student managers helped identify tutors doing exceptionally well and helped select the tutor of the month. They also helped with hiring, initial scheduling, coordinating shift covers, scheduling changes and updating policies and procedures each semester. Student managers allowed Gorilla's director to focus on higher-level aspects of the centre by off-loading the day-to-day administrative tasks to them. Through Gorilla's use of student managers and Dog's extensive leadership team, both centres allot a substantial amount time available to hiring, training and evaluation, as well as having time to consider innovations.</p> <p>In addition, each centre was able to be more efficient with their time through collaboration. Horse's position as a sub-centre of the Academic Support Services extended their network to other on-campus tutor centres. In addition, Gorilla's director and Dog's director and one of Dog's assistant directors were given time to dedicate to research on mathematics centres which led to collaboration with mathematics centre leadership at other institutions. By forming research partnerships with centre leaders at other institutions, Gorilla and Dog were able to draw on those resources for innovation. In addition, through conducting research on mathematics tutoring, Gorilla's and Dog's leaders were able to draw on best-practice findings from existing literature, as well as findings from their own research. While Horse did not have the time for external relationships at the time of the study, through the addition of a part-time assistant director, the leadership team now has additional time for collaboration and innovation. The amount of centre leadership time can provide varying constraints or affordances for enacting organizational capacity processes of tutor hiring, training and evaluation.</p> <p> <emph>Organizational capacity</emph>. The scope of a centre's mission dictated appropriate employee hiring, training and evaluation processes. Horse endeavoured to answer any mathematics or statistics questions from any student and, thus, needed tutors who could answer a wide range of questions which led to a focus on tutor's mathematical knowledge. On the other hand, Gorilla's and Dog's missions targeted fewer courses which allowed them to hire and train tutors based on ability to assist students on a few key courses. In addition, each centres' input of ability to hire and manage tutors impacted their processes.</p> <p>Gorilla, Dog and Horse all directly hired undergraduate tutors; however, Dog and Horse also had graduated assistants (GAs) who were required to work in the centre. Dog and Horse did not have any input with regard to the hiring or assignment of GAs. This inability to directly hire GAs meant they could not seek out specific desirable characteristics which affect tutor/student interactions and thus behavioural outcomes.</p> <p>Undergraduate tutor hiring practices varied among centres. Horse focused on the number of mathematics courses taken and grades received with availability, 'eagerness' and diversity considered second. For Gorilla and Dog, while grades were important, they were not the primary factor in hiring decisions. Gorilla's director 'value[ed] personality and trainability over high grades'. Dog's assistant director was not 'just looking for someone with good grades, but someone who tried to understand the concepts of the course, asked questions, participated and got along with their peers'. The centres' differences in qualifications were reflected in their hiring practices. Gorilla, Dog and Horse required tutors to submit a list of mathematics courses taken and grades earned. Gorilla and Dog also required applicants to provide an instructor reference. References were contacted to not only comment on students' mathematical performance in the class but also their participation and communication skills including how they got along with peers. Applicants at all three centres were interviewed by a centre leader; however, the focus on the interviews differed. Due to Horse's need for tutors who could answer a wide range of questions, the director prioritized mathematics knowledge in reviewing applications but during the interview process, the director focused on enthusiasm and people skills. On the other hand, because Gorilla's tutors only needed to be familiar with content from a few courses, Gorilla was able to consider applicants based on other criteria. The affordance combined with Gorilla's director's time to conduct research and network led Gorilla to implement practices which were grounded in research. Gorilla's hiring interviews focused on tutors' trainability on pedagogical techniques as demonstrated by performance on a mock-tutoring scenario. Dog conducted hiring interviews similarly. Research has shown that good grades in mathematics courses are not enough to be an effective teacher ([<reflink idref="bib3" id="ref155">3</reflink>]). Teachers also need to have productive dispositions and beliefs to support development and enactive of productive mathematics teaching processes ([<reflink idref="bib61" id="ref156">61</reflink>]). These dispositions and beliefs are difficult to change ([<reflink idref="bib20" id="ref157">20</reflink>]); so, targeting tutors already holding productive dispositions may lead to greater uptake in desired tutoring practices.</p> <p>Training also varied between the three centres. Drawing on their research knowledge, Gorilla and Dog aimed to develop tutors with well-rounded mathematical knowledge for teaching ([<reflink idref="bib4" id="ref158">4</reflink>]), consisting of both content and mathematics specific pedagogical knowledge. In addition, with tutors serving a narrow range of students, training was able to target specific knowledge and skills required for a few courses. Horse on the other hand had to prepare their tutors to answer any mathematics or statistics questions that were brought to the centre, as well as questions on statistical software. As such, Horse focused on content and general pedagogical skills. At Horse, undergraduates received the same tutor training as GAs, focusing on centre procedures with optional training on specialized course content and general pedagogy topics such as student disabilities and handling difficult situations. However, since data collection, Horse's assistant director has given leadership time to innovate and expand training. At Dog, training was different for GAs and undergraduates; however, both experienced content and pedagogical training. Undergraduates received tutor-specific pedagogical training conducted in partnership with the academic support group on campus, while mathematics content and mathematics pedagogy specific training occurred within the mathematics centre. Both GAs and undergraduate tutors received continuous content training via attending lecture and weekly course led by the course coordinator. The course coordinator would go over upcoming content, focusing on productive conceptual understandings of the material, ways of student thinking and teaching techniques. Further, because GAs' training for teaching was also conducted by centre leadership, it reflected the active-learning pedagogy encouraged in tutoring. At Gorilla, training also entailed a mix of content and pedagogy. Training at Gorilla included 8 h per semester which tended to 'involve a lot of role play and analysis of tutoring transcripts'. Gorilla's director sought to develop these skills through training akin to video clubs ([<reflink idref="bib63" id="ref159">63</reflink>], [<reflink idref="bib64" id="ref160">64</reflink>]; [<reflink idref="bib54" id="ref161">54</reflink>]; [<reflink idref="bib55" id="ref162">55</reflink>]; [<reflink idref="bib34" id="ref163">34</reflink>]; [<reflink idref="bib56" id="ref164">56</reflink>]) with an aim to produce reflective practitioners ([<reflink idref="bib57" id="ref165">57</reflink>]; [<reflink idref="bib34" id="ref166">34</reflink>]). Video clubs have been shown to improve teacher's pedagogical skills ([<reflink idref="bib63" id="ref167">63</reflink>], [<reflink idref="bib64" id="ref168">64</reflink>]; [<reflink idref="bib55" id="ref169">55</reflink>]; [<reflink idref="bib56" id="ref170">56</reflink>]). In addition, reflective practitioners continuously look to improve their practice ([<reflink idref="bib57" id="ref171">57</reflink>]; [<reflink idref="bib34" id="ref172">34</reflink>]).</p> <p>Evaluation of undergraduate tutors was similar at all three centres. Due to the frequent presence of a centre leader in the centre, most evaluation and feedback to undergraduates occurred informally and in-the-moment. When tutors were not meeting expectations or received negative feedback, leadership would meet with the tutor. Horse and Gorilla also had more formal feedback mechanisms. Both Horse and Gorilla collected student feedback that was shared with tutors. Gorilla also collected one tutoring observation for each tutor and then met with each tutor to give personalized feedback. With regard to GA evaluation, Dog and Horse were also reliant on the broader university/departmental structure for the centre leadership's role in GA evaluation. At Dog, the departmental structure and culture of active-learning pedagogy allowed the centre leaders to report performance concerns with GAs to the course coordinators or graduate committee. At Horse, the director was the supervisor of the GAs working in the centre and could directly discuss performance issues or take away the student's assistantship.</p> <p>We have described how each centre's mission and inputs of centre leadership time and the ability to hire/manage tutors impacted organizational capacity of tutor hiring, training and evaluation. Each centre's inputs and organizational capacity will subsequently impact outputs and behavioural outcomes (Fig. 1). Next, we continue along the chain of performance dimensions to examine outputs and behavioural outcomes.</p> <p>5.2.2 Outputs to behavioural outcomes</p> <p>In this section, we examine each centre's outputs, behavioural outcomes and the impact of the centres' missions on outputs and selection of behavioural outcome measures. For outputs, we focus on the input/output ratio of tutor hours per visit and number of courses served. We then discuss various behavioural outcome measures that would align with each centre's mission but report only on the quantitative regression model.</p> <p> <emph>Outputs</emph>. We begin by examining the outputs dimension using measures of the input/output ratio of tutor hours per visit and number of courses served. To examine each centre's efficient use of tutor funding, the tutor hours (input) was divided by the number of student visits (output). This ratio was then converted to average time in minutes a tutor could spend per student visit for each of interpretation (Fig. 4). Compared with other institutions, on average, the time tutors at Gorilla and Dog could spend with a student was lower than other centres, while at Horse, this measure was higher than other centres. A higher-than-average measure compared with other centres would indicate they have sufficient resources to serve students while a higher-than-average ratio may indicate inefficient use of resources. The number of courses served (Table 5), as dictated by the centre's mission (Table 6), varies across the three centres. Dog's mission was to serve calculus students and served only five courses. Gorilla's mission entailed assisting students in first- and second-year courses, generally through differential equations and served 12 courses. Horse, however, had a broad mission to assist with any mathematics question. Officially, they served 13 courses; however, any student could use the centre. As previously noted, the differences in missions impacted tutor hiring, training and evaluation. We now explore how these differences in varying outputs, in turn impacts behavioural outcomes.</p> <p>Graph: Fig. 4 Input–output ratio of tutor hours/visit for centres as sorted by observability.</p> <p> <emph>Behavioural outcomes</emph>. When selecting behavioural outcome measures, it is important that they align with the centre mission. Gorilla and Horse both mention affective goals of belongingness and confidence which should lend to positive student responses to mathematics via an attitudinal survey. To measure student learning and skill building beyond course grades, a conceptual pre/post-test could be employed. In addition, all three centres mention success in mathematics courses as an aspect of their mission. While this can be measured in many ways, for this study, we used a regression model to examine the relationship between centre usage and course grade. Based on the model, after controlling high school GPA in mathematics courses and standardized test scores, for each visit to the centre, students' final grades are predicted to increase 0.015 grade points at Gorilla (<emph>F(<reflink idref="bib3" id="ref173">3</reflink>, 2733)</emph> = <emph>209.1, P</emph> < <emph>0.001, R<sups>2</sups></emph> = <emph>0.186</emph>) and 0.035 grade points at Dog (<emph>F(<reflink idref="bib3" id="ref174">3</reflink>,1000)</emph> = <emph>115.5, P</emph> < <emph>0.001, R<sups>2</sups></emph> = <emph>0.257</emph>). While Horse's quantitative model did not indicate a significant effect of visits on mathematics course final grades, this is only one of many possible ways to measure the behavioural outcome of student success. In particular, due to the small number of variables included in the model, centres with access to additional relevant predictors may find different results after controlling for other variables. In addition, Horse's model includes only students in the 13 targeted courses and as such does not include students who received help from the centre outside of those courses. In addition, as inputs, organizational capacity and outputs flow into behavioural outcomes, Gorilla and Dog may have each benefited from the centres' higher amounts of time leadership dedicated to their centres and a focus on fewer courses which allowed for more extensive and targeted training.</p> <p>5.2.3 Horse's instability: Network and institutional legitimacy</p> <p>Horse's performance assessment should also take into consideration its instability (Table 3). [<reflink idref="bib39" id="ref175">39</reflink>] do not discuss how the overlap of instability and high observability impacts performance assessment. It may be the case that instability overshadows observability. If the environment is constantly changing, appropriate tutor interactions are also in constant flux creating difficulties with supervision (observability). Not only Horse lacked stability based on lack of course coordination, but also the centre served a wide range of students aiming to answer any mathematics question. Due to Horse's instability, network and institutional legitimacy should be considered as a more appropriate performance assessment.</p> <p>In examining Horse's network, Horse's leadership was well connected to other tutor centres on campus. Horse's director was supervised by the associate vice provost, who held monthly meetings with all academic support group leaders on campus. During these meetings, tutor centre leaders would share updates on any changes, as well as usage data. Horse's Director used this network to gather ideas for changes to the mathematics centre and market the centre's services, which can help advance the centre mission. With regard to institutional legitimacy, it may be beneficial for Horse's director to consider whether its services align with their mission. Horse's director reported that while the centre was required to serve students in the calculus and statistics sequences, serving students outside of the sequences and fielding mathematics/statistics questions from courses outside of the mathematics department were optional services provided by the Horse.</p> <p>5.2.4 Relationship with funders</p> <p>We now explore how centres' relationship with funders, an input, can be paired with the centre's public value and performance assessment to justify increased funding or build support for change in centre practices. Performance assessments are often used as leverage for funding ([<reflink idref="bib14" id="ref176">14</reflink>]), and both Dog and Horse had been successful in using data to support additional funding in the past.</p> <p>Centres with strong ties to their funders will likely have greater success in gaining additional funding. Gorilla and Dog both had strong relationships with administration. At both institutions, at least one centre leader reported directly to the department chair. In addition, Dog's leadership had administrative connections to the dean and provost, who were influential in obtaining funding for the centre, and took an interest in centre activities. At Dog, the department chair participated in tutoring events, the dean would walk by the centre time to check on its use, and the provost was previously a professor in the mathematics department. These relationships gave leadership at Gorilla and Dog direct lines to funding decision-makers.</p> <p>Leadership at Gorilla and Dog recognized they did not have sufficient resources as they stated that given additional funding, they would spend it on acquiring additional tutor hours. To justify additional funding, Gorilla and Dog can present to funders a performance assessment including the previous analysis to demonstrate funding for tutors is the primary obstacle for improved centre performance and achieving each centre's mission. For example, Dog's assistant director stated that, 'during crowded times it would be inaccurate to say that our tutoring was "student-centred" and "engages students as active participants in their learning process".' When busy, the tutors often worked through the problem on the board with a small group and, instead of helping students understand a concept, the tutor would move on to the next in line. However, 'when less busy some tutors definitely provide student-centred instruction.' Providing evidence that the centre's tutors enacted active learning techniques when the centre was less busy but struggled to meet that mission objective when busy could help to demonstrate it was not due to training or other factors that tutors failed to implement active learning, but rather time per student, which could be addressed by additional funding. While the behavioural outcome performance can be used as evidence that Dog and Gorilla are fulfilling their missions, it is not known to what extent the predicted increase in grade points per visit would change if students had more time with a tutor per visit. Dog might also use evidence of the relationship of tutor behaviours to the number of students in the centre to argue for additional funding.</p> <p>Gorilla and Dog have strong ties to instructors in the department who can voice their perceived value of the centre to administrators. At Dog, GAs were required to hold office hours in the centre; at Gorilla, this was optional. In addition, leadership at both centres taught courses in the mathematics department and worked closely with course coordinators. At Gorilla, the Director had weekly contact with course coordinators who also occasionally facilitated tutor trainings. At Dog, the centre leaders themselves were coordinators, and tutors attended lectures and course instructor meetings. These close ties with instructors added to each centre's perceived value within the department and integrated the centre into the mathematics department network.</p> <p>At Horse, leadership may take into consideration whether to reduce their services to only the courses they are required to serve. Removing these optional services may allow Horse to focus on its required mission, creating less instability for tutors and improving tutors' ability to help targeted students. However, before making change, Horse's director should take into account the provost and other stakeholders' perceived value of the centre. If the perceived value is the centre's ability to help anyone with a mathematics or statistics question it would be ill-advised to narrow the scope of tutoring. Horse's strong relationship with the provost's office puts them in a good position to have these conversations.</p> <hd id="AN0162272513-16">5.3 Centres with low observability</hd> <p>Client satisfaction outcomes, organizational capacity and institutional legitimacy are focal performance dimensions for centres with low observability (Hamster, Bird, Dolphin). Due to the inability to observe tutors' work, centres with low observability must rely on client satisfaction outcomes to evaluate if the centre is meeting its' mission. Client service outcomes include quantitative measures such as percent of eligible students using services and average visits per student attending, as well as qualitative client feedback. While quantitative measures are easier to collect, qualitative feedback can explain quantitative results. These client service outcomes can point to areas within organizational capacity (such as marketing, tutor hiring/training) which are positively or negatively impacting client satisfaction (Fig. 1). We discuss how changes to organizational capacity can improve client satisfaction outcomes within the analysis. We split our analysis into two sections. The first section examines client satisfaction as it relates to 'getting students in the door' as measured by the percent of eligible students attending. In the second section, we examine client satisfaction in terms of retention as measured by average number of visits per student attending.</p> <p>5.3.1 Client satisfaction: getting students in the door</p> <p>The percent of eligible students attending (Fig. 5) is predominantly influenced by marketing. Positive student experiences and instructor, as well as parent, messaging to attend tutoring are forms of marketing that help bring students to the centre and that create a sense of normalcy regarding help-seeking. Beyond marketing, the services offered should meet student needs. The percentage of visiting students also benefits from a convenient physical space location.</p> <p>Graph: Fig. 5 Client satisfaction measure as percent of eligible students using services for centres sorted by observability.</p> <p>Compared with other institutions, Hamster's percent of eligible students using their services fell around the middle for the six centres (Fig. 5). The primary marketing tool was instructor announcements, which may be an effective form of marketing since instructors have a captive class audience and are best positioned to recommend tutoring. Hamster had a strong network among instructors as instructor offices were located on the same building floor as the centre tutors where GAs and the centre's two Co-Directors 'work[ed] closely' with instructors and coordinators. This strong relationship served as a reminder of the centre's services and perhaps gave the centre a sense of legitimacy as instructors were familiar with centre leadership and tutors. Furthermore, Hamster's relationship with the campus academic support group aided in the convenience of the physical location and contributed to offering services that met student needs. The mathematics centre was accessible to students, located within a 5-min walk of the middle of campus, and close to mathematics classrooms. The centre also reserved a table to be used for mathematics tutoring appointments offered by the academic support group. Students did not have to go out of their way to attend tutoring, and there was one central location for all mathematics tutoring. Furthermore, Hamster's drop-in services complemented the academic support group appointments. In addition, Hamster's hours of operation were based on student demand, and courses served focused on difficult and higher enrolment service courses.</p> <p>Bird and Dolphin both had a low percent of eligible students using services compared with other institutions (Fig. 5). In order to pinpoint specific areas for improvement, centres can survey eligible students who are not using their services. While Bird did not conduct such a survey, Dolphin found many students did not attend tutoring because 'they weren't comfortable asking for help.' This indicates the centre had an image issue. Students may have been concerned about being judged by tutors, peers or instructions. To lower the stigma associated with tutoring and improve their image, Dolphin could look to their marketing. Specifically, Dolphin could seek to normalize tutoring with personal testimonies of students who have used the services. Video testimonials of senior students or tutors who have attended tutoring would demonstrate that even successful students use tutoring, and it is not shameful. In addition, Dolphin may need to improve the instructors' image of the centre. As seen from Hamster, instructors can be positive marketing resources for the centre, but Dolphin reported instructors were sending messaging (either explicit or implicit) that tutoring was only for struggling students. Students may have avoided attending tutoring to prevent being seen or identifying as being 'bad at mathematics'. The centre could partner with instructors to ensure they are sending positive messaging, are making in-class announcements and are referring all, not just struggling, students to tutoring. The centre may find providing instructors with research-based evidence of the benefits of tutoring or student testimonials and local data may improve instructors' image of tutoring. In addition, improving the departmental administration's image of tutoring could help with top-down tutor messaging to instructors.</p> <p>5.3.2 Client satisfaction: retention</p> <p>There are also many factors which impact average visits per students attending. The physical space (output) in terms of adequate seating and square foot per student, as well as comfort influence the students' experience. In addition, long wait times due to an insufficient number of tutors would lower satisfaction. The most important factor influencing students' desire to return may be their interactions with the tutors. The behaviours of tutors depend partly on a centre's ability to hire and manage tutors (input). Centres unable to be selective in their hiring or lacking the ability to manage tutors are unable to alter tutors' behaviours. In addition, centres need effective processes for hiring, training and evaluating tutors in order to promote positive student experiences. Surveying students who have attended tutoring helps understand which factors are positively or negatively impacting client satisfaction.</p> <p>While Bird had difficulties getting students in the door (low percent of eligible students visiting), once students tried tutoring, they kept coming back (high average visits per student attending). Bird had an average of 13.98 visits per student attending, high compared with other institutions (Fig. 6). We examined their centre for factors contributing to their success. Bird's director reported generally positive feedback on student surveys with common praises including '(<reflink idref="bib1" id="ref177">1</reflink>) The [centre] provides an active learning experience—students are not given answers but are forced to learn; (<reflink idref="bib2" id="ref178">2</reflink>) Tutors are welcoming, motivating and encouraging; and (<reflink idref="bib3" id="ref179">3</reflink>) Tutors provide helpful perspectives when learning.' Given the common praises, the high average visits per student is likely due to tutor hiring, training and evaluation.</p> <p>Graph: Fig. 6 Client satisfaction measure as average visits per student attending for centres sorted by observability.</p> <p>Bird's director directly hired tutors and reported there were no difficulties finding qualified tutors. This allowed the director to select applicants based on desired characteristics. Criteria used to select tutors included strong communication skills and their own experience learning. Student feedback that tutors are 'welcoming, motivating and encouraging', as well as 'provide helpful perspectives' align with the hiring criteria. In addition, the director hired tutors open to training and served as their manager which put her in a position to train and enforce tutor behaviours aligned with the centre's mission, 'to provide support for active learning'. In addition to other training, Bird's director spent 2–3 h per semester on mathematics active learning pedagogy, such as questioning and collaborative problem solving. Tutors were provided with many opportunities for feedback from both peers and the director. During training sessions, tutors performed simulated tutoring for feedback and were given feedback on end-of-semester self-evaluations. Tutors not meeting expectations for tutor effectiveness would meet with the centre leader to discuss 'how their behaviour could shift, what approaches they should take'. If the tutors did not improve, they were paired with a more expert tutor and worked fewer hours. Achievement of the centre's mission is directly reflected in student feedback that 'the [centre] provides an active learning experience—students are not given answers but are forced to learn'. The training tutors received emphasized active learning, mechanisms for tutor feedback regarding their tutor performance contributed to positive client satisfaction.</p> <p>Hamster and Dolphin had 4.58 and 4.35 visits per student attending, respectively (Fig. 6), which was low compared with other institutions. Hamster and Dolphin each had unique situations regarding student satisfaction surveys. Hamster did not conduct student satisfaction surveys; however, anecdotally, two common complaints were (<reflink idref="bib1" id="ref180">1</reflink>) there was not a tutor who could help the student with a certain subject at a given time and (<reflink idref="bib2" id="ref181">2</reflink>) 'tutors sit around and don't do anything, only give answers rather than help, and don't seem to care about helping.' These complaints indicate issues with tutor scheduling and communicating hours for specific courses with students. However, perhaps more important, complaints about tutor interactions likely stemmed Hamster's co-directors' inability to directly hire, mange, tutor and evaluate tutors.</p> <p>Hamster was unable to directly hire or supervise tutors as tutor assignments were based on teaching staff contracts and GA assignments. Tutors were 'typically well qualified, though they are not always familiar with subjects that they have not taken', leading to each tutor tutoring a different subset of courses. The centre's inability to hire tutors meant they could not seek out tutors with desirable characteristics. Had the centre been able to directly hire tutors, they may have been able to more equally distribute the mathematics qualifications of tutors to ensure tutoring was offered for all courses during open hours. In addition, because tutors were not managed by the co-directors, there were no mechanisms in place to evaluate or report tutors not meeting expectations. For example, centre leadership reported, 'ideally a tutor would at least acknowledge a student coming in and ask if they need assistance (once they're seated at settled).' However, the tutors did not meet this ideal 'at all.' Instead, 'tutors sit at tables and work on their own work or play on their phone and occasionally look around to see if anyone has their hand up.' Hamster's co-directors had no means of improving tutor performance. Furthermore, the process for training tutors at Hamster was minimal, including only how the tutoring worked. Tutors were provided online course review material, but its use was not required. The centre's inability to hire, manage and train tutors likely contributed to poor service and low return rates.</p> <p>Despite Dolphin's low rate of student return, end-of-appointment student surveys completed were typically complementary, with rare complaints about centre hours or a tutor struggling with a mathematics problem. In addition, as this was the centre's first year of operation, tracking the return rate longitudinally could be used to track improvement. To address complaints of tutor knowledge, the centre could examine tutor hiring and training used in the centre. Tutors were hired based on professor recommendation and mathematics honour society membership, likely putting them top among their peers in terms of content knowledge. However, the director could introduce processes to help tutors review content prior to session, creating a procedure by which students inform the tutor of topics they wish to cover in the session, or train tutors in how to either use resources or request help from an instructor, as needed. It may also be beneficial to set student expectations ahead of time that tutors are not instructors and may not have the answers right away but will help students work towards solving the problem together and model using resources. This type of assistance aligns with two of the centre's guiding principles: active learning and to create independent learners. Further, as a centre with low stability, Dolphin can also focus on institutional legitimacy. Ensuring centre services and processes align with the centre mission as key aspect of their performance assessment.</p> <hd id="AN0162272513-17">6 Summary and Implications for Practice</hd> <p>By using [<reflink idref="bib39" id="ref182">39</reflink>] framework adapted for mathematics centres, centres can get a much fuller picture of their performance. Previous evaluations for centres have focused primarily on outputs (number of visits), behavioural change outcomes (correlating visits to grades) and client satisfaction outcomes (student surveys). However, [<reflink idref="bib39" id="ref183">39</reflink>] framework takes into account performance dimensions outside of outputs and outcomes. These additional performance dimensions paint a fuller picture of the state of a centre and bring to light the interactions among the dimensions. For example, while centre visits at Dog had a positive effect on grades, observation of what occurred in the centre showed tutors enacted less student-centred pedagogy when the centre was crowded. The centre was using its existing resources efficiently but did not have enough funding for additional tutors. Their relationship with funders in the past had allowed them to request and receive additional funding. Centres can dig beyond outputs and outcomes to understand why these outputs and outcomes come to be.</p> <p>In addition, previous evaluations have compared tutoring versus no tutoring in order to demonstrate the need for a mathematics centre on campus. Comparing across institutions begins the process of establishing professional standards or ideals to which centres can aspire. For example, Bird can see that compared with other institutions, it has a strong performance on average visits per student attending but low performance on percent of eligible students visiting. These comparisons set benchmarks and allow the centre to turn inward to look for process improvements. Bird can see that students are satisfied once they attend, but the centre may need to improve marketing processes, the centre's image or network with instructors to get students in the door the first time. Centres can use the performance data reported in this study to determine how their centre's performance compares.</p> <p>Further, centres can use the performance dimension contingencies in [<reflink idref="bib39" id="ref184">39</reflink>] framework to select measures which align with the centre's structure. Centre leaders who are able to spend significant time observing tutors can focus on high observability focal dimensional of the end-to-end process of inputs to outcomes (Fig. 1). These centre leaders are more likely to see how the details of each tutor-student interaction may promote or constrain targeted student behavioural changes. For example, Horse can observe tutor-student interactions to determine if professional standards of interactions (accepted teaching behaviours) are occurring and then adjust training accordingly. While centres with low observability cannot observe tutor-student interactions, they can use student satisfaction surveys to locate areas for improvement. Hamster's common complaint that tutors ignored students indicates a need for improvement in tutor motivation. Based on their structure, centres can select appropriate performance dimensions from Lee and Nowell's framework which will be useful both for internal improvement and external funding support.</p> <hd id="AN0162272513-18">7 Limitations and Future Work</hd> <p>The primary limitations of this work include the cases selected and the data collected. While the cases for this study were selected purposely from the secondary data, the cases in the original study were selected from a convenience sample of math centre leaders who had attended a workshop for mathematics centre leaders, were interested in conducting research and had access to specific student data. These individuals are not representative of the broader mathematics tutor centre community. [<reflink idref="bib30" id="ref185">30</reflink>] argued for selection of 'any groups that will help generate, to the fullest extent, as many properties of the categories as possible, and that will help relate categories to each other and to their properties' (p. 49). Under diverse conditions, different aspects of the theoretical case will be salient. Future work can aim to flush out this work by examining additional cases which differ on key features of those representing in this work ([<reflink idref="bib16" id="ref186">16</reflink>]) for improved validity.</p> <p>Data were also collected for only a single fall semester. The quantitative data did not capture the full annual picture of a centre's performance. In addition, data examining tutor centres' performance over time could be used as an aspect of external validity for data triangulation or as a unit of analysis ([<reflink idref="bib23" id="ref187">23</reflink>]). As an aspect of external validity, additional sources of data would not only reduce investigator bias but also add additional angles with which to examine the phenomenon. As a unit of analysis, forming a longitudinal or extended case study ([<reflink idref="bib60" id="ref188">60</reflink>]) would have allowed for comparing and contrasting or performance within an institution. Future work can collect centre data over several semesters in order to both elaborate on the findings of this study and expand the work to demonstrate who longitudinal within centre data can aid in performance assessment. In addition, future work can examine the unit of analysis to be the broader university, conducting a case study on the collective ([<reflink idref="bib23" id="ref189">23</reflink>]). Tutor centres do not exist within a vacuum and are subject to institutional and societal influences. One could examine the processes by which the mathematics department culture or university structure impacts a tutor centre. Finally, additional studies are needed which utilize interview and survey techniques to collect data directly from students, tutors, faculty and administration, as well as researcher observations regarding tutor centre operations to add to depth of data and validity.</p> <p>Funding</p> <p>National Science Foundation Conference (DUE: 2645086).</p> <p>Acknowledgements</p> <p>Collaboration at National Science Foundation Conferences was the inspiration for this paper. The opinions expressed in this paper do not necessarily reflect the position, policy or endorsement of the supporting agency. We would also like to thank Cynthia Farthing and Daniel Moritz for their contributions to early phases of this research program.</p> <p> <bold>Carolyn Johns</bold> is the Assistant Director of the Mathematics and Statistics Learning Center at Ohio State. Her research interests broadly include mathematics tutor practices, tutor training and teacher education. Prior to working at The Ohio State University, she received an undergraduate degree from Earlham College and received a Ph.D. from The Ohio State University in August 2019.</p> <p> <bold>Cameron Byerley</bold> is an assistant professor at the University of Georgia. Formally, she co-directed a university mathematics tutoring center. Her research interests include modelling students' and citizens' mathematical thinking.</p> <p> <bold>Deborah Moore-Russo, Ph.D.</bold> is the First-Year Mathematics Director at the University of Oklahoma. Her primary research interests include the ways that mathematics topics are conceptualized, represented, visualized and communicated. She is also interested in how mathematics departments support first-year mathematics courses through tutoring centers, course coordination and other efforts.</p> <p> <bold>Brian Rickard</bold> earned a BS in Mathematics, MEd in Higher Education and Ph.D. in Educational Statistics and Research Methods at the University of Arkansas. He is a teaching assistant professor and coordinator of Finite Mathematics in the Department of Mathematical Sciences at the University of Arkansas. He conducts research in undergraduate mathematics education and university tutoring centers.</p> <p> <bold>Janet Oien</bold> is the Co-Director of the Calculus Center at Colorado State University. Her research interests broadly include mathematics tutor practices, tutor training and preparation of secondary mathematics teachers. She received her undergraduate degree and M.S. degree in Mathematics with an outside concentration in Education from Colorado State University in 2006.</p> <p> <bold>Linda Burks</bold> is the Director of the Mathematics Learning Center and a math lecturer at Santa Clara University. She received her undergraduate degree in mathematics from Virginia Tech; she earned her M.S. in mathematics and Ph.D. in mathematics education from the University of Michigan. Her research interests include mathematics tutor practices and cognitive self-regulation.</p> <p> <bold>Carolyn James, Ph.D.</bold>, is the Calculus Coordinator and Mathematics Resource Center Supervisor at University of Portland. She received her undergraduate degree in mathematics at Carleton College and her Ph.D. in mathematics education from Portland State University. Her research interests include barriers and drivers of instructional change, the relationship between instructor beliefs and practice and undergraduate mathematics tutoring.</p> <p> <bold>Melissa Mills</bold> is a teaching associate professor and the Director of the Mathematics Learning Success Center at Oklahoma State University. Her research interests broadly include mathematics tutor practices and the teaching and learning of mathematical proof at the undergraduate level.</p> <p> <bold>William Heasom, MA, MSCE</bold>, is assistant to the Director of the Mathematics Learning and Resource Center at Villanova University He also serves as collaborator with the IARPA FOCUS (forecasting counterfactuals in uncontrolled settings) project at University of Pennsylvania where he helped develop training materials for intelligence analysts. His research interests concern applying mathematical and agent-based modelling techniques to the analysis and design of hydrologic, ecological and learning systems.</p> <p> <bold>Melissa Ferreira</bold> is the Director of the Mathematics Learning and Resource Center at Villanova University. Her research interest is in university mathematics tutoring. She received undergraduate and masters' degrees in Mathematics from Villanova University.</p> <p> <bold>Behailu Mammo</bold> is a professor of mathematics and Director of the Mathematics Tutorial Center at Hofstra University, NY. His research interest focuses on number theory and examining the impact of tutoring in improving pre-service teachers' dispositions and self-efficacy. In May 2005, he received his Ph.D. from Temple University, PA. His bachelor's and master's degrees are from Ethiopia, East Africa.</p> <ref id="AN0162272513-19"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref14" type="bt">1</bibl> <bibtext> Akingbola, K. (2012) Context and nonprofit human resource management. Adm. Soc., 45, 974 – 1004. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib2" idref="ref19" type="bt">2</bibl> <bibtext> Arcavi, A. & Schoenfeld, A. H. (1992) Mathematics tutoring through a constructivist lens: the challenges of sense-making. J. Math. Behav., 11, 321 – 335. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibl id="bib3" idref="ref49" type="bt">3</bibl> <bibtext> Ball, D.L., Hill, H.C. & Bass, H. (2005) Knowing mathematics for teaching: who knows mathematics well enough to teach third grade, and how can we decide? Am. Educ., 29 14 – 17, 20–22, 43–46. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibl id="bib4" idref="ref50" type="bt">4</bibl> <bibtext> Ball, D. L., Thames, M. H. & Phelps, G. (2008) Content knowledge for teaching: what makes it special? J. Teach. Educ., 59, 389 – 407. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib5" idref="ref51" type="bt">5</bibl> <bibtext> Berry, E., Mac an Bhaird, C. & O'Shea, A. (2015) Investigating relationships between the usage of mathematics learning support and performance of at risk students. Teach. Math. Its Appl., 34, 194 – 204. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibl id="bib6" idref="ref52" type="bt">6</bibl> <bibtext> Booth, D. (2003) Open learning resource center at the University of Huddersfield in "math support for students". LTSN Mathstream. <ulink href="http://mathcentre.ac.uk/resources/mathsteam/booth.pdf">http://mathcentre.ac.uk/resources/mathsteam/booth.pdf</ulink> (accessed 18 March 2021). 16 – 17.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref36" type="bt">7</bibl> <bibtext> Bradshaw, P. (2009) A contingency approach to nonprofit governance. Nonprofit Manag. Leadersh., 20, 61 – 81. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib8" idref="ref147" type="bt">8</bibl> <bibtext> Braun, V. & Clarke, V. (2006) Using thematic analysis in psychology. Qual. Res. Psychol., 3, 77 – 101. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibl id="bib9" idref="ref2" type="bt">9</bibl> <bibtext> Bressoud, D. M., Mesa, V. & Rasmussen, C. L. (2015) Insights and Recommendations from the MAA National Study of College Calculus. MAA Press. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Byerley, C., Moore-Russo, D., James, C., Rickard, B., & Mills, M. (2019) Defining the varied structures of tutoring centers: laying the foundation for future research. The Twenty-Second Annual Conference on Research in Undergraduate Mathematics Education. Oklahoma City, OK. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Byerley, C., James, C., Moore-Russo, D., Rickard, B., Mills, M., Heasom, W., Oien, J., Farthing, C., Burks, L., Ferreira, M., Mammo, B., Moritz, D. (2020) Characteristics and evaluation of ten tutoring centers. The Twenty-Third Annual Conference on Research in Undergraduate Mathematics Education. Boston, MA. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Byerley, C., Johns, C., Moore-Russo, D., Rickard, B., James, C., Mills, M., Mammo, B., Oien, J., Burks, L., Heasom, W., Ferreira, M., Farthing, C., Moritz, D. (2021) (under review) Towards research-based suggestions for organizational decisions mathematics tutoring centre leader make. (under review).</bibtext> </blist> <blist> <bibtext> Byerley, C., Campbell, C. & Rickard, B. (2018) Evaluation of impact of calculus center on student achievement. Proceedings of the Twenty-first Annual Conference on Research in Undergraduate Mathematics Education. San Diego, CA, pp. 816 – 825. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Carmen, J. G. (2011) Understanding evaluation in nonprofit organizations. Public Perform. Manag. Rev., 34, 350 – 377. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Carroll, C. & Gill, O. (2012) An innovative approach to evaluation the University's mathematics learning centre. Teach. Math. Its Appl., 31, 199 – 214. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Charmaz, K. (2000) Grounded theory: objectivist and constructivist methods. The Handbook of Qualitative Research (N. K. Denzin & Y. Lincoln eds). Thousand Oaks, CA : Sage Publications, Inc., pp. 509 – 535. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Charmaz, K. (2014) Constructing Grounded Theory. London : SAGE Publications. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Chi, M. T. (1996) Constructing self-explanations and scaffolding explanations in tutoring. Appl. Cogn. Psychol., 10, S33 – S49. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Child, J. (1972) Organizational structure, environment and performance: the role of strategic choice. Sociology, 6, 1 – 22. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Cooney, T. J., Shealy, B. E. & Arvold, B. (1998) Conceptualizing belief structures of preservice secondary mathematics teachers. Math. Educ. Res. J., 29, 306 – 333. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Croft, A. C. (2009) Towards a culture of data collection and analysis in mathematics support centres. Queensland, Australia. <ulink href="http://www.sigma-cetl.ac.uk/index.php?section=86">http://www.sigma-cetl.ac.uk/index.php?section=86</ulink> (accessed 18 March 2021).</bibtext> </blist> <blist> <bibtext> Croft, A. C., Grove, M. & Bright, D. (2008) A resource and activity centre for mathematics students beyond their transition to higher education. MSOR Connections, 8, 11 – 16. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Denzin, N. K. (1970) The Research Act in Sociology: A Theoretical Introduction to Sociological Method. London : Butterworth. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Donaldson, L. (1996) For Positivist Organization Theory: Proving the Hard Core. UK : SAGE Publications. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Dowling, D. & Nolan, B. (2006) Measuring the effectiveness of a maths learning support centre—the Dublin City University experience. CETL-MSOR Conference 2006 Conference Proceedings Published by The Maths, Stats, and OR Network Printed by Central Printing Services. University of Birmingham, pp. 51 – 54. https://<ulink href="http://www.sigma-network.ac.uk/wp-content/uploads/2016/11/CETL-MSOR-Proceedings2006.pdf">www.sigma-network.ac.uk/wp-content/uploads/2016/11/CETL-MSOR-Proceedings2006.pdf</ulink> (accessed 18 March 2021).</bibtext> </blist> <blist> <bibtext> Dzator, M. & Dzator, J. (2020) The impact of mathematics and statistics support at academic learning centre, Central Queensland University. Teach. Math. Its Appl., 39, 13 – 28. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Eisenhardt, K. M. (1989) Building theories from case study research. Acad. Manag. Rev., 14, 532 – 550. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Gill, O. & O'Donoghue, J. (2007) Justifying the existence of mathematics learning support: measuring the effectiveness of a mathematics learning centre. Paper Presented at Adults Learning Mathematics 14 (ALM 14), Limerick, June 26–29, pp. 154 – 164.</bibtext> </blist> <blist> <bibtext> Gillard, J., Robathan, K. & Wilson, R. (2011) Measuring the effectiveness of a mathematics support service: an email survey. Teach. Math. Its Appl., 30, 43 – 52. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Glaser, A. L. & Strauss, B. G. (1968) Nursing Research. Chicago : Aldine Publications. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Graesser, A. C., Person, N. K. & Magliano, J. P. (1995) Collaborative dialogue patterns in naturalistic one-on-one tutoring. Appl. Cogn. Psychol., 9, 495 – 522. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Herman, R. D. & Renz, D. O. (2008) Advancing nonprofit organizational effectiveness research and theory. Nonprofit Manag. Leadersh., 18, 399 – 415. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Jacob, M. & Ni Fhloinn, E. (2019) A quantitative, longitudinal analysis of the impact of mathematics support in an Irish university. Teach. Math. Its Appl., 38, 216 – 229. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Karsenty, R. & Sherin, M. G. (2017) Video as a catalyst for mathematics teachers' professional growth. J. Math. Educ., 20, 409 – 413. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> King, N. & Brooks, J. (2018) Thematic analysis in organisational research. The SAGE Handbook of Qualitative Business and Management Research Methods: Methods and Challenges (C. Cassell, A. L. Cunliffe & G. Grandy eds). UK : SAGE, pp. 219 – 236. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Kyle, J. (2010) Affordability, adaptability, approachability and sustainability. Responding to the Mathematics Problem (C. M. Marr & M. J. Grove eds). St Andrews : Maths, Stats & OR Network, pp. 103 – 104. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Lawson, D. A. (2015) Mathematics support at the transition to university. Transitions in Undergraduate Mathematics Education (M. J. Grove, A. C. Croft, J. Kyle & D. A. Lawson eds). Birmingham : The University of Birmingham & Higher Education Academy, pp. 39 – 56. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Lawson, D., Grove, M. & Croft, T. (2020) The evolution of mathematics support: a literature review. Int. J. Math. Educ. Sci. Technol., 51, 1224 – 1254. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lee, C. & Nowell, B. (2015) A framework for assessing the performance of nonprofit organizations. Am. J. Eval., 36, 299 – 319. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Lepper, M. R. & Woolverton, M. (2002) The wisdom of practice: lessons learned from the study of highly effective tutors. Improving Academic Achievementt: Impact of Psychological Factors on Education (J. Aronson ed.). San Diego, CA, USA : Academic Press, pp. 135 – 158. https://doi.org/10.1016/B978-012064455-1/50010-5. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Mac an Bhaird, C., Gill, O., Jennings, K., Ni Fhloinn, E., O'Sullivan, C. (2011) The Irish mathematics support network: its origins and progression. AISHE-J, 3, 51.1 – 51.14. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> MacGillivray, H. L. & Croft, A. C. (2011) Understanding evaluation of learning support in mathematics and statistics. Int. J. Math. Educ. Sci. Technol., 42, 189 – 212. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Matthews, J., Croft, A. C., Lawson, D. A. & Waller, D. (2013) Evaluation of mathematics support centres: a literature review. Teach. Math. Its Appl., 32, 173 – 190. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Mills, M., Rickard, B. & Guest, B. (2020) Survey of mathematics tutoring centers in the USA. Int. J. Math. Sci. Educ. Technol., 1 – 21. Published online. https://doi.org/10.1080/0020739X.2020.1798525. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Mintzberg, H. (1980) Structure in 5's: a synthesis of the research on organization design. Manag. Sci., 26, 322 – 341. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Mitchell, J. C. (1983) Case and situation analysis. Sociol. Rev., 31, 187 – 211. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Ni Fhloinn, E. (2009) The role of student feedback in evaluating mathematics support centers. Proceedings of the CETL-MSOR Conference 2009. Milton Keynes, UK, pp. 94 – 98 <ulink href="http://doras.dcu.ie/21665/1/NiFhloinn2009.pdf">http://doras.dcu.ie/21665/1/NiFhloinn2009.pdf</ulink> (accessed 18 March 2021).</bibtext> </blist> <blist> <bibtext> O'Sullivan, C., Macan Bhaird, C., Fitzmaurice, O. & Ni Fhlionn, E. (2014) An Irish Mathematics Learning Support Network (IMLSN) Report on Student Evaluation of Mathematics Learning Support: Insights from a Large Scale Multi-Institutional Survey. Ireland : National Centre for Excellence in Mathematics and Science Teaching and Learning. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Parsons, S., Croft, A. C. & Harrison, M. (2011) Engineering students' self-confidence in mathematics mapped onto Bandura's self-efficacy. J. Eng. Educ., 6, 52 – 61. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Rickard, B. & Mills, M. (2018) The effect of attending tutoring on course grades in Calculus I. Int. J. Math. Educ. Sci. Technol., 49, 341 – 354. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Rojas, R. R. (2000) A review of models for measuring organizational effectiveness among for-profit and nonprofit organizations. Nonprofit Manag. Leadersh., 1, 97 – 104. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Rossi, P. H., Lipsey, M. W. & Freeman, H. E. (2004) Evaluation: A Systemic Approach, 7th edn. London : SAGE Publishing. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Rylands, I. J. & Shearman, D. (2018) Mathematics learning support and engagement in first year engineering. Int. J. Math. Educ. Sci. Technol., 49, 1133 – 1147. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Santagata, R. & Guarino, J. (2011) Using video to teach future teachers to learn from teaching. ZDM Math. Educ., 43, 133 – 145. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Santagata, R. & Yeh, C. (2014) Learning to teach mathematics and to analyze teaching effectiveness: evidence from a video and practice-based approach. J. Math. Teach. Educ., 17, 491 – 514. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Schoenfeld, A. H. (2017) Use of video in understanding and improving mathematical thinking and teaching. J. Math. Teach. Educ., 20, 415 – 432. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Schon, D. A. (1987) Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. San Francisco, CA : Jossey-Bass. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Smith, E. (2008) Pitfalls and promises: the use of secondary data analysis in educational research. Br. J. Educ. Stud., 56, 323 – 339. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Staddon, L. & Newman, L. (2003) The education drop-in centre at the University of Glamorgan, in "maths support for students". LTSN mathstream, 22 – 23.</bibtext> </blist> <blist> <bibtext> Thomas, G. & Myers, K. (2015) The Anatomy of the Case Study. Los Angeles : SAGE Publications. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Thompson, A. G. (1992) Teachers' beliefs and conceptions: a synthesis of the research. Handbook of Research on Mathematics Teaching and Learning (D. A. Grouws ed). New York : Macmillan, pp. 127 – 146. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Toepler, S. & Anheier, H. K. (2004) Organizational theory and nonprofit management: an overview. Future of Civil Society (A. Zimmer & E. Prillar eds). Wiesbaden : VS, pp. 253 – 270. Google Scholar Crossref Search ADS Google Preview WorldCat COPAC</bibtext> </blist> <blist> <bibtext> Van Es, E. A. & Sherin, M. G. (2008) Mathematics teachers' "learning to notice" in the context of a video club. Teach. Teach. Educ., 24, 244 – 276. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Van Es, E. & Sherin, M. G. (2010) The influence of video clubs on teachers' thinking and practice. J. Math. Teach. Educ., 13, 155 – 176. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Wadongo, B. & Abdel-Kader, M. (2014) Contingency theory, performance management and organisational effectiveness in the third sector: a theoretical framework. Int. J. Product. Perform., 63, 680 – 703. Google Scholar Crossref Search ADS WorldCat</bibtext> </blist> <blist> <bibtext> Wilkins, L. (2015) Maybe we could just count the boxes of chocolates? Measuring the impact of learning development mathematics support for undergraduate students. JALL, 9, A91 – A115. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Xu, Y., Hartman, S., Uribe, G. & Mencke, R. (2001) The effects of peer tutoring on undergradaute examination scores in mathematics. J Coll Read, 32, 22 – 31. Google Scholar OpenURL Placeholder Text WorldCat</bibtext> </blist> <blist> <bibtext> Yin, R. K. (2018) Case Study Research and Applications: Design and Methods, 6th edn. Los Angeles, CA : SAGE Publications, Inc. Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC</bibtext> </blist> </ref> <aug> <p>By Carolyn Johns; Cameron Byerley; Deborah Moore-Russo; Brian Rickard; Janet Oien; Linda Burks; Carolyn James; Melissa Mills; William Heasom; Melissa Ferreira and Behailu Mammo</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib43" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib67" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib13" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib50" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib10" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib38" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib11" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib19" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib24" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib39" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib51" firstref="ref22"></nolink> <nolink nlid="nl12" bibid="bib62" firstref="ref23"></nolink> <nolink nlid="nl13" bibid="bib65" firstref="ref25"></nolink> <nolink nlid="nl14" bibid="bib12" firstref="ref31"></nolink> <nolink nlid="nl15" bibid="bib45" firstref="ref32"></nolink> <nolink nlid="nl16" bibid="bib32" firstref="ref38"></nolink> <nolink nlid="nl17" bibid="bib31" firstref="ref55"></nolink> <nolink nlid="nl18" bibid="bib18" firstref="ref56"></nolink> <nolink nlid="nl19" bibid="bib40" firstref="ref57"></nolink> <nolink nlid="nl20" bibid="bib44" firstref="ref59"></nolink> <nolink nlid="nl21" bibid="bib42" firstref="ref64"></nolink> <nolink nlid="nl22" bibid="bib59" firstref="ref67"></nolink> <nolink nlid="nl23" bibid="bib25" firstref="ref68"></nolink> <nolink nlid="nl24" bibid="bib28" firstref="ref69"></nolink> <nolink nlid="nl25" bibid="bib21" firstref="ref70"></nolink> <nolink nlid="nl26" bibid="bib53" firstref="ref76"></nolink> <nolink nlid="nl27" bibid="bib33" firstref="ref77"></nolink> <nolink nlid="nl28" bibid="bib22" firstref="ref80"></nolink> <nolink nlid="nl29" bibid="bib29" firstref="ref81"></nolink> <nolink nlid="nl30" bibid="bib49" firstref="ref82"></nolink> <nolink nlid="nl31" bibid="bib15" firstref="ref83"></nolink> <nolink nlid="nl32" bibid="bib66" firstref="ref84"></nolink> <nolink nlid="nl33" bibid="bib26" firstref="ref85"></nolink> <nolink nlid="nl34" bibid="bib37" firstref="ref88"></nolink> <nolink nlid="nl35" bibid="bib47" firstref="ref89"></nolink> <nolink nlid="nl36" bibid="bib36" firstref="ref90"></nolink> <nolink nlid="nl37" bibid="bib41" firstref="ref94"></nolink> <nolink nlid="nl38" bibid="bib48" firstref="ref95"></nolink> <nolink nlid="nl39" bibid="bib52" firstref="ref112"></nolink> <nolink nlid="nl40" bibid="bib27" firstref="ref121"></nolink> <nolink nlid="nl41" bibid="bib58" firstref="ref124"></nolink> <nolink nlid="nl42" bibid="bib68" firstref="ref125"></nolink> <nolink nlid="nl43" bibid="bib46" firstref="ref126"></nolink> <nolink nlid="nl44" bibid="bib30" firstref="ref133"></nolink> <nolink nlid="nl45" bibid="bib60" firstref="ref136"></nolink> <nolink nlid="nl46" bibid="bib17" firstref="ref146"></nolink> <nolink nlid="nl47" bibid="bib35" firstref="ref150"></nolink> <nolink nlid="nl48" bibid="bib61" firstref="ref156"></nolink> <nolink nlid="nl49" bibid="bib20" firstref="ref157"></nolink> <nolink nlid="nl50" bibid="bib63" firstref="ref159"></nolink> <nolink nlid="nl51" bibid="bib64" firstref="ref160"></nolink> <nolink nlid="nl52" bibid="bib54" firstref="ref161"></nolink> <nolink nlid="nl53" bibid="bib55" firstref="ref162"></nolink> <nolink nlid="nl54" bibid="bib34" firstref="ref163"></nolink> <nolink nlid="nl55" bibid="bib56" firstref="ref164"></nolink> <nolink nlid="nl56" bibid="bib57" firstref="ref165"></nolink> <nolink nlid="nl57" bibid="bib14" firstref="ref176"></nolink> <nolink nlid="nl58" bibid="bib16" firstref="ref186"></nolink> <nolink nlid="nl59" bibid="bib23" firstref="ref187"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1372619
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Performance Assessment for Mathematics Tutoring Centres
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Johns%2C+Carolyn%22">Johns, Carolyn</searchLink><br /><searchLink fieldCode="AR" term="%22Byerley%2C+Cameron%22">Byerley, Cameron</searchLink><br /><searchLink fieldCode="AR" term="%22Moore-Russo%2C+Deborah%22">Moore-Russo, Deborah</searchLink><br /><searchLink fieldCode="AR" term="%22Rickard%2C+Brian%22">Rickard, Brian</searchLink><br /><searchLink fieldCode="AR" term="%22Oien%2C+Janet%22">Oien, Janet</searchLink><br /><searchLink fieldCode="AR" term="%22Burks%2C+Linda%22">Burks, Linda</searchLink><br /><searchLink fieldCode="AR" term="%22James%2C+Carolyn%22">James, Carolyn</searchLink><br /><searchLink fieldCode="AR" term="%22Mills%2C+Melissa%22">Mills, Melissa</searchLink><br /><searchLink fieldCode="AR" term="%22Heasom%2C+William%22">Heasom, William</searchLink><br /><searchLink fieldCode="AR" term="%22Ferreira%2C+Melissa%22">Ferreira, Melissa</searchLink><br /><searchLink fieldCode="AR" term="%22Mammo%2C+Behailu%22">Mammo, Behailu</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Teaching+Mathematics+and+Its+Applications%22"><i>Teaching Mathematics and Its Applications</i></searchLink>. Mar 2023 42(1):1-29.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://teamat.oxfordjournals.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 29
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Performance+Based+Assessment%22">Performance Based Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Instruction%22">Mathematics Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Tutoring%22">Tutoring</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Nonprofit+Organizations%22">Nonprofit Organizations</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Tutorial+Programs%22">Tutorial Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Organizational+Change%22">Organizational Change</searchLink><br /><searchLink fieldCode="DE" term="%22Organizational+Climate%22">Organizational Climate</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1093/teamat/hrab032
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0268-3679<br />1471-6976
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Evaluation has become a common, and even an expected, practice across undergraduate mathematics tutoring centres in the USA, UK and other countries. However, these evaluation efforts could benefit greatly from leveraging organizational theory research. In this study, we situate mathematics tutoring centres as non-profit organizations (NPOs) to consider how an organization performance assessment framework can be adapted to study mathematics tutoring centre data and characteristics. We use qualitative and quantitative data, collected from six mathematics tutoring centres and adapt Lee & Nowell's (2015, "Am. J. Eval.," 36, 299-319) NPO performance framework to situate our study. Using thematic analysis, the research team underwent iterative cycles of data collection and analysis to code for alignment with Lee and Nowell's framework. By adapting Lee and Nowell's framework to mathematics centres, each of the six centres was given a more relevant lens to consider its performance. Regardless of the university setting, previous evaluations for centres have focused primarily on outputs (e.g., number of visits), behavioural change outcomes (e.g., correlating visits to grades) and client satisfaction outcomes (e.g., student surveys) that ignore the particular context of a centre. However, Lee and Nowell's framework takes into account additional performance indicators that provide a more nuanced understanding of a centre's performance by bringing to light the interplay among its various dimensions. Lee and Nowell's framework allows centres to look beyond outputs and outcomes to understand why these outputs and outcomes come to be. The use of this adapted performance framework, for the six mathematics centres in this study, allows an interpretation on a variety of dimensions using relevant data while indicating possible areas for change for each centre.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1372619
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1372619
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/teamat/hrab032
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 29
        StartPage: 1
    Subjects:
      – SubjectFull: Performance Based Assessment
        Type: general
      – SubjectFull: Mathematics Instruction
        Type: general
      – SubjectFull: Tutoring
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Nonprofit Organizations
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Tutorial Programs
        Type: general
      – SubjectFull: Organizational Change
        Type: general
      – SubjectFull: Organizational Climate
        Type: general
    Titles:
      – TitleFull: Performance Assessment for Mathematics Tutoring Centres
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Johns, Carolyn
      – PersonEntity:
          Name:
            NameFull: Byerley, Cameron
      – PersonEntity:
          Name:
            NameFull: Moore-Russo, Deborah
      – PersonEntity:
          Name:
            NameFull: Rickard, Brian
      – PersonEntity:
          Name:
            NameFull: Oien, Janet
      – PersonEntity:
          Name:
            NameFull: Burks, Linda
      – PersonEntity:
          Name:
            NameFull: James, Carolyn
      – PersonEntity:
          Name:
            NameFull: Mills, Melissa
      – PersonEntity:
          Name:
            NameFull: Heasom, William
      – PersonEntity:
          Name:
            NameFull: Ferreira, Melissa
      – PersonEntity:
          Name:
            NameFull: Mammo, Behailu
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 0268-3679
            – Type: issn-electronic
              Value: 1471-6976
          Numbering:
            – Type: volume
              Value: 42
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Teaching Mathematics and Its Applications
              Type: main
ResultId 1