View in EDS HTML Full Text PDF Full Text

What's It Like When the Participatory Evaluator Is a 'Genuine' Stakeholder?

Saved in:
Bibliographic Details
Title: What's It Like When the Participatory Evaluator Is a 'Genuine' Stakeholder?
Language: English
Authors: Mathison, Sandra
Source: American Journal of Evaluation. Win 2001 22(1):29-35.
Peer Reviewed: Y
Page Count: 7
Publication Date: 2001
Document Type: Journal Articles
Reports - Descriptive
Descriptors: After School Programs, Evaluation Methods, Evaluation Utilization, Evaluators, Interpersonal Relationship, Parents, Stakeholders
ISSN: 1098-2140
Abstract: Discusses what it means to be a caring stakeholder and a participatory evaluator in the context of public and private domains using the example of a parent who is an evaluator of an afterschool program. Offers a perspective on the capacity of individual evaluators to deal with role strain. (SLD)
Entry Date: 2002
Accession Number: EJ640223
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFOzEuj9oZcZ6PEjLcNwQU5AAAA4TCB3gYJKoZIhvcNAQcGoIHQMIHNAgEAMIHHBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDN_oQlFMrobYV-GgdAIBEICBmQZ3LfkickNIkmdF9DCIJwJQu6ZhcuUTGC_votB4m6AUP0ALfbvccdUxir6O_50ySZ-c5cH2gYZal1oohzZHgE7LN4-oItgPNc0RzQ04jbp9cGMoZrO1MSYD8TWhciZ5L8xecqt7bXfCslyg9xRbcjhBfvccbgcZ3OtgS7A6lNQZ3rW_7C-IBhWZuzflSue6tYD9ZoHq9pansw==
Text:
  Availability: 1
  Value: <anid>AN0005231246;3SD01JAN.01;2001Sep27.08:05;v5.4</anid> <title id="AN0005231246-1">WHAT'S IT LIKE WHEN THE PARTICIPATORY EVALUATOR IS A 'GENUINE' STAKEHOLDER? </title> <hd id="AN0005231246-2"> ABSTRACT </hd> <p>In this reflective, analytic story I discuss what it is like to be a parent and an evaluator in a participatory, democratic evaluation of an afterschool program. The story tells of tensions between these roles, and how these tensions can instruct participatory evaluators in pursuing goals of democracy, community, and justice in evaluation. This discussion can be situated among a number of recent articles in AJE that have focused on analyses of participatory or empowerment evaluation. (See Schnoes, Murphy--Berman, & Chambers, 2000; and the exchanges between Smith, 1999 and Cousins & Earl, 1999.) </p> <hd id="AN0005231246-3">INTRODUCTION</hd> <p>I approach the double aluminum doors of the school gymnasium, predisposed to be annoyed. It is a cold New York winter day and with my gloves still on I bang on the door with the palm of my hand. To knock would be fruitless. Inside the gymnasium, I hear yelling, laughter, the sounds of children in the Afterschool Program. I wait for someone to open the door, but no one does. More annoyed now, I bang harder on the door. And wait, again. Eventually, the door is pushed open, slowly, as if in slow motion. It opens slowly because it is heavy for the little girl, maybe 5 or 6 years old, to push open. I thank her and, in my mind, I run through my reactions to this daily occurrence. </p> <p>I am annoyed at the school for making it difficult for parents to pick up their children at the end of the day. Yes, I have heard the discussion about school safety, but isn't it just a ruse, because small children are the ones to open the door--it isn't necessarily monitored by a responsible adult. I am annoyed with the Afterschool Program staff for complying with the school, for being too gutless to advocate for smoother operations of their program. I sign out my son on the sheet provided, few words are exchanged with the staff, and I convince him it is time to go home. He is reluctant to leave; he is having a good time playing with his friends. </p> <p>As we leave, I think well, finally, the access issue will be resolved now that I and my doctoral students will be doing an evaluation of this Afterschool Program. During the spring semester, I and a group of doctoral students, enrolled in an advanced course on participatory evaluation, conducted an evaluation of this program. I, as the professor, negotiate pro bono evaluations for community agencies to provide students with "real" experiences doing evaluation. In this course, and in these evaluations, the focus is on participatory evaluation that involves as many stakeholders as possible in all aspects of the evaluation, including deciding what the evaluation ought to focus on, questions to be asked, kinds of data to be collected, and how to communicate the understandings that develop from the evaluation. Guba and Lincoln's (1989) Fourth Generation Evaluation is a main text, and we use the construction of claims, concerns, and issues to start, focus, and continue the evaluation. The intent of these participatory evaluations is to create dialogue and deliberation among stakeholders, with an eye to empowering those who may have been disenfranchised in the program context, and to create a sense of community responsibility for quality programming. We use a form of participatory evaluation, sometimes referred to as transformative (see House & Howe, 1999, Mathison, 1996, 2000, and Whitmore, 1998 for more detail on the approach to participatory evaluation referred to here). We use deliberative forums extensively throughout these evaluations (see MacNeil, 2000 for a description of deliberative forums). </p> <p>I hesitated about this particular evaluation. I approached the Child Care Director for the YMCA, which runs most of the city's afterschool programs, about possible evaluations. Although there are many options within the Y's programming, the director was keen to have the students look at what she considered her best afterschool program. I resisted, suggesting alternatives. She insisted, feeling it would be a great experience for all. I relented, and we moved ahead with the evaluation. </p> <p>It is my practice to supervise the evaluations in these courses, to work with the doctoral students in planning and conducting the evaluations, but to leave most of the field work to the students. In the past, even when things have been rocky, when my personal intervention would perhaps have made things go more smoothly, I have resisted direct intervention (offering instead counsel and advice) and allowed students to do their own work. I was to discover that I had been able to maintain this distance, at least in part, because I was not what I would call a "genuine" stakeholder in previous programs. Not so for the Afterschool Program. My role in this evaluation deviated from those I have done as part of the class experience in the past. I took a substantial role in doing the evaluation, including collecting data, conducting deliberative forums with stakeholders, and writing reports. I acted as an evaluator, not just a supervisor of an evaluation class project. Various stakeholders gave either my evaluator or parent self more prominence--I believe other parents and school staff saw me as a parent, while the staff and director of the Afterschool Program saw me more often as an evaluator. </p> <p>My son, 5 years old, goes to this Afterschool Program. I am a parent who has been actively involved in my son's care and schooling; his caregivers and teachers can be assured of my involvement. When I enrolled my son in this program, I requested a meeting with the Child Care Director to inquire about the specifics of the program. At the time, she remarked how unusual my request was. In this program, I am a "genuine" stakeholder (in the sense of real, not necessarily good). * </p> <p>Guba and Lincoln (1989) quite rightly tell us that evaluators are stakeholders too. But, in the end, evaluators are "artifactual" stakeholders (in the sense that their stake is created by the existence of an evaluation, not necessarily that it is accidental or bad). Evaluators do not have a fundamental stake in the program--their stake ends when the evaluation ends. When we completed the evaluation, my students no longer had a stake in the program--they were finished with it and gave little thought to the operations and future of the Afterschool Program. I, however, continue to be a stakeholder because my son still goes to the Afterschool Program. In fact, my longevity as a stakeholder extends beyond that of most on-site staff of the program, many of whom work for a year and sometimes less in the program. </p> <hd id="AN0005231246-4">OVERCOMING THE NIMBY (NOT-IN-MY-BACK-YARD) PHENOMENON</hd> <p>One of the reasons I relented on this evaluation was some reflection on what the value of democratic, participatory forms of evaluation is. In my teaching and scholarship, I promote the notion that evaluation is about building community, about diverse stakeholder groups working through visions of both what is and what ought to be, that through this process we can arrive at more fair, more just, more caring communities within which we live and work. If this is good for others, I reasoned, I should be committed enough to the idea that it is good for me, too. Good not just from my perspective as an evaluator, but also from my perspective as a citizen. </p> <p>NIMBYs are folks who advocate for all sorts of social goods, in principle--organic pest control, community group homes, reduction in fossil fuel use--but when it impacts their lives personally these goods lose their value. When the rainy summer produces an infestation of mosquitoes or when a community home for AIDS patients moves in next door, the NIMBYs' social responsibility is subverted to individual rights. Vowing not to be a NIMBY, I and my students embarked on the evaluation of the Afterschool Program. </p> <hd id="AN0005231246-5">PRIVATE VERSUS PUBLIC</hd> <p>The notion of public and private domains is discussed in cultural anthropology, especially of other cultures, and often related to gender (see, e.g., Rosaldo & Lamphere, 1974). One common manifestation of this distinction is the domains of work (as public) and home (as private). This distinction interrelates with evaluation practice in a number of ways. Most important for this discussion is the observation that as professional evaluators doing evaluation we operate in a public domain; this is our work life. Others in an evaluation, such as service recipients, may be operating in a private domain--for example, when a mental health program strives to transition hospitalized, mentally ill patients back into the community by providing support with finances, finding a place to live, shopping for clothes, and so on. Such issues are more connected to home life. </p> <p>Culturally speaking, both the private and public domains are rule-governed and open for inquiry, although what is apparent from ethnographies of other cultures is the sometimes greater difficulty in accessing the private domain, such as when one is interested in Moslem women, for example. It is called the private domain because it is a realm of social life within which we can somehow be ourselves, not having to provide explanations and rationales to outsiders for who we are, what we believe, and how we act. We have the veil of privacy. </p> <p>Evaluation practice, it seems, is part of the cultural public domain, but intrudes on the private domain of some, at least some of the time. The meeting of the public and private domains for me personally became clear in the Afterschool Program evaluation. My role as the overseer of this evaluation is clearly in the public domain, but my role as a parent is clearly in the private domain. </p> <p>What I realized in this evaluation is the much greater sense of vulnerability that "genuine" stakeholders experience when an evaluation ventures into their private domain. Matters that we assume to be private become public, and possibly open to criticism. For example, our expectations about what afterschool care ought to be may reflect a range of possibilities in child-rearing practices, for example, how discipline ought to be handled. One issue in this evaluation was the overuse of "time-out" as the means for disciplining kids. While it was not my sense that my son was subjected to time-outs, I perceived the possibility that he could be, and as such perceived (along with many other parents) the use of time-out to be excessive and indiscriminate. Other parents did not share this view--one father said, during a deliberative forum, that time-out made absolutely good sense because this is a practice used in all American homes, and therefore kids are familiar with the form of discipline. Time-out is never used in my home. Rather than using time-out, I might suggest a strategy of helping kids find alternative activities that are engaging and that take them out of the situation that is creating the misbehavior. </p> <p>An evaluator operating within the public domain can and does deal with such differences in perception, perhaps by collecting more data or mediating these differences, to come to a sense of what might be. As a parent and evaluator in this situation, the importance of these differences is apparent to me in a new way. The "time out is OK" view is a threat to my parental conception of good child-rearing practices--for my child especially. I, as a parent, want to defend and promote my position; I, as an evaluator, want all stakeholders to deliberate about discipline, with an eye to mutual respect and ultimately to agreement about how to handle discipline. I assume these intents need not be mutually exclusive; but this experience of being a genuine stakeholder has created for me a new appreciation for the complexity of how and to what end stakeholders are asked to share and make sense of their own and others' perspectives. </p> <p>It is clear from just this one example that otherwise noncontroversial issues often mask more deeply held beliefs. For example, a behaviorist perspective on learning and development may underlie the views of the parent who felt comfortable with the use of time-out, and a more humanist perspective may underlie the strategies I as a parent would feel comfortable with. As evaluators promoting dialogue and deliberation among stakeholders, we need to recognize that we will venture into potentially different foundational beliefs--a very substantial challenge. The lesson here is not to avoid or fear this likelihood, but rather to anticipate that deliberation will be about these values and to be respectful but open about that as the focus. Although our deliberative forum did not venture into a discussion of perspectives on learning and development, it could have. Perhaps it should have. This experience suggests to me these foundational beliefs are a critical element in creating a community that has a reasonably coherent view of what it will accept, with little grumbling, as good practice. As an evaluator, operating within one's private domain is a clarifying and humbling experience--one that is instructive for all of us who advocate participatory forms of evaluation. </p> <hd id="AN0005231246-6">SILENCING AS A PARENT/ADVOCATING AS AN EVALUATOR</hd> <p>On an afternoon while we were conducting the evaluation, I arrive to pick up my son. I find all of the children seated within a circle in the middle of the gymnasium. They are silent--no children speak to one another and no staff person is speaking to them--and they are spaced about arms-length from one another. I pause, wondering what sort of activity this is. As I look for the sheet to sign out my son, I ask a staff person, "What are they doing?" Looking a little sheepish, she replies, "They are having a quiet time." A number of thoughts run through my mind--why would sitting doing nothing constitute quiet time?--and then I realize they are being punished. This is "group time out." I, as a parent, am immediately outraged--the principle of punishing all kids for the violations of some seems to me an unsophisticated and unethical response. Really I am thinking this for only a matter of seconds and I mutter, "You are kidding, right?" The staff person tells me then the kids just couldn't listen, and so the only way to deal with them was to make them sit silently. I stare at her in amazement, pausing as I contemplate two alternative responses: (<reflink idref="bib1" id="ref1">1</reflink>) as a parent, by pursuing the matter further on the spot, asking questions, and directly challenging this practice judging it to be inappropriate, or (<reflink idref="bib2" id="ref2">2</reflink>) as an evaluator, taking in the information and thinking about ways to explore the issue further as the evaluation progresses. </p> <p>I respond in the latter way, although this is punctuated by my saying to my son in a too loud voice, "Come on Colin, it is time to go home." My initial response in this situation was to feel frustrated and angry as a parent, because my role as an evaluator took precedence and therefore silenced me as a parent. I appeased my parent self by invoking my evaluator self who has great confidence that the participatory evaluation approach would out these practices, and that as a lone parent I need not be the conscience of the Afterschool Program. </p> <p>Participatory, deliberative forms of evaluation have great potential to create and sustain a community that values certain principles, activities, and actions. In this moment of tension between responding as a parent or as an evaluator, lies a decision about what might be the best long-term strategy for dealing with programmatic issues. Many social programs (unintentionally, I think) remain inadequate because people respond as individuals rather than as a collective. In this case, pitching a fit as a parent can be easily contained by the program director and staff, and it is likely that my dissatisfaction with this "group time out" strategy would remain hidden from most other parents. Raising the issue as part of the participatory evaluation, which provides continual opportunities through deliberative forums for all stakeholders to put issues and concerns on the table for discussion, makes the matter public and one that must be dealt with by the collective. </p> <p>As an aside, but to illustrate the overwhelming tendency to respond in individualist ways, during the year following this evaluation the Afterschool Program decided midyear that kids in Pre-K could not attend. This left four families in a bind for afterschool care, and with very short notice to find an alternative. Several parents without a Pre-K child and those with Pre-K children lobbied local politicians, who helped us get a waiver from a never before enforced regulation, so that Pre-K children could continue to attend the Afterschool Program. We asked all parents to sign a petition to help us in this effort, and communicated our thanks to them in a letter. A parent of a 4th grader came up to me and asked, "Isn't Colin in kindergarten? Why would you care about this? He wasn't going to get kicked out of the program, was he?" This parent marveled at the suggestion that, as parents with children in the Afterschool Program, we all bear some responsibility for the whole program, not just the circumstances for our own children. </p> <hd id="AN0005231246-7">THE NEW SCHOOL YEAR</hd> <p>During the second week of school, I arrive to pick up my son from the Afterschool Program. I approach the double aluminum doors of the school gymnasium, and bang on the door with the palm of my hand. To knock would be fruitless. Inside the gymnasium, I hear yelling, laughter, the sounds of happy children. I wait for someone to open the door, but no one does. More annoyed now, I bang harder on the door. And wait, again. When someone finally opens the door, I meet the new site supervisor (the fourth site supervisor in less than a year). In passing, I mention last year's evaluation to her. Somewhat to my surprise she says, "Yes, I've read the report," and proceeds to tell me how some of the staff didn't like some aspects of the evaluation process. While acknowledging some things could have been done better and with greater sensitivity to the staff, I ask her what her thoughts are on the issues raised in the evaluation. We have a sustained conversation about changes she feels have and can be made, I push on some other issues which she doesn't mention, and I offer to help her and the staff in whatever way I can to continue to think about and act on issues and concerns raised. It is a positive interaction, and now I have acted as my evaluator self in encouraging the YMCA Afterschool Program to use the evaluation, advocating for the potential and usability of evaluation, and as my parent self in doing what I can to assure a good childcare option for my son. </p> <p>Operating as a genuine stakeholder and an evaluator in this context often required choosing a dominant role at any given time--always with a feeling of frustration and conflict. But over time and on balance the cost of making such choices seems modest. My contributions to the evaluation of the Afterschool Program as a parent and as an evaluator were different, and as a total package enhanced the value and sustainability of the evaluation. I'm not a NIMBY, and I have new and convincing evidence of the value and possibilities of participatory evaluation. </p> <p> <bold> American Journal of Evaluation, </bold> Vol. 22, No. 1, 2001, pp. 29-35. All rights of reproduction in any form reserved. ISSN: 1098-2140 Copyright (C) 2001 by American Evaluation Association. </p> <hd id="AN0005231246-8">NOTES</hd> <p>The evaluation of the YMCA Afterschool Program was conducted in collaboration with Chris Cruz, Peggy Jenkins, Rick Parkany, and Shelly Withers, doctoral students in the Department of Educational Theory and Practice, State University of New York at Albany. </p> <hd id="AN0005231246-9">REFERENCES</hd> <p>Cousins, J. B., & Earl, L. M. (1999). When the boat gets missed: Response to M. F. Smith. American Journal of Evaluation, 20(<reflink idref="bib2" id="ref3">2</reflink>), 309-17. </p> <p>Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage. </p> <p>House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks: Sage. </p> <p>Mathison, S. (1996). Evaluation as a democratizing force in schools. The International Journal of Social Education, 11, 40-48. </p> <p>Mathison, S. (2000). Promoting democracy through evaluation. In D. Hursh & E. W. Ross (Eds.), Democratic social education. New York: Falmer Press. </p> <p>MacNeil, C. (2000). Surfacing the realpolitik: Democratic evaluation in an antidemocratic climate. In K. Ryan & L. DeStefano (Eds.), Evaluation as democratic process: Promoting deliberation, dialogue & inclusion. New Directions for Evaluation, 85 (pp. 51-62). San Francisco: Jossey-Bass. </p> <p>Rosaldo, M. Z., & Lamphere, L. (Eds.). (1974). Woman, culture, and society. Stanford: Stanford University Press. </p> <p>Schnoes, C. J., Murphy-Berman, V., & Chambers, J. M. (2000). Empowerment evaluation applied: Experiences, analysis, and recommendations from a case study. American Journal of Evaluation, 21(<reflink idref="bib1" id="ref4">1</reflink>), 53-64. </p> <p>Smith, M. F. (1999). Participatory evaluation: Not working or not tested? American Journal of Evaluation, 20(<reflink idref="bib2" id="ref5">2</reflink>), 295-308. </p> <p>Whitmore, B. (Ed.). (1998). Understanding and practicing participatory evaluation. New Directions for Evaluation, 80. San Francisco: Jossey-Bass. </p> <aug> <p>By Sandra Mathison </p> <p></p> <p>Sandra Mathison * Associate Professor, Department of Educational Theory and Practice, SUNY at Albany, 1400 Washington Avenue, Albany, NY 12222; Tel.: (<reflink idref="bib518" id="ref6">518</reflink>) 442-4986; E-mail: smathison@uamail.albany.edu. </p> </aug> <nolink nlid="nl1" bibid="bib1" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib2" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib518" firstref="ref6"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ640223
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: What's It Like When the Participatory Evaluator Is a 'Genuine' Stakeholder?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Mathison%2C+Sandra%22">Mathison, Sandra</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22American+Journal+of+Evaluation%22"><i>American Journal of Evaluation</i></searchLink>. Win 2001 22(1):29-35.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 7
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2001
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22After+School+Programs%22">After School Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Utilization%22">Evaluation Utilization</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Interpersonal+Relationship%22">Interpersonal Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Parents%22">Parents</searchLink><br /><searchLink fieldCode="DE" term="%22Stakeholders%22">Stakeholders</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1098-2140
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Discusses what it means to be a caring stakeholder and a participatory evaluator in the context of public and private domains using the example of a parent who is an evaluator of an afterschool program. Offers a perspective on the capacity of individual evaluators to deal with role strain. (SLD)
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2002
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ640223
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ640223
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 7
        StartPage: 29
    Subjects:
      – SubjectFull: After School Programs
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Evaluation Utilization
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Interpersonal Relationship
        Type: general
      – SubjectFull: Parents
        Type: general
      – SubjectFull: Stakeholders
        Type: general
    Titles:
      – TitleFull: What's It Like When the Participatory Evaluator Is a 'Genuine' Stakeholder?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Mathison, Sandra
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2001
          Identifiers:
            – Type: issn-print
              Value: 1098-2140
          Numbering:
            – Type: volume
              Value: 22
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: American Journal of Evaluation
              Type: main
ResultId 1