H A RVA R D FAM I LY R E S E A R C H P RO J E C T
Volume V, Number 2/3 1999 THE EVALUATION EXCHANGE
Emerging Strategies in Evaluating Child and Family Services Free Quarterly Newsletter From the Director's Desk
Heather B. Weiss Director, HFRP I would like to begin by thanking all who responded to our reader survey. We appreciated hearing your comments about The Evaluation Exchange--what works and what doesn't--and how we can do better. Our goal is to produce an interactive newsletter, bringing together different voices and providing a forum for readers to share new ideas about evaluation. The information we received through the survey was therefore of great value to us. A brief summary of the survey results is on page 16. In response to your comments, we will be making some changes to the format of the newsletter--look for those in the next issue! ©2000 President and Fellows of Harvard College
Published by HARVARD FAMILY RESEARCH PROJECT Harvard Graduate School of Education 38 Concord Avenue Cambridge, MA 02138 Tel: (617) 495-9108 Fax: (617) 495-8594 E-mail: hfrp_gse@harvard.edu http://gseweb.harvard.edu/~hfrp All rights reserved. This newsletter may not be reproduced in whole or in part without written permission from the publisher. The Harvard Family Research Project gratefully acknowledges the support of The Annie E. Casey Foundation, The F.B. Heron Foundation, the Ewing Marion Kauffman Foundation, and the W.K. Kellogg Foundation. The contents of this publication are solely the responsibility of the Harvard Family Research Project and do not necessarily reflect the views of the funders. In this issue, we include several articles on methodological topics, particularly those involving complex initiatives or problems. First, HFRP consultant Julia Coffman writes about using a logic model approach to evaluate a large and diverse foundation initiative, in our Theory and Practice section. We also include two articles on complex methodological issues in our Promising Practices section. The first, by James Kee of George Washington University, explains the difference between cost-benefit analysis and cost-effectiveness analysis, including some basic examples and tips about when to use each. The second, by Ellen Taylor-Powell at University of Wisconsin-Cooperative Extension, discusses what to think about when evaluating collaboratives. In our Questions and Answers section this time, we speak with Michael Scriven, professor of psychology at Claremont University and immediate past president of the American Evaluation Association, about the challenges to evaluation in the coming years. We highlight two evaluations in our Evaluations to Watch section. The first, by Laura Pinsoneault and James Sass of Alliance for Children and Families, describes an ongoing evaluation of a national replication of the Middle School Families and Schools Together Program. The second evaluation, by Donna Peterson and her colleagues at The University of Arizona, describes the organizational evaluation of the Cooperative Extension System's capacity to support programs for children, youth, and families at risk. Our Beyond Basic Training section includes an article by Jill Chopyak on the community action research Continued on page 16 IN THIS ISSUE
METHODOLOGY
o o Simplifying Complex Initiative Evaluation................. 2 At What Price? Benefit-Cost Analysis and Cost-Effectiveness Analysis in Program Evaluation ................................. 4 Evaluating Collaboratives: Challenges and Practice...................................... 6 Interview with Michael Scriven ........................ 8 Evaluating the Middle School Families and Schools Together Program ................... 10 Evaluating Cooperative Extension's Capacity to Support At-Risk Children, Youth, and Families ............... 12 Community-Based Research: Research for Action ...................................... 14 The Evaluation Exchange Readers' Survey ...................... 16 Quality of Life: A Framework for Examining the Impact of Welfare Reform .................. 17 Electronic Mailbox ................. 18 New and Noteworthy.............. 19 o o o o o o o o o The Evaluation Exchange 1 VOLUME V, NUMBER 2/3 1999 T H E O RY A N D P R A C T I C E Simplifying Complex Initiative Evaluation
Introduction E valuators are increasingly facing the challenge of evaluating complex initiatives that are both multi-site and multi-level. These initiatives typically involve a number of different programs or organizations that have unique strategies and goals but are working toward a broader, common agenda. Both private and public sector funders are increasingly investing in complex initiatives with the intent that through them they will achieve a more strategic impact than they could make by funding individual programs. For example, complex initiatives can act as a catalyst for the connection and integration of different types of services or activities that may be needed to achieve broad-based change. In general, complex initiatives are expected to achieve more in terms of outcomes than would be possible with the sum of their individual parts. In the last decade much has been learned about ways to evaluate complex initiatives. Theory-of-change and cluster evaluation have recently emerged as developing evaluation approaches that can aid evaluators facing the daunting task of designing and implementing complex initiatives such as comprehensive community initiatives. These approaches have helped evaluators make conceptual leaps in understanding how to address specific design challenges. However, there is still much to be learned and shared. Of particular importance is achieving progress in describing complex initiative evaluations that are both efficient and effective, with due regard for reasonable time and resource boundaries. Additionally, there is a need for acquiring and sharing information on how to build management and reporting structures that complement these approaches. In this article, we highlight two major lessons learned from theory-of-change and cluster evaluation about how to evaluate complex initiatives. In addition, within the context of our experiences evaluating the W.K. Kellogg Foundation's (WKKF) De- volution Initiative,1 we share tips on how we applied those lessons to meet our own evaluation challenges. The Devolution Initiative (DI) was created in 1996 to respond to some of the information and governing challenges associated with welfare reform and health care devolution--the passing of responsibility for policy and service development and management from the national level to the state and local levels. Specifically, the ers and citizens, and building the capacity of communities and their citizens to participate in and inform the policy process. Lessons and Tips from Tested Approaches
Some clear lessons can be gleaned from the work that has been completed on evaluating complex initiatives so far. We present two of these lessons below. For each, we provide practical tips on how these lessons might be applied, based on our experience with the Devolution Initiative. Lesson: Articulate the Complex Initiative's Theory It helps immensely to begin a complex initiative evaluation by articulating the broad theory (or theory of change) that weaves together the initiative's many strands. This theory can serve as a framework for interpreting the initiative's various layers and parts. The evaluation literature includes a wealth of information on ways to tease out program theory. The logic model is one of the most frequently used tools for this task, and there are many evaluation resources available on how to develop logic models. The models developed using these resources generally share as common ground the identification of initiative activities and how they relate to short-, intermediate-, and long-term outcomes. Tip: Begin with a Broad Conceptual Model of the Initiative's Theory We began our evaluation by attempting to develop a logic model to articulate the Devolution Initiative's theory of change. We found, however, that our desire to work with stakeholders to build a theory of change that fully articulated the short-, intermediate-, and long-term outcomes for each Initiative layer (individual, community, state, national) conflicted both with our time constraints (we needed to quickly develop and begin implementing an evaluation design) and with the fact that the DI was too early in its development to do that. Because the DI was still developing, the stakeholders did not want to articulate too much of the strategic detail or give a sense of false precision by making premature decisions about outcomes.
WKKF saw a need to support states and local governments in their efforts to take on these new responsibilities. Consistent with WKKF's mission to "help people help themselves," the DI strives to help citizens learn what is and is not working in various states with respect to welfare reform and health care so they can participate in the development and implementation of more effective policies in their communities. The more than 25 national, state, and local grantee organizations involved in the DI work toward this goal by building a base of knowledge about the impacts of devolution, disseminating findings to diverse target audiences that include policymak- 1 Note that this evaluation is still in progress. The Evaluation Exchange 2 HARVARD FAMILY Because the DI was still developing, the Initiative's stakeholders did not want to articulate too much of the strategic detail or give a sense of false precision by making premature decisions about outcomes. As a result, rather than enumerating all of the DI's outcomes as a typical theory articulation process might do, we began by developing a broad conceptual model of the DI, one that illustrated its goals (or activities, depending how you look at it) and the general non-causal relationships between them (see Figure 1). The advantages to using this type of model were substantial. First, the model's early timing and quick development were critical. Because the DI was in its beginning stages, the model served as an effective socialization tool for getting an early shared understanding among stakeholders of what the Initiative was trying to achieve. Using a more traditional model that identified and achieved consensus on specific outcomes at this early point in time would likely have been met with frustration, and might have stalled the evaluation early on. A second advantage of this model was its simplicity and broadness. Each grantee organization was able to understand how its individual activities fit within the Initiative's broader context. Another advantage of the model was its usefulness as a starting point for later theory articulation. As we describe below, while the model did not identify specific outcomes at first, we were able to use it later as the basis for identifying outcomes that could be attached directly to the model and further guide the evaluation. Finally, probably the most important feature of the model was its sustainability. The DI evolved substantially over time, Figure 1 WKKF Devolution Initiative Devolution Grantees WKKF Inform Policy Agenda and Decisions Develop Information Disseminate Information Build local participation in decision making Assess information needs Reach groups/ individuals/media at state/local levels Build National, State, and Local Capacity both strategically and in terms of the number of organizations involved. Despite these changes, the overall model changed only slightly, because Initiative changes for the most part could be accommodated beneath the level of this overarching structure. This was critical because as we describe below, the model became the organizing structure for the evaluation's design and management. Lesson: Use the Initiative's Theory as a Framework for Designing the Evaluation Connecting the evaluation design to the initiative's theory makes sense. The theory typically lays out what needs to be assessed in terms of both process and outcomes. Once you have that necessary focus, you can proceed through standard evaluation design steps, which may include identifying benchmarks or indicators connected to those outcomes and the methods needed to track them. This lesson can be difficult to apply, but keeping certain principles in mind will help. For example, you want to make sure that if you focus on the theory's parts (boxes in the logic model), you don't lose sight of their connections (arrows in the logic model). In addition, you want to develop a design that is easy to manage and keeps data collection and reporting focused on what is being learned about the initiative's theory as a whole and not only on its parts. Tip: Develop Evaluation Objectives Linked Directly to the Model Our first and most important step in the design process was to break the Devolution Initiative's model into evaluation objectives in order to make the evaluation in general more manageable (see Box1). The objectives focused on distinct pieces of the model and their relationships to one another. As a result, the objectives highlighted the importance of examining the links between the main model components. For example, Objective 2 involved several parts of the model. It was concerned with grantee information development and dissemination activities in terms of the types of information developed and the various mechanisms used to disseminate the inContinued on page 15 BOX 1
Objective 1: Objective 2: Objective 3: Objective 4: Objective 5: Devolution Initiative Evaluation Objectives
Examine WKKF and Devolution grantee roles in information development and dissemination. Examine links between information development, dissemination, and target audiences. Examine capacity-building activities. Examine the link between building capacity and increasing state and local participation in policymaking. Examine the Initiative's success in informing the policy agenda. RESEARCH PROJECT 3 VOLUME V, N UMBER 2/3 1999 P RO M I S I N G P R AC T I C E S At What Price? Benefit-Cost Analysis and Cost-Effectiveness Analysis in Program Evaluation
Introduction I n our current age of accountability, public and private sector funders are increasingly concerned with the relative costs and benefits of the programs they fund. Benefit-cost (or cost-benefit) and cost-effectiveness analyses can be useful quantitative tools to help address these concerns. However, they differ in their purposes, and each has strengths and limitations. Benefit-cost analysis is an applied branch of economics that attempts to assess service programs by determining whether total societal welfare has increased (in the aggregate, people have been made better off) because of a given project or program. It can be used in evaluations of existing programs to assess their overall success or failure, to help determine whether the programs should be continued or modified, and to assess the probable results of proposed program changes. Benefit-cost analysis consists of three steps: (1) determine the benefits of a proposed or existing program and place a dollar value on those benefits; (2) calculate the total costs of the program; (3) compare the benefits and the costs. Cost-effectiveness analysis is an alternative to benefit-cost analysis that relates the cost of a given alternative to specific measures of program objectives. A costeffectiveness analysis helps to compare costs to units of program objectives and may be the first step in a benefit-cost analysis if the analyst then decides to attempt to place a dollar value on the benefits. Unlike benefit-cost analysis, cost-effectiveness analysis does not produce a "net benefit" number, with benefits exceeding costs or costs exceeding benefits. However, a costeffectiveness analysis can determine that a program which costs $1 million produces ten units of outcome x, twelve units of outcome y, and twenty units of outcome z. Or, if the units are alike, it can determine the cost per unit of outcome. An example of these two methods of analysis using a hypothetical dropout prevention program is presented in Box 2. Challenges in Conducting Benefit-Cost and CostEffectiveness Evaluations
Identifying and Measuring Costs Identifying and measuring costs, and in the case of benefit-cost analysis, quantifying and placing a dollar value on the benefits, is the biggest challenge to the evaluator trying to conduct these types of analyses. Direct costs (such as personnel, materials, and equipment) are often relatively easy to account for. Indirect costs (such as over- head, costs to other providers supporting the intervention, and costs to participants) as well as capital costs (such as buildings and computers) can be more difficult to calculate. Finally, intangible costs (such as the value of wilderness) are those for which the evaluator either cannot assign an explicit price or chooses not to. Lack of assigned price does not mean that intangible costs are unimportant; indeed, in presenting any results of these types of analyses, the evaluator should point out the intangible costs and benefits, thereby enabling the decision maker to consider these as he or she examines those benefits and costs that are quantified. When identifying any benefit or cost, it is important to state its nature clearly, to state how it is being measured, and to list any assumptions made in the calculation of the dollars involved. BOX 2 Hypothetical Cost-Effectiveness and Benefit-Cost Results for Dropout Prevention Strategies Cost-Effectiveness
The cost-effectiveness of each dropout prevention strategy is determined by dividing the cost for each strategy by its effectiveness (e.g. the percentage increase in the number of students graduating).The result is the cost for each percent increase in the number of students graduating.
Strategy Mentoring After-School Sports Costs $80,000 $65,000 Effectiveness 10 5 C/E Ratio $8,000 $13,000 Benefit-Cost
The benefit-cost for each dropout prevention strategy is determined by calculating each strategy's benefits (e.g. estimates of future earning increases of participants who stayed in school) and costs (e.g. personnel, materials, equipment) and then subtracting the benefits from the costs to get the net benefit for each strategy. The benefit-cost ratio can also be computed by dividing the dollar value of benefits by the costs (the higher the ratio, the more efficient the program in economic terms).
Strategy Mentoring After-School Sports Costs $80,000 $65,000 Benefits $95,000 $75,000 Net Benefits $15,000 $10,000 B/C Ratio 1.188 1.154 The Evaluation Exchange 4 HARVARD FAMILY Identifying and Measuring Benefits Identifying benefits can also be tricky. As with costs, there are direct, indirect, and intangible benefits. In the case of benefitcost analysis, placing a dollar value on the benefits is also a challenge. The evaluator might choose a market value, when one is available, or a surrogate such as willingness to pay. Because of the redistributional nature of government programs, public agencies and those who evaluate them must be concerned with who benefits as well as the amount of benefits in addition to the costs. Where quantifying benefits is difficult, costly, or viewed as inappropriate, cost-effectiveness analysis can be used. Cost-effectiveness evaluation does not require that the evaluator place a dollar value on the benefits. This is particularly useful in cases where the benefit of a program is "lives saved." While there are various ways to place a dollar value on a life saved or lost, each is controversial. In contrast to a benefit-cost analysis, a cost-effectiveness evaluation would calculate the cost of the program per life saved without making a judgment about the dollar value of that life. The evaluator would then present the results to the decision maker who must decide whether an outcome is worth the dollar cost when viewed in light of alternative uses for the funds. A major challenge in cost-effectiveness analysis is the fact that programs frequently generate more than one type of benefit. For example, an education system might target more than one population group in the school system. When conducting a costeffectiveness analysis comparing programs with multiple benefits, the evaluator may need to place weights on the relative benefits to assist the decision maker in making comparisons. If this is not done, the comparison becomes quite subjective. Yet assigning weights often becomes at least as problematic as assigning dollar values to each benefit: how do the benefits to one population group outweigh those to another, for example? Boundaries Another challenge in conducting benefitcost and cost-effectiveness analyses is determining the geographic scope of an analysis. While the focus may be within a certain jurisdiction, such as a state, there may be benefits or costs that spill over to neighboring jurisdictions. It might be tempting to ignore spillover effects, but this can be unwise since spillovers often have political consequences. The question for the evaluator is whether to consider only those benefits and costs that accrue to the population within the jurisdiction for which the evaluator is doing the analysis. Detail One of the biggest dangers in these analyses, as in many other areas of evaluation, is the "black box" syndrome. Instead of laying out the relevant issues, assumptions, and concerns, the evaluator may be tempted to hide the messiness of the analysis from the decision maker, presenting a concise answer as to the net benefits or costs or cost-effectiveness. However, it is the detail--the assumptions involved and the sensitivity of the analysis to particular assumptions--which may be of most use to the decision makers in judging the value and usefulness of the evaluator's work. from the accumulation of new data against the costs associated with the data collection. Thus, any analysis should begin by assimilating existing data to determine whether it is sufficient. The more intangible the benefit (for example saved wilderness), the more likely it is that a cost-effectiveness analysis will be of greater use to decision makers. This type of analysis can help them assess whether a cost is justifiable, when compared with other uses of the same funds. It is important to note that benefit-cost analysis and cost-effectiveness analysis could lead to different conclusions about the same program, depending upon how benefits are valued in dollar terms. However, if the evaluation is concerned with a program with a single objective (or closely related objectives), programs or alternatives achieving the highest cost-effectiveness should also achieve the highest benefit-cost ratio. Neither benefit-cost analysis nor costeffectiveness analysis is a panacea. Both require judgments on measurement issues that should be brought to the attention of the decision maker. However, both techniques are useful to provide a format for analysis that can lead to better decisions. o
James Edwin Kee Giant Food, Inc. Professor of Public/Private Management School of Business and Public Management George Washington University Deciding Between CostEffectiveness Analysis and Benefit-Cost Analysis
Those faced with deciding between the two types of analysis may find it helpful to keep three basic questions in mind: 1. How will you use the results? Benefitcost analysis enables you to compare strategies that do not have the same outcomes, or to compare strategies across different areas of public expenditure (e.g., health, welfare, justice). Cost-effectiveness analysis is useful for comparing strategies that are trying to achieve the same objective (e.g., increased graduation rates). 2. What resources do you have? Benefitcost analyses typically require more resources, because they take more time for analysis and involve significant methodological expertise (often in economics), such as the capacity for determining the discounted present value of a stream of benefits and costs. 3. How difficult are costs and benefits to value? While you may want to have as much information as possible on both benefits and costs, you must weigh the value of the increased accuracy gained For Further Reading
Kee, James Edwin. (1994). Benefit-cost analysis in program evaluation. In Joe S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer (eds.). Handbook of practical program evaluation. San Francisco, CA: Jossey-Bass, Inc. Levin, Henry. (1983). Cost-effectiveness: A primer. Thousand Oaks, CA: Sage Publications. (This is one of the volumes in the New Perspectives on Evaluation series.) Yates, Brian. (1996). Analyzing costs, procedures, processes, and outcomes in human services: An introduction. Thousand Oaks, CA: Sage Publications, Inc. RESEARCH PROJECT 5 VOLUME V, N UMBER 2/3 1999 P RO M I S I N G P R AC T I C E S Evaluating Collaboratives1: Challenges and Practice D uring the past years, the faculty and staff of University of Wisconsin-Cooperative Extension have increasingly found themselves engaged in collaborative work, playing unfamiliar roles or interacting in new situations with new players. The venue might be a community-based initiative, an interagency partnership, or an interdisciplinary work team in the area of agriculture, family living and nutrition education, 4-H and youth development, or community and economic development. Familiar evaluation practices--ones largely built on a discrete or distinct program delivered by one agency--do not fit this new context. Collaboration is widely heralded as a mechanism for leveraging resources, dealing with scarcities, eliminating duplication, capitalizing on individual strengths, and building capacities. It offers the possibility for increasing participation and ownership, strengthened by the potential for synergy and greater impact. Yet, for all of us working in and with collaboratives, the challenges are numerous. Several, in particular, are stretching us to think about evaluation in new ways. the members work together. Implementation may never be complete and is often difficult to track. In this setting, process becomes particularly important. Conventional program evaluation and many current funders focus on activities that are delivered. We find, however, that special attention must be paid to the workings of the collaborative if it is to successfully deliver activities and programs. This includes the capacities, operations, and climate of the collaborative. In this outcomes era, we are finding it necessary to educate our constituents and fun- Evolving Nature
Collaboratives are dynamic and flexible, changing as they develop. They may look different from year to year. The membership or the roles of members may change, bringing new direction and emphasis. Some collaboratives have a clearly defined start and finish time, but many do not. They may not even start out being a collaborative. There is no grand plan. Rather, the work and direction of the group are invented as Collaboration is widely heralded as a mechanism for leveraging resources, dealing with scarcities, eliminating duplication, capitalizing on individual strengths, and building capacities. featured in a quarterly column in the local newspaper. In another case, an interdisciplinary work team has enlisted one member to evaluate its process and progress. This member summarizes the minutes of meetings and uses content analysis to keep track of decisions, actions, and achievements; observes team interactions and discussions; and conducts an annual survey of members to assess levels of satisfaction, capacity development, and operations. Ongoing feedback is provided to the team, and annual reports are submitted to administration. In yet another example, a community partnership is using the group member survey developed by Cooperative Extension to engage members in a process of self-assessment and learning. Members were informed of the evaluation opportunity and they committed to completing the survey. Teams of members volunteer to present and discuss the results at a series of meetings that engage the full group in interpretation and action related to the findings. Broad Goals and Expected Outcomes
Collaboratives form for many reasons. Some seek to develop and sustain resilient families or communities. Others are initiated to provide a particular service, to leverage resources, to coordinate efforts, or to effect greater integration of activities or services. Other collaboratives instigate social activism, or seek to create consensus around politically charged issues such as land use or school improvements. Often groups may not have a clearly defined or single purpose or one that all agree to or understand. Coming to a shared understanding of the collaborative's goals and expected outcomes is critical. Time is needed to discuss and negotiate this vision as the collaborative forms and as the collaborative evolves, since initial expectations, context, or membership may change. Creating a logic model--mapping the collaborative jour- 1 A collaborative is defined as a group working together to achieve a shared vision. Members engage in a process where they constructively explore their differences and search for (and implement) solutions that go beyond their own limited vision of what is possible. [Gray, Barbara. (1989). Collaborating: Finding common ground for multi-party problems. San Francisco, CA: Jossey-Bass, Inc.] ders about process and the linkage between process and outcomes. Integrating process evaluation into a collaborative provides information for internal decision making, visibility, legitimization, and accountability. There are several ways in which this has been done. In one case, a community-based initiative guards 15 minutes at the end of every meeting to engage in a "how are we doing" process. Members facilitate a questionand-answer period on a rotating basis and write up and share the results. Accomplishments and issues being worked on are The Evaluation Exchange 6 HARVARD FAMILY ney--makes explicit the ideas members hold about what results are desired and how to achieve them. We have found that drawing the logic model, either individually or as a group, is a fun and useful process. We use newsprint and allow members to use any metaphor, design, or thought process to show the chain of events and desired final outcomes. Members then share their pictures; similarities and differences are noted and discussed as the beginning of building consensus on expected outcomes and the strategy for achieving them. Stakeholders are involved in the process to spread understanding and ownership. Multiple Outcomes
Collaboratives often struggle with defining and measuring outcomes. As part of the logic modeling, differentiating among types of outcomes--which outcomes; for whom--helps members set realistic expectations. There may be outcomes for individuals that include changes in attitudes, knowledge, skills, behavior, actions, and/or lifestyles for clients, community residents, collaborative members themselves, and/or service providers. There may be outcomes for groups such as changes in interactions, values, or behaviors of families, the work group, the community group, or the collaborative itself. Often, collaboratives are focused on agency or organizational outcomes such as changes in service delivery, resource generation and use, practices, and policies. Some collaboratives are interested in system outcomes in which agencies, departments, or whole organizations work in new ways, behave differently, share resources, and provide services in an integrated fashion. Finally, collaboratives may be focused on outcomes for communities, including changes in norms, policies, or actions at a community-wide level. Collaboratives may lead to the institutionalization of change or the empowerment of individuals or groups. We find that collaboratives often have impact in more than one area, and the unanticipated outcomes are significant in terms of human and social capital development. Asking questions about outcomes-- what outcomes, for whom, when might we see them, how will we know it, what else is happening that we didn't anticipate?-- helps us detect and document results. One example of this is a family preservation and support initiative that collects longitudinal data through a series of interviews with participating families to document their evolving ability to deal with stress, access community resources, and reduce family isolation (family outcomes). In another case, a collaborative of youth-serving agencies, working to open up membership to youth, compiled membership data before and after the two-year effort to assess changes in membership (agency outcomes). In yet another example, a community gardening initiative collects data through observation, interviews with participants and leaders, a sample survey of residents, and logs kept by staff to assess interaction among neighbors and changes in attitudes, helping behaviors, and responsibilities concerning the neighborhood (community outcomes). Individual Performance
Collaboratives are built on the premise that the whole is greater than the sum of the individual parts, but individuals make up the collaborative and many have performance appraisal systems that require evidence of individual performance. How can one assess individual performance without undermining the essence of the collaborative? Collaborative members, who need to report to their supervisors or are concerned with personal accountability, are using a variety of techniques for defining their own contribution to the team effort and outcomes. These include using a log book or diary to track one's inputs, activities, outcomes, and impact; using meeting minutes or other documentation to determine the role and influence of individual members; engaging members in evaluating each other in a nonjudgmental process; and using surveys, group discussions, or interviews with key stakeholders to collect data on member contributions and influence on outcomes. When individual members are responsible for a particular activity, it might also be possible to evaluate that event/activity and link the results to the member's effort. is it the most appropriate approach? Some groups and/or communities are not ready for collaboration; some problems do not need a collaborative approach; and sometimes an individual's background and/or the agency's mission do not fit that of the collaborative. As we look at this dimension, a variety of evaluative questions emerge that relate to the context and readiness for collaboration in the community and external environment, in the organization, and among stakeholders. We think about this as feasibility evaluation that typically occurs at the beginning of an initiative and is often informal. Some collaboratives are using a readiness questionnaire to ascertain a number of factors, such as the willingness of the community and/or individuals to work together, levels of cooperation and trust, history of previous work, potential barriers to success, and the availability of leadership. Besides the initial feasibility evaluation, each time the collaborative changes, adds new members, or begins a new initiative, it seems important for the group to discuss the feasibility of each--the potential for success, the resources needed, and the compatibility. Again, we are finding it important to help partners and funders consider whether collaboration is the best approach and what technical assistance may be necessary. Our approach as partners in collaboratives across Wisconsin is to view evaluation as learning and as a shared process among members and stakeholders. Evaluation in the collaborative context becomes a collaborative process itself. The purpose, direction, and expectations for evaluation are negotiated among the collaborative members. When evaluation becomes a part of the collaborative, it provides the focus, feedback, and learning to support continuous progress and growth. o
Ellen Taylor-Powell Evaluation Specialist University of Wisconsin-Cooperative Extension Parts of this article are taken from Ellen Taylor-Powell, Boyd Rossing, and Jean Geran. (1998). Evaluating collaboratives: Reaching the potential. Madison, WI: University of Wisconsin-Cooperative Extension. (available at http://www.uwex.edu/ces/pdande/ evaluat.html) Collaboration as a Panacea
Collaboratives are being promoted, expected, or required everywhere. Yet, it seems that a critical initial question is often overlooked: Is a collaborative warranted-- RESEARCH PROJECT 7 VOLUME V, N UMBER 2/3 1999 QUESTIONS AND ANSWERS Interview with Michael Scriven I n this issue, we speak with Michael Scriven, professor of psychology at Claremont Graduate School and immediate past president of the American Evaluation Association. Dr. Scriven has written or edited books, periodicals, and articles in the areas of philosophy and psychology, word processing, turbine engines, artificial intelligence, critical thinking, and evaluation. In this article he shares with us some of his insights about the challenges facing evaluation, about evaluation as a distinct discipline, and about links between evaluation and practice, including organizational learning. (1) What do you see as some of the major challenges facing evaluation and evaluators in this new century? (2) Should evaluation be considered a I think the biggest challenge we have is deseparate discipline? veloping recognition of evaluation as an autonomous discipline. We know that in I think the importance of evaluation as a evaluation, there is a body of knowledge, separate discipline is slowly being recoga common logic that people will have to nized. The field of statistics evolved out of master to do this work well. mathematics in the same We have increasingly been way. But there are as yet recognizing the highly speno universities that are deTo get greater cialized skills that are also veloping evaluation derecognition for the required, but we need to go partments or even chairs beyond that. field of evaluation, in evaluation. It is imporFor example, knowing tant, I think, that we went we still have to when to use what type of from 9 to 23 national asevaluation approach is a sociations of professional improve our skill--and it's a skill that evaluators worldwide in professional takes some knowledge of the past year. To get greater the underlying logical disrecognition for the field of training. tinctions between grading, evaluation we still have to ranking, and apportioning. improve our professional On the other hand, cost training. For in-service analysis is a skill that has been developed training, there are now certificates of advery well by specialists, although it is still vanced study and summer institutes in not widely practiced in a sophisticated form which evaluators can upgrade their skills. by most evaluators. So there is room for We need to improve on that system with improvement in the basic skills as well as graduate degrees that involve a major in the logical skills. Another skill, one that is evaluation, not just in educational evaluastill not well developed anywhere, is knowtion or policy analysis. ing how to set standards after you deterPartly because we are still treating evalmine performance levels. Increasingly, uation as simply a "tool" discipline, we however, we are realizing that evaluation have so far virtually overlooked two other is different from the standard scientific parimportant areas in evaluation: the use of adigm in that in evaluation, we rely on investigative skills rather than on hypothesis testing. Finally, evaluation has a set of ethical standards unique to its work. I think another challenge we have is that we must break from the widespread attachment to the idea of interactive or collaborative evaluation as the standard model. While collaborative approaches to evaluation are important, some very good evaluations are done completely separately from the program being evaluated--they are not in any sense collaborative. In some cases, this is out of sheer necessity (e.g. in historical evaluation), but in other cases it may be a more deliberate choice (e.g., in contexts where the preservation of independence is crucial). evaluation in other disciplines (such as physics) and meta-evaluation (evaluation of evaluation). Physicists have to evaluate everything they deal with: theories, data, instruments, scientific papers and proposals for funding, candidates, and students. They learn how to do this as part of their scientific training; but, unlike everything else in that training (e.g. the mathematics they use), it is never explicitly addressed as a logical discipline. The history of science makes it clear that very large improvements in practice occur from explicating implicit principles. We already see this in the improvements in proposal evaluation that have been shown to be possible in the sciences. (3) Some argue that evaluation needs to be better linked to the policymaking process. What is your opinion about this? I do agree that program evaluation needs to be better linked with policymaking about the program evaluated. But our duty in this area is to produce valid, comprehensible, and appropriate evaluations. The evaluator's business is simply that of determining the merit, worth, or significance of what he or she is looking at. The service we provide is telling what is or is not working and, sometimes, finding the reasons why. I don't think we should be in the business of telling policymakers what they should be doing next--that is the role of policy analysts. These people look at the broader picture-- taking into account the decision makers as well as the many other variables involved in a political decision. These variables are not addressed in program evaluation. I do think the program/policy analyst needs better evaluation skills, but this is still the person who has the responsibility and the knowledge to make policy recommendations. For example, I was sole evaluator for a community foundation. They asked me to look at their youth leadership program, a program that trained youth to be leaders in the community. In my evaluation, I found The Evaluation Exchange 8 HARVARD FAMILY several things: there was no evident need to train youth in the community; the approach they used had no basis in theory; and there was no data from three years of work that the program had any effect. My tendency was to say, "Shut it down." However, in reality the situation depended on some political matters. The board of trustees for this foundation is completely unpaid. As a result, the members each implicitly get a "wildcard"--a program they like and want to fund because they think it is a good thing, without debate. That was the case with the youth leadership program--no matter what the evaluator says, the program is not going to be abandoned. This is the reality of the political process, of real decision making. That is not to say that evaluators can have no influence or should not seek to inform the policy process. Evaluation has a major contribution to make, but we need to reach the right audience and we need to be realistic in our ambitions. What we should be good at is servicing policymakers who need answers about what works and why. We can also inform policymakers through legislative evaluators, those state and federal legislative offices that do program evaluation or policy research for legislators. These are the unseen beavers who get the dam built. They respond to committee and legislator requests for information on particular topics. If evaluators want to get their stuff out there and see it better used, these are the folks that they ought to be reaching--and these are the folks who are often more accessible. (4) Some argue that evaluation needs to be better linked to on-the-ground practice. What is your opinion about this? I think we are a remarkable discipline in that the usual academic versus application distinction has been almost entirely absent. It's still true that there is room for improvement in codifying best practices. There need to be mechanisms (e.g., research secretaries on each project), to link evaluation findings to methodological work in a field. Some are trying by linking with training and higher education institutions. But we also need to take some action on our own--publish in other journals beyond academic ones, make sure our findings reach program/policy analysts, and attend congressional hearings. (5) How do you think evaluation (and accountability requirements) can be better used for organizational learning and continuous improvement of programs? People involved in "organizational learning" reforms need good program evaluation skills and good personnel evaluation skills. The detail of this learning is tricky-- how do you really decide what the lesson was? How do you design a study to learn results? What is the methodology involved? These are tough evaluation questions, and the skills are not taught in business schools. Organizations have to develop a plan to integrate policymaking and programming--what questions they want answered, how data will be reviewed, how data will be used. A good evaluator helps with these questions. One of the most recent changes has been the focus on performance measurement. I don't think this focus on outcomes is bad--I've seen programs take years to get up and running, and then have nothing to show for their work. Performance measurement is important--it is the bottom line. However, performance measurement as it has become lately is inflating one aspect of evaluation into the whole. We used to make exactly the complementary error, in the guise of process evaluation. I think you need to balance the two; you cannot ignore how you got to the outcomes. The most important way to do that is to negotiate expectations early on in the contracting process. o
Karen Horsch Research Associate HFRP JOB OPENING Research Associate
The Harvard Family Research Project, a research department at the Harvard Graduate School of Education, is hiring a full-time staff member to direct research efforts for its quarterly newsletter, The Evaluation Exchange, and to conduct research on other projects related to evaluation. The Research Associate will: · Plan and direct development of The Evaluation Exchange: review literature and current work in the area of evaluation and identify topics of current interest. Identify, recruit, and work with authors. Write original articles for each issue of the newsletter. · Develop and implement plan for expansion of newsletter audience and for development of "spin-off" evaluation-related products, including working papers, reports, evaluation tools, and articles for publication in professional journals. · Conduct research on other HFRP projects related to evaluation design, evaluation approaches, and evaluation utilization. Assist in research design, development of data collection protocols, data collection, and data analysis. Write reports and present research results at various national conferences. A Master's degree and five or more years of professional experience are required, with a doctorate preferred. Applicants should also have experience with program evaluation and/or participatory research and familiarity with qualitative research methodology and/or current evaluation theory, as well as excellent writing skills and publication record. The Research Associate reports to the Director and works closely with other research staff. The position is grant-funded through 6/30/01, with strong likelihood of continuation. To apply, send cover letter and resume to: Harvard University Resume Processing Center Requisition # 5103 11 Holyoke Street
Cambridge, MA 02138 RESEARCH PROJECT 9 VOLUME V, N UMBER 2/3 1999 E VA L U AT I O N S TO WAT C H Evaluating the National Replication of a Prevention Program for Youth and Their Families: Middle School Families and Schools Together
Introduction
n 1997, the Alliance for Children and Families, with major support from the DeWitt Wallace­Reader's Digest Fund, began replicating and evaluating the middle school adaptation of the Families and Schools Together (FAST) program. This program adaptation was developed in 1991 by staff of Family Service in Madison, Wisconsin, as part of a Center for Substance Abuse Prevention High Risk Youth Grant. Middle School FAST is a school-based, whole-family program intended to increase the likelihood of youth success in the home, in the school, and in the community. Program activities build positive, respectful, and supportive relationships for middle school youth, their parents, and other family members. Middle school youth are recruited for the program and begin participation in a school-based youth group facilitated by a youth advocate. Four weeks into