WHY CONDUCT A PROGRAM EVALUATION? FIVE REASONS

WHY CONDUCT A PROGRAM EVALUATION?

FIVE REASONS WHY EVALUATION CAN HELP AN OUT-OF-SCHOOL TIME

PROGRAM

Part 1 in a Series on Practical Evaluation Methods

Allison J. R. Metz, Ph.D.

BACKGROUND

Program evaluation is a valuable tool for program managers who are seeking to strengthen the quality of

their programs and improve outcomes for the children and youth they serve. Program evaluation answers

basic questions about a program’s effectiveness, and evaluation data can be used to improve program

services. In this brief, we define program evaluation, address common concerns program managers and

practitioners have regarding evaluation, and outline five major reasons why conducting a program

evaluation can benefit an out-of-school time program.

WHAT IS AN EVALUATION?

Program evaluation is a systematic method for collecting, analyzing, and using information to answer

basic questions about a program.

While there are many different types of program evaluations, and

many terms are used to describe them, evaluations are typically divided into two major categories: process

evaluations and outcome evaluations.

 Process evaluations assess whether an intervention or program model was implemented as

planned, whether the intended target population was reached, and the major challenges and

successful strategies associated with program implementation.

 Outcome evaluations determine whether, and to what extent, the expected changes in child or

youth outcomes occur and whether these changes can be attributed to the program or program

activities.

COMMON CONCERNS ABOUT PROGRAM EVALUATION

Program managers often express legitimate concerns or fears when considering a program evaluation.

While these concerns are valid, they can often be addressed fairly readily. Examples include:

 Evaluation will divert resources away from the program. While it is true that evaluation will take

some upfront resources (typically about 10% to 20% of an overall program budget is needed to

conduct a process and

outcome evaluation),

what a program can learn from an evaluation can

help streamline its resources to focus on “what works” for program participants and improve

outcomes.

 Evaluation will be too complicated. While some evaluations are complex, evaluation designs can

be simple and straightforward. An independent evaluator or consultant can help develop an

evaluation design that is most appropriate for a program given the program’s service model, the

evaluation questions the program wishes to address, and the program’s resources for conducting

an evaluation.

…information for program providers who

are considering evaluating their programs.

October 2007

Publication #2007-31

 Evaluation will be an additional burden on staff. In order to minimize the potential burden on

program staff, evaluation activities can be incorporated into ongoing program management

activities. Also, when feasible and appropriate, evaluation data can be collected by an outside

evaluator.

 Evaluation will produce negative results. Finding out “what does not work” is as important as

finding out “what does work.”

 Evaluation is just another form of program monitoring. Program monitoring assesses whether a

program is in compliance with specified performance standards (e.g., number of participants

served), while an evaluation assesses whether expected outcomes were achieved.

FIVE MAJOR REASONS TO CONDUCT AN EVALUATION OF AN OUT-OF-SCHOOL TIME PROGRAM

As noted above, program managers often express valid concerns about conducting a program evaluation.

While many of these concerns can be overcome or greatly minimized, it is also important to note that

there are several benefits to conducting a program evaluation that outweigh the potential issues associated

with these concerns. Below, we outline five major reasons why conducting a program evaluation will

serve the best interests of a program, as well as the children and youth served.

Reason #1: A program evaluation can find out “what works” and “what does not work.” A process

or outcome evaluation enables program managers to answer basic questions about a program’s

effectiveness, including:

 Are participants benefiting from program services?

 Are recruitment strategies working?

 Do staff have the necessary skills and training to deliver services?

 Are participants satisfied with the program?

 Are some sub-groups benefiting, but not others (for example, boys versus girls)?

Knowing “what works” helps program managers to focus resources on the essential components of the

program model that benefit participants and volunteers; knowing “what does not work” allows program

managers to improve and strengthen their service delivery models. Not knowing what is working may

waste valuable time and resources.

Reason #2: A program evaluation can showcase the effectiveness of a program to the community

and funders. Evaluation findings can demonstrate to a community and to funders that a program is

worthwhile. Sharing findings within the community can serve as a good outreach tool for attracting

collaborative partners, recruiting participants and volunteers, and building trust with families and

community members. Also, funders often require that a program evaluation be conducted when they agree

to fund a program, and some funders will not fund, or re-fund, a program until an evaluation has been

conducted and outcomes have been demonstrated.

Reason #3: A program evaluation can improve staff’s frontline practice with participants.

Improving how frontline staff members deliver services to children and youth will increase the likelihood

that a program will achieve positive outcomes with program participants. Conducting an evaluation of a

program can allow a program manager to systematically assess staff’s performance, and figure out where

staff members are succeeding and where they may need more support or training. An evaluation can also

provide staff with opportunities to discuss the challenges they face and offer potential solutions.

Evaluation questions may include:

 Do staff have the necessary skills to work effectively with program participants?

 What types of additional training would benefit staff?

 Are staff receiving the ongoing coaching and mentoring necessary to do their work?

 Do staff have the necessary supports to function effectively?

Reason #4: A program evaluation can increase a program’s capacity to conduct a critical self-

assessment and plan for the future. Conducting an evaluation either internally or with an outside

evaluator will build an organization’s capacity to conduct critical self-assessments, including conducting

staff and program needs assessments, measuring staff performance, and assessing whether program

objectives have been met. This will strengthen program operations and, consequently, improve outcomes

for those served. Knowing how and for whom the program is effective and ways services can be

strengthened are essential building blocks for an organization’s strategic plan. Having the goal and the

capacity for self-assessment allows for ongoing reflection and planning and helps create a continuous

learning organization.

Reason #5: A program evaluation can build knowledge for the out-of-school time field. Contributing

to the evidence-base on what works in out-of-school time programming benefits everyone trying to make

a difference in the lives of the children and youth who participate in these programs. Sharing knowledge

with peers on what has been learned about programs can ensure that other program managers and staff

avoid mistakes and that successful and effective strategies are replicated.

INAL THOUGHTS

While conducting an evaluation may seem complicated, expensive, or even overwhelming, it is important

to remember that program evaluations serve as tools to improve programs. Simply put, program

evaluations are conducted to make programs better. Evaluations benefit programs at every stage of

implementation. For start-up programs, evaluations can provide process data on the successes and

challenges of early implementation; and, for more mature programs, evaluations can provide outcome

data on program participants. While evaluation is not without challenges, the information obtained from a

program evaluation can help to streamline and target program resources in the most cost-efficient way by

focusing time and money on delivering services that benefit program participants and providing staff with

the training they need to deliver these services effectively. Data on program outcomes can also help

secure future funding. Finally, sharing findings and lessons learned will assist other out-of-school time

programs in achieving their goals.

REMEMBER THE 5 REASONS EVALUATION CAN HELP A PROGRAM

Reason # 1: A program evaluation can find out “what works” and “what does not work.”

Reason # 2: A program evaluation can showcase the effectiveness of a program to the community

and to funders.

Reason # 3: A program evaluation can improve staff’s frontline practice with participants.

Reason # 4: A program evaluation can increase a program’s capacity to conduct a critical self

assessment and plan for the future.

Reason # 5: A program evaluation can build knowledge for the out-of-school time field.

Source: Adapted from the Program Manager’s Guide to Evaluation, U.S. Department of Health and Human Services,

Administration for Children and Families

EXT STEPS: ADDITIONAL RESOURCES FOR PROGRAMS

The Program Manager’s Guide to Evaluation. This guidebook, developed by the Administration for

Children and Families, provides program managers with information and instruction on how to use

evaluation to improve programs and benefit staff and families.

Available online at:

http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/pmguide/foreword_pm

guide.html

W.K. Kellogg Foundation – Evaluation Toolkit. The W.K. Kellogg Foundation has developed an

evaluation toolkit designed to give guidance to programs embarking on evaluation.

Available online at:

http://www.wkkf.org/Default.aspx?tabid=90&CID=281&ItemID=2810002&NID=2820002&Langua

geID=0

United Way of America – Measuring Program Outcomes: A Practical Approach. United Way of

America has developed a step-by-step evaluation manual for health, human service, and youth and family

serving agencies.

Available online at: http://national.unitedway.org/outcomes/resources/mpo/

U.S. Department of Health and Human Services, Administration for Children and Families, Administration on Children,

Youth, and Families (2003). The Program Manager’s Guide to Evaluation. Washington, DC.

Ibid.

The Program Manager’s Guide to Evaluation from the Administration for Children and Families recommends that 15% to

20% of total funds allocated for a program be used for conducting a process and outcome evaluation. However, if a program is

conducting only a process evaluation or a single component of an evaluation, such as developing a logic model, or if the

program has a small operating budget, then a smaller percentage of funds may be needed.

See Bronte-Tinkew, J., Allan, T., Joyner, K. (2007). How to Choose an Evaluator. Research to Results Brief, Washington,

DC: Child Trends.

Adapted from The Program Manager’s Guide to Evaluation, U.S. Department of Health and Human Services (2003).

SPONSORED BY: The Atlantic Philanthropies

4301 Connecticut Ave, NW, Suite 350, Washington, DC 20008, www.childtrends.org