Abstract:
This video-based case explores how an impact evaluation for a promising after-school program in New York City went awry. In 2006, New York City's Mayor Michael Bloomberg created the Center for Economic Opportunity (CEO). Designed as an innovation fund, CEO tested anti-poverty programs by applying an evidence-based approach. One of CEO's programs, Teen ACTION, was developed to provide after-school service-learning opportunities to at-risk youth. Teen ACTION's model was based on a rigorously evaluated program which had proven to significantly reduce rates of teen pregnancy and course failure among participants when compared to a control group. But when it came time to evaluate Teen ACTION's impacts, CEO ran into significant hurdles. With rare candor, CEO officials Kristin Morse and Carson Hicks discuss on video how flaws in the program's evaluation strategy, coupled with random unforeseen events like the Swine Flu, rendered the evaluation results unusable.
The case includes a short backgrounder on Teen ACTION and two videos (totaling approximately 9 minutes) designed to be used during the class to frame specific discussion pastures. In the first video, Carson Hicks, Director of Programs and Evaluation at CEO, and Kristin Morse, CEO's Deputy Executive Director, describe the evaluation design and the implementation challenges the team encountered. The second video reveals what the CEO team decided to do once they realized the results of the initial evaluation were flawed. Information on accessing the videos is available to instructors in the teaching plan.
Learning Objective:
The challenges the CEO team experienced with the Teen ACTION evaluation illustrate key issues related to conducting impact evaluations, such as importance of good evaluation design, key problems of data collection (including low response rates and social desirability bias), internal validity, external validity, and statistical power.
Please note: the video portion of this case is included in the teaching plan and is intended for instructors to use in class. Here is more information on how to access teaching plans.
