Case #2027.0

New York City’s Teen ACTION Program: An Evaluation Gone Awry

Publication Date: August 29, 2016
Current Stock:

Educator Access

A review copy of this case is available free of charge to educators and trainers. Please create an account or sign in to gain access to this material.

Permission to Reprint

Each purchase of this product entitles the buyer to one digital file and use. If you intend to distribute, teach, or share this item, you must purchase permission for each individual who will be given access. Learn more about purchasing permission to reprint.

This video-based case explores how an impact evaluation for a promising after-school program in New York City went awry. In 2006, New York City's Mayor Michael Bloomberg created the Center for Economic Opportunity (CEO). Designed as an innovation fund, CEO tested anti-poverty programs by applying an evidence-based approach. One of CEO's programs, Teen ACTION, was developed to provide after-school service-learning opportunities to at-risk youth. Teen ACTION's model was based on a rigorously evaluated program which had proven to significantly reduce rates of teen pregnancy and course failure among participants when compared to a control group. But when it came time to evaluate Teen ACTION's impacts, CEO ran into significant hurdles. With rare candor, CEO officials Kristin Morse and Carson Hicks discuss on video how flaws in the program's evaluation strategy, coupled with random unforeseen events like the Swine Flu, rendered the evaluation results unusable.

The case includes a short backgrounder on Teen ACTION and two videos (totaling approximately 9 minutes) designed to be used during the class to frame specific discussion pastures. In the first video, Carson Hicks, Director of Programs and Evaluation at CEO, and Kristin Morse, CEO's Deputy Executive Director, describe the evaluation design and the implementation challenges the team encountered. The second video reveals what the CEO team decided to do once they realized the results of the initial evaluation were flawed. Information on accessing the videos is available to instructors in the teaching plan.

Learning Objective:
The challenges the CEO team experienced with the Teen ACTION evaluation illustrate key issues related to conducting impact evaluations, such as importance of good evaluation design, key problems of data collection (including low response rates and social desirability bias), internal validity, external validity, and statistical power.

Please note: the video portion of this case is included in the teaching plan and is intended for instructors to use in class.  Here is more information on how to access teaching plans.

Other Details

Teaching Plan:
Available with Educator Access
Video Producer:
Patricia Garcia-Rios
Case Author:
Anjani Datla
Faculty Lead:
Dan Levy
Pages (incl. exhibits):
United States