I started with OPP more than a year and a half ago. I came on board at an exciting time as OPP started on their evaluation journey after many years of managing organizational performance - using data to track performance and outcomes of our signature Pathways to Success program. Organizational performance management makes sure that we meet our goals in an effective and efficient manner. Evaluation moves beyond whether we meet our goals or not, to asking the question, is Pathways to Success making a difference in the lives of young people. Positive findings of an evaluation are powerful proof that we are making a difference in the lives of youth. It can also show a program's strengths and weakness and ways to improve to have an even greater impact in the lives of youth.
To do an evaluation right, especially the first time around, there are a few important elements to incorporate into the evaluation. First, you want to make sure that the theory of change is clear, outlining how the activities that you are doing is leading to the outcomes you expect. For example, a hypothetical theory of change could outline that classroom instruction (an activity) is responsible for graduating from high school (an outcome). Second, you want to make sure that what is outlined in your theory of change is what is actually happening in the program. This second element is called a process or formative evaluation. Third, you will need to compare the youth from your program to similar youth who have not attended your program. If your youth are doing better than youth who have not attended your program, it shows that your program is making a difference. This third element is called a summative or impact evaluation.
OPP has embarked on the evaluation journey. We have hired Metis Associates, Inc., a reputable company with an established history of conducting evaluations with similar organizations. Our formative evaluation began in October 2012 and will be completed in June 2014. The formative evaluation takes place in three phases: Phase 1 involves reviewing documentation about Pathways to Success, reviewing our data management system, developing data collection tools, and getting approval to proceed to Phase 2. Phase 2 involves the actual collection of data and finally Phase 3 will cover data analysis and reporting. OPP has completed Phase 1 of the evaluation and moved onto Phase 2. We are currently in the middle of our data collection activities.
As part of Phase 2, Metis also provides quarterly reports based on their data collection. In May 2013 we received our first quarterly report from Metis. The report outlined the evaluation process thus far, including the different methods of collecting data (surveys, observations, interviews) as well as thinking through how our different pathways align with the services we provide. The report also summarized some early, preliminary feedback, including:
Staff had reported that our data management system, ETO, is comprehensive and the data entered was accurate. As with any system, there were areas identified that could do with some further refinement. We continue to work on ways to grow the system to meet the ever growing needs of the staff and the organization.
Youth reported that they highly valued their relationships with their Youth Development Specialist (YDS) and are satisfied with their YDS overall. In support of this, staff and youth noted the importance of building trust. Youth also rated mutual respect as an important quality they would like to see in an OPP staff member.
I look forward to the continuing the evaluation process!
Sean Seepersad is the Senior Director of Research and Organizational Development at Our Piece of the Pie.