#program implementation

  Temperature Gauge- Taking a sheet of easel paper, draw an image of a thermometer. Throughout the workshop, you can ask the participants for their “temperature”: How are they being influenced or affected by what’s being shared? Are participants feeling calm, or are they aroused? Are they bored? This will allow you to see if you need to go to move on to another activity or spend more time exploring.

  The Problem Hat- This is a great evaluation example from the Peace Building toolkit, created by Caritas.org. Taking a hat, sheets of paper, and pens/pencils, participants write down an answer to the following: “I am having difficulty with…”. After writing down their difficulty, collect the sheets of paper and placed them into a hat. Participants will then take turns pulling out a sheet of paper and reading what’s written. Participants will then offer a solution that can help increase understanding.

  Bright Dots and Pipe Cleaners- For “Bright Dots”: Take several sheets of your easel paper. Write down an evaluation statement. An example statement could be “I feel confident in applying what I learned today to my school/work life.” Taking red, yellow, and green stickers, the participants will place one sticker under the statement. Red signifies “no”, yellow means “maybe/not sure”, and green is for “yes”. For “Pipe Cleaners”: At the beginning of the workshop, have three boxes placed in an area of the room that’s clearly visible to the participants. One box is marked as “yes”, the second is marked as “not sure”, and the third is marked as “no”. Above the boxes, write an evaluation statement. An example statement could be “I know a lot about the topic of today’s workshop”. Upon entering the room, give the participants one pipe cleaner. Instruct the participant to place their pipe cleaner in the appropriate box. During a group activity, make note of how many pipe cleaners are in each box before removing them. At the end of the workshop, replace the original evaluation statement with a new one. An example can be “I know more about the topic now compared to what I knew before.” Pass out the pipe cleaners again, and as the participants are leaving, have them place their pipe cleaner in the appropriate box.

  Positive and Delta- Using your easel paper, chalkboard or whiteboard, divide a sheet of paper or board space into two sections. On the left side, write “+” (what went well) at the top. On the right side, write “Δ” (improvements for next time). Ask participants to share what they enjoyed about the workshop (inviting them to give feedback on not only the activities, but the flow of the workshop) as well as their recommendations for what can be made better. When asking for feedback on what can be improved for next time, stress to participants to focus on the overall workshop, and not entirely on the logistics. There may be certain logistics (i.e., having the workshop in the morning vs. in the afternoon, physical temperature of the room, not having food, etc.) that can be easily changed compared to other logistics.

  Recommendations

  Brief is better- The longer your evaluation is, the less likely you’ll get quality feedback. Take into consideration that your participants have been sitting for a while and may be itching to leave, whether they enjoyed the workshop or not. If you feel you must have a longer evaluation, allow enough time at the end of the workshop for the participants to complete it.

  Combine different types of evaluations- Generally, written evaluations are used at the end of a workshop, but you an also evaluate throughout the workshop. Poll Everywhere or “Temperature Gauge” can be used mid way through the workshop, followed by a written evaluation at the end.

  Be mindful of how you phrase your evaluation questions- How you phrase your evaluation questions can result in what types of responses you receive. “What didn’t you like about today’s workshop?” sounds different from “In what ways can this workshop be improved?” If a participant writes down or verbally states something they didn’t like, challenge them to come up with 1-2 suggestions for improvement.

  Look at your feedback for common themes- Of course, as the workshop facilitator, you expect to get more positive feedback back than negative, and it may feel discouraging if you see that you’ve gotten more recommendations for improvement compared to what went well. Instead of taking this as a fail, look for common themes as this can be something to consider for the next set of participants. Did several participants not understand a particular aspect of the workshop? Did some participants prefer more group discussion as opposed to individual activities?

  And We’re Done!

I hope you’ve found the Planning and Facilitating Valuable Works series helpful. If you want more in-depth knowledge on creating a workshop (or if you are interested in inviting me to facilitate a workshop for you), read more about my consulting services and contact me to see how we can work together.

RAISE YOUR VOICE: What are some creative ways that you gather feedback from workshop participants? What are some creative ways you've seen workshop facilitators evaluate a workshop?  Share below in the comments!

************************************************************************************************************

If you like this and other posts, subscribe to the Raise Your Voice weekly newsletter to receive resources, advice, and tips to help you raise your voice for women and girls of color.

************************************************************************************************************

Related Posts

Planning & Facilitating Valuable Workshops (Part One): Things to Consider Before Your Begin

Planning & Facilitating Valuable Workshops (Part Two): How to Structure Your Workshop for Maximum Impact

Before we get into the differences between monitoring and evaluation, here is why each element is important:

Evaluation helps us to see what went well and what needs to be smoothed out. In order to be as effective as possible and to promote lasting change, you need to know where your program or workshop stands right now. Sometimes after you conduct an evaluation, you’ll find that all you need to do is a little tweaking, or you may have to do a complete overhaul. Secondly, evaluation helps you to plan for the future when it comes to implementing more programs or workshops.  Third, evaluation also shows your program or workshop’s strengths as well as its hiccups, showing you where the gaps are and how to fill them in.

Monitoring allows you to see how the program or workshop is doing while you’re in the thick of it. You’re making observations, including how the participants are responding to the subject matter, how smoothly the activities flow, if the room is set up to allow movement and if you’ve made things accessible, and if your program is tailored to the appropriate audience, among other things.

The terms “monitoring” and “evaluation” are often used interchangeably, but they do have some key differences. Monitoring occurs during the implementation of your program or workshop’s activities. As mentioned above, you’re constantly collecting data on the overall design and flow and making early analyses.  Evaluation, on the other hand, typically happens at the completion of the workshop or program, keeping in mind that the time frame of a workshop or program can vary. The primary focus of evaluation is to not only provide feedback on why an aspect of your workshop or program was successful (or not), but to also present recommendations on how to improve.

Putting it All Together

There are different types of monitoring and evaluation, with each one building on the last. Based on the Centers for Disease Control's guidance on evaluating HIV prevention programs, these types make up the foundation for effective monitoring and evaluation. They include:

Formative evaluation (the base)- Collects data on the needs of the target audience in order to determine how best to design your workshop or program to address needs, as well as examines the appropriateness of your workshop or program.

Process monitoring- Collects data on the characteristics of the target audience, what services are provided to them, and what resources are available.

Process evaluation- Collects more detailed data on how the activities of the program or workshop were delivered, if the implementation reached the intended audience, and what barriers were observed. (Did you reach the number of people you said that you would?)

Outcome monitoring- Collecting data about outcomes, including behavior change and learned skills. (Were the outcomes expected?)

Outcome evaluation- Similar to outcome monitoring, but compares the audience receiving the workshop or program to a similar group that has not participated in the workshop or program. (Did both groups have similar outcomes? Is what you’re seeing a result of the activities or is it from something else?)

Impact evaluation (the top)- What was the broader impact of the activities in your workshop or program?

Five Key Things to Keep in Mind

Approach monitoring and evaluation from a positive standpoint: Some common misconceptions about evaluation and monitoring are that it’s too expensive, too academic, too time-consuming, too exclusive, and too useless. This couldn’t be further from the truth. If you approach monitoring and evaluation from this viewpoint, it will forever be daunting and you’ll never want to do it, which would be a disservice for you and the communities you work with. Evaluation and monitoring can be engaging, cost-effective, participatory, and very useful when you allow yourself to be open to the process. When you come from a mindset that it’s going to help you improve what you’re doing, you’ll see the usefulness of it.

Vary your tools: Some common ways to conduct monitoring and evaluation include focus groups, needs assessments, surveys, key informant interviews, logic models, public forums, data collection, and even direct observation. Each method has its pros and cons, and can often be based on timing, funding, and even the culture of your target audience (i.e., If your program is for young Latina women between the ages of 18-24, will the evaluator be of the same race/cultural background? This can greatly effect the outcome). 

Ask for help: Many groups and organizations feel that they have to monitor and evaluate all on their own. This also isn't true. I got my start in monitoring and evaluation after college, when I was a graduate research assistant at the Morehouse School of Medicine's Prevention Research Center, then later on as a  graduate student at the Columbia University School of Social Work. In fact, some of the best ways to conduct evaluations is to solicit the assistance of college students, graduate students or volunteers who are eager to gain experience. You can also contact your local college or university, as there are many professors who do this type of work. There are also consultants like myself who are willing and able to assist as well.

Share your findings (and tailor it to your recipients): Sharing your findings is a great way to generate support and buy-in from your community. It’s not just you or your organization that are the only stakeholders. Other stakeholders include funders, policy makers, community leaders, schools, and clients. There are a variety of ways to share what you’ve learned, including fact sheets, social media, public forums, TV, and evaluation reports. You have to tailor how you share your findings to your audience. For example, an evaluation report is probably best suited for funders or executive directors.

Embrace the hiccups: It’s not the end of the world if you discover that your workshop or program wasn’t as effective as you thought it would be. Rather than starting over from scratch, analyze the elements that you’ve gotten the most hiccups on based on feedback (oral feedback, post evaluation, etc.) Try seeing if that activity can be placed in another section of the program or workshop before you scrap it altogether. Really pay attention to the feedback you get from participants. If you’re implementing your program or workshop to the same type of participants and different groups make the same recommendation for how to improve, it may mean that this is a key area to look into. You can always turn a hiccup into a strength.

Monitoring and evaluation can definitely be overwhelming, but not impossible. It can make the difference between a successful program or workshop and one that’s not. Next week, I’ll share with you an evaluation technique that many groups and organizations have found extremely useful, and even a lot of fun!  

  Ready to delve more deeply in strategically improving your programs and workshops? Contact me to learn more about my consulting services. 

Related Posts

4 Proven Ways to Create Effective Programs & Workshops for Youth

*Specific- Hone in on what you want to achieve, and avoid being too broad. From one of my past workshops, I developed this objective: "By using interactive activities, storytelling, and skill-sharing, this workshop will help Black girls, women, and the organizations that serve them to think more critically about the importance of prioritizing self-care and leave with tangible tools to incorporate self-care practices into their lives." While not the best example, I think this shows the importance of being as specific as possible.

*Measurable- How will you know that your program or workshop was successful? For example, "I want to make good grades" is not as measurable as "I want to earn a 3.5 grade point average by the end of the spring semester". How will you know if you've achieved your goal? Teacher evaluations, a final transcript, and final exam grades are some ways to measure your success. In the context of a program or workshop, how you measure your progress depends on several factors, such as the length of your program. Some programs last for 6 weeks while others only last a few days.

*Achievable- Does what you want to do even possible? Again, this may depend on the length of your program or workshop. One thing facilitators tend to do (and I've done in on many occasions) is cram in as much information as possible when you find out how much time you really have. Do you have the resources necessary? Do you have the monetary and staff support? Do you have adequate time to do what you want to do? Do you have the skills to do what you want? If you're developing a program that will span several sessions, you can break down a particular topic and discuss one aspect of it in one session, another aspect in the second session, and so on. If you're only doing a workshop, continue to focus on one aspect and consider creating another workshop on a different aspect. Or, you can touch on one aspect while providing resources for other aspects of your topic (hand-outs, websites, contact information, etc.)

*Relevant- Is the goal of your program or workshop in alignment with your mission or the "bigger picture"? If your workshop focuses on helping youth share their stories of being bullied, how does it fit into your mission of bullying prevention? Make sure your objectives fit into your overall goals.

*Time-specific- How much time do you have to meet your objectives? Is it a 4 week program or a 90 minute workshop? Knowing this will help you get as clear as possible on what you need to do or if you have the luxury of spacing things out.

2. WWWWWWH?: Global Learning Partners, Inc. has created a 7-step design progress used to get you thinking more about developing your program or workshop. WWWWWWH also ties into the S in S.M.A.R.T. What does WWWWWWH stand for?

*Who?- Who will be attending? What is the age range? How much do they know about the subject matter? These questions will guide you into creating content that is most relevant.

*Why?- Why are you developing this program or workshop? What is happening now for you to believe that a program or workshop in the subject matter in necessary? Has there been a spike in sexually transmitted infections among teen girls in your community? Is there an increase of queer young women of color becoming pregnant in your school? These are just some examples. After the completion of the program or workshop, what will the participants do differently?

*When?- What time of day do you plan to conduct your program or workshop? How often? What's the duration? The time of day is one of the most important factors in developing a program or workshop. Will it be during school hours, after school, or on a Saturday morning? If you have more flexibility, you can ask participants for their preferred time, especially if you're doing a program. When it comes to a workshop, you can always request a certain time but be prepared that you may not get the time that you want.

*Where?- Will it be in a classroom or a conference room? Is is accessible for individuals who may need wheelchair access? Is it near public transportation? Do you need permission to use the space?

*What?- What content will be taught? Does the order of the exercises involved make sense? What skills, knowledge, behaviors, and attitudes will help you to gauge if the content you're sharing is being well received? Remember this: Even if you have all the time in the world to do what you want, it's probably best not to overwhelm your participants.

*What For?- What do the participants have to do in order to demonstrate that they're getting the information? (Hint: S.M.A.R.T.)

*How?- What is the best way to present your content that is engaging and informative? Which brings us to #3...

3. Creativity is key: Develop your program and workshop in a way that honors the ways in which people process information. Some people learn better when they see something (PowerPoint presentation, video clip, detailed notes, visualizations) while others need to use their hands (artwork, puzzles) and others prefer to hear what they're learning (music, reading aloud information). When you develop the content for your program or workshop, consider having 3 separate exercises that speak to each type of learning style.

4. Gather feedback before, during, and after: When we ask for feedback, it's typically after the program or workshop is completed. You can gather feedback in a variety of ways, including pre-tests and post-tests and verbal feedback. Outside of a pre-test, feedback typically isn't gathered in the beginning, and no one really thinks to gather feedback while the content distribution is in progress. Here are 2 creative ways I've used for gathering feedback during a portion of a program or during a workshop:

*Poll Everywhere- This is one of the most creative ways to gather feedback from participants, and it utilizes something that young people tend to do often: sending text messages.It's a fun, creative way to gather feedback, and it's very customizable and it allows for your participants to be anonymous. It's also free for an audience number of 40.

*Temperature gauging- Throughout the content sharing, you can ask the participants for their temperature, meaning how are they being influenced or affected by what's being shared. Are participants feeling calm, or are they aroused? Are they bored? This will allow you to see if you need to go to move on to another exercise or spend more time exploring.

For more information on content building and to delve more deeply in strategically improving your programs and workshops, contact me to discuss my consulting services.

Related Posts

Do You Know Who You’re Talking To?: Effective Messaging for Young Women of Color

What Young Women of Color Really Want