Ten Simple Steps to Plan Your Next Program Evaluation
In this data-driven world, many organizational leaders understand the importance of using data to evaluate whether their programs are achieving intended goals. However, conducting a program evaluation can be a daunting process, and many leaders are unsure of where to begin. Here at Category One Consulting, we have found that developing a comprehensive evaluation plan can be a great way to make the process more manageable, while also addressing roadblocks to success early. Today, we wanted to share ten simple steps to help you develop a comprehensive plan to guide your next evaluation.
Using this simple ten-step process to develop a comprehensive evaluation plan will provide you with a roadmap to guide your evaluation.
Describe your program. This critical first step outlines what a program does, why it exists, and its implementation. These elements frame why and how a program should be evaluated. Make sure you document the program's purpose, the need it is addressing, and your target population. Include a list of core activities that drive the mission and the resources that help you maximize impact.
Identify your goals and research questions. Evaluation goals establish the overall evaluation purpose while research questions specify what you hope to learn. These components set the foundation and focus of the plan and help determine the design and data sources. First, think about the purpose of your evaluation to identify your goals. Evaluation frameworks tend to fall into two main categories – impact evaluations, which answer if a program is effective in achieving outcomes, and implementation evaluations, which assesses whether a program is being implemented as intended. Then, identify your research questions. Your research questions should align with the evaluation purpose. Remember to keep it simple. Try to think about your questions broadly at first, and then get more specific later after determining the best ways to answer them.
Determine the intended outcomes. Intended outcomes are the results a program aims to accomplish. Identifying specific outcomes in your plan helps you understand whether your program is achieving its goals. First, think about what success looks like for your program, and align intended outcomes with your ultimate goals. Think about the needs you are trying to address or what you are trying to change to identify intended outcomes. Then, consider how it should change. Finally, think about who or what will benefit from this change. Putting all three pieces together will help you identify clear and meaningful intended outcomes. Don't forget to identify outcomes over the short-, intermediate-, and long-term. To get to the "big" outcomes we often think of during evaluation, we first must think about the smaller steps required to get there.
Identify your data sources. Data sources are the measures and metrics used to collect and organize data. Identifying specific data sources in your plan will help determine what measurable elements you can use to understand how well your program is achieving its goals. When identifying data sources, take time to translate your theoretical goals and outcomes into practical and measurable indicators. First, think about what data sources are already available. Do any metrics you currently report seem to fit with any of your research questions? Next, identify the key performance indicators that can help you achieve your evaluation goals and answer your research questions. It often helps to consider the outcomes you developed and identify the measurable or observable elements you can track to measure success.
Select an evaluation design. The evaluation design informs how data will be collected and analyzed to help you answer your research questions. It is crucial to select a design that can answer your research questions using the indicators and data sources you identified. There are several common evaluation designs, including experimental, quasi-experimental, longitudinal, and cross-sectional. Each design has important uses, so make sure to do some research to fully understand which one makes the most sense for your evaluation.
Determine a data collection strategy. A data collection strategy will help you understand how and when your data will be accessed or collected to build a realistic evaluation timeline. Collection strategies include considering data ownership, data access, and the deadline for existing and new data collection. Collecting new data usually means you can get the exact data you need. However, don't underestimate the time required to develop new tools or create new documentation systems. If you hope to use several new tools, consider professional help with tool development.
Define the data analysis plan. A data analysis plan serves as the roadmap for organizing and analyzing your evaluation data. Including a data analysis plan in your evaluation plan will ensure you can answer your research questions. Think about the broad categories of quantitative and qualitative analysis that will help you answer your research questions. Quantitative analyses might include descriptive analyses to provide a snapshot of what's happening, comparative analyses to show differences between groups, or relational analyses to show meaningful relationships between factors. Qualitative analysis methods provide a broad framework to translate less defined data (e.g., experiences, quotes, feedback) into defined themes and explanations.
Develop a dissemination strategy. Dissemination strategies are plans for sharing your evaluation findings with potential audiences and can help you ensure you share results with all relevant stakeholders at the right level of detail. Consider who would benefit most from your results. These stakeholders could be staff, program recipients, funders, and other partner organizations. Understand the importance of tailoring your results to your audience. Even the most invested stakeholder only has time and capacity for so much information.
Determine timelines and responsibilities. Determining the timeline and responsibilities for evaluation tasks helps to translate your plan into action. Setting deadlines enables a realistic plan while assigning responsibilities allows for understanding and task accountability. List the essential tasks for each stage of the evaluation and assign a deadline and responsible party to each. Responsible parties could be individuals, teams, contractors, or outside agencies. For any task that relies on outside parties (e.g., accessing another agency's data), be sure to build in extra time for communication, processing, and completion.
Use your evaluation plan. Now that you've spent time developing a comprehensive evaluation plan, it's time for action. First, save your evaluation planning materials in an easily accessible location so you can update and document your progress over time. Review the evaluation plan to ensure the content fits well with your program needs. Share your evaluation plan with staff, funders, and other relevant stakeholders to promote alignment and transparency. Finally, follow the evaluation plan to develop data collection tools, gather and analyze data, and manage timelines and responsibilities for critical steps.
Using this simple ten-step process to develop a comprehensive evaluation plan will provide you with a roadmap to guide your evaluation. Just remember, regularly reviewing progress with all responsible parties will help ensure your plan leads to a successful evaluation. If you have any questions about developing a plan to guide your next program evaluation, feel free to reach out!