This is the third in a series of six articles that discuss outcome measurement: what it is, how to do it, and most importantly, how it will help your organization. The content of these articles draws heavily on a framework designed by the United Way of America, in addition to the author’s experience and research. While this is not the only system for measuring outcomes, it has been proven effective by many organizations in both Canada and the United States.
In last month’s article, we looked at the preparation needed to effectively implement outcome measurement in your organization. This month we will look at defining outcomes and logic models for the programs you have selected for measurement. This includes identifying and selecting meaningful outcomes and designing a logic model to describe your program, while building the consensus needed for success. After finalizing the logic model, you will be ready to plan how to measure your program’s outcomes.
It’s fine to define
With your working group prepared and all relevant stakeholders bought in to this undertaking, you can now define your selected programs. This definition will form the basis for all subsequent work, and is therefore a critical aspect of this project. The benefits of this step reach far beyond outcome measurement, however. The ability to communicate your programs concisely will help with succession planning, strategic planning, program marketing, and fundraising efforts.
By defining your programs, you will also be identifying which outcomes you would like to measure. During this process you will likely identify many outcomes, with only some of these being selected for measurement. This is appropriate; your success will depend on selecting the outcomes that best represent the change intended by your program. Select the wrong outcomes and your efforts will be at best meaningless – and at worst harmful – to the development of your program. There is no “right” number of outcomes, so don’t worry about how many you select. You also should not worry about how you will measure them or what targets you should set. That will be addressed later. For now, you should be focused on developing a meaningful and complete program description.
Your workgroup can accomplish this by following four simple steps:
- Solicit feedback broadly
- Draft a logic model for your program
- Select outcomes for measurement
- Get feedback on the selected outcomes, refining the logic model as required
Before we get underway, let’s review the concept of outcomes.
Reviewing outcomes
Outcomes are the often intangible and personal changes that are intended by a program’s activities. Some outcomes occur immediately, while others occur later as a consequence of earlier outcomes. Outcomes can therefore be organized into a logical sequence of initial, intermediate and long-term outcomes. The time frame for this progression will vary depending on the work being performed, but the outcomes will always be related in an if-then relationship.
Initial outcomes are usually changes in the knowledge, attitude or skill of the participants. Intermediate outcomes are usually changes in behavior as a result of initial outcomes. Long-term outcomes are usually changes to a participant’s status, condition, or quality of life as a result of the changed behavior. Just as there is no “right” number of outcomes, there is no “right” designation for an outcome. An intermediate outcome for one program may be an initial outcome for another; what matters is the cause-and-effect relationship linking the outcomes.
To paraphrase William Shakespeare: “Outcomes are neither initial nor long-term, but thinking makes it so.”
Gathering input
Once everyone in the workgroup understands outcome hierarchy, you can begin asking others about the program. Programs are different things to different people, and varied perspectives are essential if you want to capture a complete expression. Be sure to seek input from as many internal and external sources as possible. Approach this step as an enormous brainstorming session: gather all input without judgment, as you will be refining the information you receive in the following steps. Watch for unintended positive or negative outcomes, as your program may be achieving things you had not considered; this is very valuable information. A negative outcome is a harmful change occurring with participants. Tracking a negative outcome is necessary to manage for them, and should be considered due diligence.
Resources for this research include:
- Internal documentation, such as program or organizational materials (including complaint logs)
- Internal stakeholders, such as front-line staff and volunteers, board and committee members, program management staff
- Organizations with similar missions or programs, or who offer services frequently used by previous participants
- Qualified outside observers, such as consultants (especially helpful if you are having trouble articulating exactly how a program is helping)
- Current and past participants (or caregivers, where appropriate), focusing on their expectations and actual experiences
Drafting a logic model
Once you have all of the meaningful input you can reasonably acquire, refine this information by drafting a logic model. A logic model is a visual program description listing all inputs, activities, outputs, and outcomes in logical if-then relationships. This can be presented as a table or flow-chart, although the exact format is not important. Inputs, activities, and outputs are generally listed in three unstreamed groups, where as outcomes should be streamed to clarify the logical relationships.
Logic models are useful for understanding program components, outcomes, and scope, streamlining communication for those unfamiliar with the program, such as new staff and funders. They’re also ideal for identifying which inputs, activities, and outputs are needed for ongoing program management, particularly when outcomes are not adequately realized. As with all measurement, this should be performed with care, as poorly chosen metrics can result in misguided management.
Not all of the input you have gathered is required in the logic model; you will likely need to adjust, consolidate, or discard some of what you have received. Choose outcomes that best reflect the program’s intended purpose, and rework what you have to remove overlap and duplication.
You must also consider a program’s influence over an outcome. A program will have a great influence over initial outcomes, but less influence over long-term outcomes; other factors, such as socio-economic status or medical condition, will play an increasing role. Your long-term outcomes should be far-reaching enough to account for the full effect of the program, but not so distant as to be consumed by external factors. You don’t want to hold your program accountable for things beyond its control.
Exclude outcomes that meet any of the following criteria:
- Are irrelevant to the purpose of the program
- Are dependant on the actions of another program or organization
- Include an audience not targeted by the program
- Are insignificantly impacted by the program in an ideal scenario
If your research has identified negative outcomes, you need to track their occurrence in order to avoid them. Negative outcomes are not usually included in logic models, as they don’t describe the program’s purpose and may interfere with the use of the logic model externally. It is best to record these in accompanying documentation that can be separated when the logic model is needed for purposes other than internal management.
Selecting outcomes
Once you have drafted your logic model, you will need to select outcomes for measurement. Don’t consider how you will measure them, or even if you are able to. Instead, focus on outcomes that reflect the intended benefits to participants, and be sure to include any negative outcomes that have been identified. Initial outcomes should almost always be included for measurement, as they are often critical in ongoing program management.
Individual outcomes can be evaluated by answering the following questions:
- Can we reasonably expect the program to have a non-trivial influence on this outcome?
- Would measuring this outcome describe the successes or failings of this program?
- Will the various program stakeholders consider this a valid outcome?
Once you have vetted the individual outcomes, you can evaluate the selected outcomes collectively by answering the following questions:
- Do the outcomes adhere to a logical if-then relationship?
- Do these relationships describe the intended purpose of the program?
- Do the long-term outcomes represent meaningful change for participants?
- Are potential negative outcomes identified and included?
Building consensus
Once your logic model is drafted and outcomes are identified for measurement, present this work to and solicit feedback from anyone who has a thorough understanding of the program in question, and anyone else whose buy-in is critical for continued or future planning and implementation. Use this feedback to refine your logic model and selected outcomes, and then return for further feedback. Repeat this process as many times as is necessary; the consensus built during this stage will make the remaining work easier and increase the likelihood that the resulting analysis will be used.
Next month, we will navigate the crux of outcome measurement: how to actually measure the outcomes you have selected.
Eli Bennett has been serving the Canadian philanthropic sector for seven years. A graduate of Humber’s Fundraising and Volunteer Management postgraduate program under Ken Wyman, Eli has extensive experience raising millions of dollars through various media across Canada. Currently, Eli is applying his passion for objective management to service provision and program design. If you have any questions on applied measurement in the philanthropic sector, please contact Eli at elibennett@gmail.com.
