This is the fifth in a series of six articles that discuss outcome measurement: what it is, how to do it, and most importantly, how it will help your organization. The content of these articles draws heavily on a framework designed by the United Way of America, in addition to the author’s experience and research. While this is not the only system for measuring outcomes, it has been proven effective by many organizations in both Canada and the United States.
In last month’s article, we learned how to select outcome indicators and identify factors that can influence outcome achievement. This month, we will discuss how to design and pre-test your data collection instruments and methods.
Sourcing your data
At this point, you should have a thorough understanding of your program, including the outcomes sought and the indicators used to measure them. Before you design the instruments and methods you’ll use to measure those indicators, you must first identify the sources from which you can draw the needed data. These sources will vary depending on the work you do. Some common sources include:
- Records, from your organization or from outside sources.
- Specific individuals, such as participants, staff, or others who can offer relevant input.
- General public or targeted populations.
- Trained observers who can assess the relevant factors.
- Mechanical tests and measurements providing empirical data.
Have your workgroup brainstorm the possible sources of indicator data. Once these possible sources are identified, ask “will this data source provide useful, reliable information about the chosen outcome and indicator?” It is useful to keep a record of the potential data sources and the decisions made about them to inform any future changes to the measurement system.
Planning your approach
Once you know where to look for the appropriate data, you will need instruments to acquire the data and must know how to use them properly. Your selection of data collection instruments will be informed by the sources you have identified. How you collect your data is important; consider seeking input from people experienced in the data collection method you choose. You may want to consider outsourcing this work to a professional or experienced volunteer.
Data collection presents a trade-off between cost and return; you will need to carefully balance the resources you expend with the benefit the resulting data delivers. You will want to consider the money and time required, as well as the training needed to collect the data using various methods. Anticipated response rates and data reliability can vary with the method chosen, and this too must be considered. In the end, you want data that is useful to program managers and credible to those outside of the program, while remaining feasible and efficient.
If an indicator seems too difficult to measure, explore alternatives before dropping the indicator. Should you decide not to measure an indicator, keep the indicator in the logic model and state the reason measurement was abandoned. This will provide future decision-makers the ability to make informed choices about changes down the road, and provides an easy case for funders interested in learning more about your program outcomes.
Tuning your instruments
Before designing your data collection instruments, save some time by reviewing existing material. There are many books published on the subject, and many established scales, observation guides and questionnaires that can be adapted for your purposes. Organizations such as Imagine Canada can provide information and advice, as can foundations, colleges, universities, and other charities. By reviewing pre-existing material, you will learn different approaches to sample selection, training of collectors, timing for data collection, cultural diversity, and other challenges.
When you start developing your own data collection instruments, don’t collect information just because it seems interesting. Extraneous inclusions will waste resources every time you measure, so collect only what you need and none of what you don’t.
Check your work by first ensuring you will capture everything you need:
- Compare the questions to the outcome indicators. Are all indicators accounted for by the questions?
- Draft the charts, tables and graphs you intend to use when presenting the data. Can you identify the specific questions that provide this data?
If you answer no in either case, you are missing essential questions.
Next, trim what is not needed. If a question does not relate directly to an indicator, eliminate it unless you are sure you need it and you have a plan to use the data. Before finalizing your data collection instruments, be sure to get input from the person responsible for data entry.
Define your procedures
With your data collection instruments developed, you must define how the instruments will be used. This includes when the data will be captured, who will be included in measurement, and what the collectors need to know to do their work. You should also give careful consideration to the need for privacy protection.
Timing of measurement must be tied to program delivery, as this ensures consistency across all samples, keeping the data meaningful. Consider collecting data both before and after participation in the program and follow up six to eighteen months after completion of the program, although this will depend on the nature of your work and the outcomes chosen. While long-term outcomes are the most important to measure, they often pose significant measurement challenges; follow-up beyond two years is often best performed via ad hoc studies rather than regular measurement.
You also need to define the scope of your measurement sample. If your program works with individuals, how will you handle those who don’t complete the program, or who apply but aren’t accepted? For programs that target a population, such as awareness or crime reduction campaigns, will you measure the entire group or a targeted sample? Be sure to consider statistical rules on sample sizes and sample selection; this is an area where your skilled volunteer or hired professional can be very helpful.
In application of measurement, consistency is paramount. Select a limited number of people who have not worked directly with participants to be thoroughly trained on clearly defined procedures, including an understanding of the purpose and intended use of the data. Have your data collectors practice the procedures before they go live, and compare the results of the mock sessions to evaluate for consistency in application and results. Ensure those collecting data are bound by a confidentiality agreement, and take any necessary steps needed to protect the privacy of individuals.
Check your work with a pre-test
A pretest is a very limited trial to uncover problems in the draft stage of planning. This modest investment can save you a great deal of frustration and wasted effort later. You can pre-test as instruments and their associated procedures and sampling methodologies become available; you don’t need to test them all at once. Run through everything as drafted, letting everyone involved know that it is a pre-test.
Assess for the following:
- Wording and content of questions
- Adequacy of response categories
- Clarity and comprehensiveness of instructions
- Layout and format
- Length of time required
- Ease of data entry
After running the pre-test, revise as required. If you make substantial changes, repeat the pre-test to ensure no additional changes are needed. Once your pre-tests are concluded, you are now ready to pilot your outcome measurement system. In next month’s final article we will bring all of your planning to life!
Eli Bennett has been serving the Canadian philanthropic sector for seven years. A graduate of Humber’s Fundraising and Volunteer Management postgraduate program under Ken Wyman, Eli has extensive experience raising millions of dollars through various media across Canada. Currently, Eli is applying his passion for objective management to service provision and program design. If you have any questions on applied measurement in the philanthropic sector, please contact Eli at elibennett@gmail.com.
