Exam 6: Evaluating Program Effectiveness: Connecting Process and Outcomes

Correct Answer:

Verified

Data management systems range from the collection of information from program participants onto paper forms that is then tallied by hand to the use of specialized applications on smart phones or computers that collect information directly from the intended program beneficiaries and then translate the raw data into answers to specific questions defined by the program administrators. Many organizations employ some electronic data recording, using a Microsoft Excel spreadsheet or a Google sheet, with the spreadsheet set up to tally totals, compute means, or otherwise provide some summary information. The development of a data management system should be clearly tied to the types of questions that the program needs to answer or would like to be able to answer in the future. To create a useful data management system, it is helpful to have some skills in working with spreadsheets. Most programs we have worked with have some capacity to use Microsoft Excel or Google sheets. While using a database program like Microsoft Access can be helpful, most programs do not have the ability to use and modify Access databases effectively (nor do we), so use of Access would require outside expertise to be consistently involved. This adds cost to the evaluation and is contrary to our general inclination to help programs develop the capacity to conduct their evaluations themselves with the need for minimal outside help or ongoing support.
In helping programs create a basic data management system, you need a clear idea of what questions need to be answered. There's no point in developing a complex system that can answer questions in which program stakeholders have no interest, because that will require more work, and runs the risk of being confusing and underutilized. Yet, at the same time, the development of a data management system provides a good opportunity to revisit the current and potential needs for data.
The data management system provides the framework to enable the program to answer questions of importance. A good system allows the data to be collected with as little time and effort as possible and then automates the calculations and comparisons that are needed to answer questions. Summary statistics and graphs can then be provided in real time and copied and pasted into reports to share with stakeholders. Investing time and energy into the creation of a data management system that provides accurate and up-to-date information about the program can help the program answer questions about how effective the program is, as well as how its implementation contributes to its effectiveness.

Correct Answer:

Verified

The first step in understanding the role the program might play in effecting change is to be clear about what change is intended, desired, and/or likely, given the program's logic model or the theory of change for the program. It helps if the theory of change is sufficiently detailed to specify the program elements or actions that are used to effect types of changes. Even when there is a clear and specific theory of change or logic model, before developing an evaluation methodology it is important to clarify the specific desired outcomes expected by the program. It is also important to clarify what the most important potential outcomes of the program might be, because we rarely have enough resources to conduct a fully comprehensive evaluation that answers all the questions that stakeholders might have.
Having a good partnership with direct and open communication is particularly helpful at this point, to consider what program "effects" might occur under different sets of circumstances--for different participants, for those not directly participating in the program (e.g., parents might be affected by a program that is explicitly designed to improve the behavior of children), for those who are culturally different from the mainstream, or for participants of different genders or ages. Such discussions should lead to the identification or development of the specific indicators or measures to allow us to determine to what extent those changes are occurring. Measures need to be reliable and valid as much as possible, with the recognition that there are times when a desired outcome is not easily measured and not clearly defined. An assessment of change generally implies that we measure the construct in question at least 2 times, so we can see if there is a difference between a "pretest" or baseline value and a "posttest" value. There are many variations on a "pretest-posttest" design that involve such measurement at two points in time and allow a direct assessment of change among the program participants. However, there are many cases when having a "pretest" does not make sense or is not possible. When assessing the difference between observations at different times (i.e., pre- vs. posttest) or among different groups (e.g., treatment vs. comparison group), it is desirable to determine if that difference is statistically significant, or not likely to occur simply by chance. The use of the term "desirable" is very deliberate, although some researchers would suggest that it is "essential," because without statistical significance, you should not consider the difference as a real difference. Since we assume that readers of this book have a basic background in research design and statistical analysis, we will not spend time explaining the meaning of statistical significance. However, it is important to recognize that there are many times when it is unlikely that a difference will be statistically significant, because of a low number of participants, unreliable measures or interventions (different participants get different "treatments"), highly variable samples of participants or a host of other reasons. In those cases, there may be a "real difference," not due to chance, but we just cannot detect it. Yet, if we have a sizable apparent (but not statistically significant) difference, under some circumstances this might be appropriately used by a program manager as a reason to modify the program.

Correct Answer:

Verified

The key principles utilized in the development of a data management system are:
Assess the leadership and staff's capabilities and motivation early on to determine whether they have the necessary ability and willingness to change their processes for collecting and managing data. We can support skill building and technical competence enhancement, but an unwillingness to try something different can bring some challenge. One common circumstance we face is that, in some programs, the staff are clearly and unalterably opposed to using electronic devices to collect data when interacting with program participants, because they feel that it disrupts important relationship-building interactions. It can be useful to demonstrate to staff how they can become able to type or write on a tablet as easily as they can write on paper, but sometimes that does not overcome their reluctance. While direct entry can certainly save time, a compromise can be to have staff use paper when interacting with the program participants, and then use what they have recorded on the paper to enter the data into the tablet or a computer. Alternatively, program management may decide to shift responsibilities to enable those who are willing to enter data directly as they interact with participants to do so. Regardless of the strategy used, it is important to monitor the accuracy of the data and the impact on relationships to ensure that there are minimal negative side effects of changes in procedures.
Create a system that allows data to be recorded as close as possible to the time that the data are generated. Direct, electronic recording of the data at the time the data are being collected is certainly an efficient use of time, but it is also necessary to consider the impact of that on relationships and on the quality of the data more generally. In one setting, we urged staff to collect data from preschoolers using a tablet to allow the children to view and respond verbally to the stimuli and the staff person would touch the screen to record the response. However, when the staff tried that approach, they found that the children would constantly try to touch the screen, often making responses that were inconsistent with their verbal responses, and they would then become upset when not allowed to touch the screen. This process became more disruptive and time consuming than desirable, so they reverted to paper versions of the stimuli, with staff keeping the tablet in their hands to record responses.
Record the most detailed, granular data possible. Although it is tempting to record data only in the form that is currently needed for decision-making (e.g., if there are certain items that are summed into a scale, to only record the scale score that is used for a decision about a participant's progress), there are several reasons to avoid this practice. First, this often means that a staff person has to manually add up the scores and record the sum, and the more calculations the staff person has to make, the greater likelihood of errors. Second, sometimes it is useful to modify measures. It is not uncommon to find that measures have items that are not useful for a given set of circumstances, and using a smaller number of items may provide clearer results and save time (however, it is crucial to be judicious about modifying measures if they are standardized and you are trying to generalize to other populations based on that measure). In sum, collecting more granular data, with the management system manipulating it (via code, formulae, or functions) to create scores, provides more flexibility in the use of the data, at little to no extra cost, and sometimes with a reduction in the cost of data entry and use.
Use multiple sheets within the system to simplify its use and protect the data. For example, it is useful to have one or several sheets for data entry (different sheets might be used for different measures, or different data collection periods), with data protections built in to help ensure that persons entering the data enter it accurately (using the data protection features of the software to create drop-down menus where possible). Other sheets can be used for formulae that compute scores as needed, and for comparisons of different subgroups (e.g., males versus females). Other sheets can be used to construct graphs or tables that can be directly copied and pasted into reports. For more sophisticated users, there are "business intelligence" applications that can help create data dashboards that allow a snapshot of program statistics, but these can be more expensive in time and money than many smaller nonprofits are able to support.