Exam 5: Ethical Considerations

Correct Answer:

Verified

According to the principle of common good and equity, "evaluators strive to contribute to the common good and advancement of an equitable and just society." This is probably the most wide-ranging of the Guiding Principles, given that it targets notions of the public interest and the public good. Of relevance to collaborative evaluation is the principle's assertion that "evaluators identify and make efforts to address the evaluation's potential threats to the common good especially when specific stakeholder interests' conflict with the goals of a democratic, equitable, and just society." The notion of social justice thus becomes salient here. From an evaluation perspective, it has been argued that this concept encompasses: Ensuring that research priorities reflect the . . . needs of all socioeconomic communities, particularly vulnerable communities; and promoting institutions and social practices that support capacities for self-determination and ensure respectful interactions. This is best achieved if the individuals and communities who are the object of the research participate as partners at all stages of the research process. Viewed from a social justice vantage point, the Common Good and Equity principle puts forth an ambitious agenda, one that should be distinctively compelling to collaborative evaluators. Evaluators are seen as having an ethical responsibility to engage stakeholders in ways that might extend beyond the narrowly defined parameters of a particular study and address the implications of that study for the achievement of positive community/social change. From a related standpoint, this can be regarded as a shift in emphasis from minimizing individual risk (i.e., do no harm) to "considering the benefits of evaluation to society" as a whole. Discerning these implications can be a difficult task, given the multiple stakeholders involved and the competing/conflicting interests they can represent. Indeed, extracting the nature of the overall public good in such circumstances is probably more of an ideological endeavor than an ethical one. What does seem clear, however, is that collaborative evaluators have embraced, ethically and otherwise, an orientation that is more rather than less inclusive with respect to defining the public good within the context of their work. At least this is what occurs at the level of espoused theory. The extent to which this emphasis is reflected in actual practice is an empirical question, one for which little beyond anecdotal data are available.

Correct Answer:

Verified

The core statement of Systematic Inquiry is that "evaluators conduct data-based inquiries that are thorough, methodical, and contextually relevant." In order to fulfill this principle, evaluators should, among other things, "adhere to the highest technical standards appropriate to the methods being used." Perhaps the most prominent one involves the commitment to "highest technical standards." Evaluators typically have much more training in research methodology than the stakeholders with whom they work. They know what designs are likely to produce the most defensible conclusions regarding a program's impact, what data-collection strategies hold the greatest promise for obtaining valid evidence in a given context, and, overall, what the chances are that a particular evaluation can answer the fundamental questions that stakeholders bring to the project. Of course, these stakeholders may possess local knowledge and insight that can inform discussion of all these issues. This challenge can be especially daunting when the advocacy comes from powerful stakeholders. In his survey of AEA members using case vignettes, it was found that the more power or influence a stakeholder group was perceived to have over the logistics of an evaluation, the more willing the AEA respondents were to modify the evaluation's design to accommodate the stakeholder's concerns. This result suggests that the impact of stakeholder input on evaluation design may be less related to the technical merit of that input than it is to the political clout of those offering it. We thus have a cautionary tale. In theory, engagement-oriented evaluation approaches may strive to honor the contributions of all relevant stakeholders. In practice, however, it is probable that not all stakeholders are treated equally, a dynamic of which evaluators might not even be fully aware.
The other major ethical consideration raised by Systematic Inquiry involves the assertion that evaluators "communicate methods and approaches accurately, and in sufficient detail, to allow others to understand, interpret and critique their work . . . [and] discuss in contextually appropriate ways the values, assumptions, theories, methods, results, and analyses that significantly affect the evaluator's interpretation of the findings." Stakeholders not only differ in their power, they also vary in their exposure to evaluation and to experiences (e.g., formal education) that can have implications for their ability to readily grasp the nuances of evaluation practice. These differences can place a significant communication burden on evaluators who are committed to substantively engaging a wide variety of stakeholders. Presentations may need to be modified for different audiences and include the preparation of multiple written reports of the same evaluation. The labor-intensiveness of these activities can tempt the evaluator to forego them, in the hope that a "one-size-fits-all" approach will suffice. This hope is risky, however, unless it derives from experience working with the stakeholders in question. In the absence of such experience, using a standardized approach can estrange rather than engage the very groups with which one is attempting to collaborate. Thus, in the preceding example, a report that was not responsive to the advisory board's modest level of methodological knowledge would likely be confusing and intimidating to these stakeholders, who in turn might be reluctant to ask questions of the evaluator that could lead to greater understanding. In these cases, the evaluator's "espoused theory" of collaboration is not consistent with his/her "theory in use," an inconsistency that has ethical significance in that it constitutes a failure to implement a core component of one's evaluation philosophy.

Correct Answer:

Verified

According to the Integrity principle, "Evaluators behave with honesty and transparency in order to ensure the integrity of the evaluation." This principle applies the notion of transparency to every phase of the evaluation from entry/contracting through utilization of results. Stakeholders should have a thorough understanding of the major decisions affecting the design and implementation of the evaluation, the reasons for those decisions, and the consequences of those decisions. Furthermore, the results of the evaluation must be accurately reported, and realistic steps taken to prevent those results from being distorted or misapplied by key stakeholders in the aftermath of the evaluation. In partnership-oriented evaluations, evaluators need to facilitate discussion of these considerations and their implications if truly shared decision-making is to be achieved.
Ensuring the integrity of an evaluation is never a rote task, but in the case of collaborative evaluation it can be especially challenging. Developing a high level of trust among participating stakeholders is essential and requires "relationship-building, spending time together listening to each other's concerns, interests, and needs, and incorporating them into the research agenda." A conflict of interest occurs when "a primary interest (e.g., validity of research) is unduly influenced by a secondary interest (e.g., financial gain, academic success, community leadership)." For instance, evaluating a program that is directed by a close friend would generally be regarded as a conflict of interest for the evaluator (and the director). Or consider an evaluator who expects that a lucrative new contract would be forthcoming if he or she produced a positive evaluation of a particular program. In both cases, it is reasonable to believe that the evaluator might be tempted to skew the study's findings (violating integrity and honesty) in order to maintain a personal relationship or produce financial benefits.
As the number of evaluation stakeholders increases and the networks within which they operate become more elaborate and complex, the "opportunities" for conflicts of interest to occur undoubtedly grow as well. Thus, virtually by definition, collaborative evaluations generate a landscape populated by stakeholders who are likely to see their vested interests linked to the evaluation. The existence of a conflict of interest does not necessarily mean that unethical behavior is inevitable, but it does set the stage for such behavior. Moreover, where conflicts of interest are involved, the appearance of impropriety can be as damaging to an evaluation as the reality of misconduct. Thus, the evaluator needs to ensure that measures taken to address conflict of interest issues are in fact viewed by stakeholders as credible.