Deck 11: Network Analysis in Evaluation Research

Question 1

Provide a review of the term network analysis. Make certain that you indicate how it might be utilized in addressing evaluation questions and different types of evaluation.

Answer 1

Network analysis is an approach to research that involves theory and methods designed to characterize and explain patterns of relationships (e.g., friendships, referrals) between a set of actors (e.g., people, organizations) in a system (e.g., workplace, coalition). Network analysts start by operationally defining the actors and relationships within a particular system. Next, they collect network data that assess the relationships between each pair of actors in the system. Finally, they analyze these network data, calculating metrics that characterize patterns of relationships across the whole system (i.e., network-level metrics), relationships between pairs of actors (i.e., dyad-level measures), and/or different actors' network positions (i.e., actor-level metrics).
Network analysis can be particularly powerful for answering evaluation questions that are relational in nature and can be applied to both process evaluations of program implementation and/or impact evaluations of program effectiveness. In process evaluations, network analysis is useful when evaluators and their community partners have questions about how relationships affect program adoption or implementation. In outcome evaluations, network analysis is useful when evaluators and their community partners have questions about how a program affects participants' relationships. Network analysis can also easily be integrated with participatory approaches to evaluation that involve community partners in the framing of questions, evaluation design, data collection, and analysis.

Question 2

Explain network analysis in terms of program evaluation

Answer 2

Analysis of network data is possible at multiple levels. At the highest level, the structure of the network as a whole can be described by metrics like its density (i.e., how many relationships are present), reciprocity (i.e., if A $\rightarrow$ B, how often does B $\rightarrow$ A), and transitivity (i.e., if A $\rightarrow$ B and B $\rightarrow$ C, how often does A $\rightarrow$ C). Analysis at the network level can be useful for examining broad patterns of change in a system over time, for example, to evaluate whether a program intended to build relationships was effective. At a lower level, it is possible to explore how two actors are positioned with respect to one another in the network. Analysis at the dyad level can be useful for thinking about how actors might affect one another. Finally, at the lowest level, metrics that characterize the position of individual actors can be examined. Here, the focus is often on actors' centrality or the extent to which they occupy an important position in the network. There are many different conceptions of what constitutes an important position, and thus many different measures of centrality, which makes selecting the correct measure particularly important.
Network analysis can also be usefully applied in the field of program evaluation. Process evaluations provide information on how a program works, typically with the goal of understanding "how a program achieves what it does." While process evaluations can be used solely for explanatory purposes (which would not require network analysis), they are also sometimes used for formative purposes to improve programs. However, certain types of process evaluation questions are relational in nature and might benefit from a network analytic approach. For example, evaluators and/or community partners might be interested in asking: Are participants more likely to attend program meetings if their friends also attend these meetings? Answering this question involves collecting relational data on participants' friendships as well as friends' program meeting attendance. Impact evaluation provides information about the extent to which a program works (i.e., has an impact), typically with the goal of making an evaluative judgment about the program. Network analysis is relevant to impact evaluation in cases in which the evaluated program is expected to have an impact on the network or some feature of the network. For example, an evaluator might be interested in asking: Does a new social skills program lead to classrooms with more friendships between students? Here, the outcome of interest--classroom friendships--involves collecting relational data that enumerate who is friends with whom in the classroom.

Question 3

After completing your reading of the material in this chapter, provide discussion to justify the application of network analysis to a participatory approach to evaluation.

Answer 3

Both process and impact evaluations can differ in the extent to which they employ participatory or partner-oriented approaches. Three dimensions of participatory evaluation include partner selection (who participates?), depth of participation (i.e., the extent to which they participate), and control of the evaluation process (i.e., who makes the decisions?). Network analysis can be readily employed in process and impact evaluations that vary along each of these dimensions. However, when evaluators and their community partners adopt participatory or partner-oriented approaches, some special considerations are warranted (described below):
Who participates?: Network analysis can shed light on both of these considerations and can therefore be a powerful strategy for identifying potential partners for participatory and partner-oriented evaluations. First, because network analysis assesses the relationships between all actors in a setting, it provides a map of all the different subgroups or cliques that exist within that setting. For participatory or partner-oriented evaluations, it may be helpful to ensure that partners are drawn from each of these subgroups. Second, network analyses can be useful for determining who has power or influence. These methods can be used to determine who the opinion leaders are in a setting that might be important to have on board as evaluation partners.
To what extend do they participate?: When employing network analysis in participatory or partner-oriented evaluations, it is important to consider how and when community partners are involved and to take extra steps to ensure meaningful collaboration. For example, if community partners are involved in framing evaluation questions, evaluators may need to facilitate conversations about whether their interests focus on understanding individual actor attributes or relationships between these actors. If the latter, evaluators would need to work with community partners to consider and select relevant network methods and analyses that match these partners' evaluation questions.
Who makes the decisions?: Participatory or partner-oriented evaluations can also differ in the degree of decision-making control wielded by the evaluator or the community partners. Providing training in network analysis to community partners can build their capacity to lead evaluation studies that involve the collection and analysis of network data.