Lions and tigers and… network evaluation? Oh my!
Kerstin Tebbe, Founder, Collective Mind
EVALUATION… UGH. We know evaluation is important — how do we know if we’ve achieved any outcomes if we can’t measure them? — but it’s oh so complicated. And that’s before we even consider the in-built complexity of the networks we’re trying to evaluate. There are so many factors to consider and so many different ways to look at how to measure, monitor, and assess networks. Oh boy.
Let’s find a straightforward starting point to help us. We can try to navigate through the weeds by establishing a broad framework for evaluation that focuses on 1) what to measure and 2) how to measure it.
Now let’s break it down once more. For networks, we can measure two core components of the “what” —
● internal benchmarks — of the network’s a) efficiency and b) effectiveness
● external benchmarks — of the network’s impacts
— each with potentially unique methods for doing so (the “how”).
Let’s look at each “what” briefly and just dip our toe in the water of the “how”.
First, internal benchmarks
The internal benchmarks of efficiency and effectiveness focus on organizational strategy and operations in a more traditional sense.
Efficiency can be defined as the way the network is set up and operates in order to minimize the transactional costs of collaboration. This includes various operational systems and processes as well as the organizational components that contribute to or structure those operations such as:
● network management including systems and procedures for communications, IT, HR, etc.
● network membership including how the membership is organized, how members are oriented to network goals, how member commitments are made and followed through upon, as well as member roles and responsibilities
● network governance including structures and processes for how strategy is determined and decisions are made
● network leadership including the ability of formal (and perhaps informal) leaders to serve as brokers, and facilitators, to foster relationships among members, and build trust and consensus
● resources and sustainability including human and financial resources and how they’re developed and managed
Effectiveness is the ability of the network to meet the needs and expectations of its members. The members are a network’s raison d’être: if the network is not serving its members, it ceases to exist as such. Network effectiveness thereby focuses on engagement, participation, and involvement of members and the relationships with and between members. Measures of effectiveness include:
● the degree to which relevant parties are included in the network and its activities
● the degree to which the network is supported by key actors both inside and outside the network
● the ability of members to commit to joint action
● the degree to which members commit to the collective whole
● the degree to which participants are open in their communications with each other[1]
Halfway there! But…
Don’t forget the external benchmarks!
A network’s impact is the influence and effects that the network has on the complex problem it seeks to address. Impact for a network means progress towards achieving its purpose as articulated in the objectives laid out in its strategy as well as unexpected outcomes.
Where specific activities are undertaken by the network to reach its strategic objectives, traditional measures of the degree to which tasks are accomplished can be used to measure the achievement of those objectives. However, networks’ impacts go beyond task completion.
Understanding a network’s influence on a complex problem requires measuring systems change. One way of doing this is to focus on the elements of systems change, for example, as laid out by John, Kania, Mark Kramer, and Peter Senge which include: policies, practices, resource flows, relationships and connections, power dynamics, and mindsets.[2] Measures are needed to identify where and how the network may have influenced any of these elements of systems change.
A major challenge in assessing a network’s impact is that of determining correlation and/or causation. Given the complex systems that network’s work within and the diversity of activities and processes at play at any given time under a network’s auspices, determining causation — wherein we can explicitly ascertain that an action or process produced the outcome — is almost impossible. Correlation, wherein there is an evident relationship or connection between things, is likely the most robust type of outcomes we can find. That is not to say that correlation isn’t relevant to achieving impact, it is. Likewise, unanticipated outcomes should also be considered as evidence of impact. Identifying unanticipated outcomes requires nontraditional methodologies like outcome mapping or outcome harvesting.
For all three of these areas — efficiency, effectiveness, and impact — we must identify the appropriate indicators and relevant methodologies. Once we determine what we are assessing, we have to determine how to do so. Again, given the central role of the membership within networks, whatever methods are used must always be participatory and engage as broad a relevant group of members and other stakeholders as possible.
Good work, you made it! Well that wasn’t so bad, was it? :-)
[1] From “Evaluating Network Arrangements: Toward Revised Performance Measures”.
[2] The Water of Systems Change. John, Kania, Mark Kramer, and Peter Senge.
For this edition on network capacity, we looked at three sources:
“Evaluating Network Arrangements: Toward Revised Performance Measures”
“Do Networks Really Work? A Framework for Evaluating Public-Sector Organizational Networks”
