Nonprofit organizations and program evaluators are increasingly being called upon to evaluate multi-stakeholder initiatives. These initiatives often depend upon collaboration among various organizations and agencies. As a consequence, the need to evaluate collaborations/coalitions has become an important requirement for a wide-range of contemporary program evaluations.
In a recent article, “Evaluating Collaboration for Effectiveness: Conceptualization and Measurement,” (American Journal of Evaluation, 2015, Vol. 36, p. 67-85) Lydia I. Marek, Donna-Jean Brock and Jyoti Savla discuss the customary difficulties in assessing collaborations, including, “lack of validated tools, undefined or immeasurable community outcomes, the dynamic nature of coalitions, and the length of time typical for interventions to effect community outcomes…the diversity and complexity of collaborations and the increasingly intricate political and organizational structures that pose challenges for evaluation design” (p.68).
Building on previous research by Mattesich and Monesy (Collaboration: What Makes it Work? Fieldstone Alliance, 1992) which outlines a six category inventory of characteristics typical of collaborations ( i.e., environment, membership characteristics, process/structure, communication, purpose, and resources), Marek, et al., argue for seven key factors that typify successful inter-organizational collaborations and coalitions:
1. Context—the shared history among coalition partners, the context in which they function, and the coalition’s role within the community
2. Membership—individual coalition members’ skills, attitudes, and beliefs that together contribute to, or detract from, successful outcomes
3. Process and organizational factors like flexibility and adaptability of members, and members’ clear understanding of their roles and responsibilities
4. Communication—formal and informal communication among members, and communication with the community
5. Function—the determination and articulation of coalition goals
6. Resources—the coordination of financial and human resources required for the coalition to achieve its goals
7. Leadership—strong leadership skills, including organizing and relationship – building skills
Marek et al. offer a tool for assessing the effectiveness of collaboration, a 69 item Collaboration Assessment Tool (CAT) survey instrument, which asks a series of questions for each of the factors identified above. For example, the “function” domain of the survey asks organizational respondents the following questions:
- This coalition has clearly defined the problem that it wished to address
- The goals and objectives of the coalition are based upon key community needs
- This coalition has clearly defined short-term goals and objectives
- This coalition has clearly defined long-term goals and objectives
- Members agree upon the goals and objectives
- The goals and objectives set for this coalition can be realistically attained
- Members view themselves as interdependent in achieving the goals and objectives of this coalition
- The goals and objectives of this coalition differ, at least in part, from the goals and objectives of each of the coalition members
“Evaluating Collaboration for Effectiveness: Conceptualizations and Measurement,” offers a valuable tool for assessing the often complex inter-organizational relationships that constitute multi-stakeholder collaborations. The CAT offers a system for gathering data for evaluating the quality of collaborations, and implicitly suggests an outline of the key characteristics of successful collaborations. The CAT tool will be useful for organizations who are embarking on collaborations, and for program evaluators who are charged with evaluating the success of such inter-organizational collaborations. To learn about our evaluation methods visit our Data collection & Outcome measurement page.
Resource
“Evaluating Collaboration for Effectiveness: Conceptualization and Measurement,” Lydia I. Marek, Donna-Jean Brock and Jyoti Savla, (American Journal of Evaluation, 2015, Vol. 36, p. 67-85)
