Organizations vs. Programs
Organizations are social collectivities that have: members/employees, norms (rules for, and standards of, behavior), ranks of authority, communications systems, and relatively stable boundaries. Organizations exist to achieve purposes (objectives, goals, and missions) and usually exist in a surrounding environment (often composed of other organizations, individuals, and institutions.) Organizations are often able to achieve larger-scale and more long-lasting effects than individuals are able to achieve. Organizations can take a variety of forms including corporations, non-profits, philanthropies, and military, religious, and educational organizations.
Programs are discreet, organized activities and actions (or sets of activities and actions) that utilize resources to produce desired, typically targeted, outcomes (i.e., changes and results). Programs typically exist within organizations. (It may be useful to think of programs as nested within one or, in some cases, more than one organization.) In seeking to achieve their goals, organizations often design and implement programs that use resources to achieve specific ends for program participants and recipients. Non-profit organizations, for example, implement programs that mobilize resources in the form of activities, services, and products that are intended to improve the lives of program participants/recipients. In serving program participants, nonprofits strive to effectively and efficiently deploy program resources, including knowledge, activities, services, and materials, to positively affect the lives of those they serve.
What is Program Evaluation?
Program evaluation is an applied research process that examines the effects and effectiveness of programs and initiatives. Michael Quinn Patton notes that “Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs in order to make judgements about the program, to improve program effectiveness, and/or to inform decisions about future programming. Program evaluation can be used to look at: the process of program implementation, the intended and unintended results produced by programs, and the long-term impacts of interventions. Program evaluation employs a variety of social science methodologies–from large-scale surveys and in-depth individual interviews, to focus groups and review of program records.” Although program evaluation is research-based, unlike purely academic research, it is designed to produce actionable and immediately useful information for program designers, managers, funders, stakeholders, and policymakers.
Organization Development, Strategic Planning, and Program Evaluation
Organization Development is a set of processes and practices designed to enhance the ability of organizations to meet their goals and achieve their overall mission. It entails “…a process of continuous diagnosis, action planning, implementation and evaluation, with the goal of transferring (or generating) knowledge and skills so that organizations can improve their capacity for solving-problems and managing future change.” (See: Organizational Development Theory, below) Organization Development deals with a range of features, including organizational climate, organizational culture (i.e., assumptions, values, norms/expectations, patterns of behavior) and organizational strategy. It seeks to strengthen and enhance the long-term “health” and performance of an organization, often by focusing on aligning organizations with their rapidly changing and complex environments through organizational learning, knowledge management, and the specification of organizational norms and values.
Strategic Planning is a tool that supports organization development. Strategic planning is a systematic process of envisioning a desired future for an entire organization (not just a specific program), and translating this vision into broadly defined set of goals, objectives, and a sequence of action steps to achieve these. Strategic planning is an organization’s process of defining its strategy, or direction, and making decisions about allocating its resources to pursue this strategy.
Strategic plans typically identify where and organization is at and where it wants to be in the future. It includes statements about how to “close the gap,” between its current state and its desired, future state. Additionally, strategic planning requires making decisions about allocating resources to pursue an organizations strategy. Strategic planning generally involves not just setting goals and determining actions to achieve the goals, but also mobilizing resources.
Program evaluation is uniquely able to contribute to organization development–the deliberately planned, organization-wide effort to increase an organization’s effectiveness and/or efficiency. Although evaluations are customarily aimed at gathering and analyzing data about discrete programs, the most useful evaluations collect, synthesize, and report information that can be used to improve the broader operation and health of the organization that hosts the program. Additionally, program evaluation can aid the strategic planning process, by using data about an organization’s programs to indicate whether the organization is successfully realizing its goals and mission through its current programming.
Brad Rose Consulting works at the intersection of evaluation and organization development. While our projects begin with a focus on discrete programs and initiatives, the answers to the questions that drive our evaluation research provide vital insights into the effectiveness of the organizations that host, design, and fund those programs. Findings from our evaluations often have important implications for the development and sustainability of the entire host organization.
Organizations: Structures, Processes, and Outcomes, Richard H. Hall and Pamela S Tolbert, Pearson Prentice Hall, 9th edition.
Utilization Focused Evaluation, Michael Quinn Patton, Sage, 3rd edition, 1997
What a Strategic Plan Is and Isn’t
Ten Keys to Successful Strategic Planning for Nonprofit and Foundation Leaders
Types of Strategic Planning
Understanding Strategic Planning
Five Steps to a Strategic Plan
Five Steps to a Strategic Plan
(See our previous blog posts: “Humans Need Not Apply: What Happens When There’s No More Work?” and “Will President Trump’s Wall Keep Out the Robots?”) Today we’d like to refer readers to an important article, “Dark Factory”, that recently appeared in The New Yorker. “Dark Factory” explores the growing impact of robotics and AI on the manufacturing and service sectors of the U.S. Economy.
In “Dark Factory”, Sheelah Kolhatkar discusses her visit to Steelcase, the manufacturer of office furniture. Steelcase, like much of American manufacturing, has had its economic ups and downs over the years. Kolhatkar describes how, in recent years, the company has increasingly employed robotics as a means to improve manufacturing efficiency, and as a result, now relies on fewer workers than it has in the past. Representative of an ever-growing number of manufacturing companies in the U.S., Steelcase employs fewer and fewer high school graduates, and now seeks college educated employees with technological skills, so that these higher skilled workers can supervise an expanding army of manufacturing robots.
As Kolhatkar shows, while efficiency gains are good for Steelcase and other manufacturing companies that employ robots and AI— and even, in some cases, make work more tolerable and less grueling for the remaining employees on the shop floor— the net effect of these technologies is to displace large swaths of the work force and to shift wealth to the owners of companies. Kolhatkar cites research that shows that the use of industrial robots, like those at Steelcase, is directly related to decline in both the number of manufacturing jobs and declining pay for remaining workers.
Kolhatkar also discusses “dark factories”— factories and warehouses whose use of robots and AI are so extensive that they need not turn on the lights because there are so few human workers. While such factories and warehouses are not yet wide-spread, major U.S. corporations are looking to use robotics and AI to run nearly employee-less operations. Although some companies may not be eager to begin utilizing robotic warehouses, competitive pressure is sure to compel U.S. companies to implement fully robotized facilities, or lose competitive battles with other nations that do adopt these technologies.
AI and Robotics Not Limited to Manufacturing
The result of the growing use of robotics and AI in the U.S. is, of course, the declining demand for workers in what where once fairly labor-intensive human-dominated work environments. (Manufacturing now employees only about 10% of the US workforce, and these jobs are under constant threat by new technologies.) Although displaced manufacturing workers often seek jobs in the service sector, this sector is now hardly immune to automation. MacDonald’s, for example, is introducing “digital ordering kiosks” where customers electronically enter their orders and pay for their meals. MacDonald’s is expected to automate in 5500 restaurants by the end of 2018. Uber and Google continue to invest in the development of autonomous driving technologies, and the U.S. trucking industry is eager to adapt autonomous vehicles so that it can reduce substantial labor costs associated with trucking. Amazon has purchased Kiva, a robotics company, and is developing robots that can zoom around Amazon warehouses and fulfill orders. (A Deutsch Bank report estimates that Amazon could save 22 million dollars a year in each of its warehouses, simply by introducing warehouse robots to replace human workers.)
As Kolhatkar’s “Dark Factory” shows, while the future looks increasingly promising for the shareholders of companies who introduce labor-displacing robotics and AI, it doesn’t appear quite so sunny for those humans who must work for a living—especially in the manufacturing and service sectors of the U.S. economy. Like the “dark factories” that promise to displace them, for many workers, the future too, will be dark.
“Welcoming our New Robotic Overlords”, Sheelah Kolhatkar, The New Yorker, October 23 2017
“AI, Robotics, and the Future of Jobs”, Pew Research Center
“Artificial intelligence and employment”, Global Business Outlook
“Advances in Artificial Intelligence Could Lead to Mass Unemployment, Experts Warn”, James Vincent, The Independent, Wednesday 29 January 2014
What is an “Outcome”
“Outcomes,” i.e., specific changes or results, are what programs seek to produce, and what funders seek to fund. Programs and initiatives, especially in the nonprofit sector, exist to produce valuable and desired changes. While the measurement of outcomes is essential to evaluation, a key question for both programs and for evaluators is, ‘What counts as a desired outcome?”
Typically, education and nonprofit programs strive to see improvements in things like’s students’ reading scores, homeless persons employability, young people’s job-readiness, availability of affordable housing for home-seekers, etc. While the desirability of such outcomes appear “natural” or taken-for-granted, outcomes are, in fact, “agreed upon” events, states, or entities.
Who determines the outcomes that are to be evaluated?
For those who administer and run programs, it is often the case that such programs are highly dependent upon what funders value and desire to see changed. Although desired changes are usually viewed as self-evident, outcomes are a socially and politically defined entity. They depend upon a negotiated understanding of what constitutes a valuable change, and how is such change should be measured or indicated. Additionally, as many program leaders and educators will attest, changes that are desirable, are often shaped by the ability of proponents and researchers to measure such changes. (I’m reminded here, of Einstein’s famous quote: “Not everything that counts can be measured, and not everything that can be measured counts.”) As Heather Douglas points out, “ We must value something to find it significant enough to measure, to pluck it from the complexity of human social life, and to see it as a set of phenomena worthy of study.” (See Heather Douglas, “Facts, Values, and Objectivity”).
Of course, funders alone don’t determine the outcomes that programs produce. Program stakeholders can have a significant influence on what constitutes a desired outcome (See our previous blogpost, “The Importance of Understanding Stakeholders”). In an educational program, for example, a wide range of stakeholders may influence the desired outcomes of programming. Parents, teachers, community members, state and federal policy makers, business interests, politicians—may all influence what counts as a desirable outcome of education. Therefore, what stakeholders value (i.e., view as significant), is often what is viewed as the desired outcome of a program.
It’s important for program leaders and staff, and for evaluators to discuss and identify which changes programming seeks to affect. While evaluators deploy a range of methods to indicate or measure such changes, what counts as a desirable change, as a desirable “outcome,” is a question as critical to the success an evaluation as it is to the success of a program.
The Measure of Reality: Quantification and Western Society 1250-1600, Alfred W. Crosby. Cambridge University Press, 1997.
For a radical critique on the power of the funders of nonprofits to exercise influence on nonprofits goals and outcomes, see “How Liberal Nonprofits Are Failing Those They’re Supposed To Protect,” by William Anderson
Every program evaluation is conducted in a context in which there are parties (persons, organizations, etc.) who have an interest, or a “stake,” in the operation and success of the program. In the corporate world, a “stakeholder” is any member of the “groups without whose support the organization would cease to exist.” (see Corporate Stakeholder). More recently, the idea of “a stakeholder” has been broadened to include “any group or individual who is affected by, or who can affect the achievement of, an organization’s objectives.” (The Stakeholder Theory Summary.)
Indeed, in the not-for-profit world, stakeholders may include an array of persons and organizations including funders, community members, program participants, family members, volunteers, staff, government agencies, and the broader public.
Stakeholders in non-profits usually fall into one of three categories of legal statuses:
- Constitutional stakeholders such as board members or trustees of the non-profit organization
- Contractual stakeholders, including paid staff, or any business, group or individual that has a formal relationship with the organization.
- Third-party stakeholders including all the people and groups that may be affected by what the organization does. That includes businesses, the local government, and the citizens who live in the community. (See What is a Stakeholder of a Non-profit.)
Nonprofit stakeholders may range from those who support an organization, to those who oppose an organization. Stakeholder can include advocates, supporters, critics, competitors, and opponents. In its analysis of stakeholders in policy change efforts, the World Bank uses the categories of “promoter,” “defender,” “latents”, and “apathetics.” (See What is a Stakeholder Analysis.)
Conducting a stakeholder analysis is very useful for both evaluation and strategic planning efforts. Identifying various stakeholders’ interests in an organization’s mission and programming can help non-profit leaders and staff to be sure that their efforts and initiatives are achieving desired goals. They can also be useful in ensuring that the needs of those served are being directly met. For both evaluation and strategic planning purposes, a stakeholder analysis is an important process for achieving a shared understanding of each stakeholder’s specific interest in, and relevance to, the work of the non-profit or educational organization.
Brad Rose Consulting has developed a unique approach to stakeholder analysis, one that can be extremely useful as organizations examine the purposes, goals, specific activities, and desired outcomes of their work. We often work with organizations to implement stakeholder analyses. These analyses are helpful both in identifying where an organization is, at any given point in time, and for identifying where it wants to go in the future. You can see our basic stakeholder analysis form here.
Surveys can be an efficient way to collect information from a substantial number of people (i.e., respondents) in order to answer evaluation research questions. Typically, surveys strive to collect information from a sample (portion) of a broader population. When the sample is selected via random selection of respondents from a specified sampling frame, findings can be confidently generalized to the entire population.
Surveys may be conducted by phone, in-person, on the web, or by mail. They may ask standardized questions so that each respondent replies to precisely the same inquiry. Like other forms of research, highly effective surveys depend upon the quality of the questions asked of respondents. The more specific and clear the questions, the more useful survey findings are likely to be. Good surveys present questions in a logical order, are simple and direct, ask about one idea at a time, and are brief.
Surveys can ask either closed-ended or open-ended questions. Closed-ended questions may include multiple choice, dichotomous, Lickert scales, rank order scales, and other types of questions for which there are only a few answer categories available to the respondent. Closed-ended questions provide easily quantifiable data, for example, the frequency and percentage of respondents who answer a question in a particular way.
Alternatively, open-ended survey questions provide narrative responses that constitute a form of qualitative data. They require respondent reflection on their experience or attitudes. Open-ended questions often begin with: “why,” “how,” “what,” “describe,” “tell me about…,” or “what do you think about…”(See Open Ended Questions.) Open-ended survey questions depend heavily upon the interest, enthusiasm, and literacy level of respondents, and require extensive analysis precisely because they are not comprised of a small number of response categories.
Administering Surveys and Analyzing Results
Surveys can be administered in a variety of ways; in-person, on the phone, via mail, via the web, etc. Regardless of specific venue, it’s important to consider from the point of view of the respondent the factors that will maximize respondents’ participation, including accessibility of the survey, convenience of format, logicality of organization, and clarity of both the survey’s purpose and its questions.
Once survey data are collected and compiled, analyses of the data may take a variety of forms. Analysis of survey data essentially entails looking at quantitative data to find relationships, patterns, and trends. “Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc…That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. (See Community Tool Box “Collecting and Analyzing Data”) While data analysis usually entails some kind of statistical/quantitative manipulation of numerical information, it may also entail the analysis of qualitative data, i.e., data that is usually composed of words and not immediately quantifiable (e.g., data from in-depth interviews, observations, written documents, video, etc.)
The analysis of both quantitative and qualitative survey data (the latter typically collected in surveys from open-ended questions) is performed primarily to answer key evaluation research questions like, “Did the program make a difference for participants?” Effectively reporting findings from survey research not only entails accurate representation of quantitative findings, but interpretation of what both quantitative and qualitative data mean. This requires telling a coherent and evocative story, based on the survey data.
Brad Rose Consulting has over two decades of experience designing and conducting surveys whose findings compose an essential component of program evaluation activities. The resources below provide additional sources of information on the basics of survey research.