Surveys can be an efficient way to collect information from a substantial number of people (i.e., respondents) in order to answer evaluation research questions. Typically, surveys strive to collect information from a sample (portion) of a broader population. When the sample is selected via random selection of respondents from a specified sampling frame, findings can be confidently generalized to the entire population.
Surveys may be conducted by phone, in-person, on the web, or by mail. They may ask standardized questions so that each respondent replies to precisely the same inquiry. Like other forms of research, highly effective surveys depend upon the quality of the questions asked of respondents. The more specific and clear the questions, the more useful survey findings are likely to be. Good surveys present questions in a logical order, are simple and direct, ask about one idea at a time, and are brief.
Surveys can ask either closed-ended or open-ended questions. Closed-ended questions may include multiple choice, dichotomous, Lickert scales, rank order scales, and other types of questions for which there are only a few answer categories available to the respondent. Closed-ended questions provide easily quantifiable data, for example, the frequency and percentage of respondents who answer a question in a particular way.
Alternatively, open-ended survey questions provide narrative responses that constitute a form of qualitative data. They require respondent reflection on their experience or attitudes. Open-ended questions often begin with: “why,” “how,” “what,” “describe,” “tell me about…,” or “what do you think about…”(See Open Ended Questions.) Open-ended survey questions depend heavily upon the interest, enthusiasm, and literacy level of respondents, and require extensive analysis precisely because they are not comprised of a small number of response categories.
Administering Surveys and Analyzing Results
Surveys can be administered in a variety of ways; in-person, on the phone, via mail, via the web, etc. Regardless of specific venue, it’s important to consider from the point of view of the respondent the factors that will maximize respondents’ participation, including accessibility of the survey, convenience of format, logicality of organization, and clarity of both the survey’s purpose and its questions.
Once survey data are collected and compiled, analyses of the data may take a variety of forms. Analysis of survey data essentially entails looking at quantitative data to find relationships, patterns, and trends. “Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc…That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting. It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. (See Community Tool Box “Collecting and Analyzing Data”) While data analysis usually entails some kind of statistical/quantitative manipulation of numerical information, it may also entail the analysis of qualitative data, i.e., data that is usually composed of words and not immediately quantifiable (e.g., data from in-depth interviews, observations, written documents, video, etc.)
The analysis of both quantitative and qualitative survey data (the latter typically collected in surveys from open-ended questions) is performed primarily to answer key evaluation research questions like, “Did the program make a difference for participants?” Effectively reporting findings from survey research not only entails accurate representation of quantitative findings, but interpretation of what both quantitative and qualitative data mean. This requires telling a coherent and evocative story, based on the survey data.
Brad Rose Consulting has over two decades of experience designing and conducting surveys whose findings compose an essential component of program evaluation activities. The resources below provide additional sources of information on the basics of survey research.
Resources
The American Statistical Association, “What is a Survey?”
“Survey Questions and Answer Types”
“Writing Good Survey Questions”
“How to Ask Open-ended Questions”
“Guide to Survey Research,” University of Colorado
