In Part 1 yesterday, Tom Miller provided an overview of how citizen surveys can be of value to planners and planning commissioners. In today’s installment — and the next two — Miller covers his 9 “rules” for completing a citizen survey.

1. Identify Why a Survey is Needed and What It is Intended to Do?

One of the first things a community must do before it conducts a survey is to agree on the answer to the question: What do we want to learn? By developing a statement that clearly explains why the survey is being undertaken, the community will have a much easier time planning the survey, analyzing responses, and disseminating the results.

Part of being able to answer “What do we want to learn?” is having an idea of how the results of the survey will be used.

Generally speaking, citizens will respond energetically to surveys that will be used to guide the development of projects, evaluate programs, and prioritize expenditures. When the results of a survey indicate overwhelming public support for a particular course of action, the sponsoring agency or governmental body should be prepared to take steps to implement that course of action.

2. Determine How Much Your Community Can Spend

If money ran from the tap like water, there would be no need to select a sample of residents to participate in your survey. Instead, you would conduct a census, tracking down the opinion of each and every qualified resident.

Surveys always represent a compromise between precision and possibility. Budgets generally exclude the value of staff or volunteer time but, if you can, it is wise to know the complete cost of doing a survey. These days a scientific survey tends to cost between $8,000 and $15,000 for the basics.

Hiring a Consultant

The decision to conduct a survey should not be taken lightly. Questions to consider when making this decision include the following:

Who will oversee the development and administration of the survey?
Do the people who might work on the survey have the right expertise?
Do they have enough time available?

And, finally, have funds been budgeted to obtain outside expertise where needed?

The “ownership” of the survey is also important. The more those who may be affected by the survey results feel “connected” to the survey during its design, the more likely they will accept recommendations based on the survey’s results.

Not every community will want to conduct a survey on its own, or feel capable of doing so. These communities can get help from private consultants, universities, and/or organizations such as regional planning agencies. Most communities use consultants to conduct their surveys.

Once the decision to hire a consultant has been made, the next issue is “which consultant”? Your consultant should be:

1. Someone who understands you — and whom you understand.
2. Someone who can work with diverse groups and who can explain the benefits and limitations of various survey research methods.
3. A good writer and a clear speaker.
4. Someone who can explain the 95 percent confidence interval.
5. Someone who will challenge the useful-ness of questions.
6. Someone who can tell you how to check and control for non-response bias.
7. Someone who can accomplish statistical re-weighting of the sample.
8. Someone who won’t insist on highlighting all statistically significant differences if they don’t matter.
9. Someone who can get the right descriptive statistics out of a computer.
10. Someone who knows the difference between a pretty graph and a clear graph.
11. Someone who is willing to document meticulously all survey research methods.
12. Someone who is willing to take pieces of the project, if you are planning to handle some of it in-house

3. Put a Team in Place to Analyze the Results

It may seem too early in the process to be worrying about what to do when the survey is complete, especially when you haven’t even constructed the data collection instrument yet. Nevertheless, the weak link in survey research is almost invariably the use to which results are put.

Identify a panel of staff and citizens who are charged with making recommendations to the planning director, city manager, or city council about the meaning and use of the results. Let the panel members know that they will be expected to determine if the results merit nothing more than “watchful waiting” or if action is required. While the panel will be advisory in nature, members will need to be prepared to justify their recommendations by reference to the survey results and, perhaps, other sources.

Meaning Comes from Comparison

We don’t know what is tall or what is small without comparing. As you determine residents’ perceptions of your community’s quality of life and quality of services, you need to know, “how good is good enough”?

Evidence is clear that residents tend to give favorable ratings of community service delivery, in fact ratings more favorable than most local government officials anticipate. ¹

Given how easy it is to find ratings of 60 or higher on a scale where 0 = very bad and 100 = very good, it is essential to understand what typical ratings are for various aspects of community life.

The approach our firm takes to address this issue is by providing “peer city” comparisons — comparing a community’s survey results to those from other similarly sized or located places. This form of comparison may show, for example, that while your city’s street cleaning services rating of 70 (on the 100 point scale) was in the “good” range, that rating was, nevertheless, among the lowest given in comparably-sized cities (where ratings between 75 and 90 for street cleaning services were the norm).

This kind of comparison is also fairer than comparing ratings within your city of various departmental services. Results from peer cities provide the kind of interpretive richness that comes from no other source and will help you and staff to understand if you ought to celebrate or regroup.

In Part 3 tomorrow, Tom Miller continues his review of 9 “rules” for completing a citizen survey, with:

4. Identify the target population and sample.
5. Determine how many people should be surveyed, and how to reach them.
6. Ask the right questions in the right way.

Thomas I. Miller, Ph.D., is founder and President of National Research Center, Inc., a survey research firm located in Boulder, Colorado. An expert in research and evaluation methods, Miller is the co-author of Citizen Surveys: A comprehensive guide to making them matter, published by the International City/County Management Association in 2009.

His firm, which specializes in surveys that permit communities to compare their results with “peer” communities, maintains an integrated database of over 500 surveys completed by about a one half million residents in 44 states.

Miller would be pleased to respond to readers’ questions about the article through our PlannersWeb Linkedin group page; he can also be reached at: 303-444-7863.