Do surveys have any value?

6 posts / 0 new
Last post
Do surveys have any value?

I just had a woman from a survey company in my house asking me questions about local government.

It seemed to me there was so much wrong with what she was doing, that I wonder if I can ever take the findings of such surveys seriously again.

Here are just some of the things that would skew the results:

1. she hurried me for my answers, forcing me to tick boxes before I had had a chance to read all the options or seriously consider what I really felt

2. she asked many questions where there were really two possible answers: what I really felt and what I wanted the local council sponsoring the survey to hear. Sometimes I went with the former, sometimes with the latter.

3. in a few cases she forced me to choose from options that did not include anything that accurately reflected my view

4. she forced me to choose five topics from a list of 20, when I felt like choosing 8 and couldn't distinguish between their relative importance

5. in some cases I found myself giving answers that sounded like the right thing to say rather than answers that reflected my true feelings

6. in some cases I didn't really understand what was being asked of me.

If you multiply this blurring of the data many times over and then add in human error and statistical error and respondents who choose to be perverse, and respondents who are just having a bad day, surely what you get at the end of it all is a pile of meaningless garbage?

Firstly it should be clear that a survey is no basis for democracy; it is the bureaucratic ear turned rather deafly and very briefly towards the people. Secondly surveys must be very carefully written in order not to just give the results the surveying organisation wants. It maybe that your survey was written by someone with no idea how to conduct a survey for accuracy or representation of views. A decent degree in any of the following: political studies; psychology; sociology would be a useful start for someone involved in this work, but failing that a little intelligence, self-awareness and common sense. Thirdly, like the famous name Mass Observation suggests, surveys will usually present an apparently clear picture which will under-represent shades of opinion and nuanced ideas. The survey will suggest the existence of an everyman, in other words the results, presented with sufficient authority, will be reified. If someone tries to bring some startlingly different view toa council's attention through a survey, their response will most likely end up being considered as statistical freaks, a footnote at best.
I take part in quarterly surveys for my local authority - a few years ago I volunteered to be part of the group doing this. After volunteering I got a job with the same authority (I'm still allowed to do the surveys - I'm still a resident and a council tax payer). Only then did I realise how skewed the surveys are. I'm a Homelessness and Social Services Advice Worker, and access to services is a big issue at the moment. For financial reasons, the authority is moving towards call centres and their walk in equivalents. So, for example, the survey offers two alternatives: would you rather have a 'one stop shop' where you could get information on all council services, or different offices all over the city centre? Obviously, anyone is going to answer 'one stop shop'. Our departments aren't located in one centre, however. So the alternatives should be: would you rather have a separate one stop shop with general advisers who can print you forms off the computer but answer no specific questions, or clear directions to another building where someone who knows what they are talking about can assist you with the relevant form and answer your questions? I'm still doing the surveys, writing indignant comments all over them and sending letters complaining about the questions every time. But, as Kropotkin says, I expect my responses are regarded as totally unrepresentative, and when my time is up, I don't really see a lot of point in asking to remain on the panel.
My company does quite a lot of local consultation work, with various different groups of people. The stuff we do is generally specific combinations of focus groups and targeted questionnaires rather than mass surveys but the principles are similar. In answer to the question, surveys in themselves aren't usually a very good way of you - the person surveyed - communicating your opinion to people in power (or whoever's doing the survey). For all the reasons you mention and more. Surveys are a way for an organisation to get information out of you. In the case of the stuff we do, we (or the people who employ us) genuinely do want to know people's honest responses to the questions we're asking them. But that's a very different thing to giving people a vote on what happens. The mistake is when - particularly government/local government agencies - conflate consulting people in a bid to improve services with giving people democratic control over decision-making. They're completely different things - and they're both important - but it often suits politicians to suggest that they're the same.

 

Buk, I am sure I'm not the only one to have big reservations about focus groups. There seem to be so many ways in which they can get things wrong. People not wanting to say anything different or controversial even if it's what they really feel; people wanting to sound cool rather than be honest; people being influenced by what those around them are saying; people being naturally conservative and resistant to new ideas; people reacting to topics from a theoretical perspective rather than from within a real-life context; people being asked to give their views while placed in an unfamiliar and possibly disorienting environment. Maybe there are clever ways of filtering some of this out, but I personally wouldn't trust the findings of a focus group.
I have big reservations about focus groups, too. It depends what you want them to do. They're not a good way of getting quantitative data (finding what a percentage of people think about a particular thing). They are good way of getting and anecdotal understanding of people's experiences and how people respond to different ideas and products or services. Aside from: "people being asked to give their views while placed in an unfamiliar and possibly disorienting environment" the other points you raise are all factors in any instance where you're asking people for their opinion. They're not all problems. "people being influenced by what those around them are saying;" and "people being naturally conservative and resistant to new ideas;" are both factors to consider when you're offering people a product, service or political platform so if a focus group helps you understand the ways that people are influenced by those around them (and which people influence them) then that's good to know. Similarly with natural conservatism and resistance to new ideas. It's good to know why people are resistant to ideas and try and work out how you can tackle that. Whether or not I'd trust a focus group, depends on what I was being asked to trust it to tell me.

 

Topic locked