Opinion Polls: Caveat Emptor

Do you trust opinion polls?

  • Yes

    Votes: 1 9.1%
  • No

    Votes: 3 27.3%
  • Sometimes

    Votes: 7 63.6%

  • Total voters
    11
  • Poll closed .

Statsman

PI Member
Joined
Oct 30, 2015
Threads
18
Messages
436
Likes
430
#1
This is an OP I posted elsewhere some time ago, but I think it's worth repeating.

Every time there’s a new poll published, political forums go into an entirely predictable pattern of posting:

Great, accurate poll. My party is doing well.
Crap, unreliable poll. My party is not doing well.

To a certain extent, this is understandable, but could be avoided by standing back and asking a few simple questions.
  • What questions are asked?
  • When?
  • What is the sample size?
  • How was the sample selected (random, quota, self-selecting)?
  • What was the mode (face-to-face, phone, e-mail, other)?
  • What is the margin of error?
  • Who did it?
  • Who paid for it?

If you want to know if a pot of soup needs more salt, you just need to stir it well and then taste a spoonful, not eat the whole pot. If you want to gauge the opinion of an entire population, you don’t have to ask everyone, just a representative sample. Quota samples will reflect the composition of the larger pot by pre-selecting quotas of groups to ask (gender, age, social class, geography, etc.) to fit the census breakdown along those lines. So, if 30% of the population are working-class women over 40, then the sample should reflect this. Random samples will just ask 1,000 (or whatever the sample size is going to be) people, and then adjust to reflect. So, if they find that only 200 of the sample are working-class women over 40 but it should be 220, then they weigh the responses of that 200 so that each of them counts as 1.1 person. Small representative samples will always be better than big, self-selecting ones.

Then they work out the margin of error (MoE), which is always an approximation as it is based on an ideal situation. For example, statistical theory confirms that for a sample of 1,000 with a 100% response rate on a question where the population is split 50/50, the poll will have an MoE of +/- 3% 19 times out of 20. Clearly, this never happens, but the theory acts as a guideline for an MoE that isn’t too far off.

Despite what some people think, genuine, serious polling companies have no interest in being wrong, so polls that are conducted by one of them are more likely to be right than polls conducted by randomwebsire.ru. Equally, most serious media organisations want to be as accurate as possible, so, like them or not, a poll commissioned by RTE is more likely to be accurate than one commissioned by a company that wants to sell a product.

So, why do polls on, say, voter intention in Ireland, that are published within a week of each other often show very different trends? We’re back to these questions:
  • What questions are asked?
  • When?

Clearly, the question asked will have a strong influence. Just think about the difference between being asked ‘Which party is currently doing the best job?’ and ‘Which party would you vote for if there were a GE tomorrow?

Equally, time plays a role. Would you expect the same response rate the day before a big scandal as you’d get the day after?

So, most polls are accurate enough to within the MoE for the questions that were asked at the time they were asked. If you don’t like the result, then the problem is probably more with you than with the poll. And always remember, polls are not predictors of election results, they are samples of how salty the soup is at the moment you test it.

If you’d like more detail, this link is very useful:

A Journalist
 
OP
OP
Statsman

Statsman

PI Member
Joined
Oct 30, 2015
Threads
18
Messages
436
Likes
430
#6
By the bye, I understand Ipsos MRBI have been doing a landline phone poll this week.
 
Joined
May 6, 2018
Threads
2
Messages
213
Likes
158
#7
By the bye, I understand Ipsos MRBI have been doing a landline phone poll this week.
And therefore excluding anyone under 50.

Seriously, we have not used the thing for years, in fact, when I changed broadband provider and the obligatory phone line was mentioned, we did not bother plugging it in to the router. Purely because we never use it and it adds clutter.
 
Joined
May 6, 2018
Threads
2
Messages
213
Likes
158
#8
Re opinion polls?
Brexit, trump and the last two referenda , esp the 8th repeal have shown how wrong they can be.

And, depending on your perspective, how dangerous.

I would also question polls which are done during the day, door to door and so exclude all regular workers .
 
OP
OP
Statsman

Statsman

PI Member
Joined
Oct 30, 2015
Threads
18
Messages
436
Likes
430
#9
And therefore excluding anyone under 50.

Seriously, we have not used the thing for years, in fact, when I changed broadband provider and the obligatory phone line was mentioned, we did not bother plugging it in to the router. Purely because we never use it and it adds clutter.
They'll get enough under 50s, they have to to mix the soup properly, but it excludes a certain kind of under 50, for sure.
 
OP
OP
Statsman

Statsman

PI Member
Joined
Oct 30, 2015
Threads
18
Messages
436
Likes
430
#10
Re opinion polls?
Brexit, trump and the last two referenda , esp the 8th repeal have shown how wrong they can be.

And, depending on your perspective, how dangerous.

I would also question polls which are done during the day, door to door and so exclude all regular workers .
Brexit and the Trump popular vote were both within the margin of error for the final polls.

Referendum polls exclude the 'home to vote' section of the electorate, and the fact that younger people voted in larger numbers than you would expect based on the % of the total pool they represent.
 
Joined
May 6, 2018
Threads
2
Messages
213
Likes
158
#11
Brexit and the Trump popular vote were both within the margin of error for the final polls.

Referendum polls exclude the 'home to vote' section of the electorate, and the fact that younger people voted in larger numbers than you would expect based on the % of the total pool they represent.
How big was "home to vote" in reality? Also all the polls show a "shy voter" risk (Bradley effect) and in the case of the two referenda, they need to understand milennials better.
 
Top Bottom