Can I Poll Now? Should I Poll Now?

, , ,

Everything is up in the air! No one knows what’s going to happen next! At Quirk Global Strategies, we’re used to unpredictability.

We’ve developed strategies to manage it. No matter where in the world we’re working, every project starts with a discussion of the research goals. What questions need to be answered? By whom? How will the research be used? When? Answering these basic questions lays the foundation for the project and makes it easier to answer more complicated design questions down the road. This process helps us adjust our research plans to changing circumstances.

A meter apart, ladies

All survey research is a snapshot in time. Using it to predict views in the future is always a mistake. In the Coronavirus era, when the long term social, health and economic impacts have yet to fully hit, trying to use today’s data to predict what people will be thinking in months, or even weeks, is a waste of money.

Views on issues can be harder to shift than you’d think. As a political pollster in the US in September 2001, I assumed that the 9/11 attack was the kind of event that would radically shift perceptions on political topics. After the initial feelings of fear and insecurity wore off, pollsters found that voters’ priorities for elected officials or party preferences had changed little. This is the closest analogy in my political career to what we’re facing now. We can expect the economic and social impacts of COVID-19 will likely be much more far reaching than 9/11’s. But right now, we just don’t know.

Once you decide on the goals of your research, ask yourself these questions:

             Do You Need to Know Now?

If you need to understand how the current environment is shaping public opinion on a policy question today, or if you need to know how consumers have adapted their behavior to physical distancing, you should consider moving forward. If you’re planning to launch an advocacy campaign in six months or a year, you should probably wait, especially if you have limited resources to change your strategy or go in the field again.

            Do You Have the Resources to Respond to the Findings?

There’s nothing worse than binning a costly research project because the landscape has shifted, rendering your data useless. Can your plan be changed if the data reveal something unexpected or counterintuitive? If the answer is no, you should wait. Good research often reveals such findings, particularly in an uncertain environment. It’s wasted if you can’t incorporate it.

             Do You Have the Resources to Poll Again?

For many campaigns, knowing what people are thinking right now, at the beginning of the crisis, is critical information. A snapshot of attitudes in Spring 2020 will provide a baseline for tracking attitudinal shifts later in this year and in the years to come. It might also reflect the “worst case scenario” for your issue, which is useful to know when preparing a campaign. Depending on your timeline and your goal, you will probably need to poll again to update your assumptions. Add that to your budget.

            Is Some Data Better Than No Data?

In the difficult environments where QGS typically works, we don’t let the perfect be the enemy of the good. We always operate under budgetary, security and time restrictions that force us to make concessions. But since we fully understand the goals of each project, we know which methodological trade-offs we can live with and which we cannot.  We adjust our sampling plan to adapt to realities on the ground, report our methodology transparently and adjust our analysis and strategic recommendations accordingly. If you need actionable data, even if it’s not perfect, you have more flexibility in your data collection options.

            Who Do You Need to Talk to?

The answer to this question will help you decide which data collection mode is optimal, given your research goal.

None of this, sorry

If you need a large, proportionate, general population sample in a country where the only way to collect data is via face-to face-interviews, you’ll need to wait. Face-to-face interviews and traditional focus groups are simply off the table right now. Sadly, it’s impossible to predict when interviewers will be able to go out in the field and discussants can sit around a table talking to a live moderator.

The good news is telephone survey research to mobiles and landlines is thriving in Europe, the Gulf and North America. Interviewers stationed safely at home are calling voter file or RDD samples, like always. There are even anecdotal reports of marginal response rate improvements. If your universe has high mobile/landline penetration, phone surveys remain the best way to collect a random sample of a general population universe.

Data collectors in many middle- and low-income countries are on the cusp of being able to field random sample mobile surveys, particularly of urban populations. Before we commit to this approach, however, we need a full understanding of which groups are underrepresented (usually older, rural, lower SES) and which are overrepresented (younger, urban, higher SES) in these samples. We pay particular attention to the gender split: In some places it’s easier to interview women on the phone than in person. In others, men control access to the mobile phone. Then we decide, based on the goals of the research, if we can live with the trade-off.

            Non-Probability Options Abound

If you’re interested in non-random views of urban, younger, educated, higher SES, respondents with mobile phones or internet access, there is no shortage of methodologies available. This is particularly true in high- and middle-income countries but these populations are accessible even in many low-income countries via panels as well. Methodologies such as online surveys, SMS surveys, IVR and social media analytics can also be combined to give a richer, more contextual view of the landscape. Keep in mind, these modes sacrifice randomness and are not a substitute for a proportional sample.  Review the research goal then decide if it matters.

           Quirk Global Strategies Can Help

So should you poll now? Return to the goal of your research. Look at your budget and decide which trade-offs you can tolerate and the ones you cannot. Email us and we can help you think through your options and suggest the best one. We might even suggest waiting.

/by

Global Survey Research in the Time of Coronavirus

, , ,

Curious about how opinion polling can proceed when entire countries are closed down? I was, so I reached out to data collectors around the world to better understand what’s possible now and what kind of projects should wait.

Pollsters who use telephone interviews are in a good position to keep on working. Unfortunately, those who rely on interviewers who need to move through communities and interact with people in their homes are going to have a rougher time. In the short term, researchers will have to put those projects on hold or consider new methodologies.

Telephone Surveys Are Happening

From the early days of COVID-19, phone banks serving high-income countries of North America, Europe, and the Middle East started adapting. Obviously, big rooms filled with interviewers sitting in front of terminals are out of the question. In response, data collectors I’ve spoken to quickly set their interviewers up on secure systems that allow them to call from home. This is good news for pollsters and interviewers.

Given the unrelenting demand from election pollsters, this decision was a no brainer for US-based firms. Those with global calling capacity and multi-lingual interviewers can transfer also projects to offices in countries where workforces have been less affected by shutdowns. Response rates might also improve because people stuck at home could be more willing to respond to surveys. In the US at least, it would be hard for response rates to get worse.

For those who field in high-income countries, this is all good news. It’s important to keep a couple of things in mind, however. In the US, ordinary election year demands stretch collectors’ capacity up to and often beyond limits. Having limited remote interviewers will exacerbate the problem. Management capacity will also be stretched as quality control and training demands grow. Patience, oversight and regular communications with data collectors are critical.

If your project in high-income countries in North America, Europe or the Middle East needs to move forward, then experienced phone houses with strong management can get it done. If you can hold off, or wait for a lull in your collector’s schedule, the quality of your data will likely be higher. I analyze my data by day as it comes in with an eye for anomalies. It’s also smart to build in extra days for call-backs, or in case things go haywire and you need to replace interviews or do extra QC.

Face-To-Face Surveys Are Problematic

As a specialist in survey research in conflict and post-conflict environments and low/middle income countries, about 80% of my projects rely on face-to-face (F2F) interviews. Phone interviews are not an option in places where mobile penetration is low or disproportionately distributed.  In normal times, in-person interviewers face obstacles such as transport difficulties, bad weather and conflict-related threats. Those problems are manageable. But add a highly contagious disease spread by personal contact and F2F data collection suddenly becomes untenable. It could be this way for while.

Whether F2F data collection is viable depends on a country’s infection rate and whether restrictions on movement and social contact have been put in place. This fast changing situation makes planning almost impossible. For example, I have a survey ready to field in India. Up until a week ago data were being collected normally in many states and slowly, and with extra precautions, in others. On 25 March, the entire country was locked down to stop the spread of the virus. Thankfully, we had opted to wait until the situation clarified and had not begun fieldwork.

The situation in many African countries is also ambiguous. Some countries have few infections and life continues normally, for the time being. Other countries are suffering badly from outbreaks. Ukraine has a serious outbreak and F2F data collection is on hold. Quant work in the Philippines is also on a slowdown, if not entirely stopped.

Experienced, ethical data collectors are in the best position to offer advice on the local situation. The final judgment always belongs to the researcher however. No one — interviewer or respondent — should be put in danger for the sake of survey research.

Data collectors in some F2F countries have begun experimenting with phone surveys and online panels. The possibility for error and biased samples remain serious concerns. Having a complete understanding of how these samples under- or over-represent populations is critical; some populations will simply not be reachable by these modes. Survey work in difficult environments often requires methodological tradeoffs. If you can live with increased and unpredictable error from non-probability samples, or if you can afford to be experimental, I say give it a try.

I Can Field a Survey. Should I?

Of course, pollsters need to consider ethical questions before fielding surveys during a global pandemic. Should interviewers be sent into the field, handhelds ready, to conduct interviews in people’s homes? Absolutely not. Should worried or scared respondents be pestered with questions about topics of less importance than life, death and economic survival? I hear the same argument against conducting surveys in conflict zones. Often, the concern is overstated. Many people — especially in low- and middle- income countries — are not asked their opinions asked about anything. They’re usually happy to oblige. Additionally, respondents can be more forthcoming in times of insecurity and unpredictability. Data collected during this period will be a real “snapshot of a strange time” and will be fascinating to track over time.  As I do in conflict zones or closed spaces, I let respondents tell me if they are uncomfortable or unwilling to participate. I look at response rates, drop-offs and interviewer comments before I make judgements about respondent willingness.

The need to understand public opinion during and after this crisis is not going to go away. Economic, political and social disruption could shape perceptions in unpredictable ways in low/middle income countries and upper-income countries alike. Views that once seemed hardened and unlikely to shift may change radically, or not at all. Whether you’re working in a high income country of North America, Europe or MENA, or looking at surveys in lower/middle income countries, Quirk Global Strategies can help you sort through your options. Contact us through the link on this site.

/by

“The Polls Were Wrong” Is the Easiest Answer

, ,

Since Sunday’s election I have been in the improbable position of defending Turkish pollsters. To be clear, I have had many, many quibbles with their lack of methodological transparency and the perception of bias that causes.

Unlike other forms of public opinion or policy research, however, election polling provides a day of reckoning: you’re right or you’re wrong. It’s there for everyone to see. If you can’t get the election right, within the margin of error of your sample, no one should pay attention to your data. If you repeatedly get it right, you deserve a degree of credibility.

Here’s the thing: several Turkish pollsters were pretty damn close to getting the 7 June parliamentary election correct. And when I say “correct” I mean “correct within the appropriate margin of error in the days immediately before the election.” Therefore, their November election results which missed the mark should not be dismissed outright, especially since multiple pollsters reported similar results.

I’m going to digress a bit. It’s absolutely fundamental to understand the basic principles of probability sampling if you’re going to comment on pollsters’ performance.

A poll reflects voters’ views on the days the survey was conducted within a margin of error. Here’s what that means: if you draw a truly random sample of, let’s say, n=1000 people within a universe (Turkey, for example), and you write the questions and implement the survey in ways that diminishes biases inherent in survey research and your sample reflects the demographics of your universe, your data will, within a standard margin of error of plus or minus 3.1 percentage points (characterized, shorthand, as MoE +/- 3), reflect the views of that universe on that day. That means the results of any data point could vary three points higher or three points lower. This is critical to take into consideration when declaring a poll “right” or “wrong” relative to election results or stating a candidate/party “is ahead in the polls.”

Two important takeaways:

• The only way to achieve a margin of error of zero is to interview every single person in the universe (Turkey, for example). That’s impossible and is why we rely on probability sampling. The trade-off is we have to accept and accommodate the margin of error in our analysis. If we fail to do that, we’re wrong. Period.

• Pre-election polls are not designed to project what will happen on election day (you can do that through modeling, but it’s risky). This is why everyone (especially candidates who are about to lose) says the only poll that matters is the one on election day — it’s the only one that’s a 100% accurate report of voters’ views with no margin of error.

If you don’t believe all this, go take a statistics class and then we’ll argue about it. It’s science, not magic. Also, please do not give me an exegesis on academic research. Like these pollsters, I work in the real world with budgets and time constraints.

So, let’s look at the last three public polls taken before the 7 June election. I chose these three because 1) fieldwork was conducted the week or two before the election and 2) they shared their sample sizes so we know the margin of error. (There may be others, but I found these data here). We want to look at polls conducted as close as possible to the election because they’ll capture the effects of late-breaking campaign dynamics. (Also, not rounding is an affectation. I round. Personal opinion).

 

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

MAK

44

25

16

19

n=2155

+/- 2.1

18-26 May

SONAR

41

26

18

10

n=3000

+/-1.8

25 May

Gezici

39

29

17

12

n=4860

+/-1.4

23-24 May

Andy Ar

42

26

16

11

n=4166

+/- 1.5

21-24 May

June Results

41

25

17

13

n/a

0

7 June

I draw two conclusions.

First, putting aside ORC which overrepresented AKP and underrepresented HDP, Konda and Gezici were pretty damn close to the final result (by that I mean close to within the MoE), considering data was collected a week before election day.

Secondly, though it can be risky to compare data collected by different operations, their data are very similar, which suggests they are using similar methodology and making similar assumptions. That’s the way it should be.

Next, let’s look at publicly released data for the November election. I borrowed most of these data from the delightful James in Turkey and he did not always include the margin of error. I will take that up with him at a future date. Let’s assume pollsters without sample size indicated interviewed between n=3000 and n=5000 (that’s what they did in June), so the margin of error will be between +/-1 and +/-2

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

Andy R

44

27

14

13

n=2400

+/-2

24-29 Oct

Konda

42

28

14

14

n=2900

+/1.8

24-25 Oct

A&G

47

25

14

12

n=4536

+/1.4

24-25 Oct

Metropoll

43

26

14

13

n=

15 Oct

Gezici

43

26

15

12

n=

15 Oct

ORC

43

27

14

12

n=

15 Oct

AKAM

40

28

14

14

n=

15 Oct

Konsensus

43

29

13

12

n=

15 Oct

Unofficial Final

49

25

12

11

N/A

+/-0

1 November

AKP’s final number falls outside all the polls’ MoE, except A&G’s. The next closest, Andy R, conducted the latest fieldwork so was in the best position to capture emerging trends, such as a surge in AKP support. Andy R still underreported AKP support by five percentage points. That’s a lot. A&G didn’t release any tracking data so it’s hard to know if it’s an outlier or ahead of the others in capturing the AKP surge. The latter is possible and I will address it in a future post.

If consistent sampling methodologies and questions are used, it’s possible track data over time to see if it changes. Big unexplainable differences from one dataset to another could indicate a problem in the methodology. I like it when pollsters provide election tracking data. It suggests sound sampling and alerts us to important trends in public opinion.

For fun, let’s take a look at two of those who did:

KONDA

AKP

CHP

MHP

HDP

June 7 Results

41

25

17

13

Aug 8-9

44

26

15

13

5-6 Sept

42

25

16

12

3-4 Oct

41

29

15

12

17-18 Oct

42

28

15

13

24-25 Oct

42

28

14

14

Unofficial November Final

49

25

12

11

GEZICI

AKP

CHP

MHP

HDP

June 7 Results

41

25

17

13

3-4 Oct

41

28

17

14

17-18 Oct

41

27

16

13

24-25 Oct

43

26

15

12

Unofficial November Final

49

25

12

11

Not only are these two pollsters consistent over time, they are also consistent with the final June results and compare favorably with each other. Nothing in either of their datasets suggests a big shift in opinion toward AKP (they do indicate an AKP trend, which is plausible). Yet, inthe end, their last polls are wrong wrong wrong about the November result. That’s really troubling.

How could pollsters who nailed it in June have missed it in November? How can they be consistent over time and with each other and be wrong on election day? Falling back on “the polls are wrong” as analysis is simply inadequate. If you’re going to disregard months of consistent data, you should provide an explanation for how it went wrong.

I honestly can’t give an adequate explanation. Because I have other things to do and you have short attention spans when it comes to statistics, I will address what I think are the three most likely polling error culprits in future posts. These include (in random order of in likelihood):

• Errors in methodology (this will address the absurd argument that since UK and US pollsters were wrong, it follows that polls in Turkey are also wrong. I can’t believe this is even necessary)

• Errors in analysis (not reporting or considering Undecideds or softening support, which is my current theory of choice)

• Election dynamics that cannot be captured by polling

NOTES: If you want to look at a few other pollsters’ June data, here it is. I don’t think it’s totally fair to judge their accuracy based on data collected weeks before election day, but, with the exception of under-representing HDP, most of them (except MAK) actually are pretty close and provide more evidence of the consistency of public opinion. Being off on HDP can be forgiven because HDP had what campaign people refer to momentum and it is plausible HDP’s support increased in the final weeks.

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

MAK

44

25

16

19

n=2155

+/- 2.1

18-26 May

SONAR

41

26

18

10

n=3000

+/- 1.8

25 May

Gezici

39

29

17

12

n=4860

+/- 1.4

23–24 May

Andy Ar

42

26

16

11

n=4166

+/- 1.5

21-24 May

June Results

41

25

17

13

N/A

0

7 June
/by

Want to Increase the Credibility of Election Polling in Turkey?

,

Remember that time you asked me how to increase the credibility of public polling in Turkey? No? Well, it turns out I have thoughts on the matter. Here they are.

Transparency, Transparency, Transparency:  This is the single most important factor. Given the amount of flawed data out there, every pollster who releases election polls publicly should voluntarily provide the following information for the sake of increasing public trust in the science. Reporters should ask for it. Not all of it needs to be reported by the media — it typically isn’t — but it provides important information, especially to professionals and academics, about how data were collected and processed. Allowing outsiders to review and discuss the methodology increases the rigor of the research. Ultimately, the result is greater public confidence in polling data.

Here’s the type of information that would be helpful. (The first four bullets should be reported in every media story that references a poll, without exception)

  • Sample size, sample type and universe: Here’s an example. “A national (or urban, or regional sample of 25 provinces) sample of n=2000 adults in Turkey over age 18.” If the pollster diminished the sample to include only likely voters, he or she should explain how that determination was made.
  • Fieldwork Dates: Knowing when the data was collected provides important context about events that occurred in the political environment and could affect perceptions of the candidates (i.e., a deadly mine disaster, or a huge corruption scandal). Fielding dates also tell you if there was enough time to return to selected respondents who weren’t available on the first try. With a large sample, a day or two in the field isn’t enough for callbacks. Therefore, the data are biased toward those who answer their phones or their doors on the first try.
  • Margin of error for the sample as a whole: “The margin of error for the n=2000 sample is 2.19% at the 95% level of confidence. The margin of error for demographic and geographic subgroups varies and is higher.”
  • Who’s the Funder: This is critical information. Who’s paying for a survey may impact the credibility of the data. It may not. But you have no way of judging if you don’t know who’s coughing up the dough. In the US, few pollsters would jeopardize their reputation for accuracy and reliability by swinging data in favor of a well-funded or powerful interest (some would and have, but it’s an exception, not a rule) but revealing who’s paying for the research is standard. Even if Turkish pollsters don’t monkey with the numbers (and lots don’t) the perception that pre-election polling is cooked is well-founded, pernicious and must be addressed if opinion research is going to be used as a credible tool for illuminating public policy debates and elections.
  • How the interviews were conducted and how respondents were selected: Were interviews conducted using face-to-face interviews? If so, how were respondents selected? Were the interviews conducted by telephone? What proportion of landlines versus mobile numbers was used? How many efforts were made to call selected numbers back if there was no answer? What times of day were interviews conducted?  If the answer is “online poll,” step away from the story.
  • Response rates: What percentage of selected respondents participated in the survey? This varies depending on the country and sometimes, the type of survey. The pollster should reveal what standard response rates are in Turkey for similar surveys. An abnormally high or low response rate should raise red flags.
  • Question wording and order: How a question is asked and where it appears in a survey directly affect responses. Respondents should not be “primed” to answer a particular way. Therefore, a vote preference question should be one of the first respondents are asked. The list of candidates should be presented exactly as names appear on the ballot, with no extraneous information provided that voters won’t see when they enter the polling station. The percentage of respondents who answered “don’t know” or “undecided” (a critical data point in election polling) should also be reported and if the “don’t know” response was prompted or unprompted.
  • Quality Control: How many interviews were verified in the field by supervisors or called back to make sure they really took the survey?  I know it’s hard to believe but sometimes interviewers are lazy and fake interviews! Quality control is expensive, technical and time consuming and is why methodologically sound polling is expensive. Rigorous quality control by outsiders reduces the chances that data are falsified, especially in the processing phase where someone *might* be tempted to place a finger on the scale. Opening data sets to outside scrutiny is a good way to expose and prevent this.

 

  • Sampling and weighting procedures: It’s easy to baffle non-specialists with statistics but polling isn’t rocket science and random sampling procedures are guided by industry standards. Pollsters should reveal if their samples are stratified and by what variables. They should share how sampling points were selected. They should also reveal if the final data were weighted and by what factors.

 

Wow! This sounds like a lot of work! But one of the most interesting outcomes of the 2012 election in the US, in which a high profile, well-respected research outfit (Gallup, in about as epic a scandal as pollsters are allowed to have) got the election wrong, was the degree of public scrutiny Gallup allowed of its methodology to figure out what happened. I’m sure it was painful — no one likes to admit they did things wrong — but the result is better, and more credible, public research. Gallup’s reputation took a hard hit, but they dealt with it the best way they could. If you really want to learn more about what happened to Gallup — and why wouldn’t you? Pollsters are awesome — read this report.

 

Given that major Turkish pollsters, including a well-respected one, got the Presidential election wrong, this issue isn’t going away soon. Historic low turnout figures –preliminarily 74% — might have thrown some pollsters for a loop but it shouldn’t have, given the timing and the dynamics of the election. The challenge always, as US pollsters have found, is trying to predict which voters will cast ballots and which will stay home. Turkish pollsters, who already face credibility issues, need to confront this issue with transparency.

**Quirk Global Strategies isn’t in the business of public polling. We’re strategic pollsters, which means private clients use our data to guide their internal political or communications strategies (though not in Turkey, usually). This is an important distinction.

/by

How to Be a Good Consumer of Public Polls

, ,

Here we are again, weeks out from another Turkish election, arguing on Twitter and in bars about the pre-election polls on the Presidential race between the shouty guy and the bread guy. As much as we’d like to, we can’t really ignore this election, so wouldn’t it be great if someone explained how to tell if a poll in the paper is credible or not?

It’s your lucky day! Here are some basic, but important, concepts to understand before you write about, argue about, print, or tweet publicly released election polls (everywhere, too, not just Turkey).

 

  • How many people were interviewed? It amazes me how few press articles include this mandatory information. A nationally representative sample should include at least 800 randomly selected respondents, which has a margin of error (MOE) of 3.5% at the 95% level of confidence.* A larger sample size does not necessarily mean the survey is better (academics may argue otherwise, but their research goals are different), so don’t fall into that trap. For example, the margin of error for a n=2000 sample is 2.2% (compared to 3.5% for n=800). That’s not a big difference and won’t matter that much except in the closest elections. However, if the pollster is sharing data from smaller demographic or geographic subgroups within the national sample (men, women, Kurds or Istanbullus, for example), a larger sample size becomes more important. Remember, the MOE increases as the number of interviews decrease. If Istanbul makes up 19% of the country (and in a nationally representative sample, it will) in an n=800 sample, there will be only 152 interviews among Istanbullus, with a MOE of 8%. If the sample is n=2000, there will be 380 interviews (MOE 5%) among Istanbullus. I’m slightly more comfortable with the latter data than the former because the margin of error is smaller. Do you like to play around with sample sizes? I do! There’s an app for that.
  • Who paid for it?  This is Turkey so this is probably the single most important question. In the US, major media outlets (and think tanks) commission credible research firms to conduct election surveys (the “CNN/Washington Post poll,” for example), the results of which papers report as news. Given they are in the business of reporting things that are more or less true, they have a lot at stake by getting the numbers right. The media in Turkey operate according to different principles. That a media outlet reports data tells us little more than in whose favor the numbers are likely to have been cooked. Methodologically sound research is expensive in Turkey — $20,000 to $30,000 for data collection alone — and for-profit research firms are unlikely to undertake survey work for fun, even if they say they do. Someone’s paying for it and if you can’t find out who, don’t report it.

 

  • Who was interviewed? Election polls are designed to predict election outcomes. It sounds harsh, but non-voters’ opinions don’t matter. Therefore, only likely voters should be polled. Because voting is compulsory in Turkey, election participation is very high (88%-90%) so nearly all adults are eligible to participate in an election survey. In contrast, election polling in the US is extremely complicated: only about 50% of Americans are eligible to vote (by virtue of having registered), and among those, participation rates vary from the extremely low (15% in low-interest primaries) to the less low (about 65% in presidential elections). Predicting who should be included in a sample of likely voters is extremely challenging. Misreading the composition of the electorate was one of the reasons major polling firms got the US election in 2012 wrong. Because of its timing (10 August, mid-vacation), uniqueness (it’s the first time Turkish voters have directly elected a president) and low interest in the candidates among the tatıl-class, Turkey’s presidential election presents a unique challenge to election pollsters. Is there going to be substantial drop-off in participation among certain types of voters who won’t bother to return to Istanbul from Bodrum’s beaches to vote? Maybe! Pollsters who care about accuracy will take this into account. They should explain how they’re addressing this issue, and how, if at all, they’re diminishing their samples to exclude those who won’t vote. Ask! Ask! Ask!

 

  • How did they conduct the interviews? Generally, in probability samples (the only kind that produces representative data and the only kind I will discuss), a respondent is selected at random to participate in either a face-to-face (F2F) or telephone interview. F2F has always been the norm in Turkey because of low phone penetration but that’s changing quickly as more and more people obtain mobile phones. Mobile sampling is becoming more and more common. Both methodologies have biases and you should know which methodology the pollster uses so you can be aware of them. I can go on for days about the pros and cons of each (it’s a wonder I have any friends at all). Online, web-only surveys are bogus. If you ever want to start a flame war with me on Twitter, report on an online survey like this one without using the word “worthless.”
  • What’s the polling firm’s track record? Accuracy is a pollster’s currency. The great thing about election polling is there’s a day of reckoning. You either get it right and can be smug (it’s science!) or you’re wrong and no one should listen to you anymore. Given the dearth of credible election polls in Turkey, calling previous election results correctly boosts a pollster’s credibility even more in my book. As far as I know, and I don’t know everything, one firm did that publicly in the March local elections: Konda. Why data released by firms that got a recent election completely wrong are treated as credible is a mystery to me. It’s easy to check.

 

This isn’t all there is, but it’s plenty and you don’t have to be a specialist to interpret it (as long as you understand probability sampling). Having the answers to these questions will make it easier to assess the quality of the polls you see in the Turkish press and on Twitter. Armed with this information, you’ll have the tools to be able to say “this poll sounds like BS. I’m not going to report/tweet it,” thus depriving bogus pollsters of the media oxygen they need to survive. If you can’t get answers to these questions, don’t report the data.

 

TOMORROW (or some day in the near future)! How to Make Public Election Polling in Turkey More Credible 

 

*If your universe (total number of potential respondents) is greater than a couple hundred people, the margin of error is the same for a random sample of n=800, regardless if you’re surveying a city with a population of 1500 people or a country of 78 million. If you don’t understand why this is, or what a margin of error is, get thee to a Stats 101 course and don’t start arguments you’re going to lose.

**Quirk Global Strategies isn’t in the business of public polling (or academic research). We’re strategic pollsters, which means private clients use our data to guide their internal political or communications strategies (though not in Turkey). This is an important distinction. Strategic pollsters who collect bogus numbers give bad advice, lose elections and don’t get hired again. Therefore, we strongly oppose BS numbers. You can be certain that strategic polling is being done in Turkey — most likely on behalf of AKP — but you and the twitter loudmouths you follow are unlikely to get your hands on it.

 

/by

Legit Pre-Election Poll in Georgia Shows Saakashvili Up

, , ,

There’s been a spate of pre-election polls from Georgia lately, most of them not worth the paper they’re written on. All have been released by various interests to demonstrate their electoral strength and have little to no basis in reality. The only consequence of their release has been to increase public skepticism of polls.

President Mikail Saakashvili’s party (UPM) released poll a few weeks ago that was greeted with howls of derision, but for a lot of wrong reasons. Most people I talked to categorically refused to believe data publicly released by BCG National Research, the firm that polls for the president’s party (a high ranking official in the government I met insisted that’s different than the serving as president’s pollster, which suggested a bit of disingenuousness on his part, but we’ll not quibble here. Also, BCG is run by the wife of the head of the Central Election Commission, a particularly stinky connection in this part of the world). The poll showed the president with a comfortable lead. Big surprise.
Read more

/by

Shining Light in Dark Places: Increasing Pollster Disclosure

Two weeks ago, I posted a rather glib post about fake public polls in Ukraine, arguing that from a strategic point of view or as evidence of stolen elections, they’re meaningless. However, in the long run, the spate of fake pre-election polls make it that much harder to get information on what Ukrainians really think about issues facing the country. That’s a problem.

Read more

/by