Worried About Qualitative Research During Coronavirus? I am

, , , , ,

With traditional focus groups off the table during physical distancing, qualitative researchers face options that may require too many tradeoffs to make the effort worth the costs. The challenges for qualitative research could be even greater than those facing quant, particularly for social or opinion research projects.

If you’re interested in challenges facing quantitative researchers, check out my last two posts.

What Are My Options?

If your qualitative needs can be met with chatroom-style online focus groups, you’re in good shape. It continues to be a solid technique for ad and message testing with larger audiences and for mixing qual and quant methodologies. Recruiters report that homebound respondents have the time and inclination to participate in a variety of formats. This is good news.

No FGDs for now, whether in a room or under a tree

Focus group discussions present a greater challenge however. The most valuable part of a focus group is the personal interaction between participants that an engaged moderator can elicit through thoughtful probing and skillful management of group dynamics. If the success of your project depends on getting beyond top-of mind responses, generating new ideas, and exploring feelings about abstract concepts, remote in-person focus groups on Zoom or other proprietary platform fall short. I worry that so much illuminating data will be distorted or lost that it’s worth asking “what’s the point?”

Do You Enjoy Zoom Meetings? I Don’t

Think of your last Zoom(s) with eight friends or colleagues. Where you satisfied with the quality of interaction with others on the call? Did you enjoy it? How engaged were you with the topic? After more than two months of physical distancing, Zoom interactions are proving to be, at best, unsatisfying and at worst, exhausting. There are good reasons for this.

Axios technology editor Scott Rosenberg articulated my misgivings about video-conferencing as a qualitative tool.  Here’s what he writes:

Videoconferencing imposes cognitive and psychological frictions and aggravates social anxieties. As experts in human-computer interaction point out, using Zoom means putting on a show for others without being able to rely on the cues we primates depend on in physical encounters.

  • There’s usually a slight audio lag, as well as mute-button mistakes and “your internet connection is unstable”-style dropouts.
  • We’re also often opening a chunk of our homes for others to view, and that can trigger social worries.
  • By showing us our own image as well as others’, Zoom ensures that we will critique ourselves in real time.
  • On top of standard-grade performance anxiety, the “big face” image that Zoom uses by default in its “speaker view” can trigger a “fight-or-flight” surge of adrenaline, writes Jeremy Bailenson, founding director of Stanford’s Human Computer Interaction Lab.
  • If you switch to the “Hollywood Squares”-style “gallery view,” you’re confronted with a sea of separated faces, which is not how evolution has adapted us to group interactions.
  • As M. Sacacas observes,you can’t really achieve true eye contact with anyone: If you look right into someone else’s eyes, you will appear to them as if you aren’t looking right at them — to achieve that, you have to look right at the camera.
  • Nonetheless, the whole experience of a videoconference feels to us like an extended bout of direct staring at other people staring back at us. That’s draining, which is why it’s not what actually happens when we meet in person, where we only occasionally look right at one another.

How will participants react when deprived the visual and social cues that help them interpret the reactions of others? Will feeling uncertain, frustrated, insecure, or tired influence their opinions, particularly on sensitive or political topics? How will these dynamics shape strategy built on the data? Additionally, Rosenberg’s analysis is helpful for contextualizing the reactions of citizens of high-income countries who are comfortable interacting with technology. How will citizens of low- or medium-income countries, or those with different cultural expectations for social interaction, respond? I don’t know if we know the answers to these questions.

Traditional focus groups have many biases, all of which are well-known and factored into qualitative analysis. We know people respond to questions differently in a group setting than they would alone. Online video-conferencing multiplies these biases plus adds others that we’ve barely begun to understand. Not only are there different biases to adapt to, the medium itself diminishes the greatest advantage of a focus group: group/moderator interaction.

 So What Can We Do?

Because qualitative research can’t grind to a halt because of physical distancing, we have to come up with ways to mitigate these problems. Here are some suggestions.

  • Assess whether videoconferencing is the right tool for the job. Does the topic demand strong rapport among moderator and participants to explore controversial social topics ? If so, your findings may lack depth, mislead or entirely miss important findings.
  • Write a shorter, simpler moderator’s guide. This medium is not appropriate for a two-hour long discussion on constitutional reforms.
  • Recruit fewer participants. Aim for quality of responses, rather than quantity.
  • Use a moderator well-trained and experienced in managing the complicated dynamics of this type of discussion. A high energy, animated moderator might be better able to engage remote participants than one with a more low-key personality.
  • Manage client expectations. Video-conference groups and in-person groups are not interchangeable. They will provide different kinds of data. Clients who expect traditional focus group data are likely to be disappointed,
  • To avoid the impulse to compare data collected in a traditional focus group and that from a video-conference, start fresh with a new project.
  • Be highly cognizant of data privacy protocols. Compliance with privacy laws apply more stringently to shared and stored video.

It’s going to be hard to convince me that video-conference focus groups, despite being absolutely possible, are advisable for all projects. This is particularly true for social and opinion research projects. All researchers have to adapt to our changing environment. The first question before launching a project should always be “what is my research goal and what tools can I use to reach it?” Then you can decide if having some data is better than having no data. Contact Quirk Global Strategies and we can help you decide.

 

/by

Can I Poll Now? Should I Poll Now?

, , ,

Everything is up in the air! No one knows what’s going to happen next! At Quirk Global Strategies, we’re used to unpredictability.

We’ve developed strategies to manage it. No matter where in the world we’re working, every project starts with a discussion of the research goals. What questions need to be answered? By whom? How will the research be used? When? Answering these basic questions lays the foundation for the project and makes it easier to answer more complicated design questions down the road. This process helps us adjust our research plans to changing circumstances.

A meter apart, ladies

All survey research is a snapshot in time. Using it to predict views in the future is always a mistake. In the Coronavirus era, when the long term social, health and economic impacts have yet to fully hit, trying to use today’s data to predict what people will be thinking in months, or even weeks, is a waste of money.

Views on issues can be harder to shift than you’d think. As a political pollster in the US in September 2001, I assumed that the 9/11 attack was the kind of event that would radically shift perceptions on political topics. After the initial feelings of fear and insecurity wore off, pollsters found that voters’ priorities for elected officials or party preferences had changed little. This is the closest analogy in my political career to what we’re facing now. We can expect the economic and social impacts of COVID-19 will likely be much more far reaching than 9/11’s. But right now, we just don’t know.

Once you decide on the goals of your research, ask yourself these questions:

             Do You Need to Know Now?

If you need to understand how the current environment is shaping public opinion on a policy question today, or if you need to know how consumers have adapted their behavior to physical distancing, you should consider moving forward. If you’re planning to launch an advocacy campaign in six months or a year, you should probably wait, especially if you have limited resources to change your strategy or go in the field again.

            Do You Have the Resources to Respond to the Findings?

There’s nothing worse than binning a costly research project because the landscape has shifted, rendering your data useless. Can your plan be changed if the data reveal something unexpected or counterintuitive? If the answer is no, you should wait. Good research often reveals such findings, particularly in an uncertain environment. It’s wasted if you can’t incorporate it.

             Do You Have the Resources to Poll Again?

For many campaigns, knowing what people are thinking right now, at the beginning of the crisis, is critical information. A snapshot of attitudes in Spring 2020 will provide a baseline for tracking attitudinal shifts later in this year and in the years to come. It might also reflect the “worst case scenario” for your issue, which is useful to know when preparing a campaign. Depending on your timeline and your goal, you will probably need to poll again to update your assumptions. Add that to your budget.

            Is Some Data Better Than No Data?

In the difficult environments where QGS typically works, we don’t let the perfect be the enemy of the good. We always operate under budgetary, security and time restrictions that force us to make concessions. But since we fully understand the goals of each project, we know which methodological trade-offs we can live with and which we cannot.  We adjust our sampling plan to adapt to realities on the ground, report our methodology transparently and adjust our analysis and strategic recommendations accordingly. If you need actionable data, even if it’s not perfect, you have more flexibility in your data collection options.

            Who Do You Need to Talk to?

The answer to this question will help you decide which data collection mode is optimal, given your research goal.

None of this, sorry

If you need a large, proportionate, general population sample in a country where the only way to collect data is via face-to face-interviews, you’ll need to wait. Face-to-face interviews and traditional focus groups are simply off the table right now. Sadly, it’s impossible to predict when interviewers will be able to go out in the field and discussants can sit around a table talking to a live moderator.

The good news is telephone survey research to mobiles and landlines is thriving in Europe, the Gulf and North America. Interviewers stationed safely at home are calling voter file or RDD samples, like always. There are even anecdotal reports of marginal response rate improvements. If your universe has high mobile/landline penetration, phone surveys remain the best way to collect a random sample of a general population universe.

Data collectors in many middle- and low-income countries are on the cusp of being able to field random sample mobile surveys, particularly of urban populations. Before we commit to this approach, however, we need a full understanding of which groups are underrepresented (usually older, rural, lower SES) and which are overrepresented (younger, urban, higher SES) in these samples. We pay particular attention to the gender split: In some places it’s easier to interview women on the phone than in person. In others, men control access to the mobile phone. Then we decide, based on the goals of the research, if we can live with the trade-off.

            Non-Probability Options Abound

If you’re interested in non-random views of urban, younger, educated, higher SES, respondents with mobile phones or internet access, there is no shortage of methodologies available. This is particularly true in high- and middle-income countries but these populations are accessible even in many low-income countries via panels as well. Methodologies such as online surveys, SMS surveys, IVR and social media analytics can also be combined to give a richer, more contextual view of the landscape. Keep in mind, these modes sacrifice randomness and are not a substitute for a proportional sample.  Review the research goal then decide if it matters.

           Quirk Global Strategies Can Help

So should you poll now? Return to the goal of your research. Look at your budget and decide which trade-offs you can tolerate and the ones you cannot. Email us and we can help you think through your options and suggest the best one. We might even suggest waiting.

/by

Global Survey Research in the Time of Coronavirus

, , ,

Curious about how opinion polling can proceed when entire countries are closed down? I was, so I reached out to data collectors around the world to better understand what’s possible now and what kind of projects should wait.

Pollsters who use telephone interviews are in a good position to keep on working. Unfortunately, those who rely on interviewers who need to move through communities and interact with people in their homes are going to have a rougher time. In the short term, researchers will have to put those projects on hold or consider new methodologies.

Telephone Surveys Are Happening

From the early days of COVID-19, phone banks serving high-income countries of North America, Europe, and the Middle East started adapting. Obviously, big rooms filled with interviewers sitting in front of terminals are out of the question. In response, data collectors I’ve spoken to quickly set their interviewers up on secure systems that allow them to call from home. This is good news for pollsters and interviewers.

Given the unrelenting demand from election pollsters, this decision was a no brainer for US-based firms. Those with global calling capacity and multi-lingual interviewers can transfer also projects to offices in countries where workforces have been less affected by shutdowns. Response rates might also improve because people stuck at home could be more willing to respond to surveys. In the US at least, it would be hard for response rates to get worse.

For those who field in high-income countries, this is all good news. It’s important to keep a couple of things in mind, however. In the US, ordinary election year demands stretch collectors’ capacity up to and often beyond limits. Having limited remote interviewers will exacerbate the problem. Management capacity will also be stretched as quality control and training demands grow. Patience, oversight and regular communications with data collectors are critical.

If your project in high-income countries in North America, Europe or the Middle East needs to move forward, then experienced phone houses with strong management can get it done. If you can hold off, or wait for a lull in your collector’s schedule, the quality of your data will likely be higher. I analyze my data by day as it comes in with an eye for anomalies. It’s also smart to build in extra days for call-backs, or in case things go haywire and you need to replace interviews or do extra QC.

Face-To-Face Surveys Are Problematic

As a specialist in survey research in conflict and post-conflict environments and low/middle income countries, about 80% of my projects rely on face-to-face (F2F) interviews. Phone interviews are not an option in places where mobile penetration is low or disproportionately distributed.  In normal times, in-person interviewers face obstacles such as transport difficulties, bad weather and conflict-related threats. Those problems are manageable. But add a highly contagious disease spread by personal contact and F2F data collection suddenly becomes untenable. It could be this way for while.

Whether F2F data collection is viable depends on a country’s infection rate and whether restrictions on movement and social contact have been put in place. This fast changing situation makes planning almost impossible. For example, I have a survey ready to field in India. Up until a week ago data were being collected normally in many states and slowly, and with extra precautions, in others. On 25 March, the entire country was locked down to stop the spread of the virus. Thankfully, we had opted to wait until the situation clarified and had not begun fieldwork.

The situation in many African countries is also ambiguous. Some countries have few infections and life continues normally, for the time being. Other countries are suffering badly from outbreaks. Ukraine has a serious outbreak and F2F data collection is on hold. Quant work in the Philippines is also on a slowdown, if not entirely stopped.

Experienced, ethical data collectors are in the best position to offer advice on the local situation. The final judgment always belongs to the researcher however. No one — interviewer or respondent — should be put in danger for the sake of survey research.

Data collectors in some F2F countries have begun experimenting with phone surveys and online panels. The possibility for error and biased samples remain serious concerns. Having a complete understanding of how these samples under- or over-represent populations is critical; some populations will simply not be reachable by these modes. Survey work in difficult environments often requires methodological tradeoffs. If you can live with increased and unpredictable error from non-probability samples, or if you can afford to be experimental, I say give it a try.

I Can Field a Survey. Should I?

Of course, pollsters need to consider ethical questions before fielding surveys during a global pandemic. Should interviewers be sent into the field, handhelds ready, to conduct interviews in people’s homes? Absolutely not. Should worried or scared respondents be pestered with questions about topics of less importance than life, death and economic survival? I hear the same argument against conducting surveys in conflict zones. Often, the concern is overstated. Many people — especially in low- and middle- income countries — are not asked their opinions asked about anything. They’re usually happy to oblige. Additionally, respondents can be more forthcoming in times of insecurity and unpredictability. Data collected during this period will be a real “snapshot of a strange time” and will be fascinating to track over time.  As I do in conflict zones or closed spaces, I let respondents tell me if they are uncomfortable or unwilling to participate. I look at response rates, drop-offs and interviewer comments before I make judgements about respondent willingness.

The need to understand public opinion during and after this crisis is not going to go away. Economic, political and social disruption could shape perceptions in unpredictable ways in low/middle income countries and upper-income countries alike. Views that once seemed hardened and unlikely to shift may change radically, or not at all. Whether you’re working in a high income country of North America, Europe or MENA, or looking at surveys in lower/middle income countries, Quirk Global Strategies can help you sort through your options. Contact us through the link on this site.

/by

“The Polls Were Wrong” Is the Easiest Answer

, ,

Since Sunday’s election I have been in the improbable position of defending Turkish pollsters. To be clear, I have had many, many quibbles with their lack of methodological transparency and the perception of bias that causes.

Unlike other forms of public opinion or policy research, however, election polling provides a day of reckoning: you’re right or you’re wrong. It’s there for everyone to see. If you can’t get the election right, within the margin of error of your sample, no one should pay attention to your data. If you repeatedly get it right, you deserve a degree of credibility.

Here’s the thing: several Turkish pollsters were pretty damn close to getting the 7 June parliamentary election correct. And when I say “correct” I mean “correct within the appropriate margin of error in the days immediately before the election.” Therefore, their November election results which missed the mark should not be dismissed outright, especially since multiple pollsters reported similar results.

I’m going to digress a bit. It’s absolutely fundamental to understand the basic principles of probability sampling if you’re going to comment on pollsters’ performance.

A poll reflects voters’ views on the days the survey was conducted within a margin of error. Here’s what that means: if you draw a truly random sample of, let’s say, n=1000 people within a universe (Turkey, for example), and you write the questions and implement the survey in ways that diminishes biases inherent in survey research and your sample reflects the demographics of your universe, your data will, within a standard margin of error of plus or minus 3.1 percentage points (characterized, shorthand, as MoE +/- 3), reflect the views of that universe on that day. That means the results of any data point could vary three points higher or three points lower. This is critical to take into consideration when declaring a poll “right” or “wrong” relative to election results or stating a candidate/party “is ahead in the polls.”

Two important takeaways:

• The only way to achieve a margin of error of zero is to interview every single person in the universe (Turkey, for example). That’s impossible and is why we rely on probability sampling. The trade-off is we have to accept and accommodate the margin of error in our analysis. If we fail to do that, we’re wrong. Period.

• Pre-election polls are not designed to project what will happen on election day (you can do that through modeling, but it’s risky). This is why everyone (especially candidates who are about to lose) says the only poll that matters is the one on election day — it’s the only one that’s a 100% accurate report of voters’ views with no margin of error.

If you don’t believe all this, go take a statistics class and then we’ll argue about it. It’s science, not magic. Also, please do not give me an exegesis on academic research. Like these pollsters, I work in the real world with budgets and time constraints.

So, let’s look at the last three public polls taken before the 7 June election. I chose these three because 1) fieldwork was conducted the week or two before the election and 2) they shared their sample sizes so we know the margin of error. (There may be others, but I found these data here). We want to look at polls conducted as close as possible to the election because they’ll capture the effects of late-breaking campaign dynamics. (Also, not rounding is an affectation. I round. Personal opinion).

 

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

MAK

44

25

16

19

n=2155

+/- 2.1

18-26 May

SONAR

41

26

18

10

n=3000

+/-1.8

25 May

Gezici

39

29

17

12

n=4860

+/-1.4

23-24 May

Andy Ar

42

26

16

11

n=4166

+/- 1.5

21-24 May

June Results

41

25

17

13

n/a

0

7 June

I draw two conclusions.

First, putting aside ORC which overrepresented AKP and underrepresented HDP, Konda and Gezici were pretty damn close to the final result (by that I mean close to within the MoE), considering data was collected a week before election day.

Secondly, though it can be risky to compare data collected by different operations, their data are very similar, which suggests they are using similar methodology and making similar assumptions. That’s the way it should be.

Next, let’s look at publicly released data for the November election. I borrowed most of these data from the delightful James in Turkey and he did not always include the margin of error. I will take that up with him at a future date. Let’s assume pollsters without sample size indicated interviewed between n=3000 and n=5000 (that’s what they did in June), so the margin of error will be between +/-1 and +/-2

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

Andy R

44

27

14

13

n=2400

+/-2

24-29 Oct

Konda

42

28

14

14

n=2900

+/1.8

24-25 Oct

A&G

47

25

14

12

n=4536

+/1.4

24-25 Oct

Metropoll

43

26

14

13

n=

15 Oct

Gezici

43

26

15

12

n=

15 Oct

ORC

43

27

14

12

n=

15 Oct

AKAM

40

28

14

14

n=

15 Oct

Konsensus

43

29

13

12

n=

15 Oct

Unofficial Final

49

25

12

11

N/A

+/-0

1 November

AKP’s final number falls outside all the polls’ MoE, except A&G’s. The next closest, Andy R, conducted the latest fieldwork so was in the best position to capture emerging trends, such as a surge in AKP support. Andy R still underreported AKP support by five percentage points. That’s a lot. A&G didn’t release any tracking data so it’s hard to know if it’s an outlier or ahead of the others in capturing the AKP surge. The latter is possible and I will address it in a future post.

If consistent sampling methodologies and questions are used, it’s possible track data over time to see if it changes. Big unexplainable differences from one dataset to another could indicate a problem in the methodology. I like it when pollsters provide election tracking data. It suggests sound sampling and alerts us to important trends in public opinion.

For fun, let’s take a look at two of those who did:

KONDA

AKP

CHP

MHP

HDP

June 7 Results

41

25

17

13

Aug 8-9

44

26

15

13

5-6 Sept

42

25

16

12

3-4 Oct

41

29

15

12

17-18 Oct

42

28

15

13

24-25 Oct

42

28

14

14

Unofficial November Final

49

25

12

11

GEZICI

AKP

CHP

MHP

HDP

June 7 Results

41

25

17

13

3-4 Oct

41

28

17

14

17-18 Oct

41

27

16

13

24-25 Oct

43

26

15

12

Unofficial November Final

49

25

12

11

Not only are these two pollsters consistent over time, they are also consistent with the final June results and compare favorably with each other. Nothing in either of their datasets suggests a big shift in opinion toward AKP (they do indicate an AKP trend, which is plausible). Yet, inthe end, their last polls are wrong wrong wrong about the November result. That’s really troubling.

How could pollsters who nailed it in June have missed it in November? How can they be consistent over time and with each other and be wrong on election day? Falling back on “the polls are wrong” as analysis is simply inadequate. If you’re going to disregard months of consistent data, you should provide an explanation for how it went wrong.

I honestly can’t give an adequate explanation. Because I have other things to do and you have short attention spans when it comes to statistics, I will address what I think are the three most likely polling error culprits in future posts. These include (in random order of in likelihood):

• Errors in methodology (this will address the absurd argument that since UK and US pollsters were wrong, it follows that polls in Turkey are also wrong. I can’t believe this is even necessary)

• Errors in analysis (not reporting or considering Undecideds or softening support, which is my current theory of choice)

• Election dynamics that cannot be captured by polling

NOTES: If you want to look at a few other pollsters’ June data, here it is. I don’t think it’s totally fair to judge their accuracy based on data collected weeks before election day, but, with the exception of under-representing HDP, most of them (except MAK) actually are pretty close and provide more evidence of the consistency of public opinion. Being off on HDP can be forgiven because HDP had what campaign people refer to momentum and it is plausible HDP’s support increased in the final weeks.

AKP

CHP

MHP

HDP

Sample Size

MOE

Date

MAK

44

25

16

19

n=2155

+/- 2.1

18-26 May

SONAR

41

26

18

10

n=3000

+/- 1.8

25 May

Gezici

39

29

17

12

n=4860

+/- 1.4

23–24 May

Andy Ar

42

26

16

11

n=4166

+/- 1.5

21-24 May

June Results

41

25

17

13

N/A

0

7 June
/by

Want to Increase the Credibility of Election Polling in Turkey?

,

Remember that time you asked me how to increase the credibility of public polling in Turkey? No? Well, it turns out I have thoughts on the matter. Here they are.

Transparency, Transparency, Transparency:  This is the single most important factor. Given the amount of flawed data out there, every pollster who releases election polls publicly should voluntarily provide the following information for the sake of increasing public trust in the science. Reporters should ask for it. Not all of it needs to be reported by the media — it typically isn’t — but it provides important information, especially to professionals and academics, about how data were collected and processed. Allowing outsiders to review and discuss the methodology increases the rigor of the research. Ultimately, the result is greater public confidence in polling data.

Here’s the type of information that would be helpful. (The first four bullets should be reported in every media story that references a poll, without exception)

  • Sample size, sample type and universe: Here’s an example. “A national (or urban, or regional sample of 25 provinces) sample of n=2000 adults in Turkey over age 18.” If the pollster diminished the sample to include only likely voters, he or she should explain how that determination was made.
  • Fieldwork Dates: Knowing when the data was collected provides important context about events that occurred in the political environment and could affect perceptions of the candidates (i.e., a deadly mine disaster, or a huge corruption scandal). Fielding dates also tell you if there was enough time to return to selected respondents who weren’t available on the first try. With a large sample, a day or two in the field isn’t enough for callbacks. Therefore, the data are biased toward those who answer their phones or their doors on the first try.
  • Margin of error for the sample as a whole: “The margin of error for the n=2000 sample is 2.19% at the 95% level of confidence. The margin of error for demographic and geographic subgroups varies and is higher.”
  • Who’s the Funder: This is critical information. Who’s paying for a survey may impact the credibility of the data. It may not. But you have no way of judging if you don’t know who’s coughing up the dough. In the US, few pollsters would jeopardize their reputation for accuracy and reliability by swinging data in favor of a well-funded or powerful interest (some would and have, but it’s an exception, not a rule) but revealing who’s paying for the research is standard. Even if Turkish pollsters don’t monkey with the numbers (and lots don’t) the perception that pre-election polling is cooked is well-founded, pernicious and must be addressed if opinion research is going to be used as a credible tool for illuminating public policy debates and elections.
  • How the interviews were conducted and how respondents were selected: Were interviews conducted using face-to-face interviews? If so, how were respondents selected? Were the interviews conducted by telephone? What proportion of landlines versus mobile numbers was used? How many efforts were made to call selected numbers back if there was no answer? What times of day were interviews conducted?  If the answer is “online poll,” step away from the story.
  • Response rates: What percentage of selected respondents participated in the survey? This varies depending on the country and sometimes, the type of survey. The pollster should reveal what standard response rates are in Turkey for similar surveys. An abnormally high or low response rate should raise red flags.
  • Question wording and order: How a question is asked and where it appears in a survey directly affect responses. Respondents should not be “primed” to answer a particular way. Therefore, a vote preference question should be one of the first respondents are asked. The list of candidates should be presented exactly as names appear on the ballot, with no extraneous information provided that voters won’t see when they enter the polling station. The percentage of respondents who answered “don’t know” or “undecided” (a critical data point in election polling) should also be reported and if the “don’t know” response was prompted or unprompted.
  • Quality Control: How many interviews were verified in the field by supervisors or called back to make sure they really took the survey?  I know it’s hard to believe but sometimes interviewers are lazy and fake interviews! Quality control is expensive, technical and time consuming and is why methodologically sound polling is expensive. Rigorous quality control by outsiders reduces the chances that data are falsified, especially in the processing phase where someone *might* be tempted to place a finger on the scale. Opening data sets to outside scrutiny is a good way to expose and prevent this.

 

  • Sampling and weighting procedures: It’s easy to baffle non-specialists with statistics but polling isn’t rocket science and random sampling procedures are guided by industry standards. Pollsters should reveal if their samples are stratified and by what variables. They should share how sampling points were selected. They should also reveal if the final data were weighted and by what factors.

 

Wow! This sounds like a lot of work! But one of the most interesting outcomes of the 2012 election in the US, in which a high profile, well-respected research outfit (Gallup, in about as epic a scandal as pollsters are allowed to have) got the election wrong, was the degree of public scrutiny Gallup allowed of its methodology to figure out what happened. I’m sure it was painful — no one likes to admit they did things wrong — but the result is better, and more credible, public research. Gallup’s reputation took a hard hit, but they dealt with it the best way they could. If you really want to learn more about what happened to Gallup — and why wouldn’t you? Pollsters are awesome — read this report.

 

Given that major Turkish pollsters, including a well-respected one, got the Presidential election wrong, this issue isn’t going away soon. Historic low turnout figures –preliminarily 74% — might have thrown some pollsters for a loop but it shouldn’t have, given the timing and the dynamics of the election. The challenge always, as US pollsters have found, is trying to predict which voters will cast ballots and which will stay home. Turkish pollsters, who already face credibility issues, need to confront this issue with transparency.

**Quirk Global Strategies isn’t in the business of public polling. We’re strategic pollsters, which means private clients use our data to guide their internal political or communications strategies (though not in Turkey, usually). This is an important distinction.

/by

How to Be a Good Consumer of Public Polls

, ,

Here we are again, weeks out from another Turkish election, arguing on Twitter and in bars about the pre-election polls on the Presidential race between the shouty guy and the bread guy. As much as we’d like to, we can’t really ignore this election, so wouldn’t it be great if someone explained how to tell if a poll in the paper is credible or not?

It’s your lucky day! Here are some basic, but important, concepts to understand before you write about, argue about, print, or tweet publicly released election polls (everywhere, too, not just Turkey).

 

  • How many people were interviewed? It amazes me how few press articles include this mandatory information. A nationally representative sample should include at least 800 randomly selected respondents, which has a margin of error (MOE) of 3.5% at the 95% level of confidence.* A larger sample size does not necessarily mean the survey is better (academics may argue otherwise, but their research goals are different), so don’t fall into that trap. For example, the margin of error for a n=2000 sample is 2.2% (compared to 3.5% for n=800). That’s not a big difference and won’t matter that much except in the closest elections. However, if the pollster is sharing data from smaller demographic or geographic subgroups within the national sample (men, women, Kurds or Istanbullus, for example), a larger sample size becomes more important. Remember, the MOE increases as the number of interviews decrease. If Istanbul makes up 19% of the country (and in a nationally representative sample, it will) in an n=800 sample, there will be only 152 interviews among Istanbullus, with a MOE of 8%. If the sample is n=2000, there will be 380 interviews (MOE 5%) among Istanbullus. I’m slightly more comfortable with the latter data than the former because the margin of error is smaller. Do you like to play around with sample sizes? I do! There’s an app for that.
  • Who paid for it?  This is Turkey so this is probably the single most important question. In the US, major media outlets (and think tanks) commission credible research firms to conduct election surveys (the “CNN/Washington Post poll,” for example), the results of which papers report as news. Given they are in the business of reporting things that are more or less true, they have a lot at stake by getting the numbers right. The media in Turkey operate according to different principles. That a media outlet reports data tells us little more than in whose favor the numbers are likely to have been cooked. Methodologically sound research is expensive in Turkey — $20,000 to $30,000 for data collection alone — and for-profit research firms are unlikely to undertake survey work for fun, even if they say they do. Someone’s paying for it and if you can’t find out who, don’t report it.

 

  • Who was interviewed? Election polls are designed to predict election outcomes. It sounds harsh, but non-voters’ opinions don’t matter. Therefore, only likely voters should be polled. Because voting is compulsory in Turkey, election participation is very high (88%-90%) so nearly all adults are eligible to participate in an election survey. In contrast, election polling in the US is extremely complicated: only about 50% of Americans are eligible to vote (by virtue of having registered), and among those, participation rates vary from the extremely low (15% in low-interest primaries) to the less low (about 65% in presidential elections). Predicting who should be included in a sample of likely voters is extremely challenging. Misreading the composition of the electorate was one of the reasons major polling firms got the US election in 2012 wrong. Because of its timing (10 August, mid-vacation), uniqueness (it’s the first time Turkish voters have directly elected a president) and low interest in the candidates among the tatıl-class, Turkey’s presidential election presents a unique challenge to election pollsters. Is there going to be substantial drop-off in participation among certain types of voters who won’t bother to return to Istanbul from Bodrum’s beaches to vote? Maybe! Pollsters who care about accuracy will take this into account. They should explain how they’re addressing this issue, and how, if at all, they’re diminishing their samples to exclude those who won’t vote. Ask! Ask! Ask!

 

  • How did they conduct the interviews? Generally, in probability samples (the only kind that produces representative data and the only kind I will discuss), a respondent is selected at random to participate in either a face-to-face (F2F) or telephone interview. F2F has always been the norm in Turkey because of low phone penetration but that’s changing quickly as more and more people obtain mobile phones. Mobile sampling is becoming more and more common. Both methodologies have biases and you should know which methodology the pollster uses so you can be aware of them. I can go on for days about the pros and cons of each (it’s a wonder I have any friends at all). Online, web-only surveys are bogus. If you ever want to start a flame war with me on Twitter, report on an online survey like this one without using the word “worthless.”
  • What’s the polling firm’s track record? Accuracy is a pollster’s currency. The great thing about election polling is there’s a day of reckoning. You either get it right and can be smug (it’s science!) or you’re wrong and no one should listen to you anymore. Given the dearth of credible election polls in Turkey, calling previous election results correctly boosts a pollster’s credibility even more in my book. As far as I know, and I don’t know everything, one firm did that publicly in the March local elections: Konda. Why data released by firms that got a recent election completely wrong are treated as credible is a mystery to me. It’s easy to check.

 

This isn’t all there is, but it’s plenty and you don’t have to be a specialist to interpret it (as long as you understand probability sampling). Having the answers to these questions will make it easier to assess the quality of the polls you see in the Turkish press and on Twitter. Armed with this information, you’ll have the tools to be able to say “this poll sounds like BS. I’m not going to report/tweet it,” thus depriving bogus pollsters of the media oxygen they need to survive. If you can’t get answers to these questions, don’t report the data.

 

TOMORROW (or some day in the near future)! How to Make Public Election Polling in Turkey More Credible 

 

*If your universe (total number of potential respondents) is greater than a couple hundred people, the margin of error is the same for a random sample of n=800, regardless if you’re surveying a city with a population of 1500 people or a country of 78 million. If you don’t understand why this is, or what a margin of error is, get thee to a Stats 101 course and don’t start arguments you’re going to lose.

**Quirk Global Strategies isn’t in the business of public polling (or academic research). We’re strategic pollsters, which means private clients use our data to guide their internal political or communications strategies (though not in Turkey). This is an important distinction. Strategic pollsters who collect bogus numbers give bad advice, lose elections and don’t get hired again. Therefore, we strongly oppose BS numbers. You can be certain that strategic polling is being done in Turkey — most likely on behalf of AKP — but you and the twitter loudmouths you follow are unlikely to get your hands on it.

 

/by

Stop Reporting the Bilgi Poll!

,

Like many of you, I have visited Gezi Park over the last few days. While walking around, I noticed that a lot of the protesters are young and they seem new to the business of protesting. They had strongly held views on a lot of topics but are not overtly political.

My observation is about as scientifically valid as the poll released by Bilgi University earlier this week. I’m not going to repeat the findings. That so many respected journalists are citing and retweeting it without mentioning (or probably even looking to see) that, according to the exceedingly vague methodology statement, it’s a 20 hour online survey of 3000 people, is vexing. I’m going to assume (probably incorrectly, but I’m struggling to be generous) that there’s more information about the methodology in the Turkish, but when I saw the word “online” that’s when I clicked “close tab.”

Polling 101: Online surveys are representative of nothing except the universe of people who 1) knew about it, 2) had internet access during the 20 hours it was open, 3) felt like responding.  Participants were not randomly selected; they choose to participate, which makes them different at least one way from those who did not. It’s called selection bias.

Even worse, it appears that a lot of folks are repeating data from the poll because “it seems to make sense.” That’s confirmation bias, which is also sloppy.

If you really have to cite that poll, I suggest phrasing it thusly, “According to a worthless online survey of Gezi Park protesters publicly released by Bilgi University, which you’d think, as an academic institution would know better……”

There are ways to randomly select a sample of protesters and find out more about their demographics and attitudes. It’s time consuming and expensive, like good research usually is. Wait until someone does that, then report it.

I have something to say approximately every four years. I’m like a pollster cicada.

/by

The Challenge of Exit Polls in Unstable Countries: Kenya

, ,

Two excellent examples in the news (more or less) this week about conducting polls in unstable countries. (Tomorrow, the latest poll from Georgia).

The first was a snippet in an interesting Slate article by Alex Halperin called What’s Going on In Kenya?

Unfortunately, one bit of data has not surfaced. The International Republican Institute, a democracy-fostering nonprofit funded by the U.S. government—and despite the name, officially nonpartisan—commissioned an Election Day exit poll but has declined to release the results. Two people familiar with the results told me that they showed Odinga with a substantial lead over President Kibaki—one reported eight points, the other nine points. One has only to remember the United States’ 2004 elections to know how fallible exit polls are, but a U.S.-sponsored survey would have weight here and could have given the ECK pause before it called the election so disastrously.

Ken Flottman, an official in the IRI’s Nairobi office, said the data would serve additional purposes, such as studying voter demographics. The organization issued a statement criticizing the vote counting but does not mention its data. It missed an opportunity to advance its mission of promoting democracy and fair elections.

There’s been limited news coverage or reaction in the blogosphere so far to this except from knee-jerk reactions from people who know little about IRI’s mission or the purpose of exit polls. Furthermore, drawing conclusions about the fallibility of an exit poll in Kenya based on the 2004 election in the US or any other country, as Halperin does, is specious.

Halperin asks a fair question, though: Where’re the data?

Read more

/by

Is it Possible to Conduct Solid Pre-Election Research in a Place Like Georgia?

,

Simple question, complex answer.

The Caucasus Research Resources Center, with whom I met on my recent trip to Georgia, starts to get to the heart of the matter in this very good post on how to evaluate pre-election polls. To boil Hans’ argument down, the burden on is on the pollster to publicly disclose as much information about the data collection process as possible. Of course, the media has to also report the results responsibly, which is almost as big a hurdle in these countries as disclosing basic information about sample sizes, margin of error, interviewing techniques and, most importantly, funders.

This sort of disclosure is necessary anywhere, whether it’s Ukraine, Azerbaijan, Georgia, Armenia (Onnik at the Armenia Election Monitor has been posting on this topic quite a bit lately) or even Iowa. That’s why Pollster.com has been a strong proponent of The Disclosure Project, which pressures U.S. pollsters to reveal more about their methodology. This is even more important in countries where pre-election opinion polls are relatively new and neither the media nor voters are very sophisticated poll consumers.

Conducting methodologically sound polling in a highly politicized environment like Georgia or Azerbajian is difficult, but not impossible (and I do put Georgia and Azerbaijan in the same category in that regard– I was shocked at how polarized the pre-election environment is in Tbilisi. The pre-election atmosphere in Georgia has much more in common with Azerbaijan’s prior to the 2005 election than it does with Ukraine’s 2006 or 2007 pre-election period, which is depressing). Just like in campaign finance, disclosure is the the first and most important step to increasing public confidence in the process.

People need to understand that polling is neither good nor bad. It’s simply a tool that can be put to both legitimate and nefarious purposes. Polls are fundamentally democratic because they give ordinary people a voice, but disclosure helps an informed citizenry assess whether their voices are truly being heard or are being manipulated.

/by

Projects in the News: ABC/BBC Survey in Afghanistan

, ,

In October, I traveled to Kabul on behalf of Charney Research in New York to oversee the pre-tests and interviewer training for a nationwide survey conducted on behalf of ABC news, BBC News and ARD of Germany.

The results, which were released today, are interesting for a number of reasons– particularly the wealth of tracking data from 2006 (a project for which I also traveled to Kabul for pre-tests and trainings) and 2005. As the ABC story (which is more insightful than the BBC’s) emphasizes, Afghans are increasingly critical of US efforts, with only 42% positive, down from 57% in 2006. More than half (53%) disapprove of the job the US is doing. It’s important to note, however, that the presence of US troops isn’t what is drawing Afghans’ ire (71% support their presence), it’s their performance. Civilian deaths, especially in the Southwest, understandably, turn Afghans away from US and NATO forces. This is an important finding with implications for US policy there.
Read more

/by