Posts

What Happens When It Takes a Long time for Russia to Lose?

, , , , , , ,

This blog only enters my mind when I need to remember how to post on it, usually every six month or so. One of my last posts, which I wrote more than a year ago, is entitled What Happens When Russia Loses?

“Yikes,” I thought. “That was a more optimistic time. I might have to write a new post called What I Really Meant to Say Was…”

Actually, the only part I’d change is the title. I’d make it What Happens When It Takes a Long Time for Russia to Lose? In addition to the good things that could happen in the region when Russia loses (all still true), there are, I should point out, some bad things that will happen if it takes a long time. They are happening now.

FOUR BAD THINGS (a non-exhaustive list)

OBVIOUSLY, Ukrainians continue to suffer unbearable losses

Even if there’s a “peace agreement,” an undefeated, revanchist Russia will attack Ukraine again once it has reconstituted itself after staggering losses in its stupid war of aggression. Ukraine will not live peacefully until Russia is militarily and politically incapable of invading it. The longer this war goes on, the more Ukrainians die. They will not stop fighting.

Emboldened regional autocrats

In a recent webinar hosted by Coda Story, certified smart person Peter Pomerantsev (my advice: read everything he writes on influence operations, especially his three books on the topic) argued persuasively that an emboldened Russia provides space for openly pro-Russian regimes (Belarus, Georgia) to be more aggressive. For example, the pro-Russian Georgian Dream ruling party has rammed through a copy of Russia’s foreign agent law, despite massive public protests. Regimes that keep a finger to the wind, like Turkey, Kyrgyzstan, Hungary and Serbia, will use an emboldened Russia for cover to shore up their own corrupt structures. All have or are planning a similar Foreign Agent Law. To them, there’s no downside to stepping into Russia’s footprints and the benefit – defanged civil society in no position to challenge them – is obvious. Why is this bad? Read this report that I helped write about what happened in Russia after passage of its Foreign Agent law.

Incremental progress toward democratic development in Moldova, Georgia, Armenia and Kazakhstan, at best, is halted

At best, Russia will continue to meddle, through influence operations, election fraud and political interference. It will be impossible for these countries to chart their own paths. At worst, irredentist Russia will reabsorb them; the USSR with a more palatable name.

An emboldened Russia threatens Europe

As a resident of France, I find this one compelling. A war with a Russia that has subsumed the Ukrainian arms industry, army and geography is a much stronger opponent for Europe than one that Ukraine defeats and sends back to its 1991 borders. This is Professor Timothy Snyder’s “1938 Argument.” Helping Ukraine defeat Russia now means Russia can’t threaten Poland and the Baltics (something it already does regularly) and next, France, Germany and Italy. President Macron (link in FR) gets what is at stake. If you understand French, he explains it in simple terms.

DOES IT GO TO 11?

For Russia, the absence of defeat is victory. If I wrote that 2023 post again, I would turn the volume on the sense of urgency up to 11. We’re seeing, right now, today, what happens when Russia’s defeat is delayed by slow American aid delivery and absurd restrictions on how Ukraine uses weapons to defend itself. Ukraine is the biggest victim by a huge margin. But other countries, like Georgia and Moldova, are in Russia’s crosshairs too. It’s not going to get better, especially for Europe. Give Ukraine the tools it needs to finish the job.

Contact Quirk Global Strategies to talk about new and clever ways to measure public opinion in complicated places.

 

 

/by

Never Interrupt Your Enemy When It’s Making a Mistake

, , , , , , ,

Sometimes when I’m feeling dark, I pretend I am a Russian “political technologist” designing influence operations in Ukraine. I try to deconstruct what Russia is trying to achieve and why. I scratch my head. Do they know their efforts to undermine Ukrainians’ commitment to defending themselves is dumb and ineffective? Do they do any research at all?

Russians have had little success in undermining Ukrainians’ support for the war effort or their leaders. Pre-full scale invasion, however, Russians were pretty good at causing division in Ukrainian politics, particularly around major strategic issues like EU and NATO membership. Russia used its television channels, Ukrainian political allies and Ukraine’s natural political divisions to prevent leaders from building consensus about Ukraine’s future direction. For a number of reasons, it has become much harder lately for Russia to get traction.

Four Major Changes in Ukraine

I can identify four major changes in the Ukrainian information environment to help explain why. These are important for anyone trying to understand Ukrainian public opinion since the full-scale invasion.

  • Ukraine blocked Russian TV channels from Russia in 2014 and shut down Russian-language channels in Ukraine owned by Russian-aligned oligarchs in 2021.
  • Ukraine banned 11 Russian-backed political parties in March 2022.
  • Ukrainians are sophisticated information consumers.
  • Ukrainians are not receptive to and/or are openly hostile toward messaging they perceive as coming from Russia.

No more Russian TV channels

Television remains the most effective tool to shape mass public opinion. Accordingly, the single most important factor limiting Russian influence over public opinion in Ukraine has been the closure of Russian channels from Russia and Russian channels from Ukraine. Kremlin narratives and coordinated messages simply do not reach a mass audience in Ukraine like they used to. (Ukraine’s United TV Marathon has succeeded, to some degree, in achieving this. That’s a different post).

Sure, Russia has access to Telegram. Everyone does. Telegram channels are useful for delivering targeted messages to receptive audiences who are looking to confirm their biases, who distrust official narratives or who want to fill information gaps. All kinds of channels fill this space, however, making it quite fragmented and noisy. Russia holds no particular advantage, like it did when Kremlin-friendly TV stations pumped garbage 24/7 into millions of Ukrainian living rooms, unchallenged.

Russia-backed political parties shut down

 Ukraine’s democratic institutions provided cover for Russian-backed political parties and their parliamentary faction, allowing them to undermine democracy from within for years. The largest, Opposition Platform/For Life, enjoyed legitimate support from voters in Ukraine’s East and South and held 44 seats in the Rada. However, it was backed by Viktor Medvedchuk, a pro-Moscow Ukrainian oligarch with close ties to Putin. Banning these parties, while controversial, inhibited Russia’s ability to take advantage of Ukraine’s democratic system to promote the Kremlin’s agenda.

Ukrainians are sophisticated information consumers

 Ukrainians have become sophisticated information consumers, a process that predates the full-scale invasion. Going back to at least 2016, Ukrainian civil society and educational institutions developed tools to help Ukrainians identify information sources designed to manipulate them. Some approaches worked, some didn’t. Some Ukrainians will never be reached and will remain vulnerable to, or even embrace, Russian influence. Many more, however, learned to pay close attention to who might be fighting for their attention and why.

Most Ukrainians now consume information primarily in Ukrainian, either via Telegram channels or United TV Marathon. They can identify Russia-friendly channels and commenting bots by their bad grammar and spelling as easily as native English speakers can spot Russicisms. Ukrainians also know that Telegram channels are the main source of unproven, manipulative information. They say they read with caution and double check against other sources, evidence of their information resilience.

Most Ukrainians are deeply hostile toward Russia and its narratives

 Pre-full scale invasion, a substantial number of Ukrainians used to hold, if not positive attitudes toward Russia, at least neutral views, particularly in the East and South. No longer. I suppose Russia can count that as an accomplishment of sort. No one else – certainly not the West — could have turned so many Ukrainians against it so thoroughly and most likely, irreversibly.

Ukrainians swat down messages they suspect might be coming from Russia, including common imperialist tropes that few questioned. Not only are Ukrainians generally not receptive, but the narratives often infuriate them. That’s not fertile ground for your influence operation to take root.

Most importantly, Russians not only lack the tools to influence Ukrainians’ opinions, they lack an understanding of their audience. Russians are trying – and failing — to undermine Ukrainians’ faith in a democratic system in which most Russians cannot perceive value. If they could relate to the importance of independence to Ukrainians and the complex relationship Ukrainians have with their democratically elected leaders, Russians might rethink their approach. The enormous gap between Russia’s and Ukraine’s worldviews makes it difficult for Russians to understand post full-scale invasion Ukrainians well enough to create anything but the clumsiest narratives.

May Russia continue to fail

None of this is to say that influence operations can’t work to undermine Ukrainians’ support for the war effort, build resentment toward their leaders and sow divisions in society. I’ve also thought a great deal about how that can be done. Russia, however, isn’t doing a very good job at it. Let’s hope they keep it up.

 

Christine Quirk is a France-based opinion research consultant and strategist who has been managing strategic qualitative and quantitative research projects in Ukraine since 2006 for a variety of clients. She has observed hundreds of focus groups on topics such as Ukrainians’ view of the full-scale invasion, elections and the political environment, how Russian influence operations shape Ukrainian public opinion, LGBTQI rights, women’s political participation and many others. Contact Quirk Global Strategies for more information about our work.

 

 

/by

What Happens When Russia Loses?

, , , , , ,

What a horrible, stupid war. As historians debate whether militaries make the same mistake over and over again, we here on the opinion-research side of things run the same risk. But we don’t like to lose, so we do what we can to keep from being surprised by events.

Do you want to be taken by surprise by what happens when Russia loses this war?

Russia is Going to Lose

What do I mean by lose? Russia is militarily, politically and economically weakened to the point where it is less able to exert direct influence over its neighbors’ political systems, less effective at shaping public opinion and unable to invade its neighbors.

Plenty has been written about how the West misread what happened in 1991. Amidst the triumphalism of having “won the cold war” and reaching “the end of history,” too many policies privileged Russia’s interests at the expense of newly independent states including, but not only, Ukraine. Subsequently, many have struggled to develop rule of law, equal rights, uncontrolled media and fair elections under Russian influence.

Understand What’s Going to Happen Next

I’ve long been bearish on Ukraine’s ability to defeat Russia. I’ve thought a lot about how, post-victory, Ukrainians will approach politics without Russia occupying territory, without political parties led by corrupt Russian interests, and with a population deeply wounded by Russian atrocities.

We know post-victory Ukraine is going to be very different. I’ve already got questions I want to ask: How will the (generally flawed and unpopular) political parties respond to the challenges of reconstruction? From what quarters will the next generation of leadership emerge? Who or what will fill the vacuum – long dominated by Russian-oriented parties – in eastern Ukraine? We’ll only know by asking Ukrainians what they think.

I’ve also got questions about how dynamics in Belarus, Georgia, Moldova, Serbia and Hungary might change without the incentives provided or imposed by Russia. A weakened Russia may be less able to back up Lukashenka’s dictatorial control over Belarus. Will this give opposition forces the opening they’ve been positioning themselves for years to run through? But what do Belarusians think about a post-Russia future?

Georgians just killed a draft foreign agent law, a version of which has decimated Russian civil society by, among other things, playing on Soviet-era fears of foreign spying. Energized, could Georgians take the giant leap of evicting a weakened Russia from territory it has occupied since 2008? Who do Georgians think can lead that effort?

Russia is not going away. Indeed, Russia itself could be wracked by internal conflict and competition for power that’s unlikely to favor democrats. But what’s the spillover of that going to be in the North Caucasus and Siberia? Seems like it’s worth trying to understand what people there think about their future.

Ask the Questions and Listen to the Answers

Like at the end of the cold war, Russia’s failed war on Ukraine could cause liquefaction of the political soil in Eastern Europe and Central Asia. Political dynamics that have been frozen in amber for decades could disappear, reconstitute or explode into chaos. Arguments in favor or against such scenarios are legitimate and worthy of debate.

However, unlike 30 years ago, there’s zero excuse for assuming we know (or not caring!) what people in countries from the Balkans to the Chinese border think about such big changes. Listening to people that have been living with Russian influence makes it less likely we’ll be taken by surprise by the fallout from another major event, like Russia losing to Ukraine.

 

Contact Quirk Global Strategies to talk about new and clever ways to measure public opinion in complicated places.

 

 

/by

CATI Experimentation In India

, , , ,

2020 has been something else, hasn’t it?  It’s been discouraging trying to run every project from home — even from the Côte d’Azur. I’ve built my business on my ability to engage personally with people, and the environments that shape their opinions. Without this interaction, I feel like I’ve lost one of the most important tools I use to design insightful research. Couple that with the challenges presented by a near-complete shutdown in quantitative and qualitative data collection in many countries, it’s a challenging time to be a pollster.

Everyone knows lots of things aren’t working. But let’s take a look at what is. The transition to CATI interviewing in some places, sped up by COVID, is a genuine bright spot.

CATI Phone Surveys

If you have survey projects in Western Europe or North America, you’re in great shape. Other than slower turnaround times and increased technical demands placed on data collectors, phone interviewing continues using CATI (Computer-Assisted Telephone Interviewing)  Strategic questions about whether it’s a good idea to poll in such an unpredictable environment remain but there aren’t health concerns or technical barriers to worry about.

It’s more complicated for my clients. I work in low- and middle-income countries where interviewers administer general population surveys face-to-face (F2F). It’s the only way to sample rural, poorer or older populations. COVID ended this type of data collection overnight almost everywhere. Even though the main tool I use to conduct surveys disappeared, my need to understand public opinion did not. Responsible field firms have scrambled to figure out how to stay afloat and design alternative means to get the work done safely.

During the dark depths of France’s total lockdown I spent a lot of time on helpful ESOMAR and WAPOR webinars. Designed to help research professionals figure out how to operate in this new environment, the seminars brought together data collectors from around the world to talk about what was working and what was not.

There was plenty of bad news. Low and poorly-distributed mobile telephone penetration in parts of Africa, Eurasia and Latin America means that nationally representative samples aren’t possible for now in lots of countries.

But some vendors highlighted the progress they’ve made in the transition to CATI in countries where mobile penetration has been growing to the point random sampling is becoming practical. Accordingly, the Big Kids, which include certain departments of USG, Pew and Gallup —  are driving the transition. They have been helping local data collectors build technical capacity. improve quality control and retrain interviewers. This benefits the small fish like me. Thanks guys!

Case Study: India Makes the CATI Transition

Over the past 10 years I have fielded multiple F2F surveys in India. Because of its geographic and demographic complexity, it had always been one of the most difficult countries to randomly sample with anything but a gold-plated budget. Clients with normal budgets have to accept trade-offs during sample design. With a survey scheduled to field in mid-March, my client and I had already come to terms with those trade-offs. I designed a sample that balanced the goals of the research with the confines of budget and time using F2F. Sadly, COVID lockdowns pushed the survey into data collection purgatory, unlikely be released unless I found an alternative to F2F.

Since I last fielded an India survey a few years ago, CATI has become a viable data collection option. Thanks to my Zooming, I learned that mobile penetration is 90%. The darkness of French lockdown lightened.

With the client’s consent, I prepped the survey to switch modes. Most significantly, we translated it into 11 languages, compared with the four or five I’d used in previous face-to-face surveys, which sampled fewer states. After translation backchecks (all of them!) and field testing, fieldwork started.

CATI Interview distribution in India

What a nationally representative sample looks like in India. Beautiful!

To my delight, fieldwork produced a genuinely national sample that includes Indian states proportionate to their contribution to the population as a whole, at only slightly greater cost. Some weighting was applied to balance demographics but nothing alarming.  So, not only has CATI become a viable option in India, it’s actually preferable to the old way. Hey, thanks COVID!

Where Else Is the CATI Transition Happening?

Advances in mobile penetration have made a big difference in sampling complex countries like India, as well as in Indonesia and the South Pacific. In these countries, the distribution of respondents among hundreds of sparsely-populated islands make random sampling using F2F complicated and expensive. I have been doing CATI surveys for internal use with a client’s proprietary software in Bangladesh for the last few years, so I know it’s possible. Professional data collectors are doing so as well. Firms are also conducting promising experiments in Kenya and Nigeria. If you’re curious about other countries, contact me and I can look into options.

Lest my exuberance seem irrational, CATI is off the table in many places in Africa, Eurasia, Latin America and the Middle East. Because they exclude too many important voices, I resist the siren call of online panels, SMS and email surveys. Younger, urban and well-off respondents are easy to sample because they have access to these tools and are comfortable using them. However, such voices are already over-represented in debates over public policy. I won’t let COVID distort the debate even further by opting for samples that by design exclude the rural, the poor, the illiterate and those without adequate bandwidth.

Next week: Online qual: the Bad and the Ugly. Hopefully some good, too.

 

 

 

 

 

/by

Meeting Survey Respondents Where They Are

, , , ,

Election polling in the US has a problem. Or, depending on who you ask, more than one problem.  Since the 2016 election, many big brains have tried to fix these issues with varying degrees of success.

I have problems too, but mine aren’t the same. Pre-COVID, my biggest problems were long fielding times, QC issues and inaccessible sampling points. I don’t have to deal with response rates in the low single digits.

For once this post isn’t about my problems. It is about what I have learned from other folks’ attempts to answer the question: what is the best way to obtain a random, proportionate sample when the tools we’ve always used (for North Americans, telephone surveys) aren’t working very well anymore?

Mixed Mode: Is it the Future?

The answer seems to be some form of “mixed mode” data collection. This means giving respondents several ways to participate in the survey. It could be by telephone, SMS, face-to-face, web, or even snail mail.

I’m not going to get in to the pros (there are some) and the cons (there are many) of each mode, particularly in low- and middle-income countries. Internet, smartphone and literacy distribution vary widely. Although the combination and proportionality are different for every country, mixed mode could mitigate some sampling and data collection problems that pre-date and/or have been exacerbated by COVID. Letting people respond on their terms, rather than mine, seems like an obvious step to take.

Meet Respondents Where They Are

While watching this very interesting University of Chicago panel, it occurred to me that even though I haven’t got the same worries as US election pollsters who can’t get people to answer the phone, I could make my surveys better by providing response options that are more convenient for respondents. These tools exist and COVID has forced us to start using them more!

                      Staying on top of things during the storm is hard

For example, if younger participants prefer online surveys, they should have that option. If it is better to reach older, technophobic or less literate respondents through face-to-face interviewing in their homes, then get out the Kish grid. With careful questionnaire design to mitigate design effects and framing that accounts for selection effects, it is possible to do both in the same survey. Creative use of incentives could also help.

Substituting one mode for another is rarely a solution. I’ve been resisting this approach through this whole COVID year. Running an online-only survey and expecting a representative sample in a country where many people lack smartphones, internet access or have difficulty reading is a big mistake. Younger, better-educated urbanites are already overrepresented in policy discussions. And don’t get me started about panels. Most aren’t, ahem, Pew-level quality.

Shorter Surveys Are Good

There’s more! Shifting away from long and complex face-to-face surveys will force clients to accept shorter questionnaires. As I’ve said before, making busy, less-educated, sometimes food insecure respondents slog through an hour-long survey on constitutional reform does not produce good data. Telephone and online surveys forcing simpler, shorter instruments is an unmitigated good.

COVID has forced us to rethink the ways we’ve always collected data. If one mode appeals more to one group and another group is easier to reach using different mode, why not figure out how to use them both? It’s worth putting in the extra thought. Respondents do us a big favor by answering our questions. It is incumbent on us to make it as easy as possible.

Contact QGS and we can talk about ways to make mixed mode work for your issue and country.

 

/by

Help! I’m underspent! What can I do?

, , ,

What a problem to have! Lately I’ve been talking to program managers who, having no travel expenditures and no in-person programming, marvel at the unusual but serious problem of being underspent.

No one wants to give that money back. No one wants that money back. Are there ways to strategically put it to use to strengthen your current programs, identify gaps and inform future activities? Yes, there are!

Opinion Research: a Strategic Use of Your Underspend

Consider allocating some of it to opinion research.

Being a pollster, I would say that, wouldn’t I? A strategic research program focused on target audiences and program partners can provide insight on the impact of your current work and help you design better future programs.

Specifically, research can:

  • Help with fundraising by revealing new ideas to present to funders;
  • Provide needs assessments for political parties or civil society;
  • Add depth and context to your usual evaluations and indicators;
  • Help you understand how your programs are perceived by participants/local partners and non-participants;
  • Give you a more nuanced understanding of the political and social context in which you are operating;
  • Test collateral materials with key audiences, particularly if you are concerned about sensitivity or unintended responses;
  • Get below the top of mind, rote responses om post-activity evaluation questionnaires;
  • Identify and refine target audiences for programs or outreach;

Online Qualitative Is Cost-Effective For Many Audiences

Online qualitative research, which includes focus groups and in-depth interviews, may cost less than you think (may!).

Focus groups are great for groups of partners or program participants with similar degrees of experience with a topic or a program. With online focus groups, there’s no need to travel or manage complicated logistics to collect the views of participants or partners spread around the country. That’s a huge cost saver. Data collection firms have become adept at managing the fieldwork and technical aspects. If you can get your target audience on a properly organized platform and hire a professional moderator, you can do focus groups.

Online platforms facilitate in-depth interviews with opinion leaders or specialized audiences. These are the types of participants who may not feel comfortable being honest in a group setting, are in competition for your grants, or who have specialized knowledge you want to explore in-depth. You can use your own staff as interviewers or use a professional with the simultaneous translation feature that some platforms make possible.

Some Research Is Almost Always Better Than None

You’ll never hear the argument from QGS that online research is superior to in-person. From a data collection, management and strategic perspective, it’s got a lot of shortcomings. However, you will hear us argue that some research can be better than no research. Even qualitative research that make some concessions on methodology can provide valuable insights that boost the effectiveness of your programs and help you get more funding. (The same is not always true for quantitative). This is especially true with niche audiences, like program participants.

You don’t have to (though you certainly can!) bring in a research consultant to help you manage the implementation. However, having someone experienced who can guide your thinking and design instruments that maximize your resources can ensure your limited funds are spent efficiently. QGS has conducted many of these types of evaluations in a variety of programming contexts. After watching your groups or conducting interviews, we can produce an analytical report with strategic recommendations for improvements or next steps.

We Can Help You Think Through it

Almost all D&G programming benefits from qualitative research in one dimension or another. There’s no excuse not to do it for large programs that are making big investments in communications and targeting. But it can really benefit small programs as well. Take advantage of it! If you are curious about what’s possible, contact QGS. We can give you some ideas.

/by

How to Navigate a Polluted Information Environment

, , , , , , ,

One of my favorite arguments against political opinion research in autocracies has been that surveys measure little more than the degree to which state-driven propaganda has penetrated. When voters hear no opposing views in the media and political opponents are marginalized to the point of irrelevance, public opinion almost always favors the autocrat. How could it not?

Wait—wasn’t the internet supposed to have levelled the field? Weren’t social media and messaging platforms supposed to have given oppositional movements the tools to circumvent state-driven communications and reach voters directly?

That argument hasn’t been viable for at least a decade, if it ever was. Even the dumbest autocrats have proven better able exploit the opportunities presented by social media than their less well-organized and -funded opponents. As tools have become more sophisticated, they’ve evolved.

The Savvy Autocrat Uses Disinformation

The predictability of the autocratic media space has vanished. By economically strangling independent media and cultivating dependent media outlets, autocrats have more tools to influence public opinion at their disposal than they did when they dictated content to newspapers, radio and TV stations directly. The autocrat who can combine a nominally “free” but compliant media and hire savvy social media operators who understand the perverse incentives of audience engagement gets a head start.

                                                   It’s complicated

The modern autocrat now has extraordinary capacity to shape public opinion, often beneath the radar. Politically-aligned and compliant mass media outlets launder and amplify social media-driven fringe themes, targeted to receptive audiences. When they turn their attention to democratic processes and allies, the results can be devastating. Perhaps you’ve heard of “Stop the Steal.” If not, google it. It’s been in the news. It will serve as a textbook example of how to use disinformation to build public support for solutions to problems that don’t exist.

Unlike in the “old days,” the information environment in autocracies is extremely fluid. Within a country, domestic and foreign players can team up to spread mutually beneficial disinformation. Those players may have aligning, but temporary, political and economic interests; alliances can be transactional. Different messages on different topics can be micro-targeted at diverse audiences with no accountability.

Sometimes just a “firehose of BS” is all that’s needed to confuse citizens and encourage them to check out. True, false or contradictory, disinformation exhausts the audience to the point it concludes “everyone is lying, you can’t believe anything these days.”  That’s good enough for those for whom muddied waters count as a victory.

Sink or Swim in Polluted Waters

But let’s say you want to advance a policy, or shore up public support for democratic processes. In a polluted environment, understanding how the public view the issues is important, but it’s only part of the picture. You need also to understand the forces that are shaping their views. This is especially true in backsliding democracies where autocrats benefit from the confused and checked-out population they created.

Here are some key questions to answer and track. What and whose strategic objective is disinformation designed to achieve? What messages are getting traction? Where are they coming from and who are the targets? How are platforms’ punitive actions reshaping the information environment? Governments, businesses and advocacy organizations need to understand this.

“But my topic isn’t terribly controversial or political. Why should I care?” If you are operating in a politicized environment, and your agenda or company has the potential to challenge powerful economic interests, you need to fully understand not only public opinion, but also the potential forces arrayed against you. Knowing this, you can launch your advocacy campaign, voter education program or build your brand knowing in advance where the risks lie and the best way to respond.

Thinking Beyond Traditional Qual and Quant

Strategic disinformation is making the places QGS works in and the topics we work on increasingly complicated. Because we care about actionable data, we’ve been thinking a lot about how to expand the definition of “measuring public opinion” to meet these new realities. Our work on disinformation projects in Ukraine, a testing ground for techniques that are now common everywhere, has taught us a lot about which tool is best suited to answer which question. Because disinformation dynamics are different in every country, there is no one-size-fits-all approach. It’s hard to figure out!

When it comes to measuring the impact of disinformation, traditional qualitative and quantitative are not the only, or the best, tools. It’s a multifaceted problem that requires a multifaceted solution: social media mining, media monitoring, usage studies all help illuminate the sources of disinformation and determine which messages are cutting through and which can be ignored. We can combine these tools to give the fullest picture of your messy information environment.

Contact us and we can help you understand the challenge you’re facing

 

 

/by

QGS Looks Ahead, Not Back

, , , , ,

Looking Forward to 2021. May It Be Better than 2020

Some platform reminded me I started Quirk Global Strategies in Istanbul 14 years ago this week. Looking back, the process that led to that decision could generously be described as “ad hoc.” As it turns out, I didn’t have “a year-long travel shutdown and projects shelved because of a virus” in the threat assessment I didn’t make. While I am thankful that the year wasn’t as catastrophic as it could have been, I hope to rise out of a defensive crouch in 2021.

Above all else, the disruption of 2020 showed me that the hands-on work I do in the field is the foundational value I provide my clients. All my clients benefit from the context I gain in the field and the problem-solving skills I develop. Here are few of the other things I’ve learned.

Polls Still Work

When I hear someone say “polls don’t work,” I look at them with the same degree of respect as I do someone who says “the earth is flat.” Survey research is based on the science of statistics, the principles of which are well-established. Nothing has changed. Furthermore, polls are a tool. You wouldn’t say “hammers don’t work,” unless you’re trying to hammer a screw. In that case you’d be right and you’d be better off trying a screwdriver.

I don’t know if this image belongs to Jason Boxt of 3W Insights, but that’s who I took it from.

For a variety of complicated reasons, probability sampling by telephone has become a less useful tool for some purposes in some places. One purpose that everyone has opinions about is measuring vote intentions of the polarized, diverse US electorate. But that’s not the purpose of most survey research. I have lots of problems with data collection in places I work, but the challenges facing US pollsters are not among them (btw the public pollsters did pretty well in the recent special Senate election in Georgia). Those of us who do strategic research can keep hammering away at nails with reasonable confidence that, as long as we keep a sharp eye on quality control and COVID-related obstacles, our data and the conclusions we draw from it are sound.

Telephone Interviews Continue to Replace Face-to-Face

Interviews once done face-to-face are increasingly being conducted on mobile phones via CATI. In the short run, I still have reservations about the representativeness of phone samples from many Eurasian, African and Middle Eastern countries. However, in the long run, this a positive development. Data collection will be faster, easier to monitor and, once mobile penetration is evenly distributed, might even be more representative than face-to-face samples in some places. Additionally, telephone interviews require shorter, simpler questionnaires. This will result in better quality data. Hearing poorly educated, busy respondents trying their best to power through a 50 minute face-to-face questionnaire on constitutional reform in pre-tests is shame-inducing. With phone interviews, respondents can give their opinion on a bad questionnaire by hanging up. 

Online Qualitative Is Terrible

There, I said it. We can’t just stop doing qualitative research because in-person groups are impossible. Unfortunately, Zoom is the only option we have right now. I’ve learned ways to mitigate its shortcomings. But we shouldn’t for one minute forget what we’re losing when we collect five or six people for a discussion that looks and sounds like Hollywood Squares. So much quality data get left on the table! It actually pains me.

Everyone knows group dynamics can undermine qualitative research. Now that it works differently, I’ve seen how group dynamics can also be the brightest illuminator. I miss it. Zoom participants tend respond to the moderator rather than the person sitting across from them. The result is a lot of “going around the table,” which I hate. I miss seeing reactions of discomfort or alarm or enthusiasm to a fellow participant’s comment, particularly on a sensitive topic. Additionally, being able to google the answer distorts participants’ perceptions, making them appear more engaged or more knowledgeable than they are, especially on politics. Bored online participants have a thousand distractions right in front of them that they don’t have when sitting around a table together. Shorter groups with fewer participants mean less time for depth and breadth, less opportunity for that lightning bolt to strike when an engaged group generates useful ideas.

I miss out, too, as an analyst. Of course it’s possible to listen to groups via a video or audio link. I’ve done it many times when it’s too dangerous to travel or when observer facilities are improvised. But I miss the dynamic of the “back room” where I can check and crosscheck ideas with my local colleagues and moderators. I miss hearing them react to participants’ thoughts and the context they give. They make me a better analyst and project designer.

In-depth Interviews Work Well Online

Here’s one positive thing I’ve learned. Online platforms like Zoom facilitate near-simultaneous translation. This has made it easier for me to personally conduct in-depth interviews with non-English speakers. I wouldn’t do it for all projects. But for program assessments or KIIs, I can dig more deeply and formulate follow-ups that synthesize what I’ve heard respondents tell me. It’s also fun to interact with new and interesting people while I am isolated at home.

Relationships Matter

I recognize that waiting out a pandemic in the Côte d’Azur does not engender much sympathy

I received an email from a colleague in Georgia thanking me for a referral. He lamented the current situation and let me know that his firm has managed to hang on. I appreciated hearing the update. Having spent 14 years building these relationships, I miss them. I’ve skipped my usual trips to Washington, New York and London. The blocks of time I would have spent in Ukraine, Bangladesh, Iraq or Turkey I’ve instead spent in my apartment, staring at my monitor. The isolation has been one of the hardest parts of the last year. My clients and partners are my friends and are the best part of my job. Listening, having meals and drinks and sharing long car rides with them ground my insights in real life. Zoom can’t replace this loss.

As soon as I am vaccinated, I’ll get back on the horse. I’m looking forward to seeing as many of you out there as possible. Contact me if you’re looking for help navigating this complicated new world. Wishing everyone a more engaging, more productive 2021.

 

 

 

 

/by

Doing the Best We Can with Online Focus Groups

, , , , ,

Six months into socially distant qualitative research and what have we learned? We’ve learned it’s hard. Online focus groups have a lot of flaws, many of which cannot be mitigated, especially in lower- and middle-income countries. But, like a lot of annoying things these days, they aren’t going away. We have to figure out how to collect the best data we can from them.

The default approach cannot be to transfer in-person groups to an online platform. Here are some useful questions to ask during the project design phase

  • How will the quality of the data collected about this topic be affected by the limitations of the platform?
  • Who is being excluded from this project because they lack the technological capacity to participate? How will that affect the strategy that will be based on the research?
  • Can we collect the data we need in the amount of time we have, given shorter and smaller groups?
  • How can we take advantage of the fact that, logistically speaking, geography doesn’t matter anymore?

I’ve learned these lessons in the last few months.

Groups Must Be Shorter

Lengthy Zoom interactions are known to be enervating for everyone. It is much easier for online participants to become distracted, lose the thread of the conversation or leave the group entirely. Disengaged, bored participants – already a problem with many public policy topics — result in poor data. Making things worse, online moderators lack many of the usual tools they use to engage in-person participants.

Groups Must Be Smaller

Think of how you behave on a big zoom call. Can you keep track of who is saying what every minute? Can you listen deeply without distraction so you can probe meaningfully on what that person is trying to communicate without the benefit of body language? Do you tune out and check twitter? Limiting groups to five to seven participants makes life easier for moderators and less boring for participants who have to wait their turns to speak.

Know Who is Excluded

Online groups require participants to have a broadband or 4G internet connection, probably a tablet or monitor and the savvy to know how to use them. Many people in lots of countries don’t have these. Know who your platform excludes, then decide if that’s acceptable according to the goals of the project. Younger, wealthier, urban and tech savvy voters already tend to be overrepresented in public policy discussions. Platforms that by design exclude their older, rural and less connected counterparts exacerbate this dynamic. This is less of a concern in high- and middle-income countries. It is a serious concern in low-income countries or those where digital access is unevenly distributed.

A Quality Recruit Is More Important than Ever

With a smaller group, each voice carries greater weight. That’s why everyone has to be qualified. Spend more time on screener design and prescreening to ensure every participant is the exact person you need.

Prepare for Technical Problems

Resolving technical issues takes up valuable time, distracts participants and the moderator and disrupts the flow of the discussion. Be prepared to redo groups plagued by technical problems. 

The Platform Exacerbates Moderator Flaws

Moderators who are poor at fostering a group dynamic or controlling dominators in person will struggle to do so online. If you rely on the same moderators, investing in a course for them to update their skills could benefit them and you.

Topics that Are Illuminated by Group Dynamics Are Less Well Suited to Online

Idea generation, group dynamics and evolution of opinion as respondents learn from each other are what make in-person discussions so valuable for developing insightful strategies. Online participants are more likely act as disconnected individuals responding to moderator-presented stimuli rather than a group that responds to ideas and thoughts of other participants. For a researcher who works on sensitive political and social topics, this is a huge loss. The most revealing groups are the ones where participants struggle with a complex topic and engage actively with prompts from the moderator and the views of their fellow participants.

The “Back Room” Doesn’t Exist

It’s more difficult to guide the reactions of inexperienced observers. It’s also harder to seek clarifications, answer immediate questions or make alterations on the fly among staff. I miss hearing nationals’ reactions to the discussion and benefitting from their insights while watching online groups. Their insights improve my analyses.

Time and Space Don’t Matter

If your moderators, field team and observers don’t have to travel, you have more flexibility on logistics, scheduling and geography. You also don’t have to coordinate and travel to remote regions to hear rural perspectives. Even groups that must be held in a specific area can be more geographically diverse when travel times, public transport and traffic considerations don’t matter. Homebound respondents might have more flexibility, allowing for a broader range of times for scheduling groups.

Have concerns about conducting online groups? Wonder if they are appropriate for your project and country? Contact QGS. We can help.

We can't just move in-person groups online

                            We can’t just move in-person groups online

 

/by

Worried About Qualitative Research During Coronavirus? I am

, , , , ,

With traditional focus groups off the table during physical distancing, qualitative researchers face options that may require too many tradeoffs to make the effort worth the costs. The challenges for qualitative research could be even greater than those facing quant, particularly for social or opinion research projects.

If you’re interested in challenges facing quantitative researchers, check out my last two posts.

What Are My Options?

If your qualitative needs can be met with chatroom-style online focus groups, you’re in good shape. It continues to be a solid technique for ad and message testing with larger audiences and for mixing qual and quant methodologies. Recruiters report that homebound respondents have the time and inclination to participate in a variety of formats. This is good news.

No FGDs for now, whether in a room or under a tree

Focus group discussions present a greater challenge however. The most valuable part of a focus group is the personal interaction between participants that an engaged moderator can elicit through thoughtful probing and skillful management of group dynamics. If the success of your project depends on getting beyond top-of mind responses, generating new ideas, and exploring feelings about abstract concepts, remote in-person focus groups on Zoom or other proprietary platform fall short. I worry that so much illuminating data will be distorted or lost that it’s worth asking “what’s the point?”

Do You Enjoy Zoom Meetings? I Don’t

Think of your last Zoom(s) with eight friends or colleagues. Where you satisfied with the quality of interaction with others on the call? Did you enjoy it? How engaged were you with the topic? After more than two months of physical distancing, Zoom interactions are proving to be, at best, unsatisfying and at worst, exhausting. There are good reasons for this.

Axios technology editor Scott Rosenberg articulated my misgivings about video-conferencing as a qualitative tool.  Here’s what he writes:

Videoconferencing imposes cognitive and psychological frictions and aggravates social anxieties. As experts in human-computer interaction point out, using Zoom means putting on a show for others without being able to rely on the cues we primates depend on in physical encounters.

  • There’s usually a slight audio lag, as well as mute-button mistakes and “your internet connection is unstable”-style dropouts.
  • We’re also often opening a chunk of our homes for others to view, and that can trigger social worries.
  • By showing us our own image as well as others’, Zoom ensures that we will critique ourselves in real time.
  • On top of standard-grade performance anxiety, the “big face” image that Zoom uses by default in its “speaker view” can trigger a “fight-or-flight” surge of adrenaline, writes Jeremy Bailenson, founding director of Stanford’s Human Computer Interaction Lab.
  • If you switch to the “Hollywood Squares”-style “gallery view,” you’re confronted with a sea of separated faces, which is not how evolution has adapted us to group interactions.
  • As M. Sacacas observes,you can’t really achieve true eye contact with anyone: If you look right into someone else’s eyes, you will appear to them as if you aren’t looking right at them — to achieve that, you have to look right at the camera.
  • Nonetheless, the whole experience of a videoconference feels to us like an extended bout of direct staring at other people staring back at us. That’s draining, which is why it’s not what actually happens when we meet in person, where we only occasionally look right at one another.

How will participants react when deprived the visual and social cues that help them interpret the reactions of others? Will feeling uncertain, frustrated, insecure, or tired influence their opinions, particularly on sensitive or political topics? How will these dynamics shape strategy built on the data? Additionally, Rosenberg’s analysis is helpful for contextualizing the reactions of citizens of high-income countries who are comfortable interacting with technology. How will citizens of low- or medium-income countries, or those with different cultural expectations for social interaction, respond? I don’t know if we know the answers to these questions.

Traditional focus groups have many biases, all of which are well-known and factored into qualitative analysis. We know people respond to questions differently in a group setting than they would alone. Online video-conferencing multiplies these biases plus adds others that we’ve barely begun to understand. Not only are there different biases to adapt to, the medium itself diminishes the greatest advantage of a focus group: group/moderator interaction.

 So What Can We Do?

Because qualitative research can’t grind to a halt because of physical distancing, we have to come up with ways to mitigate these problems. Here are some suggestions.

  • Assess whether videoconferencing is the right tool for the job. Does the topic demand strong rapport among moderator and participants to explore controversial social topics ? If so, your findings may lack depth, mislead or entirely miss important findings.
  • Write a shorter, simpler moderator’s guide. This medium is not appropriate for a two-hour long discussion on constitutional reforms.
  • Recruit fewer participants. Aim for quality of responses, rather than quantity.
  • Use a moderator well-trained and experienced in managing the complicated dynamics of this type of discussion. A high energy, animated moderator might be better able to engage remote participants than one with a more low-key personality.
  • Manage client expectations. Video-conference groups and in-person groups are not interchangeable. They will provide different kinds of data. Clients who expect traditional focus group data are likely to be disappointed,
  • To avoid the impulse to compare data collected in a traditional focus group and that from a video-conference, start fresh with a new project.
  • Be highly cognizant of data privacy protocols. Compliance with privacy laws apply more stringently to shared and stored video.

It’s going to be hard to convince me that video-conference focus groups, despite being absolutely possible, are advisable for all projects. This is particularly true for social and opinion research projects. All researchers have to adapt to our changing environment. The first question before launching a project should always be “what is my research goal and what tools can I use to reach it?” Then you can decide if having some data is better than having no data. Contact Quirk Global Strategies and we can help you decide.

 

/by