The changing world of market research

While the players and technology have evolved and the scale of data has increased exponentially, valuable insights still require strategic thinking by humans

By Mike Jewell

“He who asks is a fool for five minutes.
He who does not ask is a fool forever.”

Chinese proverb

Market research in Taiwan goes back to about 1980, but organised market research really began in the early 1900s in the US, when ad agencies and marketing departments developed basic quantitative and qualitative research methods, recognising that knowledge of consumer preferences was key in increasingly competitive markets.

In the 1920s, major ad agencies were formally investigating how and why consumers used brands, so that they could recommend appropriate advertising copy to manufacturers and Arthur Nielsen was pioneering the measurement of radio audiences and subsequently TV viewing. Meanwhile, Daniel Starch was stopping Americans on the street to ask if they had read certain publications and if they remembered any of the adverts, as a way of assessing advertising effectiveness. Surveys were born.

Those early surveys have now been replicated in some form in almost every sector of activity in scores of countries around the world. Uncovering what people think, believe, want and need and interpreting this information is fundamental in guiding brands, organisations and even governments when making decisions.

“Public opinion shapes our destinies and guides the progress of human affairs.”

Frank B. Kellogg

According to ESOMAR, the worldwide association of the data, research and insights community, survey research is a US$41 billion business and the total insights market, including analytics, is worth US$80 billion. In Taiwan, the sector has evolved rapidly since I joined Frank Small & Associates in 1986 and today is a mature industry, worth around US$100 million annually.

The infrastructure is a mix of international research giants, regional players, local boutique agencies, academic institutions and specialist service providers taking care of a diverse client base of multinational and local corporations across a spectrum of business sectors, such as household goods, food and beverages, cosmetics, automobiles, pharmaceuticals, retailing, hospitality, telecommunications, financial services, technology, media and many more. In addition, central and local government, NGO’s, think tanks and political parties are constantly surveying the mood of the population to guide their policy making.

Since 2000, there has been considerable M&A activity, as Research International, Synovate, NFO and Millward Brown have been merged into larger groups and four global agencies dominate the scene – Nielsen, Kantar and Ipsos and IQVIA, the leader in the specialist pharma sector, with a cumulative 60% market share.

There has been minimal growth in spending on survey research in recent years, partly because of the sluggishness of the economy, but also due to a switch in investment focus by major multinationals into the ever-growing China market, with cheaper adapted rather than homegrown marketing solutions being preferred for Taiwan.

Despite this lack of growth, which developed markets worldwide are experiencing, market research is going through a period of rapid and fundamental change, more dramatic than at any time in its hundred-year history.

The internet, smartphones and Big Data
 

Technology has revolutionised market research in the last decade. It has changed people, clients’ expectations, research methods and analysis, and it has given us big data.

The idea of accessing and storing large amounts of information for analytics has been around a long time, but it gained real momentum in the early 2000s, when we all started living our lives through smartphones and organisations woke up to the possibilities of harnessing it for their benefit.

Global devotion to mobiles and the internet is mind-boggling.

• 4.6 billion people spent a total of 1.2 billion years looking at the internet in 2019
• 45% of the planet’s population own a smartphone
• 3.8 billion people use social media and 2.5 billion are locked into Facebook
• In 2017 the average number of social media accounts was 7 per person and one third of daily internet browsing time is devoted to social media
• In Taiwan 88% of the population are active social media users…
• …and 19 million of us spend almost 4 hours a day on the internet on our mobiles, and then another 4 hours via other devices.

Nowadays, almost every action leaves a digital trail. We generate data whenever we go online, when we carry our GPS-equipped smartphones, when we communicate with our friends through social media or chat apps, and when we shop. We leave digital footprints with every digital action and all this activity generates staggering amounts of data. Every two days we create as much new data as we did from the beginning of time up until the year 2000.

• 40,000 searches are launched on Google every second
• Every minute 300 hours of new video show up on YouTube
• Within that same minute, users send 31 million messages and view 2.7 million videos on Facebook, despatch over half a million tweets, share 2 million photos on Snapchat and make almost 1 million Tinder swipes
• This year the total of amount of data in existence will reach 40 zettabytes (1 zettabyte = 1,000,000,000,000,000,000,000 bytes) and we are producing an additional 2.3 zettabytes every day.

Companies are enthusiastically embracing big data and investment in analytics is growing rapidly, at around 10% a year, in sharp contrast to regular survey spending.

An outstanding example of the value of big data analytics is Taiwan’s response to the Covid-19 pandemic. As soon as news of the virus broke, the authorities started integrating data from two disparate sources, the immigration and customs database and the national insurance database, to create a unified big data platform. Through this they successfully identified coronavirus cases based on clinical visits, travel history, and symptom patterns to provide real-time alerts to targeted population segments about possible infected persons. The well-thought out, unified post-SARS organisational and command structure was able to combine this analysis with the rapid introduction of coherent action plans, to mobilise a properly prepared and well-resourced health infrastructure and engage the willing participation of the populace to deal with the pandemic more effectively than almost any other country.

In the commercial environment, the full value of big data is taking longer to realise. A major problem with so much data is working out what to use and how. IT market intelligence specialist IDC has estimated that only 22% of data has the potential to be of any use to businesses, and that currently only 0.5% is actually being analysed.

In Taiwan, interest in data analytics is no less keen, although the level of investment has not yet caught up with traditional survey spending, as companies wrestle with how to weave all the different pieces of knowledge into a single stream of actionable recommendations. Nevertheless, they recognise there can be immense benefit in synthesising all available information sources, beyond simply relying on survey findings.

“The environment is changing lots, because of digitisation, new technology and so on, so we’re thinking if there are better ways to collect insights from consumers. Let’s not just limit ourselves to those traditional research methods. How can we leverage data from different sources and deliver valuable insights to drive our business decisions, product innovation and even channel strategy?”

Taiwan insurance company

This broadening of focus is changing the research landscape. Increasingly, buyers are seeing research agencies as consultants, specialists in data interpretation, able to separate the meaningful 22% of data from the useless 78% and integrate both secondary information (known previously as “desk research”) with conventional survey output. At the same time, data and analytics companies with tech-enabled systems and automation are muscling in on the research agencies’ turf and providing organisations with key patterns, trends, and associations to guide their strategy.

The coronavirus pandemic again provides a good illustration of how monitoring behaviour through our digital footprints provides valuable guidance about changing habits under the abnormal conditions. Apple, Google and Moovit have made mobility data from transport apps freely available, to help authorities understand if people have heeded government advice to go out less. Hundreds of thousands of data points give a much more robust picture of public behaviour than could be obtained through surveys.

In Taiwan Nielsen, Kantar, Ipsos and others are competing against new analytic tools from Google and Facebook in particular, which help track user profiles and advertising response. However, there is still a vital role for research, since analytics only show “what” people do but not “why”. Research can go much deeper.

The leading research agencies have been criticised for reacting slowly to the competition from data analytics companies, as Didier Truchot, the global CEO of Ipsos recently acknowledged.

“We weren’t positioned in the most dynamic segment of the market. Using secondary data wasn’t well developed in the large international companies, which tend to dedicate their coverage to sell traditional market research projects. The established companies have been very conservative.”

The agencies counter this criticism by pointing to the efforts they are (now) making by utilising their expertise in analysing data so clients can extract the maximum value from across all their sources of information, internal and external, not just from survey results.

Alan Chien, MD of Ipsos Taiwan observed, “It’s not research any more for most of the clients. It’s more about information, or so-called business intelligence. They don’t treat market research as such important information as ten years ago. They have more partners now able to offer more information in different formats. For the last couple of years our strategy is to go beyond traditional market research, i.e. surveys. A lot of clients have their own big data analysis teams, trying to integrate their internal data, but they are struggling, so we’re helping them, coaching them how to pull out a portion of the data and demonstrating how to analyse that.”

There is also intense focus on harnessing consumer sentiment from social media, and several local companies have sprung up, such as i-Buzz and e-Land, offering “social listening” analysis, alongside more established firms like Eastern Online’s research centre and Ipsos’ international partner, Synthesio.

One local consumer goods manufacturer said they are switching budget away from surveys because “we believe that, from consumers’ daily conversations, we can find some clues to develop our new products.”

Social listening offers a massive pool of consumer feedback to draw on in almost real time, but the sheer number of comments can be overwhelming. Focused interrogation by experienced analysts, working within precisely defined parameters is essential if any sense is to be made of the data. A general search for a popular consumer brand will generate millions of mentions, but the vast majority (90%+) will be irrelevant.

Yvonne Wang, CEO of Kantar Insights discussed this trend. “Many clients ask if they can leverage big data, not just traditional marketing research. They even think of social listening replacing survey data. However, depending on their objectives, sometimes you can’t just use social listening as there’s too much, both good and bad which might lead to false conclusions. One client tried to do it themselves but the outcome was shallow and they realised that a trusted middle agency is critical to deliver the real insights. It really depends on your objectives. If you want to get at shoppers’ needs or preferences or decide what product to launch, I won’t recommend this way. If you just want to see if a campaign has some impact or how people talk about it, maybe we can use social listening data but still better for the marketing research consultant who understands the market and category to manage it. Clients want to leverage big data, but it can’t replace market research. It can help give a deeper or broader viewpoint, but you can’t just use big data to make a conclusion. It’s too risky. Clients still rely on research agencies for insights, because we know how to screen and interpret data in the context of the business questions. What we need to do is sharpen our consulting skills. Maybe in the past we just provided key findings and recommendations from survey results. Now clients expect us to be their consultants. That’s what we’re transforming into and offering more comprehensive insights by incorporating multiple data sources.”