The changing world of market research
While the players and technology have evolved and the scale of data has increased exponentially, valuable insights still require strategic thinking by humans
By Mike Jewell
“He who asks is a fool for five minutes.
He who does not ask is a fool forever.”
Market research in Taiwan goes back to about 1980, but organised market research really began in the early 1900s in the US, when ad agencies and marketing departments developed basic quantitative and qualitative research methods, recognising that knowledge of consumer preferences was key in increasingly competitive markets.
In the 1920s, major ad agencies were formally investigating how and why consumers used brands, so that they could recommend appropriate advertising copy to manufacturers and Arthur Nielsen was pioneering the measurement of radio audiences and subsequently TV viewing. Meanwhile, Daniel Starch was stopping Americans on the street to ask if they had read certain publications and if they remembered any of the adverts, as a way of assessing advertising effectiveness. Surveys were born.
Those early surveys have now been replicated in some form in almost every sector of activity in scores of countries around the world. Uncovering what people think, believe, want and need and interpreting this information is fundamental in guiding brands, organisations and even governments when making decisions.
“Public opinion shapes our destinies and guides the progress of human affairs.”
Frank B. Kellogg
According to ESOMAR, the worldwide association of the data, research and insights community, survey research is a US$41 billion business and the total insights market, including analytics, is worth US$80 billion. In Taiwan, the sector has evolved rapidly since I joined Frank Small & Associates in 1986 and today is a mature industry, worth around US$100 million annually.
The infrastructure is a mix of international research giants, regional players, local boutique agencies, academic institutions and specialist service providers taking care of a diverse client base of multinational and local corporations across a spectrum of business sectors, such as household goods, food and beverages, cosmetics, automobiles, pharmaceuticals, retailing, hospitality, telecommunications, financial services, technology, media and many more. In addition, central and local government, NGO’s, think tanks and political parties are constantly surveying the mood of the population to guide their policy making.
Since 2000, there has been considerable M&A activity, as Research International, Synovate, NFO and Millward Brown have been merged into larger groups and four global agencies dominate the scene – Nielsen, Kantar and Ipsos and IQVIA, the leader in the specialist pharma sector, with a cumulative 60% market share.
There has been minimal growth in spending on survey research in recent years, partly because of the sluggishness of the economy, but also due to a switch in investment focus by major multinationals into the ever-growing China market, with cheaper adapted rather than homegrown marketing solutions being preferred for Taiwan.
Despite this lack of growth, which developed markets worldwide are experiencing, market research is going through a period of rapid and fundamental change, more dramatic than at any time in its hundred-year history.
The internet, smartphones and Big Data
Technology has revolutionised market research in the last decade. It has changed people, clients’ expectations, research methods and analysis, and it has given us big data.
The idea of accessing and storing large amounts of information for analytics has been around a long time, but it gained real momentum in the early 2000s, when we all started living our lives through smartphones and organisations woke up to the possibilities of harnessing it for their benefit.
Global devotion to mobiles and the internet is mind-boggling.
- 4.6 billion people spent a total of 1.2 billion years looking at the internet in 2019
- 45% of the planet’s population own a smartphone
- 3.8 billion people use social media and 2.5 billion are locked into Facebook
- In 2017 the average number of social media accounts was 7 per person and one third of daily internet browsing time is devoted to social media
- In Taiwan 88% of the population are active social media users…
- …and 19 million of us spend almost 4 hours a day on the internet on our mobiles, and then another 4 hours via other devices.
Nowadays, almost every action leaves a digital trail. We generate data whenever we go online, when we carry our GPS-equipped smartphones, when we communicate with our friends through social media or chat apps, and when we shop. We leave digital footprints with every digital action and all this activity generates staggering amounts of data. Every two days we create as much new data as we did from the beginning of time up until the year 2000.
- 40,000 searches are launched on Google every second
- Every minute 300 hours of new video show up on YouTube
- Within that same minute, users send 31 million messages and view 2.7 million videos on Facebook, despatch over half a million tweets, share 2 million photos on Snapchat and make almost 1 million Tinder swipes
- This year the total of amount of data in existence will reach 40 zettabytes (1 zettabyte = 1,000,000,000,000,000,000,000 bytes) and we are producing an additional 2.3 zettabytes every day.
Companies are enthusiastically embracing big data and investment in analytics is growing rapidly, at around 10% a year, in sharp contrast to regular survey spending.
An outstanding example of the value of big data analytics is Taiwan’s response to the Covid-19 pandemic. As soon as news of the virus broke, the authorities started integrating data from two disparate sources, the immigration and customs database and the national insurance database, to create a unified big data platform. Through this they successfully identified coronavirus cases based on clinical visits, travel history, and symptom patterns to provide real-time alerts to targeted population segments about possible infected persons. The well-thought out, unified post-SARS organisational and command structure was able to combine this analysis with the rapid introduction of coherent action plans, to mobilise a properly prepared and well-resourced health infrastructure and engage the willing participation of the populace to deal with the pandemic more effectively than almost any other country.
In the commercial environment, the full value of big data is taking longer to realise. A major problem with so much data is working out what to use and how. IT market intelligence specialist IDC has estimated that only 22% of data has the potential to be of any use to businesses, and that currently only 0.5% is actually being analysed.
In Taiwan, interest in data analytics is no less keen, although the level of investment has not yet caught up with traditional survey spending, as companies wrestle with how to weave all the different pieces of knowledge into a single stream of actionable recommendations. Nevertheless, they recognise there can be immense benefit in synthesising all available information sources, beyond simply relying on survey findings.
“The environment is changing lots, because of digitisation, new technology and so on, so we’re thinking if there are better ways to collect insights from consumers. Let’s not just limit ourselves to those traditional research methods. How can we leverage data from different sources and deliver valuable insights to drive our business decisions, product innovation and even channel strategy?”
Taiwan insurance company
This broadening of focus is changing the research landscape. Increasingly, buyers are seeing research agencies as consultants, specialists in data interpretation, able to separate the meaningful 22% of data from the useless 78% and integrate both secondary information (known previously as “desk research”) with conventional survey output. At the same time, data and analytics companies with tech-enabled systems and automation are muscling in on the research agencies’ turf and providing organisations with key patterns, trends, and associations to guide their strategy.
The coronavirus pandemic again provides a good illustration of how monitoring behaviour through our digital footprints provides valuable guidance about changing habits under the abnormal conditions. Apple, Google and Moovit have made mobility data from transport apps freely available, to help authorities understand if people have heeded government advice to go out less. Hundreds of thousands of data points give a much more robust picture of public behaviour than could be obtained through surveys.
In Taiwan Nielsen, Kantar, Ipsos and others are competing against new analytic tools from Google and Facebook in particular, which help track user profiles and advertising response. However, there is still a vital role for research, since analytics only show “what” people do but not “why”. Research can go much deeper.
The leading research agencies have been criticised for reacting slowly to the competition from data analytics companies, as Didier Truchot, the global CEO of Ipsos recently acknowledged.
“We weren’t positioned in the most dynamic segment of the market. Using secondary data wasn’t well developed in the large international companies, which tend to dedicate their coverage to sell traditional market research projects. The established companies have been very conservative.”
The agencies counter this criticism by pointing to the efforts they are (now) making by utilising their expertise in analysing data so clients can extract the maximum value from across all their sources of information, internal and external, not just from survey results.
Alan Chien, MD of Ipsos Taiwan observed, “It’s not research any more for most of the clients. It’s more about information, or so-called business intelligence. They don’t treat market research as such important information as ten years ago. They have more partners now able to offer more information in different formats. For the last couple of years our strategy is to go beyond traditional market research, i.e. surveys. A lot of clients have their own big data analysis teams, trying to integrate their internal data, but they are struggling, so we’re helping them, coaching them how to pull out a portion of the data and demonstrating how to analyse that.”
There is also intense focus on harnessing consumer sentiment from social media, and several local companies have sprung up, such as i-Buzz and e-Land, offering “social listening” analysis, alongside more established firms like Eastern Online’s research centre and Ipsos’ international partner, Synthesio.
One local consumer goods manufacturer said they are switching budget away from surveys because “we believe that, from consumers’ daily conversations, we can find some clues to develop our new products.”
Social listening offers a massive pool of consumer feedback to draw on in almost real time, but the sheer number of comments can be overwhelming. Focused interrogation by experienced analysts, working within precisely defined parameters is essential if any sense is to be made of the data. A general search for a popular consumer brand will generate millions of mentions, but the vast majority (90%+) will be irrelevant.
Yvonne Wang, CEO of Kantar Insights discussed this trend. “Many clients ask if they can leverage big data, not just traditional marketing research. They even think of social listening replacing survey data. However, depending on their objectives, sometimes you can’t just use social listening as there’s too much, both good and bad which might lead to false conclusions. One client tried to do it themselves but the outcome was shallow and they realised that a trusted middle agency is critical to deliver the real insights. It really depends on your objectives. If you want to get at shoppers’ needs or preferences or decide what product to launch, I won’t recommend this way. If you just want to see if a campaign has some impact or how people talk about it, maybe we can use social listening data but still better for the marketing research consultant who understands the market and category to manage it. Clients want to leverage big data, but it can’t replace market research. It can help give a deeper or broader viewpoint, but you can’t just use big data to make a conclusion. It’s too risky. Clients still rely on research agencies for insights, because we know how to screen and interpret data in the context of the business questions. What we need to do is sharpen our consulting skills. Maybe in the past we just provided key findings and recommendations from survey results. Now clients expect us to be their consultants. That’s what we’re transforming into and offering more comprehensive insights by incorporating multiple data sources.”
“Speed is VERY, VERY important”
There is immense pressure for the insights and intelligence that drive growth to be delivered at the ever-increasing speed of business decision making. Agencies are expected to compress research timings to match the more agile tech start-ups. The trend has been towards gathering fast responses to a limited number of questions, so that decisions can be made swiftly based on immediate feedback, rather than waiting weeks, even months for detailed, well-crafted analysis from carefully designed surveys.
The availability of user-friendly DIY survey tools like SurveyMonkey, Google Surveys, Qualtrics, Surveycake and so on is making the landscape even more crowded, as consulting firms turn to these no frills solutions for quick responses to inform their consultancy, and these platforms even give clients the opportunity to create their own surveys, bypassing the traditional agencies.
Research firms are recognising this new reality. In some cases, they are developing their own DIY solutions or partnering with tech-based providers; in other cases, they are taking clients’ DIY data and integrating it into broader analyses.
One senior agency researcher explained: “Clients are getting more impatient, because they can’t wait anymore, as decision making has been speeded up to support the business. Often they don’t need sophisticated information, so they’re experimenting with DIY solutions. It may not be perfect, but it’s good enough for a fast decision. It’s a trade-off. If it’s something strategic and really critical, they’ll still opt for the safety of a properly designed survey; but for low risk decisions, they’ll choose the DIY option.”
Technology is changing people
Apparently, we check our phones every 12 minutes for news, emails, social media updates and communications from friends, but we are increasingly resistant to phone calls, especially as there are so many unwanted intrusions from telemarketers and, worse, scammers. The Pew Research Center in the US has seen response rates for phone surveys plummet from 36% in 1997 to just 6%.
Research suppliers are actively promoting switching to online surveys and the adoption of “mobile first” design to improve response and data quality. 45% of surveys globally in 2018 were completed online (ESOMAR), while Ipsos Taiwan reports that over 25% of online surveys are answered on mobiles and that figure is growing.
But the smartphone world demands a radically different approach – a 10-minute survey is about as much as we can tolerate, for, although the amount of time we spend on our mobiles is increasing, our concentration per screen is diminishing.
This is a difficult transition, since it often involves wholesale re-engineering of long-established methodologies, which clients are reluctant to change, fearing a loss of continuity. However, these surveys often use over-long questionnaires that are not even suitable for desktops, so research agencies are working with their clients to manage this necessary change.
“Mobile first research is more than making a survey available online and accessible by any device. Mobile first research is designed to be conducted on a mobile phone. This means that the look, feel and usability of the survey will be best suited to a mobile phone. If it works on the smallest of screens, it will also work on the largest. Research should be designed as mobile first so that respondents can easily and comfortably complete surveys on the device of their choice. This means aligning survey design with how people use their mobile phones in day-to-day life. Mobile first research should be short, simple and direct. By putting consumers and their smartphones at the heart of survey design, it is possible to provide quality insights with greater ease and speed.”
Ipsos: Mobile First Survey Design
Kantar’s research-on-research indicates that mobile first design reduces survey dropout rates by 40%+.
“Traditional research gives traditional answers”
The industry is continually looking to overcome the shortcomings of established methodologies, turning increasingly to behavioural science for answers.
Big data shows patterns and trends, but it cannot show companies what frustrated their customers, what they were looking for, but could not find, or why they ultimately decided to go with one choice over another.
Unfortunately, standard question-and-answer methodologies have also posed problems for researchers for many years.
“The shift towards behavioural analysis is a long-standing trend, because we don’t trust what people say they’ll do. We want to know what people are going to do, so increasingly we look at their behaviour. Behaviour leads attitude change.”
Mark Earls, marketing consultant and writer.
All the leading agencies are investing in creating expert hubs to supply specialist knowledge to their op-cos. At the same time, boutique agencies specialising in qualitative research are starting up, adapting their skills to the individual needs of a few key clients. They are more willing to try new approaches, although Taiwan buyers still feel there is a lack of first world innovation in the techniques being introduced. Innovation is a two-way street, however, and untested methodologies are high risk, when budgets are under pressure, so many clients are unwilling to invest without the reassurance of knowing what their return will be.
Perhaps the most important evolution in present day thought is the recognition that consumers do not exist in a vacuum. It is just as important to understand consumers’ context as it is to study their choices and behaviour. Traditional focus groups, with their inherent flaws, are being replaced by extensive use of observation, ethnography and neuroscience approaches such as eye tracking and facial coding.
New conversational insight solutions, using smartphone technology, can capture behaviour and opinions in real time, while also adding context with videos, images and emojis, which are often richer, more compelling, and higher fidelity than traditional research output.
Ipsos highlights a range of information from mobile first surveys that provide further guidance for decision making.
- Geo-location data which can:
- track peoples’ journeys (e.g. how far they travel, how they travel, locations of interest)
- trigger studies when close to important locations (e.g. retail locations)
- support media studies for Out of Home advertising.
- Image, video and audio capture to add richer insights than traditional open ends
- Voice and chat surveys to enable hands-free survey responses. This can be especially valuable for target audiences like mothers of babies and can make longitudinal studies (like diaries) much easier for people to complete.
- Passive metering enables the monitoring of mobile behaviour, providing information on how to improve digital experiences.
- Near Field Communications (the technology behind Apple and Google Pay) can be leveraged for ‘in-the-moment’ research simply and effectively.”
What is the future for insights?
The future of the insights business is going to be focused on becoming faster, more efficient, and more scalable. It will be a partnership between machine learning, AI processing and human perception, with algorithms taking over routine tasks and researchers applying their experience to selecting and incorporating the right data and formulating specific recommendations from broad insights.
The biggest opportunity is to continue to integrate technology with the strategic thinking of humans. The need for deeper qualitative analysis and consultant skills will be more important than ever.