One of the greatest challenges in this industry is to fully understand what constitutes realistic expectations. In a data-driven world, the risk of data-less claims about sample quality, survey processes, and incidences of fraud can be dangerous for your research and ultimately your bottom-line.
I was recently referred to a blog post that presented desktop sample as the black sheep of research while falsely placing mobile app solutions on an impervious pedestal. For this reason, it's important to take a closer look at claims about mobile panel and its relation to desktop-based sample (in terms of quality). I have long been a massive advocate for a mobile-first solution. In fact, I built the industry’s first app-based panel solution at Instant.ly in 2010 and conducted numerous ROR initiatives proving the merit of mobile research. I get on my mobile soap box whenever the opportunity presents itself.
The truth is, many traditional sample suppliers have both desktop and mobile sample, so it's disingenuous to compare the two as separate entities operating in a vacuum. According to The Pew Research Center, mobile penetration rates are nearly 80 percent in the US; the typical “desktop” panel sees daily mobile traffic as high as 50 percent!
So, what am I getting at here? In a nutshell, I know mobile research. I know what it is and I know what it is not. Which is why, I felt the blog post I referenced earlier was a mischaracterization of our ongoing dialogue and thought leadership on quality issues in our industry. It is disingenuous to make claims that mobile app sample is the only solution. Fear-mongering buyers into your methodology by making false claims and twisting context is a dangerous strategy.
Instead, I recommend using consumer data and ROR benchmarks to demonstrate the merits of your methodology, while also sharing potential pitfalls or short-comings that should be considered for thoughtful adoption.
We all must drink the “water” so to speak; it’s up to panel companies to collaborate with research agencies and decide how best to filter the water, regardless of how we tap the source.
From my perspective, there is a seat at the table for both desktop and mobile sample and I encourage clients daily to create a strategy that leverages both these sources. The blog I mention here indicated that app-based mobile sample has 0 percent fraudulent activity and quite frankly this is 100 percent wrong!
It is often difficult to get accurate estimates of survey noise, incidences of fraud, and other issues that, while undesirable, do happen. The same can be said for mobile panels that, because they are newer, offer even less transparency and clarity into how susceptible they are to these issues.
Desktop-based sampling has been in practice since the late 90s. It is safe to say this mode is mature and there are several approaches that can be employed to filtrate unengaged and/or fraudulent behaviors. Mobile apps, conversely, are in a relative new-state and therefore require more investigation as it relates to MR-specific strategies for fraud mitigation. The point is that both modes are susceptible to fraudulent activity and require pro-active strategies to manage against this unsavory by-product of simply doing research in the 21st century!
One of Innovate’s primary goals is to have these difficult conversations and address the elephant in the room that many overlook or actively ignore. There can be issues with sample, and it’s only through diligence and constant technological innovation that we can address and overcome these issues to provide the best quality and maintain a high level of engagement with research participants.
What Should Sample Look Like?
There are no widely accepted benchmarks for us to evaluate survey noise and sampling error, as each company typically defines what the acceptable threshold is for their business.
However, we can look to the largest companies in the space to evaluate what they see in their data. Recently, at the Insights Leadership Conference in Palm Beach, Florida, a leading and reputable full-service firm highlighted that 7 percent of their sample was rejected: 3 percent from unengaged respondents, and another 4 percent due to fake data via IP masking and botting.
While not a perfect state; the firm indicated that 7 percent does not impact the integrity of their overall results and they are actively leading the industry in exploring strategies to reduce this footprint on their data. Some noise is ok, but less noise is always better! It should be noted that 7 percent came from both desktop and mobile-based samples.
How Clean Is the Water?
Sample buyers want to know that the sample they purchase is as clean as possible. They want accurate, non-misleading indicators that inform how accurate their surveys will be and what adjustments will be needed.
Sample suppliers have been addressing these issues for decades, and while there are certainly areas for improvement, it’s the acknowledgement of these issues that allows us to deal with them effectively. As a newer platform, mobile app-based solutions are dealing with their own wave of issues, starting on the ad side.
According to DataVisor, a leading fraud detection company, mobile app ad spending reached $5.7 billion in 2016 and averaged a fraudulent install rate of 5.3 percent, costing mobile marketers a whopping $300 million. While advertisers and researchers are improving their ability to detect fraud, fraudsters are working hard to improve technologies that take advantage of these gaps.
This includes malicious apps, install farms, mobile device emulators, click injection apps, cloud datacenters, and proxy servers, all working to make it as difficult as possible to discern between legitimate and fraudulent inventory. Fraud is just as prevalent in the mobile space as we see in the desktop environment.
For example, here’s a video of a Chinese mobile click farm that was uncovered earlier this year. Mashable covered a different story where a click farm had over 500 cell phones and 350,000 SIM cards working diligently to generate "fake" page views, likes, and shares through the social media app WeChat. Scary stuff indeed!
The Long-Term Challenges of Advancing Technologies
As fraud detection and sampling technologies advance for both online and mobile panels, so too do the methods used by fraudsters to infiltrate and take advantage of these networks.
It is our role to stay as many steps ahead of these issues as possible, while accurately representing the frequency of these incidences for the companies with which we engage. Data-driven decision-making that provides actionable resources for companies purchasing sample will improve the conversation we are having and offer key insights that right now aren’t always clear.
So, why did I write this blog? Simply put, to set the record straight on false claims being made in our industry. I’ve always been a big believer that we have a responsibility to educate the clients we serve and contribute to innovative methods that can help us move our industry forward. Don’t point fingers at the problem; be part of the solution! As such, I’m excited to announce our ongoing partnership with the Data Quality Genome Initiative. This group includes top suppliers and agencies whose goal is to champion new mitigation approaches and educate researchers on the various permutations of fraud taking form in both desktop and mobile samples.