Bots & Bad data…How much do you really know?
The significance of precise surveys from market research panels cannot be overstated. Essential business decisions hinge on the data obtained from online surveys, and if that data lacks quality, the decisions will be flawed.
Nearly half of respondents in a standard online study using market research panels were identified as fake. Indeed, with a straightforward, 10-minute, simple-to-complete questionnaire, they found 46% of responses to be invalid. Here are two clear examples of how this was determined:
Legitimate respondents reported an average monthly medical care expenditure of $568, while the disqualified respondents claimed their average spending was $9,069—16 times more than the actual figure. Valid participants took an average of 80 seconds to read a 200-word concept statement, whereas disqualified participants spent only 11 seconds (around 18 words per second). This was 40% of respondents who didn’t properly read the statement before answering questions about it. If that isn’t alarming enough, the concern for data accuracy intensifies when targeting a more specific audience than the general population. Market research panel firms find it more challenging to source hard-to-reach respondents, heightening the risk of problematic participants and consequently, poor data.
Types of Problematic Research Participants
The reality is that conducting research using market research panels may expose clients to various problematic participants. However, by knowing what to watch for, these troublesome individuals can be identified and removed:
Outliers. These respondents’ performance or behavior deviates significantly from the average. They might not belong to your target user group or could be exceptional in some other way. Speed readers. A key indicator of a fraudulent participant, these respondents’ quick response times suggest they didn’t read with real comprehension. Cheaters. These participants aim to get paid without genuinely engaging. They can be spotted by their random answers, such as stating “12345” when asked how much they spent on your product last year. Straightliners. These individuals opt for the easiest path, often marking “agree strongly” to every statement or selecting every brand as “familiar.” Professionals. These respondents don’t represent “regular” users as they participate in too many studies too often. Their extensive survey exposure makes them attuned to researchers’ goals, potentially skewing their answers. Bots. Not human participants, survey bots are algorithms designed to complete online surveys automatically to earn payment. A 2020 Pew Research Center study found that checks for speed and attention alone fail to catch most fraudulent respondents. In fact, 84% of their bogus respondents passed a trap question, while 87% passed checks for responding too quickly.
With statistics like these, the necessity for various techniques and quality measures in all online surveys is evident. The integrity of research depends on questionnaire design, continuous data monitoring during collection, thorough final data reviews, and the willingness to eliminate any identified fraudulent participants. Crucially, market research panel companies and their clients must collaborate closely to detect fraudsters.
Tips on Questionnaire Design
Incorporate multiple traps in the questionnaire to identify fraudulent respondents. Panel suppliers can assist early on by reviewing the questionnaire wording.
Use CAPTCHAs to exclude bots. Ask the same demographic question at both the beginning and the end of your questionnaire for comparison. Include fake brands in brand awareness and usage questions. Utilize both numerical and open-ended text questions. Formulating answers in their own words helps identify legitimate participants. Spot potential “professionals” by asking screener questions about recent study participation. Eliminate anyone who marks 0-3 months or 0-6 months.
Ensure the overall survey completion length isn’t too demanding to prevent respondent burnout.
Tips on Monitoring Field Collection
A field partner should continuously monitor raw survey data during field collection to identify problematic participants. Reviewing open-end responses and tracking the time spent on individual questions to eliminate speeders ensures good quota integrity and helps establish separate cells with equal demographic distribution for monadic testing.
Tips on Final Data Review
A final data review is a critical process step. This review helps identify fraudulent respondents and ensures any additional responses after the last field check are considered.
Look for responses outside the normal range and multiple metrics, such as time on task (identifying those who move too quickly or too slowly) and task success (low success rates with fast task times). Respondents with multiple outlying metrics should be viewed with suspicion.
Bots, while challenging to identify, can be detected by behaviors such as:
Impossible timestamps Failure to answer required questions Identical open-ended responses from different “participants” Inconsistent responses to identical questions Impossible data values An additional check—or pre-check—to achieve the most robust results is to exceed the desired final sample size, anticipating the need to discard fraudulent responses. Skipping this precaution could jeopardize timeframes and survey outcomes.