If you collect behavioral or market research observations online using Qualtrics surveys, data quality is a serious concern. Unlike in-person labs, paid data collection online is highly susceptible to fraud. In this post, we will revisit established best practices and adapt them to recent threats posed by Large Language Models (LLMs) like ChatGPT.
Browser Automation
Many scripting languages support automated browser interaction. These tools have important and legitimate uses in web software development and testing. Unfortunately, they can also be misused to provide shortcuts for survey respondents looking to maximize collection of incentive payouts with minimal effort. These are the bots.
As a means to commit fraud, simple automation has many shortcomings and detection is relatively easy, as responses are rarely plausible. For example, Likert scale questions might have no variation throughout your survey (“flatlining”) or fluctuate in a completely random manner (“noise”). Simple telling statistics like this should be easy to detect, given the right survey design choices.
Bots tend to hurry through surveys. Automation is usually much faster than human input. On detection, look for ways to distinguish human behavior from automation. Add timers. Always measure the time it takes to enter text (Qualtrics has a built-in timer question type). Does it look reasonable? You can even measure the time between keystrokes. Intervals that are too fast and too uniform indicate automated text entry.
Add delays to navigation buttons, if possible. Bad actors employing bots and LLMs profit from speeding though surveys. By slowing them down, your survey becomes a less attractive target.
Bots also “see” web pages differently than human respondents. To take advantage of this, you can hide questions using cascading style sheets (CSS) or JavaScript. A simple bot reading the HTML may not realize that the question is hidden and answer it anyway. Here’s a JavaScript code example that uses the Qualtrics API:
this.getQuestionContainer().style.display = "none";
Another classic strategy is to include open-ended text entry questions. A simple bot cannot generate realistic natural language. This is where the game has changed. LLMs are now able to generate plausible text in seconds, defeating easy detection.
Large Language Models
Deep learning products like OpenAI’s ChatGPT have demonstrated the ability to not only generate plausible open-ended text responses, but could be trained to save a memory context that maintains some consistency among preference, personality, or demographic questions. It could even perform well against typical attention check challenges or coherency analysis. That said, sophistication like this requires substantial engineering effort at the present time and remains unlikely in practice. While these more advanced uses are sure to increase eventually, LLM-generated text responses are a bigger concern now.
To prevent or detect LLM-generated text entry, our best strategies are to control and evaluate user input activity. For prevention, you can disable the ability to paste from the computer’s clipboard. This can be implemented simply in a web browser using JavaScript and will block most low effort respondents. For example, see the following code using the Qualtrics JavaScript API:
this.getQuestionContainer().addEventListener('paste', e => e.preventDefault());
Evaluate the text content. LLM-generated text often has a distinctive style and voice. It’s too wordy, too long, and too eager to just fill up space with clause after clause. This is subjective and can be tough to evaluate yourself, but there are services that can help label text with methods from Machine Learning and Natural Language Processing (NLP). Another obvious tell is that LLMs sometimes end at a specific character count, even breaking off mid-sentence. Look for partial closing sentences or text entries that are all the same length.
Trust and Oversight
It’s nice to have trust in your respondents, but that’s not always possible. For example, in-person labs can trust that their subjects aren’t taking shortcuts through active proctoring. Also, labs can better confirm an individual’s identity. In a lab, we rarely need to worry much about fraud.
At the other end of the trust spectrum are unmoderated online labor markets like MTurk. Here you are collecting observations from a large pool of profit seeking adversaries. Any surveys distributed here in the “wild” would have to be heavily fortified.
Somewhere in between the two are moderated panel providers. The best ones have rigorous identity verification steps and will actively monitor their pool for signs of fraud, proactively weeding out problem accounts.
Use the Standard Features
If you’re using Qualtrics, learn about and use the included features. Timers, mentioned above, help detect unnatural behavior and can be used to add navigation delays.
Under survey options, you can add fraud prevention defenses like preventing multiple submissions, RelevantID (digital fingerprint and reputation service), and reCAPTCHA (user input behavioral analysis).
Input validation is another often overlooked feature that can greatly improve the quality of the data you collect. This can be as simple as requiring that a response be a number, meet a minimum length, or conform to a standard format like an email address or date. Qualtrics offers extensive options. Not only will your data require less post hoc cleaning, but bots may struggle to follow your input instructions and stall out.
Best Practices Checklist
- Inattention—use attention check questions.
- Low effort—analyze choice responses for coherence, use open ended questions.
- Speeding—use timers that record user input duration.
- Identity verification—ask about location, demographics, pre-screen responses.
- Trust—Are you using a trusted panel with targeted distribution?
- Behavior—reCAPTCHA service and/or keystroke monitoring
- Controlled interaction—Navigation delays and UI controls like paste prevention.
- Content evaluation—input validation and NLP.
For your best chance of success, plan ahead. Do your best to include questions designed for effective bot detection. Define data quality and fraud detection metrics ahead of time and develop an action plan.
Recommended Services
- Qualtrics—secure online survey instrument
- Prolific—online panel provider
- Roundtable—AI-Native fraud prevention using NLP and behavioral metrics
Good luck. We will be posting more information as technologies emerge and evolve.