No Easy Fix for Bogus Respondents in Online Opt-In Polls

This study tests how well three screening methods - trap questions (attention checks), CloudResearch’s Sentry automated prescreener, and matching respondents to a commercial voter file - identify and remove bogus respondents from online opt-in surveys. A large opt-in survey of 11,114 U.S. adults fielded Nov. 14-19, 2024, was used; respondents who supplied names and addresses were matched to a voter file after the fact. Data quality was evaluated using three metrics (yea-saying, quality of open-ended responses, and response-order effects) and by looking at effects on 2024 voter turnout and vote-choice estimates. Trap checks flagged 1,963 cases (18%), Sentry would have failed 5,369 completes (nearly half), and 49% of respondents gave contact info of whom 73% matched to the voter file (3,977 matched, 7,137 screened out by matching).

Purging bogus cases generally lowered error but produced no foolproof solution. Trap questions and the automated prescreener produced similar, modest improvements in data quality; Sentry failures were driven mostly by poor open-ends and yea-saying. Matching to voter files increased overall error because it removed many valid respondents who declined to provide or could not be matched to registration records. All three approaches modestly increased overestimation of Democratic support in 2024 - a consequence of bogus respondents’ tendency to report voting for the winning candidate - which shows trade-offs between removing fraud and preserving valid cases.