Research Wonder: Do Web Surveys Change the Way We Interpret?

by | May 20, 2016 | Opinion / Editorial, Research Wonder, Survey Methodology

In the recent conference proceedings of the 2016 Computer Human Interaction (CHI) Conference in Human Factors in Computing Systems, Geoff Kaufman and Mary Flanagan present an eye opening piece of research on web survey methodology and the different reaction our brain has to content presented on digital vs. non-digital platforms.

Their work raises excellent questions about how we interact with content that is presented electronically vs. on paper – and raised awareness around the fact that even if we do everything we can to make the stimulus comparable visually, when dealing with paper vs. electronic, the context of the platform may be going beyond simple design.

Specifically, using the Construal-level Theory of Psychological Distance, the authors found evidence to suggest that respondents on a digital device will process information with a lower-level construal than those on paper.  This means that those using digital will be more focused on the trees over the forest.

In thinking about this, I can’t help but wonder how this research may inform us in the field of survey methodology.


If digital devices nudge survey responders to a more concrete & detailed way of looking at things, what does this mean for measures where we are asking respondents to quantify their behavior over some fixed period of time?  Will respondents be more susceptible to recent events when estimating a long fixed term period if they respond to a survey via a web-based survey than if they respond via a paper survey?


When we are asking about risky behaviors, does a low-level construal result in less risk taking about the reporting of such behaviors?  Do we see lower reports of risky behaviors in web surveys vs. on paper as a result?


How much of what we call “mode effect” in survey methodology can be explained by this difference – especially when an effort have been put forth to create comparable survey instruments between web-based surveys and paper surveys?

Like usual with these “wonders”… I don’t know the answers (yet).  But this paper certainly did give me something to think about this week!

What’s a Research Wonder?  Read this to find out…

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 


Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 


To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  



As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

Scott D. Crawford

Scott D. Crawford is the Founder and Chief Vision Officer at SoundRocket. He is also often found practicing being a husband, father, entrepreneur, forever-learner, survey methodologist, science writer & advocate, and podcast lover. While he doesn’t believe in reincarnation, he’s certain he was a Great Dane (of the canine type) in a previous life.