Evaluating Nonresponse Bias in a Longitudinal Study of Healthy Adults Receiving Genome Sequencing

by | Dec 12, 2017 | Survey Operations

We know your survey is exciting and will divulge important findings; you know your survey is exciting and will divulge important findings. But darn those who don’t know this and so don’t participate. Or worse – those who do know this, but choose not to participate for some other reason. Their nonresponse can create uncertainty in how accurate our survey results actually are.  If only we could survey robots or well-trained dogs who followed our instructions instead of people!

All kidding aside, nonresponse can and does make us wonder about the accuracy of survey results, going so far as even making all the effort we put into creating surveys for naught because we just can’t rely on the survey’s accuracy.

We don’t have to tell you how troublesome this can be for researchers and scientists: our studies rely heavily on survey results and we must be able to defend their accuracy.

And it’s not just that “not enough” people respond to our surveys: nonresponse can be problematic if those who do end up responding to our surveys turn out to be different from those who do not in some systematic way – especially if that systematic way is correlated with the topic of the study. When differences exist – we call this nonresponse bias.  Studies have found that nonresponse bias is very difficult to pin down – it emerges differently with different topics and populations.

Download Evaluating Nonresponse Bias in a Longitudinal Study of Healthy Adults Receiving Genome Sequencing POSTER

Name


We presented a poster in October at the ASHG Conference in Orlando, Florida on Evaluating Nonresponse Bias in a Longitudinal Study of Healthy Adults Receiving Genome Sequencing (You can download the poster by filling our the form on this page) and we discussed how the PeopleSeq Consortium study (designed to measure economic, medical and behavioral outcomes after elective genome sequencing of healthy individuals) may have been impacted by nonresponse.

Understanding nonresponse bias helps methodologists design questionnaires and studies to deal with the always-present chance such bias can become an issue in a survey research study.

In short, by conducting a nonresponse analysis within the PeopleSeq study, we found that white females were overrepresented in our survey respondents, meaning that nonresponders were more likely to be male and non-white – a difference that may matter for some analyses. However, we also found very little evidence for bias in age. (See the poster for more details).

The key takeaway? There are differences between those who respond to surveys and those who don’t.

Yet before we decide to throw up our hands and weep bitter tears at the chance our survey is biased in some ways, we need to delve deeper.  So far, this study includes people who are a very select group to start with (those who have elected to participate in full genome sequencing). And, so far, we have not found any evidence that the bias is having an impact on any substantive outcomes being studied.

Armed with this information, we need to: (a) learn how to increase participation of those who turn out to be underrepresented, and (b) identify how much differential nonresponse may be actually influencing research findings.

Finally, the very existence of this evidence of nonresponse bias shows how important it is to have a survey tool and data collection design that is scientifically informed. And who has these nifty tools? SoundRocket of course! We help our clients create surveys that can tell nonresponse bias to stay home/you can’t sit with us, so that no matter who responds (and how many or few of them) results will be useful and actionable.

Contact us today to learn how we can help you save your surveys from the scourge or nonresponse bias!

(Yes, I’m now asking you yet again to download the poster for more information. Because I’m persistent like that.)

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 

 

Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 

 

To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  

 

 

As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

SoundRocket

Understanding human behavior—individually and in groups—drives our curiosity, our purpose, and our science. We are experts in social science research. We see the study of humans as an ongoing negotiation between multiple stakeholders: scientists, research funders, academia, corporations, and study participants.