Survey research is a part of the scientific process – and even a science on its own.  So why do researchers abandon the science when they implement their studies?  An astronomer would not go to the hobby store to buy a telescope to study the galaxy.  A geneticist would never purchase non-sterile test tubes from an unknown source to capture saliva samples from research subjects.  So why do social scientists routinely treat their own data collection tools this way? 


The traditional scientific method as it is applied to the social sciences.

The traditional scientific method as it is applied to the social sciences.

Your social science research must embrace the scientific method throughout.  Look at what you do and ask yourself – is it science-based or are you satisficing in the implementation of your survey research?

By employing the scientific method, you can easily correct the following six mistakes that are routinely made during the implementation of social science research:

  1. Results cannot be used because the purpose of the current study was lost.
  2. Inconsistencies in study replication leave comparisons with previous literature and research lacking.
  3. The required data to test your hypotheses was not collected.
  4. Sources of error during the implementation of data collection (sample coverage, sampling, nonresponse, survey instrument, respondent environment & interpretation, interviewer effects, etc.) reduce data usefulness.
  5. Processing errors (coding, data editing & validation, weighting errors, etc.) increase bias, and impact your ability to appropriately analyze and interpret the data.
  6. A lack of documentation and transparency reduces the likelihood that you and others will repeat and cite your work.

There is no perfect data collection – the unpredictability of humans will always introduce errors.  But this does not mean you should not try.  Apply the same scientific rigor to your survey data collection that you do in your field.

Bring a survey methodologist onto your team, and work with people who understand the process and can correct these six problems before you even know they exist.

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 

 

Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 

 

To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  

 

 

As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

SoundRocket

Understanding human behavior—individually and in groups—drives our curiosity, our purpose, and our science. We are experts in social science research. We see the study of humans as an ongoing negotiation between multiple stakeholders: scientists, research funders, academia, corporations, and study participants.