Survey Data Collection Part 2: 12 Pre-launch Steps to Quality Data

by | Aug 10, 2015 | Business Leadership, Social Science Business, Survey Methodology, Survey Operations

NOTE:  If you have not yet read Part 1 of our survey data checklist, please do so now.

Use this as a checklist for your own studies – or as a tool to evaluate survey research collaborators.  Researchers who do not consider each of these elements will make disastrous mistakes.  A solid researcher is not one who is perfect – it is one who learns from past mistakes.  Here are the final six (of twelve) areas to consider prior to data collection launch:

SEVEN: PROGRAMMING

Programming is where social science meets computer science, and communication between these two paradigms is critical to quality data.  Understanding what needs to be programmed, and why, provides survey programmers with the ability to raise issues, suggest alternatives, and engage in the research process from an entirely different perspective.  Programmers and survey methodologists should be side by side working towards the same goal, not separated by organizational structures and walls.

A key step in programming is testing.  Use white, grey, and black box testing techniques, and include both automated and manual testing processes.  Make sure that what is programmed is what is intended, and that what is intended makes sense methodologically.

EIGHT: SURVEY SAMPLE MANAGEMENT

A major gap in off-the-shelf and other do-it-yourself tools for survey research is sample management.  Sample management is generally customized for most studies and so should be evaluated as aggressively as the survey instrument itself.  The sample management system should capture paradata (process data generated as a result of the data collection process) and other key metrics that support the data collection effort.  Don’t assume that it does – review the data just as you do the survey data.  The sample management system is also the heart of all data about the respondent that we have going in to each survey effort.

NINE: THE SURVEY

Separating out the questionnaire from the survey is a key step in catching problems in social science research.  Look at the survey as the actual instrument that is used to capture the data using the specified questionnaire.  The design of the survey, the mode of the survey, and the process of implementation are part of the overall survey implementation and can impact quality.

TEN: RESPONDENT COMMUNICATIONS

Communicating effectively with respondents about the research that is being conducted can have massive influences on study quality.  Approach every communication as a potential mechanism for either contamination or success. Evaluate and monitor communication timing, mode, promises made, and related extras like incentive descriptions.  They are all important communications worth reviewing closely as well as documenting carefully so that they may even be considered in the analysis and interpretation of the data.

ELEVEN: HUMAN SUBJECTS

Approach every research project with your own eye to protecting respondents using Human Subjects Committee (aka Institutional Review Boards, Ethics Committees, etc.) requirements.  Often researchers fall into a mode of purely satisfying checkboxes that the HSCs present, missing where the real problems may arise.  Ask yourself what the impact on human subjects will be in your study.  Be hard on yourself – if you were to be called in front of Congress to testify on how your research participants are treated in your study, could you unequivocally state that you protected them to the best of your ability?

Consider all research subjects as if they were your own close family members – you may find that they are more responsive to your research when you do.  Refuse to participate in any non-ethical research or anything that leads to excessive harm to respondents, regardless of who has given permission for the research to be conducted.

TWELVE: SUPPORT

Supporting survey research is often a 24/7 effort.  In today’s world, this may mean going beyond normal business hours.  However, the key here is expectations.  Have you set the correct expectations with your respondents and your extended research team as to the support that you will provide?

Also, consider that the best support provided is a proactive approach to a good quality study design as well as good usability of all respondent communications and survey designs.

I hope you find this list useful in your research.  These twelve checklist items have been learned the hard way – lessons that no researcher needs to learn on their own again.

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 

 

Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 

 

To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  

 

 

As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

Scott D. Crawford

Scott D. Crawford is the Founder and Chief Vision Officer at SoundRocket. He is also often found practicing being a husband, father, entrepreneur, forever-learner, survey methodologist, science writer & advocate, and podcast lover. While he doesn’t believe in reincarnation, he’s certain he was a Great Dane (of the canine type) in a previous life.