Five Ways to Use Historical Survey Data to Improve Quality in a Survey

by | Jun 15, 2016 | Survey Methodology

When you find yourself surveying a population where significant information is known about those who are in the study prior to them completing the survey (such as in a longitudinal survey, a panel, or when the respondents are part of a known group such as a membership organization), such historical survey data can be used as part of the survey instrument design.  

While it is tempting to do whenever possible, when using such data, care should be given to how and when it is used – as it is not always recommended.  Using previously collected data may introduce a wide range of errors, which may in some cases be more significant than the intended gain in quality.  

Some of the possible issues with using historical survey data include:

  • Flaws in the collection or maintenance of the original data source (these could include all forms of error and may be the result of the data being collected for alternative purposes – not for research),
  • Changes to respondents experiences between when the original data was collected/generated and the current questionnaire,
  • Context changes / environmental changes that may have influenced how the responded answered previous questions,
  • Respondent recall / memory problems,
  • Respondent memory changes that could lead to rejection of previous answers, 
  • Respondent confidentiality concerns when a topic is sensitive.

As such, care should be taken to only use existing data in a survey when it is methodologically important to do so (there is clear improved data quality as a result of doing so), and it should never be done just because it is possible to do.

Here is some general guidance about when and how I recommend using historical survey data in a current survey:

  1. To drive sample selection.  When you are looking to learn more about a specific characteristic, you may use previous data to identify individuals who you would like to follow-up with some additional questions.  For example, a questionnaire entirely devoted to asking about being a cancer survivor could be fielded to a sample of people who are known to be cancer survivors.
  2. To drive question logic.  When you have data that you do not expect will have changed, it can be effectively used to drive question logic in the current survey.  For example, those who indicated that they have ever been diagnosed with high blood pressure at Time 1 may now be asked about medications they have taken for high blood pressure in the past, without having to ask about the diagnosis again.
  3. To provide parameters for validation checks.  Historical survey data can help validate later responses.  This must be done with care because it is possible that the previous data may be incorrect or have changed, and a validation triggered now may frustrate respondents.  I would only do this if the value in obtaining a response that is guided by the validation is greater than the possible loss of the respondent for the study if it causes frustration.
  4. To provide context for the respondent.  Often it can be helpful to provide context for respondents, so that their response is as comparable to that of others as possible.  If you goal is to learn more about a vehicle that the respondent identified as having been purchased in a Time 1 survey, and you want to ask more questions about that car today at Time 2, you may want to use their previous responses about the car to set the context for them.  For example, “When you responded to a survey last Fall, you told us about your Blue Buick Enclave that you were using as your primary vehicle.  Thinking about that vehicle….”
  5. To ease the burden of updating routine data.  Sometimes you may make the survey response task much simpler for the respondent if you provide their historical survey data back to them, allowing them to keep, or change, the item depending on their current situation.  For example, when responders are being asked several questions about each family member, it may be helpful to pre-load in a list of family member names to the survey, and present them to the respondent, asking if there are any changes to their immediate family before you proceed.  Another example of this is when you are capturing preferred contact information, or best times of day for a telephone follow-up, or other administrative items which are not likely to change a lot, but also not of significant value to the research.  Presenting those responses back to the respondent, and asking them to simply click “next” to accept the same responses as before can reduce respondent burden, yet give you an opportunity to identify updated information.  However, be careful with this approach.  If this approach is used for anything that is too specific, or too sensitive, or that is likely to change, you can be increasing the burden in doing this.  Also, because respondents will be most likely to simply accept what is entered, this should never be done if you truly wish to capture a valid response that may change.  While this can be useful in specific situations, it is not a normally recommend approach.

Again, care should always be taken to only use historical survey data when it is methodologically important to do so.  If you are considering using previous data, build the case for why it is important to do so.  If you truly reduce burden (through asking fewer questions as a result) and you know that you will get better data by doing so, then this is a wonderful technique to use in today’s highly programmatic surveys.  

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 

 

Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 

 

To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  

 

 

As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

SoundRocket

Understanding human behavior—individually and in groups—drives our curiosity, our purpose, and our science. We are experts in social science research. We see the study of humans as an ongoing negotiation between multiple stakeholders: scientists, research funders, academia, corporations, and study participants.