Examining the Feasibility of Using SMS When Surveying College Students

by | Apr 11, 2018 | Alcohol & Other Drugs, Climate Surveys, Higher Education, Innovative Methodologies, Survey Methodology

Text messages (also known as Short Message Service, or SMS) are more and more becoming the go-to medium of communication. This especially is the case for today’s college students, who seem to conduct their social and even business lives completely via their smartphone. Scott Crawford and the team at SoundRocket looked at the data surrounding the efficacy of using SMS when surveying college students, resulting in a presentation at the 2013 American Association of Public Opinion Research (AAPOR) Conference.

THE QUESTION

As college students move more and more toward SMS as a primary means of communication, could SMS be a feasible mode to contacting students regarding surveys, and thus increase survey response? More specifically, could surveys using SMS more closely integrate survey participation in students’ routines?

SOUNDROCKET’S INSIGHTS INTO THE CHALLENGES OF USING SMS

We found the following to be the main challenges that pop up when using SMS technology as part of a survey’s data collection protocol. Challenges include:

  • Making sure students gave proper consent for their cell numbers to be used.

  • SMS message length and how to best to link the survey.

  • And more.

THE RESEARCH AND ABSTRACT WAS LEAD BY AUTHORS:

  • Scott D. Crawford, Research Consultant / Founder, SoundRocket, Ann Arbor, MI

  • Colleen A. McClain, University of Michigan Institute of Social Research, Ann Arbor, MI

  • Sara O’Brien, Research Director, Survey Sciences Group LLC, Ann Arbor, MI

  • Toben F. Nelson, Division of Epidemiology and Community Health, School of Public Health, University of Minnesota, Twin Cities, MN

ARTICLE ABSTRACT

Surveying undergraduate student populations is easily done through the use of their University provided email address – often Universities will engage in research and provide such contact to researchers.  It is expected that the University can/will use email addresses to request students participate in surveys.  However, when it comes to cell phones (and SMS messages), that expectations is not necessarily there.  Universities have a higher burden (typically) to meet to demonstrate that the students’ have consented to cell phone use – and, often Universities are reserving cell phone SMS messaging for emergency situations.  Many Universities today pledge to only use cell phones for emergency contacts unless students specifically sign up for other alert services.

In this study, we sent undergraduate students at a large midwestern who had participated in a prior study of mental health and alcohol use (and who had agreed to a follow-up study) an email and letting them know that we would “like to have the option of contacting you via text/SMS message.” We informed them that their phone number would be used only for the survey and would be destroyed once the study was completed.

Those who agreed to allow SMS were included in the follow-up study — almost 20 percent provided us with their mobile phone number.  A small number (about 2.5%) who agreed to provide their mobile number did not in fact provide it. 

We found that those who were more likely to give consent were younger, female undergraduate students. Those who consented also reported higher levels of technology use.

We randomized this population into two groups: those who would receive the survey via text and those who received it via email.   To tease out whether the short bit.ly URL that we used in the SMS message could have an impact on the response rate, we randomized those who did not give permission to contact them via SMS into two groups: one group received bit.ly (shortened) URLs and the second received emails with regular-length URLs.

We made sure the survey itself was short: students were asked to answer just 11 questions.

We had hypothesized that contact via text would be associated with a lower response rate and this turned out to be the case (46% RR in the SMS group compared to a 67% RR in the email contact).  We also found among the group who did not approve SMS (so they got email invitations), the use of the short bit.ly URL resulted in a higher response rate (77% vs. 57%).

However, among those who didn’t approve SMS were much more likely (27% vs. 2%) to break off from the survey.  It appears that the bit.ly URL facilitated entry to the survey, but what was gained in response was lost in break-offs.

FINAL THOUGHTS

Surveying students via SMS/text has its challenges: it’s not an optimal mode of contact. However, we find that where permission is obtained, it may have value when used to supplement protocols for the right type of data collection and with the right respondents.

We believe further research is needed in order to:

  • Optimize the integration of SMS and other mobile technologies more effectively into research protocol.

  • Identify the particular context in which SMS messaging may be useful (invites vs. reminders; longitudinal studies vs. cross-sectional; etc.).

  • Test protocol elements more rigorously via tighter control over the experiment.

THE SOUNDROCKET PROMISE

You can’t accomplish your research goals without great data. We at SoundRocket are committed to making sure that the data collection process for your surveys proceeds smoothly and securely. Learn more about how the SoundRocket Method can help your research!
Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 

 

Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 

 

To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  

 

 

As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

Scott D. Crawford

Scott D. Crawford is the Founder and Chief Vision Officer at SoundRocket. He is also often found practicing being a husband, father, entrepreneur, forever-learner, survey methodologist, science writer & advocate, and podcast lover. While he doesn’t believe in reincarnation, he’s certain he was a Great Dane (of the canine type) in a previous life.