Text messages (also known as Short Message Service, or SMS) are more and more becoming the go-to medium of communication. This especially is the case for today’s college students, who seem to conduct their social and even business lives completely via their smartphone. Scott Crawford and the team at SoundRocket looked at the data surrounding the efficacy of using SMS when surveying college students, resulting in a presentation at the 2013 American Association of Public Opinion Research (AAPOR) Conference.
THE QUESTION
As college students move more and more toward SMS as a primary means of communication, could SMS be a feasible mode to contacting students regarding surveys, and thus increase survey response? More specifically, could surveys using SMS more closely integrate survey participation in students’ routines?
SOUNDROCKET’S INSIGHTS INTO THE CHALLENGES OF USING SMS
We found the following to be the main challenges that pop up when using SMS technology as part of a survey’s data collection protocol. Challenges include:
-
Making sure students gave proper consent for their cell numbers to be used.
-
SMS message length and how to best to link the survey.
-
And more.
THE RESEARCH AND ABSTRACT WAS LEAD BY AUTHORS:
-
Scott D. Crawford, Research Consultant / Founder, SoundRocket, Ann Arbor, MI
-
Colleen A. McClain, University of Michigan Institute of Social Research, Ann Arbor, MI
-
Sara O’Brien, Research Director, Survey Sciences Group LLC, Ann Arbor, MI
-
Toben F. Nelson, Division of Epidemiology and Community Health, School of Public Health, University of Minnesota, Twin Cities, MN
ARTICLE ABSTRACT
Surveying undergraduate student populations is easily done through the use of their University provided email address – often Universities will engage in research and provide such contact to researchers. It is expected that the University can/will use email addresses to request students participate in surveys. However, when it comes to cell phones (and SMS messages), that expectations is not necessarily there. Universities have a higher burden (typically) to meet to demonstrate that the students’ have consented to cell phone use – and, often Universities are reserving cell phone SMS messaging for emergency situations. Many Universities today pledge to only use cell phones for emergency contacts unless students specifically sign up for other alert services.
In this study, we sent undergraduate students at a large midwestern who had participated in a prior study of mental health and alcohol use (and who had agreed to a follow-up study) an email and letting them know that we would “like to have the option of contacting you via text/SMS message.” We informed them that their phone number would be used only for the survey and would be destroyed once the study was completed.
Those who agreed to allow SMS were included in the follow-up study — almost 20 percent provided us with their mobile phone number. A small number (about 2.5%) who agreed to provide their mobile number did not in fact provide it.
We found that those who were more likely to give consent were younger, female undergraduate students. Those who consented also reported higher levels of technology use.
We randomized this population into two groups: those who would receive the survey via text and those who received it via email. To tease out whether the short bit.ly URL that we used in the SMS message could have an impact on the response rate, we randomized those who did not give permission to contact them via SMS into two groups: one group received bit.ly (shortened) URLs and the second received emails with regular-length URLs.
We made sure the survey itself was short: students were asked to answer just 11 questions.
We had hypothesized that contact via text would be associated with a lower response rate and this turned out to be the case (46% RR in the SMS group compared to a 67% RR in the email contact). We also found among the group who did not approve SMS (so they got email invitations), the use of the short bit.ly URL resulted in a higher response rate (77% vs. 57%).
However, among those who didn’t approve SMS were much more likely (27% vs. 2%) to break off from the survey. It appears that the bit.ly URL facilitated entry to the survey, but what was gained in response was lost in break-offs.
FINAL THOUGHTS
Surveying students via SMS/text has its challenges: it’s not an optimal mode of contact. However, we find that where permission is obtained, it may have value when used to supplement protocols for the right type of data collection and with the right respondents.
We believe further research is needed in order to:
-
Optimize the integration of SMS and other mobile technologies more effectively into research protocol.
-
Identify the particular context in which SMS messaging may be useful (invites vs. reminders; longitudinal studies vs. cross-sectional; etc.).
-
Test protocol elements more rigorously via tighter control over the experiment.