Photo by Michael Barera (CC BY-SA 4.0)
.jpg){kind=link}
Soundrocket conducted an interview with University of Michigan professor Bill Axinn. Bill is a valued client and collaborator, and this post outlines Dr. Axinn‘s research, how he first came to find Scott Crawford, eventually collaborating with Soundrocket on survey methodology and survey research support. This is Part 1 of a two-part interview; next, check out Part 2!
So tell me, who are you? What do you do? Tell me a little bit about yourself.
Bill Axinn: My name is Bill Axinn. I’m trained as a sociologist demographer, but I specialize in primary data collection, mixed method data collection and survey research. I’m a professor at the University of Michigan in Sociology, and at the Institute for Social Research in both the Survey Research Center and the Population Studies Center. I also have a courtesy appointment at the Ford School of Public Policy.
What are some of the big projects you’re working on these days?
Bill Axinn: I have two streams of research, one of them in South Asia, primarily in Nepal – and another one in the United States. Both of them focus on family formation processes. So marriage and premarital cohabitation, childbearing, contraceptive use, how people get from being single people to being people with families. And I do a lot of that work in Nepal with a long term family panel study there.
Right now I am working in the US on a few different projects. One of them is helping out the team that does the panel study of income dynamics with some family formation measures. Another is working with a team of survey methodologists who have invented a new approach to nationally representative web surveys. So you know, web surveys are all the rage, but doing one that’s nationally representative of the entire US population is tricky. These folks have put in – really, decades of work – into the methods needed to accomplish this. The design is based on the US National Survey of Family Growth, which is the nation’s data on topics like childbearing, fertility, contraceptive use – as well today’s topics, forced intercourse, and and those kind of dimensions of sexual assault.
Anyway, that National Survey of Family Growth is a very big engagement of the federal government in a face-to-face, nationally representative survey. What we’ve done is pioneered the methods to replicate the same measures, but in a web survey format that can be done more quickly, more cheaply – with specialized populations or nationally representative. And so we’re very excited about that as a tool that will accelerate the science of studying families, childbearing, all of that.
That’s wonderful. So you know, we’re talking a bit about SoundRocket and your work with them. I’m just curious, when did you first meet Scott Crawford? How did you connect with them, and why did you start working with them on survey management?
Bill Axinn: So I had heard of Scott Crawford, because he had been a student at the University of Michigan’s Survey Research Center. We have a Master’s and PhD program in Survey Methodology, and he did that program. And so I knew of him because I work with some of the faculty in that program. I also work a bunch with our survey operations unit of the survey research center here, and they have worked with Scott on projects.
So I knew of him and of his work, including his work on campus-focused surveys. And I think a key piece of that reputation was the effort to apply high quality methods to survey data collection for institutions. Survey research for institutions is not my main thing, but my understanding of it is of course, there are a lot of institutions. that want to do things as cheaply and quickly as possible to kind of check a box – on an ‘I did this, leave me alone’ kind of framework. And Scott’s work kind of differentiates itself from that type of approach by applying good methods using sound tools, honestly.
So the backstory on what happened is the Office of the President of University of Michigan contacted me when I was the Director of the Survey Research Center, and asked if we had the wherewithal to do a really high quality – I think the phrase was ‘gold standard’ — campus climate survey of a topic like sexual misconduct. And in that context, at the time, we were conducting the National Survey of Family Growth, the US federal statistical data collection on that topic. And there’s also a National Academy of Sciences report on the topic of conducting sexual misconduct surveys with high quality – with excellence. And one of the things that that report concluded is that asking about sexual misconduct in the context of criminal behavior, or in a crime survey, it’s often done, generally leads to under-reporting of sexual misconduct… because a lot of the experiences of sexual misconduct are so personal, so intimate, oftentimes part of an existing relationship. But people don’t automatically think of it as a type of criminal behavior, even though it’s a type of assault against a person’s will.
And so, I knew that background a little bit, I knew that the US National Survey of Family Growth provided a good quality to it because it asked about sexual assault in the context of family, your reproductive health history (that means things like your history of various kinds of sexual experiences, sexually transmitted diseases or other things, stuff like that). And so I knew that that was a good context for asking questions about sexual assault.
When the University of Michigan asked me about this, I said, ‘Yeah, I think we could do that,’ but I knew immediately, you don’t want the University of Michigan’s Survey Research Center to conduct University of Michigan’s survey on sexual misconduct. So we knew we wanted a third party.
But we knew enough about data security – that there are a bunch of third party vendors who use products that are cheap and widely available – survey data collection products, questionnaire products, web survey products – easily available, but not high security. And we knew that Scott Crawford’s company, SoundRocket, used web survey software that was identical to the high security software that we sometimes use for web surveys at the Survey Research Center at the University of Michigan.
Knowing his reputation for good quality methods, knowing that he was trained in methods by the faculty with whom I was collaborating, and knowing they have the software tools to conduct this, I contacted Scott about the possibility of collaborating to do the University of Michigan’s survey on sexual misconduct.
And the first one was a few years ago, 2016, 2017?
Bill Axinn: 2015, actually. So I was the Director of the Survey Research Center – my term ended in 2014. And the query came in, I would say, early-to-mid 2014. If you look back at what was going on in the United States, at that time, the US Department of Justice was conducting investigations of over 100 different US universities – on the way they handle allegations of sexual misconduct and their rates of sexual misconduct – there was growing awareness that sexual assault on campuses was very high, and growing awareness that that somehow campuses needed to take some accountability for that and attempt to improve the situation.
And I think in that context – and I’m giving you my interpretation of the events I experienced in that context. My sense is that the University of Michigan as one of the nation’s leading public institutions wanted to be one of the leaders in addressing that very tough topic. And they wanted to have a data-driven, scientific approach to addressing it. And so I think that was the setting for launching this work in 2014 that ultimately resulted in the survey that we’re discussing.
That’s wonderful – because I know this is super important. And as you’re saying, it’s a topic that needs high quality, high security research. It answers the question of why the top survey research center in the world would use an outside company like SoundRocket to help conduct the research. And I think you explain that – the security, the objectivity.
Bill Axinn: I’ll tell you it goes a level deeper, and we have replicated it in other surveys – and we do this for other institutions as well. By having a third-party, independent firm – first of all, they can be monitored for the way they treat the human subjects engaged in research. Second, they keep all the identifying information. So in order to invite somebody to a survey, you have to know who that is. And as we’ll probably get to in a few minutes, in order to do a good job of contacting the people who were scientifically selected to represent the population – the target population – you have to have more than one way to contact them. Email addresses are nice, but if you have a phone number, mailing address, those are very important tools for establishing contact with the people who you want to have completed the survey.
Well, you have to have those records to conduct the survey, they have to be connected to who has completed the survey and who has not completed the survey in order to know that you’ve finished. And what you really don’t want is for that information to be in the hands of, in some cases, the employer, in other cases, the institution on which those human beings are reporting. So if it’s students, they’re reporting on their experiences on campus, and they’re reporting on the way that their university behaved in handling those experiences. You don’t want that institution to have any way to identify those individuals.
Those individuals have to be able to report knowing that they can say what happened to them with complete confidence that their answers will be 100% confidential, 100% secure, and there’s no way the institution about which they’re reporting can discover their individual identity. And so that’s crucial, I would say, in all of these studies of sexual misconduct on the campus, but in most institutional surveys, we really want to know the truth.
And so that ability to de-link the data collection – and frankly, we’ve used it in Diversity, Equity and Inclusion surveys at Michigan, also – so that people, faculty, staff, and students can really report their experiences. Those experiences are reported back to the institution in a de-identified report that only summarizes people’s answers with statistics. It does not expose the identity of any individuals, and cannot be linked back to the email, the phone number, the address or whatever contact with those individuals. And that function of this third party to be the barrier between ‘Yes, you can answer concretely, specifically, candidly, and know that your identity will never get to the institution or whichever party,’ that was a crucial part of having a third party play that role.
And do the survey respondents know that? Do they understand that security?
Bill Axinn: Well, you know, whenever you’re dealing with a large-scale communication effort – trying to explain something to potential respondents – whether they know it or not is partly a consequence of our ability to communicate that. So, you know, part of the success in communicating that, I would argue, is not only to communicate it, but to try doing it more than one way. You know, some people understand something one way other people understand another way. You and I are speaking, you are going to translate this into writing. And I think those are two different forms of trying to say the same thing – in making sure that we understand each other.
And that’s a lot of what the communication step of the survey invitation is about: trying to make sure that they understand that not only is it important that they participate (because they represent many other people like them, the way survey sampling works), but also that their confidentiality will be protected and they can speak frankly.