Use this as a checklist for your own studies – or as a tool to evaluate survey research collaborators. Researchers who do not consider each of these elements will make disastrous mistakes. A solid researcher is not one who is perfect – it is one who learns from past mistakes. Here are the final six (of twelve) areas to consider prior to data collection launch:
Programming is where social science meets computer science, and communication between these two paradigms is critical to quality data. Understanding what needs to be programmed, and why, provides survey programmers with the ability to raise issues, suggest alternatives, and engage in the research process from an entirely different perspective. Programmers and survey methodologists should be side by side working towards the same goal, not separated by organizational structures and walls.
A key step in programming is testing. Use white, grey, and black box testing techniques, and include both automated and manual testing processes. Make sure that what is programmed is what is intended, and that what is intended makes sense methodologically.
EIGHT: SURVEY SAMPLE MANAGEMENT
A major gap in off-the-shelf and other do-it-yourself tools for survey research is sample management. Sample management is generally customized for most studies and so should be evaluated as aggressively as the survey instrument itself. The sample management system should capture paradata (process data generated as a result of the data collection process) and other key metrics that support the data collection effort. Don’t assume that it does – review the data just as you do the survey data. The sample management system is also the heart of all data about the respondent that we have going in to each survey effort.
NINE: THE SURVEY
Separating out the questionnaire from the survey is a key step in catching problems in social science research. Look at the survey as the actual instrument that is used to capture the data using the specified questionnaire. The design of the survey, the mode of the survey, and the process of implementation are part of the overall survey implementation and can impact quality.
TEN: RESPONDENT COMMUNICATIONS
Communicating effectively with respondents about the research that is being conducted can have massive influences on study quality. Approach every communication as a potential mechanism for either contamination or success. Evaluate and monitor communication timing, mode, promises made, and related extras like incentive descriptions. They are all important communications worth reviewing closely as well as documenting carefully so that they may even be considered in the analysis and interpretation of the data.
ELEVEN: HUMAN SUBJECTS
Approach every research project with your own eye to protecting respondents using Human Subjects Committee (aka Institutional Review Boards, Ethics Committees, etc.) requirements. Often researchers fall into a mode of purely satisfying checkboxes that the HSCs present, missing where the real problems may arise. Ask yourself what the impact on human subjects will be in your study. Be hard on yourself – if you were to be called in front of Congress to testify on how your research participants are treated in your study, could you unequivocally state that you protected them to the best of your ability?
Consider all research subjects as if they were your own close family members – you may find that they are more responsive to your research when you do. Refuse to participate in any non-ethical research or anything that leads to excessive harm to respondents, regardless of who has given permission for the research to be conducted.
Supporting survey research is often a 24/7 effort. In today’s world, this may mean going beyond normal business hours. However, the key here is expectations. Have you set the correct expectations with your respondents and your extended research team as to the support that you will provide?
Also, consider that the best support provided is a proactive approach to a good quality study design as well as good usability of all respondent communications and survey designs.
I hope you find this list useful in your research. These twelve checklist items have been learned the hard way – lessons that no researcher needs to learn on their own again.