When I first heard about the new artificial intelligence (AI) chat tool called ChatGPT (created and offered as a free research tool by OpenAI), I was excited to play with it. Social media was abuzz with others in professional fields who had identified ways that this technology could greatly improve their workday. Could AI offer anything to a survey professional?
But I had my first adventure with this technology when I was hungry and wondering what I could do with my leftover turkey beyond making another sandwich. So, first I asked ChatGPT (or “Bot”, as I have come to refer to them in my mind) for some ideas. I was rewarded with a creative list of options.
After reviewing the options, I provided Bot with a list of other ingredients I had in my refrigerator and asked them to refine the suggestions to include one or two of those other items. The Bot responded with an updated list personalized for the ingredients I had on hand. I told Bot I would try the Turkey Chili option, and I asked for a recipe. Within moments I had a nicely formatted ingredient list and recipe.
Unfortunately, I never got to try that recipe out, as my family revolted about once again having leftover turkey. So pizza was ordered.
After this reasonably positive experience, I wondered whether we had reached the day when artificial intelligence could take on the job of being an AI survey methodology consultant.
Could ChatGPT help design a questionnaire?
Or maybe, to start, a question? I opened a new chat with Bot and typed out the following question:
“Please write a question for a survey that captures participant gender.”
Within moments, I had a response:
“What is your gender?
Options:
- Male
- Female
- Non-binary/gender nonconforming
- Prefer not to say”
I was impressed—but I was also concerned. The question did address non-binary and gender non-conforming options. This made it more inclusive than most free questionnaire resources. I did not expect that. However, this response still needed details. It needed guidance relevant to the question’s purpose and the expectations of data quality that I may have.
So, I started a new chat and thought about this next test as a conversation between two survey methodologists: Scott (myself) and “Bot”. In this case, I treated Bot as if they were a very smart and eager student who had unlimited access to the resources available at the university library.
I learned that there is great value in asking the right questions about how to ask the right questions.
The following transcript shows the evolution of a question across 11 pages, which took no more than about 10-15 minutes to complete. Bot made some errors, and they apologized and corrected themselves when that was pointed out. While the resulting question may not be perfect—I was trying not to lead it to a specific answer. Rather, I pointed out issues and concerns and asked for solutions.
Unlike the original suggestion, this is a set of questions that I would consider using in an actual questionnaire, such as one of our diversity, equity, and inclusion campus climate surveys.
Enjoy the exchange—but meanwhile—
Here are a few tips that I would recommend considering when using AI technology to assist in survey research:
- It’s all about the questions. Quality questions will give you quality questionnaire items. Garbage in—garbage out.
- Expect to go back and forth in a conversation.
- Start broadly, but include specific issues relevant to the question you want. Then refine and narrow your follow-up questions and prompts.
- Ask for references when you feel they are needed. (I didn’t show that lengthy side conversation that I had with Bot, but it was able to provide references to support most of its suggestions.)
- Expect it will make a mistake—do not accept its content “as is.” Always review its recommendations.
Enjoy my conversation with ChatGPT on the ins and outs of designing a survey question surrounding gender? We’re always happy to discuss these and other innovations in tech and survey research; contact us!