This post was written in collaboration with Jesus Arrue
Why develop live televideo interview methods?
In simpler times, pre-pandemic, we conducted user comprehension and other cognitive interview studies through in-person interviews. We would meet with research participants at central interviewing facilities, usually in rooms equipped with one-way mirrors and video cameras.
Participants would be given a set of hard copy materials or a laptop with pre-loaded content to review. A moderator would then ask questions about what they had just read. Key concepts would be tested to ensure that the content communicated the intended message clearly and effectively. The in-person protocol would allow interviewers to gauge not only if the participant could read the materials, but also if they understood them. The physical environment and existence of a researcher would apply gentle pressure on the participant to remain engaged in the process.
Participating in user comprehension studies requires significant focus. It is not a task that can be done on the phone while cooking a meal. It requires a quiet space without distractions—as distractions can be comprehension killers.
Until recently, conducting these studies online using commercial self-administered panels has failed to effectively measure comprehension. Perhaps leaving study participants to navigate the study protocol on their own in their own environment allows for too many distractions. For this reason, pre-COVID-19, in-person data collection with hard copy materials was the expectation from the FDA when conducting user comprehension studies for an FDA premarket submission (e.g. 510k).
A push toward innovation in survey science
In March of 2020, as the COVID-19 pandemic swept the world, in-person research protocols came to a screeching halt. Interviewing facilities closed. Researchers could not travel. And study participants were banned from public locations that could put them at risk of infection.
In response, almost overnight, live video conferencing and ‘Zoom-ing’ (capital Z) were adopted as the ‘new normal’. While these remote conferencing services existed pre-pandemic, they were predominantly used for remote work purposes. COVID-19 brought this technology into the homes of everyday Americans.
Knowing that fully self-administered user comprehension surveys do not work, we had to devise a pandemic-friendly workaround for our user comprehension study protocol. Importantly, we needed to satisfy the FDA’s regulatory requirements for a 510k submission.
Insights from established survey modes
When devising our approach we first looked to traditional survey modes and why they work.
In face-to-face (FTF) methods, interviewers use the back and forth nature of the conversation to confirm what has been said has been understood, a process called grounding. FTF interviews allow both verbal and non-verbal cues to establish this base understanding, making it the gold standard for comprehension studies.
However, there are benefits to self-administered questionnaires (SAQs) as well. To start, they are a lower-cost option. Usually distributed as web-based surveys, they allow researchers to survey populations who are otherwise difficult to reach and they can ease the disclosure of sensitive information which may be more difficult in front of an interviewer. Unfortunately, SAQs are plagued by unit and case-level non-response, without the interviewer facilitating engagement.
The conversational style of telephone interviews offers similar advantages to FTF interviews, however, without the physical presence of an interviewer, respondents may multitask during the interview, deviating their attention from the ongoing interview, or may break off early if they don’t feel engaged. Logistically, telephone-based user comprehension studies require the study materials to be delivered by mail or online prior to the interview.
A new acronym: LTIs
We gathered up the positive elements from each of these traditional modalities, and combined them into a new survey mode we call Live Televideo Interviews, or LTIs for the survey acronym junkies. Our Live Televideo Interview protocol uses FTF (with video) technology, self-administered surveys, and video or web delivered materials.
Here’s how it works:
- People are recruited to participate by matching specific demographic criteria.
- Participants complete a web-based baseline questionnaire, which documents their demographics and allows for the collection of any pre-test measures that are desired before exposure to the documents being tested.
- Participants are scheduled to join a Zoom live televideo interview session, as Zoom is the most widely used video conferencing provider.
- The interviewer presents the material for user comprehension testing by sharing their computer screen on Zoom and handing over permission to control the screen. This allows participants to scroll and navigate through a document as if they were doing so on their own computer.
- The interviewer may intervene to keep the participant on track and may assist if they have difficulty navigating the screen.
- The interviewer directs the participant to a post-test questionnaire that is designed to evaluate their comprehension of the material. Participants can navigate back and forth through the material as they proceed through the post-test questionnaire.
LTIs offer many benefits. Studies are no longer limited to specific geographic areas around interview facilities, as participants can engage with the study from anywhere with high-speed internet access and a computer with a camera.
During the entire Zoom interview, participants can see the interviewer’s face and the interviewer can see the participant. This visual contact allows for non-verbal signals to enhance communication and grounding during the study.
Lastly, to the glee of quality control staff and methodologists, the entire interview with the participant’s face, screen navigations, and all audio is recorded. These recordings can be used as primary data, as a tool for interviewer quality control and training, and serve as evidence of work completed.
Live Televideo Interviews emerged as a necessity during the pandemic, but just like the term “Zooming” and jokes about hoarding toilet paper, it looks like they are here to stay.