Finding this post on or after Friday, November 16, 2018? This discussion of user comprehension studies is part of a 5-part series, which you can now find in its entirety as one post here. Feel free to read on, however, if you prefer to read it in daily chunks.
Authors
Scott D. Crawford, M.A., Shawn Fayer, M.Sc., M.S., C.G.C., Robert C. Green, MD, MPH
Note: See Part 1 for further details about these authors.
What Are User Comprehension Studies and Why are they Required by the FDA?
Unlike tests that are overseen by health care professionals where a conversation may take place about the meaning of results, direct-to-consumer tests bypass the traditional patient-doctor relationship and deliver results directly to patients (or consumers). A well-known example of a direct-to-consumer test is a pregnancy test that one may purchase at any pharmacy or similar retail store.
The FDA is charged with identifying and reducing risks to consumers with such direct-to-consumer tests. To do that, they require test manufacturers to conduct “controls” or “special controls” prior to marketing a test for a specific purpose — all designed to reduce risk. One common special control is the user comprehension study. Such studies are designed to ensure that the information being presented in the test report is done so in a way such that the average consumer understands and interprets the results accurately.
User comprehension studies are defined as a special control in all FDA DTC authorization processes. Most recently, in the Genetic Health Risk (GHR) protocol, the FDA identifies risks to the user of obtaining these reports as including “incorrect understanding of the device and test system” and “incorrect interpretation of the test results”. To help mitigate these risks, they propose special controls for the test manufacturer to take – Special Control #3 focuses on the comprehension of the test results.
Specifically, they require that naive, untrained users must be used to conduct comprehension testing. The testing must include sufficient numbers of cases and represent diversity of age and education level.
The user comprehension testing must evaluate a representative sample of the contents that are generated as the report, which must include a public facing website including test details, definition of terms, a pre-purchase page, a frequently asked questions page, technical details, as well as the test report itself.
The GHR authorization specifies that participants must be evaluated to ensure that they are capturing 90% comprehension on items in the following five domains (topics):
-
Test limitations
-
Test purpose
-
Appropriate actions (next steps)
-
Other factors that may have an impact on the test results
-
Test results
According to the FDA, the details of what concepts should be included in the report within these domains is left to a “physician and/or genetic counselor that identifies the appropriate general and variant-specific concepts contained within the material tested in the user comprehension study.”
For example – the purpose of the test is a domain that the FDA requires. However, they do not indicate what the specific purpose for the test should be. Using 23andMe’s publicly presented example report for Parkinson’s Disease, we can find several pieces of information that describe the purpose of the test. The concepts of two genetic variants, increased risk of developing the condition, as well as specific gene variants that the test looks for are examples of content that relate to the test purpose. The decision to include these concepts (and not others) was made, presumably, by a physician and/or genetic counselor who understands how to communicate these kinds of results to patients.
The FDA leaves the specific questions and methods for how to capture comprehension among these domains to the professionals (presumably to the expert questionnaire designers/researchers) involved in designing the study. However, they do provide some details that they feel are important things to consider in the design. We recommend you talk with someone who has experience in such designs to learn more about these details. However, if you would like to drink it directly from the FDA fire hose, most of these details above can be found on pages 20-22 in the GHR DEN160026 Reclassification Order.
User Comprehension for Personal Genome Service Pharmacogenetic Reports (PGSPR)
This brings us to the recent authorization for 23andMe’s PGSPR. There is no difference in primary concept to the previous GHR protocol described above— however, one change in language does show up that may have an impact on those involved in user comprehension studies for these new reports.
Where the GHR authorization defines the domains of the special control user comprehension study (limitations, purpose, next steps, other factors, and results), the language used in the PGSPR authorization around user comprehension is that it must cover “all result comprehension concepts that are critical for safe use of the device.” They do not specifically reference the domains of comprehension previously used in the GHR protocol.
It is too early to tell what this may actually mean. More details may be revealed once the FDA Decision Summary on the PGSPR is released. The Decision Summary is a follow-up document that comes after an initial authorization and often includes more complete details that relate to the specific authorization given to 23andMe.
Return tomorrow for part 3 in this series – “Decoding FDA DTC Authorizations: Part 3 – The November 1 Warning Shots”
The contents of this post are not intended as, nor should be considered legal, regulatory, health, or any other form of advice for any specific situations – they simply represent the opinions of these authors.