Peer Review as a Necessary but Unscientific Process – Can we just do some iterative science already?

by | Sep 21, 2015 | Business Leadership, Higher Education, Human Subjects / IRB, Opinion / Editorial, Reviews, Social Science Business

Peer review is wonderful, in theory.  Scientists reviewing other scientists’ work to evaluate whether the science was applied thoroughly, implemented well, and interpreted effectively can be a wonderful way to allow the best science through.  But the Reproducibility Project clearly demonstrated that something is broken – when over a quarter of the published studies reviewed could not be replicated.

It is not a surprise to most.  Humans are involved.  We make mistakes.  We have our own biases, as was so well described recently by Philip Ball in a piece in Nautilus.

So I begin this post with the assumption that the current system is not perfect.  And neither is what I am going to suggest… Is it not time to explore different methods, methods that take advantage of today’s world of rapid communication and self-publication?

Is the current process for peer review no longer consistent with the scientific method itself?  If not, then why did it take such an effort to make a point that a lot of the science done is not as straightforward?

“The scientific method is a cycle. When we turn that cycle, we learn. When we learn, the science moves forward.”

The scientific method is a cycle.  When we turn that cycle, we learn.  When we learn, the science moves forward.  The faster we can facilitate the process, the greater our advancements will be.

How does the current peer review process fit into this method?  The peer review process limits available literature, it does not effectively provide a platform for hypotheses declaration, and it makes access to data sometimes more difficult. It often reduces space available for variations in analysis and interpretation due to length restrictions and editing. It limits others’ ability to replicate.

Traditional Scientific Method in the Social Sciences.

Traditional Scientific Method in the Social Sciences.

The technical infrastructure is there with platforms such as the Open Science Framework and the Dataverse Project, and blogging/publication platforms such as, Tumblr, Blogger, and Squarespace (the platform for this blog).  Some have even written about using blogs to communicate your science: 9 Reasons Why Running A Science Blog Is Good For You.

I don’t suggest that we drop the traditional peer review process – it certainly has its place for seminal career defining research that has already survived the test of critique over time through incremental steps.  But for those incremental pieces themselves, those smaller building blocks that eventually lead to the “big one”, is there not a better way?

Why don’t we start putting those findings out there on our own, with full transparency so that they can be reproduced or refuted?  Leave the door open for others to replicate (and declare their successes or failures to do so right alongside).  Allow for feedback and critiques.  Let your research stand the test of time (or pass as we collectively learn new things).

Let’s consider the dissemination of results as part of the scientific method itself – not an end by which we simply measure our prestige.

Currently, the FDA only regulates true direct-to-consumer (DTC) genetic tests, which have no health care provider involved either before or after testing. Consumer-initiated, physician-mediated genetic tests are considered lab developed tests (LDTs), which currently do not require FDA oversight. 


Our Study Design

Our study was designed to simulate the experience of an everyday person who is considering doing a health-related genetic test. For this reason, we only reviewed website contents presented to a consumer before ordering a test. By limiting our data collection to pre-test content, instead of digging around or contacting the companies to fill in missing data points, gaps in public-facing information that consumers use to make ‘informed’ decisions were revealed.  

Also, while a genetic counselor supervised the project, a research assistant (RA) conducted most of the website investigations. The RA was familiar enough with genetics and genetic testing to understand and identify the information presented on the websites, but has not had the clinical exposure that might create bias from knowing how specific tests work “behind-the-scenes”. 


To Sum Up

We set out to understand the landscape of health-related consumer genomics testing from the public perspective. By limiting our research (by design) to public-facing pre-test website content, we could not complete our data collection as set out in the protocol. However, this uncovered an important observation: consumer genomics websites are highly variable in content, readability and ease of use. 

This begs the question, if we can’t find basic test information on a consumer genomics website, how does a consumer have enough information to make an informed choice about testing? 

Stay tuned for Part 2 in this series, where we will dig into our study findings and reveal our most interesting observations.  



As experts in FDA user comprehension studies for consumer genomics companies seeking 510(k) clearance, we are interested in how everyday people access and understand health content that is meant for them. If you need help optimizing your consumer-directed health communications, we’ve got the in-house expertise and experience to meet your needs. Let’s chat

About the Author

Scott D. Crawford

Scott D. Crawford is the Founder and Chief Vision Officer at SoundRocket. He is also often found practicing being a husband, father, entrepreneur, forever-learner, survey methodologist, science writer & advocate, and podcast lover. While he doesn’t believe in reincarnation, he’s certain he was a Great Dane (of the canine type) in a previous life.