When Your Science Doesn’t Overcome Your Bias

I read a fair few science papers and not all of them contain what would be thought of as ‘good science’, mostly because there is something lacking in the methodology or the way conclusions are drawn. In some cases what is lacking is an understanding of how ones personal bias can cloud your judgement.

I’ve talked about biases before but here is an example of a bias that is common in our industry:

Natural is best for the skin so I am going to run an experiment to find out how much better my natural product is than a synthetic one. 

By highlighting that as a common bias I am not saying that I think there is no way this can be true or that I also want it to be true and think it likely to be so.  I’m just saying that we all have an emotional attachment to these words, the words are loaded and when we see the ‘natural vs synthetic’ question posed, our brains flood with ideas (or conceptions) that we will inevitably bring to whatever we do next.

As a science trained individual I have to be careful not to fall into the ‘don’t be stupid, everything is chemical’ mindset and thus refuse to give any discussions along these lines the attention and academic rigour they deserve.  Further, I have to avoid projecting what I think the words ‘natural’ and ‘synthetic’ mean onto the general public without explicitly explaining them.  I want to stop there for a moment to point that out more clearly,  we throw words like ‘natural’ and ‘synthetic’ around in this space as if it is universally known what we mean by them but actually, most people don’t have a clue and neither should they given that neither word has a really good and clear boundary around it. There simply is no logical and definitive way of defining these things when you really think about it. Both are human constructs and as such, have meanings that can shift depending on where humanity is sitting on the issues relating to how these words are used at the time.

I’ll leave you to ponder that  and carry on in the direction that I originally intended, that is about bias science.

So this morning I read this.

I found the above to be, in my mind, a good example of where personal bias had clouded the interpretation of the results and even prevented the development of a robust set of experimental conditions.  Now before I go on I must state that just because I found this paper to display bias that doesn’t mean that any of the products under the test conditions are bad or otherwise deficient. I’m just critiquing the method and interpretation rather than the subject or aims.

I debated in my head about what would be the most appropriate way to critique a paper like this without pulling it to shreds and then looking like I was attacking it (not my aim) so I decided that, in the name of promoting good science, I’d just stick to an outline of where I felt the experiment fell short. The aim of this is to inform you, my reader, of how you might avoid falling into these traps and how you might actually progress your understanding of your product rather than potentially just wasting time and money.

The experiment in brief.

1. The big question was whether modern-day synthetic cosmetics are a main cause of long-term damage to the skin microbiome.
2. The premise put forward was that there is a thought that synthetic chemicals are one factor that disrupt the skin micro   biome.
3. The investigation undertaken pitched a truly natural, a ‘claims-to-be-natural-but-isn’t’ and a synthetic product against each other in a body wash challenge.
4. The conclusion drawn was that this work indicates that synthetic ingredients have an effect on skin microbiome biodiversity.

The main points that made me go ‘huh?’ were:

• On point 1. The experimental protocol was to wash twice a day for 4 weeks. That is not a long-term study so can’t answer the question. A way around this would have been for the microbial analysis to continue post the 4 week time frame, maybe for a further 2 months with results of the micro biome mapped against the pre-experiment levels to see how the skin had responded once the product phase was over. As there are no results post the 4 week time frame we are unable to gauge whether the changes experienced persisted or constituted long-term damage. I would expect that long-term damage be demonstrated by a longer-term change in microbiome towards one that more closely resembled a dysfunctional skin sample.  That then brings up another question, what specifically does a dysfunctional skin sample look like for that skin area (given that the micro biome changes all over the body).
• On point 2 I felt there needed to be a much deeper analysis of what constituted a synthetic chemical for the purpose of this study given the importance of that definition in drawing up a roadmap for future work.  I’ve looked at the brands webpage and they point out their philosophy there.  For the record they include soap (as in bars of soap), alcohol, alkalis,  fermented products (as separate from alcohol),  xanthan gum or antiseptics.  This list presents a number of other challenges in its self-given that the brands own product doesn’t meet it – the current cleaner formulations contain several essential oils which have antiseptic activity.   With such an unformed and scattered concept of what constitutes natural and synthetic I would find it hard to draw any meaningful conclusions from this study that relate to the ingredient profile of the products.  This again means the purpose of the study can’t be addressed.
• On point 3 see point 2. The truly natural product does not meet the brands definition of a natural product as it contains antiseptics.
• The conclusions (point 4) reached are misleading in my opinion.  It is true that the testing did show a micro biome response to the test protocol but that is all we can conclude.  This article from 2007, which has been cited 568 times so far tested the microbial diversity on the forearm of 6 healthy subjects and found there to be great diversity between subjects and diversity on the same subject over time.  They found that some microbes appear to be more transient than others, responding to varying external and internal factors while other microbes are more settled.  If this is true it would be unwise to read too much into small microbiota changes over a short time frame in a small sample size.

The bottom line.

It is true that us humans don’t have all the answers to how our micro biome is affected by cosmetic products, ingredients or even treatment protocols (habits).  It absolutely is likely that everything we do to our skin has an effect but we must not forget that what we do to our skin includes the types of clothing we wear, where we live (weather/ how suited the climate is to our skin),  our home environment, our general health and our other habits (food, drink, sleep, stress etc).   There seems to me to be a thought-trap that many brand owners fall into that centres around cosmetic products being the be-all-and-end-all of skin health.  Mainly because that’s what they are selling, that is what is within their control and making better products (or having a point of difference) is what helps them to sell. However, this is a trap and micro biome science absolutely will be the undoing of these brands as microbes respond both to what happens on the skin and under it in a homeostatic feedback loop that informs our immune system according to this paper. 

I think it is great that cosmetic brands of all shapes and sizes are looking towards a scientific approach to back up their claims and progress their thinking and I really hope we see more of that in future.  However, if your biases shape your science your science won’t be very robust and that’s not going to help anyone long-term.

Amanda