The Microbiome Metadata Crisis (MMC) project aims to quantify the proportion of microbiome studies that are reproducible. What we have realized in the microbiome field is that microbiome research is in the middle of a serious metadata crisis. A majority of microbiome studies are not reproducible at all, specifically because authors do not upload their metadata. This is a serious problem, especially when considering the challenges the microbiome field faces towards becoming a translational interface for biomedicine. And while there have been published reviews and letters on this topic, no one has bothered to go out and calculate just how reproducible the microbiome field is. Is it 20%? 50%? As such, we want to put a number on just how many studies are reproducible, to what level are they reproducible, and share these important findings with the scientific community.
REPRODUCIBILITY IN THE SCIENCES
reproducibility: independent researches can obtain the same results using the same methods and data; ensures reliability of findings
there are different types of reprodicibility:
methodological reproducibility
replication
inferential reproducibility
WHY DOES REPRODUCIBILITY MATTER?
builds trust in scientific findings
filters out errors, biases, and fraud
promotes transparency in methods and analysis
facilitates cumulative knowledge
CHALLENGES TO REPRODUCIBILITY
not sharing data/access restrictions
vague methods
unshared code/unspecified software dependencies
publication bias
human error
SOLUTIONS TO IMPROVE REPRODUCIBILITY
sharing data in public repositories
sharing code scripts
detailed and shared documentation
pre-registering studies and publishing negative results
following community standards for formatting data and methods
FIGURES
NUMBER OF PAPERS PER YEAR
This bar graph illustrates how the number of papers per year included in our meta-analysis out of #### total papers included
NUMBER OF PAPERS PUBLISHED BY COUNTRY
This figure illustrates nodes from which the studies included in our project stem from. Notably, the United States and China are two countries that are represented the most in our project.
NUMBER OF PAPERS PUBLISHED PER YEAR BY SEQUENCING TYPE
This bullet graph reveals the trend in the type of sequencing used in the papers included in our study. With this graph, an increase in the use of shotgun sequencing is observed up until 2023, when it decreased, but increased again for 2024.
REPRODUCIBILITY OVER TIME
This bullet graph reveals the trends with reproducibility over time. For a study to be reproducible, independent researchers must be able to obtain the same results using the same methods and data.
DISTRIBUTION OF REPRODUCIBILITY ACROSS TIME
Our analysis included finding a paper's metadata, making sure that the SampleID in SRA/genomic data base matches the metadata, and that the entry in the genomic data base is present and differentiable.
FLOW CHART
This flow chart depicts the steps taken during out project to decide if studies were replicable.
