Research Reproducibility: Data Management, Methods, and Archiving

What the reproducibility crisis taught us

In 2015 the Open Science Collaboration re-ran 100 psychology studies and found that only about 36 percent produced significant results the second time, against 97 percent in the originals. The Reproducibility Project: Cancer Biology later closed with a similar verdict: of the experiments it set out to repeat, it could complete far fewer than planned, largely because the original papers did not describe their methods in enough detail to even attempt them. A 2016 Nature survey of 1,576 researchers found that more than 70 percent had failed to reproduce another scientist's experiment, and more than half had failed to reproduce one of their own.

The lesson is not that most science is fraudulent. It is that ordinary practice quietly loses information. A dropped outlier that never got logged, an analysis script that lived only on one laptop, a package that silently changed behavior between versions: each loss is small, and together they make a finding impossible to rebuild. Reproducibility is the discipline of not losing that information in the first place.

Reproducibility, replicability, and robustness are three different questions

These words are used loosely, and the confusion hides real distinctions. The most widely cited convention comes from the 2019 U.S. National Academies report, Reproducibility and Replicability in Science.

• Reproducibility (computational reproducibility): same data, same analysis, same result. Someone takes your dataset and your code and regenerates the numbers in your tables. This is the floor, and it is entirely within your control.
• Replicability: new data, same methods, consistent result. An independent team collects a fresh sample, applies your procedure, and finds the same effect. This tests whether the finding generalizes beyond your particular sample.
• Robustness: same data, different reasonable analysis, consistent result. Your conclusion survives a defensible change in model specification, exclusion rule, or covariate set. Multiverse and specification-curve analyses probe this directly.

You cannot promise replicability, because that depends on nature. You can and should guarantee reproducibility, and you can demonstrate robustness. Keeping these three separate tells reviewers exactly what your open materials do and do not claim.

Reproducibility starts on day one, not at submission

The single biggest mistake is treating reproducibility as a submission task. By then the provenance is already gone: you no longer remember why 14 participants were excluded, which version of the raw export produced Figure 3, or whether the abstract's p value came before or after the reviewer-requested robustness check. Provenance is the chain that connects a reported number back through the analysis, the processed dataset, the raw file, and the protocol that generated it. It must be captured as you work, because it cannot be reconstructed afterward.

Concretely, that means every processing step is a script rather than a manual edit in a spreadsheet, every figure names the exact input that produced it, and every claim in the manuscript can be traced to the evidence and source behind it. This is the same discipline that underpins an evidence base you can trace from claim back to source: the argument and the data share one audit trail. A research operations platform exists precisely to hold that chain together so nothing floats free of its origin.

The prize is not a tidy folder at the end. It is that at any moment you can point at a number in your paper and trace, in one unbroken path, the raw file behind it.

Write a data management plan, then make the data FAIR

The data management plan

A data management plan (DMP) is a short document, usually two to three pages, that you write before collection rather than after. Funders including the NIH (whose 2023 Data Management and Sharing Policy makes one mandatory), the NSF, and the European Commission's Horizon Europe now require them, and tools such as DMPTool and DMPonline provide per-funder templates. A useful DMP answers a handful of questions:

• What data will exist, in what formats, and at what volume? Raw instrument output, processed tables, code, and materials all count.
• How will files be named and organized? Agree a machine-readable convention on day one, for example a dated, zero-padded scheme so files sort correctly and never collide.
• Where does it live during the project, and how is it backed up? Follow the 3-2-1 rule: three copies, on two kinds of media, with one offsite.
• What are the licensing, consent, and ethics constraints on sharing? Human-subjects data may need de-identification or controlled access.
• Where will it be archived at the end, under what license, and who is responsible for the deposit?

The FAIR principles

FAIR (Wilkinson et al., 2016) is the reference standard for how research data should be published. It describes data that is:

• Findable: it carries a persistent identifier (a DOI) and rich metadata, indexed so search engines and databases can locate it.
• Accessible: the metadata, and where possible the data, are retrievable by a standard open protocol; if access is restricted, the conditions are stated and the metadata stay public.
• Interoperable: it uses open, documented formats (CSV over proprietary binary, standard vocabularies and units) so other tools can read it.
• Reusable: it carries a clear license, provenance, and enough documentation (a data dictionary or codebook) that someone else can legitimately reuse it.

FAIR is not the same as open. Sensitive data can be FAIR while remaining access-controlled: the metadata is public and findable, and the access route is documented. The European Commission's phrasing captures the balance well: as open as possible, as closed as necessary.

Document methods and protocols so a stranger can rerun them

The Cancer Biology project stalled mainly on under-specified methods. The test is simple: could a competent researcher in your field, with only your paper and supplements, run your study without emailing you? Almost no first draft passes it.

Write protocols as executable recipes. Platforms like protocols.io give each protocol its own DOI and version history, so you can cite the exact procedure you followed. For every method, record reagent sources and catalog numbers, instrument settings, software and its version, and every parameter whose default you silently relied on. For analyses, state the exclusion rules and the order in which they were applied, the statistical model in full (not "a mixed model" but the fixed and random effects, the estimator, and the software), and how missing data were handled. Reporting guidelines exist for most designs: pick the checklist that fits your study design, such as CONSORT for trials, PRISMA for systematic reviews, ARRIVE for animal work, and STROBE for observational studies. Journals increasingly require them at submission.

Version your data and analysis code, and freeze the environment

Version control from the first line

Put your analysis code under version control before you write the second line of it. Git gives every state of the project a commit hash you can cite, so "the analysis" is never ambiguous. Commit early and often with messages that explain why, not just what. Tag the exact commit that produced the submitted manuscript, for example v1.0-submission, so the version behind your numbers is fixed in the record.

Data needs versioning too, and Git alone handles large or binary files poorly. Use Git LFS, or purpose-built tools like DVC (Data Version Control) or git-annex, which keep a small pointer in Git while the bytes live in object storage. The principle is the same as for text: raw data is read-only and never edited in place; every transformation is a tracked, rerunnable step, so the path from raw to processed is a graph you can replay rather than a folder you have to trust.

Freeze the computational environment

The same code on two machines can produce two answers. Three things drift over time and must be pinned:

1. 1Random seeds. Set and record the seed for every stochastic step (resampling, cross-validation splits, MCMC, weight initialization). An unrecorded seed means even your own figure is not reproducible.
2. 2Package versions. "It worked last year" fails because a dependency changed under you. Capture exact versions: a requirements file or lockfile plus a virtual environment in Python, renv in R, a Project.toml and Manifest.toml in Julia.
3. 3The whole environment. For real portability, put the environment in a container (Docker or Singularity/Apptainer) or a Binder configuration, so the operating system, system libraries, and interpreter are pinned together. Archive the container image or its recipe alongside the code.

Pre-registration and registered reports

Reproducibility protects the computation; pre-registration protects the inference. By time-stamping your hypotheses, sampling plan, and analysis before you see the data, you separate confirmatory tests (planned) from exploratory ones (discovered), and you remove the room for p-hacking and HARKing (hypothesizing after the results are known). You can register on the OSF, on AsPredicted, or on ClinicalTrials.gov, which is mandatory for clinical trials.

A registered report goes further: you submit the introduction, methods, and analysis plan for peer review before collecting any data. If the question and design are judged sound, the journal issues an in-principle acceptance, committing to publish regardless of whether the results turn out significant. This attacks publication bias at its root and is now offered by more than 300 journals. Pre-registration does not forbid exploration; it just labels it honestly, so the claims you defend in the manuscript are marked as confirmatory or exploratory rather than blurred together.

Persistent identifiers, repositories, and availability statements

A URL rots; a persistent identifier does not. Three of them matter:

• DOI (Digital Object Identifier): a permanent handle for a dataset, a paper, a protocol, or a tagged software release. This is what makes an artifact citable and Findable.
• ORCID: a persistent identifier for you, the researcher, disambiguating your name across a whole career and every institution you pass through. Put it on every submission.
• RRID (Research Resource Identifier): stable identifiers for reagents, cell lines, antibodies, and software, so the exact resource you used is never ambiguous.

Deposit in a repository that mints DOIs and commits to long-term preservation, not a personal website or a lab Dropbox:

• Zenodo (run by CERN): general purpose, and it integrates with GitHub so tagging a release automatically archives a snapshot and mints a DOI. Free, up to 50 GB per record.
• OSF (Open Science Framework): ties together pre-registration, materials, data, and preprints for a whole project in one place.
• Figshare and Dryad: widely accepted general and data-focused repositories; Dryad specializes in the datasets underlying publications and curates them.
• Discipline repositories where they exist (GEO for gene expression, PDB for protein structures, ICPSR for social science) are preferred, because their metadata and curation are field-specific.

Your paper links to all of this through a data availability statement and a code (or materials) availability statement. Do not write "available on request", which studies repeatedly show fails in practice: one analysis recovered fewer than a third of datasets that had been promised that way. State instead where the data lives, its DOI, its license, and any access conditions. Choose licenses deliberately: CC0 or CC BY for data, and an OSI-approved license (MIT, BSD, GPL) for code.

The archive-ready package: a reproducibility checklist

At submission, assemble a single self-contained package, the thing another researcher downloads to rebuild your results. If you kept provenance throughout, this is an afternoon of assembly rather than a month of archaeology. A complete package contains:

1. 1A README at the top that states what the project is, how the folders are organized, and the exact command that reproduces everything.
2. 2Raw data, read-only, with a data dictionary or codebook defining every variable, its units, and its coding.
3. 3Analysis code, version controlled, that runs from raw data to every table and figure with no manual steps in between.
4. 4The environment spec: a lockfile or container recipe, plus the recorded random seeds.
5. 5Protocols and materials: stimuli, questionnaires, and instrument settings, ideally each with its own DOI.
6. 6A pre-registration link, if any, with a clear map from registered analyses to reported ones and any deviations noted.
7. 7A LICENSE file, kept separate for data and for code.
8. 8Metadata and identifiers: the deposit DOI, your ORCID, and RRIDs for the key resources.

The same discipline pays off long after publication. When you write the thesis chapter or the follow-up paper, a well-kept archive means your methods and results are already documented and citable, and regenerating a figure for a revision is one command rather than a week of reconstruction. If you write in LaTeX, you can wire figures and computed numbers to compile straight from your analysis outputs, so the manuscript and the code stay in lockstep and the archive stays connected to the evidence behind every claim.

Reproducibility is a maintained chain, not a bolt-on

Everything above reduces to one habit: never let a reported result float free of the process that produced it. Keep the seed with the analysis, the analysis with the data, the data with the protocol, and the claim with its evidence, from the first raw file to the final DOI. Do that continuously and the archive-ready package is a byproduct; skip it and no amount of end-stage effort can rebuild what was lost along the way.

This is exactly the chain Research Woven is built to maintain. It holds one continuous thread from source to highlight to note to evidence to claim to manuscript to citation to submission, and it computes provenance rather than guessing at it. Because the connections are kept as you work, you can stand behind any number in your paper and show its raw file in a single path. When your whole project is one maintained chain from raw data to final claim, reproducibility stops being a scramble at the end and becomes the shape of the work. Pair it with a disciplined research workflow from first idea to published paper and the archive builds itself.