Git-Annex

One scenario where DataLad shines is managing datasets that are larger than what a single Git repository can deal with. The combination of git-annex’s capabilities to separate Git hosting from data hosting in extremely flexible ways with DataLad’s approach to orchestrating collections of nested repositories as a joint “mono repo” is the foundation for that. One example of such a large dataset is the WU-Minn HCP1200 Data, a collection of brain imaging data, acquired from more than a thousand individual participants by the Human Connectome Project (HCP)....

Two years ago, my colleagues published FAIRly big: A framework for computationally reproducible processing of large-scale data. In this paper, they describe how to partition a large analysis (their example: processing anatomical images of 42 thousand subjects from UK Biobank), using DataLad to provision data and capture provenance, so that individual results can be reproduced on a laptop, even though a cluster is needed to run the entire group analysis. The article is accompanied by a workflow template and a tutorial dataset....

If we are being honest, hosting DataLad datasets is not the most straightforward thing in the world. Sure, through the power of git-annex, and with the help of a range of DataLad extension packages they can be put pretty much anywhere. But the fact that it is possible does not imply simplicity. There used to be a time when GitLab supported git-annex repositories directly. But with the arrival of Git LFS it got removed....

Managing my music collection was the very first thing I tried with git-annex. That was more than 13 years ago, and till today I continue to use the very same git-annex repository for this purpose. Screenshot of the first commit’s git log of my oldest git-annex repository that is still in active use today. I still have a clone on my laptop. I had others on machines like the small box plugged into the wall with speakers connected and running mpd....