Promoting open science: a better tool for version controlling mega binary data files - git-mega

In recent years, the drive towards open science has been gaining momentum, with researchers recognizing the importance of transparent and accessible data sharing. However, version controlling large binary data files has remained a challenging hurdle in achieving comprehensive open science practices. Existing solutions such as Git LFS have provided some relief, but they struggle to handle mega binary data (a few hundred gigabytes to a few terabytes) with problems such as network traffic jams, long waiting times to clone repositories, consuming a large amount of disk space with multiple copies, costing too much money for hosting data. This talk introduces "git-mega," a cutting-edge tool specifically developed to meet the needs of version-controlling mega binary data files.

By capitalizing on the latest advancements in data storage and retrieval, git-mega efficiently manages large binary files within Git repositories, ensuring they remain lightweight and effortlessly accessible. Leveraging unique algorithms, git-mega minimizes the impact of mega binary files on version control operations, enhancing repository cloning speed and responsiveness.

Key features of git-mega include intelligent data deduplication, seamless data synchronization with existing storage systems, and comprehensive metadata management. Through its user-friendly interface, researchers can effortlessly integrate git-mega into their existing workflows, eliminating the complexities often associated with handling mega binary data files.

Moreover, git-mega adheres to the FAIR (Findable, Accessible, Interoperable, and Reusable) principles, fostering a culture of data sharing and collaboration within the scientific community. Researchers can confidently share their binary data files, knowing that git-mega facilitates effortless discovery, accessibility, and reproducibility of scientific findings.

This talk discusses the architecture, implementation, and performance evaluation of git-mega, comparing it with existing solutions to showcase its superiority in managing mega binary data files. By promoting transparent data sharing and fostering collaboration, git-mega brings researchers one step closer to realizing the full potential of open and reproducible scientific endeavors.