ML Data Version Control and Reproducibility at Scale

### Overview In this talk, Einat Orr introduces the core concepts and operational challenges of data reproducibility at scale for data science and machine learning. She explains that while developers typically track code in Git and log parameters and metrics in tools like MLflow, managing datasets usually remains an error-prone, manual process involving copying files or renaming folders. Orr presents LakeFS, an open-source data version control system providing Git-like semantics (commits, branches, merges) over object storage without copying underlying data. A recorded demonstration presented by Amit walks through an end-to-end computer vision workflow using LakeFS, Git, and MLflow.

### Key Topics - **Challenges of ML Reproducibility**: Managing data versions manually through ad-hoc copies or lists of filenames leads to time loss, lack of auditability, and poor collaboration across data science teams. - **Zero-Copy Metadata Abstraction**: LakeFS adds an abstraction layer over object stores (AWS S3, Google Cloud Storage, Azure Blob, MinIO) by managing immutable pointers, deduplicating storage, and supporting branch creation in milliseconds without data copying. - **Local Workflow via lakeCTL**: Data scientists can use the lakeCTL CLI to clone a LakeFS branch locally, edit data, and commit changes back to remote storage. - **Git and LakeFS Co-existence**: Separating code tracking in Git from large data versioning in LakeFS using `.gitignore` and auto-generated YAML metadata files containing repository, branch, and commit IDs. - **Computer Vision Pipeline Demo**: Implementing a Medallion architecture (raw, bronze, silver, gold) for image segmentation using Kaggle's Airbus Ship Detection dataset, Delta Lake, Petastorm, PyTorch Lightning, and MLflow.

### Technical Details - **Zero-Copy Mechanism**: Importing or branching creates pointers rather than duplicating objects. Commits capture content hashes, enabling immutable version history and garbage collection to prune unreferenced commits and reduce overall storage. - **Demonstration Pipeline**: - Amit configures LakeFS Cloud with credentials, an endpoint, and an S3 bucket. - The demo uses the Airbus Ship Detection Challenge dataset from Kaggle (~200,000 satellite images, ~32 GB). For local experimentation, a subset of 100 images and corresponding masks is imported into LakeFS. - An experiment branch (`local-experiment-1`) is created simultaneously in LakeFS and Git. - Using `lakectl local`, the branch data is cloned to the local container environment. Git ignores the bulky local data directory while staging an auto-generated LakeFS YAML file containing the repository name, branch name, and LakeFS commit ID alongside the Jupyter notebooks. - A Medallion architecture processes the data: Raw images are read and converted into Delta Lake tables (Bronze), resized and enhanced alongside mask images (Silver), and joined to produce bounding/count data (Gold), with each stage tracked and tagged in LakeFS. - Models are trained using PyTorch Lightning and Petastorm (open-sourced by Uber), with parameters, metrics, run artifacts, and LakeFS linkage metadata tracked in MLflow.

### Takeaways - Data version control systems like LakeFS resolve the gap between code tracking (Git) and experiment tracking (MLflow) by providing a scalable, Git-like metadata layer over cloud object storage. - Teams avoid high storage costs and manual errors through zero-copy branching and automated pointer-based commits. - Integrating the LakeFS metadata YAML file into Git repositories enables tight coupling of code, data state, and model parameters for end-to-end reproducibility.