Approach 1 Using Git LFS

Github has file size limits of 100MB. Files with a size of 50MB trigger a warning message but can still be pushed through.

Getting errors while pushing large files in git.

Git Large File Storage (LFS) is an open-source Git extension that allows users to store large files and binary files separately in the main Git repository. Instead of storing actual files or Binary Large Objects (blobs) in the Git repository itself, Git LFS replaces them with text pointers. The actual file contents are stored on a remote server, such as GitHub.com or GitHub Enterprise. This allows users to work with large files in a Git repository without bloating the repository size.

Example:

If there are existing large files in your repository that resist Git further from accepting your changes and you would like to keep them in GitHub. In that scenario, you need to first remove those files from the repository and then add them to Git LFS locally. Let us see those steps in detail.

Step 1: Install Git LFS on your system.

(i) Download Git LFS.
(ii) Locate the downloaded file and install Git-LFS on your system.

GIT LFS Installation

(iii) Verify that the installation was successful:

C:\Windows\System32>git lfs install
Git LFS initialized.

Step 2: Configure Git Large File Storage.

(i) Open Git Bash.
(ii) Set the directory path to an existing repository path, where you want to use Git LFS.
(iii) Select the file types you would like Git LFS to manage, You can configure additional file extensions at any time by using the following command:

git lfs track 

For example, to associate a .psd file, enter the following command:

git lfs track "*.psd"                

Here I want to track a .glb file. Let’s try to configure it.

Tracking .glb file using Git LFS

Step 3:

(i) Commit your local .gitattributes file into your repository.
(ii) Add a file to the repository matching the extension you’ve associated:
(iii) Commit and push your changes into remote.

Now, we will add a .glb file and with this .gitattributes file into our repo.

git add path/to/file.glb
git add .gitattributes

Changes we make in the remote repository.

GIT LFS Management

We Didn’t get much of what was happening, only added the local files to the remote server.

Git LFS handles large files by storing references to the file in the repository, but not the actual file itself. To work around Git’s architecture, Git LFS creates a pointer file that acts as a reference to the actual file (which is stored somewhere else). GitHub manages this pointer file in your repository. When you clone the repository down, GitHub uses the pointer file as a map to go and find the large file for you.

Large file located.

One important point is to remember, to commit your local .gitattributes file into your repository.

Relying on a global .gitattributes file associated with Git LFS may cause conflicts when contributing to other Git projects.

In each Git repository where you want to use Git LFS, select the file types you’d like Git LFS to manage (or directly edit your .gitattributes). You can configure additional file extensions at any time.

Optimizing Git for Large Binary Files

Version Control Systems are a category of software tools that help in recording changes made to files by keeping track of modifications done in the code.

Table of Content

  • What is large binary files?
  • The Challenge of Large Binary Files in Git
  • Why do we need to optimize binary files in Git?
  • Strategy for optimizing Git for Large Binary Files:
  • Approach 1: Using Git LFS:
  • Approach 2: Using Git-Annex
  • Differences Between Git LFS and Git-Annex:
  • Approach 3: Git-Submodules

Purpose of Version Control System:

  • Multiple people can work simultaneously on a single project. Everyone works on and edits their copy of the files and it is up to them when they wish to share the changes made by them with the rest of the team.
  • Version control provides access to the historical versions of a project. This is insurance against computer crashes or data loss. If any mistake is made, you can easily roll back to a previous version. It is also possible to undo specific edits that too without losing the work done in the meantime. It can be easily known when, why, and by whom any part of a file was edited.

Git is a free and open-source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. When you do actions in Git, nearly all of them only add data to the Git database.

Similar Reads

What is large binary files?

Large binary files or Binary File Objects (BLOBS) are complex large files, any external assets you add to your projects like images, videos, and animated content (.blb, .fbx), unlike other data strings that only contain letters and numbers....

The Challenge of Large Binary Files in Git

Git was originally designed to handle primarily text-based files efficiently. While it excels at managing source code, it can struggle with large binary files such as images, videos, compiled binaries, or datasets. These files can significantly increase repository size, making cloning, pushing, and pulling operations slower and more resource-intensive. Moreover, storing large files directly in the Git repository can lead to performance degradation over time, impacting the productivity of development teams....

Why do we need to optimize binary files in Git?

Let us consider a scenario now, Suppose you are working on a large-scale multi-modules project, and the project itself contains some large files or generates some large files in some of its phases....

Strategy for optimizing Git for Large Binary Files:

Approach 1: Using Git LFS:...

Approach 1: Using Git LFS:

Github has file size limits of 100MB. Files with a size of 50MB trigger a warning message but can still be pushed through....

Approach 2: Using Git-Annex

git-annex is an annotated git repository. git-annex is a distributed file synchronization system (development began in 2010). It aims to solve the problem of sharing and synchronizing collections of large files independent of a commercial service or even a central server....

Differences Between Git LFS and Git-Annex:

Subject Git LFS Git Annex Definition Git LFS is an open source Git command line extension. git-annex is a distributed file synchronization system. Working Principle Git LFS stores large files outside of the Git repository, while maintaining references to those files within the repository. Git Annex also takes large files out of Git’s history, but it handles this by storing symbolic links to the files in .git/annex Storage The files managed by Git LFS are stored as Git objects both in .git/lfs/objects and in the working repository, which can result in duplicated files and increased disk space usage. The actual data of the large files is stored in a separate backend, such as S3 or rsync. Protocols Git LFS supports both SSH and HTTPS protocols for accessing repositories. Git Annex works only through SSH. Programming Language Git LFS is primarily implemented by Go. Git Annex is written in Haskell....

Approach 3: Using Git-Submodules

It often happens that while working on one project, you need to use another project from within it. Then why not you can do similar things If you have some large files inside your project. You can manage the files in a separate repo and then use git-submodule to pull them into your project in the same way....

Conclusion:

In conclusion, Optimizing large binary files plays a crucial role, through the implementation of various techniques as discussed above you can able to improve the performance, reduce storage requirements, and enhance overall system efficiency....

Contact Us