How to useGit-Annex in GIT

git-annex is an annotated git repository. git-annex is a distributed file synchronization system (development began in 2010). It aims to solve the problem of sharing and synchronizing collections of large files independent of a commercial service or even a central server.

A repository, or Git project, enclose the entire collection of files and folders associated with a project, along with each file’s revision history. The file history appears as snapshots in time called commits. The commits can be organized into multiple lines of development called branches. git-annex provides as a supplement to this, A way of storing files alongside a git repo without checking them in. (git-annex manages files with git, without checking their contents into git.)

Big files are sometimes a problem and if you would like to capture large files, you can use it.

How does it work?

git-annex needs to be installed both on the server and client side.
Supported OS: Debian, Ubuntu, Fedora, Android, MacOS, Windows, Linux etc.

Note: The documentation says that git-annex on Windows is still in beta.
To download git-annex on Windows, it is recommended to download first Python on your machine.

There are two convenient ways to install git-annex. The first is downloading the installer from git-annex’ homepage. The other is to deploy git-annex via the DataLad installer. The latter option requires the installation of the datalad-installer Python package. Once Python is available, it can be done with the Python package manager pip. Open a command prompt and run:

Step 1: Download datalad-installer.

C:\Windows\System32>python -m pip install datalad-installer
Collecting datalad-installer
  Downloading datalad_installer-1.0.5-py3-none-any.whl.metadata (23 kB)
Downloading datalad_installer-1.0.5-py3-none-any.whl (31 kB)
Installing collected packages: datalad-installer
Successfully installed datalad-installer-1.0.5

Step 2: Afterwards, open another command prompt in administrator mode and run:

C:\Windows\System32>datalad-installer git-annex -m datalad/git-annex:release
2024-04-27T12:32:19+0530 [INFO    ] datalad_installer: Writing environment modifications to C:\Users\DELL\AppData\Local 
\Temp\dl-env-k4bfxefa.sh
2024-04-27T12:32:19+0530 [INFO    ] datalad_installer: Installing git-annex via datalad/git-annex:release
2024-04-27T12:32:19+0530 [INFO    ] datalad_installer: Version: None
2024-04-27T12:32:20+0530 [INFO    ] datalad_installer: Downloading https://github.com/datalad/git-annex/releases/ 
download/10.20240227/git-annex-installer_10.20240227_x64.exe
2024-04-27T12:32:25+0530 [INFO    ] datalad_installer: Running: 'C:\Users\DELL\AppData\Local\Temp\tmp9kbisx43\ 
git-annex-installer_10.20240227_x64.exe' /S
2024-04-27T12:32:25+0530 [INFO    ] datalad_installer: git-annex is now installed at C:\Program Files\Git\usr\bin\git-annex.exe

This will download a recent git-annex, and configure it for your Git installation. The admin command prompt can be closed afterwards, all other steps do not need it.

It is recommended to also set the following git-annex configuration:

git config --global filter.annex.process "git-annex filter-process"

Example:

In the following example, we will create a storage named database it is a directory on a storage server where we are going to keep a copy of all our large binary files with the help of git and git-annex.

Step 1: After you have a git repository, you run git annex init to set up the git-annex metadata. This is run once in each repository in the git-annex network:

Step 2: Check for available branches and try to switch to the master branch.

Step 3: Configure git-annex storage server

Content and changes can be created on remote clones of the repository, i.e. local computers of lab members and collaborators. Such contents and changes need to be pushed to the storage-server, to be shared. For this reason, the storage server needs to be properly configured to allow that, in two steps. The first is:

git config receive.denyCurrentBranch updateInstead

If you do not plan to push changes from remote, then you do not need this configuration. Notice that, if you push changes to the repository after enabling the previous configuration, the working tree of the repository will not be updated. See below how to enable the automatic update of the working tree. The second step is:

git annex wanted . standard
git annex group . backup

Step 4: You can add small files using git (full content in git), and large files with git annex add.

git add smallFile.txt
git annex add largeFile.txt
git commit           # metadata: commit message, author, etc.

Step 5: Adding public accessibility from the web

At this point, we will discuss how to access the repository when the storage server directory with the data is exposed via the web server.
Run the following commands:

....create repository on github....
git remote add github_project <github-URL-to-repository> // If any problem is there, skip the step
and create repository from github.
git push -u github_project master
git push -u github_project git-annex

Do you want to check further How git-annex works?

See, In the GitHub repo, Your git-annex branch will look like this:

Do you remember that we have added largerFile.txt inside git-annex, but where it is? I can’t see anything. Let us check the master branch.

See, Here is the culprit. Nothing has changed. Really?
Let’s check their content.

Yeah, the small file content is a little bit okay, what about large file?

Do you get the difference?

Git Annex takes large files out of Git’s history, but it handles this by storing symbolic links to the files in .git/annex or /annex/objects.

Version Control Systems are a category of software tools that help in recording changes made to files by keeping track of modifications done in the code.

Purpose of Version Control System:

• Multiple people can work simultaneously on a single project. Everyone works on and edits their copy of the files and it is up to them when they wish to share the changes made by them with the rest of the team.
• Version control provides access to the historical versions of a project. This is insurance against computer crashes or data loss. If any mistake is made, you can easily roll back to a previous version. It is also possible to undo specific edits that too without losing the work done in the meantime. It can be easily known when, why, and by whom any part of a file was edited.

Git is a free and open-source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. When you do actions in Git, nearly all of them only add data to the Git database.