Migrating from git-media to git-lfs

Caution: This procedure has a huge limitation that the history of binary files are completely lost. (In our case this was not a problem, since we basically never overwrite binary files in the project.)

Objective

Migrate repos repos_gm (git-media) to repos_lfs (git-lfs).

Prerequisites

Setup git-lfs on the remote repository. (We used GitHub in our case.)
Install git-lfs on the client side.

Import binary files from git-media repos

$ mkdir git-lfs-migration-test
$ cd git-lfs-migration-test

$ git clone git@github.com:orgfoo/repos_gm .
$ $EDITOR .git/config
 ...
 [remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
-       url = git@github.com:orgfoo/repos_gm
+       url = git@github.com:orgfoo/repos_lfs
...

Set file types to track

Check existing file types.

$ find ../repos_gm/ -type f -print \
  | grep -v '.git' \
  | sed 's/^.*\([\.\/][^.]*\)$/\1/g' \
  | sort -u
.csv
.epub
.erb
.gif
.haml
.md
.mobi
.pdf
.png
.rb
.ruby-version
.txt
.yml
/Rakefile

Start tracking files.

$ git lfs track "*.pdf" "*.png" "*.epub" "*.mobi" "*.gif"
$ git add .gitattributes
$ git commit -m 'Configure file types to track with git-lfs'
$ git push origin master

Remember to push chanegs in .gitattributes, to enable file tracking by git-lfs hereafter. Otherwise, you may get file size exceeding error from GitHub later when you push.

Copy binary files

Import files (dryrun).

$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    echo cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done

Import files.

$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done
...
$ git status

Add, commit, and push: Simple and easy case

If your files are small enough, the following may suffice.

$ git add **/*.{pdf,png,epub,mobi,gif}
$ git commit -m 'Restore binary files overwriting git-media text files'
$ git push origin master

If you have large amount of files, you may have to use some kludge to push. See the next section.

Pitfalls

Git LFS at GitHub has a limit of 1GB in storage and transfer for now. If you need more, you have to purchase data pack ($5/mo for 50GB according to current pricing).
Sometimes GitHub fails to receive your push.
```
remote: fatal: pack exceeds maximum allowed size
error: pack-objects died of signal 13
```
Try pushing one commit per push to reduce pack size. You may have to split large commits into small ones.

See http://stackoverflow.com/questions/15125862/github-remote-push-pack-size-exceeded.
GitHub file size cap seems to have effect on Git LFS-enabled repos. Not yet confirmed though.

Add, commit, and push: A case with large amount of data

In our case, we had 5.8GB of binary data. We purchased a data pack, and did the following after setting up .gitattribute by git lfs track.

$ git checkout master
$ git checkout -b git-lfs-migration
$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done

Suppress ssh interactively asking your passphrase.

$ eval `ssh-agent` && ssh-add

Add and push one directory at a time (dry run).

$ (cd somepath; \
  for d in `ls -a | grep 'somemagicnumber'`; do; \
    git add --dry-run $d && \
    git commit --dry-run -m \
      "Restore binary files overwriting git-media text files ($d)" && \
    git push --dry-run origin git-lfs-migration; \
  done)

$ ^C

Add and push one directory at a time.

$ (cd somepath; \
  for d in `ls -a | grep 'somemagicnumber'`; do; \
    git add $d && \
    git commit -m \
      "Restore binary files overwriting git-media text files ($d)" && \
    git push origin git-lfs-migration; \
  done)

Things should look like this.

[git-lfs-migration xxxxxxx] Restore binary files overwriting git-media
text files (foo)
 2 files changed, 0 insertions(+), 0 deletions(-)
 rewrite somepath/foo/bar.png (100%)
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 24.55 MiB | 237 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
To git@github.com:orgfoo/repos_lfs
 * [new branch]      git-lfs-migration -> git-lfs-migration
...

(Actually we didn’t eventually take this workaround in our case due to some site specific problem, but it may work in other cases.)