Caution: This procedure has a huge limitation that the history of binary files are completely lost. (In our case this was not a problem, since we basically never overwrite binary files in the project.)

Objective

Migrate repos repos_gm (git-media) to repos_lfs (git-lfs).

Prerequisites

  • Setup git-lfs on the remote repository. (We used GitHub in our case.)
  • Install git-lfs on the client side.

Import binary files from git-media repos

$ mkdir git-lfs-migration-test
$ cd git-lfs-migration-test

$ git clone git@github.com:orgfoo/repos_gm .
$ $EDITOR .git/config
 ...
 [remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
-       url = git@github.com:orgfoo/repos_gm
+       url = git@github.com:orgfoo/repos_lfs
...

Set file types to track

Check existing file types.

$ find ../repos_gm/ -type f -print \
  | grep -v '.git' \
  | sed 's/^.*\([\.\/][^.]*\)$/\1/g' \
  | sort -u
.csv
.epub
.erb
.gif
.haml
.md
.mobi
.pdf
.png
.rb
.ruby-version
.txt
.yml
/Rakefile

Start tracking files.

$ git lfs track "*.pdf" "*.png" "*.epub" "*.mobi" "*.gif"
$ git add .gitattributes
$ git commit -m 'Configure file types to track with git-lfs'
$ git push origin master

Remember to push chanegs in .gitattributes, to enable file tracking by git-lfs hereafter. Otherwise, you may get file size exceeding error from GitHub later when you push.

Copy binary files

Import files (dryrun).

$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    echo cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done

Import files.

$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done
...
$ git status

Add, commit, and push: Simple and easy case

If your files are small enough, the following may suffice.

$ git add **/*.{pdf,png,epub,mobi,gif}
$ git commit -m 'Restore binary files overwriting git-media text files'
$ git push origin master

If you have large amount of files, you may have to use some kludge to push. See the next section.

Pitfalls

  1. Git LFS at GitHub has a limit of 1GB in storage and transfer for now. If you need more, you have to purchase data pack ($5/mo for 50GB according to current pricing).

  2. Sometimes GitHub fails to receive your push.

     remote: fatal: pack exceeds maximum allowed size
 error: pack-objects died of signal 13

    Try pushing one commit per push to reduce pack size. You may have to split large commits into small ones.

    See http://stackoverflow.com/questions/15125862/github-remote-push-pack-size-exceeded.

  3. GitHub file size cap seems to have effect on Git LFS-enabled repos. Not yet confirmed though.

Add, commit, and push: A case with large amount of data

In our case, we had 5.8GB of binary data. We purchased a data pack, and did the following after setting up .gitattribute by git lfs track.

$ git checkout master
$ git checkout -b git-lfs-migration
$ for f in ../repos_gm/somepath/*/*.{pdf,png,epub,mobi,gif}; do;
    cp $f `echo $f | sed 's/..\/repos_gm\///g'`;
  done

Suppress ssh interactively asking your passphrase.

$ eval `ssh-agent` && ssh-add

Add and push one directory at a time (dry run).

$ (cd somepath; \
  for d in `ls -a | grep 'somemagicnumber'`; do; \
    git add --dry-run $d && \
    git commit --dry-run -m \
      "Restore binary files overwriting git-media text files ($d)" && \
    git push --dry-run origin git-lfs-migration; \
  done)

$ ^C

Add and push one directory at a time.

$ (cd somepath; \
  for d in `ls -a | grep 'somemagicnumber'`; do; \
    git add $d && \
    git commit -m \
      "Restore binary files overwriting git-media text files ($d)" && \
    git push origin git-lfs-migration; \
  done)

Things should look like this.

[git-lfs-migration xxxxxxx] Restore binary files overwriting git-media
text files (foo)
 2 files changed, 0 insertions(+), 0 deletions(-)
 rewrite somepath/foo/bar.png (100%)
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 24.55 MiB | 237 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
To git@github.com:orgfoo/repos_lfs
 * [new branch]      git-lfs-migration -> git-lfs-migration
...

(Actually we didn't eventually take this workaround in our case due to some site specific problem, but it may work in other cases.)

Other Articles