Splitting a git repository into two projects and reintegrating them

I have a Django app which is proving very useful so I need to split it out into its own repository so that I can share it between multiple projects. As the client (CCCS), generously and sensibly keep their non sensitive project code as open source, I can present this as a real example. The app is the 'documents' app which manages an S3 repository containing many thousands of documents.

This is what I need to do:

  1. Create a new repository/project for the app
  2. Replace the app in the source project with an appropriate link to the new separate app
  3. Add the app to other projects.

To achieve this I will use the fairly new 'git subtree' command. Ultimately, the app might be installed using pip but, for the time being, I want to be able to have several projects use the latest code as necessary.

(cccs)~/wk/cccs/split $ git clone git@github.com:cccs-web/core.git
Cloning into 'core'...
remote: Counting objects: 5126, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 5126 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5126/5126), 20.69 MiB | 149.00 KiB/s, done.
Resolving deltas: 100% (2112/2112), done.
Checking connectivity... done.
(cccs)~/wk/cccs/split $ cd core
(cccs)~/wk/cccs/split/core $ git checkout documents
Branch documents set up to track remote branch documents from origin.
Switched to a new branch 'documents'
(cccs)~/wk/cccs/split/core $ git subtree split -P documents -b documents_only
Created branch 'documents_only'
c742072c69c6a40c821ac4722f4b2d0965e11e83
(cccs)~/wk/cccs/split/core $ cd ..
(cccs)~/wk/cccs/split $ mkdir documents
(cccs)~/wk/cccs/split $ cd documents
(cccs)~/wk/cccs/split/documents $ git init
Initialized empty Git repository in /home/paul/wk/cccs/split/documents/.git/
(cccs)~/wk/cccs/split/documents $ git pull ../core/ documents_only
remote: Counting objects: 197, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 197 (delta 106), reused 160 (delta 103)
Receiving objects: 100% (197/197), 302.29 KiB | 0 bytes/s, done.
Resolving deltas: 100% (106/106), done.
From ../core
 * branch            documents_only -> FETCH_HEAD

Now I have a nice new repo with just the documents stuff in it at ~/wk/cccs/split/documents. At this stage I create a new repo to hold it on GitHub so I can hook it up. Then I hook it up thus:

(cccs)~/wk/cccs/split/documents $ git remote add origin git@github.com:cccs-web/docmeta.git
(cccs)~/wk/cccs/split/documents $ git push --set-upstream origin master
Counting objects: 197, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (74/74), done.
Writing objects: 100% (197/197), 302.29 KiB | 0 bytes/s, done.
Total 197 (delta 106), reused 197 (delta 106)
To git@github.com:cccs-web/docmeta.git
 * [new branch]      master -> master
Branch master set up to track remote branch master from origin.

Now I have a new docmeta project for the document manager (I'll sort out the naming of the app later). Next I need to fix the source repo so that it no longer has the documents files for itself but links to my new repo appropriately instead. The documents branch on the cccs/core repo has been merged into staging but the initial deployment will be to another site, which is why it is not yet in the master branch. We'll be making all of our updates to the documents branch. Lets start by getting rid of the old stuff:

(cccs)~/wk/cccs/split/core $ git rm -rf documents
rm 'documents/__init__.py'
rm 'documents/admin.py'
rm 'documents/importer.py'
rm 'documents/migrations/0001_initial.py'
rm 'documents/migrations/0002_auto__add_bibtexentrytype__add_cccsentrytype__del_field_document_creat.py'
...
(cccs)~/wk/cccs/split/core $ git commit -m "Removed old documents folder"
[documents cd0fad9] Removed old documents folder
 31 files changed, 34745 deletions(-)
 delete mode 100644 documents/__init__.py
...

Now we have to add in our subtree project appropriately. We'll rename the app to docmeta in the process because using the documents subtree causes conflicts because I named the working branch 'documents' (meh).

(cccs)~/wk/cccs/split/core $ git remote add -f docmeta_origin https://github.com/cccs-web/docmeta
Updating docmeta_origin
warning: no common commits
remote: Counting objects: 200, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 200 (delta 107), reused 199 (delta 106)
Receiving objects: 100% (200/200), 302.57 KiB | 168.00 KiB/s, done.
Resolving deltas: 100% (107/107), done.
From https://github.com/cccs-web/docmeta
 * [new branch]      master     -> docmeta_origin/master

How do I update the cccs repository with docmeta changes?

Lets make a change and have a go:

(cccs)~/wk/cccs/split/documents $ git add README.md 
(cccs)~/wk/cccs/split/documents $ git commit -m "Added readme"
[master 6c6469d] Added readme
 1 file changed, 3 insertions(+)
 create mode 100644 README.md
(cccs)~/wk/cccs/split/documents $ git push
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 337 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To git@github.com:cccs-web/docmeta.git
   c742072..6c6469d  master -> master

Now, thanks to our remote setting, to pull the latest change into cccs, I go to the docmeta_master branch and pull. Then I can merge those changes into any other branch as appropriate:

(cccs)~/wk/cccs/split/core $ git checkout docmeta_master
Switched to branch 'docmeta_master'
Your branch is up-to-date with 'docmeta_origin/master'.
(cccs)~/wk/cccs/split/core $ git pull
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (1/1), done.
remote: Total 3 (delta 2), reused 3 (delta 2)
Unpacking objects: 100% (3/3), done.
From https://github.com/cccs-web/docmeta
   3a6de28..9e4ec79  master     -> docmeta_origin/master
Updating 3a6de28..9e4ec79
Fast-forward
 README.md | 1 +
 1 file changed, 1 insertion(+)
(cccs)~/wk/cccs/split/core $ git checkout documents
Switched to branch 'documents'
Your branch is ahead of 'origin/documents' by 2 commits.
  (use "git push" to publish your local commits)
(cccs)~/wk/cccs/split/core $ git merge --squash -s subtree --no-commit docmeta_master
Squash commit -- not updating HEAD
Automatic merge went well; stopped before committing as requested
(cccs)~/wk/cccs/split/core $ git commit -m "Merged in docmeta update"
[documents ea3fdb4] Merged in docmeta update
 1 file changed, 1 insertion(+)

How do I push changes I make in the cccs repo up into the docmeta repo?

This will be an irrestistable temptation with the files just sitting in the repo as a subtree. Lets change the readme there and see about pushing it back into the docmeta repo:

(cccs)~/wk/cccs/split/core $ echo 'More detail please!' >> docmeta/README.md 
(cccs)~/wk/cccs/split/core $ git add docmeta/README.md 
(cccs)~/wk/cccs/split/core $ git commit -m "Updated docmeta/README.md"
[documents 9426ca2] Updated docmeta/README.md
 1 file changed, 1 insertion(+)
(cccs)~/wk/cccs/split/core $ git checkout docmeta_master
Switched to branch 'docmeta_master'
(cccs)~/wk/cccs/split/core $ git merge -s subtree documents
Waiting for Emacs...
Merge made by the 'subtree' strategy.
 README.md | 1 +
 1 file changed, 1 insertion(+)
(cccs)~/wk/cccs/split/core $ git push docmeta_origin HEAD:master
Username for 'https://github.com': pwhipp
Password for 'https://pwhipp@github.com': 
Counting objects: 3830, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1975/1975), done.
Writing objects: 100% (3827/3827), 11.78 MiB | 84.00 KiB/s, done.
Total 3827 (delta 1643), reused 3763 (delta 1599)
To https://github.com/cccs-web/docmeta
   9e4ec79..1580127  HEAD -> master

Last but not least, how do other developers bring their repos in line?

The good news here is that they don't have to do anything. A pull will bring in the update and they can push changes as normal.

However, if they make changes to the docmeta app and want to push those up to the docmeta repo, they will need to set up the branch and split their changed content into it so that the above notes can be used to push the changes:


Most of the information for this post came from the git book and from this stack overflow question.


Comments

There are currently no comments

New Comment

required

required (not published)

optional

Australia: 07 3103 2894

International: +61 410 545 357