#submodules

List submodules in a git repository

I have a git repository that has several submodules in it. How do I list the names of all the submodules after git submodule init has been run?

The git submodule foreachcommand could echo the names of the submodule, but that only works once they have been checked out which has not happened after the init step. There are more steps in the chain that need to happen before they can be checked out…

I ran into an interesting issue this past week.  When I was searching around for solutions, I didn't come across any, so I wanted to share this here so I'll have documentation for the future and so that others might come across this if they need it.  I have my thesis hosted in a private github repository (thanks to the awesome student package github offers).  Unfortunately, my advisor wanted my code to be in our research group's repository, which is:

huge

not structured remotely like my current repo

full of things that I don't care about and don't need for my daily work (i.e. other peoples' research)

not hosted on a public facing server

On top of this, it has none of my history.  I looked into doing a subtree merge to solve this, but I would still have the problem of my Github repo stagnating (as the research groups repo is several gigabytes and I believe they stop you from putting anything too huge on github) as well as the changes to directory structure and the fact it would no longer be public-facing (i want this because i could make it NOT private at some point, as well as do things like push and pull from home without a vpn).

The best way would probably be to use what's called a submodule, which lets you map a section of a repo to another repo and push/pull at will, but my advisor doesn't like this as it complicates the repo, and if my secondary repo is ever taken down (because I delete my git account or something) then the group repo may end up with an old version and no way to get the updated, now deleted, code.  Another option is a subtree merge, which unfortunately would fully merge the repos and would result in my github filling with the huge group repo.

To solve this I thought about how git is structured, and came up with a solution: Pull from github into my local copy of the group's repo.  I never push to github from the group's repo and continue working in my current setup, pulling into the group repo from time to time.

To set this up, I moved the contents of my git directory into another directory to avoid merge conflicts and to keep my files together when they were pulled into the group repo.  To do this I created a new directory and moved the contents into the folder.  This causes one of your commits to show a move history, which can be undesirable but I didn't care.  To fix this (if it's undesirable) I found resources on git filter-branch, but I decided it really didn't matter to me.  

My git directory was actually in a bad place, so I additionally had to pull in directories from the parent directory, before moving my entire repository folder up and deleting the original (Thesis/Assets/<repo here> moved to Thesis/<repo here>/Assets by moving the contents of assets into Thesis/Assets/Assets, renaming Assets to Thesis, pulling contents of Thesis into the new subdirectory Thesis, then finally replacing the parent with the child).

This then made it a simple task to fetch from github into the group's repo and merge into my branch there.  When I want to update my code, I push to github from my working repo and pull from github into the group repo.  Again, this mimics the behavior of submodules as I understand them, but in this case they weren't quite an option.  I'll try to argue my case but for now this is it.

Some downsides include:

I have to be careful to only pull into the group repo or else I'll end up with more copies of the huge repo

I have to manually pull from github into the group repo

Same problems as submodules in that I have to make sure the group repo is up to date

When you add a git folder folder into git folder folder - as you might do for using other open source libraries in your code - you'll find that the .git meta folders will interfere with each other. If you want to keep all libraries attached to their respective git origins you need to add them as Submodules. The difference is that those submodule library folders need to be treated independently from your own code. You need to initialise and fetch their content separately. Here's how it works.

Add a git folder as submodule (instead of simply cloning it into your existing git folder = bad), example ...

git submodule add [email protected]:MrRio/jsPDF.git jsPDF

Go to the folder of this library. You'll find it's still empty. Init and Fetch its contents

git submodule init git submodule update

That's it. Now you can commit and publish your own topmost folder while keeping the references to the other containing git repositories intact.

With Subversion, you can check out a new svn repo into any directory.  Even if that directory is an svn repo.  It doesn't really matter,  They can all be embedded in each other, and everything works fine.

Git doesn't support that, since there is only one .git folder for the whole repo instead of one per directory.  It gets confuse-ed.

They solve that with 'submodules', which is an overcomplicated, hard-to-use solution to the problem, just like everything git does.  I love them very much.

I have a common pattern that I like to use at work; something like this:

/          <----- 'master' repository   /docs   /releases   /src        <----- a different repository (maybe)!   /dependencies     /toolchain       <------ a different repository!     /flash_burner  <------ yet another repository!

The /src directory might be a different repository, specifically if it was given to us by a customer with version control history, and we want to retain that.

Each directory under /dependencies is a separate repository, because those are source branches that are required to build/use this project, but are developed independently and possibly shared between multiple projects.

In git, this isn't all that hard to setup.  You just add each sub-repository as a git submodule:

# git submodule add <git server>:flash_burner ./dependencies/flash_burner # git submodule init ./dependencies/flash_burner # git submodule update

Now you have a different git repo checked out into the directory ./dependencies/flash_burner.  Now you should also probably add... something.  'commit -a', and push if you have a remote server.  Because now you have a .gitmodules file tracking all of your submodules, and that's just a regular file in the repo now.

But, alas, checking this whole conglomeration out is kind of a bitch.  It looks something like:

# git clone <remote repo, the parent of everything> # git submodule init <path to a submodule> # git submodule init <path to next submodule...> ... # git submodule update

And then you still aren't necessarily done.  A submodule is linked to a single revision, the one that was the head when you created it.  If you want to actually track the head, you need to cd to each submodule and 'git checkout master'.

So, the point of this whole post is that you don't have to cd to each repo, actually, just run this nonsense:

git --git-dir=<path to submodule> --work-tree=<path to submodule> checkout master

And that point of that, dear reader, is that now you make a 'checkout_submodules.py' script at the top-level of your master repo, which contains a list of all submodules, inits them all, updates them all, and checks out the head.