Git Superproject and Submodules: A Real World Example (LuaDoc with CMake)

git-logo_textmedium170px-cmake_logo_textmediumlang-128_textmedium


In my previous entry on Git Superprojects and Mercurial Forests, I didn't have any examples. I meant to do that, but didn't finish that. Having a horrible memory for these things, I regret that very much because it meant I didn't have a good reference to go back to.

Not wanting to repeat that mistake again, I have implemented a Git Superproject and Submodules for a real world project that I needed to get fixed. Furthermore, the project is relatively lightweight and potentially cross-platform, so I have designed the example so you can participate too by accessing a public repository.

The project is LuaDoc. It is a documentation generator that is similar to Doxygen, except that it is designed explicitly for Lua. (Doxygen currently lacks Lua support.)

Unfortunately for me, the straight out of the box build system didn't do me much good because the Makefiles that come with all the projects are Mac OS X framework unaware and also Universal Binary unaware. And I could only find prebuilt binaries for Windows. (I was disappointed I couldn't find a Debian Etch package.) Furthermore, LuaDoc has a handful of dependencies: Lua, LuaFileSystem, and LuaLogging. LuaLogging depends on LuaSocket and LuaSQL. 

So to fix this, I decided to bite the bullet and reimplement the build system. To do this, I used CMake since it is cross-platform (as Lua is) and has knowledge of Mac OS X Frameworks and Universal Binaries. I resurrected a technique I proposed in the early days of the OpenSceneGraph CMake adoption to coordinate independent projects. I introduced a notion of a Unification Script which is a specially written CMake script that sits at the directory above all the projects and coordinates the build for all the subprojects. The subprojects have their own self-contained CMake build scripts so they can be built independently, but I introduce a few special hooks to detect if they are being coordinated by the Unification Script. If so, the Unification Script can provide additional information so the separate projects know how to work together instead of depending on finding global system pre-installed components. I originally implemented the Unification Script to handle OpenSceneGraph's dependencies on Producer and OpenThreads, but the solution was rejected after Producer was dropped and SVN externals was used to blend OpenThreads inside the OSG source tree. (Almost nobody uses OpenThreads outside of OSG, so nobody notices when the stand-alone OpenThreads build system breaks.)

With 6 subprojects to coordinate, the idea of a Unification Script seemed useful.

So the hard part was to implement CMake build systems for each project. After doing that, I created a regular public Git repository for each one. They can be found here:

Git Repositories

Lua:

http://www.assembla.com/spaces/lua/trac_git_tool

LuaSocket:

http://www.assembla.com/spaces/luasocket/trac_git_tool

LuaSQL:

http://www.assembla.com/spaces/luasql/trac_git_tool

LuaFileSystem:

http://www.assembla.com/spaces/luafilesystem/trac_git_tool

LuaLogging:

http://www.assembla.com/spaces/lualogging/trac_git_tool

LuaDoc:

http://www.assembla.com/spaces/luadoc/trac_git_tool


Keep in mind that each of these projects are self-contained with their own respective CMake build systems. Since downloading and building 6 independent projects can be a significant number of steps, this is where a Superproject or Forest can be very useful. The idea is to coordinate all 6 projects into a single aggregate project. This aggregate project will also contain a Unification script to coordinate the build process. 

To create the SuperProject, I adapted the tutorial found here:

http://git.or.cz/gitwiki/GitSubmoduleTutorial


Creating a SuperProject:

The first step is to create a new SuperProject directory and create a regular Git repository out of it. This is where I add my Unification script and other support files that belong to the SuperProject as a whole, and not the individual subprojects.

$ mkdir LuaDocSuperProject
											$ cd LuaDocSuperProject/
											$ cp /Location_of_Unification_Script/CMakeLists.txt .
											$ cp /Location_of_some_notes/Notes.txt .
											$ cp /Location_of_uninstall_scr/cmake_uninstall.cmake.in .
											$ git init
											$ git add .
											$ git commit -m "Initial SuperProject repository creation"
											$ git remote add origin git@git.assembla.com:luadocsuperforest.git
											$ git push origin master
											


You can find the repository at Assembla I pushed to located here:

LuaDoc Git SuperProject Repository

http://www.assembla.com/spaces/luadocsuperforest/trac_git_tool


To test the repository, I remove the repository and re-clone it from Assembla.

$ cd ..
											$ rm -rf LuaDocSuperProject/
											$ git clone git@git.assembla.com:luadocsuperforest.git LuaDocSuperProject
											$ cd LuaDocSuperProject/
											


With everything there, the next step is to add the submodule projects to this repository using the "git submodule" command. Unlike the Git tutorial, I add the public server URLs instead of a local copy.

$ git submodule add git@git.assembla.com:lua.git lua
											$ git submodule add git@git.assembla.com:luasocket.git luasocket
											$ git submodule add git@git.assembla.com:luasql.git luasql
											$ git submodule add git@git.assembla.com:luafilesystem.git luafilesystem
											$ git submodule add git@git.assembla.com:lualogging.git lualogging
											$ git submodule add git@git.assembla.com:luadoc.git luadoc
											

 

You will see all the submodule directories and files are now inside. The next thing to do is commit the change and then push it back to the server.
$ git commit -m "Added Submodules"
											$ git push origin master

 

Your Turn:

And that's it for creating. Now you can join in. You can clone my SuperProject and try it for yourself. 

Unfortunately, there is one extra step you will need to take. You need an ssh-key to access my repository. It is possible to access without ssh, via 'git clone git://git.assembla.com/luadocsuperforest.git', but because I used 'git submodule add' with the URLs that use access through ssh-keys (so I can write back changes to the public repository), if you clone without a key, you will get the SuperProject, but will be unable to get the submodules as you will be asked for a key. I don't see a clean way of changing this to work with non-ssh access without restricting the ability to push back changes.

I have uploaded an ssh key-pair so you can access the repository. The key only permits read-only access so you won't be able to push changes with it.

http://www.assembla.com/spaces/luadocsuperforest/documents

To use, you want to grab the 'private ssh key' ('AssemblaPublicKey' without the .pub extenstion). To use, place this in your .ssh directory (with suggested limited permissions of 0600). Use 'ssh-add AssemblaPublicKey' to add the key. The password is 'AssemblaPublicKey' (no quotes).


Once the key is activated, you should be able to clone the repository:

$ git clone git@git.assembla.com:luadocsuperforest.git MyLuaDocSuperProject
											$ cd MyLuaDocSuperProject


You will notice the submodule directories are empty. There are 2 more steps for a SuperProject:

$ git submodule init
											$ git submodule update


As the Git tutorial says, notice that inside the submodules, if you do a "git branch", you are not on a branch, so you want to do a "git checkout master" if you want to make a change. For example:

$ cd lua
											$ git checkout master
											# Make some change
											$ git commit -a -m "Updated submodules."
											$ git push  # only if you have write access
											


Then you must update the superproject as well:

$ cd ..
											$ git add lua
											$ git commit -m "Updated submodule lua."
											$ git push  # only if you have write access
											

Note the Gotchas section of the tutorial. Do not do a "git submodule update" until all submodules have published their changes.


Building via CMake:

Now for actually building LuaDoc with the CMake Unification script. As usual with CMake, you are encouraged to do out-of-source builds. So starting at the directory about MyLuaDocSuperProject:

$ mkdir BUILD
											$ cd BUILD
											$ ccmake ../MyLuaDocSuperProject
											# Configure with the GUI and Generate.
											$ make
											$ make install  # may need to run with higher access
											


If all works well, everything should build and things will be installed. To uninstall, I have implemented an uninstall target so you can just do:

$ make uninstall  # may need to run with higher access
											
There is also a file generated in your BUILD directory called install_manifest.txt which lists all the files installed in case something is broken.


Running:

Make sure you set up your environment corrently, e.g.:

$ export LUA_CPATH='/usr/local/lib/lua/5.1/?.so;;'
											$ export LUA_PATH='/usr/local/share/lua/5.1/?.lua;;'
											$ export PATH=/usr/local/bin:$PATH

The executable to LuaDoc is called luadoc (luadoc.exe for Windows). Assuming your path is setup correctly, you should just be able to type that in and run it. You will get a prompt of different command switches and arguments you are expected to pass. Find a Lua script and give it a try.


If you are having problems, to test if Lua and the modules were installed correctly, you might try running lua and loading LuaSocket:

$ lua
											Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
											> socket = require("socket")
											> print(socket._VERSION)
											LuaSocket 2.0.2

If lua runs, but the module fails to load, there is probably something wrong with your LUA_*PATH. (Or something was installed to the wrong location.)


Next Time:

For comparison, we will repeat this setup using the Mercurial Forest extension in the sequel article entitled: Mercurial Forests: A Real World Example (LuaDoc with CMake).


(Update 2009/09/04:  New entry on Mercurial Subrepos)


Copyright © PlayControl Software, LLC / Eric Wing