Skip to content
Print

In Search of a Better Build System

26
Nov
2007

There’s a consistent problem with developing applications of any reasonable size, a problem which has dated back even before the early days of C.  The problem is that any application of significance will be composed of several source files.  In fact, reasonable applications are often found to be composed of thousands if not hundreds of thousands of files.  Back in the day, it was felt (for some reason which escapes me) that it would be poor practice to type “gcc -Wall -o filename.o filename.cpp” several thousand times every time the app needed to be recompiled.

So from very early on, developers have been writing tools to aid in the build process.  Some of these tools (most of them) were somewhat ad hoc and specialized.  The most common example which springs to mind is a simple script, which handles the compilation:

#!/bin/sh
 
for f in *.c; do
  name=`echo $f | sed 's/.c//'`
  gcc -Wall -o ${name}.o ${name}.c
done

The limitations of such an approach should be obvious.  For one thing, you can only use this script on a single directory which contains all source files.  This is very rarely the case.  More importantly, there is no linking or dependency checking taking place.  This means that with the exception of very simple applications, this script will outright fail every time.  Of course, you could modify the script extensively to hard-code the dependency information, check for file modification, etc.  However, this would be a long, dull process which would have to be repeated for every application you write.  Not a very productive way to spend your time.

And so was born make.  Make has a number of advantages over the hand-scripted method.  It allows for (fairly) easy dependency specification, it will only compile modified files, it lets you ensure everything happens in the proper order, it’s more maintainable, etc.  However at its core, Make is almost exactly a wrapper around the hand-script method.  As such, it suffers from many of the same limitations, such as a cryptic syntax and a dependence on the underlying shell.  Make is a far cry from hand scripting everything, but it’s hardly the silver bullet developers were looking for.

So as the years went by, more and more solutions were devised, though few of them caught on to the extent that Make had.  To this day, Make is still the de facto standard for C and C++ build systems.  Its dominance is so pervasive that I have even found Java applications which are built using Make, though thankfully these are far and few between.  With Java on the scene, attention turned to a new effort which attempted to unseat Make as the reigning champion of the build tools: Ant.

Ant based its syntax on XML, breaking completely with Make’s bash roots and focusing on the task rather than the dependency.  Because of this clean break, and due to the fact that the Ant interpreter itself is written in Java, Ant is entirely platform agnostic.  An Ant build script written for a Java project can be run on any platform, anywhere (as long as Java is installed).  This immediately gave it a huge boost over Make as it finally enabled developers on platforms such as Windows and MacOS 9 (and earlier), platforms without the advantages afforded by a real shell.  Ant’s rise to dominance in the field of Java build systems was so rapid and so completely unchallenged that it still remains the “proper” way to build a Java application.  Every Java developer has Ant installed, and as such it has become something of a lingua franca in build script land.

Unfortunately Ant, like Make suffers from a number of shortcomings.  Its XML syntax for one, while instantly recognizable and familiar to 99% of developers on this planet, poses problems with verbosity and expressiveness.  For example, Ant doesn’t provide any real mechanism for variables, bona fide procedures or any way to execute arbitrary code without a clean break into Java (a custom Ant Task).  While this tends to keep build scripts somewhat uniform and easily understandable (when you’ve seen one build.xml, you’ve seen them all), it also forms a crippling limitation in many ways.  I’ve used Ant a lot in my time, and let me tell you it can be a real pain.  For simple builds (javac a bunch of source files, copy one or two resources, zip the result, etc) it’s quite sufficient, but headaches set in when dealing with anything complex like chained builds, subprojects or library dependency management.  You can make it work, but it’s not pretty.

The Maven project was started to try to address some of these problems (among other things).  Maven provides full-stack dependancy management (even resolving and downloading third-party libraries), build management, conventions enforcement, IDE interop and so on.  A number of people would say that Ant is completely superceded by Maven, and that Maven is the only way to go for any new Java project.  Unfortunately, like so many successful Java projects of its day (a few examples Spring to mind), Maven refused to maintain its focus.  Instead of being an incredibly simple build system and dependency management tool, Maven has tried to branch out and become the all-encompassing tool to solve everything.  I know I haven’t even scratched the surface of its capabilities in my limited exposure, but I can say that I’ve seen enough.  Maven is amazing, but way way to invasive for my tastes.  It has a knack of making the simple things cryptic, the hard things harder and the complex things impossible.  (by impossible I mean without resorting to hackery like invoking Ant from within Maven or calling out to a shell script)  Now I realize flame wars have been fought on this very subject, but I have to conclude that Maven is just too much and too overwhelming for easy use (and hence, wide adoption).

Fortunately for me, the rise of dynamic languages has brought about some wider options.  Ruby in particular has become a favorite for many different build tool projects (Rake, Raven, etc).  Most interesting is the effort underway to provide a “non-sucky Maven”.  The Buildr project, currently in incubation at Apache, is basically seeking to be a build system which enables trivial application of the most common case (builds and dependencies), as well as the flexibility of a Turing-complete language (Ruby) to make possible just about any build task, no matter how esoteric.

Buildr’s promise is indeed alluring, and at first glance it seems to deliver.  The DSL syntax of the build file is intuitive and easy to grasp.  With this it bundles the full power of Maven, allowing it to be a drop-in replacement for any pre-existing Maven project.  Well, almost.  Buildr doesn’t allow for things like dependency checkout from a source code repository.  It also retains one of Maven’s biggest failings in that it unduly enforces a rigid directory structure.  While it is possible to override this restriction, Buildr’s documentation isn’t exactly clear on how, and to be honest I still haven’t figured out how to get some things working.  Buildr is promising, but not perfect.

Another flaw suffered by all of the new, “avant garde” build tools is that not all of them can be expected on every developer’s machine.  Back in the days of Make and Ant, every developer knew that every other developer could handle a build.xml file and use it to get a fully functional build out the other end.  Unfortunately, while Maven has made tremendous strides in popularity, it is still no where near as ubiquitous as Ant.  Buildr is even less common, additionally requiring the separate installation of a full Ruby runtime, as well as the “buildr” gem.  These considerations are less significant for a small commercial product, where all of the developers are in close contact and outside interference is rare.  However, for the open-source project, standardization in the required tools is critical, otherwise new developers would have no way to contribute.

Unfortunately it’s a bit of a Catch-22.  Even assuming Buildr manages to make good on its promise of being “a build system that doesn’t suck”, it has to gain in popularity before it will be accepted as the de facto standard for Java project builds.  But to gain popularity, Buildr must be accepted by the community.  It’s a tightly knit, closed circle driven by managers remaining content with “the way it’s always been done” and developers refusing to chance the success of their project on the hope that all parties concerned will also show forward thinking in their tool set.  I really don’t envy the Buildr project leads in their task to promote their tool.

So where does this leave me?  Practically speaking, I’m still going to stick with Ant.  Buildr may be interesting, and Maven may be powerful, but neither is the standard yet.  Perhaps even more importantly: I’m lazy.  I know Ant.  I know it very well and I would have to see some pretty clear benefits (and an easy introductory road) to switch my build system of choice.  Right now, I don’t see either of those.  Maven is infamous for having a very steep learning curve.  And as I mentioned before, Buildr’s documentation leaves something to be desired.

I want to help usher in the next era of Java development as much as the next guy.  But I’m not willing to sacrifice the now for the sake of a future which may take a totally different form when it arrives.  So I will (reluctantly) stick with my old standby.  Perhaps someday I’ll come across a build tool which really impresses me enough to make me switch.  Until then you can continue to listen to me whine and complain about the difficulties of whipping Ant into shape.  How lucky for both of us.  :-S

Comments

  1. Another very nice buildsystem is OMake ( http://omake.metaprl.org/ ).

    It keeps a lot of makes syntax and strenghts while beeing a clean functional and object oriented language. It can do cool tricks like clean beeng “delete all artefacts can be regenerated by omake”.

    Nils Monday, November 26, 2007 at 3:55 am
  2. Build systems require a LOT more than dependency tracking that make provided, especially with the advent of continuous integration. I consider a build environment for any significantly large project to be a software project of its own and Make is its “assembly language”. We need higher orders of abstractions and Ant’s XML just didn’t cut it. I’ve gone through several systems the last few years (Scons was a strong “almost got it”) and have finally settled on Boost.Build v2 ( http://boost.org/boost-build2/ ) which has an extension of Perforce’s Jam system and excellent environmental awareness & extensibility. Its being re-written in Python and has reportedly gotten even faster. The DSL for Boost’s bjam tool supports the declarative syntax that is more appropriate for a build system’s rules than regular programming languages. Needs better examples on extensibility in the documentation but its been adopted by some critical projects and organizations and improves on a weekly basis. I think this one’s got traction…

    Ben Scherrey Monday, November 26, 2007 at 8:01 am
  3. Thanks.
    I’ve settled on
    1. using ant for build/running tests
    2. use maven for dependency management only, by using ant tasks that can pull out the maven dependencies- beyond that, maven never feels like I get the time I put into it back out.
    3. use atlassian’s cruise control : bamboo

    Looking forward to what the ant/ivy partnership may bring…

    James Law Wednesday, November 28, 2007 at 3:10 pm
  4. FWIW, I happily use Automake for my Java applications. It does the job, while getting out the way, and letting me concentrate on the code, rather than figuring out the build system. It’s as primitive as make is, but it does the one job well that I care about from a build system: let me build/rebuild the code as I need it, while making it easy for others to repeat what I’ve done, and making it easy for me to ship source code tarballs letting them do that.

    It doesn’t have dependency management, but that’s what apt-get build-deps is for, without having to hack it into the build tool. It doesn’t have explicit JUnit support, but since automake files are very similar to regular makefiles, just a lot simpler, it’s trivial to make it run junit, or anything else without having to hack the build tool first. It does not have CI support, but there is no lack of CI tools like buildbot that take care of all that, without me having to hack the build tool to support it.

    I guess my point is that a good build system should do one thing well. That’s entirely sufficient.

    Dalibor Topic Thursday, November 29, 2007 at 6:25 am
  5. @Dalibor

    A good point about the purpose for a build system. In this respect, I think automake does a much better job than many modern, java-specific build systems (showing its maturity). Unfortunately, most people (myself included) find automake fairly cryptic, not to mention inflexible in some areas such as build packaging and so on.

    apt-get build-deps is great for C/C++ apps, but it just doesn’t cover everything with Java. There are so many third-party Java libraries, and a given application may use dozens of them. Obtaining them all in the correct version is a pain to say the least. To maven’s credit, it does a good job in solving this problem…most of the time. The more obscure libraries are much harder to deal with in a Maven build. I’m sort-of looking forward to playing with Ivy/Ant to see how it holds up in this respect.

    Daniel Spiewak Thursday, November 29, 2007 at 9:46 am
  6. Nothing wrong with waiting until you need something before getting it. Just because many of our tools are “free software” doesn’t mean that we ‘infovores’ shouldn’t be wary of unchecked consumerism and greed! In fact we need to be even more careful of gorging because the barrier-to-consumption is so low.

    I wish you could be more specific in which ways you feel that Maven isn’t maintaining focus. I’ve been reading about Maven, although haven’t yet used it; I have a couple of strong developer friends who very much like Maven – they feel that the rigidity of the file structure is a small price to pay for the convenience the platform offers. And from the look of it, I tend to agree (although I have to say the 22-step build model is a bit intimidating!)

    WRT directory layout rigidity, there is something to be said for a one-size-fits all convention when it “doesn’t really matter”. Who cares where the directories are? If the world really agreed on suitable conventions our tools could nicely present our files to us however we like (and indeed, most IDEs can already do this).

    Josh Rehman Saturday, December 15, 2007 at 9:50 pm
  7. http://www.nabble.com/Project-Layouts-td14588782.html

    I didn ‘t test it. But Assaf announced the Project Layouts feature.

    Alexandre Bairos Thursday, January 3, 2008 at 7:38 am

Post a Comment

Comments are automatically formatted. Markup are either stripped or will cause large blocks of text to be eaten, depending on the phase of the moon. Code snippets should be wrapped in <pre>...</pre> tags. Indentation within pre tags will be preserved, and most instances of "<" and ">" will work without a problem.

Please note that first-time commenters are moderated, so don't panic if your comment doesn't appear immediately.

*
*