- Code Commit - http://www.codecommit.com/blog -

In Search of a Better Build System

Posted By Daniel Spiewak On November 26, 2007 @ 1:06 am In Java | 7 Comments

There’s a consistent problem with developing applications of any reasonable size, a problem which has dated back even before the early days of C.  The problem is that any application of significance will be composed of several source files.  In fact, reasonable applications are often found to be composed of thousands if not hundreds of thousands of files.  Back in the day, it was felt (for some reason which escapes me) that it would be poor practice to type “gcc -Wall -o filename.o filename.cpp” several thousand times every time the app needed to be recompiled.

So from very early on, developers have been writing tools to aid in the build process.  Some of these tools (most of them) were somewhat ad hoc and specialized.  The most common example which springs to mind is a simple script, which handles the compilation:

for f in *.c; do
  name=`echo $f | sed 's/.c//'`
  gcc -Wall -o ${name}.o ${name}.c

The limitations of such an approach should be obvious.  For one thing, you can only use this script on a single directory which contains all source files.  This is very rarely the case.  More importantly, there is no linking or dependency checking taking place.  This means that with the exception of very simple applications, this script will outright fail every time.  Of course, you could modify the script extensively to hard-code the dependency information, check for file modification, etc.  However, this would be a long, dull process which would have to be repeated for every application you write.  Not a very productive way to spend your time.

And so was born make.  Make has a number of advantages over the hand-scripted method.  It allows for (fairly) easy dependency specification, it will only compile modified files, it lets you ensure everything happens in the proper order, it’s more maintainable, etc.  However at its core, Make is almost exactly a wrapper around the hand-script method.  As such, it suffers from many of the same limitations, such as a cryptic syntax and a dependence on the underlying shell.  Make is a far cry from hand scripting everything, but it’s hardly the silver bullet developers were looking for.

So as the years went by, more and more solutions were devised, though few of them caught on to the extent that Make had.  To this day, Make is still the de facto standard for C and C++ build systems.  Its dominance is so pervasive that I have even found Java applications which are built using Make, though thankfully these are far and few between.  With Java on the scene, attention turned to a new effort which attempted to unseat Make as the reigning champion of the build tools: Ant [1].

Ant based its syntax on XML, breaking completely with Make’s bash roots and focusing on the task rather than the dependency.  Because of this clean break, and due to the fact that the Ant interpreter itself is written in Java, Ant is entirely platform agnostic.  An Ant build script written for a Java project can be run on any platform, anywhere (as long as Java is installed).  This immediately gave it a huge boost over Make as it finally enabled developers on platforms such as Windows and MacOS 9 (and earlier), platforms without the advantages afforded by a real shell.  Ant’s rise to dominance in the field of Java build systems was so rapid and so completely unchallenged that it still remains the “proper” way to build a Java application.  Every Java developer has Ant installed, and as such it has become something of a lingua franca in build script land.

Unfortunately Ant, like Make suffers from a number of shortcomings.  Its XML syntax for one, while instantly recognizable and familiar to 99% of developers on this planet, poses problems with verbosity and expressiveness.  For example, Ant doesn’t provide any real mechanism for variables, bona fide procedures or any way to execute arbitrary code without a clean break into Java (a custom Ant Task).  While this tends to keep build scripts somewhat uniform and easily understandable (when you’ve seen one build.xml, you’ve seen them all), it also forms a crippling limitation in many ways.  I’ve used Ant a lot in my time, and let me tell you it can be a real pain.  For simple builds (javac a bunch of source files, copy one or two resources, zip the result, etc) it’s quite sufficient, but headaches set in when dealing with anything complex like chained builds, subprojects or library dependency management.  You can make it work, but it’s not pretty.

The Maven [2] project was started to try to address some of these problems (among other things).  Maven provides full-stack dependancy management (even resolving and downloading third-party libraries), build management, conventions enforcement, IDE interop and so on.  A number of people would say that Ant is completely superceded by Maven, and that Maven is the only way to go for any new Java project.  Unfortunately, like so many successful Java projects of its day (a few examples Spring to mind), Maven refused to maintain its focus.  Instead of being an incredibly simple build system and dependency management tool, Maven has tried to branch out and become the all-encompassing tool to solve everything.  I know I haven’t even scratched the surface of its capabilities in my limited exposure, but I can say that I’ve seen enough.  Maven is amazing, but way way to invasive for my tastes.  It has a knack of making the simple things cryptic, the hard things harder and the complex things impossible.  (by impossible I mean without resorting to hackery like invoking Ant from within Maven or calling out to a shell script)  Now I realize flame wars have been fought on this very subject, but I have to conclude that Maven is just too much and too overwhelming for easy use (and hence, wide adoption).

Fortunately for me, the rise of dynamic languages has brought about some wider options.  Ruby in particular has become a favorite for many different build tool projects (Rake, Raven, etc).  Most interesting is the effort underway to provide a “non-sucky Maven”.  The Buildr [3] project, currently in incubation at Apache, is basically seeking to be a build system which enables trivial application of the most common case (builds and dependencies), as well as the flexibility of a Turing-complete language (Ruby) to make possible just about any build task, no matter how esoteric.

Buildr’s promise is indeed alluring, and at first glance it seems to deliver.  The DSL syntax of the build file is intuitive and easy to grasp.  With this it bundles the full power of Maven, allowing it to be a drop-in replacement for any pre-existing Maven project.  Well, almost.  Buildr doesn’t allow for things like dependency checkout from a source code repository.  It also retains one of Maven’s biggest failings in that it unduly enforces a rigid directory structure.  While it is possible to override this restriction, Buildr’s documentation isn’t exactly clear on how, and to be honest I still haven’t figured out how to get some things working.  Buildr is promising, but not perfect.

Another flaw suffered by all of the new, “avant garde” build tools is that not all of them can be expected on every developer’s machine.  Back in the days of Make and Ant, every developer knew that every other developer could handle a build.xml file and use it to get a fully functional build out the other end.  Unfortunately, while Maven has made tremendous strides in popularity, it is still no where near as ubiquitous as Ant.  Buildr is even less common, additionally requiring the separate installation of a full Ruby runtime, as well as the “buildr” gem.  These considerations are less significant for a small commercial product, where all of the developers are in close contact and outside interference is rare.  However, for the open-source project, standardization in the required tools is critical, otherwise new developers would have no way to contribute.

Unfortunately it’s a bit of a Catch-22.  Even assuming Buildr manages to make good on its promise of being “a build system that doesn’t suck”, it has to gain in popularity before it will be accepted as the de facto standard for Java project builds.  But to gain popularity, Buildr must be accepted by the community.  It’s a tightly knit, closed circle driven by managers remaining content with “the way it’s always been done” and developers refusing to chance the success of their project on the hope that all parties concerned will also show forward thinking in their tool set.  I really don’t envy the Buildr project leads in their task to promote their tool.

So where does this leave me?  Practically speaking, I’m still going to stick with Ant.  Buildr may be interesting, and Maven may be powerful, but neither is the standard yet.  Perhaps even more importantly: I’m lazy.  I know Ant.  I know it very well and I would have to see some pretty clear benefits (and an easy introductory road) to switch my build system of choice.  Right now, I don’t see either of those.  Maven is infamous for having a very steep learning curve.  And as I mentioned before, Buildr’s documentation leaves something to be desired.

I want to help usher in the next era of Java development as much as the next guy.  But I’m not willing to sacrifice the now for the sake of a future which may take a totally different form when it arrives.  So I will (reluctantly) stick with my old standby.  Perhaps someday I’ll come across a build tool which really impresses me enough to make me switch.  Until then you can continue to listen to me whine and complain about the difficulties of whipping Ant into shape.  How lucky for both of us.  :-S

Article printed from Code Commit: http://www.codecommit.com/blog

URL to article: http://www.codecommit.com/blog/java/in-search-of-a-better-build-system

URLs in this post:

[1] Ant: http://ant.apache.org

[2] Maven: http://maven.apache.org

[3] Buildr: http://incubator.apache.org/buildr/

All content copyright © 2010 Daniel Spiewak. Distributed under Creative Commons.
All code (unless otherwise stated) is licensed under the BSD License.