I am about to break a promise. Almost three years ago, I promised myself that I would never write an article about monads. There are too many such articles already; so many, in fact, that people are often confused by the sheer proliferation. Everyone seems to have a different take on the subject, meaning that those attempting to learn the concept for the first time are stuck trying to reason out the commonalities between burritos, space suits, elephants and desert Bedouins.

I’m not going to add to this menagerie of confusing analogies. The fact is that none of these parallels are entirely accurate. None of them convey the whole picture, and some of them are blatantly misleading in important respects. You will never come to understand monads by pondering Mexican food and the Final Frontier. The only way to understand monads is to see them for what they are: a mathematical construct.

Math (or not)

Here’s the thing about monads which is hard to grasp: monads are a pattern, not a specific type. Monads are a shape, they are an abstract interface (not in the Java sense) more than they are a concrete data structure. As a result, any example-driven tutorial is doomed to incompleteness and failure. The only way to really understand is to take a step back and look at what monads mean in the abstract rather than the concrete. Take a look at the following Ruby snippet:

def foo(bar)
  puts bar
  bar.size
end

Just as a quick Ruby refresher, we can rewrite this code in the following way:

def foo(bar)
  puts bar; bar.size
end

Ruby has this neat convention (which is shared by most modern languages) which causes the final expression in a method to be turned into the implicit return statement. Thus, the foo method will take a parameter, print it to standard out and then return its size. Fairly simple, right?

Here’s the puzzler: what is the semicolon (;) doing? It’s tempting to say that it’s just a separator, but theoretically speaking, there’s something much more interesting going on here. Let’s switch to Scala and add some Christmas trimmings:

def foo(bar: String) = {
  ({ () => println(bar) })()
  ({ () => bar.length })()
}

Just in case you’re unfamiliar with Scala, I’d like to make it clear that we are not required to enclose every statement inside its own lambda (anonymous function). I’m just doing that to make a point.

This function does exactly the same thing as the Ruby version. Well, the parameter is a bit more constrained since we require a String rather than accepting anything defines size, but moving past that… The major difference from what we had previously is that each statement is wrapped inside its own anonymous function, which we immediately call. We can again play the same semicolon trick that we used in Ruby. However, because the statements are actually functions, we can go a step further:

def foo(bar: String) = {
  ({ () => println(bar) } andThen { () => bar.length })()
}

(note: the andThen method isn’t defined for functions of 0-arity, but we’re going to pretend that it is and that it works the same as it does for functions of one argument. If it makes you feel better, you can pretend that these are both one-argument functions taking Unit as a parameter, the theoretical implications are the same, it just requires more syntax)

Notice that we haven’t actually used the semicolon (although we could have). Instead, we’re combining two functions together and invoking them at the very end. The semantics we’re using to do the combination are such that the first function will be evaluated, then its result (()) discarded and the second function evaluated with its result returned. For those following along at home, we could easily define andThen in the following way:

def funcSyntax[A](f1: () => A) = new {
  def andThen[B](f2: () => B) = f1(); f2()
}

In a way, we have defined a method which literally encapsulates the effect of the semicolon “operator”, allowing us to apply it directly to functions, rather than dealing with it indirectly at the level of statements. That’s kind of a cool thought, but the important point is that we are first executing the first function, discarding its result and then executing the second function, returning its result.

It should be clear that we could extend this to any number of functions. For example:

def foo(bar: String) = {
  ({ () => println("Executing foo") } andThen
   { () => println(bar) } andThen
   { () => bar.length })()
}

Still with me? Congratulations, you’ve seen your first monad.

You Could Have Invented Monads! (and maybe you already have)

This certainly isn’t a monad in the traditional sense, but if we worked at it, we could show that the monadic axioms do hold. The significant point here is what this monad is doing: combining one thing together with another in sequence. In fact, this is what all monads do, deep down. You start out with Thing One, and you have a function which will (given One) will give you Thing Two. Monads let you combine Thing One and your function together, producing a final resultant Thing. Let’s look at some more code:

case class Thing[+A](value: A)

This is about the simplest container imaginable (in fact, it is precisely the simplest container imaginable, but that’s not relevant now). We can wrap up values inside of Thing, but that’s about it:

val a = Thing(1)
val b = Thing(2)

Now, let’s switch into design mode for a moment. Imagine that we find ourselves writing a lot of code which looks like this:

def foo(i: Int) = Thing(i + 1)
 
val a = Thing(1)
val b = foo(a.value)        // => Thing(2)

We’re starting with a Thing, and then we’re using the value inside of that Thing to call a function which gives us a new Thing. If you think about it, this is actually a very common pattern. We have a value, and then we use that value to compute a new value. Mathematically, this is pretty much the same as the following:

def foo(i: Int) = i + 1
 
val a = 1
val b = foo(a)              // => 2

The only difference between these is that the first version wraps everything in Thing, while the second version is using “bare” values.

Now, let’s extend our imagine just a bit and assume that we have a good reason for wrapping everything inside of Thing. There could of course be any number of reasons for this, but basically it boils down to the notion that Thing might have some extra logic which does interesting things with its value. Here’s the question: can we come up with a nicer way of going from a to b? Basically, we want to encapsulate this pattern as a more general tool.

What we want is a function which pulls the value out of Thing and then calls another function with that value, returning the result of that function call (which will be a new Thing). Since we’re good object-oriented programmers, we will define this as a method on class Thing:

case class Thing[+A](value: A) {
  def bind[B](f: A => Thing[B]) = f(value)
}

So if we have a Thing, we can pull its value out and use it to compute a new Thing, all in one convenient step:

def foo(i: Int) = Thing(i + 1)
 
val a = Thing(1)
val b = a bind foo          // => Thing(2)

Notice that this is a lot cleaner that our original version, while still performing exactly the same function. Thing is a monad.

The Monad Pattern

Any time you start with something which you pull apart and use to compute a new something of that same type, you have a monad. It’s really as simple as that. If it sounds like I’m describing almost all of your code, then good, that means you’re starting to catch on. Monads are everywhere. And by “everywhere”, I do mean everywhere.

To understand why this is, let’s look at what it is that makes Thing a monad:

val a = Thing(1)

The first thing is that I can wrap up a value inside of a new Thing. Object-oriented developers might call this a “constructor”. Monads call it “the unit function”. Haskell calls it “return” (maybe we shouldn’t try to figure out that one just yet). Whatever you call it though, it comes to the same thing. We have a function of type A => Thing; a function which takes some value and wraps it up inside a new Thing.

a bind { i => Thing(i + 1) }

We also have this fancy bind function, which digs inside our Thing and allows a function which we supply to use that value to create a new Thing. Scala calls this function “flatMap“. Haskell calls it “>>=“. Again, the name doesn’t matter. What’s interesting here is the fact that bind is how you combine two things together in sequence. We start with one thing and use its value to compute a new thing.

It’s as simple as that! If you’re like me, then you’re likely asking the following question: if it’s so simple, what’s all the fuss about? Why not just call this the “using one thing to compute another” pattern? Well, for one thing, that’s far too verbose. For another, monads were first defined by mathematicians, and mathematicians love to name things. Mathematics is all about finding patterns, and it’s hard to find patterns if you can’t affix labels once you’ve found them.

More Examples

I said that monads are everywhere (they are!), but we’ve only looked at two examples so far. Let’s see a few more.

Option

This might be the most famous monad of all, quite possibly because it’s one of the easiest to understand and by far the easiest to motivate. Consider the following code:

def firstName(id: Int): String = ...    // fetch from database
def lastName(id: Int): String = ...
 
def fullName(id: Int): String = {
  val fname = firstName(id)
  if (fname != null) {
    val lname = lastName(id)
    if (lname != null)
      fname + " " + lname
    else
      null
  } else {
    null
  }
}

Here again, we have a fairly common pattern. We have two functions (firstName and lastName) which are responsible for producing some data which may or may not be available. If the data is available, then it is returned. Otherwise, the result of these functions will be null. We then use these functions to do something interest (in this case, compute a full name). Unfortunately, the fact that firstName and lastName may or may not produce a useful value needs to be handled explicitly with a set of nested ifs.

At first blush, it seems this is the best that we can do. However, if you look very closely, you can find the monad pattern buried in this code. It’s a little more complicated than last time, but it’s still there. Let’s try wrapping everything in Thing to make it clear:

def firstName(id: Int): Thing[String] = ...    // fetch from database
def lastName(id: Int): Thing[String] = ...
 
def fullName(id: Int): Thing[String] = {
  firstName(id) bind { fname =>
    if (fname != null) {
      lastName(id) bind { lname =>
        if (lname != null)
          Thing(fname + " " + lname)
        else
          Thing(null)
      }
    } else {
      Thing(null)
    }
  }
}

See it now? As I said, monads are everywhere. Here’s the really useful bit though: every time we bind, the very first thing we do inside the function is test the value to see if it is null, why not move that logic into bind? Of course, we can’t really do that without changing Thing into something different, so we will define a new monad called Option:

sealed trait Option[+A] {
  def bind[B](f: A => Option[B]): Option[B]
}
 
case class Some[+A](value: A) extends Option[A] {
  def bind[B](f: A => Option[B]) = f(value)
}
 
case object None extends Option[Nothing] {
  def bind[B](f: Nothing => Option[B]) = None
}

If you block out everything except Some, this looks a lot like our old friend, Thing. The main difference is that Option has two different instantiations: Some, which contains a value, and None, which doesn’t contain a value. Think of None as being just an easier way of writing Thing(null).

What’s interesting is that Some and None need to have two different definitions of bind. The definition of bind in Some looks a lot like the definition in Thing, which makes sense as Some and Thing are almost identical. However, None defines bind to always return None, ignoring the specified function. How does this help us? Well, let’s return to our fullName example:

def firstName(id: Int): Option[String] = ...    // fetch from database
def lastName(id: Int): Option[String] = ...
 
def fullName(id: Int): Option[String] = {
  firstName(id) bind { fname =>
    lastName(id) bind { lname =>
      Some(fname + " " + lname)
    }
  }
}

All of those nasty if statements have disappeared. This works because firstName and lastName now return None when they fail to fetch the database record, rather than Thing(null). Of course, if we try to bind on None, the result will always be None. Thus, the fullName function returns the combination of firstName and lastName inside of a Some instance only if neither firstName nor lastName return None. If either one returns None, then the result of the whole thing will be None.

For those keeping score at home, we have “accidentally” stumbled upon Groovy’s safe-dereference operator (?.), Raganwald’s andand for Ruby and many, many more. See? Monads are everywhere.

IO

Anyone trying to understand monads will inevitably run into Haskell’s IO monad, and the results are almost always the same: bewilderment, confusion, anger, and ultimately Perl. The fact is that IO is a rather odd monad. It is fundamentally in the same category as Thing and Option, but it solves a very different problem.

Here’s the deal: Haskell doesn’t allow side-effects. At all. Functions take parameters and return values; you can’t just “change” something outside the function. As an example of what this means, let’s return to our earlier Ruby code:

def foo(bar)
  puts bar
  bar.size
end

This function takes a value, calls its size method and returns the result. However, it also changes the standard output stream. This is pretty much the same as if we had a global array hanging around which we just mutated in-place:

STDOUT = []
 
def foo(bar)
  STDOUT += [bar]
  bar.size
end

Haskell doesn’t have variables of any sort (imagine if Scala didn’t have var, or if every variable in Java were final). Since we don’t have any variables, we can’t change anything in-place. Since we can’t change anything in-place, there’s no way we can define a puts function. At least, not the puts we’re used to.

Let’s switch back to Scala. Our goal here is to define a println function which doesn’t rely on any sort of mutable state (ignoring for the moment the fact that the standard output stream is always going to be mutable, since there’s no way we can “change” the user’s screen by cloning the physical display). One thing we could do is wrap up our standard output stream as a Vector which we carry along with our functions:

def foo(bar: String, stdout: Vector[String]) = {
  val stdout2 = println(bar, stdout)
  (bar.length, stdout2)
}
 
def println(str: String, stdout: Vector[String]) = stdout + str

Theoretically, we could write all of our println-enabled functions in this way, passing in the current stdout and receiving the new state as a result. At the end of the day, the entire program would produce a result in addition to the final state of stdout, which could be printed to the screen by the language runtime.

This works (at least for println), but it’s ridiculously ugly. I would hate it if I had to write code like this, and early Haskell adopters felt exactly the same way. Unfortunately, things only get worse from there. Our trick with Vector[String] may work for the standard output stream, but what about standard in? At first blush, it seems as if the readLine function wouldn’t be as bad as println, after all, we aren’t changing anything! Unfortunately, something is clearly being changed at some level, otherwise repeated calls to readLine would yield the same result (clearly not the case).

Graphics updates, networking, the list goes on. It turns out that any useful program is going to need to have side-effects, otherwise there’s no way for all of that usefulness to get outside the program where we can see it. So, Haskell’s designers (more specifically, Phillip Wadler) needed to come up with a way to solve this problem not only for standard out, but for all side-effects.

The solution is actually quite simple. To solve the standard out problem with println, we just passed a Vector[String] around, “modifying” it by returning the new state along with our regular return value. Here’s the inspiration: what if we did that for the entire universe? Instead of passing around just a plain Vector[String], we pass around Universe:

def foo(bar: String, everything: Universe) = {
  val everything2 = println(bar, everything)
  (bar.length, everything2)
}
 
def println(str: String, everything: Universe) = everything.println(str)

As long as the language runtime is able to somehow give us an instance of Universe which behaves the way we would expect, then this code will work just fine. Obviously, the runtime can’t really package up the entire cosmos and allow us to get new versions of it, but it can cheat a bit and pretend to give us the entire universe. The language runtime is allowed to implement the println function on the Universe object in whatever way it deems best (hopefully by actually appending to standard out). Thus, we let the runtime perform whatever magic is necessary while we remain blissfully ignorant of any and all side-effects.

This solves our problems alright, but it’s horrible in almost every other respect. We have this manual threading of the Universe, which is both verbose and painful. Even worse, this sort of thing is very error prone (i.e. what happens if we “modify” the universe and then go back and change an older version of it? do we get two, parallel universes?). The heart of our problem now is that we are using the old version of the universe to compute a new version of the universe, and we’re doing that manually. We’re taking something (in this case, the universe) and using its value to compute a new something (the new version of the universe). Sound familiar?

Phillip Wadler’s inspiration was to take advantage of the monad pattern to solve this problem. The result is the IO monad:

def foo(bar: String): IO[Int] = {
  println(bar) bind { _ => IO(bar.length) }
}
 
def println(str: String): IO[Unit] = {
  // TODO insert magic incantations
}

Of course, we can’t actually implement println, since this fake language isn’t allowed to involve itself in side-effects. However, the language runtime could provide a native (read: cheat code) implementation of println which performs its side-effects and then conjures a new version of IO (i.e. the modified universe).

The only catch with this design is that we can never get anything out of IO. Once you start down the dark path, forever will it dominate your destiny. The reason for this is one of language purity. Haskell wants to disallow side-effects, but if it were to allow us to pull values out of IO, then we could very easily bypass its safe-guards:

def readLine(): IO[String] = {
  // TODO insert magic incantations
}
 
def fakeReadLine(str: String): String = {
  val back: IO[String] = readLine()
  back.get      // whew!  doesn't work
}

As you can see, if we could pull values out of IO, then the whole exercise would become a waste of time, since it would be trivially easy to hide side-effects within wrapper functions.

Of course, none of this is particularly relevant to Scala or Ruby, much less Java! Scala doesn’t restrict side-effects. It allows you to call println whenever you feel like it, and it provides a way of declaring mutable variables (var). In a sense, Scala is hiding a hypothetical IO monad. It’s as if all Scala code is implicitly inside of IO, and thus implicitly threading the state of the universe from one moment to the next. In light of this fact, why should we care at all about how IO works? Mostly because it’s a monad which is very different from Thing and Option. I briefly considered using State, but that’s too much ancillary complexity when we’re trying to focus on the core idea of using the value from one thing to compute another thing.

Now What?

So, we’ve identified the monad pattern. We can spot it in code and employ it in an impressive range of situations, but why are we bothering with all the ceremony? If we’re already using monads all over the place without realizing it (e.g. semicolon), then why do we have to bother futzing with all of this nomenclature? Simply put: why do we care that Option is a monad so long as it just “does the right thing”?

Well, the first answer to that lies in the nature of mathematics, and by extension, computer programming. As I said, before, mathematics is all about identifying patterns (well, you also play with those patterns to generate larger, more intricate patterns, but that’s really just a means to an end). Once you identify a pattern, you name it. That’s just what mathematicians do. Imagine if Newton hadn’t named the “derivative”, and he simply insisted on calling it “an expression for the line tangent to a given line relative to a particular value of x, where x is some variable in the expression for the given line.” For one thing, calculus textbooks everywhere would be about 50 times longer. For another, we probably never would have been able to see “the derivative” as an abstract entity. Partial derivation would have never been devised. Integrals, differential equations, infinite series, and almost all of physics might never have happened. None of these consequences have anything to do with the name, that’s just a label. Rather, it was the fact that Newton was able to see (and represent) the derivative in the abstract, as a mathematical shape to be manipulated and applied in novel ways.

If you understand the Option monad, then you can use the Option monad. You can see places in your code where it can be applied, and you will reap enormous benefits as a result. However, if you understand monads as an abstraction, then you will not only understand Option, but also IO, State, Parser, STM, the list goes on. Or at least, you will understand the fundamental properties of these constructs (the rest is details). You will begin to see places where you are doing monadic things even when those things don’t fit exactly into the restricted mold of Option or State. This is where the true utility can be found.

Besides the (vast) improvements in your thought process, there’s also a more immediate practical upshot. It’s possible to define functions which work on monads in the generic sense, rather than specializing in one or the other. Just as Swing programming would be impossible if you had to rewrite every function for each specific instance of Component, there are many things which would be impossible (or at least, very very impractical) if you had to rewrite them for each specific monad. One such function is sequence:

trait Monad[+M[_]] {
  def unit[A](a: A): M[A]
  def bind[A, B](m: M[A])(f: A => M[B]): M[B]
}
 
implicit object ThingMonad extends Monad[Thing] {
  def unit[A](a: A) = Thing(a)
  def bind[A, B](thing: Thing[A])(f: A => Thing[B]) = thing bind f
}
 
implicit object OptionMonad extends Monad[Option] {
  def unit[A](a: A) = Some(a)
  def bind[A, B](opt: Option[A])(f: A => Option[B]) = opt bind f
}
 
 
def sequence[M[_], A](ms: List[M[A]])(implicit tc: Monad[M]) = {
  ms.foldRight(tc.unit(List[A]())) { (m, acc) =>
    tc.bind(m) { a => tc.bind(acc) { tail => tc.unit(a :: tail) } }
  }
}

There are a lot of ways to pretty this up, but I wanted to be as explicit as possible for the sake of illustration. The general function of sequence is to take a List of monad instances and return a monad instance of a List of those elements. For example:

val nums = List(Some(1), Some(2), Some(3))
val nums2 = sequence(nums)           // Some(List(1, 2, 3))

The magic is that this function works on any monad:

val nums = List(Thing(1), Thing(2), Thing(3))
val nums2 = sequence(nums)           // Thing(List(1, 2, 3))

In this case, Monad (the trait) is an example of a typeclass. Basically, we’re saying that there is this general idea of a monad, and any monad will define two functions: unit and bind. In this way, define functions which operate on monads without knowing which specific monad we’re manipulating. Think of it like the adapter pattern on steroids (sprinkled with a goodly portion of Scala’s implicit magic).

This is reason number two for understanding the general concept of a monad, rather than just the specific: you suddenly become able to define tons of nifty utility functions. For more examples of this, you need look no further than Haskell’s standard library.

Those Pesky Axioms

Congratulations! You just made it through an entire monad tutorial without ever having to worry about the monadic axioms or what they mean. Now that you’ve got the concept down (hopefully), you can graduate to the axioms themselves.

As it turns out, the monadic axioms are really quite intuitive. We’ve actually been assuming them all along without ever really saying so. The axioms define how the unit (constructor) and bind (composition) functions are supposed to behave under certain situations. Think of them a bit like the laws governing integer addition (commutativity, associativity, etc). They don’t tell you everything about monads (actually, they don’t tell you much at all from an intuitive standpoint), but they do give you the basic, mathematical underpinnings.

Excited yet? No, I didn’t think so. Well, here they are anyway, defined in terms of the Monad typeclass we used earlier:

def axioms[M[_]](implicit tc: Monad[M]) {
  // identity 1
  def identity1[A, B](a: A, f: A => M[B]) {
    val ma: M[A] = tc.unit(a)
    assert(tc.bind(ma)(f) == f(a))
  }
 
  forAll { (x, f) => identity1(a, f) }        // sort-of ScalaCheck
 
  // identity 2
  def identity2[A](ma: M[A]) {
    assert(tc.bind(ma)(tc.unit) == ma)
  }
 
  forAll { m => identity2(m) }
 
  // associativity
  def associativity[A, B, C](m: M[A], f: A => M[B], g: B => M[C]) {
    val mf: M[B] = tc.bind(m)(f)
    val mg: M[C] = tc.bind(mf)(g)
 
    val mg2: M[C] = tc.bind(m) { a =>
      tc.bind(f(a))(g)
    }
 
    assert(mg == mg2)
  }
 
  forAll { (m, f, g) => associativity(m, f, g) }
}

The first two axioms (the “identity” ones) are basically saying that the unit function is a simple constructor with respect to the bind function. Thus, when bind “pulls apart” the monad and passes the value to its function parameter, that value will be precisely the value that unit puts into the monad. Likewise, if the function parameter given to bind simply takes the value and wraps it back up inside the monad, then the final result is exactly the same as if we had left the monad alone.

The third axiom is the most complicated to express, but I think it’s actually the most intuitive. Basically, this axiom is saying that if you first bind with one function, then you bind the result against another function, that’s the same as applying the first function to the value inside the monad and then calling bind on the result of that application. This isn’t exactly associativity in the classical sense, but you can sort of think about it in that way.

One of the useful consequences of the second law comes up (quite frequently) when you have one bind nested inside the another. Whenever you find yourself in that situation, you can move the nested bind outside of the outer bind, flattening your code slightly. Like so:

val opt: Option[String] = Some("string")
 
opt bind { str => 
  val innerOpt = Some("Head: " + str)
  innerOpt bind { str => Some(str + " :Tail") }
}
 
// is the same as...
 
opt bind { str => Some("Head: " + str) } bind { str => Some(str + " :Tail") }

The rewritten code has a much more “sequential” feel (the essence of monads!) and is almost always shorter than the nested form.

As I said before, the axioms are very, very intuitive once you understand the abstract concept of a monad. They may not be intuitive to express, but their consequences are very easy to understand and quite natural in practice. So, don’t spend a lot of time trying to memorize the axioms. Your time will be better spent contemplating the way that semicolon works.

Conclusion

Monads are not scary. They are not complex, academic or esoteric. Monads are an abstract, mathematical label affixed to a pattern found in almost all code. We all use monads every day. The hardest part in understanding monads is recognizing that the hardest part isn’t so hard after all.

I sincerely hope that this latest, dubious venture down the well-trod path of monad exposition has proven a fruitful one. I can say without hesitation that the understanding and perspective which comes from a solid grasp of monads is invaluable in very practical, down-to-earth coding (even in staid languages like Java!). Monads impart an understanding of the very fabric of sequential computation and composability, and if that isn’t sufficient motivation to learn, I don’t know what is.

For those coming from a primarily-Java background, the concept of an interactive shell may seem a bit foreign.  Basically, an interactive shell — or REPL, as it is often called — is a line-by-line language interpreter which allows you to execute snippets of code with immediate result.  This has been a common tool in the hands of dynamic language enthusiasts since the days of LISP, but has only recently found its way into the world of mainstream static languages such as Scala.

One of the most useful applications of these tools is the testing of code (particularly frameworks) before the implementations are fully completed.  For example, when working on my port of Clojure’s PersistentVector, I would often spin up a Scala shell to quickly test one aspect or another of the class.  As a minor productivity plug, JavaRebel is a truly invaluable tool for development of this variety.

The problem with this pattern of work is it requires the interactive shell in question to be pre-configured to include the project’s output directory on the CLASSPATH.  While this isn’t usually so bad, things can get very sticky when you’re working with a project which includes a large number of dependencies.  It isn’t too unreasonable to imagine shell invocations stretching into the tens of lines, just to spin up a “quick and dirty” test tool.

Further complicating affairs is the fact that many projects do away with the notion of fixed dependency paths and simply allow tools like Maven or Buildr to manage the CLASSPATH entirely out of sight.  In order to fire up a Scala shell for a project with any external dependencies, I must first manually read my buildfile, parsing out all of the artifacts in use.  Then I have to grope about in my ~/.m2/repository directory until I find the JARs in question.  Needless to say, the productivity benefits of this technique become extremely suspect after the first or second expedition.

For this reason, I strongly believe that the launch of an interactive shell should be the responsibility of the tool managing the dependencies, rather than that of the developer.  Note that Maven already has some support for shells in conjunction with certain languages (Scala among them), but it is as crude and verbose as the tool itself.  What I really want is to be able to invoke the following command and have the appropriate shell launched with a pre-configured CLASSPATH.  I shouldn’t have to worry about the language of my project, the location of my repository or even if the shell requires extra configuration on my platform.  The idea is that everything should all work auto-magically:

$ buildr shell

This is exactly what the interactive-shell branch of my Buildr fork is designed to accomplish.  Whenever the shell task is invoked, Buildr looked through the current project and attempts to guess the language involved.  This guesswork is required for a number of other features, so Buildr is actually pretty accurate in this area.  If the language in question is Groovy or Scala, then the desired shell is obvious.  Java does not have an integrated shell, which means that the default behavior on a Java project would be to raise an error.

However, the benefits of interactive shells are not limited to just the latest-and-greatest languages.  I often use a Scala shell with Java projects, and for certain things a JRuby shell as well (jirb).  Thus, my interactive shell extension also provides a mechanism to allow users to override the default shell on a per-project basis:

define 'my-project' do
  shell.using :clj
end

With this configuration, regardless of the language used by the compiler for “my-project”, Buildr will launch the Clojure REPL whenever the shell task is invoked.  The currently supported shells and their corresponding Buildr identifiers:

• Clojure’s REPL — :clj
• Groovy’s Shell — :groovysh
• JRuby’s IRB — :jirb
• Scala’s Shell — :scala

It is also possible to explicitly launch a specific shell.  This is useful for situations where you might want to use the Scala shell for testing some things and the JRuby IRB for quickly prototyping other ideas (I do this a lot).  The command to launch the JIRB shell in the context of my-project would be as follows:

$ buildr my-project:shell:jirb

As a special value-added feature, all of these shells (except for Groovy’s, which is weird) will be automatically configured to use JavaRebel for the project compilation target classes if it can be automatically detected.  This detection is performed by examining REBEL_HOME, JAVA_REBEL, JAVAREBEL and JAVAREBEL_HOME environment variables in order.  If any one of these points to a directory which contains javarebel.jar or points directly to javarebel.jar itself, the configuration is assumed and the respective shell invocation is appropriately modified.

Best of all, this support is implemented using a highly-extensible framework similar to Buildr’s own Compiler API.  It’s very easy for plugin implementors or even average developers to simply drop-in a new shell provider, perhaps for an internal language or even some unexpected application.  The core functionality of shell detection is integrated into Buildr itself, but this in no way hampers extensibility.  For example, I could easily create a third-party .rake plugin for Buildr which added support for a whole new language (e.g. Haskell).  In this plugin, I could also define a new shell provider which would be the default for projects using that language (e.g. GHCi).

Open Question

The good news is that this feature has been discussed extensively on the buildr-user mailing-list and the prevailing opinion seems to be that it should be folded into the main Buildr distribution.  Exactly what form this will take has yet to be decided.  The bad news is that there is still some dispute about a fundamental aspect of this feature’s operation.

The question revolves around what the exact behavior should be when the shell task is invoked.  Should Buildr detect the project (or sub-project) you are in and automatically configure the shell’s CLASSPATH accordingly?  This would give the interactive shell access to different classes depending on the current working directory.  Alternatively, should there be one all-powerful shell per-buildfile configured at the root level?  This would allow your shell to remain consistent throughout the project, regardless of your current directory.  However, it would also mean that some configuration would be required in order to enable the functionality.  (more details of this debate can be found on the mailing-list).

Additionally, what should the exact syntax be for invoking a specific shell?  Rake 0.8 allows tasks to take parameters enclosed within square brackets.  Thus, the syntax would be something more like the following:

$ buildr collection:shell[jirb]

In some sense, this is more logical since it reflects the fact that a single task, shell, is taking care of the work of invoking stuff.  On the other hand, it’s a little less consistent with the rest of Buildr’s tasks, particularly things like “test:TestClass” and so on.  This too is a matter which has yet to be settled.

All in all, this is a pretty experimental branch which is very open (and desirous) of outside input.  How would you use a feature like this?  Is there anything missing from what I have presented?  What design path should be we take with regards to project-local vs global shell configurations?

If you feel like adding your voice to the chorus, feel free to leave a comment or (better yet) post a reply on the mailing-list thread.  You’re also perfectly free to fork my remote branch at GitHub to better experiment with things yourself.  The root of the whole plate of spaghetti is the lib/buildr/shell.rb file.  Bon appetit!

I’m told that this issue came up at the JVM languages conference this past week.  The problem of integrating very separate languages in a natural way is not an easy one, even when all languages in question are on the JVM.  So far, the only solution that anyone has been able to come up with is just to use Java as an intermediary.  After all, JRuby integrates with Java, as does Clojure, therefore JRuby has at least some path of integration with Clojure and vice versa.

The problem is that such integration is not really idiomatic.  As the title of this post implies, we’re going to consider the example of Scala and JRuby.  I’ve already written about how to create a Scala DSL for calling JRuby directly, so let’s look at the other side of the problem: it’s certainly possible to use Scala classes from within JRuby, but it isn’t exactly a pleasant proposition.  Let’s imagine that I want to make use of my Scala port of Rich Hickey’s excellent immutable persistent vector data structure:

import com.codecommit.collection.Vector
 
vec = Vector.new
vec = vec.send('$plus'.to_sym, 1)
vec = vec.update(0, 2)
 
puts vec.apply 0       # 2

Straightforward, but ugly.  Let’s break this down a little bit…  The import and instantiation are both self-explanatory, so we come to the rather cryptic invocation of send.  In case you don’t know, this is a Ruby method which makes it possible to invoke arbitrary methods on a given object, including those with illegal characters.  I happen to know that Scala will compile the + operator to a method named $plus within class Vector.  JRuby is perfectly happy to handle and forward this call as necessary, despite the fact that you could never actually declare this method in pure Ruby ($ has special meaning).  Thus, this invocation is the same as the following Scala snippet:

vec = vec + 1

In other words, append 1 to the vector and assign the resulting new vector back into vec.

Moving on, we come to the invocations of update and apply.  These should be a bit more comprehensible to those familiar with Scala’s intricacies.  In short, these methods are how you overload the parentheses and parentheses/assignment operators.  Thus, our last two lines correspond as follows:

vec = (vec(0) = 2)
println(vec(0))

This was just a trivial example, you can imagine how a more complicated Scala API could be neigh unusable in JRuby.  Intuitively though, it shouldn’t have to be this way.  After all, JRuby supports many of the same syntactic power as Scala: it has some operator overloading, closures and even more complicated features like mixins.  With all of this common ground, surely there is a way to make the two coexist more naturally?  I mean, optimally we could have something like this:

import com.codecommit.collection.Vector
 
vec = Vector.new
vec = vec + 1
vec = (vec[0] = 2)       # problematic
 
puts vec[0]              # 2

What we have just done is informally define a desired syntax for a domain-specific problem.  Does that sound like an internal DSL to anyone else?

Our goal is to create a simple Ruby API that can be required into any JRuby application to improve the integration with Scala.  To do this, we will need to find a way to convert Ruby features into their corresponding Scala features by going through Java.  Once we find this translation, we can use meta-programming and dangerously-cool Monkey Patching to augment JRuby’s existing Java integration.  In this way, we don’t have to start our integration layer from scratch, we can just “pretty up” JRuby’s existing features, taking advantage of the fact that Scala classes are Java classes.

Step One: Operators

From our example above, converting the invocation of a $plus method into a Ruby + operator seems to be the easiest challenge to tackle.  Ruby does support operator overloading; unfortunately, this support is incredibly limited when compared with Scala’s awesome power.  For example, in Scala it is possible to invent arbitrary operators, a feature which is heavily used in the Scala standard library.  Ruby has no such capability, operators are implemented using conventional techniques and hard-coded rules in the parser.

In order to avoid blowing this experiment completely out of proportion, we’re only going to implement translations for the set of conventional Ruby operators.  It would be possible to also implement Scala operators which are made by combining existing Ruby operators (e.g. the ++ operator) using techniques developed for the Superators gem, but to do so would be extremely difficult.

We saw in our original example that the Scala + operator is translated into an invocation of the $plus method.  It stands to reason that we could make use of this fact in our translation from Ruby to Scala.  The trick is finding a comprehensive list of Scala’s operators and what methods they compile to.  Fortunately, I had already investigated these translations for a different project:

Scala Operator	Compiles To
=	$eq
>	$greater
<	$less
+	$plus
-	$minus
*	$times
/	div
!	$bang
@	$at
#	$hash
%	$percent
^	$up
&	$amp
~	$tilde
?	$qmark
|	$bar
\	$bslash

Repeated iterations of an operator become corresponding repetitions of the equivalent character sequence.  Thus, ++ becomes $plus$plus.  This suggests a very natural strategy for converting Ruby operator invocations into Scala: string substitution.  We can easily Ruby operator calls using method_missing, substitute the appropriate character sequences and then attempt the modified call against the same object.  This idea, when translated into Ruby, is almost as simple as that:

OPERATORS = {"=" => "$eq", ">" => "$greater", "<" => "$less", \
        "+" => "$plus", "-" => "$minus", "*" => "$times", "/" => "div", \
        "!" => "$bang", "@" => "$at", "#" => "$hash", "%" => "$percent", \
        "^" => "$up", "&" => "$amp", "~" => "$tilde", "?" => "$qmark", \
        "|" => "$bar", "\\" => "$bslash"}
 
alias_method :__old_method_missing_in_scala_rb__, :method_missing
def method_missing(sym, *args)
  str = sym.to_s
 
  str = $&[1] + '_=' if str =~ /^(.*[^\]=])=$/
 
  OPERATORS.each do |from, to|
    str.gsub!(from, to)
  end
 
  if methods.include? str
    send(str.to_sym, args)
  else
    __old_method_missing_in_scala_rb__(sym, args)
  end
end

Right at the end, after we have transformed the method name to convert any Ruby operators into their Scala equivalents, we actually check to see if the method exists instead of blindly sending the invocation.  The reason for this is to avoid creating an infinite loop in cases where a method really doesn’t exist.  We can rely on the presence of bona fide methods due to the way that JRuby proxies Java objects into Scala.

Astute readers will notice the extra bit of regular expression checking right at the head of the method.  We didn’t cover this in our target example, but this transformation is none-the-less quite important.  Both Scala and Ruby provide mechanisms to simulate assignment to class members through method calls.  In Ruby, you just suffix the method name with =, whereas in Scala the correct suffix is _=.  This extra transformation looks for those situations and converts to the appropriate result.  Thus, Scala var fields are now accessible within JRuby using the same syntax as standard attr_reader/attr_writer methods in Ruby.

Two other operators which might be useful to enable are the [] and []= operators, generally used for collections access.  Scala doesn’t actually support these operators, reserving square brackets for declaring type parameters.  However, it does provide a very similar syntax with parentheses.  As a refresher, here is how we might use an array within Scala:

val arr = new Array(5)
arr(0) = 5
arr(1) = 4
arr(2) = 3
arr(3) = 2
arr(4) = 1
 
println(arr(1))        // 4

At compile-time, this translates into corresponding invocations of the apply and update methods of class Array.  Specifically, apply is used to retrieve data, while update is used to assign it.  These are the direct corollaries to Ruby’s [] and []=.  It would only be natural to translate invocations of these operators into corresponding calls to apply and update, and we can do this simply by extending our method_missing just a little bit:

  # ...
 
  gen_with_args = proc do |name, args|
    code = "#{name}("
 
    unless args.empty?
      for i in 0..(args.size - 2)
        code += "args[#{i}], "
      end
      code += "args[#{args.size - 1}]"
    end
 
    code += ')'
  end
 
  if str == '[]'
    eval(gen_with_args.call('apply', args))
  elsif sym.to_s == '[]='
    eval gen_with_args.call('update', args)            # doesn't work right
  elsif methods.include? str
    send(str.to_sym, args)
  else
    __old_method_missing_in_scala_rb__(sym, args)
  end
end

The gen_with_args proc is merely a nifty little utility closure used to cut down on redundancy without creating an entire method.  It actually generates a String of Ruby code which invokes the specified method with the given arguments.  This is required because JRuby’s Java integration is only so smart.  It will try to properly convert invocations of n-arity Java methods when Ruby arrays are passed as arguments, but it can only do so well.  The safest route is to just call the method, passing each element of the array as a successive argument.

All of this looks quite nice, and it seems to satisfy our requirements, but there is just one problem: Ruby doesn’t behave itself with respect to the []= operator.  Rather than taking the sensible approach and allowing the receiving class to define its return value, it actually ignores whatever value is returned from the []= method and forces it to be the final argument.  In other words, the above code will work, but it might not integrate with Scala in quite the way we would expect.  Consider our original motivating example:

import com.codecommit.collection.Vector
 
vec = Vector.new
vec = vec + 1
vec = (vec[0] = 2)       # problematic
 
puts vec[0]              # 2

Well, I said that this would be problematic.  The issue is the value of the expression vec[0] = 2 is not a new Vector instance as returned by the Vector#update method, but actually the Fixnum value 2.  Thus, the snippet above can never work.  In what is probably the most bone-headed feature of the entire language, Ruby forces every invocation of []= to return the assigned value.  This works fine (sort-of) for mutable, side-effecting implementations like Array and Scala’s ListBuffer, but it completely falls apart for immutable, functional collections like Vector.  So much for a language that “makes me smile”…

The solution of course is to always be aware of these cases where the update method does not return Unit and just use update directly rather than []=.  Thus, a working version of the given snippet would be as follows:

import com.codecommit.collection.Vector
 
vec = Vector.new
vec = vec + 1
vec = vec.update(0, 2)
 
puts vec[0]              # 2

By calling update directly, we ensure that its return value is captured and assigned back into vec.  Kind of ugly, but unavoidable.

Step Two: Mixins

Mixins are probably the most useful and code-saving feature in the entire Scala language.  Like interfaces, they provide the flexibility of multiple inheritance, but they also bring with that the DRYness of shared definitions between subtypes.  In fact, it may even be safe to say that the overwhelming power of the Scala collections library is owed primarily to traits.  After all, without the inherited method definitions from Iterable, each collection would have to re-implement foldLeft, map, flatMap and each of the myriad of common methods defined by collections.

It just so happens that Ruby also has mixins of a very similar variety.  It seems natural then that we should be able to use Scala mixins within JRuby just as if they were standard Ruby modules.  Unfortunately, Java does not have mixins, meaning that unlike operator overloading, there isn’t a nice and easy technique we can use to convert between the two worlds.  The good news is that JRuby does give us a bit of a head start in making this integration work: it’s interface implementation syntax.

Ruby doesn’t support interfaces directly, but JRuby allows us to use the standard include syntax on a Java interface.  Once we have included the interface and implemented its methods, we can pass an instance of our Ruby class into Java methods expecting instances of that interface.  JRuby takes care of all of the magic required to proxy the method calls.  Syntactically, it looks something like this

class DoSomething
  include java.lang.Runnable
 
  def run
    puts 'Running!'
  end
end

This is how we want our mixin integration to work.  We should be able to include a Scala trait into a Ruby class, implement any abstract methods and then expect everything to work per-normal.  In order to accomplish this, we’re going to need to override the include method to check to see if its target is a Scala trait.  If that is the case, then include should create proxies for all of the defined methods within the trait.  Basically, we have to do the same thing for Scala traits that Ruby does automatically for its modules.

Fortunately, this too is an area where Ruby’s notion of “open classes” will come to our rescue.  Not only does Ruby allow us to add methods to classes at runtime, it also allows us to redefine core methods within the standard library; in this case, Module#include.

class Module
  alias_method :__old_include_in_scala_rb__, :include
  def include(*modules)
    modules.each do |m|
      clazz = nil
      begin
        if m.java_class.interface?
          cl = m.java_class.class_loader
          mixin_methods_for_trait(cl, cl.loadClass(m.java_class.to_s))
        end
      rescue
      end
 
      # ...
    end
 
    modules.each {|m| __old_include_in_scala_rb__ m }
  end
end

The most important thing to see here is regardless of whether or not the module in question is a trait, we still forward the inclusion onto the old definition.  This is critical, because it means that JRuby is still able to create an interface proxy for that class, allowing us to pass instances of the including class into Scala and have them treated as instances of the trait in question.

The key to our actual inclusion of the defined methods is the way in which Scala compiles traits.  Consider the following:

trait Test {
  def method1(): Unit
 
  def method2() {
    println("In method2()")
  }
}
 
class Usage extends Test {    // mixin
  def method1() {
    println("In method1()")
  }
}

Scala will compile this into the bytecode equivalent of the following Java code:

public interface Test {
    public void method1();
 
    public void method2();
}
 
public class Test$class {     // actually an inner-class of Test
    public static void method2(Test t) {
        System.out.println("In method2()");
    }
}
 
public class Usage implements Test {
    public void method1() {
        System.out.println("In method1()");
    }
 
    public void method2() {
        Test$class.method2(this);
    }
}

It’s very clever if you think about it.  Through a bit of compile-time magic, Scala is able to keep the method definitions within traits centralized (rather than inlining everything) as well as maintain full interface-level interop with Java.  It is actually possible to define a Java method which accepts as a parameter an instance of a particular trait.  Since traits are interfaces, javac knows how to deal with this definition and the JVM has no problems in actually dispatching the call.

We can use this implementation detail to enable our mixin of traits into JRuby classes.  We’re already testing within include to see whether or not the “module” at hand is in fact an interface, it would be a simple matter to also check for the existence of a class of the form “TraitName$class“.  If such a class exists, we could loop through all of its public static members and create corresponding proxy methods within the including class.  All of this is done within the mixin_methods_for_trait method:

def mixin_methods_for_trait(cl, trait_class, done=Set.new)
  return if done.include? trait_class
  done << trait_class
 
  clazz = cl.loadClass "#{trait_class.name}$class"
 
  trait_class.interfaces.each do |i| 
    begin
      mixin_methods_for_trait(cl, i, done)
    rescue
    end
  end
 
  clazz.declared_methods.each do |meth|
    mod = meth.modifiers
 
    if java.lang.reflect.Modifier.isStatic mod and \
	    java.lang.reflect.Modifier.isPublic mod
      @@trait_methods ||= []
 
      unless meth.name.include? '$'
        module_eval "\
def #{meth.name}(*args, &block)
  args.insert(0, self)
  args << block unless block.nil?
 
  args.map! do |a|
    if defined? a.java_object
      a.java_object
    else
      a
    end
  end
 
  begin
    scala_reflective_trait_methods[#{@@trait_methods.size}].invoke(nil, args.to_java)
  rescue java.lang.reflect.InvocationTargetException => e
    raise e.cause.message.to_s      # TODO  change over for 1.1.4
  end
end
"
 
        @@trait_methods << meth
      else
        define_method meth.name do |*args|    # fallback for methods with special names
          args.insert(0, self)
 
          begin
            meth.invoke(nil, args.to_java)
          rescue java.lang.reflectInvocationTargetException => e
            raise e.cause.message.to_s
          end
        end
      end
    end
  end
end

I know, the amount of code here is a little daunting, but it’s really not that bad.  Essentially, this just implements our intuition about how mixins should work in Ruby.  The only thing about the algorithm that we haven’t already considered is how to deal with the super-traits of traits.  Since the inheriting of method definitions is only done through static proxying, there would be no reason for Scala to compile an inherited relationship between the definition class of a sub-trait and its super-trait.  To get around this problem, we explicitly check for super-interfaces and then mixin their methods first.  This is to allow for methods which are inherited by sub-traits and then overridden.

The final little tidbit here is the way in which the proxy methods are created.  Ruby does allow us to add methods to classes at runtime, but it is unfortunately rather restrictive in how those methods can be added.  In a nutshell, there are two main techniques: module_eval and define_method.  By using define_method, we are able to avoid String evaluation and keep our editor’s syntax-highlighting happy with our sources.  Unfortunately, methods created using define_method have a very critical limitation: they cannot accept blocks.  Thus, any mixed-in trait methods which took function values as parameters would be unusable from within Ruby.

This inconsistency is fixed in Ruby 1.9, but until then we still need a way of proxying higher-order Scala methods.  Thus, we default to using module_eval.  Unfortunately, this technique also comes with its own set of issues; specifically: illegal characters.  As we saw previously, any Scala method with a non-alphanumeric name will have to use the $ character to denote certain symbols.  However, Ruby assigns special significance to symbols like $ and @, making it impossible to use them within method names.  The only way around this restriction on method names is to create such special methods using define_method, which brings us right back to our first set of problems.

The only solution I could come up with for Ruby 1.8 was to use module_eval by default, unless the method name contained a $ character, in which case define_method would be used.  With this technique, almost every case is covered.  The only remaining issue would be if a method with a non-alphanumeric name took a function as a parameter.  In this case, the method would be unusable from Ruby.  However, even a cursory glance over the Scala standard library shows that this is an extremely rare case, one which can be safely made “difficult” if it means improving the integration in other areas (trait mixins).

One final unfortunate note on this method: it doesn’t work with JRuby 1.1.3 and later (the current release is 1.1.4).  As reported by JRUBY-2999, including two separate Java interfaces with conflicting methods will actually cause the interpreter to crash.  At the time of the writing, this bug has not been resolved, either in 1.1.4 or in the latest SVN sources.  Since traits quite often have methods which overlap with inherited methods, there is really no way to get around this bug.  Thus, the library described in this article was developed with JRuby 1.1.2 and will only work with that release or any upcoming releases which fix the regression.  Interestingly, this means that the library must deal with a different, less critical JRuby bug which makes it impossible to raise Java exceptions.  As this exception bug does not entirely prevent the use of Scala mixins, it is preferable to the more serious interpreter-crash regression.

Now that we have mixins at our disposal, it is possible to further improve our Scala integration by implementing a scala_object method for Array and Hash, one which converts these objects into a form which can be passed to Scala methods expecting Seq and Map, respectively.

class BoxedRubyArray
  include Scala::RandomAccessSeq
 
  def initialize(delegate)
    @delegate = delegate
  end
 
  def apply(i)
    @delegate[i]
  end
 
  def length
    @delegate.size
  end
end
 
class Array
  def scala_object
    BoxedRubyArray.new self
  end
end

The Hash proxy is analogous, but much more verbose.  The key to this entire implementation here is the enabling of Scala mixins within JRuby.  All we have to do is mixin the RandomAccessSeq trait, implement apply and length, and suddenly we have a first-class Scala collection backed by a Ruby Array.  Not only does this allow us to use some of Scala’s higher-order magic on Ruby arrays, it also enables previously impossible usages like the following:

import com.codecommit.collection.Vector
 
vec = Vector.new
vec = vec + 1 + 2 + 3 + 4 + 5
 
arr = [6, 7, 8, 9, 0]
cat = vec.concat arr.scala_object

At the end of this snippet, cat is a Scala Iterable with contents [1, 2, 3, 4, 5, 6, 7, 8, 9, 0].  Remember that concat is actually a method within Vector which itself expects an instance of Iterable.  Fortunately, we now have a method to convert a Ruby array into just such an Iterable, which we then pass to concat achieving our final result.

Step Three: Closures

Both Ruby and Scala have this notion of closures, which are assignable, anonymous functions with access to their enclosing scope.  Closures are most often used as a syntactic device for passing and/or storing a bit of code for later invocation.  A common example that even Java has to deal with would be event handling, where a specific code block must be executed every time a button is pressed.  Scala and Ruby take this concept a little further, providing higher-order functions like map and implementing conventional iteration using foreach and each (respectively).

Optimally, we should be able to call Scala methods passing Ruby blocks where Scala function values are required.  As it happens, even without special Scala integration magic, we can already get very close to this:

vec = Vector.new
# ...
 
vec.foreach do |e|
  puts e
end

This will print every element in the vec instance.  This works because JRuby allows blocks to be passed to methods accepting interfaces.  Since Scala’s function values are in fact subtypes of traits Function0, Function1 and so on (according to arity), JRuby is able to recognize the method signatures appropriately and proxy the values.

Unfortunately, JRuby doesn’t innately understand Scala traits, so it doesn’t know that the compose method should be proxied to an implementation within the trait.  I assume that JRuby will proxy every method in the target interface to the block in question (I haven’t actually tested that).  Assuming this is the case, the moment that foreach (or any other method) attempts to invoke compose on a proxied JRuby block, the result will be a ClassCastException.  It just so happens that foreach behaves itself and there is nothing to worry about, but we cannot make that guarantee in general.

The solution of course is to do for Ruby’s Proc what we already did for Array and Hash: provide a wrapper which uses the Functionn mixin and delegates to the Proc in question.  However, unlike Array, there is no single “Function” mixin that we can use.  Instead, we must create a separate wrapper for each function arity supported by Scala…all 22 of them.  Fortunately, Ruby’s meta-programming capabilities help us out here, allowing us to define classes within a loop and then dynamically select the correct wrapper class name based on the Proc arity:

module ScalaProc
  class ScalaFunction
    def initialize(delegate)
      @delegate = delegate
    end
 
    def apply(*args)
      @delegate.call args
    end
  end
 
  for n in 0..22    # sneaky, but much more concise
    eval "\
class Function#{n} < ScalaFunction
  include Scala::Function#{n}
end
"
  end
end
 
class Proc
  def to_function
    eval "ScalaProc::Function#{arity}.new self"
  end
 
  def java_object
    to_function
  end
 
  def scala_object
    to_function
  end
end

The only abstract method within any of the Functionn traits is apply, taking the same number of arguments as the function arity.  It’s easy enough to implement this function once within the ScalaFunction superclass, which means that all we have to do within each of the arity-specific wrappers is mixin the relevant Function.

Now that we have this conversion between Ruby Proc(s) and Scala function values, we can make use of it in situations where it becomes necessary.  For example, let’s say that I have a Scala method like the following:

def doAndMultiply(by: Int)(f: (Int)=>Int) = {
  f compose { (_:Int) * by }
}

Simply put, this method takes two curried parameters, an Int along with a function value taking another Int and returning an Int.  This method then returns a new function which itself takes an Int, first multiplies it by the original first Int parameter, then applies the given function to the result.  Got all that? 

We can call this function from Scala in the following way:

val comp = doAndMultiply(3) { _ + 2 }
comp(5)      // => 17

Now that we have our super-fancy Proc conversion, we are able to use an almost identical call syntax from within Ruby.  Notice how we are able to take advantage of the way in which Scala compiles curried methods to achieve a JRuby syntax which looks almost exactly like block-standard Ruby:

add = proc {|x| x + 2 }
comp = doAndMultiply(3, add.scala_object)
 
comp.call 5        # => 17

And if doAndMultiply is a method brought in via a mixin, we can do even better:

comp = doAndMultiply 3 do |x| 
  x + 2
end
 
comp.call 5        # => 17

This works because of the way in which we coerce parameters within the proxies for the mixed-in methods.  Recall from earlier:

args.map! do |a|
  if defined? a.java_object
    a.java_object
  else
    a
  end
end

The very reason we have to explicitly map over the method arguments is to satisfy this particular case.  JRuby’s Ruby-to-Java coercion is pretty smart, but it doesn’t seem to be up to this particular challenge (believe me, I tried).  Thankfully, a little bit of benign check-and-coerce later, our arguments are none the worse for wear and in a form that Scala can chew on.

Conclusion

As you may have guessed, I have already taken the liberty of implementing the framework described in this article.  It even has a few features I didn’t discuss, like auto-detecting the default Scala installation and the ability to use Scala function values as if they were Ruby Proc(s).  All in all, it makes for a very slick way of working with Scala libraries from within JRuby scripts in an intuitive, idiomatic fashion.  Hopefully, this should help to encourage the use of these two fascinating technologies together in future projects.

• Download scala.rb (does not work with JRuby 1.1.3 or 1.1.4)

Now this article had just finished a long-winded presentation of type inference and compilation steps, so I’m quite certain that the author was aware of Scala’s type system.  The more likely target of the “dynamic typing” remark would be Scala’s implicit conversions mechanism.  I have heard this language feature described many times as being a way of “dynamically” adding members to an existing class.  While it would be incorrect to say that this feature constitutes a dynamic type system, it is true that it may be used to satisfy many of the same design patterns.  Consider the facetious example of a string “reduction” method, one which produces an acronym based on the upper-case characters within the string:

val acronym = "Microsoft Certified Systems Engineer".reduce
println(acronym)            // MCSE

The immediate problem with this snippet is the fact that string literals are of type java.lang.String, a class which comes pre-defined by the language.  The only way to ensure that the above syntax works properly is to “add” the reduce method to the String class separate from its definition.  In a language such as Ruby or Groovy which have dynamic type systems, we could simply open the class definition and add a new method at runtime.  However, in Scala we have to be a bit more tricky.  We can’t actually add methods to an existing class, but we can define a new class which contains the desired method.  Once we have that, we can define an implicit conversion from our target class to our new class.  The Scala compiler sees this and performs the appropriate magic behind the scenes.  In code, it looks like this:

class MyRichString(str: String) {
  def reduce = str.toCharArray.foldLeft("") { (t, c) =>
    t + (if (c.isUpperCase) c.toString else "")
  }
}
 
implicit def str2MyRichString(str: String) = new MyRichString(str)

This contrasts quite dramatically with the Ruby implementation of the same concept via open classes (somewhat less-graciously known as “Monkey Patching”):

class String
  def reduce
    arr = unpack('c*').select { |c| (65..90).include? c }
    arr.pack 'c*'
  end
end
 
puts 'HyperText Transfer Protocol'.reduce       # HTTP

No visible type conversion is taking place here, all we did is add a method to an existing class and trust that the runtime can figure out the rest.  Indeed, for this application, we don’t really need anything else.  However, as anyone with experience implementing internal domain-specific languages will tell you, seldom is life as simple as adding a few methods to an existing class.  Consider a more complicated scenario where we need to overload the < operator on integers to operate on String values, returning true if the length of the string is less than the integer value, otherwise false.  In Scala, we would once again make use of the implicit conversion mechanism, this time with an even more concise syntax:

implicit def lessThanOverload(i: Int) = new {
  def <(str: String) = str.length < i
}

In fact, we don't even need to go this far.  It is possible to create an implicit conversion from String to Int defined on the length of the String.  This would allow existing method implementations within the Int class to operate upon String values:

implicit def str2Int(str: String) = str.length

As a matter of interest, this particular situation can be managed by one of the most convoluted and verbose languages on the market, C++:

bool operator<(const int &i, const std::string &str)
{
    return str.length() < i;
}

Despite the seemingly-dynamic nature of the problem, the statically typed language camp seems well represented in terms of solutions.  Ironically, this sort of problem is one which will be exceedingly difficult to solve in a language like Ruby.  This is primarily because method overloading is an innately static device.  That's not to say that overloading is impossible in a dynamically typed language (Groovy), but it's not easy.  To see why, let's consider the most natural implementation of our operator problem in Ruby:

class Fixnum
  def <(str)
    str.size < self
  end
end

Intuitively, this may seem like the right way to approach the problem, but the results of such an implementation would be disastrous.  At the very least, the first time anyone attempted to perform a < comparison targeting an integer, the interpreter will overflow the call stack.  In fact, any time any code uses the less-than operator on an instance of Fixnum, the interpreter will crash.  The reason for this is the invocation of < upon str.size within our "overloaded" definition.  This call creates a very tight recursive loop which will very quickly eat through all available stack frames.  We can avoid this problem by reversing the comparison like so:

class Fixnum
  def <(str)
    self >= str.size
  end
end

Now we don't have to worry about stack overflow, but in the process we have accidentally redefined integer-to-integer comparison in a very strange way:

irb(main):006:0> 123 < 'test'
=> true
irb(main):007:0> 123 < 123
=> true

Clearly, more effort is going to be required if we are to put to rest our little dilemma.  As it turns out, the final solution is surprisingly ugly and verbose:

class Fixnum
  alias_method :__old_less_than__, '<'.to_sym
  def <(target)
    if target.kind_of? String
      __old_less_than__ target.size
    else
      __old_less_than__ target
    end
  end
end

Whatever happened to Ruby as a "more elegant" language?  The unfortunate truth is that in order to emulate method overloading based on input type, we must hold onto the old method implementation while we implement a type-sensitive facade in its place.  The alias_method invocation literally copies the old less-than operator implementation and provides us with a way of referencing it within our later redefinition.  And what happens if someone else happens to monkey patch Fixnum and (for whatever reason) uses the identifier "__old_less_than__"?  Well, then we have problems.  It's like the old days of Lisp macros and endless identifier collisions.

It is true that this was an example specifically contrived to make Ruby look bad.  I could have implemented the overload using Groovy's meta-classes and been reasonably certain that everything would work out fine, but that's not the point.  The point is that there are a surprising number of situations where static typing serves not only to check for errors but also to allow extension patterns which would be otherwise impossible (or very, very difficult).  Dynamic typing isn't the panacea of extensibility that its proponents make it out to be, sometimes it isn't quite up to the task.

In fact (and this is where we come to my Digg-friendly point), I would submit that Scala (and to a lesser extent, C++) have created a mechanism for controlled extensibility which is more powerful than Ruby's open classes design.  That's not to say that there aren't situations which are easily solved using open classes and entirely intractable using only implicit conversions, but in my experience these scenarios are very rare.  In fact, I believe that it is far more common to run against a problem like my contrived overload which is greatly simplified through the use of static typing.

Ironically enough, some of Ruby's greatest pundits are starting to come around to the belief that a more controlled and well-defined model of class extension is required.  ParseTree is a Ruby framework which provides mechanisms for dynamically manipulating the AST of an expression prior to evaluation.  Conceptually, it is very similar to Lisp's macros and peripherally related to .NET's expression trees (used in LINQ).  ParseTree is used by a number of complex Ruby domain-specific languages, including Ambition, a fact which is extremely telling of how great the need is for just such a tool.  Having myself attempted a domain-specific language for constructing queries, I can state categorically that to do such a thing solely on the basis of open classes would be nearly impossible.  Even if successful, such a framework would be extremely volatile, sensitive to the slightest change in the Ruby core library, either caused by update or by other packages injecting their own meddlesome implementations into runtime classes.

Lex Spoon (co-author of Programming in Scala) once said that any language which seriously targeted domain-specific languages would have to create some sort of implicit conversion mechanism.  At the time, I was skeptical, convinced that Ruby (and similar) would always have the upper-hand in the area of class extension due to their dynamic treatment of modules and classes.  However, after some serious dabbling in the field of internal domain-specific languages, I'm beginning to come 'round to his point of view.  Implicit conversions are far from a weak imitation of Scala's dynamically typed "betters", they are a powerful and controlled way of extending types far beyond anything which can be easily accomplished through open classes.

Here’s the crux of the problem: people don’t test well.  Even with nice, behavior-driven development as facilitated by frameworks like RSpec, very few developers sufficiently test their code.  This isn’t just a problem with dynamic languages either, no one is safe from the test disability.  In some ways, it’s a product of laziness, but I think in most cases, good developers just don’t want to work on mundane problems.  It’s boring having to write unit test after unit test, checking and re-checking the same snippet of code with different input.

In some sense, it is this problem that compilers and static type systems try to avert, at least partially.  The very purpose of a static type system is to be able to prove certain things about your code simply by analysis.  By enabling the compiler to say things using the type system, the language is providing a safety net which filters out ninety percent of the annoying “no-brainer” mistakes.  A simple example would be invoking a method with the wrong parameters; or worse yet, misspelling the name of the method or type altogether.

The problem is that there are some problems which are more simply expressed in ways which are not provably sound.  In static languages, we get around this by casting, but such techniques are ugly and obviously contrived.  It is this problem which has given rise to the kingdom of dynamic languages; it is for this reason that most scripting languages have dynamic type systems: simple expression of algorithm without worrying about provability.  In fact, there are so many problems which do not fit nicely within most type systems that many developers have chosen to eschew static languages altogether, claiming that static typing just gets in the way.

Unfortunately, by abandoning static types, these languages lose that typo safety net.  It’s too easy to make a trivial mistake in a dynamic language, buried somewhere deep in the bowls of your application.  This mistake could easily be averted by a compiler with validating semantic analysis, but in a dynamic language, such a mistake could go unnoticed, conceivably even making it into production.  For this reason, most dynamic language proponents are also strong advocates of solid, comprehensive testing.  They have to be, for without such testing, one should never trust dynamic code in a production system (or any code, for that matter, but especially the unchecked dynamic variety).

Most large, production systems written in languages like Ruby or Groovy have large test suites which sometimes take hours to run.  These suites are extremely fine-grained, optimally checking every line of code with every possible kind of input, so as to be sure that mistakes are caught.  This is where the flexibility of dynamic typing really comes back to haunt you: extra testing is required to ensure that silly mistakes don’t slip through.  The irony is that a lot of developers using dynamic languages do so to get away from the “nuisance” of compilation, when all they have done is trade one inconvenience for another (testing).

Given this situation, it’s not unreasonable to conclude that what dynamic languages really need is a tool which can look through code and find all of those brain-dead mistakes.  Such a tool could be run along with the normal test suite, finding and reporting errors in much the same way.  It wouldn’t really have to be a compiler, so the tool wouldn’t slow down the development process, it would just be an effective layer of automated white-box testing.

But how could such a thing be accomplished in a language like Ruby?  After all, it is a truly dynamic language.  Methods don’t even exist until runtime, and sometimes only if certain code paths are run.  Types are completely undeclared, and every object can potentially respond to any method.  The answer is to perform extremely permissive inference.

It was actually a recent post by James Ervin on the nomenclature of type systems which got me thinking along these lines.  It should be possible by static analysis to infer the structural type of any value based on its usage.  Consider:

def do_something(obj)
  if obj.to_i == 0
    obj[:test]
  else
    other = obj.find :name => 'Daniel'
    other.to_s
  end
end

Just by examining this code, we can say certain things about the types involved.  For instance, we know that obj must respond to the following methods:

• to_i
• [Symbol]
• find(Hash)

In turn, we know that the find(Hash) method must return a value which defines to_s.  Of course, this last bit of information isn’t very useful, because every object defines that method, but it’s still worth the inference.  The really useful inference which comes out of to_s is the knowledge that this method sometimes returns a value of type String (making the assumption that to_s hasn’t been redefined to return a different type, which isn’t exactly a safe assumption).  At other times, do_something will return whatever value comes from the square bracket operator ([]) on obj.  This bit of information we must remember in the analysis.  We can’t just assume that this method will return a String all the time, even if to_s does because method return types need not be homogeneous in dynamic languages.

Now, at this point we have effectively built up a structural type which is accepted by do_something.  Literally, we have formalized in the analysis what our intuition has already told us about the method.  There are some gaps, but that is to be expected.  The key to this analysis is not attempting to be comprehensive.  Dynamic languages cannot be analyzed as if they were static, one must expect to have certain limitations.  In such situations where the analysis is insufficient, it must assume that the code is valid, otherwise there will be thousands of false positives in the error checking.

So what is it all good for?  Well, imagine that somewhere else in our application, we have the following bit of code:

do_something 42

This is something we know will fail, because we have a simple value (42) which has a nominal type we can easily infer.  A little bit of checking on this type reveals the fact that it does not define square brackets, nor does it define a find(Hash) method.  This finding could be reported as an error by the analysis engine.

Granted, we still have to account for the fact that Ruby has things like method_missing and open classes, but all of this can fall into the fuzzy area of the analysis.  In situations where it might be alright to pass an object which does not satisfy a certain aspect of the structural type, the analysis must let it pass without question.

You can imagine how this analysis could traverse the entire source tree, making the strictest inferences it can and allowing for dynamic fuzziness where applicable.  Since the full sources of every Ruby function, class and module are available at runtime, analysis could be performed without any undue concern regarding obfuscation or parsing of binaries.  Conceivably, most trivial errors could be caught without any tests being written, taking some of the burden off of the developer.  There is a slight concern that developers would build up a false sense of security regarding their testing (or lack thereof), but I think we just have to trust that won’t happen, or won’t last long if it does.

Most advanced Ruby toolsets already have an analysis somewhat similar to the one I outlined.  NetBeans Ruby for example has some fairly advanced nominal type inference to allow things like semantic highlighting and content assist.  But as far as I know, this type inference is only nominal, and fairly local at that.  The structural type inference that I am proposing could conceivably provide far better assurances and capabilities than mere nominal inference, especially if enhanced through successive iteration and a more “global” approach (similar to Hindley/Milner in static languages).

One thing is certain, it isn’t working to just rely on developers being conscientious with their testing.  With the rapid rise in production systems running on dynamic languages, it is in all of our best interests to try to find a way to make these systems more stable and reliable.  The best way to do this is to start with code assurance and try to make it a little less painful to catch mistakes before deployment.

Specs takes full advantage of Scala’s flexible syntax, offering a very natural format for structuring tests.  For example, we could write a simple specification for a hypothetical add method in the following way:

object AddSpec extends Specification {
  "add method" should {
    "handle simple positives" in {
      add(1, 2) mustEqual 3
    }
 
    "handle simple negatives" in {
      add(-1, -2) mustEqual -3
    }
 
    "handle mixed signs" in {
      add(1, -2) mustEqual -1
      add(-1, 2) mustEqual 1
    }
  }
}

We could go on, of course, but you get the picture.  This code will lead to the execution of four separate assertions in three tests (to put things into JUnit terminology).  Fundamentally, this isn’t too much different than a standard series of unit tests, just with a slightly nicer syntax.

Specs defines a domain-specific language for structuring test assertions in a simple and intuitive way.  However, this is hardly the only framework for BDD.  Perhaps the most well-known such framework is RSpec, which answers a similar use-case in the Ruby programming language.  Our previous specification could be rewritten using RSpec as follows:

describe AddLib do
  it 'should handle simple positives' do
    add(1, 2).should == 3
  end
 
  it 'should handle simple negatives' do
    add(-1, -2).should == -3
  end
 
  it 'should handle mixed signs' do
    add(1, -2).should == -1
    add(-1, 2).should == 1
  end
end

The end-result is basically the same: the add method will be tested against the given assertions (all four of them) and the results printed in some sort of report form.  In this area, RSpec is significantly more mature than Specs, generating very slick HTML reports and nicely formatted console output.  This isn’t really a fundamental weakness of the Specs framework however, just indicative of the fact that RSpec has been around for a lot longer.

These two frameworks are interesting of course, but they are merely implementations of a much larger concept: behavior-driven development.  I’ve never been much of a fan of unit testing.  It’s always seemed to be incredibly dull and a very nearly fruitless waste of effort.  As much as I hate it though, I have to bow to the benefits of a self-contained test suite; and so I press on, cursing JUnit every step of the way.

BDD provides a nice alternative to unit testing.  At its core, it is not much different in that test groupings and primitive assertions are used to check all aspects of a test unit against predefined data.  However, there is something about the “flow” of a behavioral spec that is considerably easier to deal with.  For some reason, it is far less painful to devise a comprehensive test suite using BDD principles than conventional unit testing.  It seems a little far-fetched, but BDD actually makes it easier to write (and more importantly, formulate) exactly the same tests.

It’s an odd phenomenon, one which can only be caused by the storyboard flow of the code itself.  It is very natural to think of distinct requirements for a test unit when each of these requirements are being labeled and entered in a logical sequence.  Moreover, the syntax of both Specs and RSpec is such that there is very little boiler-plate required to setup an additional test.  Compare the previous BDD specs with the following JUnit4 example:

public class MathTest {
    @Test
    public void testSimplePositives() {
        assertEquals(3, add(1, 2));
    }
 
    @Test
    public void testSimpleNegatives() {
        assertEquals(-3, add(-1, -2);
    }
 
    @Test
    public void testMixedSigns() {
        assertEquals(-1, add(1, -2));
        assertEquals(1, add(-1, 2));
    }
}

JUnit just requires that much more syntax.  It breaks up the logical flow of the tests and (more importantly) the developer train of thought.  What can be worse is this syntax bloat makes it very tempting to just group all of the assertions into a single test – to save typing if nothing else.  This is problematic because one assertion may shadow all the others in the case of a failure, preventing them from ever being executed.  This can make certain problems much more difficult to isolate.

Logical flow is extremely important to test structure.  BDD frameworks provide a very nice syntax for painlessly defining comprehensive test suites.  The really wonderful thing about all of this is that BDD is available on the JVM, right now.  There’s nothing stopping you from writing your code in Java as you normally would, then creating your test suite in Scala using Specs rather than JUnit.  Alternatively, you could use RSpec on top of JRuby, or Gspec with Groovy.  All of these are seamless replacements for a test framework like JUnit, and requiring of far less syntactic overhead.

The growing move toward polyglot programming encourages the use of a separate language when it is best suited to a particular task.  In this case, several languages are available which offer far more powerful test frameworks than those which can be found in Java.  Why not take advantage of them?

A while back, I evaluated Buildr as a potential replacement for Ant and Maven2 and eventually came to the unfortunate conclusion that it just wasn’t ready.  At the time, Buildr was still hampered by a number of annoying limitations (such as being unable to reconfigure the default directory structure).  I’m happy to say that many of these problems have been fixed in the 1.3 release, as well as a large number of feature additions which I hadn’t thought to request.  Despite all that, I’m still forced to conclude that Buildr simply isn’t (yet) suitable as my build tool of choice.

Now before you stone me for utter heresy, try to hear me out.  I’m not dissing Buildr in any way, I really want to use it, but I just can’t justify moving over to it in the face of all of its current issues.  What’s really unfortunate is that all of these issues can be summed up with a single word: transitivity.

One of Maven’s killer features is the ability to resolve the entire transitive dependency graph.  Now I’ll grant that it does this with varying degrees of success, but for the most part it’s pretty smart about how it fixes up your CLASSPATH.  As of 1.3, Buildr claims to have experimental support for transitive dependency resolution, but judging from the problems I encountered in my experimentation, it’s barely even deserving of mention as an experiment, much less a full-fledged feature.

To understand why transitive dependencies are such a problem, it is of course necessary to first understand the definition of such.  In a word (well, several anyway), a transitive dependency is an inherited dependency, an artifact which is not depended upon directly by the project, but rather by one of its dependencies.  This definitions carries recursively to dependent parents, grand-parents and so on, thus defining a transitive dependency graph.  Consider it this way:

In the diagram, MyCoolProject is the artifact we are trying to compile.  The only dependencies we have actually specified for this artifact are the DBPool and Databinder artifacts.  However, the Databinder artifact has declared that it depends upon both the Wicket and the ActiveObjects artifacts.  ActiveObjects doesn’t depend upon anything, but Wicket has dependencies SLF4J and on the Servlets API.  Thus, our original goal, MyCoolProject, has transitive dependencies Wicket, SLF4J, Servlets and ActiveObjects.  Quite a bit more than we thought we were asking for when we declared our dependency on Databinder.

In general, this sort of transitive resolution is a good thing.  It means that instead of specifying six dependencies, we only had to specify two.  Furthermore, we observed the DRY principle by not re-specifying dependency information already contained within the various packages.  As with any “good thing”, there’s definitely a valid argument regarding its down sides, but on the whole I’m quite fond of this feature.

In Maven, this sort of thing happens by default, which can lead to some very confusing and subtle CLASSPATH problems (conflicting packages, etc).  Buildr takes a slightly different approach in 1.3 by forcing the use of the transitive method:

repositories.remote << 'http://www.ibiblio.org/maven2'
 
define 'MyCoolProject' do
  project.version = '1.0'
 
  compile.with transitive('xmlrpc:xmlrpc-server:jar:3.0')
 
  package :jar
end

It’s easy to see why Buildr is raising such a ruckus in the build genre.  It’s syntax is elegance itself, and since it’s actually a proper scripting language (Ruby), there’s really nothing you can’t do with it.  Having to remember to specify the default repository is a bit of a pain, but it’s certainly something I can live with.  The real problem with this example is a little less subtle: It doesn’t work.

If you create a buildfile with the above contents and then run the buildr command, the results will be something like the following:

Downloading org.apache.xmlrpc:xmlrpc:jar:3.0
rake aborted!
Failed to download org.apache.xmlrpc:xmlrpc:jar:3.0, tried the following repositories:

http://www.ibiblio.org/maven2/

(See full trace by running task with --trace)

After a fairly significant amount of digging, I managed to discover that this problem is caused by the fact that Buildr attempts to download a corresponding JAR file for every POM it resolves.  This seems logical until you consider that many large projects (including Databinder, Wicket, and Hibernate) define POM-only projects which exist for the sole purpose of creating transitive dependencies.  They’re organizational units, designed to allow projects to depend upon one super-artifact, rather than a dozen sub-projects.  It’s a very common practice, and one which Buildr completely fails to handle appropriately.

After some prodding on the Buildr dev mailing-list, Assaf admitted that this is an issue worth looking at and provided a temporary workaround (pending a rework of the functionality in 1.4):

def pom_only(a)
  artifact a do |task|
    mkpath File.dirname(task.to_s)
    Zip::ZipOutputStream.open task.to_s
  end
end

This was about the point that I started wondering if maybe Ant/Ivy would be a better bet, at least for the time being.  To use this workaround, you must call the pom_only method once for every POM-only project in the dependency graph.  Usually, this means you must invoke Buildr repeatedly and find the troublesome artifacts by trial and error.  Not exactly a “just works” solution.

Pressing forward however, I unearthed a deeper, even more insidious issue: an intermittent failure to generate Eclipse project meta.  I’m not sure if this is due to the POM-only dependencies or just bad juju, but whatever the reason, it’s annoying.  I’ve raised the issue on the Buildr mailing lists, but so far no response.  Basically, what happens is something like this:

C:\\Users\\Daniel Spiewak\\Desktop\\MyCoolProject> buildr eclipse
(in C:/Users/Daniel Spiewak/Desktop/MyCoolProject, development)
Completed in 0.499s
C:\\Users\\Daniel Spiewak\\Desktop\\MyCoolProject>

Not exactly the most helpful output.  In case you were wondering, this process did not create the Eclipse metadata.  It’s interesting to note that calling buildr idea (to create project metadata for IntelliJ) seems to work just fine.  Whatever causes the bug, it seems to be specific to just the Eclipse project generator.

Buildr is a remarkable project.  It shows potential to someday become the de-facto build system, possibly even unseating Ant.  Unfortunately, that day is not today.  There are too many odd wrinkles and unpredictable errors to really call it a “finished product”.  Hopefully, the Buildr dev team will continue their excellent work, eventually producing a tool worthy of serious consideration.  Until then, I guess that I’m (still) stuck with Ant.

Update: It seems that the issues with transitive dependencies have been resolved in the latest Buildr versions in their SVN.  I’m looking forward to when this all becomes stable and publicly consumable!

• Download sample buildfile

There’s a lot of material out there on how to replace Java with Ruby “glue code” in your application.  The so-called “polyglot programming” technique states that we should embrace multiplicity of language in our applications.  Java may be very suitable for the core business logic of the application, but for actually driving the frontend UI, we may want to use something more expressive (like Ruby).  JRuby provides some powerful constructs which allow access to Java classes from within any Ruby application.  For example:

require 'java'
 
JFrame = javax.swing.JFrame
JButton = javax.swing.JButton
JLabel = javax.swing.JLabel
 
BorderLayout = java.awt.BorderLayout
 
class MainWindow < JFrame
  def initialize
    super 'My Test Window'
 
    setSize(300, 200)
    setDefaultCloseOperation EXIT_ON_CLOSE
 
    label = JLabel.new('You pushed the button', JLabel::CENTER)
      label.visible = false
    add label
 
    button = JButton.new 'Push Me'
    button.add_action_listener do
      label.visible = true
    end
    add(button, BorderLayout::SOUTH)
  end
end
 
window = MainWindow.new
window.visible = true

Not a terribly complex example, but it illustrates some of the major advantages of JRuby.  Notice how clean and concise this code is.  It wouldn’t have been much longer had I done this using Java, but it would certainly have been less readable.  Ruby is absolutely perfect for this sort of use case (driving a UI).

As I said though, there are a myriad of examples showing this sort of thing.  As such, it’s not a very interesting topic for a posting.  What the masses have failed to cover, however, is how to accomplish the opposite: calling from Java into Ruby.

Likely the reason this topic has received less attention is because Java is the language will the veritable zoo of libraries and frameworks.  The amount of effort and research that has been put into Java simply dwarfs the comparative immaturity of the Ruby offerings.  Given the disparity, why would you even want to call into Ruby from Java?  This conclusion seems logical until one remembers that almost any application which uses Ruby for the frontend must actually pass flow control to Ruby at some point.  This means calling some sort of Ruby code.

The Java Way

There is some information available on the JRuby Wiki.  The wiki article really should include the caveat that “some experimentation may be required.”  Sufficient information is available, but it is neither intuitive nor convenient.  From Java, the syntax for executing an arbitrary Ruby statement looks like this:

ScriptEngineManager m = new ScriptEngineManager();
ScriptEngine rubyEngine = m.getEngineByName("jruby");
ScriptContext context = engine.getContext();
 
context.setAttribute("label", new Integer(4), ScriptContext.ENGINE_SCOPE);
 
try {
    rubyEngine.eval("puts 2 + $label", context);
} catch (ScriptException e) {
    e.printStackTrace();
}

It’s a typical Java API: over-bloated, over-designed and over-generic.  What would be really nice is to have a syntax for accessing Ruby objects that is as seamless as accessing Java from Ruby.  I want to be able to call Ruby methods and use Ruby classes with the same ease that I can use Java methods and classes.  In short, I want an internal DSL for Ruby.

Unfortunately, Java is a bit constrained in this regard.  Java’s syntax is extremely rigid and does not lend itself well to DSL construction.  It’s certainly possible, but the result is usually less than satisfactory.  We could certainly construct an API around the the Java Scripting API (JSR-233) which provides more high-level access (such as direct method calls and object wrappers), but it would be clunky and only a marginal improvement over the original.

The good news is there’s another language tightly integrated with Java that has a far more flexible syntax.  Rather than building our JSR-233 wrapper in Java, we can avail ourselves of Scala’s power and flexibility, hopefully arriving at a DSL which approaches native “feel” in its syntax.

The Scala Way

Since we’re attempting to construct a tightly-integrated API for language calls, the most effective route would be to apply techniques already discussed in the context of DSL design.  As always, we start with the syntax and allow it to drive the implementation:

// syntax.scala
import com.codecommit.scalaruby._
 
object Main extends Application with JRuby {
  require("test")
 
  associate('Person)(new Person(""))
 
  println("Received from multiply: " + 'multiply(123, 23))
  println("Functional test: " + funcTest('test_string))
 
  val me = new Person("Daniel Spiewak")
  println("Name1: " + me.name)
  println("Name2: " + (me->'name)())
 
  me.name = "Daniel"
  println("New Name: " + me.name)
 
  println("Person#toString(): " + me)
 
  val otherPerson = 'create_person().asInstanceOf[AnyRef]
  println("create_person type: " + otherPerson.getClass())
  println("create_person value: " + otherPerson.send[String]("name")())
 
  eval("puts 'Ruby integration is amazing'")
 
  def funcTest(fun:(Any*)=>String) = fun()
}
 
class Person(name:String) extends RubyClass('Person, name) {
  def name = send[String]("name")()
  def name_=(n:String) = 'name = n
}

And the associated Ruby code:

# test.rb
class Person
  attr_reader :name
  attr_writer :name
 
  def initialize(name)
    @name = name
  end
 
  def to_s
    "Person: {name=#{name}}"
  end
end
 
def test_string
  'Daniel Spiewak'
end
 
def multiply(a, b)
  a * b
end
 
def create_person
  Person.new 'Test Person'
end

Obviously we’re going to need some heavy implicit type conversions.  The important thing to note is that we don’t see any residue of the Java Scripting API, it’s all been encapsulated by our DSL.  We’ve taken an API which is oriented around single-call, low-level invocations and created a high-level wrapper framework which allows method calls, instantiation and even some form of type-checking.

Starting from the top, we see a call which should be familiar to Rubyists, the require statement.  In our framework, this method call is just a bit of syntactic sugar around a call to eval(String).  This semantics are basically the same as within Ruby directly, with the exception of how Ruby source files are resolved.  Any script file on the CLASSPATH is fair game, in addition to the normal Ruby locations.  This allows us to easily embed Ruby scripts within application JARs, libraries and other Java distributables.

Moving down a bit further, we find a somewhat mysterious call to the curried associate(Symbol)(RubyObject) method.  The purpose of this invocation will become more apparent later on.  Suffice it to say that this step is necessary to allow Scala class wrappers around existing Ruby classes.

On the next line of interest, we see for the first time how the framework allows for seamless Ruby method invocation.  Unlike Ruby, Scala doesn’t allow us to simply handle calls to non-existent methods.  Because of this limitation, we have to be a bit more clever in how we structure the syntax.  In this case, we use Scala symbols to represent the method.  There doesn’t seem to be a terribly good explanation of symbols in Scala, but there’s plenty of information regarding how they work in Ruby.  Since the concepts are virtually identical, techniques are cross-applicable.

The key to the whole “symbols as methods” idea is implicit type conversion.  The JRuby trait inherits a set of conversions which look something like this:

implicit def sym2Method[R](sym:Symbol):(Any*)=>R = send[R](sym2string(sym))
implicit def sym2MethodAssign[R](sym:Symbol) = new SpecialMethodAssign[R](sym)
 
private[scalaruby] class SpecialMethodAssign[R](sym:Symbol) {
  def intern_=(param:Any) = new RubyMethod[R](str2sym(sym2string(sym) + "="))(param)
}

Though we haven’t looked at it yet, it is possible to infer the purpose of the send(String) method.  It’s function is to prepare a call to a Ruby method without actually invoking it.  This distinction allows us to pass Ruby methods around as method parameters, just like standard Scala methods.  The method returned is actually an instance of class RubyMethod[R] (where R is the return type).  Scala allows classes to extend structural types like methods, allowing us to redefine the method invocation semantics for wrapped Ruby calls.

class RubyMethod[R](method:Symbol) extends ((Any*)=>R) {
  import JRuby.engine
 
  override def apply(params:Any*) = call(params.toArray)
 
  private[scalaruby] def call(params:Array[Any]):R = {
    val context = engine.getContext()
    val plist = new Array[String](params.length)
 
    for (i <- 0 until params.length) {
      plist(i) = "res" + i
      context.setAttribute(plist(i), JRuby.resolveValue(params(i)), ScriptContext.ENGINE_SCOPE)
      plist(i) = "$" + plist(i)
    }
 
    evaluate(() => if (plist.length > 0) {
      sym2string(method) + "(" + plist.reduceLeft[String](_ + ", " + _) + ")"
    } else {
      sym2string(method) + "()"
    })
  }
 
  protected def evaluate(invoke:()=>String):R = {
    val toRun = invoke()
    Logger.getLogger("com.codecommit.scalaruby").info(toRun)
 
    JRuby.handleExcept(JRuby.wrapValue[R](engine, engine.eval(toRun, engine.getContext())))
  }
}

The gist of this code is simply to assign every parameter value to an attribute in the Ruby runtime.  Attributes of ENGINE_SCOPE (as defined by JSR-233) are represented as global variables within Ruby.  These variables are named sequentially starting from zero.  (e.g. $res0, $res1, …)  As you can imagine, this technique tends to be a bit of a concurrency killer.  To keep things simple, I decided to completely ignore the issues associated with asynchronous execution.  It is certainly possible to adapt the framework to function in a multi-threaded environment, but I didn’t bother to do it.  (one of the perks of blogging is a license to extreme laziness)

Once these parameters are assigned, the method call is evaluated within the context of the runtime.  This is done by literally generating the corresponding Ruby code (done in the anonymous method) and then wrapping the return value in an instance of RubyObject (if necessary).  Note that the send(String) method does not actually kick-start this invocation process at all.  Rather, it creates an instance of RubyMethod[R] which corresponds to the method name.  This class extends (Any*)=>R, so it may be used in the normal “method fashion” – by appending parentheses which enclose parameters (if any).

Supporting Cast

At this point, it’s worth taking a moment to examine the specifics of the framework class hierarchy.  A number of classes exist to wrap around Ruby objects and methods.  We’ve already seen a few of them (RubyMethod[R] and RubyObject), but it’s worth going into more detail as to their purpose and relation to one another. 

Note that these class names often conflict with existing classes in the JRuby implementation.  This odd coincidence is precipitated by the fact that the framework seems to deal with a lot of the same concepts as the JRuby runtime (go figure).  Rather than obfuscating my class naming to avoid conflict, I just assume that you will either make use of the enhanced Scala import feature (as I have in the implementation), or just avoid using the JRuby internal classes.

• RubyObject – The root of the object hierarchy.  This abstract class is designed to encapsulate the core functionality of the generic object (roughly: send, -> and eval) as well as containing all of the implicit type conversions.  Most of the syntax-defining magic happens here (more on this later).
• JRuby – This is the primary type interface between the developer and the framework.  Classes which wish to make use of Ruby integration must inherit from this trait.  This is where the Logger (for executed statements) is initialized and deactivated.  Within the corresponding object, all of the backend resources are managed.  This is where the actual ScriptEngineManager instance lives, as well as a set of utility methods to handle wrapping and unwrapping of framework-specific objects.
• RubyWrapperObject – This implementation of RubyObject is designed to wrap around instances which already exist within the Ruby interpreter.  For example, if a Ruby method returns an instance of ActiveRecord::Base, it will be represented in Scala by a corresponding instance of RubyWrapperObject.  Note that objects which are equivalent in the Ruby interpreter are not guaranteed to be pointer-equivalent.  However, the equals(Object) method is well-defined within RubyObject, thus comparisons between RubyObject instances will return sane results.  The == method in Scala is defined in terms of equals(Object), so existing code will behave rationally.
• RubyClass - With the exception of the JRuby trait, this is likely the only class within the framework which the developer will have to reference explicitly.  This class allows developer-defined Scala classes to wrap around existing Ruby classes, providing type-safe method calls and even extended functionality.  More on this feature later.
• RubyMethod – We’ve already seen how this serves as a wrapper around calls to Ruby methods.  However, its default implementation assumes that the method is defined in the global namespace.  This is impractical for many method calls (such as dispatch on an object).
• RubyInstanceMethod – This class solves the problem of object dispatch with RubyMethod.  All of the core functionality is identical to its superclass with the exception of the generated Ruby code.  Instead of just generating a method call passing parameters, this class will generate a method call on a given Ruby object.  Thus, this class depends upon RubyWrapperObject which maintains a reference to a corresponding Ruby instance.

Alternative Dispatch

Not every method call is made on the enclosing scope.  Sometimes it is necessary to call a method in an object to which you have a reference.  For example, a method may return an instance of a some Ruby class.  This instance will be automatically wrapped by a Scala instance of RubyWrappedObject.  Since this Scala class doesn’t actually define any methods which correspond to the Ruby class, it is necessary to once more utilize the “symbols as methods” trick in method dispatch.  There are two ways to call a method on an object like this: the send[R](String) method (where R is the return type), and the -> (arrow) operator.

Using the arrow operator is a lot like normal method calls, except with symbols instead of method names.  Just like dispatch on the enclosing scope, the call is converted into an instance of RubyMethod (actually, an instance of RubyInstanceMethod) which can then be used as a standard Scala method.  The difference between using arrow and dispatching on the enclosing scope is the syntax must be a little more contrived.

Parentheses have the second-highest priority of all the Scala operators (the dot operator (.) has the highest).  This means that if we simply “follow our nose” where the syntax is concerned, we will arrive at an order of invocation which leads to an undesirable result.  Consider the following sample:

val obj = 'create_person()
obj->'name()

The first call is a standard dispatch on the enclosing scope.  The second call is what is interesting to us.  Reading this line naturally (at least to old C/C++ programmers) we would arrive at the following sequence of events:

1. Get a reference to the name method from the instance contained within obj
2. Invoke the method, passing no parameters

Unfortunately, this is not how the compiler sees things.  Because parentheses bind tighter than the arrow operator, it actually resolves the expression in the following way:

1. Get a reference to the name method contained within the enclosing scope
2. Invoke the method, passing no parameters
3. Invoke the -> method on the instance within obj, passing the result of name as a parameter

This is obviously not what we wanted.  Unfortunately, there’s no way to make the arrow operator bind tighter than parentheses.  This is a good thing from a language standpoint, but it causes problems for our syntax.

The solution is to enclose any “arrow dispatch” statement within parentheses so as to force the order of evaluation:

val obj = 'create_person()
(obj->'name)()

It looks a bit weird, but it’s the only way Scala will allow this to work.  This call now evaluates properly, calling the name method on the obj instance, passing no parameters.

There’s actually another problem associated with arrow dispatch in our DSL: Scala already has an implicit meaning for the arrow operator.  The following sample should look familiar to those of you who have worked with Scala in other applications:

val numbers = Map(1 -> "one", 2 -> "two", 3 -> "three")

By default, Scala defines the arrow operator as an alternative syntax for defining 2-tuples.  This is good for most things, but bad for us.  What we want is to define a new implicit type conversion which converts Any into a corresponding instance of RubyWrappedObject.  This would allow us to satisfy the syntax given above.  However, Scala’s 2-tuple syntax already defines an implicit type conversion for the Any type which deals with the arrow operator.  Rather than examining the context to attempt to disambiguate, the Scala compiler simply gives up and prints an error stating that the implicit type conversions are ambiguous.  This poses a bit of a problem and nearly killed the arrow operator idea in design.

The solution is actually to override Scala’s built-in conversion by defining our own conversion with the same name and signature but which provides us with the option of using our own arrow operator definition.  The behavior we want is to allow normal use of the arrow operator when dealing with Any -> Any, but convert to RubyWrappedObject and dispatch when dealing with Any -> Symbol.  After a little digging through the Scala standard library, I arrived at the following solution (defined in RubyObject):

implicit def any2ArrowAssoc[A](a:A) = new SpecialArrowAssoc(a)
 
private[scalaruby] class SpecialArrowAssoc[A](a:A) extends ArrowAssoc(a) {
  def ->(sym:Symbol) = (a match {
    case obj:RubyObject => obj
    case other => new RubyWrapperObject(other)
  })->sym
}

Notice that we extend Scala’s pre-existing ArrowAssoc[A] class (which handles the special 2-tuple syntax) and then overload the -> method to work differently with symbols.  This code now does precisely what we need.  By introducing this extra layer of indirection, as well as by overriding Scala’s existing conversion, we’re able to support the arrow syntax as shown in the above examples.

Sending Messages

There is one final form of dispatch which allows typed return values: send[R](String).  This is actually the method to which all the other dispatch forms delegate (as it is the most general).  This method is very similar to the Ruby send method which allows Smalltalk-style message passing on arbitrary objects.  The really important thing about this method though is that it will automatically cast the return value from the method to whatever type you specify, allowing you to define type-safe wrappers around existing Ruby methods in Scala:

def multiply(a:Int, b:Int) = send[Int]("multiply")(a, b)
 
val result:Int = multiply(123, 23)

send is effectively defined as a curried function since it takes a method name as a parameter and returns an instance of RubyMethod as a result.  This mimics the behavior of dispatch with symbol literals in that you can use send to generate type-safe partially-applied functions for corresponding Ruby methods.

Note that send could just as easily have taken a symbol as a parameter, rather than a string.  However, the metaphor throughout the DSL is “symbols as methods”, thus string was used to avoid logical conflict.  Scala itself was perfectly happy passing symbol literals around in addition to treating them as methods.

Class Wrapping

The final bit of code in the example now so far above us serves as a sample of how one might wrap an existing Ruby class within Scala.  Person is actually a class defined in Ruby (as you can see from the Ruby sources).  It has a read/write attribute, name, as well as an overridden to_s method.  RubyObject already contains the logic for handling calls to toString() and proxying them to Ruby’s to_s, but the name attribute must be handled explicitly in code.

The goal is basically to provide a type-safe wrapper around the Person Ruby class.  We could just as easily dispatch on the automatically wrapped instance of RubyWrappedObject using either syntax described above, but an explicit wrapper is a bit nicer.  The compiler can check things for us, and we can even add methods to the class (at least, as far as Scala is concerned) in true Ruby “open class” style.  All that is necessary to accomplish this wrapper is to extend RubyClass and to define the delegating wrapper methods:

class Person(name:String) extends RubyClass('Person, name) {
  def name = send[String]("name")()
  def name_=(n:String) = 'name = n
}

We specify which Ruby class we are wrapping as the first parameter in the constructor for RubyClass.  The parameters which follow are passed directly to the constructor of the corresponding Ruby class.  This Ruby constructor is invoked automatically, instantiating the corresponding wrapped Ruby object in the background.  Notice that we specify the name of the Ruby class using a symbol.  This is the one place in the framework that we break with the “symbols as methods” metaphor.  The consequence is a nice, clean syntax for Ruby class wrapping.  Unfortunately, it also means that wrapping a class within a non-included namespace (e.g. ActiveRecord::Base) can be a little clunky.  The only way to do it is to explicitly invoke the Symbol(String) constructor.  (this is required because Scala symbols can only contain alpha-numerics and underscores)

Once we have our wrapped class signature, it’s easy to define the delegate methods.  Scala encourages a blurring of field and method, similar to Ruby.  As such, it supports a very Ruby-esque syntax for accessor/mutator pairs.  This makes the wrapped syntax just a bit nicer.  For the accessor, we make a call to the send method, specifying the return type necessary for the wrapper.  The mutator allows us to be a bit more creative.

We don’t really need type-safe return values for a mutator.  We would normally just set the return type as Unit and ignore the result.  Thus we can once again use the symbol dispatch syntax.  Notice that this time we’re not directly treating a symbol as a method.  We’re apparently assigning a value to the symbol using the = operator (corresponds to the operator= assignment operator in C++).  This is possible through a separate implicit type conversion which generates a one-off utility instance:

private[scalaruby] class SpecialMethodAssign[R](sym:Symbol) {
  def intern_=(param:Any) = new RubyMethod[R](str2sym(sym2string(sym) + "="))(param)
}

As you can see, all this method does is generate a new symbol which includes the ‘=’ character and returns the result of dispatching on the corresponding Ruby method.  Note that mutators in Ruby are defined as “=“, thus appending “=” to the method name is the appropriate behavior.

Return Value Wrapping

There’s actually a slight problem involved in allowing Scala wrappers around existing Ruby types.  Well, not so much a problem as an inconsistency.  The problem is simply this: if a Ruby method creates an instance of a Ruby class for which there is a Scala wrapper and returns this value through the framework into Scala, one would expect this value would be wrapped into an instance of the Scala wrapper.  If you look in the example far above, there is an example of this in the create_person method.  The method creates an instance of Ruby class Person and returns it as a result.

Somehow, the framework must identify that there is a corresponding Scala wrapper and then properly create an instance.  This actually poses something of a dilemma in two ways.  Number one, Scala has no equivalent to Ruby’s ObjectSpace, so there’s no way to get a comprehensive list of all classes which have been defined.  Even if we could get this list, the corresponding Ruby class is specified in the constructor parameters to RubyClass, so there’s no way to obtain the information statically from outside the class.  Number two, we have to somehow create an instance of the Scala wrapper class without creating a corresponding instance of the wrapped Ruby class (since we already have one).  This means we need some sort of override in the RubyClass constructor.

The best solution to all of these problems is to introduce the associate method.  The usage is demonstrated at the top of the example where we associate the Person Ruby class with the Person Scala wrapper class.  More specifically, we associate the Ruby class with a pass-by-name parameter which defines how to instantiate the Scala class.  This is an important distinction as it solves our second problem of instance creation.  The framework has no way of knowing what parameters must be passed to the Scala wrapper constructor, so the instantiation itself must be passed:

associate('Person)(new Person(""))

As I mentioned previously, this is a pass-by-name parameter which means that it will not be immediately evaluated, but rather on-demand somewhere in the body of associate.  The associate method actually takes this value and wraps it in an anonymous method which invokes the instantiation each time a value of Ruby type Person must be wrapped.  Just prior to invoking the constructor, an override is put in place within the RubyClass singleton object (not shown in the class hierarchy) to prevent the creation of a corresponding Ruby instance.  This is what allows the new instance of Scala class Person to correspond with an existing Ruby value.  Here again we’re sacrificing concurrency for a hacky work-around to a complex problem.  Any sort of “proper” implementation would have to solve this problem in a more elegant way.

It Never Ends!

This post, that is.  There’s so much more I could ramble on about (I never even talked about how exceptions are handled), but this entry is already far too long.  Hopefully the material presented here only serves to whet your appetite for slicker JRuby-Scala integration and all the benefits it can bring.  I’ve packaged up the framework presented here as a downloadable archive.  The package includes the Ruby engine for the Java Scripting API as well as a jar-complete build made from the JRuby SVN.  The project may work with JRuby 1.0, but I doubt it.  Anyway, JRuby 1.1 is due shortly, so why bother.  Remember that this is extremely untested and very experimental.  (I did warn you about the concurrency issues, right?)  If this is interesting to people, I may do a proper release into an OSS project somewhere.  For right now, I just don’t have the time. 

I hope this entry gives you an idea of what’s involved in Scala DSL implementation, as well as an idea of where such a technique may be useful in your own projects.  After all, what would be better than everyone being able to write their own Rails-killer and define highly fluid APIs!

• Download unnamed-scala-ruby.zip (requires Java 6 and Scala 2.7.0)

So in that sense, XML is a mixed blessing.  Its flexibility and intuitive nature allows developers to store just about any data in a human readable, easy-to-debug manner.  Unfortunately, its verboseness often makes generating the actual XML a very frustrating and boring foray into the land of boiler-plate.  Various techniques have been developed over the years to smooth this process (e.g. manipulating a DOM tree or reflectively marshalling objects directly to XML), but on the whole, generating XML in code is just as annoying as it has always been.  We’ve all written code like this in the past:

public String toXML() {
    final String INDENT = "    ";
    StringBuilder back = new StringBuilder(
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n");
 
    back.append("<person name=\"").append(getName()).append("\">\n");
    for (Book book : getBooks()) {
        back.append(INDENT).append("<book>").append(book.getTitle()).append("</book>\n");
    }
    back.append("</person>");
 
    return back.toString();
}

Not the most pleasant of algorithms.  Oh there’s nothing complex or challenging about the code, it’s just annoying (as String manipulation often is).  Things would get a little more interesting if we actually had some sort of recursive hierarchy to traverse, but even then it would still be pretty straight-forward.  XML generation is tedious grudge-work to which we all must submit from time to time.

Domain Specific Languages

There’s a new wave in programming (especially in the communities surrounding dynamic languages like Ruby and Groovy) called “Domain Specific Language”, or DSL for short.  The DSL technique has really been around for a long time, but it’s just now finding its way to the mainstream, “Citizen Joe” developers.  This is because for decades, domain specific languages have been separate languages unto themselves, with their own syntax, quirks, and utter lack of tool support.

For example, if a company wanted to specify their business logic in a more readable format using a DSL, the would have to spend months, sometimes years of effort to build an entirely new language, just for the one application.  While there was no need to generate a truly flexible and extensible syntax (or even one which was Turing-complete), such efforts would still require an enormous amount of work to be put into trivialities like parsers, AST walkers and core libraries.  Obviously this meant that DSLs were extremely uncommon.  There are very few use-cases for something which requires so much and gains you so little.

This variety of domain specific language is called an “External DSL”.  This name stems from the fact that the language is completely independent and external to other languages.  The innovation which made the DSL attainable for the common-man is that of the “Internal DSL”.

People often struggle to precisely define internal DSLs.  In the broadest sense, an internal DSL is a language API carefully structured to satisfy a particular syntax when used in code.  There is no syntax parsing involved in implementing an internal DSL.  Rather, effort is focused on the API itself.  The more flexible the syntax of the underlying language, the more powerful and potentially intuitive the syntax of the DSL can be.  It is for this reason that languages such as Ruby, Python, Groovy and company are often used to implement internal DSLs.  These languages are defined by extremely flexible and dynamic syntax, lending themselves perfectly to such efforts.

With this in mind, it should theoretically be possible to design an “API” for XML generation that’s simple and intuitive.  The DSL could be implemented using Ruby, though actually sufficiently dynamic language could do.  Implementing an internal DSL is a notoriously difficult task, so perhaps a step-by-step walk through is in order.

Getting Started

The very first step in creating an internal DSL is to design the syntax.  Similar to how test-driven development starts with a set of unit tests and builds functionality which satisfies the tests, DSL development starts with a few code samples and builds an API to satisfy the syntax.  This step will guide all of the code we write as we implement the DSL.

One of the primary goals for our DSL syntax should be to reflect (as much as possible) the structure and essence of the XML output in the generating code.  One of the major shortcomings of the ad hoc “string concat” XML generation technique is an utter lack of logical structure in the code.  Sure your algorithm may be nicely organized and formatted, but does it really reflect the actual structure of the XML it’s generating?  Probably not.  Another major goal should be brevity.  XML generating code is extremely verbose, and the whole idea behind writing a DSL for XML generation is to elevate some of this hassle.

So, we’ve got brevity and logical structure.  As minor goals, we also may want to spend some effort making the syntax versatile.  We don’t want the algorithm to perform incorrectly if we try to generate a document with the < character in some field.  With that said however, we don't need to be overly concerned with flexibility.  Yes, our DSL should be as capable as possible, but not to the detriment of brevity and clarity.  These concerns are paramount, functionality can take a back seat.

With these goals in mind, let’s try writing a static syntax for our Person/Book example above. Bear in mind that every construct used in the syntax must be valid Ruby which the interpreter will swallow.  How it will swallow is unimportant right now.  For this step, it’s helpful to remember that the ruby -c command will perform a syntax check on a source file without actually attempting to run any sort of interpretation.

xml.person :name => 'Daniel "Leroy" Spiewak' do
  book do
    'Godel, Escher, Bach'
  end
  book do
    'The Elegant Universe'
  end
  book do
    'Fabric of the Cosmos'
  end
  book do
    "The Hitchhiker's Guide to the Galaxy"
  end
  book    # book with no title
end
 
puts xml  # prints the generated XML

Note that this is all static, there’s no dynamically generated data to muddle the picture.  We can worry about that later.

This code sample gives us a rough idea of what the syntax should look like.  According to ruby -c, it’s valid syntax, so we’re ready to proceed.  Everything looks fairly unencumbered by meta-syntax, and the logical structure is certainly represented.  Now we do a quick mental pass through the syntax to ensure that all of our functional bases are covered.  In the example we’ve used attributes, nested elements and text data.  One of our attributes is making use of illegal XML characters (double quotes).  We haven’t tried mixing elements and text yet, but it’s fairly obvious how this could be done.

Strictly speaking, we haven’t rigorously defined anything.  What we have done is given ourselves a framework to build off of.  This sample will serve as an invaluable template to guide our coding.

Parsing the Syntax

Remember I said that internal DSLs do not involve any syntax parsing?  Well, in a way I was wrong.  The next thing we need to do is walk through the syntax ourselves and understand how Ruby will understand it.  This step requires some fairly in-depth knowledge of Ruby’s (or whatever language you’re using) syntax.  Hopefully the following diagram will help to clarify the process.

I’ve annotated the areas of interest in cute, candy colors to try and illustrate roughly what the Ruby syntax parser will see when it looks at this code.  It’s important to understand this information since it is the key to translating our contrived syntax into a rigorous API.

For the sake of this discussion, I’m going to assume you have a working knowledge of Ruby and have already recognized some of the more basic syntax elements within the sample.  For example, the do/end block is just that, a Block object (roughly the Ruby equivalent of a closure).  The annotated String literal is an implicit return value for the inner block (the last statement of a block is implicitly the return value).  This syntax employs a Ruby “trick” which allows us to drop the return statement (an important trick, since you can’t return from a block anyway).

Anyone familiar with Ruby on Rails should be aware of Ruby’s convenient Hash literal syntax employed for the element attributes.  As annotated, this literal is being passed as a method parameter (Ruby allows us to drop the parentheses).  Here again we’re using little oddities in the Ruby syntax to clean up our DSL and reduce unnecessary cruft.

You should also be familiar with the blurred line between variable and method so common in Ruby.  As annotated, the xml syntax element could be either a local field, or a method that we’re invoking without the parentheses.  To clear up this point, we need to refer to the full sample above.  In no place do we declare, nor do we provide for the declaration of a local variable xml.  Thus, xml() will be a method within our API available in the global scope.

Now we get to the really interesting stuff: those mysterious undefined methods.  As with other dynamic languages, Ruby allows method invocations upon non-existent methods.  To someone from a static language background, this seems rather peculiar, but it’s perfectly legal I assure you.  In a sense, it’s much like the “methods as messages” metaphor employed by Objective-C and Smalltalk.  When you call the method, you’re just broadcasting a message to the object, hoping someone answers the call.

In Ruby, when a method is called which has no corresponding handler, a special method called method_missing is invoked.  This method is what allows us to handle messages even without a pre-defined method body.  ActiveRecord makes extensive use of this feature.  Every time you access a database field, you’re calling a non-existent method and thus, method_missing.

So based on the first non-existent method we’re calling (person(…)), we can already say something about our API.  Somewhere in the global scope, we’re going to need to have a method called xml().  This method will return an instance of a class which defines method_missing and so can handle our person(…) invocation.  It seems this implementation of method_missing will also need to optionally take a Block as the final parameter, allowing it to pass execution on to its logical children.  When referring back to our original sample, the final line seems to indicate that the instance must implement the to_s() method, allowing puts() to implicitly convert it to a String.

Implementation

All that work and we still haven’t even written a solid line of code.  What we have done though is given ourselves a clear idea of where we need to go, and a rough outline for how it can be done.  We’ve laid the foundation for the implementation by giving ourselves two critical starting points: xml() and that mysterious outer-scoped method_missing.  We may not know how we’re going to implement all this yet, but we at least have an idea on where to begin.

Starting with the easy one, let’s implement a basic framework around xml().

require 'singleton'
 
module XML
  class XMLBuilder
    attr_reader :last
 
    private
    def initialize
    end
 
    def method_missing(sym, *args, &block)
      # ...
    end
 
    def to_s
      @last
    end
 
    def self.instance
      @@instance ||= new
    end
  end
end
 
def xml
  XML::XMLBuilder.instance
end

Notice how we’re using a singleton instance of XMLBuilder?  That’s because there’s no need for us to have more than one instance exposed to the DSL users.  XMLBuilder is just a placeholder class that dispatches the first level of commands for the DSL and will assemble the result for us as it executes.  Thus, XMLBuilder can be considered the root of our DSL, the corner-stone upon which the entire API is bootstrapped.  We do however need to allow for other, private instances as we’ll see later on.

Another item worthy of note in this snippet is the non-standard method_missing signature.  This is because we will actually need the block as a proper Proc object down the line.  A block parameter (prefixed with &) is the only parameter which can follow a varargs parameter (prefixed with *) and there can only be one of them.

We can now try a first-pass implementation of method_missing.  This implementation is really just a sample with a very significant shortcoming.  The actual implementation is quite a bit more complex.

def method_missing(sym, *args, &block)
  @last = "<#{sym.to_s}"
 
  if args.size > 0 and args[0].is_a? Hash  # never hurts to be sure
    args[0].each do |key, value|
      @last += " #{key.to_s}=\"#{value.to_s}\""
    end
  end
 
  if block.nil?
    @last += "/>"   # if there's no children, just close the tag
  else
    @last += ">"
 
    builder = XMLBuilder.new
    builder.instance_eval block
 
    @last += builder.last
 
    @last += "</#{sym.to_s}>"
  end
end

Again, this is just the rough idea.  In our actual implementation we need to be concerned about things like valid attribute values (as demonstrated in our sample), proper element names, etc.

The key to the whole algorithm is the instance_eval invocation.  This single statement passes control to the next block down and starts the process all over again.  The important thing about this is evaluating the block within the context of the an XMLBuilder instance, rather than just its enclosing context.  This allows the nested block to take advantage of the same method_missing implementaiton, hence implicitly recursing further into the XML tree.  This technique is extremely powerful and absolutely critical to a lot of DSL implementations.

You’ll also notice that this is breaking the singleton pattern we established in our class design.  This is because a separate instance of XMLBuilder is required to handle each nested block within the DSL tree.  It’s very important to remember that we’re not actually exposing this instance in the public API, it’s just a tool we use within the implementation.  The API users will still only see the singleton XMLBuilder instance.

So a bit more semantically, what we’re doing is the following:

1. Handle the root non-existant method invocation
2. Deal with any attributes in the single Hash parameter (if exists)
3. Check for nested block. If found, create a new builder instance and use its context to evaluate the child block
4. Within child block evaluation, recurse to step 1 for each method invocation
5. Accumulate result from child block evaluation and return

Of course, this doesn’t deal with nested text content within elements.  However, the principles of the implementation are fairly clear and the rest is just code.

As with all internal DSLs, the user-friendly DSL syntax is supported by an API that’s ugly, hacky and heavily dependant on language quirks (such as the super-critical instance_eval).  In fact, it has been said that internal DSLs encourage the use of language quirks if it simplifies the API from the end-developer standpoint.  Of course this makes the end-developer code very clean and easy to maintain at the cost of the DSL developer’s code, which is nightmarish and horrible to work with.  It’s a tradeoff that must be considered when the decision is made to go with a DSL-style API.

Conclusion

Hopefully this was a worthwhile trek into the gory innards of implementing an internal DSL.  I leave you with one final code sample to whet your appetite for the fully-implemented API.  This is an extract from the ActiveObjects Ant build file converted to use our new DSL.  It’s interesting that this converted version is significantly cleaner than the original, XML form.

require 'xmlbuilder'
 
xml.project :name => 'ActiveObjects', :default => :build do
  dirname :property => 'activeobjects.dir', :file => '${ant.file.ActiveObjects}'
  property :file => '${activeobjects.dir}/build.properties'
 
  target :name => :init do
    mkdir :dir => '${activeobjects.dir}/bin'
  end
 
  target :name => :build, :depends => :init do
    javac :srcdir => '${activeobjects.dir}/src', :source => 1.5, :debug => true
  end
 
  target :name => :build_test, :depends => [:init, :build] do
    property :name => 'javadoc.intern.path', :value => '${activeobjects.dir}/${javadoc.path}'
  end
end
 
puts xml

This will print the following XML:

<?xml version="1.0" encoding="UTF-8"?>
 
<project name="ActiveObjects" default="build">
  <dirname property="activeobjects.dir" file="${ant.file.ActiveObjects}"/>
  <property file="${activeobjects.dir}/build.properties"></property>
  <target name="init">
    <mkdir dir="${activeobjects.dir}/bin" />
  </target>
  <target name="build" depends="init">
    <javac source="1.5" debug="yes" srcdir="${activeobjects.dir}/src"/>
  </target>
  <target name="build_test" depends="init,build">
    <property name="javadoc.intern.path" value="${activeobjects.dir}/${javadoc.path}"></property>
  </target>
</project>

The fully implemented DSL is available in a single Ruby file. Also linked are some examples to provide a more balanced view of the capabilities.

• xmlbuilder.rb – The actual DSL implementation
• antfile.rb
• htmlexample.rb
• testfile.rb

Several different solutions have been proposed to workaround this limitation.  The canonical technique involves intensifying tests and increasing test coverage.  Ruby has some excellent unit test frameworks (such as RSpec) which serve to ease the pain associated with this approach, but no matter how you slice it, tests are a pain.  Having to rely on tests to take the place of type checking in the code assurance process can be extremely frustrating.

Another, less common technique is to simply perform dynamic type checks within the method itself.  Like so:

def create_name(fname, lname)
  raise "fname must be a String" unless fname.kind_of? String
  raise "lname must be a String" unless lname.kind_of? String
 
  fname + " " + lname
end

This code explicitly checking the dynamic kind of the parameter values to ensure that they are of type or subtype of String.  The issues with this sample should be relatively obvious.

Primarily, it’s ugly!  This sort of repetitious, boiler-plate conditional checking is exactly the sort of thing Ruby tries to avoid.  What’s more is the added bulk of all of these repetitive checks (assuming you perform one check per-parameter per-method) because far more unwieldy than just improving the rspec test coverage.

While manually type checking may be a bad solution syntactically, it’s on the right track conceptually.  What we really want is some sort of assertion that the parameters are of a certain type, but that won’t overly bloat our existing code.  We need some sort of framework that will “weave in” (think AOP) its type assertions without getting in the way our our algorithms.

Well it turns out that someone’s already done this.  Eivind Eklund kindly pointed me to his type checking framework in a comment on a previous post.  The basic idea is to perform the type checking assertions, but to factor the work out into an API encapsulated by an intuitive DSL.  So rather than performing all those nasty unless statements as above, we could simply do something like this:

typesig String, String
def create_name(fname, lname)
  fname + " " + lname
end

It’s really as simple as that.  Passing the type values to the typesig method just prior to a method declaration give the cue to the Types framework to perform some extra work on each call that method.  Now we have the runtime assurances that the following code will not work (with a very intuitive error message):

create_name("Daniel", 123)

Will produce the folling output:

ArgumentError: Arg 1 is of invalid type (expected String, got Fixnum)

But the fun doesn’t stop there.  Ruby encourages the “duck typing” pattern, where algorithm developers concern themselves not with what the value is but rather what it does.  This means that the type checking really should be done based on what methods are available, not just the raw type.  It turns out that the Types framework supports this as well:

class Company
  def name
    "Blue Danube"
  end
end
 
class Person
  def name
    "Daniel Spiewak"
  end
end
 
typesig String, Type::Respond(:name)
def output(msg, value)
  puts msg + " " + value.name
end
 
c = Company.new
p = Person.new
 
output("The company name is: ", c)
output("The person is: ", p)
 
output("The programmer is: ", "a genius")    # error

Types can check not only the kind of the object but also to what methods it responds.  This is crucial to enabling its adoption into modern Ruby code bases, many of which rely heavily on this “duck typing” technique.

You can think of the Types framework just like another layer in your testing architecture.  Obviously it’s not performing any sort of static type checking (since Ruby has no compile phase).  All it’s doing is providing that extra certainty that you’re never passing something weird from somewhere in your code, something that would break your algorithm.

So what’s the catch?  Well, obviously you need to have the Types framework installed.  It’s not as easy as just typing gem install types either, since the framework actually predates Ruby Gems.  You’ll have to download the framework and then copy around the types.rb file yourself.  But this is just deployment semantics.  The more interesting issue are the limitations of the code itself.

As far as I can tell, the only restriction on the framework is that it must be used within a proper class, not in the root scope.  This means that all of my examples above would have to be enclosed in a class, rather than just copy-pasted into a .rb file and run in place.  But other than this one limitation, the framework is incredibly flexible.  I really haven’t shown you the seriously interesting stuff in terms of the API (there are more examples at the top of the types.rb file).  In many ways, Types is actually more powerful than any static type checking mechanism could be (yes, I’m even including Scala in that evaluation).

I haven’t had a chance to use Types on any serious project myself, but I can see tremendous potential, particularly for companies with large-scale Ruby/Rails deployments or even smaller projects looking for just a bit tighter code assurance.  As far as I’m concerned, there shouldn’t be a non-trivial Ruby project attempted without this lovely library, Rails or no Rails.