Skip to content
Print

XMLBuilder: A Ruby DSL Case Study

10
Mar
2008

XML is probably the most ubiquitous and most recognizable format in the modern development landscape.  It’s simple power in representing hierarchically structured data has made it the standard for representing everything from Word documents to databases.  It’s also one of the most verbose and meta-rich syntaxes known to man.

So in that sense, XML is a mixed blessing.  Its flexibility and intuitive nature allows developers to store just about any data in a human readable, easy-to-debug manner.  Unfortunately, its verboseness often makes generating the actual XML a very frustrating and boring foray into the land of boiler-plate.  Various techniques have been developed over the years to smooth this process (e.g. manipulating a DOM tree or reflectively marshalling objects directly to XML), but on the whole, generating XML in code is just as annoying as it has always been.  We’ve all written code like this in the past:

public String toXML() {
    final String INDENT = "    ";
    StringBuilder back = new StringBuilder(
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n");
 
    back.append("<person name=\"").append(getName()).append("\">\n");
    for (Book book : getBooks()) {
        back.append(INDENT).append("<book>").append(book.getTitle()).append("</book>\n");
    }
    back.append("</person>");
 
    return back.toString();
}

Not the most pleasant of algorithms.  Oh there’s nothing complex or challenging about the code, it’s just annoying (as String manipulation often is).  Things would get a little more interesting if we actually had some sort of recursive hierarchy to traverse, but even then it would still be pretty straight-forward.  XML generation is tedious grudge-work to which we all must submit from time to time.

Domain Specific Languages

There’s a new wave in programming (especially in the communities surrounding dynamic languages like Ruby and Groovy) called “Domain Specific Language”, or DSL for short.  The DSL technique has really been around for a long time, but it’s just now finding its way to the mainstream, “Citizen Joe” developers.  This is because for decades, domain specific languages have been separate languages unto themselves, with their own syntax, quirks, and utter lack of tool support.

For example, if a company wanted to specify their business logic in a more readable format using a DSL, the would have to spend months, sometimes years of effort to build an entirely new language, just for the one application.  While there was no need to generate a truly flexible and extensible syntax (or even one which was Turing-complete), such efforts would still require an enormous amount of work to be put into trivialities like parsers, AST walkers and core libraries.  Obviously this meant that DSLs were extremely uncommon.  There are very few use-cases for something which requires so much and gains you so little.

This variety of domain specific language is called an “External DSL”.  This name stems from the fact that the language is completely independent and external to other languages.  The innovation which made the DSL attainable for the common-man is that of the “Internal DSL”.

People often struggle to precisely define internal DSLs.  In the broadest sense, an internal DSL is a language API carefully structured to satisfy a particular syntax when used in code.  There is no syntax parsing involved in implementing an internal DSL.  Rather, effort is focused on the API itself.  The more flexible the syntax of the underlying language, the more powerful and potentially intuitive the syntax of the DSL can be.  It is for this reason that languages such as Ruby, Python, Groovy and company are often used to implement internal DSLs.  These languages are defined by extremely flexible and dynamic syntax, lending themselves perfectly to such efforts.

With this in mind, it should theoretically be possible to design an “API” for XML generation that’s simple and intuitive.  The DSL could be implemented using Ruby, though actually sufficiently dynamic language could do.  Implementing an internal DSL is a notoriously difficult task, so perhaps a step-by-step walk through is in order.

Getting Started

The very first step in creating an internal DSL is to design the syntax.  Similar to how test-driven development starts with a set of unit tests and builds functionality which satisfies the tests, DSL development starts with a few code samples and builds an API to satisfy the syntax.  This step will guide all of the code we write as we implement the DSL.

One of the primary goals for our DSL syntax should be to reflect (as much as possible) the structure and essence of the XML output in the generating code.  One of the major shortcomings of the ad hoc “string concat” XML generation technique is an utter lack of logical structure in the code.  Sure your algorithm may be nicely organized and formatted, but does it really reflect the actual structure of the XML it’s generating?  Probably not.  Another major goal should be brevity.  XML generating code is extremely verbose, and the whole idea behind writing a DSL for XML generation is to elevate some of this hassle.

So, we’ve got brevity and logical structure.  As minor goals, we also may want to spend some effort making the syntax versatile.  We don’t want the algorithm to perform incorrectly if we try to generate a document with the < character in some field.  With that said however, we don't need to be overly concerned with flexibility.  Yes, our DSL should be as capable as possible, but not to the detriment of brevity and clarity.  These concerns are paramount, functionality can take a back seat.

With these goals in mind, let’s try writing a static syntax for our Person/Book example above. Bear in mind that every construct used in the syntax must be valid Ruby which the interpreter will swallow.  How it will swallow is unimportant right now.  For this step, it’s helpful to remember that the ruby -c command will perform a syntax check on a source file without actually attempting to run any sort of interpretation.

xml.person :name => 'Daniel "Leroy" Spiewak' do
  book do
    'Godel, Escher, Bach'
  end
  book do
    'The Elegant Universe'
  end
  book do
    'Fabric of the Cosmos'
  end
  book do
    "The Hitchhiker's Guide to the Galaxy"
  end
  book    # book with no title
end
 
puts xml  # prints the generated XML

Note that this is all static, there’s no dynamically generated data to muddle the picture.  We can worry about that later.

This code sample gives us a rough idea of what the syntax should look like.  According to ruby -c, it’s valid syntax, so we’re ready to proceed.  Everything looks fairly unencumbered by meta-syntax, and the logical structure is certainly represented.  Now we do a quick mental pass through the syntax to ensure that all of our functional bases are covered.  In the example we’ve used attributes, nested elements and text data.  One of our attributes is making use of illegal XML characters (double quotes).  We haven’t tried mixing elements and text yet, but it’s fairly obvious how this could be done.

Strictly speaking, we haven’t rigorously defined anything.  What we have done is given ourselves a framework to build off of.  This sample will serve as an invaluable template to guide our coding.

Parsing the Syntax

Remember I said that internal DSLs do not involve any syntax parsing?  Well, in a way I was wrong.  The next thing we need to do is walk through the syntax ourselves and understand how Ruby will understand it.  This step requires some fairly in-depth knowledge of Ruby’s (or whatever language you’re using) syntax.  Hopefully the following diagram will help to clarify the process.

image1

I’ve annotated the areas of interest in cute, candy colors to try and illustrate roughly what the Ruby syntax parser will see when it looks at this code.  It’s important to understand this information since it is the key to translating our contrived syntax into a rigorous API.

For the sake of this discussion, I’m going to assume you have a working knowledge of Ruby and have already recognized some of the more basic syntax elements within the sample.  For example, the do/end block is just that, a Block object (roughly the Ruby equivalent of a closure).  The annotated String literal is an implicit return value for the inner block (the last statement of a block is implicitly the return value).  This syntax employs a Ruby “trick” which allows us to drop the return statement (an important trick, since you can’t return from a block anyway).

Anyone familiar with Ruby on Rails should be aware of Ruby’s convenient Hash literal syntax employed for the element attributes.  As annotated, this literal is being passed as a method parameter (Ruby allows us to drop the parentheses).  Here again we’re using little oddities in the Ruby syntax to clean up our DSL and reduce unnecessary cruft.

You should also be familiar with the blurred line between variable and method so common in Ruby.  As annotated, the xml syntax element could be either a local field, or a method that we’re invoking without the parentheses.  To clear up this point, we need to refer to the full sample above.  In no place do we declare, nor do we provide for the declaration of a local variable xml.  Thus, xml() will be a method within our API available in the global scope.

Now we get to the really interesting stuff: those mysterious undefined methods.  As with other dynamic languages, Ruby allows method invocations upon non-existent methods.  To someone from a static language background, this seems rather peculiar, but it’s perfectly legal I assure you.  In a sense, it’s much like the “methods as messages” metaphor employed by Objective-C and Smalltalk.  When you call the method, you’re just broadcasting a message to the object, hoping someone answers the call.

In Ruby, when a method is called which has no corresponding handler, a special method called method_missing is invoked.  This method is what allows us to handle messages even without a pre-defined method body.  ActiveRecord makes extensive use of this feature.  Every time you access a database field, you’re calling a non-existent method and thus, method_missing.

So based on the first non-existent method we’re calling (person(…)), we can already say something about our API.  Somewhere in the global scope, we’re going to need to have a method called xml().  This method will return an instance of a class which defines method_missing and so can handle our person(…) invocation.  It seems this implementation of method_missing will also need to optionally take a Block as the final parameter, allowing it to pass execution on to its logical children.  When referring back to our original sample, the final line seems to indicate that the instance must implement the to_s() method, allowing puts() to implicitly convert it to a String.

Implementation

All that work and we still haven’t even written a solid line of code.  What we have done though is given ourselves a clear idea of where we need to go, and a rough outline for how it can be done.  We’ve laid the foundation for the implementation by giving ourselves two critical starting points: xml() and that mysterious outer-scoped method_missing.  We may not know how we’re going to implement all this yet, but we at least have an idea on where to begin.

Starting with the easy one, let’s implement a basic framework around xml().

require 'singleton'
 
module XML
  class XMLBuilder
    attr_reader :last
 
    private
    def initialize
    end
 
    def method_missing(sym, *args, &block)
      # ...
    end
 
    def to_s
      @last
    end
 
    def self.instance
      @@instance ||= new
    end
  end
end
 
def xml
  XML::XMLBuilder.instance
end

Notice how we’re using a singleton instance of XMLBuilder?  That’s because there’s no need for us to have more than one instance exposed to the DSL users.  XMLBuilder is just a placeholder class that dispatches the first level of commands for the DSL and will assemble the result for us as it executes.  Thus, XMLBuilder can be considered the root of our DSL, the corner-stone upon which the entire API is bootstrapped.  We do however need to allow for other, private instances as we’ll see later on.

Another item worthy of note in this snippet is the non-standard method_missing signature.  This is because we will actually need the block as a proper Proc object down the line.  A block parameter (prefixed with &) is the only parameter which can follow a varargs parameter (prefixed with *) and there can only be one of them.

We can now try a first-pass implementation of method_missing.  This implementation is really just a sample with a very significant shortcoming.  The actual implementation is quite a bit more complex.

def method_missing(sym, *args, &block)
  @last = "<#{sym.to_s}"
 
  if args.size > 0 and args[0].is_a? Hash  # never hurts to be sure
    args[0].each do |key, value|
      @last += " #{key.to_s}=\"#{value.to_s}\""
    end
  end
 
  if block.nil?
    @last += "/>"   # if there's no children, just close the tag
  else
    @last += ">"
 
    builder = XMLBuilder.new
    builder.instance_eval block
 
    @last += builder.last
 
    @last += "</#{sym.to_s}>"
  end
end

Again, this is just the rough idea.  In our actual implementation we need to be concerned about things like valid attribute values (as demonstrated in our sample), proper element names, etc.

The key to the whole algorithm is the instance_eval invocation.  This single statement passes control to the next block down and starts the process all over again.  The important thing about this is evaluating the block within the context of the an XMLBuilder instance, rather than just its enclosing context.  This allows the nested block to take advantage of the same method_missing implementaiton, hence implicitly recursing further into the XML tree.  This technique is extremely powerful and absolutely critical to a lot of DSL implementations.

You’ll also notice that this is breaking the singleton pattern we established in our class design.  This is because a separate instance of XMLBuilder is required to handle each nested block within the DSL tree.  It’s very important to remember that we’re not actually exposing this instance in the public API, it’s just a tool we use within the implementation.  The API users will still only see the singleton XMLBuilder instance.

So a bit more semantically, what we’re doing is the following:

  1. Handle the root non-existant method invocation
  2. Deal with any attributes in the single Hash parameter (if exists)
  3. Check for nested block. If found, create a new builder instance and use its context to evaluate the child block
  4. Within child block evaluation, recurse to step 1 for each method invocation
  5. Accumulate result from child block evaluation and return

Of course, this doesn’t deal with nested text content within elements.  However, the principles of the implementation are fairly clear and the rest is just code.

As with all internal DSLs, the user-friendly DSL syntax is supported by an API that’s ugly, hacky and heavily dependant on language quirks (such as the super-critical instance_eval).  In fact, it has been said that internal DSLs encourage the use of language quirks if it simplifies the API from the end-developer standpoint.  Of course this makes the end-developer code very clean and easy to maintain at the cost of the DSL developer’s code, which is nightmarish and horrible to work with.  It’s a tradeoff that must be considered when the decision is made to go with a DSL-style API.

Conclusion

Hopefully this was a worthwhile trek into the gory innards of implementing an internal DSL.  I leave you with one final code sample to whet your appetite for the fully-implemented API.  This is an extract from the ActiveObjects Ant build file converted to use our new DSL.  It’s interesting that this converted version is significantly cleaner than the original, XML form.

require 'xmlbuilder'
 
xml.project :name => 'ActiveObjects', :default => :build do
  dirname :property => 'activeobjects.dir', :file => '${ant.file.ActiveObjects}'
  property :file => '${activeobjects.dir}/build.properties'
 
  target :name => :init do
    mkdir :dir => '${activeobjects.dir}/bin'
  end
 
  target :name => :build, :depends => :init do
    javac :srcdir => '${activeobjects.dir}/src', :source => 1.5, :debug => true
  end
 
  target :name => :build_test, :depends => [:init, :build] do
    property :name => 'javadoc.intern.path', :value => '${activeobjects.dir}/${javadoc.path}'
  end
end
 
puts xml

This will print the following XML:

<?xml version="1.0" encoding="UTF-8"?>
 
<project name="ActiveObjects" default="build">
  <dirname property="activeobjects.dir" file="${ant.file.ActiveObjects}"/>
  <property file="${activeobjects.dir}/build.properties"></property>
  <target name="init">
    <mkdir dir="${activeobjects.dir}/bin" />
  </target>
  <target name="build" depends="init">
    <javac source="1.5" debug="yes" srcdir="${activeobjects.dir}/src"/>
  </target>
  <target name="build_test" depends="init,build">
    <property name="javadoc.intern.path" value="${activeobjects.dir}/${javadoc.path}"></property>
  </target>
</project>

The fully implemented DSL is available in a single Ruby file. Also linked are some examples to provide a more balanced view of the capabilities.

Comments

  1. Hi,

    Maybe you know where can I find some info about internal DSLs in Scala? I have been looking on google and nabble but there is not that much information. Or maybe you know at least some libraries using such concepts? I would be highly interested.

    Thanks,
    Mateusz

    jau Monday, March 10, 2008 at 1:09 am
  2. Pardon me. English is not my first language. Can someone please explain what this means:

    “It’s flexibility and intuitive nature allows developers to store just about any data in a human readable, easy-to-debug manner”

    I expanded the contraction (?) to:

    “It is flexibility and intuitive nature allows developers to store just about any data in a human readable, easy-to-debug manner”

    but that does not make any sense to me. Any help would be very welcomed. This article looks great, but I cannot understand it all :-(

    Jean Francois Monday, March 10, 2008 at 4:22 am
  3. Something similar is possible in Java, too. I blogged about this some time ago: http://dschneller.blogspot.com/2007/08/building-xml-groovy-way-in-java.html

    In the meantime I have improved (internally) upon that by providing static methods that wrap the different constructors. The benefit is a “lighter” syntax. By statically importing those factory methods, you can get rid of all the “new XElement(” stuff in my examples and instead just write code like

    elem(“parent”,
    elem(“child1″, “value1″),
    elem(“child2″, “value2″)
    )

    (hope this is readable in the comments).

    However one has to admit, that the ruby source is way shorter than the implementation of XElement and XAttribute.

    Daniel Schneller Monday, March 10, 2008 at 5:39 am
  4. @jau
    There really isn’t much info available anywhere. Coderspiel has done some experimenting with DSL-like constructs here: http://technically.us/code/x/the-awesomeness-of-scala-is-implicit#lets_wrap_component_construction_whiners_in_duct_tape You can also see a bit more Scala DSL techniques in this article on DZone: http://osgi.dzone.com/news/creating-domain-specific-langu

    The good news (at least, I hope it’s good) is that I’ve been working on some stuff with internal DSLs in Scala, so there are a few articles coming down the pipe. The best one of them is done, but I can’t publish it right away. Hopefully soon, but watch the RSS. :-)

    @Jean
    Good eye! It seems that I’ve fallen victim to that subtle difference between the apostrophy as a contraction and as a possessive modifier. The correct sentence is as follows:

    “Its flexibility and intuitive nature allows developers to store just about any data in a human readable, easy-to-debug manner.”

    Does this make more sense?

    @Daniel
    Very interesting! I hadn’t seen your post before, which surprises me. It looks a lot like how functional languages construct trees, which seems fairly intuitive to me. I’ll have to give this a closer look.

    Daniel Spiewak Monday, March 10, 2008 at 10:19 am
  5. @daniel
    That’s nice to hear. I’m looking forward for any info. BTW. I really like your blog. Keep up the good work :)

    jau Monday, March 10, 2008 at 10:21 am
  6. Daniel,

    Ahh, thank you. I have not yet learned popular English “errors”. Thank you again.

    Best,
    Jean Francois

    Jean Francois Tuesday, March 11, 2008 at 1:44 am
  7. Thanks for the post. It’s really amazing. I read a lot of blogs where most of them just write pile of contents about various things but they don’t show the “code in action.” This blog mostly deals with code and that’s what makes it one of my favorite blogs of all time.

    Thanks again! Keep writing!!
    Srikanth

    Srikanth Thursday, March 13, 2008 at 10:24 pm

Post a Comment

Comments are automatically formatted. Markup are either stripped or will cause large blocks of text to be eaten, depending on the phase of the moon. Code snippets should be wrapped in <pre>...</pre> tags. Indentation within pre tags will be preserved, and most instances of "<" and ">" will work without a problem.

Please note that first-time commenters are moderated, so don't panic if your comment doesn't appear immediately.

*
*