Jun
2008
As I have previously written, I have recently been spending some time experimenting with various aspects of Scala, including some of the frameworks which have become available. One of the frameworks I have had the privilege of using is the somewhat unassumingly-titled Specs, and implementation of the behavior-driven development methodology in Scala.
Specs takes full advantage of Scala’s flexible syntax, offering a very natural format for structuring tests. For example, we could write a simple specification for a hypothetical add method in the following way:
object AddSpec extends Specification { "add method" should { "handle simple positives" in { add(1, 2) mustEqual 3 } "handle simple negatives" in { add(-1, -2) mustEqual -3 } "handle mixed signs" in { add(1, -2) mustEqual -1 add(-1, 2) mustEqual 1 } } } |
We could go on, of course, but you get the picture. This code will lead to the execution of four separate assertions in three tests (to put things into JUnit terminology). Fundamentally, this isn’t too much different than a standard series of unit tests, just with a slightly nicer syntax.
Specs defines a domain-specific language for structuring test assertions in a simple and intuitive way. However, this is hardly the only framework for BDD. Perhaps the most well-known such framework is RSpec, which answers a similar use-case in the Ruby programming language. Our previous specification could be rewritten using RSpec as follows:
describe AddLib do it 'should handle simple positives' do add(1, 2).should == 3 end it 'should handle simple negatives' do add(-1, -2).should == -3 end it 'should handle mixed signs' do add(1, -2).should == -1 add(-1, 2).should == 1 end end |
The end-result is basically the same: the add method will be tested against the given assertions (all four of them) and the results printed in some sort of report form. In this area, RSpec is significantly more mature than Specs, generating very slick HTML reports and nicely formatted console output. This isn’t really a fundamental weakness of the Specs framework however, just indicative of the fact that RSpec has been around for a lot longer.
These two frameworks are interesting of course, but they are merely implementations of a much larger concept: behavior-driven development. I’ve never been much of a fan of unit testing. It’s always seemed to be incredibly dull and a very nearly fruitless waste of effort. As much as I hate it though, I have to bow to the benefits of a self-contained test suite; and so I press on, cursing JUnit every step of the way.
BDD provides a nice alternative to unit testing. At its core, it is not much different in that test groupings and primitive assertions are used to check all aspects of a test unit against predefined data. However, there is something about the “flow” of a behavioral spec that is considerably easier to deal with. For some reason, it is far less painful to devise a comprehensive test suite using BDD principles than conventional unit testing. It seems a little far-fetched, but BDD actually makes it easier to write (and more importantly, formulate) exactly the same tests.
It’s an odd phenomenon, one which can only be caused by the storyboard flow of the code itself. It is very natural to think of distinct requirements for a test unit when each of these requirements are being labeled and entered in a logical sequence. Moreover, the syntax of both Specs and RSpec is such that there is very little boiler-plate required to setup an additional test. Compare the previous BDD specs with the following JUnit4 example:
public class MathTest { @Test public void testSimplePositives() { assertEquals(3, add(1, 2)); } @Test public void testSimpleNegatives() { assertEquals(-3, add(-1, -2); } @Test public void testMixedSigns() { assertEquals(-1, add(1, -2)); assertEquals(1, add(-1, 2)); } } |
JUnit just requires that much more syntax. It breaks up the logical flow of the tests and (more importantly) the developer train of thought. What can be worse is this syntax bloat makes it very tempting to just group all of the assertions into a single test - to save typing if nothing else. This is problematic because one assertion may shadow all the others in the case of a failure, preventing them from ever being executed. This can make certain problems much more difficult to isolate.
Logical flow is extremely important to test structure. BDD frameworks provide a very nice syntax for painlessly defining comprehensive test suites. The really wonderful thing about all of this is that BDD is available on the JVM, right now. There’s nothing stopping you from writing your code in Java as you normally would, then creating your test suite in Scala using Specs rather than JUnit. Alternatively, you could use RSpec on top of JRuby, or Gspec with Groovy. All of these are seamless replacements for a test framework like JUnit, and requiring of far less syntactic overhead.
The growing move toward polyglot programming encourages the use of a separate language when it is best suited to a particular task. In this case, several languages are available which offer far more powerful test frameworks than those which can be found in Java. Why not take advantage of them?
Comments
Very cool article
Just one thing I want to mention. If you want to have the exact equivalent of the rspec example, you can remove a “should” level and write:
object AddSpec extends Specification {
“it should handle simple positives” in {
add(1, 2) mustEqual 3
}
“it should handle simple negatives” in {
add(-1, -2) mustEqual -3
}
“it should handle mixed signs” in {
add(1, -2) mustEqual -1
add(-1, 2) mustEqual 1
}
}
And if you want purely anonymous examples, because your code reads like no one, you can write:
object AddSpec extends Specification {
add(1, 2) mustEqual 3
add(-1, -2) mustEqual -3
add(1, -2) mustEqual -1
add(-1, 2) mustEqual 1
}
(this works in the current release but I enhanced the naming in the latest SNAPSHOT release: 1.3.0-SNAPSHOT).
Cheers,
Eric (specs developer).
I wasn’t aware of that syntax. Thanks for the heads-up, Eric!
As an aside, it’s worth mentioning that the should/in syntax is one of the things that I really liked about Specs as opposed to RSpec and similar. I’m sure there’s a way to replicate the syntax somehow with rspec, but it just falls out naturally with Specs. Very nice job!
G’day Daniel,
Have you seen Reductio at http://reductiotest.org/ ?
In particular, the Scala examples? http://reductiotest.org/examples/scala
I’m a bit skeptical. Is it that much of an improvement to group your tests this way?
Eventually, you want to be pointed to a Java method so you can inspect it and debug it, so the information ’should handle mixed signs’ seems to be one extra layer of noise over “shouldHandleMixedSigns()”…
–
Cedric
@Tony
Yes, I’ve looked Reductio, but only briefly. It’s about on the same level as ScalaCheck: a testing solution which is quite obviously superior in coverage and minimalism, but also way beyond my limited intellect.
I’m planning on really digging into combinitorial testing one of these days, but at the moment it’s easier for me to stick with the conventional canned-data methodology.
@Cedric
To be honest, I was quite skeptical too when I first started playing with Specs. It’s a very difficult phenomenon to describe, one which almost requires you to experience it for yourself. Obviously the reduced syntax cruft is one benefit of DSLs like Specs, but the overall flow is really the big win. Somehow - and I’m honestly not sure how - BDD specifications are psychologically easier to formulate, easier to code and (ultimately) easier to verify correct.
Oh, and for the record: I have looked at TestNG, and I do find it to be a nicer framework than JUnit. However, it’s still a straight-up unit test framework. TestNG is to JUnit what SVN is to CVS: vastly improved, but still fundamentally the same.
You’re too hard on yourself mate
Feel free to ask any questions.
While I agree completely with your example on here, I think that the Java example is perhaps a little harsh; in JUnit 4 you now have the Hamcrest matchers which do a surprisingly good job (not nearly as good as Specs or RSpec, of course) considering that Java is the language.
Written in this style the Java example would be:
public class MathTest {
@Test
public void simplePositives() {
assertThat( add(1, 2), is( 3 ) );
}
@Test
public void simpleNegatives() {
assertThat( add( -1, -2 ), is( 3 ) );
}
@Test
public void testMixedSigns() {
assertThat( add( 1, -2 ), is( -1 ) );
assertThat( add( -1, 2 ), is( 1 ) );
}
}
It at least reads a little more sensibly, and you can specify more complex Matchers which can be used like “is”. I love the stuff that you can do with the simple-but-powerful syntax within Scala; Specs is just one example but there are many more.
Looks a lot like Hecl’s test suite in terms of how it flows, and the lack of extraneous crud getting in your way. test intro-3 { proc bar {} { return puts } proc foo {} { [bar] xyz } foo intro proccode foo } {[bar] xyz}
JTestr: http://jtestr.codehaus.org/ is also an alternative to consider.
Daniel, don’t forget Easyb (http://easyb.org) on Groovy’s options.
Great post!
As previously pointed out, the Java example is a poor straw man. JUnit 4.4 has BDD extensions (Hamcrest) and has been shipping for almost a year.
Hamcrest doesn’t really solve everything. It is a more “fluid” syntax, but it’s also significantly more verbose. It’s not too bad, but it really feels like strapping a saddle onto a dinosaur: you just can’t “fix” JUnit. It’s a nice tool because it’s so ubiquitous, but you can’t say much more for it.
Post a Comment