Skip to content
Print

Groovy’s Performance is Not Subjective

31
Mar
2008

Ah, the saga of misinterpreted micro-benchmarks!  Developers advocating one technology over another have long used micro-benchmarks and trivial examples to illustrate their point.  This is most often seen in the area of language implementation, where performance is a critical (and often emotional) consideration.  The Computer Language Benchmarks Game is a good example of this.

With the rise of the JVM as a platform for language development, we’ve been seeing a lot of benchmarks directly comparing some of these languages with each other and with Java itself.  Actually, it seems that the popular trio (as far as benchmarking goes) are Scala, JRuby and Groovy.  It’s been a bit of a favorite past-time of the blogosphere to create benchmarks for these languages and then spin up controversial posts about the results.  Back in September, Darek Young wrote a post illustrating a ray tracer benchmark which really caught my eye:

Results

ray.java
12.89s

ray.scala
11.224s

ray.groovy
2h 31m 42s

I was expecting the Groovy code to run longer than the Scala code but was shocked at the actual difference. All three versions of the code produce identical images: (fullsize here)

Wow!  I’ve seen these results dozens of times (looking back at the post), but they never cease to startle me.  How could Groovy be that much slower than everything else?  Granted it is very much a dynamic language, compared to Java and Scala which are peers in static-land.  But still, this is a ray tracer we’re talking about!  There’s no meta-programming involved to muddle the scene, so a halfway-decent optimizer should be able to at least squeeze that gradient down to maybe 5x as slow, rather than a factor of 830.

If this were an isolated incident, I would probably just blow it off as bad benchmarking, or perhaps an odd corner case that trips badness in the Groovy runtime.  Then a week later, I read this post by Pete Knego:

Test Groovy Java Java vs Groovy (times faster)
Simple counter 8.450 150 56x
Binary tree building 19.500 2.580 7.6x
Binary tree traversing 2.530 76 33x
Prime numbers 43.270 1.170 37x

All [non-decimal] times are in milliseconds.

Well, this is really disappointing. I expected Groovy to be slower but not by that much. In order to understand where does such a performance hit come from we have to peek under the hood. The culprit of all this is of course Groovy’s dynamic nature, implemented as MOP. MOP is a way for Groovy to know which class elements (fields/properties, methods, interfaces, superclasses, etc..) are defined on an object and to have a way to alter that data or invoke it. The core of MOP are two methods defined on GroovyObject: get/setProperty() and invokeMethod(). This methods are called every time you access a field or call a method on a Groovy object and quite a lot of work is done behind the scenes. The internals of the MOP are listed in MetaClass interface and implemented in six different classes.

All of this is old news, so the question is: Why am I bringing this up now?  Well, I recently saw a post on Groovy Zone by none-other-than Rick Ross, talking about this very subject.  Rick’s post was in response to two posts (here and here), discussing ways to improve Groovy code performance by obfuscating code.  Final result?

This text is being written as I was changing and trying things, I gained 20s from
minor changes of which I lost track. :-) I am currently at 1m30s (down from the
original 4m and comparing with Java’s 4s).

I’m sorry, this is acceptable performance?  This is someone who’s spent time trying to optimize Groovy, and by his own admission, Groovy is 23x slower than the equivalent Java code.  Certainly this is a far cry from the 830x slower in the ray tracer benchmark, but in this case it’s simple string manipulation, rather than a mathematically intensive test.

Coming back to Rick’s entry, he looks at the conclusion and has this to say about it:

Language performance is highly overrated

Much is often made of the theoretical “performance” of a language based on benchmarks and arcane tests. There have even been cases where vendors have built cheats into their products specifically so they would score well on benchmarks. In the end, runtime execution speed is not as important a factor as a lot of people would think it is if they only read about performance comparisons. Other factors such as maintainability, interoperability, developer productivity and tool and library support are all very significant, too.

Wait a minute, that sounds a lot like something else I’ve read recently!  Maybe something like this:

Is picking out the few performance weaknesses the right way to judge the
overall speed of Groovy?

To me the Groovy performance is absolutely sufficient because of the
easy integration with Java. If something’s too slow, I do it in Java.
And Java compared to Python is in most cases much faster.

I appreciate the efforts of the Groovy team to improve the performance,
but if they wouldn’t, this would be no real problem to me. Groovy is the
grooviest language with a development team always having the simplicity
and elegance of the language usage in mind – and that counts to me.  :-)

This is almost a mantra for the Groovy proponents: performance is irrelevant.  What’s worse, is that the few times where they’ve been pinned down on a particular performance issue that’s obviously a problem, the response seems to be along the lines of: this test doesn’t really show anything, since micro-benchmarks are useless.

I’m sorry, but that’s a cop-out.  Face up to it, Groovy’s performance is terrible.  Anyone who claims otherwise is simply not looking at the evidence.  Oh, and if you’re going to claim that this is just a function of shoe-horning a dynamic language onto the JVM, check out a direct comparison between JRuby and and Groovy.  Groovy comes out ahead in only four of the tests.

What really bothers me about the Groovy performance debates is that most “Groovyists” seem to believe that performance is in the eye of the beholder.  The thought is that it’s all just a subjective issue and so should be discounted almost completely from the language selection process.  People who say this have obviously forgotten what it means to try to write a scalable non-trivial application which performs decently under load.  When you start getting hundreds of thousands of hits an hour, you’ll be willing to sell your soul for every last millisecond.

The fact is that this is simply not the case.  Performance is important and it must be considered when choosing a language for your next project.  Now I’m not implying that it should be the sole consideration, after all, we don’t use Assembly much anymore.  But we can’t just discard performance altogether as a means of evaluation.  Developer productivity is important, but it means nothing if the end result doesn’t meet basic responsiveness standards.

It’s like the first time I tried Mingle (which is written using JRuby BTW) back when it was still in pre-beta.  The application was amazing, but it took literally minutes to render the most basic of pages.  The problem was closely related to the state of the JRuby interpreter at the time and its poor memory management with respect to Rails.  In the end, the application was completely unusable.  ThoughtWorks had produced an amazing piece of software in a remarkably short time span, but the cost was the use of a language and framework which (at the time) led to unredeemable performance.  They spared the developers at the expense of the end-users.

Both Mingle and JRuby have come a long way since those first tests.  Charles and the gang have put an intense amount of effort into optimizing the JRuby runtime and JIT compiler.  They’ve gone from an implementation which was 5-10x slower than MRI 1.8.6 to a fully compatible implementation that is usually faster than the original.  Obviously performance is achievable in a dynamic language on the JVM, so why is Groovy still so horrible?

The only answer I can think of is that the Groovy core team just doesn’t value performance.  Why else would they consistently bury their heads in the sand, ignoring the issues even when the evidence is right in front of them?  It’s as if they have repeated their own “performance is irrelevant” mantra so many times that they are actually starting to believe it.  It’s unfortunate, because Groovy really is an interesting effort.  I may not see any value for my needs, but I can understand how a lot of people would.  It fills a nice syntactic niche that other languages (such as Ruby) just miss.  But all of its benefits are for naught if it can’t deliver when it counts.

Comments

  1. Isn’t the value of Groovy that anytime you need performance you can fall back to Java?

    Alexis MP Monday, March 31, 2008 at 2:11 am
  2. I both agree and disagree. Performance IS subjective…sometimes. A system need only be “fast enough”, but yes, your points are well taken. The Groovy hippies take this to a religious extreme, at least publicly, at least sometimes. Even the ruby camp is (slowly) coming around to the fact that yes, ruby’s dog slow. Often so slow that it’s NOT “fast enough”.

    (FWIW, I like both strong typed systems, and dynamic languages. Yeah, I can’t be satisfied.)

    Michael Campbell Monday, March 31, 2008 at 3:51 am
  3. You said: “…why is Groovy still so horrible?
    The only answer I can think of is that the Groovy core team just doesn’t value performance.”

    Wrong. The Groovy Developers are avoiding the pre-mature optimization trap. First get it working, THEN make it fast.

    The next release of Groovy, Groovy v 1.6 beta 1, is due out in a few weeks. It’s much faster than the current v 1.5.4.

    Grover Monday, March 31, 2008 at 5:02 am
  4. “The only answer I can think of is that the Groovy core team just doesn’t value performance. Why else would they consistently burry their heads in the sand, ignoring the issues even when the evidence is right in front of them? ”

    Then you need to think harder or *gasp* maybe ask them. They are pretty good at answering stuff like this. I am pretty sure this is talked about on the lists fairly regularly.

    That takes away from your rant though.

    Robert Monday, March 31, 2008 at 5:38 am
  5. “a halfway-decent optimizer should be able to at least squeeze that gradient down to maybe 5x as slow”

    More like 10x slower – that’s the target for John Wilson’s Ng language. The Groovy team have made some improvements in performance (mainly call-site caching), slated for version 1.6 I think.

    John Wright Monday, March 31, 2008 at 9:11 am
  6. > The great language shootout is a good example of this

    It hasn’t been called ‘The great language shootout’ for at least 2 years.

    It hasn’t been called any kind of ’shootout’ for 11 months.

    Isaac Gouy Monday, March 31, 2008 at 9:13 am
  7. > More importantly, this test was run with the current stable version of JRuby (1.0). JRuby 1.1 (due out soon)

    Actually they were run with jruby1.1b1 by Nov 4 2007, and jruby1.1RC1 by Jan 9 2007

    Isaac Gouy Monday, March 31, 2008 at 9:20 am
  8. Daniel, as a matter of fact performance is a _BIG_ issue in groovy, just take a glance at the discussions on the Groovy dev list, the main focus of the 1.5.x and 1.6.x series is performance, performance, performance. True, at some point ‘Groovy advocates’ would dismiss performance as being a key concern to choose Groovy, after all you get more stuff and you are paying a price, but you can’t keep paying the price in every release.

    Let’s also remember that Charlie and friends have a day job working in JRuby, the Groovy picture was completely different until G2One came along 6 months ago, so yes, Groovy is becoming faster, more performant now, and we will continue boosting it as much as we can.

    My $0.02

    Andres Almiray Monday, March 31, 2008 at 9:23 am
  9. @Alexis

    Saying that Groovy isn’t slow because you can always drop down to Java is like saying that Java is suitable for OS development since you can always drop down to C. As soon as you drop into Java, you’re no longer in Groovy. Yes, I know they have simultaneous compilation which really does a good job of blurring the line between the languages, but that doesn’t change the fact that it *is* a separate language with different performance constraints.

    @Robert

    Actually, I *have* put a lot of thought into this, as well as reading mountains of opinion and technical pieces regarding the language. I’ve never conversed with any of the core Groovy devs directly, but we have gone back and forth on forums a bit. My honest evaluation was that they just didn’t find performance to be a significant concern. Alexis’s comment illustrates the response that I have often received when voicing similar concerns. Given what I have seen, could I have come to any other conclusion?

    @Isaac

    Ah, I didn’t know that the tests had been run with 1.1. I’ll update the article. Thanks for the heads up!

    As to the name, it’s just more fun to type “great language shootout” than “computer language benchmarks game”. :-) Everyone knew what I meant, and clicking on the link reveals the true name in large, hard-to-miss styling; so I don’t really think I was confusing anyone.

    @Andres

    Your comment is a perfect example of why I really respect the Groovy community, despite the fact that I don’t care for their language. :-) I post a rant literally attacking you guys, and you respond with polite rationale. I’d like to see DHH do the same if I posted a rant about Rails performance…

    It’s good to hear that performance is getting more attention. It is also a fair point that the JRuby team has been working on this full time for nearly two years, whereas Groovy was just a hobby project until recently. It should be interesting to see what the next few releases bring!

    Daniel Spiewak Monday, March 31, 2008 at 10:09 am
  10. Every tool has its powers and shortcomings nor JRuby or Groovy perform like Java (not with the current JVM implementation) any time soon,will have to wait and see if the “Da Vinci machine project” will make a change.

    Ronen Monday, March 31, 2008 at 11:30 am
  11. > As to the name, it’s just more fun to type “great language shootout” than “computer language benchmarks game”

    Would it be fun if I started posting on reddit a variety of names for you instead of Daniel Spiewak?

    If we wanted to be called “great language shootout” that’s the name we would use.

    Isaac Gouy Monday, March 31, 2008 at 11:38 am
  12. Very well. I’ve updated the article to reflect the correct name.

    Daniel Spiewak Monday, March 31, 2008 at 11:58 am
  13. You do realize that Groovy is a DYNAMIC language, don’t you? And that, indeed, raw performance matters much less that productivity with the language, particularly when you can easily fall back on Java for bottlenecks?

    Peter Monday, March 31, 2008 at 12:23 pm
  14. @Peter

    Did you read the article? :-) I’m well aware that Groovy is dynamic, but that doesn’t necessarily mean it has to be slow, and certainly not as slow as it is. As to falling back to Java, I addressed that a few comments ago.

    Daniel Spiewak Monday, March 31, 2008 at 12:28 pm
  15. Postings like this together with the comments are helpful to embody the opinion of many users and for groovy devs to get aware and to clarify things

    Ramon Monday, March 31, 2008 at 1:10 pm
  16. > Very well. I’ve updated the article to reflect the correct name.

    Thank you.

    Isaac Gouy Monday, March 31, 2008 at 1:33 pm
  17. As other posters have pointed out, the focus of Groovy 1.6 is completely around performance. And in fact the roadmap all the way up to Groovy 2.0 is around performance as we hope to get it as fast as C Python (which by the way is still no where near as fast as a statically compiled language).

    As for comparing dropping from Groovy to Java to dropping from Java to C that is a bit of a silly comparison considering the world of hell you enter when doing the latter. The integration between Java and Groovy is quite simply seamless and unmatched by any other language on the JVM.

    Finally, just as a recommendation to readers of your posts. Daniel is a Scala advocate and supporter so take what he writes with a pinch of salt. It would be like me, as a core Groovy developer, writing a “Why scala sucks article” ie. completely unsubjective and not something I would ever bother doing.

    Graeme Rocher Tuesday, April 1, 2008 at 5:53 am
  18. Good post, thanks. Also great to hear that the Groovy devs are taking care of its performance.

    LudoA Tuesday, April 1, 2008 at 6:57 am
  19. Isn’t this just the same argument everyone had about Java years ago? Like the language or not give the guys a break. They’re doing the best they can with the resources they have.

    Jason Kratz Tuesday, April 1, 2008 at 7:41 am
  20. @Graeme

    Yes, I am quite a fan of Scala, but that doesn’t pose a conflict of interest at all. As I’m sure you would agree, Scala and Groovy are hardly peers in the language world. It would be like comparing Scala to C, it just doesn’t work. Now, I do have a *different* bias against Groovy in that I’ve been a big fan of JRuby for years, but that’s something which should be obvious to anyone who scrolls a little bit down the page (hint, look at my previous post).

    Dropping from Java to C is almost trivial now with frameworks like JNA. I would also hardly agree that C is “hell” as you put it, just far more complex and annoying than Java. For some things, it is actually a very nice language.

    To follow up on your actual point though, what I was *trying* to say with the comparison is that when you drop from Groovy to Java, you’re no longer in Groovy. If you’re not programming that algorithm in Groovy, then you can’t lump its performance in with your overall “Groovy evaluation”. When looking at Groovy’s performance, it *must* stand alone, otherwise every language on the JVM could simply claim its performance was equivalent to Java’s and call it a day.

    Daniel Spiewak Tuesday, April 1, 2008 at 7:42 am
  21. Whatever the case may be you’re not posting from a position of subjectivity, denying that is a bit silly.

    Having said that you are right, Groovy’s performance must stand on its own, but let us for a moment keep the goals of Groovy’s performance realistic here. When comparing Groovy’s performance it should be compared to Python/Ruby/JavaScript etc. as they are, as you say your self, its language peers.

    Scala and Java are not peers of the Groovy language and yet you make this comparison and publish the results without including any results from an equivalent language such as JRuby or Jython to make Groovy look worse than it actually is. If you’re going to argue that you should should only compare languages “peers” then your articles should reflect that which they do not.

    Graeme Rocher Tuesday, April 1, 2008 at 11:05 am
  22. http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=jruby&lang2=groovy

    :-) Try scrolling down. I was also careful to make the point that I don’t expect Groovy’s performance to rival Java’s, but rather to be better than it is. 830x slower than Java is just insane.

    Daniel Spiewak Tuesday, April 1, 2008 at 11:08 am
  23. Did you notice the difference in memory usage between JRuby and Groovy? Its huge. JRuby used almost 3 times as much. In the microbenches I’ve run, its always used 1.5-3x more even with programs just consisting of straight up arrays of huge lengths. In a production app, I wonder how much of that memory eats into CPU time via the GC?

    Anyways, no one trying to decide on Grails vs. Rails in a production app (the two biggest reasons to use JRuby or Groovy) is going to pay attention to an obviously biased rant like this. I know because I’m making that decision now, and the performance of Groovy 1.5.4 doesn’t matter to me (in fact, I could scarcely care about the languages at all). The slowness of some Grails tests is annoying, but that seems somewhat platform-dependent and so might not have anything to do with Groovy (they are fine on WinXP, but slow on OS X).

    What I am more interested in (although not the for the above mentioned app) is whether or not Groovy – or another JVM-based dynamic language – could integrate seamlessly with Scala?

    For the hell of it, I ran the binary-trees benchmark (on my Win XP box) and got:
    JRuby 1.0: 137.407
    Groovy 1.5.4: 125.781 seconds
    JRuby 1.1RC2: 137.078

    That seemed weird, since I was at acts_as_conference in Orlando where Charles Nutter showed us how much faster 1.1 was. so I downloaded JRuby 1.1RC3 and got: 60.787

    Wow. Good job JRuby team! Of course, Java is still 40 times faster, so if I’m going to be doing number-crunching, seamless Java integration is more important to me :)

    Grant Wednesday, April 2, 2008 at 4:11 am
  24. In light of how good Groovy’s Java integration is, people tend to forget that JRuby also has some really nice Java integration mechanisms. Calling from Ruby into Java is basically seamless (treat Java classes as Ruby classes, syntax sugar, etc). You can even add to Java classes dynamically as you would with any other Ruby “open class”. It’s going the other direction which is a little clunky (see http://www.codecommit.com/blog/ruby/jruby-interop-dsl-in-scala for one way to mitigate this).

    Both Groovy and JRuby should be able to integrate with Scala as easily as they do with Java. Scala’s integration with Java is unparalleled, which means that the languages effectively interoperate “through” Java (even though no Java code is involved). The only thing about Groovy-Scala interaction which would not be supported would be simultaneous compilation. So you couldn’t have circular dependencies between Groovy and Scala classes (as you can between Groovy and Java). Other than that though, life should be joyous.

    JRuby does use quite a bit more memory than Groovy, which was something I mentioned in my first draft of the article and cut because it disrupted the flow of the argument. This is due to the way JRuby’s JIT compiler works. Groovy has a lot of front-side overhead due to the fact that all of its compilation is ahead-of-time (albiet done in the same runtime cycle). If Groovy actually had a real JIT compiler, it’s memory usage would also climb quite high (whilst its exec time performance would improve quite a bit). Note that I haven’t looked at the Groovy architecture recently, so they may have changed the AOT compiler design into a true JIT already.

    Daniel Spiewak Wednesday, April 2, 2008 at 10:14 am
  25. Daniel, when we speak of Groovy’s integration with Java it means that you can call Groovy from Java without the need of a cumbersome API (JSR 223) as it is the case with other JVM languages like JRuby and JavaFX Script, almost *any* other JVM language will let you call Java in a ’seamless’ fashion. I believe a Scala class is also able to be called from Java as if it were a regular Java class, is it not? Groovy allows that, JRuby does not.

    A joint compiler for Groovy/Scala would be interesting, I guess it may happen when Scala gains more mind share.

    Andres Almiray Wednesday, April 2, 2008 at 11:28 am
  26. I suppose there really is two different sides to integration: Java -> Groovy and Groovy -> Java. Going from Java to JRuby is certainly a painful affair. Scala provides more options since it is possible to create a DSL to wrap around JSR 223, but it’s still a bit contrived.

    Daniel Spiewak Wednesday, April 2, 2008 at 11:40 am
  27. The memory usage patterns have nothing to do with the fact that JRuby has a JIT compiler. JRuby has a JIT compiler because its interpreted. Groovy is not interpreted, but instead compiled. Yes there is a small overhead in this happening.

    The main reason JRuby has greater memory overhead is because (and this is particularly apparent in JRuby on Rails) Ruby does not support native threading. This means that in JRuby an entire new JRuby runtime needs to be loaded per thread. Groovy used Java’s threading model and doesn’t need a new runtime per thread.

    As for Java integration, I’ve said it before and I’ll say it again. Integrating well with Java doesn’t mean that you can call a Java method, many languages allow that. Groovy integrates well with Java because, as well as the joint compiler, it shares the same object model, the same APIs, the same debugger, the same profiling and monitoring tools, the same call stack etc. etc. etc.

    It is funny how you omit these details to not “disrupt the flow”. As I said before your articles can in no way be interpreted as subjective and readers should take care.

    Graeme Rocher Wednesday, April 2, 2008 at 12:02 pm
  28. I’m not sure that threading can be considered as the root of the memory usage patterns seen in the benchmarks since most of them are single-threaded. I honestly don’t remember off the top of my head how many threads Ruby spins up for a “single-threaded” application, but I don’t think it’s more than two or three, most of them within the runtime for the purpose of utility tasks like garbage collection. JRuby, being based on the Java runtime, would obviously do things a bit differently than MRI, so it may very well use a fully single-threaded approach to effectively single-threaded applications.

    I just ran a simple test in JIRB by spawning a separate thread to print a value. There was literally no overhead beyond what I would normally expect from native thread creation. JRuby threads are in fact native and have been since (don’t quote me on this) 0.4 or 0.5. If it were spinning up an entirely new runtime per thread, there would be significant overhead in the following code:

    Thread.new do
    puts ‘test’
    end

    This code executes practically instantly on my machine, whereas the JRuby interpreter has a noticeable startup time. Obviously, you’re the language expert not I, but I’m simply not seeing evidence of this threading overhead you mention.

    BTW, I think the phrase you’re looking for is “not *objective*”. Subjective and objective are opposites, objective meaning “fully-reasoned, unbiased, etc”.

    I never claimed to be objective, and certainly not when writing an opinion piece like this. I linked to the benchmarks and didn’t go into any great detail because (as you said) it would have disrupted the literary flow. I usually make the assumption that people are capable of reading things on their own. :-)

    Regarding Java integration, I can certainly see the logic of your position. The problem is that by that standard, Groovy and Scala are the *only* languages which integration with Java, something which is obviously untrue. I believe it is a gradient scale (levels of integration). Groovy and Scala certainly have tighter integration with Java than any other JVM-based language, but that is not to say that other languages (like JRuby and Jython) have *no* integration. Your implication is such and it is more than a little misleading.

    Daniel Spiewak Wednesday, April 2, 2008 at 12:22 pm
  29. Graeme, the tests I’ve run indicate that JRuby (and Ruby, to a lesser degree) just uses more memory period, even with single threads.

    By the way, I ran these tests because blog posts like this made me think that Groovy is slower than untrained chimpanzees with abacuses, and so I began to question my use of Grails. Thankfully its not that slow. At best its as slow as Ruby 1.8.6, which was still adequate for most needs. Groovy 1.6 should be much better, and of course even 1.5.4 can easily make use of Java to increase performance.

    Grant Wednesday, April 2, 2008 at 2:47 pm
  30. I’m laughing. This is an April Fool’s joke, right? It has to be. Who the hell in their wildest dreams would want to do ray tracing in Groovy? For that matter, why would you even consider doing ray tracing in Java?

    Then your other examples, with the exception of the “simple counter” are also heavily math based. Silly silly man.

    But, I’ll tell you the same thing I tell anyone else that tries to complain about Groovy performance. If performance is your primary concern, Groovy is not the best (or even a good) choice. In many cases, neither is Java. If, however, you want to enjoy programming again, and want to improve YOUR performance, Groovy is the perfect choice.

    And, I’ll also fall back on the same old rebuttal; this is the same argument we heard from C programmers about Java 5 to 10 years ago, so wait 5 to 10 years to start comparing Groovy’s performance to Java.

    Joe Kueser Friday, April 4, 2008 at 12:01 am
  31. If you read the post a little more closely, you’ll see that my primary argument was not complaining about Groovy performance, it was complaining about the very same rebuttal you gave. You can’t just throw away performance as a useless benchmark when it feels convenient.

    Oh, and like it or not, mathematical operations are at the core of any application, regardless of what it does. Yes, some are more intensive than others, but in *all* cases the math operations must be reasonably fast. I don’t know about 1.6, but in 1.5 Groovy is shamefully lax in its numeric optimizations.

    Daniel Spiewak Friday, April 4, 2008 at 8:33 am
  32. I’m a big fan of Groovy, but the top ten features I want out of the next few releases of Groovy are: performance, performance, performance, performance, performance,performance,performance,performance, performance, and performance.

    I work in bioinformatics where Perl is king. Perl works well but I hate Perl. Every night I pray to the language gods that I will never have to write another line of Perl. Since I’m fairly fond of Java Groovy looked like the answer to my prayer. It really is the ideal language for my purposes except…. it’s dog slow. I could, and still might, learn Python or Ruby as a replacement for Perl, but now that I’ve seen Groovy I really don’t want to. I want Groovy to work for me. I want the Groovish syntax, and I want to be able to seamlessly leverage all the Java libraries out there, including my own Java code base. All of the dynamic scripting languages are, of course, slow compared to C or even Java, and that’s fine. But for me, the standard for an acceptable scripting language is Perl. I can use a language that is 2x or maybe even 3x slower than Perl on tasks I commonly perform, but I can’t use a language that is 10x or 20x slower than Perl. The data I operate on varies from a few mega-bytes to a few gigabytes. It’s a tremendous difference in the amount of work I can get done if it takes 10 minutes instead of 30 seconds to process one data file.

    Speed is not just about the actual time it takes to do something, though. It’s also sociological. It is hard enough to convince colleagues, who are used to running Perl programs or C programs, to even accept something written in an obscure language with a cute name like Groovy. It becomes impossible if the script you give them also runs obviously, and glaringly, slow. If you try to remedy this by sending along some Java classes to do the heavy lifting, then you enter packaging hell. I really want to hand people a single file to solve their problem. Or rather, that’s what my customers, other scientists, demand of me.

    It is, of course, one of the best features of Groovy that it’s not just easy, but totally transparent, to mix in Java classes, and in practice I do that a lot. But very quickly it ceases to be a Groovy program in any meaningful sense. I start to feel silly even using Groovy when all my Groovy script does is create a series of Java objects and invoke Java methods on those objects.

    I find it hopeful that the performance of Groovy seems to vary wildly between different tasks. That makes me think it has some quirky bottlenecks that can be ironed out. For example, on the computer language shootout… er… benchmarks game [1], I was able to write the regex-dna benchmark so that it took 15 seconds, which compares well enough for me with Python’s 6 seconds, or Ruby’s 7 seconds. However, I was unable to come up with a Groovy implementation of the reverse-complement benchmark that was anywhere in the ball park of being competitive (the one someone else submitted is 9000x slower than Perl !! I can beat that, but all my efforts have been too poor still to bother submitting). These tasks, regex-dna, reverse-complement, k-nucleotide, are not just arbitrary benchmarks to me either. These represent very good approximations of the bread and butter of my daily work.

    I’m not commenting to rag on the Groovy developers. I’m a big Groovy fan. I want to make Groovy the next bioinformatics language of choice. I am, moreover, very impressed with what the Groovy team has done so far and rather quickly. I just want to add my voice to the chorus saying, “Performance matters, it matters a lot, and even us fanatics can’t ignore it”. I want to do whatever I can to put performance on the front burner of their future efforts.

    [1] http://shootout.alioth.debian.org/gp4sandbox/

    James Sunday, April 6, 2008 at 10:51 pm
  33. The fact is that Groovy is simply not as slow as you are making it out to be. For starters, almost all of the benchmarks I’ve seen are done without the ‘-server’ tag, which makes it run much faster. Also, if you look, you’ll notice those original benchmarks were done in Groovy 1.0.

    For instance,
    “groovy -server ray 8 64″ takes 43.859s (About 100 times slower than Java on my system)
    “groovy ray 8 64″ takes 115.735s

    In the binartrees test,
    “groovy -server binarytrees 16″ takes 41.047s
    “groovy binarytrees 16″ takes 126.000s
    Ruby 1.9: “ruby binarytrees.rb 16″ 55.891s
    JRuby 1.1: “jruby binarytrees.rb 16″ takes 60.184s
    JRuby 1.1: “jruby -J-server binarytrees.rb 16″ takes 33.047s

    I’ve got a zillion benchmarks on my system, because certain bloggers had me convinced it was unusably slow, and so wasn’t suitable for production :P Fortunately I bothered to run some tests on my own, and found out otherwise.

    From the looks of things, Groovy 1.6 just may beat out JRuby 1.1 as the fastest dynamic language which integrates easily with Java.

    Grant Monday, April 7, 2008 at 9:29 am
  34. James, my last comment was in response to Daniel, not you. You’ll probably be happy to hear that Groovy 1.6 beta is about twice as fast as 1.5.4 on my system, putting it ahead of the Ruby implementations (JRuby 1.1 and Ruby 1.9) that I’ve tried on my PC. I don’t know about Python, though.

    Also, if you prefer just to give folks one file, why not make it a .jar with combined Java and Groovy?

    Grant Monday, April 7, 2008 at 11:52 am
  35. Grant,

    Why not make it a jar file? Well, that should be a perfectly reasonable option, but alas the world isn’t perfectly reasonable. My colleagues and clients and, importantly, bosses, are often biologists who know some Perl but not much more. For reasons that are obscure to me, Java has gotten a bad reputation among this group. The biologists who write the checks just want to get a paper out about the genetic causes of epilepsy or something, and so they don’t care about the software at all, per se, just the final result. I’m just guessing, but I expect that Java got a bad rap among this group because Java programmers tended to over engineer their assignments. Whatever the case, I’m always fighting an uphill battle to use non-Perl or non-C tools. So, as much as possible, I’d prefer not to draw attention to the fact that I’m using Java based technology. I’d really rather not remind them of the fact every time they run James’ software. “Oh, great, it’s one of Jame’s jar files… how do I run that again?” I’d like to be able to give them a file that they just run like they are used to, like a Perl, Python, Ruby, or even C program. You want something to find splice sites? Sure, here it is:

    findSpliceSites input.fa splice.out

    It’s all a bit silly, I know. I mean, I REALLY know. I have to live with Ph.D’s who act as though remembering to do “java -jar” to run a jar file will overtax their abilities. It’s a bit absurd and very frustrating, but no less real for that. I strongly suspect that I’m not the only person who works in an environment where Java’s clunky command line packaging is a liability. One of my hopes for Groovy was to smooth over this prejudice, to write little Groovy-only scripts for a lot of the little programs I have to generate every day. That’ll only work, though, if the Groovy script isn’t grossly slower than they are expecting.

    James Monday, April 7, 2008 at 1:08 pm
  36. To be honest, James, Groovy sounds perfect for what you’re doing. As Grant has pointed out, performance is greatly improved in recent releases. More than that, the integration with Java is seamless, so it should fit right in with your code.

    Since you already have to package your Java application to look like a normal command line app, Groovy really isn’t going to add to that overhead. I assume you’re either using a wrapper script or cross-compilation (using GCJ or similar). With the wrapper, you can just add Groovy to whatever you’re already doing for packaging and things should work perfectly. If you’re cross-compiling, things get a little hairier, but not too much so. The Groovy compiler produces bytecode, so GCJ should be able to consume that as easily as it does any regular Java artifact.

    Daniel Spiewak Monday, April 7, 2008 at 1:12 pm
  37. Grant,

    I guess I can do that by continuing to drop Java .jar files into the groovy lib directory for my internal customers, and package up a .jar for external customers. So maybe I’m fretting overmuch about a small thing. I’m just hyper sensitive to the anti-Java skepticism of my colleagues, so I’m always trying to minimize the perceived difference in usability.

    James

    James Monday, April 7, 2008 at 1:13 pm
  38. > As Grant has pointed out, performance is greatly improved in recent releases.

    “The Groovy 1.5.4 stable release is the latest release. No milestone releases for the next version of Groovy have yet been released.” http://groovy.codehaus.org/Download

    The ‘-server’ tag /is used/ for the benchmarks game measurements.

    Isaac Gouy Friday, April 11, 2008 at 12:39 pm
  39. This Groovy performance discussion helped to me. I’m a Java programmer, but looking for simpler options, such as J/Ruby, PHP, and Groovy etc. for prototyping and personal projects. I hope someday my projects become popular enough so that I’ll have to worry about performance and scalability issues:-)

    Despite comments from several posters, that performance isn’t important, it is. Everyone likes a snappy application. Lack of snappiness is one reason Java failed to become anyone’s GUI of choice. Sun’s server-side corporate group-think never grasped this.

    Think of the top 10 to 20 most popular GUI applications used by non developers. After 10+ years of Java, only a small fraction of them are written in Java. Groovy will fail too if it isn’t snappy and competitive in performance.

    Twice before I’ve tried to get into Groovy as a command-line scripting language alternative to bash, Ruby, and Python. Each time Groovy’s sluggish performance made me stop using it. As of now, it can’t compete on the command-line with its peers.

    A slow performing language, compared to it’s other dynamic language peers, will in my opinion, remain a toy, hobby only language. That’s not the kind of language I want to invest my time learning.

    So I’m pleased to hear that Groovy developers do value performance and are working on this issue. (Thank you Groovy devs! )

    Now I’d like to comment on the emotional undertone of some comments above. I feel that Daniel Spiewak has done Groovy a service with his blog about its performance. To me Spiewak doesn’t come off as an overly biased proponent of another language out to slam Groovy. Yet, that is the feeling I get when reading Graeme Rocher’s comments to Spiewak.

    Attacking Groovy critics is unwise and will drive away people considering the language. Nobody wants to be part of a community of mean spirited people. It’s one reason for Microsoft’s lack of popularity with many developers.

    Groovy is in a difficult competition for developer hearts and minds with J/Ruby, J/Python, PHP, and Scala (and more language choices appear all the time). I suggest that if Rocher has a hard time welcoming and engaging Groovy critics in a positive way, especially thoughtful ones like Spiewak, then he should allow other Groovy community members, with better PR and relating skills, to do it instead.

    devdanke Thursday, April 17, 2008 at 2:50 am
  40. I buried the lead… this comment is about mixed Java Groovy runtime compiling from a single file… too lazy to reformat so it starts with lead.

    Groovy vs…

    I have been a bit surprised at how often Groovy performance is compared to Java. That’s just not a comparison that ever occurs to me. I wouldn’t expect Groovy to compete with Java itself, that seems silly. However, I would like it to compete, not win just compete, with other “scripting” options out there. Almost exclusively what I care about is how Groovy compares to Perl, Python, and Ruby (and not JRuby, etc.), because those are the languages my colleagues use, and those are the things they are going to compare Groovy to. If I hand them a Groovy script, and they run it, and it’s obviously abysmal, I loose credibility, which is very expensive in my profession. “James is that flake who uses that dog slow Groovy”. A lot of people sound like they don’t have to deal with these kinds of pressures, but really it’s exactly this that makes me deeply care about Groovy performance. I have to be able to give Groovy scripts to my peers with a straight face.

    I just mention this because for me Groovy primarily serves the same role as Perl, Python, and Ruby. I prefer it over these others because I write a lot of Java code too, and I like to be able to tap into that now and then in the very transparent way that Groovy lets me, as well as the rich set of Java libraries out there. But by and large, when I’m writing a “script” I want to stay in “script world” and not jump out into the world of compiled applications. So it’s important to me that my “scripting language” be able to accomplish a fair amount of work on it’s own.

    To this end, I have seen on the developer’s mailing list archive [1] a nice suggestion. The idea is to allow Java to be mixed into Groovy in a source file and compiled by the runtime compiler, so that you can run a mixed Java/Groovy text file seamlessly as a single script. That would be great for me. Then I can stay in the world of “open a text file, bang out a little program, hand it to someone” that makes up the “scripting” half of my life, but be able to tap into Java without having to completely drop out of “single file script world” and into the world of compiling and packaging jar files, etc.

    Perhaps all my whining about not wanting to compile things sounds like… well, whining. But Groovy is about convenience for me. It’s useful to me precisely because I can hack out little fifty line programs in a single file and then just run the thing. The second that becomes two twenty five line files, with products I have to package, Groovy looses more than half it’s utility to me.

    Anyway, I just wanted to mention it here in the slight hope that some momentum will build behind the mixed-Java Groovy runtime compiling. I believe it would be fairly easy to implement and it would go a very long way towards answering my own issues with Groovy performance.

    [1]

    http://archive.groovy.codehaus.org/dev/197b18fc0802210742y4b943e6budcaa64af3791f899%40mail.gmail.com

    James Friday, April 25, 2008 at 12:32 am
  41. can anyone tell me about the readability of this language(groovy!)? Is it easy to understand?
    and as well as its writtability and reliability! hope there’s someone who can answer my question.
    tnx alot.

    kim Saturday, July 12, 2008 at 12:51 am

Post a Comment

Comments are automatically formatted. Markup are either stripped or will cause large blocks of text to be eaten, depending on the phase of the moon. Code snippets should be wrapped in <pre>...</pre> tags. Indentation within pre tags will be preserved, and most instances of "<" and ">" will work without a problem.

Please note that first-time commenters are moderated, so don't panic if your comment doesn't appear immediately.

*
*