Skip to content
Print

Joint Compilation of Scala and Java Sources

5
Jan
2009

One of the features that the Groovy people like to flaunt is the joint compilation of .groovy and .java files.  This is a fantastically powerful concept which (among other things) allows for circular dependencies between Java, Groovy and back again.  Thus, you can have a Groovy class which extends a Java class which in turn extends another Groovy class.

All this is old news, but what you may not know is the fact that Scala is capable of the same thing.  The Scala/Java joint compilation mode is new in Scala 2.7.2, but despite the fact that this release has been out for more than two months, there is still a remarkable lack of tutorials and documentation regarding its usage.  Hence, this post…

Concepts

For starters, you need to know a little bit about how joint compilation works, both in Groovy and in Scala.  Our motivating example will be the following stimulating snippet:

// foo.scala
class Foo
 
class Baz extends Bar

…and the Java class:

// Bar.java
public class Bar extends Foo {}

If we try to compile foo.scala before Bar.java, the Scala compiler will issue a type error complaining that class Bar does not exist.  Similarly, if we attempt the to compile Bar.java first, the Java compiler will whine about the lack of a Foo class.  Now, there is actually a way to resolve this particular case (by splitting foo.scala into two separate files), but it’s easy to imagine other examples where the circular dependency is impossible to linearize.  For the sake of example, let’s just assume that this circular dependency is a problem and cannot be handled piece-meal.

In order for this to work, either the Scala compiler will need to know about class Bar before its compilation, or vice versa.  This implies that one of the compilers will need to be able to analyze sources which target the other.  Since Scala is the language in question, it only makes sense that it be the accommodating one (rather than javac).

What scalac has to do is literally parse and analyze all of the Scala sources it is given in addition to any Java sources which may also be supplied.  It doesn’t need to be a full fledged Java compiler, but it does have to know enough about the Java language to be able to produce an annotated structural AST for any Java source file.  Once this AST is available, circular dependencies may be handled in exactly the same way as circular dependencies internal to Scala sources (because all Scala and all Java classes are available simultaneously to the compiler).

Once the analysis phase of scalac has blessed the Scala AST, all of the Java nodes may be discarded.  At this point, circular dependencies have been resolved and all type errors have been handled.  Thus, there is no need to carry around useless class information.  Once scalac is done, both the Foo and the Baz classes will have produced resultant Foo.class and Baz.class output files.

However, we’re still not quite done yet.  Compilation has successfully completed, but if we try to run the application, we will receive a NoClassDefFoundError due to the fact that the Bar class has not actually been compiled.  Remember, scalac only analyzed it for the sake of the type checker, no actual bytecode was produced.  Bar may even suffer from a compile error of some sort, as long as this error is within the method definitions, scalac isn’t going to catch it.

The final step is to invoke javac against the .java source files (the same ones we passed to scalac) adding scalac’s output directory to javac’s classpath.  Thus, javac will be able to find the Foo class that we just compiled so as to successfully (hopefully) compile the Bar class.  If all goes well, the final result will be three separate files: Foo.class, Bar.class and Baz.class.

Usage

Although the concepts are identical, Scala’s joint compilation works slightly differently from Groovy’s from a usage standpoint.  More specifically: scalac does not automatically invoke javac on the specified .java sources.  This means that you can perform “joint compilation” using scalac, but without invoking javac you will only receive the compiled Scala classes, the Java classes will be ignored (except by the type checker).  This design has some nice benefits, but it does mean that we usually need at least one extra command in our compilation process.

All of the following usage examples assume that you have defined the earlier example in the following hierarchy:

  • src
    • main
      • java
        • Bar.java
      • scala
        • foo.scala
  • target
    • classes

Command Line

# include both .scala AND .java files
scalac -d target/classes src/main/scala/*.scala src/main/java/*.java

javac -d target/classes \
      -classpath $SCALA_HOME/lib/scala-library.jar:target/classes \
       src/main/java/*.java

Ant

<target name="build">
    <scalac srcdir="src/main" destdir="target/classes">
        <include name="scala/**/*.scala"/>
        <include name="scala/**/*.java"/>
    </scalac>
 
    <javac srcdir="src/main/java" destdir="${scala.library}:target/classes" 
           classpath="target/classes"/>
</target>

Maven

One thing you gotta love about Maven: it’s fairly low on configuration for certain common tasks.  Given the above directory structure and the most recent version of the maven-scala-plugin, the following command should be sufficient for joint compilation:

mvn compile

Unfortunately, there have been some problems reported with the default configuration and complex inter-dependencies between Scala and Java (and back again).  I’m not a Maven…maven, so I can’t help too much, but as I understand things, this POM fragment seems to work well:

<plugin>
    <groupId>org.scala-tools</groupId>
    <artifactId>maven-scala-plugin</artifactId>
 
    <executions>
        <execution>
            <id>compile</id>
            <goals>
            <goal>compile</goal>
            </goals>
            <phase>compile</phase>
        </execution>
 
        <execution>
            <id>test-compile</id>
            <goals>
            <goal>testCompile</goal>
            </goals>
            <phase>test-compile</phase>
        </execution>
 
        <execution>
            <phase>process-resources</phase>
            <goals>
            <goal>compile</goal>
            </goals>
        </execution>
    </executions>
</plugin>
 
<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.5</source>
        <target>1.5</target>
    </configuration>
</plugin>

You can find more information on the mailing-list thread.

Buildr

Joint compilation for mixed Scala / Java projects has been a long-standing request of mine in Buildr’s JIRA.  However, because it’s not a high priority issue, the developers were never able to address it themselves.  Of course, that doesn’t stop the rest of us from pitching in!

I had a little free time yesterday afternoon, so I decided to blow it by hacking out a quick implementation of joint Scala compilation in Buildr, based on its pre-existing support for joint compilation in Groovy projects.  All of my work is available in my Buildr fork on GitHub.  This also includes some other unfinished goodies, so if you want only the joint compilation, clone just the scala-joint-compilation branch.

Once you have Buildr’s full sources, cd into the directory and enter the following command:

rake setup install

You may need to gem install a few packages.  Further, the exact steps required may be slightly different on different platforms.  You can find more details on Buildr’s project page.

With this highly-unstable version of Buildr installed on your unsuspecting system, you should now be able to make the following addition to your buildfile (assuming the directory structure given earlier):

require 'buildr/scala'
 
# rest of the file...

Just like Buildr’s joint compilation for Groovy, you must explicitly require the language, otherwise important things will break.  With this slight modification, you should be able to build your project as per normal:

buildr

This support is so bleeding-edge, I don’t even think that it’s safe to call it “pre-alpha”.  If you run into any problems, feel free to shoot me an email or comment on the issue.

Conclusion

Joint compilation of Java and Scala sources is a profound addition to the Scala feature list, making it significantly easier to use Scala alongside Java in pre-existing or future projects.  With this support, it is finally possible to use Scala as a truly drop-in replacement for Java without modifying the existing infrastructure beyond the CLASSPATH.  Hopefully this article has served to bring slightly more exposure to this feature, as well as provide some much-needed documentation on its use.

Comments

  1. Groovy joint compiler doesn’t do type checking – and it sucks compared to Scala.

    Alexey Lapusta Monday, January 5, 2009 at 5:45 am
  2. Daniel:

    This is great… the fact that Scala now will allow compiling cross-dependencies with pure Java code and also the fact that you provided everyone with a handy writeup and cheatsheet.

    What bothers me is that in my mind this solves the wrong problem. The REAL objective is not allowing compilation of projects containing both Scala and Java code, but compilation of projects containing mixed code of various languages that interoperate on the JVM. Jython, Groovy, Scala, and Java should all be able to co-exist. They can effectively all use Java as an inter-language interface, and they can all compile today so long as there are no circular dependencies. But introduce circular dependencies and you are now restricted to Groovy-and-Java, or Scala-and-Java, or basically any combination of languages so long as the “youngest” compiler is aware of all the languages.

    Or you can try playing tricks with multiple compilation passes — compile certain sources in one language, then other sources in another, then back to the first language. I can say from experience that this approach gets unmanageable VERY rapidly.

    What IS the ideal solution? Perhaps a standard facility that compilers like Groovy, Scala, Jythonc, and others can implement which would allow them to be invoked just to produce the AST (perhaps in the form of a stub .class file). Does anyone reading this know of any attempts to produce such a thing?

    Michael Chermside Monday, January 5, 2009 at 6:05 am
  3. Very useful article. Thanks !

    jau Monday, January 5, 2009 at 7:08 am
  4. A unified joint compiler would be a nice thing, but based on my reading of how to use Scala’s joint compilation, the do it quite different from Groovy.

    When Groovy does a joint compile it is really a three step process: Generate Groovy Stubs, call JavaC on the .java files with the stubs as non-compiled source, compile Groovy with the JavaC Results in the classpath. The clever part from the IntelliJ guys was the stub generation. A signature equivalent java file is generated from the Groovy source and made available to JavaC Since the transform is fairly simple and does not vary based on the actual libraries used no external code needs to be referenced, it’s just a text transformation on the AST. It looks to me Scala is parsing the Java source and just creating data in it’s compiler tables to handle the symbols and verification for the Scala code.

    However, if Scala, Jython, JRuby, JavaFX Script, etc. had stub generation code then a grand unified compiler could be written, but it gets messy at n=3 if Groovy calls JRuby for example, the stubs would need to be compiled and segregated so that they don’t get exposed to the Groovy code. It would be a really neat hack and would help the case for JVM polygot programming a lot. But, alas, not enough spare time. :(

    Danno Ferrin Monday, January 5, 2009 at 10:18 am
  5. I would be thrilled to see how the Scala Eclipse plugin ppl would approach this. I am having the same issue using Groovy’s joint compiler in Groovy Eclipse.

    James E. Ervin Tuesday, January 6, 2009 at 5:53 pm
  6. Actually, joint compilation in the Scala Eclipse plugin works out of the box. Try it! :-)

    Daniel Spiewak Tuesday, January 6, 2009 at 6:29 pm
  7. BTW, Scala plugin for IntelliJ provides automatic ant script generation by project taking into account these peculiarities of Scala/Java joint compilation. :)

    Ilya Sergey Sunday, January 11, 2009 at 1:35 pm
  8. @Ilya

    Personally, I think that Buildr is really the only way to go for Scala projects. It’s faster than Maven, far easier to configure, and requires less manual tweaking than Ant. It especially shines when it comes time to include a dependency that isn’t in a Maven repo. Its only problem is the fact that it doesn’t have Scala/Java joint compilation in the mainline trunk/ (working on that).

    Daniel Spiewak Sunday, January 11, 2009 at 2:02 pm
  9. As an alternative to maven-scala-plugin, you could also use antrun, which I’m starting to think of as Maven’s escape hatch for tricky problems. In this situation, if you have Java files that depend on Scala classes, you can prevent compiler:compile from trying to compile them by putting them in src/main/scala instead of src/main/java. You can then build them as you’ve suggested with antrun. Here’s a POM snippet that worked for me:

          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-antrun-plugin</artifactId>
            <version>1.3</version>
            <executions>
              <execution>
                <id>scala-compile</id>
                <phase>compile</phase>
                <configuration>
                  <tasks>
                    <property name="build.compiler" value="extJavac"/>
                    <taskdef resource="scala/tools/ant/antlib.xml" classpathref="maven.compile.classpath"/>
                    <mkdir dir="${project.build.outputDirectory}"/>
                    <scalac srcdir="src/main" destdir="${project.build.outputDirectory}" classpathref="maven.compile.classpath">
                      <include name="scala/**/*.scala"/>
                      <include name="scala/**/*.java"/>
                    </scalac>
                    <javac srcdir="src/main" destdir="${project.build.outputDirectory}" classpathref="maven.compile.classpath">
                      <include name="scala/**/*.java"/>
                    </javac>
                  </tasks>
                </configuration>
                <goals>
                  <goal>run</goal>
                </goals>
              </execution>
              <execution>
                <id>scala-test-compile</id>
                <phase>test-compile</phase>
                <configuration>
                  <tasks>
                    <property name="build.compiler" value="extJavac"/>
                    <taskdef resource="scala/tools/ant/antlib.xml" classpathref="maven.compile.classpath"/>
                    <mkdir dir="${project.build.testOutputDirectory}"/>
                    <scalac srcdir="src/test" destdir="${project.build.testOutputDirectory}" classpathref="maven.test.classpath">
                      <include name="scala/**/*.scala"/>
                      <include name="scala/**/*.java"/>
                    </scalac>
                    <javac srcdir="src/test" destdir="${project.build.testOutputDirectory}" classpathref="maven.test.classpath">
                      <include name="scala/**/*.java"/>
                    </javac>
                  </tasks>
                </configuration>
                <goals>
                  <goal>run</goal>
                </goals>
              </execution>
            </executions>
            <dependencies>
              <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-compiler</artifactId>
                <version>2.7.5</version>
              </dependency>
            </dependencies>
          </plugin>
    
    Dan Blick Friday, July 24, 2009 at 1:40 pm
  10. Holy mind-numbing XML, Batman! I think I’m going to stick with Buildr, considering the fact that something like this (manual control over the build process) is almost trivial.

    Daniel Spiewak Friday, July 24, 2009 at 1:47 pm
  11. What about joint compilation of scaladoc and javadoc?

    Dan Monday, July 26, 2010 at 7:58 am
  12. This is great; it is just what I am trying to do. I think there might be a mistake in your ant example, though. You have:

    Are the destdir and classpath attributes reversed? You are trying to add the output from the scalac command (i.e. the resulting class files) to the build classpath for the java compilation, right?

    Patrick Kaeding Wednesday, March 23, 2011 at 10:41 pm
  13. I think you have one small typo: Your Ant snippet runs against scala/**/*.java but I think you meant java/**/*.java.

    thom Wednesday, July 27, 2011 at 10:05 am

Post a Comment

Comments are automatically formatted. Markup are either stripped or will cause large blocks of text to be eaten, depending on the phase of the moon. Code snippets should be wrapped in <pre>...</pre> tags. Indentation within pre tags will be preserved, and most instances of "<" and ">" will work without a problem.

Please note that first-time commenters are moderated, so don't panic if your comment doesn't appear immediately.

*
*