Happy Easter everyone, we’ve got a new release!  ActiveObjects 0.8 is simmering at low heat (and footprint) on the servers even as I type.  This is probably the most significant milestone we’ve released thus far in that the 1.0 stream is now basically feature-complete.  I won’t be adding any new features to this release, just polishing and testing the heck out of the old ones.  We’ve already got a suite of 91 tests and I’ve got lots of ideas for more.  Expect this framework to get very, very stable in the next few months.

Features

Despite an annoyingly cramped schedule in the last couple months, I have managed to implement a few long-requested features.  The big ticket item for this release is pluggable cache implementations.  For those of you not “in the know”, ActiveObjects has a multi-tiered cache system composed of several layers and delegates.  This isn’t entirely unusual for an ORM, which is why I hadn’t mentioned it before.  The three basic layers:

• Entity Cache - A SoftReference map from id/type pairs to actual entity values.  This means that if you run a query which returns an entity for people {id=1} and then run a second query which returns an entity for the same row, the two instances will be identical.  This is what enables the additional cache layers to function properly (and it saves on memory).
• Value Cache - This is where all the field values for a given row are stored.  This will store both field values from the database as well as entity values (corresponding to foreign key fields).  Each entity has a separate instance of this cache.  Most of the memory required by ActiveObjects is taken up here.  With a long-running application, quite a bit of the database can actually get paged into memory.  This is a really good thing, since it drastically reduces round-trips to the database.  Thanks to the SoftReference(s) in the entity cache, running out of memory due to stale cache entries isn’t a problem.
• Relations Cache - This is probably the most complex of all of the cache layers.  This cache literally caches any one-to-one, one-to-many or many-to-many result sets and spits them out on demand.  This reduces round-trips still more to the point where certain applications can be running completely from memory, even querying on complex relations.  This in and of itself isn’t that complex; the hard part is knowing when to expire specific cache entries.  This cache layer is essentially three different indexes to the same data, allowing cache expiry for a number of different circumstances.  Naturally, these structures don’t represent well as a single map of key / value pairs.  (more on this in a bit)

As of the 0.8 release, the value cache and relations cache layers have been unified under a single controller.  They’re still separate caches, but this unification allows for something I’ve been wanting to implement since 0.2: Memcached support.

High-volume applications often run into the problem of limited memory across the various server nodes.  Solutions like Terracotta can help tremendously, but unfortunately this doesn’t solve everything.  One answer is to provide every node in the cluster with a shared, distributed, in-memory map of key / value pairs.  This allows the application to page data into cluster memory and retrieve it extremely quickly.  Memcached is effectively this.  Naturally, it would be very useful if an ORM could automatically make use of just such a distributed memory cache and thus share cached rows between nodes.  Hibernate has had this for a long time, and now ActiveObjects does as well.

By making use of the MemcachedCache class in the activeobjects-memcached module (separate from the main distribution), it is possible to point ActiveObjects at an existing Memcached cluster and cause it to store the value cache there, rather than in local memory.  Once the relevant call has been made to EntityManager (setting up the cache), all cached rows will be stored in Memcached and shared between every instance of ActiveObjects running in your cluster.

Caveats

Of course, nothing is perfect.  For the moment, the major issues is that the relations cache doesn’t work properly against Memcached.  I had originally planned to just release it running against the local RAM, but this causes issue with proper cache expiry.  As such, the relations cache is disabled when running ActiveObjects against Memcached.  This doesn’t really impair functionality other than forcing ActiveObjects to hit the database every time a relationship is queried.  Since this is what most ORMs do anyway, it’s not too horrible.

I plan to have this issue resolved in time for the 0.9 release.  The biggest problem is figuring out how to flatten the multi-index structure into a single map.  Once I can do that, rewriting things for Memcached should be a snap.  (btw, if anyone wants to help out with patches in this area, you’re more than welcome!)

Databases Galore

One of the major focuses of this release was testing on different platforms.  This process uncovered an embarrassing number of bugs and glitches in some of the supposedly “supported” databases.  With the exception of Oracle, every bug I’ve been able to identify has been fixed.  So if you’ve tried ActiveObjects in the past and run into problems on non-MySQL platforms, now would be the time to give it another shot!

Speaking of Oracle, we’ve made some tremendous progress toward full support of this ornery database.  To be fair, I’m not the one responsible.  One of our community members has been tirelessly submitting patches and running tests.  We didn’t have time to get things fully sorted for this release (so don’t try AO 0.8 with Oracle unless you’re willing to risk server meltdown), but you can expect dramatic improvements in 0.9.  Once these fixes are in place, we should be able to safely claim that we support most of the database market (in terms of product adoption).

Final Thoughts

This really is an amazing release, well worth the download.  Most of the core API has remained the same, so things should be basically plug-n-play for existing applications.  The user-facing API is now in freeze and (unless some serious bug arises) will experience no changes between now and 1.0.  If you’ve been considering trying ActiveObjects for your project, now would be an excellent time.  I can’t promise no bugs, but if you point out my idiocy, I’ll certainly work to fix it!

Oh, as an aside, ActiveObjects is now in a Maven2 repository.  (thanks to some POM reworking by Nathan Hamblen)  To make use of this somewhat dubious feature, add the java.net Maven2 repository to your POM and then insert the following dependency:

<dependency>
    <groupId>net.java.dev.activeobjects</groupId>
    <artifactId>activeobjects</artifactId>
    <version>0.8</version>
</dependency>

After that, the magic of Maven will take over.  Of course, you’ll need to add the dependency for the appropriate database driver and (optionally) connection pool.  Those dependencies are left as an exercise to the reader.  Note, activeobjects-memcached isn’t in Maven yet, but it will be for the 0.9 release (either that or integrated into the core, I haven’t decided yet).

I certainly hope you enjoy this latest release.  As always, feel free to drop by the users list if you have any questions or (more likely) uncover any problems.

• activeobjects-0.8.tar.gz
• activeobjects-0.8.zip

Just in time for the Christmas holidays, I give you ActiveObjects version 0.7!  This release sports a whole slew of minor performance tweaks and bugfixes, providing better reliability.  Also, we’ve greatly increased the number and scope of the unit tests running against the core.  This gives us more confidence that the code is indeed as stable as we hope.

Actually, the big ticket item for 0.7 is the introduction of polymorphic relations.  This feature is fully tested and stable, so I don’t anticipate it introducing any issues in the coming days as we narrow our focus to 1.0.  However, no feature is so well tested as to be certainly bug free under all circumstances.  I look forward to fixing any problems you may encounter with the library!

A few minor feature additions:

• Support for persisting enum values
• One-to-one relations
• EntityManager#flush(…) now also flushes the relations cache

Looking toward the road ahead, things are really starting to settle down a bit.  The main feature I’m looking to implement for 0.8 is extensible value caching, which should allow you to determine how and where the cached field values are stored.  The main use-case behind this is support for memcached, which has been requested a few times and has actually been on the agenda since 0.3.  With memcached driving the value cache, ActiveObjects should be almost perfectly scalable.

To that end, it’s also worth mentioning that I did some profiling (using the NetBeans Profiler) using a Wicket-based web application I’m developing on the side.  I’m happy to report that the overhead imposed by ActiveObjects is extremely minimal.  In each test, the bottleneck was in the PreparedStatement#execute method (in JDBC).  Since ActiveObjects generates very streamlined and natural queries, it seems that this is really the minimal execution time possible for such database access.  Extrapolating further from this data, and given that the JDBC exec time was avg 40ms, while the execution time in the ActiveObjects API was on the order of 1ms-5ms, we can safely say that ActiveObjects is less than 10% slower than using JDBC directly (compare that to the 50% overhead claimed by ActiveRecord).  This “estimate” doesn’t even take into account the many performance advantages offered by ORMs over hand-coded database access (such as relations caching, optimal query construction, etc).

Anyway, enough shameless boasting.  As Hibernate has often pointed out, ORM benchmarks are usually worse than useless.  The only benchmark which really matters is how well the ORM performs in your application.  Try it out, see what you think.

• activeobjects-0.7.tar.gz
• activeobjects-0.7.zip

Disclaimer: For those of you who don’t know, my pet project is an ORM based on the active record pattern (ActiveObjects).  As such, my opinions are probably colored somewhat.  Take everything I say here with a grain of salt.

There seems to have been a recent resurgence of interest in Java implementations of the active record pattern in recent months.  Most traditional Java ORMs (like Hibernate/JPA) implement the data mapper pattern, as it is far easier to apply to a static language like Java.  Yet I can count a number of new (at least to me) ORM projects which attempt to fill this void.  One such project is jPersist.  From the project page:

jPersist is an extremely powerful object-relational persistence API that is based on the Active-Record pattern. jPersist is mapless and has no need for XML or annotation based mapping (all mapping is automatic and dynamic).

Sounds promising!  I’m always in favor of any framework which can reduce the boiler-plate configuration required, especially when the configuration would have been in a language as verbose as XML.  But irregardless of the method for configuration, I’m always interested in how alternative frameworks implement the active record pattern.  As they say, inspiration is one part perspiration and three parts plagiarism.

So I started to dig through the project page a bit, trying to find more info.  Unfortunately, the jPersist project page is a bit weird in how the whole thing is structured.  The page layout implies that jPersist is in fact a sub-project of some kind, which lead me to initially discount the “Documentation” link in the main nav bar.  Even after I found the documentation though, I was somewhat disappointed in its limited scope.  Effectively, the only documentation available is some basic javadoc and a “getting started” tutorial.  That’s alright though, I can work with that.  Code samples are more informative anyway, right?  I downloaded the full distribution (including javadoc and examples) and began to poke around.

The first thing I noticed upon opening the “DatabaseExample.java” file was the syntax: it’s very verbose.  Granted, most persistence APIs suffer from a less-than-DSL design, but I was still a bit taken aback at the volume of code required to accomplish a simple task.  For example, this is how you INSERT a new entity “Contact” with jPersist:

Database db = dbm.getDatabase();
db.saveObject(new Contact("alincoln", "mypasswd1", "Abraham", "Lincoln", 
        null, "alincoln@unitedstates.gov"));

This is assuming that you have defined the POJO Contact with a constructor taking all of the above arguments.  Obviously this could have been just as easily simplified into something more like this:

Contact c = new Contact();
c.setContactId("alincoln");
c.setPassword("mypasswd1");
c.setFirstName("Abraham");
c.setLastName("Lincoln");
c.setEmail("alincoln@unitedstates.gov");
 
Database db = dbm.getDatabase();
db.saveObject(c);

Still a bit verbose, but understandably so. I don’t know of any persistence API in Java that can accomplish the same thing more compactly, so I won’t hold it against jPersist. 

What really stands out at me in this example is that Contact isn’t actually a database peer, it’s a bean.  It’s just a regular, run-of-the-mill POJO created using a constructor upon which mutators are invoked.  This doesn’t really seem significant until you consider the claim from the project page (emphasis mine):

jPersist is an extremely powerful object-relational persistence API that is based on the Active-Record pattern.

Hmm, that’s odd, because it sure looks like data mapping to me.  By definition, the active record pattern requires the entities to directly peer to database rows or (more correctly) result-set rows.  You can’t just instantiate an active record entity, it must be created for you since it has to peer to the database somehow.  More importantly, to satisfy the active record pattern, entities must have the ability to persist themselves, at least from an API standpoint.  In this example, we have to haul around an instance of Database which we invoke separately to create, update, delete and query our entities.  This property is the hallmark of the data mapper pattern, which (as the name implies) involves a separate mapper which translates data from one form (entity beans) to another (database records).

Upon digging a bit further, I did manage to find “ProxyExample.java”.  The title itself seemed promising as any implementation of the active-record pattern in Java must utilize proxies at some level (JDK interface proxies or otherwise).  Digging into the source for the example, I discovered an interface Contacts extending DatabaseObject which was used further down in the following way:

Database db = dbm.getDatabase();
Contacts c = (Contacts) db.castDatabase(Contacts.class);
 
c.setResultSetConcurrency(ResultSet.CONCUR_UPDATABLE);
c.executeQuery("select * from contacts where 1 = 0");
 
c.moveToInsertRow();
c.setContactId("contactId" + System.currentTimeMillis());
c.setPassword("password");
c.setFirstName("First Name");
c.setLastName("Last Name");
c.setCompanyName("Company Name");
c.setEmail("email");
c.insertRow();

So it does seem that the active-record pattern is supported as a secondary mechanism for persistence (the documentation heavily emphasizes the data mapper style).  I do find it a bit odd though that we’re executing queries against an entity directly, cursoring around in result sets and still performing operations with a non-peered entity instance.  It’s also a bad sign that even having the documentation in front of me, as well as a pretty solid knowledge of database concepts, I’m still not entirely sure what the above code does on a semantic level.

Incidentally, if you look at the javadoc for DatabaseObject, it is basically functioning as a super-interface for any persistence class.  Database is the most notable implementation.  Thus, technically speaking we’re still not getting the active-record pattern in the above sample, we’re just being handed a data mapper which maps data from itself to the database.  You could make an argument that this satisfies the basic properties of active record, but I’m skeptical.

Along this line, I’d like to point out that the API of DatabaseObject (and by extension, Database) is heavily dependant on modification of its internal state.  For example, notice anything odd about the following snippet?

Database db = dbm.getDatabase();
db.queryObject(new Contact(), 
        "where :companyName = 'United States' order by :lastName");
 
Contact contact = null;
while (db.hasNext() && db.next() != null) {
    db.loadObject(contact = new Contact());
    System.out.println(contact);
}

Passing over the introduction of yet another framework-specific query language, notice the way the calls are structured?  Database itself is being used as an iterator.  So in a nutshell, you query the database using the queryObject(Object, String) method, which opens a result-set and saves it internally.  You then iterate over the database, loading the current object as you go.  If you want to be academic about this, what we have here is called an “internal iterator”.  As any compsci 201 student will tell you, internal iterators are bad news, both for concurrency and scalability.  Also, they tend to somewhat unintuitive APIs (iterating over a database?) and non-standard idioms.  In addition to all, this approach also imposes a hard limit on the number of queries which can be inspected at once against a given Database instance (one).  This leads to more odd idioms (like maintaining n Database instances for flexibility) and can introduce some bugs which would be very difficult to track down.

Moving on through the examples, a few more things jumped out at more, both positive and negative.  As I mentioned above, jPersist introduces its own SQL-derivative query language which allows querying for entities directly using field names (something supported by Hibernate using HQL but not by ActiveRecord or ActiveObjects).  Consider the following jPersist example:

Database db = dbm.getDatabase();
db.queryObject(new Contact(), 
        "where :companyName = 'United States' order by :lastName");

Compare to the corresponding ActiveObjects code:

EntityManager em = ...
Contact[] contacts = em.query(Contact.class, 
        Query.select().where("companyName = 'United States'").order("lastName"));

Hmm, which do you think is more readable? (hint: it ain’t ActiveObjects)

There are varying schools of thought on whether framework-specific query languages are a good idea or not.  Personally, I’m very opposed to them due to the added complexity and overhead they impose.  Why would I want to learn n different query languages rather than sticking with good old SQL?  On the flip side, I can see why such “spin-off” languages are nice when I look at examples like the above.  It’s really a matter of personal preference.

Another thing that’s really worth mentioning is the design decision made by jPersist not to avail itself of Java 5 syntax.  For example, if you want to use prepared statement parameters with a jPersist query, you must pass them in the following way:

db.queryObject(customer, "where :state = ? order by :lastName", 
        new Object[] {"arizona"});

A basic application of varargs would have simplified this API tremendously. Another example of the lack of Java 5 usage is in the active-record sample:

Contacts c = (Contacts) db.castDatabase(Contacts.class);

Notice the cast? Adding a type parameter to the castDatabase method would eliminate the cast entirely without bulking up the method call in any way.

The major advantage to not using Java 5 features is compatibility.  You can use jPersist on pre-Java 5 systems, possibly even back as far as Java 1.2 (not sure if it uses assertions or not).  Though, this advantage is of questionable value given the scarcity of such neolithic JVMs.

There’s more that’s worth mentioning, like the rather odd syntax for defining relations or the all-important schema generation, but this post is already pushing record sizes.  In brief: jPersist exhibits some interesting properties, and it certain provides a unique perspective on the field of database persistence, but I can’t say I’m particularly fond of the API.  Hopefully, the weaknesses I’ve enumerated will eventually be rectified, resulting in a really excellent library.