Disclaimer: For those of you who don’t know, my pet project is an ORM based on the active record pattern (ActiveObjects).  As such, my opinions are probably colored somewhat.  Take everything I say here with a grain of salt.

There seems to have been a recent resurgence of interest in Java implementations of the active record pattern in recent months.  Most traditional Java ORMs (like Hibernate/JPA) implement the data mapper pattern, as it is far easier to apply to a static language like Java.  Yet I can count a number of new (at least to me) ORM projects which attempt to fill this void.  One such project is jPersist.  From the project page:

jPersist is an extremely powerful object-relational persistence API that is based on the Active-Record pattern. jPersist is mapless and has no need for XML or annotation based mapping (all mapping is automatic and dynamic).

Sounds promising!  I’m always in favor of any framework which can reduce the boiler-plate configuration required, especially when the configuration would have been in a language as verbose as XML.  But irregardless of the method for configuration, I’m always interested in how alternative frameworks implement the active record pattern.  As they say, inspiration is one part perspiration and three parts plagiarism.

So I started to dig through the project page a bit, trying to find more info.  Unfortunately, the jPersist project page is a bit weird in how the whole thing is structured.  The page layout implies that jPersist is in fact a sub-project of some kind, which lead me to initially discount the “Documentation” link in the main nav bar.  Even after I found the documentation though, I was somewhat disappointed in its limited scope.  Effectively, the only documentation available is some basic javadoc and a “getting started” tutorial.  That’s alright though, I can work with that.  Code samples are more informative anyway, right?  I downloaded the full distribution (including javadoc and examples) and began to poke around.

The first thing I noticed upon opening the “DatabaseExample.java” file was the syntax: it’s very verbose.  Granted, most persistence APIs suffer from a less-than-DSL design, but I was still a bit taken aback at the volume of code required to accomplish a simple task.  For example, this is how you INSERT a new entity “Contact” with jPersist:

Database db = dbm.getDatabase();
db.saveObject(new Contact("alincoln", "mypasswd1", "Abraham", "Lincoln", 
        null, "alincoln@unitedstates.gov"));

This is assuming that you have defined the POJO Contact with a constructor taking all of the above arguments.  Obviously this could have been just as easily simplified into something more like this:

Contact c = new Contact();
c.setContactId("alincoln");
c.setPassword("mypasswd1");
c.setFirstName("Abraham");
c.setLastName("Lincoln");
c.setEmail("alincoln@unitedstates.gov");
 
Database db = dbm.getDatabase();
db.saveObject(c);

Still a bit verbose, but understandably so. I don’t know of any persistence API in Java that can accomplish the same thing more compactly, so I won’t hold it against jPersist. 

What really stands out at me in this example is that Contact isn’t actually a database peer, it’s a bean.  It’s just a regular, run-of-the-mill POJO created using a constructor upon which mutators are invoked.  This doesn’t really seem significant until you consider the claim from the project page (emphasis mine):

jPersist is an extremely powerful object-relational persistence API that is based on the Active-Record pattern.

Hmm, that’s odd, because it sure looks like data mapping to me.  By definition, the active record pattern requires the entities to directly peer to database rows or (more correctly) result-set rows.  You can’t just instantiate an active record entity, it must be created for you since it has to peer to the database somehow.  More importantly, to satisfy the active record pattern, entities must have the ability to persist themselves, at least from an API standpoint.  In this example, we have to haul around an instance of Database which we invoke separately to create, update, delete and query our entities.  This property is the hallmark of the data mapper pattern, which (as the name implies) involves a separate mapper which translates data from one form (entity beans) to another (database records).

Upon digging a bit further, I did manage to find “ProxyExample.java”.  The title itself seemed promising as any implementation of the active-record pattern in Java must utilize proxies at some level (JDK interface proxies or otherwise).  Digging into the source for the example, I discovered an interface Contacts extending DatabaseObject which was used further down in the following way:

Database db = dbm.getDatabase();
Contacts c = (Contacts) db.castDatabase(Contacts.class);
 
c.setResultSetConcurrency(ResultSet.CONCUR_UPDATABLE);
c.executeQuery("select * from contacts where 1 = 0");
 
c.moveToInsertRow();
c.setContactId("contactId" + System.currentTimeMillis());
c.setPassword("password");
c.setFirstName("First Name");
c.setLastName("Last Name");
c.setCompanyName("Company Name");
c.setEmail("email");
c.insertRow();

So it does seem that the active-record pattern is supported as a secondary mechanism for persistence (the documentation heavily emphasizes the data mapper style).  I do find it a bit odd though that we’re executing queries against an entity directly, cursoring around in result sets and still performing operations with a non-peered entity instance.  It’s also a bad sign that even having the documentation in front of me, as well as a pretty solid knowledge of database concepts, I’m still not entirely sure what the above code does on a semantic level.

Incidentally, if you look at the javadoc for DatabaseObject, it is basically functioning as a super-interface for any persistence class.  Database is the most notable implementation.  Thus, technically speaking we’re still not getting the active-record pattern in the above sample, we’re just being handed a data mapper which maps data from itself to the database.  You could make an argument that this satisfies the basic properties of active record, but I’m skeptical.

Along this line, I’d like to point out that the API of DatabaseObject (and by extension, Database) is heavily dependant on modification of its internal state.  For example, notice anything odd about the following snippet?

Database db = dbm.getDatabase();
db.queryObject(new Contact(), 
        "where :companyName = 'United States' order by :lastName");
 
Contact contact = null;
while (db.hasNext() && db.next() != null) {
    db.loadObject(contact = new Contact());
    System.out.println(contact);
}

Passing over the introduction of yet another framework-specific query language, notice the way the calls are structured?  Database itself is being used as an iterator.  So in a nutshell, you query the database using the queryObject(Object, String) method, which opens a result-set and saves it internally.  You then iterate over the database, loading the current object as you go.  If you want to be academic about this, what we have here is called an “internal iterator”.  As any compsci 201 student will tell you, internal iterators are bad news, both for concurrency and scalability.  Also, they tend to somewhat unintuitive APIs (iterating over a database?) and non-standard idioms.  In addition to all, this approach also imposes a hard limit on the number of queries which can be inspected at once against a given Database instance (one).  This leads to more odd idioms (like maintaining n Database instances for flexibility) and can introduce some bugs which would be very difficult to track down.

Moving on through the examples, a few more things jumped out at more, both positive and negative.  As I mentioned above, jPersist introduces its own SQL-derivative query language which allows querying for entities directly using field names (something supported by Hibernate using HQL but not by ActiveRecord or ActiveObjects).  Consider the following jPersist example:

Database db = dbm.getDatabase();
db.queryObject(new Contact(), 
        "where :companyName = 'United States' order by :lastName");

Compare to the corresponding ActiveObjects code:

EntityManager em = ...
Contact[] contacts = em.query(Contact.class, 
        Query.select().where("companyName = 'United States'").order("lastName"));

Hmm, which do you think is more readable? (hint: it ain’t ActiveObjects)

There are varying schools of thought on whether framework-specific query languages are a good idea or not.  Personally, I’m very opposed to them due to the added complexity and overhead they impose.  Why would I want to learn n different query languages rather than sticking with good old SQL?  On the flip side, I can see why such “spin-off” languages are nice when I look at examples like the above.  It’s really a matter of personal preference.

Another thing that’s really worth mentioning is the design decision made by jPersist not to avail itself of Java 5 syntax.  For example, if you want to use prepared statement parameters with a jPersist query, you must pass them in the following way:

db.queryObject(customer, "where :state = ? order by :lastName", 
        new Object[] {"arizona"});

A basic application of varargs would have simplified this API tremendously. Another example of the lack of Java 5 usage is in the active-record sample:

Contacts c = (Contacts) db.castDatabase(Contacts.class);

Notice the cast? Adding a type parameter to the castDatabase method would eliminate the cast entirely without bulking up the method call in any way.

The major advantage to not using Java 5 features is compatibility.  You can use jPersist on pre-Java 5 systems, possibly even back as far as Java 1.2 (not sure if it uses assertions or not).  Though, this advantage is of questionable value given the scarcity of such neolithic JVMs.

There’s more that’s worth mentioning, like the rather odd syntax for defining relations or the all-important schema generation, but this post is already pushing record sizes.  In brief: jPersist exhibits some interesting properties, and it certain provides a unique perspective on the field of database persistence, but I can’t say I’m particularly fond of the API.  Hopefully, the weaknesses I’ve enumerated will eventually be rectified, resulting in a really excellent library.

ORMs really interest me, so naturally I read a lot of material regarding ORMs of all kinds, especially Hibernate and ActiveRecord.  One of the more interesting reviews I read recently complained about the rigidity of the type system in the Rails ORM.  According to the author’s examination of the code, ActiveRecord just uses a monolithic switch/case statement to determine the appropriate Ruby type from the SQL type in the result set.  This may make sense from a simplicity standpoint, but it may not be the best approach when it comes to flexibility.

The problem with this approach is that it’s impossible to easily add new types to the ORM.  Granted, the framework authors could do it by modifying the switch/case statement(s) - and the approach does usually require more than one statement - and releasing a whole new version of the framework.  This is not a significant issue as the framework authors already have access to the full library sources.  The real trial is with third-party developers who require custom data types.

An alternative approach (suggested in the article) is to implement a series of type delegates inheriting from a common superclass, or possibly using a mixin as allowed by Ruby.  These type classes would each be responsible for a single type, handling the mapping both to and from the language-native type to the database type.  This would allow for both easy addition and modification of core types by the framework authors, but also trivial support for arbitrary types as implemented by third-party developers.

Not one to shirk good advise when I hear it, I’ve decided to go with this approach to types in ActiveObjects.  Formerly, I must admit I had gone with the multiple, giant switch/case statements.  This seemed to make sense when I first implemented the framework, but it developed, it became apparent that this was inadequate, especially if third-party types are desired.  This decision led to the refactoring of the type system and subsequent creation of the TypeManager class.

TypeManager is basically the singleton manager for the entire type system.  It maintains the list of available DatabaseType(s) and can resolve both Java classes and SQL types to the appropriate delegate.  A number of core types (VarcharType, IntegerType, etc) are added to the singleton instance of TypeManager, ensuring that basic functionality works without any extra effort on the part of the developer.  If a type other than the core types is needed, all that is necessary is to add the type delegate instance to the TypeManager prior to the type’s usage in either migrations or data access.  Thusly:

public interface Company extends Entity {
    public String getName();
    public void setName(String name);
 
    public Class<?> getJavaType();
    public void setJavaType(Class<?> type);
}
 
public class ClassType extends DatabaseType<Class<?>> {
 
    public ClassType() {
        super(Types.VARCHAR, 255, Class.class);
    }
 
    @Override
    public Class<?> convert(EntityManager manager, ResultSet res, 
                Class<? extends Class<?>&gt; type, String field) throws SQLException {
        try {
            return Class.forName(res.getString(field));
        } catch (Throwable t) {
            return null;
        }
    }
 
    @Override
    public void putToDatabase(int index, PreparedStatement stmt, 
                Class<?> value) throws SQLException {
        stmt.setString(index, value.getName());
    }
 
    @Override
    public Object defaultParseValue(String value) {
        try {
            return Class.forName(value);
        } catch (Throwable t) {
            return null;
        }
    }
 
    @Override
    public String valueToString(Object value) {
        if (value instanceof Class) {
            return ((Class<?>) value).getName();
        }
 
        return super.valueToString(value);
    }
 
    @Override
    public String getDefaultName() {
        return "VARCHAR";
    }
}
 
// ...
TypeManager.getInstance().addType(new ClassType());
 
Company[] stringCompanies = manager.find(Company.class, "javaType = ?", String.class);
for (Company c : stringCompanies) {
    System.out.println(c.getName() + " former held type " + c.getJavaType().getName());
 
    c.setJavaType(Exception.class);
    c.save();
}

The most complicated bit of the example above is the database type itself.  Yet even this delegate isn’t too horrible.  The ClassType class first specifies in its constructor which types it corresponds to, both database and Java.  Multiple Java class types can be specified, allowing for cases like IntegerType which maps to both Integer.class and int.class.

The rest of the database type is fairly self-explanatory.  There are methods to read the Java value out of a JDBC ResultSet, put the Java value back into a JDBC PreparedStatement, as well as three methods to handle some of the non-database type-sensitive operations, such as parsing a String value into a type-specific value and visa-versa.  These database non-specific conversions are required for things like parsing the value of a @Default or an @OnUpdate annotation.  Finally, getDefaultName() allows the default DDL rendering of the type to be specified.  This can be overridden in the DatabaseProvider implementation for that particular database, but the use of getDefaultName() allows for third party types that the database provider developers may not have foreseen.  Thus, it effectively opens the door to third-party types in migrations.

Of course, no example would be complete without another one to complement it!  Here’s how we could create a type delegate for the java.awt.Point class:

public class PointType extends DatabaseType<Point> {
    private static final Pattern PATTERN = Pattern.compile("x=(\\d+),y=(\\d+)");
 
    protected PointType() {
        super(Types.VARCHAR, 45, Point.class);
    }
 
    @Override
    public Point convert(EntityManager manager, ResultSet res, 
            Class<? extends Point> type, String field) throws SQLException {
        return (Point) defaultParseValue(res.getString(field));
    }
 
    @Override
    public void putToDatabase(int index, PreparedStatement stmt, Point value) 
                throws SQLException {
        stmt.setString(index, valueToString(value));
    }
 
    @Override
    public Object defaultParseValue(String value) {
        Point back = null;
        Matcher matcher = PATTERN.matcher(value);
 
        if (matcher.find()) {
            back = new Point();
            back.x = Integer.parseInt(matcher.group(1));
            back.y = Integer.parseInt(matcher.group(2));
        }
 
        return back;
    }
 
    @Override
    public String getDefaultName() {
        return "VARCHAR";
    }
}

One thing of note here which has changed from the previous example of ClassType is that the second parameter to the super constructor is now 45, instead of 255.  This parameter is actually the default precision of the SQL type when rendered into the database.  If the SQL type doesn’t have a precision or should just take the database default, a negative value should be specified for this parameter.  Another item of note is that we’re delegating work between methods in a way that I simply didn’t do for ClassType.  Because the rendering of the type in the database is in VARCHAR (String) form, we can rely upon our default String conversion methods to render into the database.  As an aside, the superclass implementation of valueToString(Object) uses the toString() method for that particular value.

As you can see, the type system in ActiveObjects is incredibly powerful and capable of satisfying many use-cases that were impossible in previous versions or other ORMs.  Hopefully this brief glimpse into advanced uses of the type system will aid you in databasing efforts.

So in the intervening time since I last updated you on ActiveObjects, I’ve been busy refactoring and repurposing some of the core.  I’ve added a full JUnit test suite, which definitely helps my own confidence about the stability of the source.  Also, there’s a whole bunch of new features that have come down the pipe which hopefully I’ll get to address in the next few posts.  So, without further ado…

The change which is probably going to cause you the most grief is the switch from @Index to @Searchable, and the addition of the @Indexed annotation.  Yes, you really did read that correctly; and no @Indexed isn’t even related to the functionality provided by the old @Index annotation.

The old @Index annotation used to handle tagging of entity methods to mark them as to be added to the Lucene full-text search index (see this post for more details).  This was a little confusing for a number of reasons, not the least of which my failure to remember the tense of the annotation name (see the comments on the indexing post).  By convention, most Java annotations are declarative in name.  Thus, the name should not be the present tense “index” but the past tense “indexed”.  So my first thought was to just refactor the annotation, but then I came into a slightly hairier name-clash.

Database Field Indexing

One of the most common techniques for optimizing your database’s read (SELECT) performance is to create indexes on certain fields.  When a field is indexed, the database will maintain some separate hash tables to enable very fast selection of rows based on the field in question.  This is a really good thing for almost all foreign keys, for example:

SELECT ID FROM people WHERE companyID = ?

Here we’re SELECTing the id field from the people table where companyID matches a certain value.  The database can execute this query fairly quickly.  In fact, the only bottle-neck is finding all of the rows which match the specified companyID value.  In a table containing hundreds of thousands of rows, one can see how this could be a problem.

The problem goes away (sort of) with the use of field indexing.  Instead of having to linearly search through the table for rows matching the companyID, the database can perform a quick hash lookup in an index and get a set of rowid(s) based on the companyID.  Simple, efficient, and incredibly scalable.  Practically, the DBMS wouldn’t take any longer to execute such a query against a table of 100,000,000 rows than it does to execute the same query against a table of 100 rows.  So, this is a really good thing right?

Well, indexes have their drawbacks.  I won’t go into all of the reasons not to use indexes, the following two points will likely suffice:

• Indexing slows down UPDATEs and INSERTs
• Indexing adds to your table’s storage requirements (all those hashes have to be put somewhere)

So perfect database performance isn’t attainable just by indexing every field, one has to be quite judicious about it.  In fact, choosing the fields to be indexed is really as much of an art as it is a science.

This long and tangled introduction actually did have a point…  ActiveObjects didn’t have any support for field indexing.  I hadn’t really considered the possibility, so I didn’t factor it into my design.  In retrospect, this was probably a bad idea.  So making up for lost time, I’ve now introduced field indexing into the library!

public interface Person extends Entity {
 
    @Indexed
    public String getName();
    @Indexed
    public void setName(String name);
 
    public int getAge();
    public void setAge(int age);
 
    public Company getCompany();
    public void setCompany(Company company);
}

When ActiveObjects generates the migration DDL for this table (running against MySQL), it will look something like this:

CREATE TABLE people (
    ID INTEGER NOT NULL AUTO_INCREMENT,
    NAME VARCHAR(255),
    age INTEGER,
    companyID INTEGER,
    CONSTRAINT fk_people_companyID FOREIGN KEY (companyID) REFERENCES companies(ID),
    PRIMARY KEY(ID)
);
CREATE INDEX NAME ON people(NAME);
CREATE INDEX companyID ON people(companyID);

This is where the aforementioned @Indexed annotation comes in.  As you can see from the resultant DDL, adding a field index is as simple as tagging the corresponding method.  Also, foreign keys are automatically indexed, to ensure maximum performance in SELECTion of relations.

So that’s the good news, the bad news is that index creation doesn’t play too nicely with migrations.  Everything works of course, and if you’re actually adding a table or field, the corresponding index(es) will also be created.  Likewise if you drop a table or field, the corresponding index(es) will also be dropped.  However, that’s about the limit of the migrations support for indexes at the moment.  JDBC has an unfortunately limitation which prevents developers from getting a list of indexes within a database.  Since I have no way (within JDBC) of finding pre-existing indexes, they must be excluded from the schema diff and thus are CREATEd or DROPped irregardless of the existing schema.  I do have a plan to fix this (along with migrations on Oracle), but I seriously doubt that it’ll be included in the 1.0 release, owing to the changes required.

Refactoring End Result

The final result of the refactoring and the adding of field indexing is that the annotation formerly named @Index is now the @Searchable annotation.  Likewise (and keeping with convention), the formerly IndexingEntityManager is now named SearchableEntityManager.  Both of these types remain in the net.java.ao package.  To allow for field indexing, the @Indexed annotation was added to the net.java.ao.schema package, owing to the fact that it only effects schema generation and doesn’t change runtime behavior in the framework at all.

Hopefully these changes won’t be too confusing (now that you’re aware of them) and will be a welcome addition to the ActiveObjects functionality.  As always, I welcome comments, suggestions and criticisms!