Skip to content
Print

Pipe Dream: Static Analysis for Ruby

30
Jun
2008

Yes, yes I know: Ruby is a dynamic language.  The word “static” is literally opposed to everything the language stands for.  With that said, I think that even Ruby development environments would benefit from some simple static analysis, just enough to catch the really idiotic errors.

Here’s the crux of the problem: people don’t test well.  Even with nice, behavior-driven development as facilitated by frameworks like RSpec, very few developers sufficiently test their code.  This isn’t just a problem with dynamic languages either, no one is safe from the test disability.  In some ways, it’s a product of laziness, but I think in most cases, good developers just don’t want to work on mundane problems.  It’s boring having to write unit test after unit test, checking and re-checking the same snippet of code with different input.

In some sense, it is this problem that compilers and static type systems try to avert, at least partially.  The very purpose of a static type system is to be able to prove certain things about your code simply by analysis.  By enabling the compiler to say things using the type system, the language is providing a safety net which filters out ninety percent of the annoying “no-brainer” mistakes.  A simple example would be invoking a method with the wrong parameters; or worse yet, misspelling the name of the method or type altogether.

The problem is that there are some problems which are more simply expressed in ways which are not provably sound.  In static languages, we get around this by casting, but such techniques are ugly and obviously contrived.  It is this problem which has given rise to the kingdom of dynamic languages; it is for this reason that most scripting languages have dynamic type systems: simple expression of algorithm without worrying about provability.  In fact, there are so many problems which do not fit nicely within most type systems that many developers have chosen to eschew static languages altogether, claiming that static typing just gets in the way.

Unfortunately, by abandoning static types, these languages lose that typo safety net.  It’s too easy to make a trivial mistake in a dynamic language, buried somewhere deep in the bowls of your application.  This mistake could easily be averted by a compiler with validating semantic analysis, but in a dynamic language, such a mistake could go unnoticed, conceivably even making it into production.  For this reason, most dynamic language proponents are also strong advocates of solid, comprehensive testing.  They have to be, for without such testing, one should never trust dynamic code in a production system (or any code, for that matter, but especially the unchecked dynamic variety).

Most large, production systems written in languages like Ruby or Groovy have large test suites which sometimes take hours to run.  These suites are extremely fine-grained, optimally checking every line of code with every possible kind of input, so as to be sure that mistakes are caught.  This is where the flexibility of dynamic typing really comes back to haunt you: extra testing is required to ensure that silly mistakes don’t slip through.  The irony is that a lot of developers using dynamic languages do so to get away from the “nuisance” of compilation, when all they have done is trade one inconvenience for another (testing).

Given this situation, it’s not unreasonable to conclude that what dynamic languages really need is a tool which can look through code and find all of those brain-dead mistakes.  Such a tool could be run along with the normal test suite, finding and reporting errors in much the same way.  It wouldn’t really have to be a compiler, so the tool wouldn’t slow down the development process, it would just be an effective layer of automated white-box testing.

But how could such a thing be accomplished in a language like Ruby?  After all, it is a truly dynamic language.  Methods don’t even exist until runtime, and sometimes only if certain code paths are run.  Types are completely undeclared, and every object can potentially respond to any method.  The answer is to perform extremely permissive inference.

It was actually a recent post by James Ervin on the nomenclature of type systems which got me thinking along these lines.  It should be possible by static analysis to infer the structural type of any value based on its usage.  Consider:

def do_something(obj)
  if obj.to_i == 0
    obj[:test]
  else
    other = obj.find :name => 'Daniel'
    other.to_s
  end
end

Just by examining this code, we can say certain things about the types involved.  For instance, we know that obj must respond to the following methods:

  • to_i
  • [Symbol]
  • find(Hash)

In turn, we know that the find(Hash) method must return a value which defines to_s.  Of course, this last bit of information isn’t very useful, because every object defines that method, but it’s still worth the inference.  The really useful inference which comes out of to_s is the knowledge that this method sometimes returns a value of type String (making the assumption that to_s hasn’t been redefined to return a different type, which isn’t exactly a safe assumption).  At other times, do_something will return whatever value comes from the square bracket operator ([]) on obj.  This bit of information we must remember in the analysis.  We can’t just assume that this method will return a String all the time, even if to_s does because method return types need not be homogeneous in dynamic languages.

Now, at this point we have effectively built up a structural type which is accepted by do_something.  Literally, we have formalized in the analysis what our intuition has already told us about the method.  There are some gaps, but that is to be expected.  The key to this analysis is not attempting to be comprehensive.  Dynamic languages cannot be analyzed as if they were static, one must expect to have certain limitations.  In such situations where the analysis is insufficient, it must assume that the code is valid, otherwise there will be thousands of false positives in the error checking.

So what is it all good for?  Well, imagine that somewhere else in our application, we have the following bit of code:

do_something 42

This is something we know will fail, because we have a simple value (42) which has a nominal type we can easily infer.  A little bit of checking on this type reveals the fact that it does not define square brackets, nor does it define a find(Hash) method.  This finding could be reported as an error by the analysis engine.

Granted, we still have to account for the fact that Ruby has things like method_missing and open classes, but all of this can fall into the fuzzy area of the analysis.  In situations where it might be alright to pass an object which does not satisfy a certain aspect of the structural type, the analysis must let it pass without question.

You can imagine how this analysis could traverse the entire source tree, making the strictest inferences it can and allowing for dynamic fuzziness where applicable.  Since the full sources of every Ruby function, class and module are available at runtime, analysis could be performed without any undue concern regarding obfuscation or parsing of binaries.  Conceivably, most trivial errors could be caught without any tests being written, taking some of the burden off of the developer.  There is a slight concern that developers would build up a false sense of security regarding their testing (or lack thereof), but I think we just have to trust that won’t happen, or won’t last long if it does.

Most advanced Ruby toolsets already have an analysis somewhat similar to the one I outlined.  NetBeans Ruby for example has some fairly advanced nominal type inference to allow things like semantic highlighting and content assist.  But as far as I know, this type inference is only nominal, and fairly local at that.  The structural type inference that I am proposing could conceivably provide far better assurances and capabilities than mere nominal inference, especially if enhanced through successive iteration and a more “global” approach (similar to Hindley/Milner in static languages).

One thing is certain, it isn’t working to just rely on developers being conscientious with their testing.  With the rapid rise in production systems running on dynamic languages, it is in all of our best interests to try to find a way to make these systems more stable and reliable.  The best way to do this is to start with code assurance and try to make it a little less painful to catch mistakes before deployment.

Comments

  1. Even in the case of Eclipse’s Java support, there are certain situations that only indicate you might be making a mistake, like when you use an enum in a switch block but don’t use all values, or a missing serialVersionUID. A method call that falls down into method_missing is maybe a similar situation.

    I’m thinking that runtime analysis might be the way to go though, especially in Ruby where you can attach methods to instances and classes at will. Resolving the ordering of these things seems too tough at a purely source code level, but at runtime you can verify which methods are available on an instance. Then maybe you could use the data from that analysis and insert it back into the source code viewer, kind of like some code coverage tools do.

    matthew Monday, June 30, 2008 at 9:21 am
  2. @matthew: I’ve thought about runtime analysis before a little, by no means thoroughly. But the problem is that you have no idea what code is safe to run. The analysis may exercise some code which deletes a whole directory of files. For example you may think you’re just resolving a class and enumerating its methods. But when that class is resolved maybe it makes a connection to a database and resets some state somewhere. Seems like dangerous and unpredictable stuff to do especially when you’re dealing with 3rd party libs.

    Maybe some sort of guided analyzer would work where you show it safe paths or maybe just dangerous paths to exclude from the analysis. This would still be work for the developer but could be easier than writing tests.

    It would be nice to see more attempts in this area forsure.

    coderrr Monday, June 30, 2008 at 10:07 am
  3. coderrr is right, there’s a fairly hard limit on what is possible with pure runtime analysis. Ruby’s highly dynamic nature actually makes this easier than it would otherwise be (subbing in a new implementation of Object.new, for example), but there’s really no way around the issue of side-effects. It is for this reason that I think that some sort of static analysis is really the only way to improve on the situation, due to the fact that static analysis doesn’t have the concern associated with external side-effects. The obvious problem being that you can’t really guarantee much about the code. ActiveRecord is a good example of a library which relies upon external conditions (a database) to define what object structure is going to look like. Clever, but very frustrating from the perspective of stability assurance.

    Daniel Spiewak Monday, June 30, 2008 at 12:56 pm
  4. Actually, this kind of type inference can be taken pretty far. Look at Traits (for Squeak), (http://doi.acm.org/10.1145/1119479.1119483) which is a simple system that does just the kind of analysis you’re talking about (and allows automatic refactoring based on this analysis, or Moose http://moose.unibe.ch/ which is an ambitious system that does many kinds of analysis on software.

    Ned Konz Thursday, July 2, 2009 at 7:35 am
  5. You probably already know this but the Perl community scoffed at the idea of a static code analyzer for Perl, that is, until Adam Kennedy produced PPI (http://search.cpan.org/dist/PPI/) which led to the creation of Perl::Critic (http://search.cpan.org/dist/Perl-Critic/ or http://perlcritic.com/)

    So, there is hope for doing something like this in Ruby.

    Matisse Enzer Saturday, July 10, 2010 at 9:18 pm
  6. That’s called Soft Typing (Static typing + Type inference + allowing indeterminable types to be checked at runtime).

    anonymous Thursday, September 9, 2010 at 5:50 pm

Post a Comment

Comments are automatically formatted. Markup are either stripped or will cause large blocks of text to be eaten, depending on the phase of the moon. Code snippets should be wrapped in <pre>...</pre> tags. Indentation within pre tags will be preserved, and most instances of "<" and ">" will work without a problem.

Please note that first-time commenters are moderated, so don't panic if your comment doesn't appear immediately.

*
*