It gets much more complicates in agglutinative languages like Hungarian and Finnish, and even languages like German which can basically synthesize arbitrary words and still remain intelligible German.

A better way to determine a language is to use an n-gram model. The distribution of n-grams across a language is a great finger print and you can identify most languages within a word or two.

Cheers,
Jesse

That aside, this is a great introduction to the topic and it was interesting to see the Scala code. I own scalainside.com, but unfortunately never got into the language enough to go full steam into it.

Well, not *quite* blind loyalty. I really, really like the benefits that immutable data structures carry with them. I mean, think about it, how many times have you done something like this by accident?

public class Bean {
private String[] data;

private String[] getData() {
return data;
}
}

There’s no setter for data, but it doesn’t really matter; anyone can call getData() and then change the internals. It’s even worse with more complex data structures like List and Map. And don’t even get me started on thread synchronization issues…

In terms of performance, immutable data structures can be pretty darn fast. My benchmarking of Rich Hickey’s persistent vector shows it to be within 5x-10x of the speed of an ArrayList on “writes” (remember, a new Vector each time), and between 1x and 2x the speed on reads. It’s hard to argue that these factors constitute a significant performance hit, especially when we remember that lookups in an ArrayList are on the order of 0.5-1ms.

It should be possible to implement a bitset immutably using much the same technique as is used in the persistent vector. However, rather than the lowest level of the trie being composed of a 32-element array, this level would be a single 32-bit integer. Because of the speed of integer copying on modern processors (constant), this sort of data structure would actually be phenomenally faster than a general-purpose Vector. Granted, this is to be expected since it is limited to bit values, but it’s still a cool conclusion to trot out.

With this sort of implementation, I don’t think you would really have to discard any of the benefits of a bloom filter. Granted, it would certainly have a higher “write” overhead than a mutable bloom filter, but its reads would be very nearly (if not exactly) as fast. The “write” overhead wouldn’t be too much of a concern because bloom filters are usually used immutably anyway. In short, I think the benefits of a fully immutable bloom filter would far outweigh the minimal performance penalty imposed by such a structure.

P.S. Oh, I’m not really claiming that immutable data structures are computationally as efficient as their mutable counterparts. Even Clojure’s persistent vector is really only log32(n) for reads, which is close to constant time but still not quite. However, in most scenarios that I work on, immutable data structures lend considerably to the correctness and logical “reasonability” of the code. Add in their inherently stellar throughput when dealing with concurrency, and I think the overall benefits are more than sufficiently compelling (at least, they are to me).

I’m not terribly familiar with how a bitset would be implemented in an immutable manner, but my guess is that you would throw away much of the performance benefit of using a BloomFilter.

Immutability is beneficial in many, but not every circumstance.

wl.processText(“This is a very long string which contains a lot of different words. The goal is to try to confuse the language processor and draw out some more contrasting results in terms of language scoring”)

=> Map(Portuguese -> 10, English -> 31, Swedish -> 5, Farsi -> 3, German -> 1, Spanish -> 7, French -> 13, Dutch -> 15, Pinyin -> 1)

And afterward:

=> Map(Portuguese -> 9, English -> 30, Swedish -> 7, Farsi -> 2, German -> 1, Spanish -> 3, French -> 13, Dutch -> 17, Pinyin -> 1)

So oddly enough, it seems that the floating width has a slightly deleterious effect on accuracy, at least with this particular string. Overall accuracy should have been improved, particularly with strings containing Russian words. The only unfortunate consequence is the memory requirements have increased due to the average width being higher than 2,000,000. Still, an interesting experiment to be sure.

Your note on performance actually makes a lot of sense. I was running this with JRuby 1.1.2. For the record, I wasn’t really expecting (or demanding) performance approaching Scala’s in JRuby, I was just somewhat surprised to see how divergent it really was.

The String conversion was a beast as well. I actually had to force the conversion by explicitly creating a new instance of java.lang.String, which makes things all the more worse.

@Asd and @Ian

I’m a little surprised that the JVM doesn’t optimize boolean[]. Considering all the other complicated optimizations performed by the JIT, this would seem to me to be low hanging fruit.

In any case though, you’re right that the JVM isn’t optimizing BloomSet down to one bit per element. Even if it could optimize boolean[], Vector isn’t using a typed array under the surface. It’s actually even worse than a straight array of primitive booleans due to the fact that every boolean has to be stored in its boxed form. Watching the memory usage of an example application using WhatLanguage shows that life isn’t *too* terrible, but it’s still a lot more than I would have liked.

Thinking about it, the best way to solve this would be (as you say) to create a new persistent data structure based on the Vector concept which stores bits in Integer masks. Conveniently, this would still preserve all of the exact properties of Clojure’s persistent vector since integers contain 32 bits. This sort of data structure would not be possible just as a “specialization” of Vector, but the same ideas could be cross-applied through (mostly) copy-paste coding.

Java’s BitSet isn’t really an option because it is mutable. I really strongly believe in using immutable data structures except where they carry a severe performance hit. Immutable data structures can be used mutably if you really need to, but they also carry all of the benefits of air-tight thread safety and dirt cheap versioning (not to mention correctness assurance in cases where a return value may expose implementation details).

@Asd (again)

It would seem that you’re right about hash. I guess that’s what happens when I try to re-tune algorithms at the last minute… :-S Originally, I had a very different version of hash based on Random (@Tim). As I was actually writing the article, I attempted to pair down the implementation just a bit so as to improve the readability. Seems I went just a little too far…

Considering that there is exactly one integer value (out of 2^32 – 1) that reproduces this issue, I think I’m pretty safe. Do you know of another function which does better? I suppose I could just right-shift everything by one place, which yields the more natural top-end result: Integer.MIN_VALUE >> 1 == Integer.MAX_VALUE.

@Tim

Character repetition frequency is an interesting way of detecting language. This would be a lot more memory-efficient than assembling bloom filters. However, I’m not entirely sure that it would be as accurate. The nice thing about the dictionary approach is fairly easy to analyze its properties and discover why it is wrong in such cases that it is. This contrasts pretty heavily with a more statistical approach, which would be very difficult to debug. I’m sure there are pathological English sentences that look a lot like French, German or even Dutch (considering the similarities in the languages).

@Thomas

I thought about letting the width float, but apparently you’re more ambitious than I am in terms of calculating optimizations. The BloomSet API is capable of accepting a configurable width, so this would only require changes in the GenLangs.scala script. Thanks for the pointer!

@Paolo

When I initially looked at the algorithm, it seemed closer to O(n^2), but somehow my second glance told me it was O(n!). I’ll correct the article. Thanks for catching my error!

If it were the factorial, even 30 hash functions would be waaaaay too much.

# –
def size_filter(capacity, error = 1 / 1000000.0)
m = ((capacity * Math.log(error)) / Math.log(1.0 / (2.0 ** Math.log(2.0)))).ceil
k = (Math.log(2.0) * m / capacity).ceil
{:k => k, :m => m}
end

p size_filter(234936, 0.03) # => {:m=>1714667, :k=>6}
p size_filter(399, 0.03) # => {:m=>2913, :k=>6}
# –

Also, after finding how poorly “fast” hashes perform on bloomfilters, I ended up just using SHA512; you get excellent distribution, and can make a lot of k’s out of all those bits.