Comments on: Is a Separate Text Search Engine a Bad Idea? http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea (permanently in beta) Sun, 29 Aug 2010 20:01:44 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: Daniel Spiewak http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2585 Daniel Spiewak Tue, 18 Dec 2007 16:26:46 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2585 That's a good point. The problem I see is the same as I illustrated in the article: service overload. PostgreSQL does support full text indexing, but if you try to put both full-text indexing and databasing into a single point of failure, you're going to have to deal with some serious scalability issues. I won't deny that it's possible to do things that way, but I don't think it's a terribly good idea for a high volume application. What would be really nice is some sort of integration between the search engine (Lucene) and the database (MySQL, PostgreSQL, it shouldn't matter). This sort of integration could allow things like your use case, without overloading the database. That’s a good point. The problem I see is the same as I illustrated in the article: service overload. PostgreSQL does support full text indexing, but if you try to put both full-text indexing and databasing into a single point of failure, you’re going to have to deal with some serious scalability issues. I won’t deny that it’s possible to do things that way, but I don’t think it’s a terribly good idea for a high volume application.

What would be really nice is some sort of integration between the search engine (Lucene) and the database (MySQL, PostgreSQL, it shouldn’t matter). This sort of integration could allow things like your use case, without overloading the database.

]]> By: Prathapan Sethu http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2584 Prathapan Sethu Tue, 18 Dec 2007 11:45:01 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2584 There are cases where you would want to further filter or sort the result set from a full-text query by another parameter stored in a database. For instance, I search for articles stored in Lucene, but want to sort/rank them by user-given ratings, which are stored in a database. I don't want to store the user ratings inside Lucene because that data changes a lot, and keeping the Lucene index updated is not easy. Multiple users cannot write to Lucene index concurrently. Another example: what if I want to sort the shopping cart items by price? So I think we need a single product which can handle both full-text and regular SQL queries and sort the results based on full-text relevance or some other db field. If only the database vendors take this seriously. I can see a huge demand for such an IR-DB product. There are cases where you would want to further filter or sort the result set from a full-text query by another parameter stored in a database. For instance, I search for articles stored in Lucene, but want to sort/rank them by user-given ratings, which are stored in a database. I don’t want to store the user ratings inside Lucene because that data changes a lot, and keeping the Lucene index updated is not easy. Multiple users cannot write to Lucene index concurrently. Another example: what if I want to sort the shopping cart items by price? So I think we need a single product which can handle both full-text and regular SQL queries and sort the results based on full-text relevance or some other db field. If only the database vendors take this seriously. I can see a huge demand for such an IR-DB product.

]]> By: Daniel Spiewak http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2405 Daniel Spiewak Fri, 12 Oct 2007 18:37:16 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2405 Well, I'll admit my experience with that kind of system is limited, but I know of other people who do have the experience and they say that it's often *easier* to keep the Lucene bits in sync than the DBMS. If you're having difficulties keeping Lucene in sync with your database (or visa versa), you may want to take a hard look at how you're doing things. Not just the underlying search-vs-database, but how you're implementing them. As I mentioned, a few ORMs can keep search indexes and databases in sync automatically. Have you tried this approach? Even if you haven't, a reasonable DAO should be able to take care of most of the pain by centralizing the Lucene update code. Lucene scales incredibly simply (since it's usually just a bunch of files sitting on a hard drive). I honestly can't see how a huge deployment would make it difficult to keep everything in sync. Maybe on the database side of life, but not with Lucene itself. Well, I’ll admit my experience with that kind of system is limited, but I know of other people who do have the experience and they say that it’s often *easier* to keep the Lucene bits in sync than the DBMS.

If you’re having difficulties keeping Lucene in sync with your database (or visa versa), you may want to take a hard look at how you’re doing things. Not just the underlying search-vs-database, but how you’re implementing them. As I mentioned, a few ORMs can keep search indexes and databases in sync automatically. Have you tried this approach? Even if you haven’t, a reasonable DAO should be able to take care of most of the pain by centralizing the Lucene update code.

Lucene scales incredibly simply (since it’s usually just a bunch of files sitting on a hard drive). I honestly can’t see how a huge deployment would make it difficult to keep everything in sync. Maybe on the database side of life, but not with Lucene itself.

]]> By: You're wrong http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2402 You're wrong Fri, 12 Oct 2007 08:19:15 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2402 You are obviously using a relatively simple database + index configuration. One DB and one index ? Well, the world is bigger than that. When you have data *that changes frequently* spread across several hundreds of servers, some of it replicated in different clusters, keeping the Lucene index(es) *up to date and consistent* becomes a huge task. I know because I'm doing it right now and it sucks. Keeping the full text search engine outside of the database is the wrong way and market requirements will force RDBMS manufacturers to do a better job at integrating them while preserving performance and scalability. You are obviously using a relatively simple database + index configuration. One DB and one index ?

Well, the world is bigger than that. When you have data *that changes frequently* spread across several hundreds of servers, some of it replicated in different clusters, keeping the Lucene index(es) *up to date and consistent* becomes a huge task. I know because I’m doing it right now and it sucks.

Keeping the full text search engine outside of the database is the wrong way and market requirements will force RDBMS manufacturers to do a better job at integrating them while preserving performance and scalability.

]]> By: owen http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2401 owen Thu, 11 Oct 2007 19:33:15 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2401 use the best tool for the job. and sometimes this means more than one tool. unless the day comes when I can write an entire program in my RDMS, lol THEN I'll be happy and singletonitized! use the best tool for the job. and sometimes this means more than one tool. unless the day comes when I can write an entire program in my RDMS, lol THEN I’ll be happy and singletonitized!

]]> By: The How-To Geek http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea/comment-page-1#comment-2400 The How-To Geek Thu, 11 Oct 2007 12:42:14 +0000 http://www.codecommit.com/blog/java/is-a-separate-text-search-engine-a-bad-idea#comment-2400 Using a lucene backend is great for a search engine, or even a site where the navigation involves browsing through mostly static content. It's not strictly limited to searching, I know of a lot of shopping sites that are built on lucene. It's pretty much useless for a site or section that needs transactional support, for instance you wouldn't use it for a shopping cart page because it's just not made for that type of thing. The index building functions are far less performant than you'd think, and you really require transactions for that type of data. I think that you have to design an application so that you can "plug in" a new search engine when MySQL fulltext starts to break down. For small applications MySQL will work perfectly (like this blog), but once you start to have a large volume of searching going on it simply can't handle the load. That's where it would be great to just "plug" a new search engine in that could utilize lucene. Using a lucene backend is great for a search engine, or even a site where the navigation involves browsing through mostly static content. It’s not strictly limited to searching, I know of a lot of shopping sites that are built on lucene.

It’s pretty much useless for a site or section that needs transactional support, for instance you wouldn’t use it for a shopping cart page because it’s just not made for that type of thing. The index building functions are far less performant than you’d think, and you really require transactions for that type of data.

I think that you have to design an application so that you can “plug in” a new search engine when MySQL fulltext starts to break down. For small applications MySQL will work perfectly (like this blog), but once you start to have a large volume of searching going on it simply can’t handle the load.

That’s where it would be great to just “plug” a new search engine in that could utilize lucene.

]]>