If you are in the NYC area on November 18, please join us!

To register, please go to: http://www.javasig.com/meeting/show/35

When: November 18, 2010 6:30PM
Where: Credit Suisse – 11 Madison Avenue, New York, NY 10010

About the NYJavaSIG: The New York Java Special Interest Group (NYJavaSIG) was founded in 1995 by Frank Greco. Today there are well over 6,500 members in the NYJavaSIG, and attendance at the monthly meetings averages over 150 attendees. The NYJavaSIG has hosted meetings for developers from New York, New Jersey and Connecticut, and has brought together members of the local Java community through its website and monthly get-togethers to share tips, techniques, knowledge, and experience.

This time around, the data set I chose was Data.gov’s catalog of datasets, which fit with the ApacheCon open source aura, and Lucid Imagination’s support of Open Source for America, which helps to advocate for open source in the US Federal Government.  The prototype built includes faceting browsing, query term suggest, hit highlighting, result clustering, spell checking, document detail, and a bonus Venn diagram visualization.

The prototype was built with these steps:

1. Install LucidWorks for Solr
2. Grab the Data.gov catalog CSV file
3. Iterate a bit with Solr’s CSV update handler (the funnest way to get data into Solr) and a little Solr schema tinkering
4. Adjusted the Solr configuration and UI templates to get a nice look and feel, adding in a document detail page and a Venn diagram visualization comparing query overlaps

Voilà (click the images for large view):

This isn’t the first time we’ve toyed with Data.gov data… earlier this year, Hoss demonstrated Solr’s stats component on another of Data.gov’s data sets.

My ApacheCon slides are published at Slideshare and embedded here:

All the code and instructions for running the entire prototype yourself can be found here: https://github.com/erikhatcher/solr-rapid-prototyping/tree/master/ApacheCon2010

First off, hats off to Lulu for the excellent hosting, coordination and marketing of the event.  You could definitely see the evidence of Lulu’s “Be Remarkable” philosophy in the event. I’d say we had roughly 30-40 people for the first time event, with a good mix of developers, technical managers and a few recruiters.  There was even a “competitor” from an unnamed proprietary vendor present.  On the application front, there was a large mix of usages represented: ecommerce, publishing, video search, procurement, biopharma, etc.

After some socialization, we kicked off the night with Lulu CEO Bob Young, who gave a short intro to Lulu as well as a warm welcome to all.  Next up, I gave a talk (slides) on what’s coming in Lucene/Solr 3.x and beyond as well as answered some questions about features and functionality.  After me, Tarun Jain of The ABB Group, one of Lucid’s first customers and the world’s largest producer of industrial robots as well as a global leader in power and industrial automation with revenues around $33B USD, gave a presentation titled “Extreme Faceting Using Solr” (slides) on their move from a legacy proprietary vendor to Solr for searching all of their customer facing (and internal) product catalog (420K SKUs with 20+ million attributes and over 6M hits per month).   After setting the stage about the content to be searched and faceted, Tarun detailed how they went from wanting to “do everything in the DB” to doing nearly everything in Solr because it was that easy.  Moreover, slide 8 details the comparison they did between Solr and a very large proprietary search vendor (one of the so called top 3).  Here are the bullet points:

1. Stress test results in Proof of concept
   1. SOLR 35 req/sec vs 2 req/sec
   2. Average response times 200 ms vs 1-7 secs
   3. CPU usage 2-3% vs 100%
2. Sadly matchup was not even close (at least for the scenarios we tested for)
3. Conclusion .. Performance of SOLR is inversely proportional to the cost
4. Winner – SOLR by a KO

After Tarun’s talk, Paul Oakes from Lulu gave an excellent technical presentation (slides) on implementing auto-suggest in Solr using jQuery.  Just for grins, he also showed how trivial it was to add Google’s much hyped “Instant” search capability to Solr as well simply by making an extra jQuery call.  Naturally, the real work behind “Instant” is in capacity planning at scale, not in the programming of a few lines of Javascript.

As for the RTP meetup in general, I would suspect we will try to meet once a quarter, but maybe more often if the group so desires.

All in all, an excellent night, in my opinion.  Best of all it was a “home” event for me, so I didn’t have to fly anywhere or bum a ride back to a hotel!

First, by choosing an open backbone like Lucene and Solr, you are free to plugin the best tool for the job; proprietary solutions often limit you to their own tools and their implementation.  Let’s face it, we can’t be good at everything, so it makes sense to be able to plug in the best of breed for something that isn’t a core competency.  For example, one could choose OpenNLP or Teragram or any other commercial vendor for these capabilities.  Solr, especially, makes it simple to plugin these capabilities through its well defined plugin architecture.  (By the way, for almost every capability out there in this realm, there is an open source alternative that warrants investigation.)

Second, intelligent search–in other words, search that goes beyond simple keyword capabilities–is the leading edge of the field and is being adopted in more and more products, just as Dr. Schabes recommends.  Whether it is intelligent query parsing, better faceting and discovery capabilities or integration with natural language processing (NLP) tools for NER (Named Entity Recognition), sentiment analysis and relationship discovery, the companies making a difference in search are those that intelligently bring together a variety of approaches to solve the problem at hand.  I believe Lucene,  Solr and open source are uniquely positioned to fuel intelligent search because they drive down the cost of experimentation simply because it takes effort to get this stuff right, much of it due to the need to understand your domain and how to translate it into a good user experience.  Furthermore, open source lets you cost effectively fill in your infrastructure and conserve your precious resources for your core competencies.  Why would you pay millions of dollars for a search engine that implements a vector space retrieval model (which most of the commercial vendors do) when you can get the same thing from Lucene for free?  If you suspect that you think Lucene isn’t as good, think again; there’s a reason it is used at the likes of Apple, AOL, Comcast, CNET, Viacom and thousands of others.  If you like bells and whistles and knowing there is a company behind your chosen solution, I’ll do you three better:  with Lucene and Solr you not only get 1) a company that offers support, training, professional services, and bells and whistles, you also get 2) the very large Apache community of users as well who constantly use/test/fix/improve the software and 3) all of the source code,  completely unencumbered, so you are free to change it as you see fit.

Finally, you get to choose whether you even need a particular capability.  On more than one occasion, I have been involved in replacement of a proprietary search package so bloated with unused add-ons that the Solr installation, containing only the required functions, needed an index that was a mere fraction of the size of the proprietary solution, resulting in:

• Less hardware to achieve the same throughput
• Less operations costs — more hardware = more hardware failures
• faster indexing, faster queries, etc.

In short, Lucene and Solr offer a cost effective and fully capable mechanism for improving the efficiency of search along the lines of the approach Dr. Shabes recommends, giving you the freedom to choose based on your idea of what works, not someone else’s.

Apache Nutch 1.0 contains almost 200 resolved issues and improvements such as Solr Integration, new indexing framework and new scoring framework just to mention a few.

Nutch 1.0 is available from here.