So, before I start into my review, let me first say congratulations to Sean, Robin, Ted, Ellen and Manning for producing such an excellent product.   The simplest praise I can give it is to put it on the same level as one of the best intro to technology books I know:  Lucene In Action.  In other words, it sets the standard by which all other Mahout books will be judged.

As for the actual book, it is broken down into 3 sections, which I like to call the “three C’s”:

1. Collaborative Filtering
2. Clustering
3. Classification

So, without further ado, let’s take a deeper look at the book in this context of the three C’s.

Collaborative Filtering

Collaborative Filtering is by far one of the most popular parts of Mahout, being used in places like Amazon and Foursquare and this section of the book, via 5 chapters,  walks you nicely through both the concepts and the practical aspects of collaborative filtering.   Chapter 2 starts by getting you up and running using the GroupLens dataset for movie recommendations.  For those unfamiliar with collaborative filtering, this makes for a nice entrance into the subject with data everyone can relate to easily.  Chapter 3 then discusses how to best model your data, while chapter 4 looks at the mechanics of actually generating recommendations from the data.

Chapters 5 and 6 then discuss the ins and outs of taking a recommendation engine into production, including details on how to scale it out using Apache Hadoop.  I found the explanation of the Hadoop based co-occurrence process (via RecommenderJob) especially useful, as I recently just committed MAHOUT-798, which uses it to build an example recommendation system based off of user interaction with email.  In fact, I relied heavily on all of the concepts in this part of the book, as I first had to extract and clean the data, then properly model it before finally running the recommendation task on EC2.

When I first got access to the MEAP for this book (quite some time ago), I did not have a lot of background in collaborative filtering and these chapters really helped fill in the practical details for me as well as provided a good foundation for the theoretical aspects behind collab. filtering.  I think this will serve others well who are looking to get started with collaborative filtering as well.

Clustering

Similar to collaborative filtering, the clustering section starts off by introducing the basic concepts and then quickly gets you up and running with an example clustering run.  Chapter 8 then gets into how best to do feature selection for clustering.  Feature selection is often one of the keys to successful clustering, so be sure to make sure you have a good grasp on the contents of the chapter before moving ahead into chapter 9, which gets into some of Mahout’s clustering algorithms.  That chapter primarily focuses on K-Means and Dirichlet, but also covers a few others.  Note, Mahout actually has a few other algorithms for clustering then the ones described, like spectral, canopy, meanshift and minhash.  Of course, some of these were added later in the book cycle, so it is hard to complain that they weren’t incorporated. Chapter 10 then covers, in my experience, one of the harder aspects of clustering, namely how to evaluate the results.  This chapter is a little bit thin, but it seems the overall field is the same, so this is not a put down on the chapter!  There simply isn’t a lot of great tools available for evaluating clustering.

Chapter 11 then adds some meat onto the bones of taking clustering to producti0n, including information on leveraging clustering in a Hadoop cluster.  Chapter 12 adds some nice concreteness to the sections by looking at clustering of real data sets from Twitter, Last.fm and Stack Overflow.  For those looking to kick the tires with some real data, be sure to check out that chapter.

Classification

Classification is very popular these days both in search and beyond, so it is great to see this set of chapters covering the topic so well in practical, accessible terms.  As you would expect, the first chapter (13) gets you up and running as well as introduces the concepts of classification.  This chapter has a great explanation of how classification works and a typical workflow for building a classifier.

Chapter 14 then delves into the details of actually training a classifier using Mahout’s Stochastic Gradient Descent algorithm as well as it’s Bayesian classifier.

The next chapter then takes a look at how best to evaluate a classifier as well as some insight into what happens when a classifier goes bad.  Be sure to check this out, as you will no doubt run into many of the issues covered.  As an aside, I couldn’t help thinking of the classic “Far Side” cartoon to the right upon reading that section heading.The penultimate classification chapter digs into the practical aspects of deploying a classifier in production, including details on working through your scale and speed requirements.  It finishes off with an example Apache Thrift based server which some may find as a useful starting point for their applications.  Finally, Mahout in Action finishes off with a Case Study of how Shop It To Me uses a Mahout classifier to provide recommendations of offers to customers.  As with any technical book, it is great to have some concrete discussion of how this stuff functions in the wild.

What’s Missing (i.e. When’s the 2nd edition coming out?)

Mahout has a number of other interesting things that are in various stages of development like frequent patternset mining, Singular Value Decomposition (feature reduction), evolutionary programming, integration libraries for input/output as well as tools for storing data in Cassandra and Mongo.  Since Mahout is developing pretty quickly, the lack of this being in the book is no fault of the authors, I’m just putting it up here so that people are aware that Mahout has more to offer, even if the three “C’s” are the most popular.

All in all, Mahout in Action is an excellent introduction to the project.  Naturally I’m biased, but, pun intended, I highly recommend the book!

For instance, for testing memory, it’s pretty easy to set up a series of tests that start with a small heap size and successively grow it until no Out Of Memory Errors (OOME) occur. Then, to be on the safe side, add 1 GB of memory to the heap.  It works for the large majority of people. Ironically, for Solr at least, this usually ends up with a heap size somewhere between 6-12 GBs for a system doing “consumer search” with faceting, etc. and reasonably sized caches on an index in the 10-50 million docs range. Sure, there are systems that go beyond this or are significantly less (I just saw one the other day that has around 200M docs in less than 3 GB of RAM while handling decent search load), but the 6-12 GB seems to be a nice sweet spot for the application and the JVM, especially when it comes to garbage collection, while still giving the operating system enough room to do it’s job.  Too much heap and garbage may pile up and give you ohmygodstoptheworld full garbage collections at some point in the future.  Too little heap and you get the dreaded OOME.  Also too much heap relative to total RAM and you choke off the OS.  Besides, that range also has a nice business/operations side effect in that 16 GBs of RAM has a nice performance/cost benefit for many people.

Recently, however, I thought it would be good to get beyond the inherent hand waving above and attempt to come up with a theoretical (with a little bit of empiricism thrown in) model for estimating memory usage and disk space.   After a few discussions on IRC with McCandless and others, I put together a DRAFT Excel spreadsheet that allows people to model both memory and disk space (based on the formula in Lucene in Action 2nd ed. – LIA2), after filling in some assumptions about their applications (I put in defaults.)   First a few caveats:

1. This is just an estimate, don’t construe it for what you are actually seeing in your system.
2. It is a DRAFT.  It is likely missing a few things, but I am putting it up here and in Subversion as a means to gather feedback.  I reserve the right to have messed up the calculations.
3. I feel the values might be a little bit low for the memory estimator, especially the Lucene section.
4. It’s only good for trunk.  I don’t think it will be right for 3.3 or 3.4.
5. The goal is to try to establish a model for the “high water mark” of memory and disk, not necessarily the typical case.
6. It inherently assumes you are searching and indexing on the same machine, which is often not the case.
7. There are still a couple of TODOs in the model.  More to come later.

As for using the memory estimator, the primary things to fill in are the number of documents, number of unique terms and information on sorting and indexed fields, but you can also mess with all of the other assumptions.  For Solr, there are entries for estimating cache memory usage.  Keep in mind that the assumption for caching is that they are full, which often is not the case and not even feasible.  For instance, your system may only ever employ 5 or 6 different filters.

The disk space estimator is much more straightforward and based on LIA2′s fairly simple formula of:

disk space used(original) = 1/3 original for each indexed field + 1 * original for stored + 2 * original per field with term vectors

It will be interesting to see how some of the new flexible indexing capabilities in trunk effect the results of this equation.  Also note, I’ve seen some applications where the size of the indexed fields is as low as 20%.

Hopefully, people will find this useful as well as enhance it and fix any bugs in it.  In other words, feedback is welcome.  As with any model like this, YMMV!

During a past ecommerce webinar with Brian Doll of Sheetmusicplus.com, I posted a checklist of items that are commonly occurring in many ecommerce applications and then I waved my hands, due to time constraints, and said Solr (and now LucidWorks) can do almost all of them out of the box and left the rest as an exercise for the reader.  (Note, the slides are available here.  Registration required.)  Well, now I have some time, so let me fill in the blanks with some more concrete examples about how to do this.

Setup

For this example, I am using real estate data freely available from the NYC government.  The reason I am interested in this data is that it is:

1. Free.
2. It has product-like data in it, as in: name, description, a bunch of metadata and price
3. It’s mostly real (I embellished it with descriptions and a few other pieces and filled in some missing pieces of data, see the Indexer class in the source code.)  In fact, it’s so real, that when setting up the app, one quickly sees how noisy the data is in terms of things like missing values, etc.  For instance, 1804 records don’t have the year built specified.

I have setup a Solr schema for this data as well as some tools for indexing the data.    To run the demo, you will need:

1. Java 1.6
2. Ant 1.7.X
3. Download ecommerce.zip (73 MB)

Once you have the prerequisites in place, take the following steps:

1. Unzip the ecommerce.zip file into the directory of your choice
2. cd lucid_ecom
3. In a separate terminal window: cd solr
   1. java -jar start.jar (just as if you were running the Solr tutorial.  Note, I am running a relatively recent version of the Solr 3.x branch)
4. Point your web browser at http://localhost:8983/solr/nyc and take a moment to familiarize yourself with the interface.

Once you have completed step 4, you should see something like:

(NOTE: I’m not a graphic designer.  I tried to create a reasonable UI w/o spending a ton of time on every last piece of it.  Also, I used the VelocityResponseWriter built into Solr.  It’s nice for prototyping, but it “ain’t” for production use.)

A pre-built index is included in the Zip file, but if you wish to index it yourself, run:

1. ant delete-all (deletes the existing content)
2. ant index

With the working application in place, let’s take a look at how to implement the various checklist items.

Implementing the Checklist

I’ve broken out each checklist item below and will cover each of them in more detail in the following subsections.

Keyword search

There really isn’t much to be said here other than Solr has built in support for querying in all the “usual” ways that one would expect out of a search engine.  Keywords, phrases, wildcards, fielded search and much, much more.  For example, try:

1. http://localhost:8983/solr/nyc?q=tottenville or just type tottenville in the search box.
2. http://localhost:8983/solr/nyc?q=5+bedrooms+%22Staten+Island%22 (5 bedrooms “Staten Island”)
3. http://localhost:8983/solr/nyc?q=5+bedrooms+borough_display%3ABro* (5 bedrooms borough_display:Bro* — Should match all 5 bedrooms in either the Bronx or Brooklyn)

Take some time and try out your own queries.  In our example, we are using the extended Dismax Query Parser, in case you want to learn more about how it works.

High Quality relevance (precision @ < 10)

In many search applications, and ecommerce is no exception, users often abandon searches when the first page of results (often the top 10) are not relevant to their query.  Thus, it is important that a search engine return good results on the first page.  While some guidance (more on this in the coming sections) can help alleviate the abandonment problem, a strong first showing is often the quickest way to more clickthroughs.  Since Solr utilizes Lucene, which implements an industry standard vector space approach to search, results are often quite good out of the box.  Nevertheless, many ecommerce applications may need one or more of the tools that Solr/Lucene provide out of the box to tweak relevance, such as:

1. Document, field, token boosting (i.e. matches in the title field are more important than matches in the description.)
2. Query term boosting (provide weights for different terms, such as synonyms.)
3. Disjunction Maximum Query scoring (aka the “dismax” parser or the extended dismax parser) for dealing with cross field matches.
4. Automatic phrase generation from multiword queries even when the user did not explicitly quote the keywords.
5. The ability to override low-level scoring information such as term frequency, document frequency, document length normalization and coordination factors.
6. Function queries (more later) to allow values in fields (such as price) to be factors in scoring.
7. Editorial Boosting/Sponsored Results (in Solr-speak it’s called the QueryElevationComponent — more later) to place specific results at the top.

Relevance tuning is a complex subject and one that is best viewed in the light of your data.  In summary, make sure you are making decisions about relevancy based on the big picture and try to avoid any local minima (i.e. tuning a specific query to the detriment of breaking lots of other queries.)  In other words, make sure your top money making queries aren’t effected by you “fixing” a one or two bad queries.  To learn more, see my articles on Improving Findability and Debugging Relevance.  With the basics out of the way, it’s time to take a look at faceting and discovery tools

Faceting/Discovery

One of Solr’s most appealing features is its out of the box support for faceting (sometimes called navigators, parametric search, guided navigation) in a number of different ways (see http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Faceted-Search-Solr for a primer.  Also see http://wiki.apache.org/solr/SimpleFacetParameters)  In the example application, the left hand nav area shows facets for things like borough (field based faceting), sale price (numeric range faceting), sale date (date range faceting) and pet friendly (facet by query).   Solr also supports “multi-select” faceting (see http://search.lucidimagination.com for an example.)  And, while there isn’t support for true hierarchical faceting in Solr yet, there are ways to achieve it through intelligent modeling of your tokens.  Last, but not least, you may find https://issues.apache.org/jira/browse/SOLR-792 useful for doing grouped faceting (color: red, size: large).

Additionally, helping customers discover items of interest goes well beyond facets.  Features like Did you mean, Related Items/Searches, Collaborative Filtering/Recommenders (see Mahout for an open source solution), Auto Suggest and others can go a long way in increasing the user’s ability to purchase items from your store.  Many of these features I’ll cover below.

Flexible language analysis tools

Multilingual support

Solr contains support for most of the commonly spoken languages in the world, including English, Chinese, French, Spanish, Korean, German, Thai and many more.  Lucene and Solr are also UNICODE compliant.

Frequent Incremental Updates

Lucene, and thus, Solr has supported incremental updates from it’s inception without the need to re-index the whole collection.  It is also very fast at making new documents available for search.  Additionally, with the combination of recent and upcoming work in Lucene, real time search should be available soon.  The one piece that is still missing is individual field update, but for certain types of fields (ratings, for instance), there may be easy workarounds.

Ratings and Reviews

In working with many ecommerce customers on Solr, there are usually questions around how to incorporate ratings and reviews into search results without skewing results or introducing too much noise.   On the ratings side, app developers often want to incorporate the aggregate rating of an item as a boost factor in the overall score.  I will discuss how to do this in detail in the section titled Editorial Relevance Controls below.  Meanwhile, on the review side, it is often the case that too much noise is introduced by including reviews “on par” with matches in the product title or description.  For instance, if I’m selling “Widget X” and a review for a different product says something like “You should also check out Widget X”, bringing back a match on that second product really isn’t all that useful for a customer searching for “Widget X”.   To deal with this noise, people often take a couple of different approaches:

1. They weight review matches lower than product matches via boosting (either at query time or indexing time)
2. They only search reviews if they don’t feel they have high quality matches for the main product search

You could also do some type of post processing analysis (NLP) of the review to see if it is on topic, but this approach likely isn’t viable for most people in most situations due to the processing power and accuracy of such a solution.  As for #2 above, see my post on Fake and Invisible Queries for more insight.

Auto-suggest

Auto suggest (aka auto complete) is one of the cheapest (in terms of development costs) mechanisms available for enhancing the chance that users find what they are looking for.  I’ve heard of vendors adding auto-suggest and having it add millions to their bottom line.  Simply by providing a drop down list of ways of completing what a user has typed so far an application can do a number of things:

1. Reduce spelling errors thus leading to lower frustration and better results sooner rather than later
2. Seed the user with items that they may want but weren’t explicitly looking for.  After all, an intelligent auto-suggest box can very easily not only give completions, but it can also hook in related items too.
3. Short-circuit search all together and go directly to a landing page for a specific search

For the demo, I implemented auto-suggest using SOLR-1316, which should be committed to trunk soon.  Note, also, there are other ways of doing auto-suggest, too, including using the TermsComponent and Faceting.  Here are the steps I went through to make auto-suggest work:

1. Applied the SOLR-1316 patch to the 3.x branch.  This required a minor tweak to the HighFreqDictionary.java file.  See patch below
2. Add the necessary piece to the solrconfig.xml.  See the /autosuggest SearchComponent in the solrconfig.xml in the appendix.
3. Decide what fields to use in building the auto-suggest index (see schema.xml).  I then “copy fielded” these into a field named suggest.  Note that I used a non-stemming analyzer.  I also used Solr’s word-based n-gram filter with a shingle base of 5 so as to give phrase suggestions too.  Note, this is intended for demonstration purposes, as you may wish to not use shingles and append terms as the user types or you may want to use a different value for n.  Also note, I did not spend much time at all on evaluating what went into the suggest field that is used as a source.  You will want to validate it and make sure it is aligned with your business goals.
4. Build the auto-suggest data structures via the Spell Checker build command (see the next section)
5. Modified the jQuery script that is in the Solr VelocityResponseWriter example to use the SOLR-1316 output instead of the TermsComponent output.  See the autocomplete.vm file for details on the Javascript.  See the next section on Did You Mean on how to make requests to the the auto-suggest component, as it uses the same mechanism as the spell checker.

Hopefully, from here you will have enough information to build you your auto-suggest capabilities.  If not, see our search site for more info, including alternate approaches to the SOLR-1316 patch.

Did You Mean?

Just like auto-suggest, spell checking can be helpful to users in finding what they are looking for, especially given the propensity of manufacturers/product designers to use incorrectly spelled words in their product name in order to better “brand” the product.  Good spell checking goes beyond merely hooking up a dictionary of terms, it is also quite important to know when to suggest a term and when not suggest a term.  Lucene/Solr has the basics of setting up spell checking covered via the SpellCheckComponent, but a good spell checking application will need to go beyond merely setting up the component in order to achieve good results.  First things first, however, let’s take a look at getting spell checking setup and then we can examine what is needed to make it better.

First, we need to configure the SpellCheckComponent in the solrconfig.xml file.  There is an example of this in the Solr tutorial example, from which I changed the distance measure from the Levenstein edit distance to the Jaro-Winkler distance.  The reason I did this is based on past experience that users tend to misspell words towards the end of the word and not the beginning, which the Jaro-Winkler distance accounts for.  My configuration looks like:

<searchComponent name="spellcheck">
 <str name="queryAnalyzerFieldType">textSpell</str>
 <lst name="spellchecker">
 <str name="name">default</str>
 <str name="field">spell</str>
 <str name="spellcheckIndexDir">./spellchecker</str>
 <str name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
 </lst>
<!-- ... -->
 </searchComponent>

The whole point of a SearchComponent such as the SpellCheckComponent is to hook it into the main Solr request processing instead of having to make a separate call.  Thus, I hooked the SpellCheckComponent into the /nyc RequestHandler so that all queries that are submitted to the “main” RequestHandler will also be spell checked.  Once the configuration is setup, the spelling index must be built (and maintained.)  This is handled by issuing an &spellcheck.build=true command to the spell checker, as in:

http://localhost:8983/solr/autosuggest?q=man&spellcheck=true&wt=xml&rows=0&indent=true&spellcheck.build=true

(Note, the &q param can be anything.)

Once the configuration is hooked up and the spell checking data structure is built, the last piece is to hook it into the UI.  (Note, I setup the solrconfig.xml to automatically do spell checking on every query request.)  To hook into the UI, I co-opted the suggest.vm file and spruced it up a bit to provide links, etc.  Other than that, it is exactly the same, since both are just different implementations of spell checking.

See the Solr wiki on the SpellCheckComponent for more information.

Related Searches/Items

In many ecommerce applications, stores position related items next to a particular item so as to inspire the user to either buy an additional item or offer an alternative.  Naturally, the “relation” is determined by the store and might take on a variety of forms, such as: accessories, enhanced versions, cheaper versions, alternatives from different manufacturers or popular items based on other users.  Similarly, a store may wish to give users not only suggestions and spelling corrections, but they may also want to give users alternative search terms or other popular searches.  For instance, if a user searches for TVs, a store may want to suggest they search for “LCD TVs” or “HD TVs”, etc.

When it comes to related items, many Solr users rely on either hand-crafting a second query (given an original query and a particular item) by using the original terms of the query and some of the terms that describe the item.  For instance, an application might use the category of the item plus some of the keywords for that item to then craft the query, submit it to Solr and then display the first few results.  This approach can also be done automatically using Solr’s built in More Like This (MLT) capability, but you may need to do some tuning to get the results you desire.  For the sake of the example, I incorporated MLT into the application.  You can see it on the left hand side, just below the map, under the “Similar Properties” heading.  The configuration of MLT was done in the solrconfig.xml file as part of the /nyc RequestHandler.  Note, in a typical application you may not wish to generate MLT results for a search query, but instead only provide them once a user chooses a particular document, as MLT can add a fair amount of overhead to the process.  Other Solr applications will often calculate related items off line or through some type of collaborative filtering approach (see Apache Mahout’s recommender capability for an open source library to do this) and either add the information to the document and re-index or integrate it at the application level.  In these cases, it’s not hard to integrate, but it is beyond the scope of this article.

As for the functionality to add related searches, there is not currently support built into Solr, but there is a JIRA issue open to track the idea.  Related searches can often be determined through a combination of log analysis (look for patterns in a user session) and synonyms or via collaborative filtering/recommenders.  Also, have a look at Mahout’s Frequent Pattern Mining capabilities.  One could also index the queries into another index (Solr core) and simply issue fuzzy queries to it.

Editorial Relevance Controls

Administration

Administration means many things to many people.  To the IT department, it means easy setup, configuration, monitoring, maintenance, scalability, fault tolerance, etc. while to the business user it means tools for manipulating results, reporting search statistics and following through on business goals.  While the latter is important, I am going to focus on the IT dept. needs for the sake of this article.  Solr is very easy for an IT person to get setup and have a baseline configuration in place.   I’ve seen customers (without my help) be up and running and searching their data in non-trivial ways in as little as 30 minutes, sometimes less.  As for monitoring, Solr comes with web pages that report status as well as JMX integration.  I’ve also seen Solr integrated nicely with Nagios, Cactus and other tools.  Lucid Imagination also partners with New Relic to offer Solr specific monitoring tools.

As for the big questions about scalability and fault tolerance, the answer is an unequivocal yes.  High traffic ecommerce sites like Zappos, Netflix, CNET, AOL and many others use Solr to server their search needs.  Solr can be setup to both handle large indexes and high query volumes.  For more information on how to do this, see Mark Miller’s excellent article on scaling Solr.

Recommendations (See Mahout)

For both online and offline recommendation calculations, see the Apache Mahout project, which has an excellent collaborative filtering library.  While integration with Solr does not yet exist, Mahout does expose web services (as well as Java APIs) for its recommender engine, so it is feasible to integrate it within an application.

Analytics and other Business Tools

Analytics are probably Solr’s weakest area, but that being said, we find that many customers already have platforms in place (like Omniture) that they can easily integrate Solr into.  This often saves business users from having to learn yet another tool.  As for other business tools, Solr likely does not have them (for instance, merchandising tools), but again, many people find it straightforward to integrate Solr into existing tools.  Also, this is an area that LucidWorks, with it’s administrative UI really can help.  It has screens and tools for doing log analysis and seeing what popular queries are, as well as popular terms and zero result values.

Looking Forward

Solr is a very popular and capable search engine for ecommerce and, looking forward, it is only getting better.  With a focus on greater features (spatial search, for instance), the latest Lucene and easier scalability, the next version of Solr promises to be even better.

Appendix A

Items needed here: schema, solrconfig, SOLR-1316 3.x branch patch

What are shingles good for?  Many people use them to create “pseudo-phrases” during the indexing process since the shingle ends up being a single token, which is then subject to the normal TF-IDF scoring that is used in Lucene.  In many cases, searching for phrases yields relevance improvements, but finding phrases at query-time can be more expensive than normal term queries, so people sometimes try to get ahead of the game and use shingles.

If you want to see shingles in action and compare them to n-grams, add the following field types to a sample Solr schema:

<fieldtype name="shingle">
 <analyzer>
 <tokenizer class="solr.WhitespaceTokenizerFactory"/>
 <filter class="solr.ShingleFilterFactory" minShingleSize="2" maxShingleSize="5"/>
 </analyzer>
 </fieldtype>
 <fieldtype name="ngram">
 <analyzer>
 <tokenizer class="solr.NGramTokenizerFactory" maxGramSize="5" minGramSize="2"/>
 </analyzer>
 </fieldtype>

Next start up your Solr instance and browse to http://localhost:8983/solr/admin/analysis.jsp and do the following steps:

1. In the Field row, choose “Type” from the dropdown and enter shingle in the text area
2. In the Field Value section, choose Verbose output and enter “The quick red fox jumped over the lazy brown dogs”.
3. Hit Submit.  You should see something like:

Output of Solr's Analysis page

As you can see, there are multiple tokens put out for each position, many of which contain multiple words as a single token.

Next, try the same sentence, but switch from “shingle” to “ngram” for the field type.  This time you should see the words split up into character groups.

For more info, see http://en.wikipedia.org/wiki/N-gram.  Note, you might also find Google Book’s ngram viewer interesting too: http://ngrams.googlelabs.com/

First off, hats off to Lulu for the excellent hosting, coordination and marketing of the event.  You could definitely see the evidence of Lulu’s “Be Remarkable” philosophy in the event. I’d say we had roughly 30-40 people for the first time event, with a good mix of developers, technical managers and a few recruiters.  There was even a “competitor” from an unnamed proprietary vendor present.  On the application front, there was a large mix of usages represented: ecommerce, publishing, video search, procurement, biopharma, etc.

After some socialization, we kicked off the night with Lulu CEO Bob Young, who gave a short intro to Lulu as well as a warm welcome to all.  Next up, I gave a talk (slides) on what’s coming in Lucene/Solr 3.x and beyond as well as answered some questions about features and functionality.  After me, Tarun Jain of The ABB Group, one of Lucid’s first customers and the world’s largest producer of industrial robots as well as a global leader in power and industrial automation with revenues around $33B USD, gave a presentation titled “Extreme Faceting Using Solr” (slides) on their move from a legacy proprietary vendor to Solr for searching all of their customer facing (and internal) product catalog (420K SKUs with 20+ million attributes and over 6M hits per month).   After setting the stage about the content to be searched and faceted, Tarun detailed how they went from wanting to “do everything in the DB” to doing nearly everything in Solr because it was that easy.  Moreover, slide 8 details the comparison they did between Solr and a very large proprietary search vendor (one of the so called top 3).  Here are the bullet points:

1. Stress test results in Proof of concept
   1. SOLR 35 req/sec vs 2 req/sec
   2. Average response times 200 ms vs 1-7 secs
   3. CPU usage 2-3% vs 100%
2. Sadly matchup was not even close (at least for the scenarios we tested for)
3. Conclusion .. Performance of SOLR is inversely proportional to the cost
4. Winner – SOLR by a KO

After Tarun’s talk, Paul Oakes from Lulu gave an excellent technical presentation (slides) on implementing auto-suggest in Solr using jQuery.  Just for grins, he also showed how trivial it was to add Google’s much hyped “Instant” search capability to Solr as well simply by making an extra jQuery call.  Naturally, the real work behind “Instant” is in capacity planning at scale, not in the programming of a few lines of Javascript.

As for the RTP meetup in general, I would suspect we will try to meet once a quarter, but maybe more often if the group so desires.

All in all, an excellent night, in my opinion.  Best of all it was a “home” event for me, so I didn’t have to fly anywhere or bum a ride back to a hotel!

As background, Solr provides a schema that describes the Fields and Field Types (FT) that are used by an application.  Field Types describe how Solr should handle the information contained in a Field.  For instance, the integer FT tells Solr to treat the contents of any Field of type integer as, you guessed it, an integer.  By integer here, I mean, good old fashioned Java ints.  Solr provides other FTs like long, double, float, string, date, as well as Text (which can be associated with Lucene’s analysis process).  Additionally, Solr provides several “sortable” FTs such as sint, slong, sdouble and sfloat.  Therein lies the confusion.  I think what happens is developers hear the word “sortable” and think they should use the sortable FT for any field they want to sort results by.  However, there is some subtlety here.  Namely, “sortable” FTs manipulate the content so that the lexicographic order is the same as the numeric order for use during search.  Sortables are thus really meant to be used when doing things like range queries (i.e. [price:2 TO 100]) and not for sorting as it relates to returning results.  Due to these required changes, sortables take up more space in the index (and in memory) then their non-sortable compadres.

What’s this got to do with schema design?  Well, this client had three fields, all defined as sortable integer FTs, as in:

1. fieldOriginal  -  The source of the content.  This was the main field used for sorting
2. fieldSearch – Copy field of Original, but rounded to the nearest 100.  This was the main field for searching.
3. fieldFacet – Copy field of Original, but rounded based on a percentage of the original value so as to provide a sliding scale for faceting.  This was the main field used for faceting.

In this case, the client was using the Original for sorting, Search for searching, and Facet for faceting.  They were not doing any range queries, so they did not need fieldSearch to be “sortable”.  Furthermore, the Original field had over 1 million unique terms, so sorting on it was taking up a good chunk of memory and disk space.  The other two fields were smaller, so the cost of sortables was not that big of a deal.  Finally, this field “pattern” was replicated for several other fields as well, some of which also had a significant number of unique terms.

Thus, simply by changing the Fields to use integers where appropriate, we significantly reduced the memory footprint and the disk space required in this client application.

So, as is always the case, play close attention to your schema design.  While the Solr example schema is pretty good out of the box, you shouldn’t just take it as gospel, either.  Spend some time thinking about your needs during design and it will likely save you much time later when debugging and testing your application.

**UPDATE**:  Note, making these changes will require you to re-index.

I’ve posted a Lucene related event schedule on my blog for people who are interested.  Of particular note are the two days of pre-conference training on both Lucene and Solr.  These are shorter ApacheCon versions of our 3 day training classes.  Obviously, we can’t cover all the material that we do in our full courses, but they are a means for people who are attending the conference or who are in the vicinity to get some Solr and Lucene training.  Of course, if you can’t make it to ApacheCon, we’d be happy to discuss alternative training opportunities.  In fact, last year, I did an on-site training at a company in Europe right around conference time.  Just drop us a line at training@lucidimagination.com and we can see what works out.