These improvements are now committed to Lucene’s trunk: you can use these models in tandem with all of Lucene’s features (boosts, slops, explanations, etc) and queries (term, phrase, spans, etc). A JIRA issue has been created to make it easy to use these models from Solr’s schema.xml.

Relevance ranking is the heart of the search engine, and I hope the additional models and flexibility will improve the user experience for Lucene: whether you’ve been frustrated with tuning TF/IDF weights and find an alternative model works better for your case, found it difficult to integrate custom logic that your application needs, or just want to experiment.

I’ll be giving a talk about how you can practically apply some of the upcoming Lucene 4 search features at Lucene Eurocon in October, and at the SFBay Apache Lucene/Solr Meetup later this month.

Some bullet points of the new scoring features:

• New ranking algorithms, in addition to Lucene’s Vector Space Model:
  • Okapi BM25 Model
  • Language Models
  • Divergence from Randomness Models
  • Information-based Models
• Added key statistics to the index format to support additional scoring models.
  • Term- and field-level statistics for collection frequencies and deriving averages.
  • Additional document-level statistics for computing normalization factors.
• Decoupled matching from ranking in Lucene’s core search classes:
  • Customize scoring without digging into the “guts”.
  • Customize explanations: essential for debugging relevance issues.
• Powerful low-level Similarity API, supporting advanced use cases:
  • Incorporate per-document values from Column Stride Fields into the score.
  • Use different scoring parameters or algorithms for different fields.
  • Fuse multiple scoring algorithms into a combined score.
• Convenient high-level SimilarityBase for everything else:
  • Write your own scoring function in one Java method.
  • Easy access to available index statistics.

For more information about this GSOC project, take a look at its wiki page

Circumstances have conspired to keep my away from this series longer then I had intended, So today I want to jump right in talking about improving the user experience by improving relevancy.

(If you are interested in following along at home, you can checkout the code from github. I’m starting at the blog_9 tag, and as the article progresses I’ll link to specific commits where I changed things, leading up to the blog_10 tag containing the end result of this article.)

Academic vs Practical

In Academia, people who study IR have historically discussed “relevancy” in terms of “Precision vs Recall” (If these terms aren’t familiar to you, then I highly suggest reading the link) but in my experience, those kinds of metrics are just the starting point. While users tend to care that your “Recall” is good (results shouldn’t be missing), “Precision” is is usually less important then “ordering” — Most users (understandably) want the “best” results to come first, and don’t care about the total number of results.

Defining the “best” results is where things get tricky. Once again, there are lots of great algorithms out there that academics debate the pros and cons of all the time, but frequently the best approach you can take to give you users the “best” results first isn’t to get a PhD in IR, it’s to “cheat” and bias the algorithms and apply “Domain Specific Knowledge” — But I’m getting ahead of myself, let’s start with a real example.

Poor Results in Our Domain

Every domain is different, and the key to providing a good search experience is making sure you really understand your domain, and how your users (and data) relate to it.

Lets look at a specific example with our ISFDB Data. One of the most famous Sci-Fi short stories ever written is Nightfall by Isaac Asimov, who later collaborated with Robert Silverberg to expand it into a novel. If a user (who knows they are searching the ISFDB) searched for the word “Nightfall” they would understandable expect one of those two titles to appear fairly high up in the list of results, but that’s now quite what they get with page #1 of our search as it’s configured right now…

1. Title: Nightfall INTERIORART – Author: Kolliker
2. Title: Nightfall: Body Snatchers ANTHOLOGY- Author: uncredited
3. Title: Cover: Nightfall One COVERART – Author: Ken Sequin
4. Title: Cover: Nightfall One COVERART – Author: Ken Sequin
5. Title: Nightfall SHORTFICTION – Author: Tom Chambers
6. Title: Glossary (Nightfall at Algemron) ESSAY – Author: uncredited
7. Title: Cover: Nightfall One COVERART – Author: Ken Sequin
8. Title: Cover: Nightfall Two COVERART – Author: Ken Sequin
9. Title: Nightfall POEM – Author: Susan A. Manchester
10. Title: Cover: Nightfall One COVERART – Author: Ken Sequin

These results aren’t terrible surprising since so far in this series we’ve put no work into relevancy tuning, we’re just searching a simple “catchall” field. Before we can improve the situation, it’s important to make sure we understand why we’re getting what we’re getting and why we’re not getting what we want.

Score Explanations

One of the most hard to understand features of Solr is “Score Explanation” — not because it’s hard to use, but because the output really assumes you understand the core underpinnings of Lucene/Solr scoring. When we enable debugging on our query we get a new “toggle explain” links for each result that let us see the score and a break down of how that score was computed — but that doesn’t let us compare with documents that aren’t on page #1. To do that, we use the explainOther option, and switch to the XML view since the velocity templates don’t currently display explainOther info. Now we can compare the explanations between the two docs we really hoped to find, and the top scoring result…

• TITLE_847094 (Nightfall INTERIORART)
  

  2.442217 = (MATCH) fieldWeight(catchall:nightfall in 274241), product of:
    1.0 = tf(termFreq(catchall:nightfall)=1)
    9.768868 = idf(docFreq=98, maxDocs=636658)
    0.25 = fieldNorm(field=catchall, doc=274241)
  

• TITLE_11852 (Nightfall NOVEL)
  

  1.7269082 = (MATCH) fieldWeight(catchall:nightfall in 11741), product of:
    1.4142135 = tf(termFreq(catchall:nightfall)=2)
    9.768868 = idf(docFreq=98, maxDocs=636658)
    0.125 = fieldNorm(field=catchall, doc=11741)
  

• TITLE_46434 (Nightfall SHORTFICTION)
  

  1.7269082 = (MATCH) fieldWeight(catchall:nightfall in 41784), product of:
    1.4142135 = tf(termFreq(catchall:nightfall)=2)
    9.768868 = idf(docFreq=98, maxDocs=636658)
    0.125 = fieldNorm(field=catchall, doc=41784)
  

The devil is in the differences, which I’ve put in bold. Without going into a lot of complicated explanation, the crux of the issue is that even though the documents we’re looking for match the word “nightfall” twice in the catchall field we’re searching (and the top scoring result only matches once) that is offset by the “fieldNorm” which reflects the fact that the catchall field is much longer for our “good” docs then for our “bad” docs.

Tweaking Our Scoring

This is one of those examples where academics doesn’t always match the reality of your domain. Typically when using the TF/IDF scoring model used in Lucene/Solr, you need a “length normalization” factor to offset the common case where a really long document inherently contains more words, so there is a statistical likely hood that the search terms may appear more times. In a nutshell: All other things being equal, shorter is better. This reasoning is generally sound, but the default implementation in Lucene/Solr can be a hinderence in a few common cases:

• A corpus full of really short documents – our ISFDB index isn’t full books, just a bunch of metadata fields
• A corpus where longer really is better – in the ISFDB data, more popular titles/authors tend to have more data, which means the catchall field is naturally longer.

There are some cool things we could do with tweaking the Similarity class to try and improve this, but the simplest thing to start with is to omitNorms on the catchall field to eliminate this factor from our scoring. With our new schema, we re-index and see some noticable changes…

1. Title: Nightfall NOVEL – Authors: Robert Silverberg, Isaac Asimov
2. Title: Nightfall and Other Stories COLLECTION – Author: Isaac Asimov
3. Title: Nightfall SHORTFICTION – Author: Isaac Asimov
4. Title: The Legend of Nightfall NOVEL – Author: Mickey Zucker Reichert
5. Title: Nightfall NOVEL – Author: John Farris
6. Title: The Road to Nightfall COLLECTION – Author: Robert Silverberg
7. Title: Road to Nightfall SHORTFICTION – Author: Robert Silverberg
8. Title: A Tiger at Nightfall SHORTFICTION – Author: Harlan Ellison
9. Title: Nightfall SHORTFICTION – Author: David Weber
10. Title: Nightfall SHORTFICTION – Author: Charles Stross

Domain Specific Biases

Omitting length norms has helped “level the field” for our docs, and in this one example it looks like a huge improvement at first glance, but that’s mainly a fluke. If you look at the score explanations now we get a lot of identical scores, and the final ordering is primarily because of the order they were indexed in.

This is where adding some Domain Specific Bias can be handy. If we review are schema, we see the views and annualviews fields which correspond to how many page views a given author/title has received (recently) on the ISFDB web site. By factoring these page view counts into our scoring, we provide some “Document Biasing” to ensure that documents which are more popular will “win” (ie: score higher) in the event of a tie on the basic relevancy score.

The most straightforward way to bias scoring is with the BoostQParser which will multiple the score of a query for each document against an arbitrary function (on that document). In it’s simplest form we can use it directly in our q param to multiple the scores by the simple sum of the two “views” fields: q={!boost b=sum(views,annualviews)}nightfall and now we get a much more interesting ordering…

1. Title: Nightfall SHORTFICTION – Author: Isaac Asimov
2. Title: Nightfall NOVEL – Authors: Robert Silverberg, Isaac Asimov
3. Title: Nightfall and Other Stories COLLECTION – Author: Isaac Asimov
4. Title: Nightfall SHORTFICTION – Author: Charles Stross
5. Title: The Return: Nightfall NOVEL – Author: L. J. Smith
6. Title: Nightfall SHORTFICTION – Author: Arthur C. Clarke
7. Title: Nightfall SHORTFICTION – Author: David Weber
8. Title: The Road to Nightfall COLLECTION – Author: Robert Silverberg
9. Title: The Legend of Nightfall NOVEL – Author: Mickey Zucker Reichert
10. Title: Nightfall Revisited ESSAY – Authors: Pat Murphy, Paul Doherty

This new ordering for the page #1 results is much more appropriate for the domain of the ISFDB, and represents a general rule of relevancy biasing: “Users unusually want to see the popular stuff.” However, users don’t usually want to have to type things like {!boost b=sum(views,annualviews)}... into the search box, so we need to encapsulate this into our config. It’s very easy to do this using Local Params, but unfortunately it does mean changing our “main” query param from q to something else.

We start by changing the defaults and invariants of our request handler so that our boost function is always used as the q param, but it uses a new qq param as the main query (whose score will be multiplied by the function). This works fine for our default query, but in order to be useful our UI also needs to be changed to know that the qq param is what is now used for the user input.

Conclusion (For Now)

And that wraps up this latest installment with the blog_10 tag. We’ve dramatically improved the user experience by tweaking our how our relevancy scores are computed based on some knowledge of our domain, particularly via Document Biases. In my next post, I hope to continue the topic of improving the user experience by using DisMax to add “Field Biases”.

I won’t be the first to point out that relevance for in-house search — i.e., without using Pagerank — is a harder nut to crack. How much harder? A recent study from Aberdeen Group, publicized this week in Information Week, provides the following stat:

At top performing companies [defined as "Best in Class", the top 20% of those surveyed], 67% of searches returned the most relevant results on the first search results page, while lower rated companies saw relevant results on the first page for only 42% of searches.

1 out of 3 searches at best don’t deliver the right search on the first results page. In other words, the best case for search=find is 67%.

Relevancy is as much art as science; the best solutions for the problem are the ones that provide a way to match the art to the science. If you need some background on relevancy, read the seminal article on relevancy and findability by Grant Ingersoll, and check out the fine presentation on the subject by Mark Bennett of New Idea Engineering delivered at the most recent SFBay Lucene/Solr Meetup we co-sponsored with the Computer History Museum in early September.

One of the best implementations of findability in Lucene and Solr I’ve come across is at Netflix. There’s a really nice discussion captured in some slides by Walter Underwood, who helped built the Solr search infrastructure at Netflix (a milestone in a very distinguished career in search). He gave a terrific presentation at that same most recent Meetup.

A key metric Walter used at Netflix to gauge finding (search relevancy effectiveness is such a mouthful) is called Mean Reciprocal Rank, or MRR. Simply put, it gives one point for a click through to the first-ranked item, 1/2 a point to the second ranked item, 1/3 of a point to the 3d ranked, etc. While it may not help find relevancy bugs, it provides a very nice aggregate picture of users’ experience finding what they look for. A good benchmark, or stretch goal, according to Walter: 0.5 MRR, with 85% of clicks on #1.

Let me be quick to say that there is much that is unique about the Netflix search use case (and much that is really, really fun). But the contrast between 85% of results selected at #1 in the results ranking, vs. 2/3 of results on the first page at best in class enterprise search implementations, leads me to wonder: what are others doing to measure relevancy and programmatically build feedback loops, automatic or otherwise? Lucene and Solr provide transparent, rich interfaces for doing this; and according to the Aberdeen study, there’s plenty of opportunity to do so.