Grant Ingersoll [search on Grant] talks to Walter Underwood, Search Software Engineer of Netflix.

Get the "behind the scenes" tour on how Netflix optimizes their search, using Solr, to improve the customer experience using autocomplete, improved findability and other techniques for over two million searches per day. Walter discusses migrating to Solr, configuring for known data search, intuiting user behavior from log files, tuning autocompletion, debugging space variants, faceted browsing, and even how Netflix can help pick a good date movie. 

"[W]hat’s interesting for me is that Solr brings out some things that were certainly possible in Lucene but maybe not so obvious. The per-field analyzers are really fantastic."

Interviewer:

Today I’m speaking with Netflix Search Software Engineer, and long-term enterprise and web search developer, Walter Underwood. Thanks for joining me, Walter.

Walter Underwood:

Glad to be here.

Interviewer:

So Walter, you have a pretty deep search background. Can you fill us in a bit on your past prior to using Lucene and Solr?

Walter Underwood:

Sure. I started out at – well, about 12 years ago I joined Infoseek and was about the third engineer, I guess, on Ultraseek Server, which is their enterprise search product and integrated search engine, and search spider, that was based on the Infoseek engine, and I was there on that product for over nine years.

As we were acquired and shifted around from Infoseek to Disney Go.com, if anyone remembers Go.com.

Interviewer:

Yeah (Chuckling.)

Walter Underwood:

To Inktomi, and then rode that roller coaster down to when Inktomi websearch went to Yahoo, so it was kind of nice being with a websearch group again after Infoseek, and then went to Verity, who of course had been a deadly competitor, and then Verity was acquired by Autonomy. So it was quite a Silicon Valley ride to have the same customers and the same product over five companies. If you included both Disney and Go.com, we called it Six Flags over Ultraseek.

Interviewer:

I think you just summed up the – a whole history of search in your career right there, or at least commercial search in the last 10 years, right? (Chuckling.)

Walter Underwood:

To move from Disney to Inktomi we had to drive past Excite and Google, as they were all in between two companies.

Interviewer:

(Laughing.) So with all that search experience what initially attracted you to use Lucene and Solr?

Walter Underwood:

When I came to Netflix we had a home-built search engine built by people who obviously had done search before. It was really fast but you couldn’t really – you couldn’t improve it because it was truly a binary search on an array of tokens in memory, and there were like three arrays, one of exact match, and one of stemmed, and one of phonetic. It worked remarkably well for what it was but it was clear that we needed something that was really flexible and tunable.

We were familiar with Lucene at Verity. We had actually looked at it and I had advocated replacing the VDK library with Lucene – well, Lucene is clearly a great library. It’s as fast as any other search library, and as flexible as you could find.

The cost of maintaining your own search toolkit was effectively – I mean the cost of getting one was zero with Lucene around, so there was no real value in VDK anymore, but so as failure with Lucene to that – you know – enough to respect it as a great platform for the work at Netflix.

We didn’t really need all of the extra fancy stuff like spidering and converting Japanese PDF’s and all that kind of stuff because we have information in a database. It’s an e-Commerce approach, and so we have really it’s just here’s the text, index it, and then tune the hell out of it, and for that purpose what I needed was a server-ized to Lucene.

So it’s like, “Okay, I’ll write a server around Lucene,” and I was getting started and Solr was announced. It’s like, “Cool, I don’t have to write a server around Lucene.”

Interviewer:

(Laughing.)

Walter Underwood:

(Laughing.)

Interviewer:

Yeah. Yeah, I think ever committer and every contributor who previously had written something like that and, you know, something like Solr just said, “Ah thanks, I don’t need to do that.”

Walter Underwood:

Well, and what’s interesting for me is that Solr brings out some things that were certainly possible in Lucene but maybe not so obvious. The per-field analyzers are really fantastic. I really wish I had thought of that for Ultraseek because it really allows you to have multiple views of the same data from a search perspective, the multiple representations.

We were running our older implementation at Netflix. The first round of Solr actually had three versions of each token, or each title. One that was exact with no stemming, and no stop word removal, one is stemmed and stopped, and then one – it was Soundex – or actually Metaphone because Soundex is terrible.

And then we can weight those separately, and of course the query gets the same processing, and so you get three levels of match. So you can weight more, exact match, more heavily. We used the boosts because we would get some benefit out of IDF but not quite enough to really, really prefer an exact match over a stem, then a stem over a phonetic to the degree you wanted.

Interviewer:

How long have you been using Solr at Netflix then?

Walter Underwood:

Netflix has been our – our on-line engine for about a year and a half now. I’ve been at Netflix for about two and a half years.

Interviewer:

So you’ve had some good running time now with Solr. Can you fill us in kind of on how you use Solr and, you know, just high-level architecture and kind of basic things that people would be interested in and, “Oh, this is how Netflix does it, that’s a good idea, and maybe they can take something away from that?”

Walter Underwood:

Sure, it’s a pretty simple architecture. We don’t do anything fancy. I’m kind of a “when in doubt, use brute force” kind of architect. We have a machine that does indexing. We index once a day because the content doesn’t really change that fast. It’s really kind of expensive to make a movie or a TV show --

Interviewer:

Yeah (Laughing.)

Walter Underwood:

-- compared to, say, YouTube or Twitter or something. You know, it’s a little harder than writing 140 characters. We do have some real-time issues but not very much. We do have some instant watching content that’s available at a specific time during the day, like Hannah Montana’s on the day afterwards at this time, but in general, we know what’s going to be available tomorrow.

So we have one index machine and we run an index once a day. We have four or five search servers which use the regular old sync scripts to pick up the new index. They actually check every hour just in case we have to index in the middle of the day, and so normally they don’t do anything but that check once an hour.

We actually did that yesterday when there was a – one of the updates on inventory didn’t run, and so we did new index in the middle of the day and it just picked it up.

In front of the five servers we have a load balancer box, so those are the middle-tier search farm. We turned on caching, HTTP caching on the middle tier and it has been amazing. During peak times we get 75 to 80 percent hit rates.

Interviewer:

Wow.

Walter Underwood:

So –

Interviewer:

Wow.

Walter Underwood:

-- yes.

Interviewer:

Yeah, that’s great.

Walter Underwood:

It’s really, really impressive. In fact we’re probably going to decommission some of those servers because they’re running at about 10 percent CPU right now.

Interviewer:

Oh wow, yeah, that’s a nice little savings right there, right?

Walter Underwood:

It’s a big savings, yes.

Interviewer:

(Chuckling.)

Walter Underwood:

That’s serious money. Our server farmer is actually, usually it’s IBM Power PC boxes. Those were chosen before I got here but I believe it was for our power consumption as much as anything else.

So then we have our web app, web farm. The Netflix web apps are very, very stateful because every, every one is personalized. Almost every page on the site is personalized because we have the ratings that we show the ratings for your movies, we show what’s available in your area. We show all kinds of stuff that’s very personalized, so it’s very heavy in that – you know – session heavy in that sense.

Interviewer:

Does the personalization play a factor in search or do you – you know, so I imagine your search side, you know, you’ve got movies, you’ve got TV shows and you’ve got all the metadata around them. Do you factor in things like the user preferences, or any of that into search?

Walter Underwood:

We don’t really do that because – what’s very interesting and actually took me a long time to get my head around is most places, when you look at the search logs, you’ll see very general queries. You’ll see queries that are sort of English categories, like “fleece jacket.”

At Netflix people don’t search for Westerns or Thrillers. They search for a specific Western. They search for Unforgiven or The Assassination of Jesse James by the Coward, Robert Ford. So over 90 percent of our queries are specific movie titles.

That’s an overwhelming amount of known item search and then the next, another three or four percent is specific people, well over 95 percent known item search. And so the only real spot where you bring in some kind of tiebreaker is when you have a lot of stuff with the same title.

So Pride and Prejudice, there’s four, five Pride and Prejudice matches. Which one do you show first?

Interviewer:

Right, which one do you show first?

Walter Underwood:

We show the one that’s most popular. We show the one what people are renting.

Interviewer:

Do you do that sorting in Solr, or is that something you do afterwards, after you get back the results.

Walter Underwood:

We do that in Solr. We pick up a global value for the popularity and just index that and add that in. It’s a function query.

Interviewer:

Ah, so it’s actually – it’s part of the score, not a sort on popularity?

Walter Underwood:

Right, right.

Interviewer:

Great, great. Yeah, that makes sense. That’s a perfect example of using function queries and Solr, you know, or pick up something that’s on the document and have it be a factor but not the only factor in scoring.

Walter Underwood:

Right. You know, there are some queries that are sort of interesting, like we have, oh, Treasure Island, there are 11 movies titled Treasure Island and we have 10 results. So we’ve got --

Interviewer:

(Laughing.)

Walter Underwood:

-- you know we can’t really show them all in front, or South Park. I don’t know how many South Park titles there are now, like 20 or something.

Interviewer:

Right.

Walter Underwood:

Scooby Doo is amazing, like 26 titles or something.

Interviewer:

(Laughing.) Right, so I was gonna say, so back to the exact match thing, people are putting in what they think is an exact match but obviously, you know, they – they don’t remember the titles right or they spell them wrong, or they get certain parts of the title correct.

How do you guys approach that problem?

Walter Underwood:

Well, again, we’re lucky in that we have – we have another entire arm of our business is an industry that’s spending millions of dollars to teach people the names of the movies.

Interviewer:

Right. (Chuckling.)

Walter Underwood:

So it’s not quite like trying to remember what the name of this particular Acer laptop is – is Acer around anymore? You know, people can spell 300 or Iron Man.

Interviewer:

Sure, but Assassination might be – might be – to go back to your earlier example, Assassination might be one that’s more problematic for people.

Walter Underwood:

Well, and the movie industry and studios don’t really get making movies findable. Anybody who had thought about Americans typing into Google would never have named a movie Ratatouille. No one can spell Ratatouille. I can only spell it because I – you know – kept testing it.

Interviewer:

You’ve looked at the search logs?

Walter Underwood:

I use it as an example. Well, what was very interesting, one of the big things we’ve done, probably the single biggest improvement in relevance has been to add the autocompletion --

Interviewer:

Right.

Walter Underwood:

-- on the search box.

Interviewer:

That makes sense.

Walter Underwood:

And the way I spotted that is I was looking at search logs and there were these searches for “rat,” and “rata,” and “ratat.”

Interviewer:

(Chuckling.)

Walter Underwood:

And I wasn’t connecting that to Ratatouille, and then there was “apoc,” you know, A-P-O-C, and it turns out that was for Apocalypto and the thing that finally got me was somebody typed in K-O-Y-A-A – oh Koyaanisqatsi, I can’t even type it – I-S-Q, and stopped, and I realized that they didn’t know – they weren’t sure whether there was a ‘U’ after the ‘Q’ in Koyaanisqatsi and there is no ‘U’ after that ‘Q,’ which is why they stopped.

And our algorithms like Metaphone and fuzzy matching all assume that somebody makes a brave attempt to spell the whole word and messes it up, but that’s not what people were doing. What people were doing was stopping typing rather than make a mistake.

Instead of being wrong, they typed as much as they could get correct and just threw it at the engine.

Interviewer:

And so short of doing a wild card query, you – they were probably left with really bad results?

Walter Underwood:

That was not working, no. “Rat” doesn’t get you to Ratatouille.

Interviewer:

Yeah.

Walter Underwood:

You know, it gets you to “rat patrol” or something. And so --

Interviewer:

Because presumably you’re not doing n-gram analysis or anything like that, right?

Walter Underwood:

Well, even if you do, there’s other stuff that matches it.

Interviewer:

Yeah.

Walter Underwood:

Like there’s too much stuff that matches “Rat.”

Interviewer:

That’s true.

Walter Underwood:

And, you know, Koyaanisq, you could probably get away with –

Interviewer:

Right.

Walter Underwood:

-- but it was clear that it is just a user behavior that I intuited from the logs, and so we knew that autocompletion was a good thing but I’d like to know what it is that I’m actually fixing. So, you know, what are people doing that will be improved by this?

And enabling autocomplete was a big boost. We measure MRR, mean reciprocal rank. We use that because for almost all of our queries there’s only one information need. Well, sometimes people are looking for Weeds or Heroes, and they want to put them all on their queue.

But most of the time they only want one of the Pride and Prejudice movies, and so MRR is basically a weighted click-through measure. The first hit gets a full click, the second hit gets half a click, third gets a third click, and so on, reciprocal down the page.

So you get penalized. You’re penalizing yourself in this metric for not putting something at the top that someone wants to click on. So our MRR measure went up, if I remember correctly, five points when we enabled autocomplete –

Interviewer:

Wow.

Walter Underwood:

-- and that is a huge move.

Interviewer:

Yeah, that’s very large (Chuckling.)

Walter Underwood:

And most of that was really, you know, directly improved total click-through, because the stuff was showing up at the first position once you got the name right. So we had basically five percent more successful searches.

We are seeing around two to two and a half million searches a day now. At that time it was more like 1.2, 1.5 million searches a day. So that’s a lot of extra convergence.

Interviewer:

Yeah, a lot more happy customers.

Walter Underwood:

Yeah, happy customers is what it’s all about.

Interviewer:

Exactly (Chuckling.) So I want to come back to the metric stuff because it sounds like you guys are doing some pretty hardcore stuff there, but first can you fill us in on how you’re doing AutoSuggest?

Walter Underwood:

Sure. This is another one where we did the first test implementations of this before we were using Solr, partly because a lot of the company was very, very busy getting the instant watching stuff going, so I was doing kind of stand-alone implementation on it.

And that was, again, an in-memory tree search. I used a ternary search tree, really the same kind of thing that AltaVista used to use inside their index, something that was oriented towards pure prefix matching and it returns – it’s a Servlet and it returns adjacent with the matches.

Interviewer:

Okay.

Walter Underwood:

And it’s loaded up from – originally it was loaded up from our content database and it’s sitting in each front end server because it has to be very, very fast, and it gets a lot of load. You know, for two and a half million – two to two and a half million queries per day, we get 25 million autocomplete queries per day. So –

Interviewer:

Because you show like the top five or the top 10 autocompletes or what?

Walter Underwood:

Well, it’s just because people are typing, right?

Interviewer:

Yeah.

Walter Underwood:

And they type 10 characters for every query that goes to the search engine.

Interviewer:

Right.

Walter Underwood:

And it needs to be very, very fast because it’s interactive. So we didn’t want to make the round-trip to the back end. The first round with this in-memory tree, we wanted to do some additional work with it. What we ended up doing to replace it was using RAM directory, was putting Lucene in memory in each FrontEnd web app, and so we put together a little query and the only time it’s slow is with a one character – is for the first character.

Interviewer:

Yeah.

Walter Underwood:

Luckily, before we switched to this we had also turned on caching for that servlet in the FrontEnd Load Balancers, and so HTTP caching for that, so that took us from 25 million requests a day to five million requests a day.

Interviewer:

Gotcha. Yeah, that’s nice. So with the RAM directory then you have – you have – obviously have documents in it. Are you just putting, say, movie titles in there or do you put in actors’ names, and all of that stuff too?

Walter Underwood:

We also put in actors, movies and actors.

Interviewer:

And then how do you – what do your fields look like? Do you analysis – any analysis on it or is it just – it’s a straight string with no real analysis?

Walter Underwood:

It’s really – it’s exact. It is by token – it’s tokenized but there’s no stop words or stemming.

Interviewer:

Okay.

Walter Underwood:

So it’s a no fancy analysis kind of thing, and the query is crafted to tack on a wildcard at the end.

Interviewer:

So you’re doing –

Walter Underwood:

So –

Interviewer:

You actually do a query against the RAM directory as opposed to –

Walter Underwood:

Right.

Interviewer:

-- just using the term enumerator or something like that?

Walter Underwood:

Right, we do a query, because we also – we have some different situations where we are – we only want to match the instant streaming things.

Interviewer:

Gotcha.

Walter Underwood:

So then we would use a Boolean prefix for the media or we actually have autocomplete in two places on the site. One is for the search box, the other is for adding items to your queue, and there, it’s interesting. It’s really a kind of navigation.

Interviewer:

Right.

Walter Underwood:

It’s not really search at all, and in that case instead of returning just Pride and Prejudice which maps to five movies, we return five entries for Pride and Prejudice with a year after them –

Interviewer:

Right.

Walter Underwood:

-- and you have to guess which one has Colin Firth in it but that –

Interviewer:

(Laughing.)

Walter Underwood:

-- we can’t do everything.

Interviewer:

So if you’re tokenizing then how do you deal with – okay, I type “Pride, space, and space Prejudice,” you know, so that you – you’re not just completing on the individual token. You have to actually complete the phrase, right?

Walter Underwood:

Right, we know what people type so we can say search for “Pride and star,” or “an star,” so we just put a wildcard on the very end of what you’ve typed.

Interviewer:

Okay.

Walter Underwood:

And there’s some fuzziness in getting it exactly right.

Interviewer:

So that’s – that’s really cool. I personally haven’t implemented autosuggest before, so I’m always curious to hear how – because I have seen people mention different approaches, so it’s good to hear somebody say, “This is what we did.”

Walter Underwood:

One of the hard parts, and we did this for the spelling suggestions at Ultraseek as well. One of the hard parts is knowing – getting a list of the right answers. So if you have enterprise documents then you’ve got all this stuff in there. You’ve no idea it’s a real thing you should complete to.

In our case we have the movie titles and the people’s names, and so despite occasional content problems, you know, we’re basically in good shape. There’s always collisions, like, we had one person, and entry for “monkeys,” and one for “The Monkees.” It’s like, no, sorry, one band, collapse those.

Interviewer:

(Chuckling.)

Walter Underwood:

Or does “N.W.A.” have periods in it or not? You know, there’s -- you’ve got two entries. So those kind of things even with professionals putting in all the data you end up with metadata issues, but having a list of the right answers is a huge difference.

Interviewer:

Yeah, data cleanliness is always a problem in search, right? Garbage-in, garbage-out kind of thing?

Walter Underwood:

Our consultants at Verity are really, really good at this. They told me that cleaning up the data was half of the work in a consulting engagement.

Interviewer:

Yeah. I can tell you a number of times I’ve gone into a place or other people I work with and, you know, “Oh, we’re having some issue with relevance on this thing,” and probably eight out of 10 times or maybe around there it’s, “Oh, they were missing some entry on a field that they were searching against, and that was the problem.”

So going back to your – you know, talking about mean reciprocal rank and all that, can you tell us more about what you guys do for relevance testing?

Walter Underwood:

Sure. Well, actually it was an interesting little story because we had this awful ad hoc engine, right, I mean really now, and replacing it with Solr we had to – we have it – we run A/B tests all the time. At any point on the Netflix site there’s probably a dozen A/B tests running, and sometimes those will have four cells, six or seven different test cells so we can work stuff out.

So we were running an A/B cell between the classic engine, what we now call “classic,” and Solr, and Solr was not doing as well. And it’s like it was just – it was driving me nuts. You know, it’s like, “Okay, it’s a really, really a better engine than this, it actually does ranking, not sorting.”

And so in order to figure it out I said, you know, it’s like, “Okay, stop tuning for two weeks and get some data, and so I started going through all of the Apache logs and working out the clicks, what did people click on to get a database of clicks so that we could re-run those, re-play the clicks off-line, and do off-line the MRR testing.

Now that misses a little bit of stuff, so let’s say if something new shows up in the new engine you don’t have any clicks for that because the old engine wasn’t showing it, so they can’t click it.

Interviewer:

Right.

Walter Underwood:

But it does – it is still very useful. In particular, you can see what’s working in the old one and not working in the new one.

Interviewer:

Okay.

Walter Underwood:

So I ran through this and I would say, “Okay, here’s the – here’s the success rates for these clicks,” and found out that they were – interestingly there was – it was almost all spacing variants. So, is “Space Ghost” one word or two? Is “Dark Star” one word or two? Is “Ghostbusters” one word or two? Is – those are all one word or two stuff.

And there was a little bit of – you occasionally get some things like “Full Metal.” They can’t even decide whether “Full Metal,” or “Dance Hall,” or “babysitter,” are one word or two because even the titles we have, so some of those are symmetric. You know, it’s “Full Metal Alchemist” and “Full Metal Jacket.” They’re different. They’re different movies but they’re also different, you know, spacing.

Interviewer:

Different spacing, yeah.

Walter Underwood:

(Chuckling.) So I put in a few hundred manual synonyms for those variants and boom, it was as good as the old digit.

Interviewer:

Nice.

Walter Underwood:

So we have been maintaining those manually. We see stuff come up like, oh, what was the – oh, Step Brothers, thank you so much, “Step, space Brothers.” It’s cute but no one can type it.

Interviewer:

(Laughing.)

Walter Underwood:

And so we’ll see those things pop up, put in a synonym and go along. One of the things on my list to do is to use shingles to do that, all that automatically.

Interviewer:

Right, and just for our listeners, shingles are an n-gram --– n-grams at the word level. So what we call them in Lucene and Solr land.

Walter Underwood:

Right. At – and in fact Infoseek had a thing which they called phrase inference, which is essentially shingles, by word and larger phrases. So things like – well, “metadata” or actually “database,” decades ago, was two words.

Interviewer:

Right.

Walter Underwood:

And then for a while, “database” was hyphenated. Now it’s one word.

Interviewer:

Yeah (Chuckling.)

Walter Underwood:

People kept – I could say that “webserver” was two words, and in fact, dictionaries may still say that “webserver” is two words along with “webmaster.” So English is very, very rich, and is kind of compounding instead of now – now noun compounds like this, and it’s just something you’ve gotta deal with.

Interviewer:

Yeah, I know in German and a lot of the Northern Germanic languages you have – the problem is even more pronounced, I believe. So –

Walter Underwood:

Well, there it’s so common that you just – that it doesn’t work at all unless you can break it up.

Interviewer:

Yeah.

Walter Underwood:

So in that case you deal with it head on. It’s sort of like all the partial matching in Arabic with missing – with the vowels not written out. It’s like, you know, it just doesn’t work at all unless you deal with that.

But for English you can – you can go quite a ways and then you find out that you’re losing four or five percent because you’re not doing this and so – so one of those things as – and as I posted in the list, on the mailing list, was to consider – to do something with really a full text parser instead of a query parser.

So where you’re a user query, you say, “This has zero query syntax in it.” There’s none.

Interviewer:

Right.

Walter Underwood:

We’re running it through a regular full-text parser, I’m gonna take the tokens out of that, convert those to a Lucene query.

Interviewer:

Right. So you’re – it’s almost like a natural language query, in other words?

Walter Underwood:

Very much a natural language –

Interviewer:

Yeah.

Walter Underwood:

-- full-text parser, really exactly the same processing all the way down. Every two months on the mailing list we come up with the, “Why aren’t synonyms working?” thing. It’s just the –

Interviewer:

(Laughing.)

Walter Underwood:

(Laughing.) Right.

Interviewer:

Right.

Walter Underwood:

And it’s because there’s different parsing at the space level. One is – there are different languages. One is English, and one is Lucene queries.

Interviewer:

Right. I don’t think people pay enough attention necessarily to the whole notion of relevance and findability. That – they just assume it works and, in a lot of cases, it does work, but Netflix is a perfect example of kind of that you have – it’s almost its own genre, you know, movie titles and people wanting to come up with names that differentiate themselves, and so how that then translates to problems for us search engineers, you know, and solving that with synonyms is a pretty common – pretty common way of doing it. So –

Walter Underwood:

One of the interesting ones we’ve found – oh, I guess it was last summer obviously was the move – that the query “Batman” was failing,” and it was because the title, The Dark Knight doesn’t have “Batman” in it.

Interviewer:

Right. So you needed a synonym for “The Dark Knight” matches “Batman?”

Walter Underwood:

Right.

Interviewer:

Yeah. Yeah, those pesky movie people don’t know how to make things findable, right?

Walter Underwood:

Well, and you can look at another – that another way though and that’s that one of the search tactics that people use is to use some kind of either fictional character or suppose they use “James Bond,” or “Batman,” or “Scooby Doo,” or something like that to find the movies because “Barbie,” “Godzilla,” these are all queries that we see, and they’re not – they’re really little, bitty tiny categories around that particular character or franchise, or whatever you’re gonna call it. It’s just a category of movie at least.

Interviewer:

Do you ever get queries that are of the style of – that – you know – where somebody would type in, “That movie with the guy and the girl about going on a date, or something like that.” (Chuckling.) You know where it’s just they’re – it’s like they’re describing the movie or, you know, “The movie with that guy named Clint Eastwood in it or something like that.” How do you handle that stuff?

Walter Underwood:

We really don’t get very many of those.

Interviewer:

Oh, really? You don’t?

Walter Underwood:

I always – no, we don’t. I was – no, I look for this and so, you know, a query like, you know, “funny movies,” someone was asking me about this and so we had about 15 or 20 queries per day for that out of two and a half million.

Interviewer:

Okay.

Walter Underwood:

So we don’t really – we don’t have a lot of that kind of discovery thing and we actually looked into this because obviously the search engine really only works for movie titles, and so we thought, “Okay, we’re just training people to only search for the stuff that works,” and so we’ve got detailed logs on new subscribers and existing users, and found there was very, very little difference between how they searched with the categories and things they searched for.

And that makes sense to me because your average web searcher does not use your site any differently than they use the rest of the Internet. They use the entire Internet the way they use Google which also means that you’re – a good design can be a bad design two years later when you don’t change it because the rest of the web is moving.

But really tuning, tuning your queries to a specific site is a really, really advanced tactic, even reformulating your query is de-bugging. It’s programming. It’s a little boolean – teeny boolean program.

Interviewer:

Yeah.

Walter Underwood:

And people are not good at de-bugging. They’re terrible.

Interviewer:

(Laughing.)

Walter Underwood:

And it’s just because it’s hard.

Interviewer:

It is.

Walter Underwood:

Even people coming in through the CS program, you’ll find people who are – you know – it’s a tiny number of people have already got the de-bugging thing and everybody else has to learn it painfully.

Interviewer:

Right. So –

Walter Underwood:

Yeah.

Interviewer:

“—painful” is the right word too.

Walter Underwood:

Right, so we really need to get people there without – you know – by leading them on, but do that de-bugging for them, and that’s where things like spelling suggestions and autocomplete come in in saying “Keep trying, here’s a thing to click on.” Here’s something that’s blue and underlined, and go for it.

Interviewer:

Yeah. Well, and it’s interesting too is – you know – de-bugging search results is especially difficult, like I – you know – my advice often is well, “If it’s on the top 10,” don’t worry about it but in your case you guys really do have a need to have, you know, a result be at a particular position, namely the number one position because you know other stuff, you know, the difference in score probably between your number one result and the other ones is probably bigger than a more fuzzy search, I guess, if you would – if you wanna call it that.

Walter Underwood:

Well, we have a lot of – we have a vast number of searches and, you know, I’m sure way over the majority of searches really only have one answer.

Interviewer:

Right.

Walter Underwood:

Now probably TV series would be the biggest exception to that, and you can think of it as a single answer but it’s really – you know – you can’t – there’s three results or – or for something like The Office there’s the UK series and there’s the U.S.

Interviewer:

(Chuckling.) Well and you almost – for those you almost need like clustering, or some type of collapsing that brings similar results together into a single result or, you know, like on Google, that indents results from that same grouping, right?

Walter Underwood:

We have thought about that. There’s – we’re making a lot of exceptions because you have super series like CSI.

Interviewer:

Yeah.

Walter Underwood:

Or Law and Order, good grief.

Interviewer:

(Laughing.) There’s a new one every week.

Walter Underwood:

Right, and so then you’ve gotta deal with those, or then you things – do things like the Mad Max movies, or Krzysztof Kieslowski’s Red, Blue, and White series --

Interviewer:

Right.

Walter Underwood:

-- which don’t even have the same words in the title. They’re red, blue and white. You know, it’s one of those where we’re a little hesitant to roll out something that works for only about half of that kind of content.

Interviewer:

Yeah. Yeah, I can see how that would be difficult.

Walter Underwood:

And it’s been pretty well right now. About 85 percent of our clicks are on the first hit, which is pretty dang successful.

Interviewer:

Yeah, and that’s pretty much --

Walter Underwood:

But that’s partly because of the content.

Interviewer:

-- and that’s pretty much just – you’re running – from what I understand it sounds like you have Solr pretty much just out of the box, right?

Walter Underwood:

Right. The one thing that we did do last Fall was – well, Summer and Fall, was spend a bunch of time working on fuzzy search. I mentioned that the phonetic matching was our old algorithm, and what we’re using now, we dropped the phonetic search, and now we’re using fuzzy, and that was one where we ran multiple test cells, and we actually implemented other fuzzy algorithms.

We implemented Jaro and Jaro-Winkler. We did some pre-filtering within grams to cut down the number of matches. We used it with a prefix and without, I mean requiring the first letter to match. It turns out requiring one – one letter to match on the front of fuzzy is a huge speed-up.

Interviewer:

Yeah.

Walter Underwood:

Now with our current – with our current load we could probably run with a full – with or without requiring that original, that first match, especially considering the efficiency in the HTTP cache.

Interviewer:

Yeah, but hopefully –

Walter Underwood:

But it wasn’t –

Interviewer:

-- people can type the first character right (Chuckling.)

Walter Underwood:

Almost all the time but it’s not always – not always the case. I mean “knight” and “night.” Was there a ‘K?’

Interviewer:

Right, yeah, and sometimes you just – you know – we’ve all done this, you’re just sloppy on the keyboard and you hit the wrong one –

Walter Underwood:

Right.

Interviewer:

-- and then you realize that after you hit the “enter” button, so we can forgive the users.

Walter Underwood:

Right. What we may do is – well, we can forgive them but we can’t just leave them stranded.

Interviewer:

Yeah.

Walter Underwood:

So we may address that with spelling suggestions.

Interviewer:

Yeah.

Walter Underwood:

So keep the first character requirement but then give a spelling suggestion that gets them to the right place.

Interviewer:

That makes sense.

Walter Underwood:

So –

Interviewer:

Anything else you guys are doing at Netflix that people might be interested, and, “Oh, that’s different, I hadn’t thought about doing search that way?”

Walter Underwood:

Well, there’s a new – there’s something that was just turned on for all of our subscribers, last, like, Monday. We have – we’ve opened up a whole new categorization scheme for the movies where you would say something is romantic, or dark, or post-apocalyptic, or – and that coming of age, or sci-fi, and so like A Boy And His Dog is all five of those.

Interviewer:

(Chuckling.)

Walter Underwood:

And so we have – not a good first date movie. I had a friend who did that but – not a good – but --

Interviewer:

Not only do we have Solr advice, we have date night movie advice.

Walter Underwood:

Dating advice.

Interviewer:

(Laughing.) That’s great.

Walter Underwood:

You know, we should have a category for first date movies. There you go.

Interviewer:

There you go.

Walter Underwood:

So when you go to the home page for Netflix as you’re a subscriber, you’ll see these categories, and if you click on a category name you’ll go into a faceted browse.

Interviewer:

Okay.

Walter Underwood:

And that is backed up with Solr.

Interviewer:

Nice, yeah, that makes sense.

Walter Underwood:

So that’s the whole new thing that we rolled out, and we were watching the performance very closely. The queries are unfortunately much more expensive than these straightforward, “show me the top 10” queries. They were running about – well, at low load, they were running about five times as much CPU.

But the caching appears to be very similar, the cache success rate seems to be very similar. I was worried about the cache hit rate being lower because there’s so many different parameters that come in.

And so if you have a – if you have a query that’s based purely on a query term, or caching where your statistics are entirely based on a query term, you’ve got a straight old Zipf power law, you know, fat head, no tail, we all know what it does. You could just look at the number of queries per hour, and you can almost guess the hit rate.

Interviewer:

Right.

Walter Underwood:

But this, with so many variables, I was worried, but it’s – it’s still got a very, very high hit rate. So I guess people still click – agree on what they’re going to click on, even though they have too many options – or enough options. It’s all new.

Interviewer:

So they can then search once they pick a facet, as well, and search within that facet?

Walter Underwood:

It’s pure browse.

Interviewer:

Oh, okay.

Walter Underwood:

This is something where we did implement a bunch of options. We implemented a bunch of things and tried not A/B testing but really prototyping in this case and integrate – it turned out certainly for our stuff that integrating search and faceted categorization was really complicated, and people were just sort of lost in it.

My guess on that is that people have treat – the search is a known item location thing, and for discovery, they don’t do that. For discovery, they go into the genre pages, or something else.

So – and I noticed that I had done this when I was trying to find a movie for my son, for a friend who was coming over and it’s like, “Oh, we’ll get something for him,” and it took me forever wandering around the genre pages because I didn’t know the name. I was looking for something good.

Interviewer:

Right (Chuckling.)

Walter Underwood:

And I ended up –

Interviewer:

You didn’t just type in “something good?”

Walter Underwood:

There’s a movie named that.

Interviewer:

There probably is.

Walter Underwood:

I ended up with the Zorro movie with Antonio Banderas.

Interviewer:

Right.

Walter Underwood:

Which is a great movie.

Interviewer:

Yeah.

Walter Underwood:

And it was perfect, and but I had to get down into a genre called “Super Swashbucklers.”

Interviewer:

That’s a genre? Okay.

Walter Underwood:

So yeah, that’s a genre.

Interviewer:

Wow.

Walter Underwood:

So when we do – so the faceted browse is much more of a discovery navigation thing. We’ve run up against this in a bigger way the last couple of years because of the instant watching content.

So because of the studio contracts, we can’t have everything on streaming that we have on disc.

Interviewer:

Right.

Walter Underwood:

It’s just not available, and so when – if somebody wants 300 or you know, sorry, or whatever it is, so you can get Ratatouille, but you can’t get – you can’t get WALL-E.

Interviewer:

WALL-E, right.

Walter Underwood:

You can get Ratatouille on streaming but you can’t get WALL-E on streaming. So but if somebody navigates down in to this, and you kind of get in this area, this little faceted zone -- and I’m waving my hands and making little balls of content in the air in front of me.

Interviewer:

So we should have done the video?

Walter Underwood:

(Chuckling.)

Interviewer:

(Chuckling.)

Walter Underwood:

Yeah. Then they can click a box that says, “Show me instant content only.”

Interviewer:

Makes sense.

Walter Underwood:

And then you could just filter that down to say, “Okay, just show me that.”

Interviewer:

Right.

Walter Underwood:

And then – because that’s a real problem if someone says, “Okay, I wanna watch this, oh, it’s not an instant, I wanna watch this, it’s not an instant.” Whatever and it’s – that’s a pretty high failure rate. You’re just shooting in the dark really trying to guess what might be available.

So the browse discovery model is much, much better for that and one of the things we’re hoping it helps with is that watch instantly discovery.

Interviewer:

The watching what?

Walter Underwood:

The watch instantly. The instant content.

Interviewer:

Yeah, the more ad hoc browsing, and more serendipitous finding of movies, “Oh yeah, let’s watch that and yeah, that makes sense.”

Walter Underwood:

My wife said that she uses search when she’s looking for a movie for herself and she uses the genre browse when she’s looking for someone else.

Interviewer:

Ah, because she knows what they might be – what they like –

Walter Underwood:

Right.

Interviewer:

-- but not a specific title. Yeah, that’s a good example. What do you see as interesting challenges kind of going forward for Solr? I mean are there things that, “Oh, I wish I had that,” or that you see coming up on your horizon?

Walter Underwood:

Well, one of the interesting Lucene and Solr challenges is real-time search, is the fast update.

Interviewer:

Yeah.

Walter Underwood:

It’s a really hard thing to get right and I know people are working on it. We don’t need it right now but it’s certainly a hard thing and something that a lot of people need.

One of the things that’s bugged me for a long time is – is that stemmers are not tuned for search. And now I know we have a lighter weight analyzer that we can use, the K stem. I have not dug into it but when I’ve looked at – kind of anecdotally looking at logs, particularly stemming verbs doesn’t help at all. So you don’t want to turn “tracking meeting” into “track meet.”

Interviewer:

Right.

Walter Underwood:

There’s a movie, Saw.

Interviewer:

(Chuckling.)

Walter Underwood:

Well, you don’t match that to “see.”

Interviewer:

Yeah.

Walter Underwood:

It’s not a verb form, just – just don’t even go there, and so I’m sure that Google and Yahoo, and MSN have stemming that’s really much more usage-based and it’s – I would expect that they’ve gone beyond stemming to more of a let’s find the appropriate word forms for this phrase, more the kind of a usage-based conflation of terms.

Interviewer:

Yeah.

Walter Underwood:

But they have a lot more data. We don’t have enough data to do that.

Interviewer:

Yeah.

Walter Underwood:

Even with two million inquiries a day which is a great, great level of traffic for an analysis, we still – I think we could get some improvement by going to a much less aggressive stemmer who really just worked on plurals and –

Interviewer:

Right.

Walter Underwood:

-- noun phrases.

Interviewer:

Yeah, like actually in our – in the lucid distribution we have an optimized version of the K stem and we usually recommend people start out with less aggressive stemming and then move, you know, given appropriate relevance tests and that, then they can try more aggressive stemming, but you know, stemming and all of that, it’s – a lot of it’s the domain of computational linguists and people just kind of – I think a lot of people start out with, “Oh, well, here’s the example that’s in Solr, those guys know what they’re doing, that must be right for me as well.”

And in reality it’s one of those areas where knowledge of your domain can really help just as you’ve pointed out. You know, in my case if I’m doing something else, you know, stemming the word “saw” to “see” would make perfect sense, but in your case, obviously it doesn’t.

So yeah, that’s – that’s really a good point.

Walter Underwood:

I was talking about the amount of traffic we have and how it’s nice for analysis. I would also mention that when I did that tuning to find the spacing variance between the classic engine and Solr, I had to get about 25 million clicks to get – say 1,000 or more queries down to – down the curve, and to get really stable statistics.

So it could take a lot of data to get down the curve and find stuff, so even when we look at a week’s worth of queries now, I cut it off at queries that happened a hundred times, and that’s not as many queries as you might think.

There’s a pretty heavy, heavy drop.

Interviewer:

Yeah. Yeah, most people search for the same thing over and over on – on – I would imagine, right? Whatever is the new releases and things like that?

Walter Underwood:

Right. I was looking at a batch of queries a couple of days ago, and so we have around nine million queries over the period and 1.2 million of those were for Trilobite.

Interviewer:

(Laughing.) Right, there you go. Now if you could just solve the problem I always seem to have with movies is I can – I’ll be somewhere totally random and a movie comes into my head, and I kind of remember the title but I don’t, and then by the time I get back home to look it up I’ve completely forgotten it.

Walter Underwood:

Well, one of the interesting things we’ve done is the last – so starting at the beginning of the Fall was open up API for Netflix.

Interviewer:

Oh, there you go.

Walter Underwood:

And that does have a search. There’s a search in the API as well.

Interviewer:

Ah!

Walter Underwood:

And that really goes straight to the stuff that we use, so you know, that will change and improve as we improve search.

Interviewer:

Right.

Walter Underwood:

That’s interesting because we’re seeing – well, we opened up, and that also opens up things like your ratings of movies and recommendations to other applications. So we opened up a Facebook connection last week, I think.

Interviewer:

Nice.

Walter Underwood:

And I really like some of the iPhone apps.

Interviewer:

I was gonna say now I solved the problem but I don’t have the phone. I have the iPod Touch, and so I’ll take notes on my iPod (Chuckling.)

Walter Underwood:

Right, right.

Interviewer:

And save them for later.

Walter Underwood:

There’s several. The one I’ve been using is called Now Playing, and so you could do things like look at what’s in the movies, what’s in the theaters, and go, “Oh, I like that,” and go add it to your queue.

Interviewer:

There you go.

Walter Underwood:

And you know you say, “Well look, I’m not gonna be able to get to the theater this week,” or this month, or whenever, but I – say I want to see Sunshine Cleaning, I want to put it on my queue, so that’s – that’s really nice to have that kind of – not a new DVD release but a new theater release.

Interviewer:

Yeah, yeah. Well I just – every now and then I go through a phase of – I’m into classic movies or something. I – these are the movies that you feel like you’re supposed to have watched because everybody, it’s part of the culture but maybe I haven’t seen them so it’s like trying to remember all of those but that – there again, that’s where your discovery mode would really fit well.

I can go look at classics or modern classics or something like that, and then discover those movies that way instead of having to remember specific titles.

Walter Underwood:

And you could have editorial lists, like I found a list of the – some film schools, like these are the movies you’re supposed to have seen.

Interviewer:

Right.

Walter Underwood:

And it’s like, “Wow, cool, maybe we should just have those things linked up to Netflix.”

Interviewer:

Yeah, definitely. All right, well, Walter that’s all I have. It was really great talking to you about Solr. Anything else you want to add in?

Walter Underwood:

Sure. One thing is that the mailing list is just great. It’s a tremendous help. I was trying to solve a problem this week and I knew it was in there but, you know, I actually wanted to get the push onto the site to production the same day, and so I rummaged around in the docs a little bit but couldn’t find it exactly.

And so I sent a request to the mailing list and went across the street to get a sandwich, and I came back and I had gotten the answer eight minutes later. I got one answer six minutes later and I got a full, complete answer two minutes after that.

Interviewer:

Right.

Walter Underwood:

Boom, and it made it onto the list. I mean they – it was fixed, on production, done.

Interviewer:

Nice.

Walter Underwood:

And so we can start another A/B test.

Interviewer:

Yeah, power.

Walter Underwood:

(Chuckling.)

Interviewer:

Power of the crowds, right.

Walter Underwood:

So –

Interviewer:

Yeah, and of course now –

Walter Underwood:

So it’s a really good group. It’s really helpful and I try to contribute back to it just because it’s – you know – it’s been a big help.

Interviewer:

Yeah, your contributions are definitely appreciated. I can definitely speak to that, and of course now you can go to Lucid’s mailing list search place where you can actually search all of the Lucene and Solr – not only the mail archives, but the wiki, and the documentation, and all of those, all in one place. So hopefully that will complement your – your future need of finding answers on Solr.

Okay. Well, Walter, I want to thank you very much for taking the time to speak with me. It sounds like you guys are really doing some really neat things with Solr there at Netflix, and I look forward to seeing more and more from Netflix and from you. Thanks.

Walter Underwood:

Thank you.