For instance, for testing memory, it’s pretty easy to set up a series of tests that start with a small heap size and successively grow it until no Out Of Memory Errors (OOME) occur. Then, to be on the safe side, add 1 GB of memory to the heap.  It works for the large majority of people. Ironically, for Solr at least, this usually ends up with a heap size somewhere between 6-12 GBs for a system doing “consumer search” with faceting, etc. and reasonably sized caches on an index in the 10-50 million docs range. Sure, there are systems that go beyond this or are significantly less (I just saw one the other day that has around 200M docs in less than 3 GB of RAM while handling decent search load), but the 6-12 GB seems to be a nice sweet spot for the application and the JVM, especially when it comes to garbage collection, while still giving the operating system enough room to do it’s job.  Too much heap and garbage may pile up and give you ohmygodstoptheworld full garbage collections at some point in the future.  Too little heap and you get the dreaded OOME.  Also too much heap relative to total RAM and you choke off the OS.  Besides, that range also has a nice business/operations side effect in that 16 GBs of RAM has a nice performance/cost benefit for many people.

Recently, however, I thought it would be good to get beyond the inherent hand waving above and attempt to come up with a theoretical (with a little bit of empiricism thrown in) model for estimating memory usage and disk space.   After a few discussions on IRC with McCandless and others, I put together a DRAFT Excel spreadsheet that allows people to model both memory and disk space (based on the formula in Lucene in Action 2nd ed. – LIA2), after filling in some assumptions about their applications (I put in defaults.)   First a few caveats:

1. This is just an estimate, don’t construe it for what you are actually seeing in your system.
2. It is a DRAFT.  It is likely missing a few things, but I am putting it up here and in Subversion as a means to gather feedback.  I reserve the right to have messed up the calculations.
3. I feel the values might be a little bit low for the memory estimator, especially the Lucene section.
4. It’s only good for trunk.  I don’t think it will be right for 3.3 or 3.4.
5. The goal is to try to establish a model for the “high water mark” of memory and disk, not necessarily the typical case.
6. It inherently assumes you are searching and indexing on the same machine, which is often not the case.
7. There are still a couple of TODOs in the model.  More to come later.

As for using the memory estimator, the primary things to fill in are the number of documents, number of unique terms and information on sorting and indexed fields, but you can also mess with all of the other assumptions.  For Solr, there are entries for estimating cache memory usage.  Keep in mind that the assumption for caching is that they are full, which often is not the case and not even feasible.  For instance, your system may only ever employ 5 or 6 different filters.

The disk space estimator is much more straightforward and based on LIA2′s fairly simple formula of:

disk space used(original) = 1/3 original for each indexed field + 1 * original for stored + 2 * original per field with term vectors

It will be interesting to see how some of the new flexible indexing capabilities in trunk effect the results of this equation.  Also note, I’ve seen some applications where the size of the indexed fields is as low as 20%.

Hopefully, people will find this useful as well as enhance it and fix any bugs in it.  In other words, feedback is welcome.  As with any model like this, YMMV!

Looking forward to seeing you there. Everyone in attendance will not only get to hear about the latest greatest enhancements to these technologies, but may also walk away with a cool Lucid t-shirt!

Here are the slides for “Rapid Prototyping with Solr”:

My next talk, titled “Solr Recipes”, was a 3 hour workshop covering the common ways to get data into Solr, configure it, and leverage it within applications.

Here are the “Solr Recipes” slides:

And finally, to a few hardcore folks, I discussed “Lucene for Solr Developers”, which more broadly covered the various ways to extend Solr. One cool example (or so I think, at least) I built for this talk that I’ve put out there as food for thought is “auto-faceting”, having a way for Solr to determine the best facets to select for a given query. See SOLR-2641 for my initial proof-of-concept implementation.

Here are the “Lucene for Solr Developers” slides:

There was so much going on that I barely got to tap into the conference experience myself, with the best part being the conversations had during meals, and between and during the scheduled talks. I was able to reconnect with many long-time friends that I’ve made through No Fluff, Just Stuff, and made many new acquaintances – I won’t name names, as this list covers most of them.

Thank you, Überconf, Jay, and fellow speakers and attendees for a stellar technical event. If you missed it, don’t despair, No Fluff, Just Stuff brings many of the same speakers and topics to an area near you. I’ll be speaking at a few NFJS events during the second half of this year, including Raleigh, NC, Boston, MA, and, surely to rival the quality and size of Überconf, The Rich Web Experience in Ft. Lauderdale, FL.

Let’s talk.

Sensis, “the search engine for Australians”, uses a modified version of this model to assess their own search processes. It has five levels:

1. Unmanaged: Set it and forget it, basically.
2. Ad Hoc: People work on the process part time, and innovation is led by individuals with an itch to scratch.
3. Monitored: There’s a defined team responsible for improvements, and it’s monitored for problems.
4. Managed: Improvements are methodically sought, and there are defined targets and metrics.
5. Optimized: This is obviously the target level, where machine learning leads to the best possible results.

On day 1 of Lucene Revolution, Craig Rees talked about Sensis‘ goal of moving up that maturity ladder as they made their data — millions of white pages and yellow pages listings — available via a developer API. When it comes to search, Sensis is currently at the “Montitored” stage, and making the move up to “Managed”.

Slides for this session:

It turns out that as much as you might like to, you can’t “jump levels” on the maturity index; you’ve got to earn it the hard way, and that’s what Sensis is doing.

Grant Ingersoll is always saying that if you’re not methodically testing your search results, you’re not testing at all, and Sensis definitely is a good example of methodically testing your results. To formulate a “scientific approach”, they created “Gold Sets” of queries and results, which enables them to tweak their Lucene settings, then compare the results with their gold set of “perfect” results.

Of course, Sensis has a lot to test for. Context is key, Craig points out; a 12 year old on his mobile phone in the schoolyard at noon probably shouldn’t get the same results as a 60 year old at home on his computer at 10pm. And context has a lot of variables, such as time of day (or even time of year), location, device, or, and this is perhaps hardest to quantify, intent.

Sensis also has other hurdles to leap. For example, while conventional wisdom says that “rare” terms should be worth more, it can sometimes backfire in their data set, which is broad but not deep. The term “flowers” is rare in “crematorium” listings, but if you search for “flowers”, a crematorium is probably NOT what you want. They also have to deal with contextual synonyms. For example, “bow” and “ribbon” are synonyms — unless you’re also looking for arrows.

So Sensis created a tool, the Search Quality Analysis and Testing system, or SQUAT. (And yes, the audience snickered when he said, “All we need is SQUAT,” but at the same time, I think most of us were just a little bit jealous). The very first question was, “Are you releasing that as open source?” The answer was, “We’re thinking about it, but not at this time.” SQUAT enables Sensis to test for a variety of terms and know exactly how good the results were. If a change doesn’t result in a positive effect on quality score, it doesn’t go into production.

And isn’t that what we all want?

Cross-posted with Lucene Revolution Blog. Nicholas Chase is a guest blogger.This is one of a series of presentation summaries from the conference.

Much as we might think of search technology as a 21st century internet thing, its back to when IBM was sued by the US government. By the early days of the Internet, search—Lycos, Infoseek, Excite, and Alta Vista–began to accelerate the virtuous cycle of requirements and innovation. Marc describes the evolution of the technology from centralized to distributed indexes; among the early players, only Lucene, Google, and Fast had distributed architectures in their initial releases.

There’s no denying the influence of Google on search, and their Page rank algorithm—a popularity-based authority metric—represented a breakthrough in search precision. Along with speed, it helped Google to its now-familiar market position. But in contrast to the public perception of internet search experience as the be-all and end-all, it is essential to understand the differences between “generic” public internet search and enterprise search. One important, little understood virtue of Google is that it is actually a combination of multiple search applications and techniques, tailored to a particular set of user behaviors; the magic is not in one technique or another, but in their deliberate combination.

Marc continues with a review of search fundamentals, precision and recall, and discusses what makes for high scores on each with an emphasis on enterprise requirements, resources, and methodologies. With the stage thus set, the focus turns to the emergence of Lucene and Solr.

He makes the case that Lucene and Solr are industrial strength search technology and are as good as, if not better than, Google for search in terms of precision, performance, and relevance. This derives from the superior attributes of open source as a development model.

That said, as good as Lucene and Solr are, they are not perfect. Issues include a focus on core functionality, at the expense of gaps in certain areas of enterprise requirements. The good news is that these addressed in a number of ways: community, consultants, commercialization, and an enterprise’s internal resources. Marc concluded his remarks with an overview of the competition and the competitive landscape, along with some thoughts on the future.

You can download the slides here.

Cross-posted with Lucene Revolution Blog. Tony Barreca is a guest blogger.This is one of a series of presentation summaries from the conference.

Welcome to Ronald Mayer’s world. In his talk on Day 2 of Lucene Revolution, he described how Forensic Logic aggregates data from local police departments, courts, and even federal agencies, so that law enforcement officers can get information on crimes and suspects not only in their own jurisdictions, but also in surrounding areas.

Slides for this session:

He’s got lots of challenges, not the least of which are the differences in data formats. You’d think there’d be a standard for this, and as he points out, there is. In fact, there are lots of them. In reality, the standards are so large that most agencies wind up using some subset, plus their own extensions. And that doesn’t count the agencies that have all of their data in Word files in a folder in someone’s computer. (It also doesn’t count data from external sources. Apparently gangs are big into MySpace. Who knew?)

Once they’ve created a transformation for adding a new agency’s data, they have a whole host of practical issues to deal with. For example, “at dusk” and “near any elementary school in the school district” are perfectly valid requests, and their system has to accomodate that. They’ve also got to be able to take a statement such as “suspect is a caucasian male, approximately 6’4″, wearing a red ball cap and black jacket, leather” and have it come up for “tall caucasian baseball cap black leather jacket”. Entity extraction and help from Basis Technology products helps associate an adjective with a noun, which makes things a bit easier.

Interestingly, Forensic also has to work in the other direction; if all of that information was encoded by field, it wouldn’t be searchable in a simple text search. So another task they’ve had to perform is to de-normalize the data back into a searchable narrative.

Forensic gets good use from Lucene, and in fact has been contributing back; they have been making heavy use of the phrase field (pf) and phrase slop (ps) parameters in the new Extended Dismax parser but what they really needed was to be able to combine several sets of them into a single query. So SOLR-2058 proposes (and implements) a new query syntax, field~slop^boost, that allows for independent pf and ps settings, such as:

pf2=important_text^10~10&pf=important_text^100&pf=important_text^100~10

Mayer says that although searches might take a second or three to return data, relevance is much more important in this case.

There are still problems to overcome. For example, relative boosts are tricky. How far away does an event have to happen before it’s as irrelevant as something that happened two years ago? Forensic continually works on refining the process, tags, synonyms, and other parameters, so at any given moment, they’re always indexing the oldest (non-reindexed) documents.

Remember that the next time you get pulled over.

Cross-posted with Lucene Revolution Blog. Nicholas Chase is a guest blogger.This is one of a series of presentation summaries from the conference.

Slides for this session:

When Mijares’s company, Canoo Engineering AG, met with Swiss newspaper publisher and media group Axel Springer, they all agreed that what Axel Springer needed was to keep readers on the sites of their most popular newspapers longer, and to drive traffic from those papers to their less-well-known brands. They also agreed that providing “related articles” was the way to do that.

What they didn’t agree on was how to pay for the development of such a service.

Still, Canoo soldiered on. Deciding on Lucene/Solr as the tool of choice was an easy decision; Lucene’s “More Like This” function would be perfect for finding articles similar to the current page.

Well, almost perfect. What Canoo discovered is that while More Like This does work out of the box, without semantics the results weren’t good enough for their purposes. Furthermore, at the moment, it’s designed to work well in English, and the “financial language” of Switzerland is German.

They solved the problem with a combination of strategies. The first was to use WMTrans, Canoo’s language tool and the foundation behind the linguistic sites Leo.de and Canoo.net, to perform linguistic analysis. They then used Lucene’s analysis pipeline to add semantics to the data using external sources such as Wikipedia in order to get better results from More Like This.

But the “funding problem” remained. Finally, they hit on a solution: they’d provide “related articles”, but using the Software as a Service model. This way, they could make the service available to other companies, and what was a one-off project became an ongoing business. Literally, the search turned from an implementation into a business opportunity, seeking new customers who are ‘more like this’.

Because of the semantics, not all companies will be able to take advantage of Canoo’s service; their system requires documents to have a significant size in order for the analysis to be accurate, though they’re working on shrinking that down. But by providing their application as a service, Canoo has expanded their market considerably. So because Canoo was able to both (a) have confidence in an open source solution that would get them close to what they needed and (b) get commercial-grade support for that product (disclosure: Canoo is a client of Lucid Imagination, a sponsor of Lucene Revolution), they were willing to take a chance on a project even though the potential client wasn’t, leading to not one project, but an ongoing business.

And that lesson I was talking about? Have faith in open source, and have faith in yourself; it can definitely pay off in the long run.

Cross-posted with Lucene Revolution Blog. Nicholas Chase is a guest blogger.This is one of a series of presentation summaries from the conference.

Here are the slides for this session.

Imagine you can see 160 years of history, all on one screen. You can zoom and pan, you can look at a particular day, you can even do a search. And when you do, the results come up not as a list, but as a heat map that shows where in history that topic appears, and how often.

That’s the application Anne Veling, BeyondTrees, and Proquest built for a prominent US newspaper.

The system covers 160 years of newspapers, with every single issue — more than 58,000! — appearing on a single canvas. When you do a search, the app sends an AJAX request that gets back a heat map specifying the color to overlay over each of the tiles, with brighter colors representing greater numbers of mentions in that particular issue.

And that’s where facets come in. After all, what is a facet anyway? It’s a count of how many results appear in a particular “bucket”. In a traditional system, you might do a search for “Indians” and get something along the lines of

...
1870 [37]
1871 [259]
1872 [135]
1873 [3]
1901 [25]
...

and so on. That’s exactly the information Veling needed for his heat map, only instead of years, each facet would represent one day, or one issue  of the newspaper.

But getting those facets posed some interesting challenges, and presented some interesting opportunities.

For one thing, this may be one of the only times in your professional life you’ll ever see a Lucene index in which absolutely nothing is stored. Because nothing is ever returned except the facet counts, nothing needs to be. (Once the user finds the issue he or she wants, the paper is returned to them through a different process.)

From there, it was partly a matter of tuning — the filterCache, set by default at 512, needed to be at least 60,000 so that the entire set of facets could be resident in memory, plus he used DocSets to reduce memory requirements — and partly a matter of knowing the data.

Rather than having each document check against all 58K+ facets to see where it belongs, he instead did a hierarchical check; first by decade, then by year, then by month, then by day. Also, because he knew that a document could belong to only one facet, he created a custom runtime collector that stops looking when it finds one. By making those changes, he was able to reduce the number of checks per document from 58560 to just 34.5. Considering that he was dealing with more than 28 million documents, it’s no surprise that he was able to increase performance by a factor of 30.

The result of all this is a beautiful system in which you can clearly see patterns such as the 1978 newspaper strike, leap years, or the fact that this renowned Sunday paper didn’t initially exist at all. You’ve probably seen those old science fiction movies where the time machine has a dial that selects years. Here, the very sleek iPad interface lets you zoom in, in an experience reminiscent of the flying function of Google earth — and instead of traveling through time, you travel through space.

Next up, Veling will work on a similar Belgian system, representing over 1.2 million books, CDs, and DVDs.

The real beauty of this project isn’t so much the actual interface (though it is beautiful) but the fact that it exists at all. This kind of innovative thinking is the direction we need to go in order to give meaning to all this information we are beginning to see.

It’s a new day, and it is great to see Lucene and Solr at the forefront of the revolution.

Cross-posted with Lucene Revolution Blog. Nicholas Chase is a guest blogger.This is one of a series of presentation summaries from the conference.

Here are the slides for this session.

With this premise in place, Steve discussed the use and integration of advanced analytics in the document-processing pipeline, focusing on the three levels to which they apply: namely, the document, sub-document, and cross-document levels.

Meta-data derivable at the document level includes identification of the language in which the document is written and the category in which is it properly placed. Steve touches on some especially interesting challenges posed by Asian languages, and mentioned the fact that this level of analysis was useful for creating document search dashboards, useful to those responsible for assessing and maintaining document search quality.

The amount of information that can be gleaned from sub-document analysis is immense. Some of the processes involved at this level include basic stemming and its cousin, lemmatization. Among the more advanced techniques are entity extraction, relationship and event extraction, sentiment analysis, and the mapping of extracted items to real-world concepts in a process called “co-reference resolution.”

Key uses of cross-document analysis include, for example, document clustering, i.e., finding a set of documents that are “more like” one another than another set would be.

An aspect of the presentation of great interest to Solr users focuses on how to integrate analytics, like those provided by Basis, into the Solr pipeline. Not surprisingly, there are a lot of ways to do this. The biggest question you need to answer is: Should I run the analytics within Solr, or should treat them as external calls?

Steve wrapped this useful talk with some approaches to both techniques, including UpdateRequest processing and a list of some tools (e.g., UIMA, GATE, and OpenPipeline) to consider when the time to implement arrives.

Cross-posted with Lucene Revolution Blog. Tony Barreca is a guest blogger.This is one of a series of presentation summaries from the conference.

Stephen Dunn and the Guardian: How being open makes them better

Stephen is from The Guardian, the second oldest newspaper in the United Kingdom. If you’ve been following along, you know that newspapers have been having a hard time of it lately, but the Guardian is in a unique (or at least very rare) position; the paper is actually owned by a trust, so short-term profits can take a back seat to long-term planning and experimentation. That gave the Guardian the freedom to experiment with their online presence.

This results in two interesting aspects of his talk. The first is technological; in 1999, when they moved online, they found that they had steeply increasing traffic. That’s good, of course, but that also meant increasing load on their database, which is not so good.

So they started doing searching through Solr, and the load on their database leveled off, even though traffic increased. So the Guardian thought, “great, what else can we move to search?” The answer turned out to be “everything”, it seems.

Basically, being able to find data using search (as opposed to using an RDBMS) enables the Guardian to go from being a publisher to being a platform. After requesting an API key, developers can basically do whatever they want with not just current Guardian data, but also past articles, curated data, and also information from a separate Politics API that records voting records and other political data.

Of course from a business standpoint, the question is, “how can this possibly be a good idea?” The answer turns out to be twofold.

First off, articles come with advertising, so even if they’re being displayed in someone else’s application, the ads are still being displayed and to a new audience, so that’s additional revenue for the Guardian. That one’s obvious.

The second reason isn’t so obvious. Basically it comes down to an acknowledgement that there are people who know more about the topics they cover than they do, and by keeping their data open, rather than locking it behind a pay wall (as some of their contemporaries have done or are doing) the Guardian makes it possible to get participation from those people. The example that he gave is the Arab Spring, where people in the region basically followed Al Jazeera and The Guardian. These people also responded to Guardian content in social media, so where their competitors might get a dozen comments, all from people in the UK, the Guardian had hundreds of tweets (or more) from people with a variety of perspectives.

The Open Platform, as they call it, also provides another way to get additional perspectives; Guardian data can be built into applications that use their Micro Apps API, which then enables those apps to be integrated back into Guardian sites and platforms (or other sites). Similarly, their API enabled them to create a WordPress plugin. That plugin then in turn enabled them to get outside subject matter experts, such as scientists, to blog for them and have other content added into the Guardian CMS.

So overall, the capabilities provided by using Lucene/Solr to search their data provides an opportunity for the Guardian to open themselves up as a platform, which in turn makes them a better publisher.

Definitely a win, and a great way to start the day.

Cross-posted with Lucene Revolution Blog. Nicholas Chase is a guest blogger.This is one of a series of presentation summaries from the conference.