Presented by Marc Sturlese, Trovit at Apache Lucene Eurocon 2011

Trovit is a global classified advertising service covering real estate, jobs and more in 27 countries worldwide. Until recently, our distributed Lucene/Solr search indexes used a customized Data Import Handler to draw data out of MySQL, but they no longer adequately handle our volumes with acceptable performance. We have moved Lucene/Solr indexes using MapReduce and came up with a new way to build indexes which is into production since months ago. Here at Trovit, we deal with many countries and different business categories, each with its own index -- and not all of them have similar size or structure.

I'll present our experience as a combined use case/tutorial, beginning with a brief introduction about the main Solr features we use at Trovit, and then move to the more complex part:

• Brief explanation of the data pipeline handled by Hadoop before our ads are indexed, with implementation details of the indexing process, deploying indexes from HDFS, etc.
• Tuning performance parameters to improve indexing speed as much as possible and keep good search performance
• Managing the effect of GC at search time as much as we can as we deal with shards
• Moving indexing time Solr features like DeDuplication to MapReduce.
• Using Solr analyzers to analyze large amounts of text outside of an indexing process

I'll also talk about how we used the phased indexing strategy to manage indexes across countries and verticals (jobs, autos, etc.) and working around limitations in SOLR-1301.

Download session slide.