For more details and to RSVP, please see http://sf-mahout-11-11.eventbrite.com/

For me personally, I’m on to Vancouver and ApacheCon for two primary things, besides of course the community bits that go with every ApacheCon:

1. Providing the ApacheCon’s first ever Apache Mahout training on Monday, November 7th.  It’s still not too late to sign up!
2. Giving a talk on alternative uses of Lucene/Solr other than traditional free text search (things like recommendation engines, classification, etc.)

For the 2nd item, I’m also interested in hearing from you, the user, about interesting things you’ve done with Lucene/Solr that fall outside the norm of free text search.  If you care to share, please leave a comment on this post.

I’d be remiss if I didn’t also plug several other Lucid Imagination employees who are speaking at ApacheCon as well:

1. Solr Flair by Erik Hatcher.  Erik will also be doing a 2 day Solr training class.  Registration is still open for this class as well.
2. Apache Solr: Out of the Box by Chris Hostetter

Lucid Imagination is also sponsoring the Lucene/Solr meetup on Wed. November 9th, so if you are in town, please feel free to drop by for a drink and a chat.

With that, I’ll simply say, I hope to see you in Vancouver in a few weeks!

-Grant

So, before I start into my review, let me first say congratulations to Sean, Robin, Ted, Ellen and Manning for producing such an excellent product.   The simplest praise I can give it is to put it on the same level as one of the best intro to technology books I know:  Lucene In Action.  In other words, it sets the standard by which all other Mahout books will be judged.

As for the actual book, it is broken down into 3 sections, which I like to call the “three C’s”:

1. Collaborative Filtering
2. Clustering
3. Classification

So, without further ado, let’s take a deeper look at the book in this context of the three C’s.

Collaborative Filtering

Collaborative Filtering is by far one of the most popular parts of Mahout, being used in places like Amazon and Foursquare and this section of the book, via 5 chapters,  walks you nicely through both the concepts and the practical aspects of collaborative filtering.   Chapter 2 starts by getting you up and running using the GroupLens dataset for movie recommendations.  For those unfamiliar with collaborative filtering, this makes for a nice entrance into the subject with data everyone can relate to easily.  Chapter 3 then discusses how to best model your data, while chapter 4 looks at the mechanics of actually generating recommendations from the data.

Chapters 5 and 6 then discuss the ins and outs of taking a recommendation engine into production, including details on how to scale it out using Apache Hadoop.  I found the explanation of the Hadoop based co-occurrence process (via RecommenderJob) especially useful, as I recently just committed MAHOUT-798, which uses it to build an example recommendation system based off of user interaction with email.  In fact, I relied heavily on all of the concepts in this part of the book, as I first had to extract and clean the data, then properly model it before finally running the recommendation task on EC2.

When I first got access to the MEAP for this book (quite some time ago), I did not have a lot of background in collaborative filtering and these chapters really helped fill in the practical details for me as well as provided a good foundation for the theoretical aspects behind collab. filtering.  I think this will serve others well who are looking to get started with collaborative filtering as well.

Clustering

Similar to collaborative filtering, the clustering section starts off by introducing the basic concepts and then quickly gets you up and running with an example clustering run.  Chapter 8 then gets into how best to do feature selection for clustering.  Feature selection is often one of the keys to successful clustering, so be sure to make sure you have a good grasp on the contents of the chapter before moving ahead into chapter 9, which gets into some of Mahout’s clustering algorithms.  That chapter primarily focuses on K-Means and Dirichlet, but also covers a few others.  Note, Mahout actually has a few other algorithms for clustering then the ones described, like spectral, canopy, meanshift and minhash.  Of course, some of these were added later in the book cycle, so it is hard to complain that they weren’t incorporated. Chapter 10 then covers, in my experience, one of the harder aspects of clustering, namely how to evaluate the results.  This chapter is a little bit thin, but it seems the overall field is the same, so this is not a put down on the chapter!  There simply isn’t a lot of great tools available for evaluating clustering.

Chapter 11 then adds some meat onto the bones of taking clustering to producti0n, including information on leveraging clustering in a Hadoop cluster.  Chapter 12 adds some nice concreteness to the sections by looking at clustering of real data sets from Twitter, Last.fm and Stack Overflow.  For those looking to kick the tires with some real data, be sure to check out that chapter.

Classification

Classification is very popular these days both in search and beyond, so it is great to see this set of chapters covering the topic so well in practical, accessible terms.  As you would expect, the first chapter (13) gets you up and running as well as introduces the concepts of classification.  This chapter has a great explanation of how classification works and a typical workflow for building a classifier.

Chapter 14 then delves into the details of actually training a classifier using Mahout’s Stochastic Gradient Descent algorithm as well as it’s Bayesian classifier.

The next chapter then takes a look at how best to evaluate a classifier as well as some insight into what happens when a classifier goes bad.  Be sure to check this out, as you will no doubt run into many of the issues covered.  As an aside, I couldn’t help thinking of the classic “Far Side” cartoon to the right upon reading that section heading.The penultimate classification chapter digs into the practical aspects of deploying a classifier in production, including details on working through your scale and speed requirements.  It finishes off with an example Apache Thrift based server which some may find as a useful starting point for their applications.  Finally, Mahout in Action finishes off with a Case Study of how Shop It To Me uses a Mahout classifier to provide recommendations of offers to customers.  As with any technical book, it is great to have some concrete discussion of how this stuff functions in the wild.

What’s Missing (i.e. When’s the 2nd edition coming out?)

Mahout has a number of other interesting things that are in various stages of development like frequent patternset mining, Singular Value Decomposition (feature reduction), evolutionary programming, integration libraries for input/output as well as tools for storing data in Cassandra and Mongo.  Since Mahout is developing pretty quickly, the lack of this being in the book is no fault of the authors, I’m just putting it up here so that people are aware that Mahout has more to offer, even if the three “C’s” are the most popular.

All in all, Mahout in Action is an excellent introduction to the project.  Naturally I’m biased, but, pun intended, I highly recommend the book!

And while I wasn’t around in the beginning, I thought I would offer up some (little) known tidbits, links, etc. about Lucene as an ode to the search library that has significantly changed the search world, as well as my own career:

1. Lucene was Doug’s way of learning Java!  How’s that for a start?  It took him 3 months, working 2 days a week to crank out the first version.
2. At the time, some commercial search engines could not do incremental updates of the index, meaning you had to re-index all your documents anytime you had an update.  Lucene has always had an incremental model, all the way through to today’s Near Real Time features that power the likes of Twitter at 1 billion+ searches and 100M+ new documents per day.
3. Field myField = Field.Text(“foo”, “bar”); anyone?  Or how about Field myField = Field.UnIndexed(“foo”, “bar”);
4. Back then, Lucene had it’s own PorterStemmer, now we just use Snowball.
5. Only 1 of the original committers still remains somewhat active.
6. Read the old FAQ!  True as it ever was.  (Mostly)
7. Lucene 2.3 drastically improved indexing performance thanks to a thorough overhaul of the innards while barely affecting the API.  4.0 will blow the doors off of previous versions in terms of speed and efficiency.
8. Lucene is Doug’s wife’s middle name.
9. Lucene has evolved from offering a single vector space scoring model to one that now offers plug-n-play ranking (BM25 anyone?)
10. Lucene is ubiquitous.  It powers search on everything from mobile devices to web scale engines.  I’ve seen indexes as small as 15% of the original content.  I’ve also seen indexes grow to several billion documents in size.  Lucene has been used as a caching store, an ORM, a cross language search engine, the guts of the popular Solr search server, the retrieval engine for IBM’s Watson as well as several commercial search engines and pretty much everything in between.
11. Did you know Apache Hadoop started as a subproject of Lucene?  Doug Cutting and Mike Cafarella first built out Hadoop in order to scale out indexing for the Apache Nutch project.  From there it was spun out to be a top level ASF project and has gone on to be the de facto choice for large scale distributed processing, much like Lucene is the de facto choice for search!  Lucene has also spun out Mahout, Tika, Lucene.NET and Lucy!

As for how Lucene’s impacted me?  In 2004, I took a job at the Center for Natural Language Processing at Syracuse University working for Dr. Liz Liddy.  My job was to build an Arabic-English cross language search engine.  Within a day or two of starting, Ozgur Yilmazel (my boss at the time) said something to the effect of “we’ll be using Lucene for the implementation.  Go learn it.”  Digging in, I quickly needed a couple of features, the biggest one being Term Vectors, so I updated a patch from an earlier version of Lucene and managed to convince the committers at the time to commit it.  From there, I kept supplying patches.  Eventually, I was asked to be a committer.  Some time after that, Yonik Seeley and Marc Krellenstein approached a bunch of the committers about starting a company and here I am today at the company we (Erik, Yonik, Marc and I) founded back in 2007, Lucid Imagination.  I feel fortunate to have the opportunity to work on hard problems in an interesting field and for that, Lucene, in no small part, I thank  you.

But enough of my self-indulgence, how has Lucene impacted you?  When did you first start using it?  What’s your biggest index or fastest QPS?   What ways have you used Lucene beyond that of a search engine?  Leave a comment and let us know.

Happy 10th Anniversary, Lucene!

How, then, can you know how much you know about search with Solr and Lucene? In the world of Apache open source, there’s a clear meritocracy of peer review: contributors, committers, and active membership in the PMC. In theory, it’s a distinction anyone of sufficient talent and single-minded focus can achieve — just like anyone of sufficient talent and single-minded focus can make it to the NBA, or win the Nobel prize, or join the New York Philharmonic.

So you probably know your stuff if you’ve won the Nobel prize, made the NBA, or played the solo for Mozart’s Clarinet Concerto at Carnegie Hall, or you’re a Lucene/Solr contributor-or-committer. But what if you have not done any of those things, how do you know you know? Equally important, how do your peers or potential employers know how well you know your open source search stuff?

While there are more professional basketball players than Lucene/Solr committers, there are many, many more capable, talented, experienced Solr/Lucene application developers who are not going to ‘go pro’ in the Apache meritocracy. And the demand for Solr application development skills is exploding as interest and uptake of the leading open source application development technology spread like wildfire through organizations large and small. (Lucene Revolution, May 25-26 in San Francisco, will be packed with these people — sign up today if you haven’t already. And read on for another special  opportunity at Lucene Revolution).

It’s exactly for the broad base of interested, committed search application developers that today we’re introducing the Lucid Certified Apache Solr/Lucene Developer Program; a certification exam designed to benchmark development skills and experience in building applications with Apache Solr.

Designed with Prometric and a team of subject matter experts comprised of Apache Lucene/Solr committers, developers, and trainers, the test is designed to rigorously assess a broad base of search skills and experience, and provide the closest reasonable approximation possible to a standard measure of  search skills and experience.  It’s delivered via Prometric.com, consists of multiple choice questions, and costs $250. The test reflects a carefully selected, broad range of topics intended to reflect the real-world challenges and landscape of search application development problems, which you can see here.

Eric Pugh, who wrote the book on Solr, says this:

“I expect that the Lucid Imagination certification will quickly become the gold standard benchmark for whether someone who claims Solr and Lucene expertise truly possesses it. Oftentimes, a buyer of services has to take the leap of faith from sales pitch to execution that the knowledge is truly there. This certification can show, without a doubt, that the holder truly has the knowledge required to deliver a successful Solr/Lucene implementation. In the open source world, there are very few marks of authenticity: committer status, published author, and now the Lucid Solr certification. Just as the CPA certification shows a high level of knowledge and ability in the accounting industry, the Lucid Imagination Solr certification demonstrates unquestionable knowledge and experience in successful Solr/Lucene search engine implementation.”

It’s important to be clear about what the certification is not:

• It’s not easy: don’t expect to take your first Solr course one day and pass the exam the next.
• It’s not a substitute for experience: if you’ve only built one Solr application, earlier this morning, using the wiki demo that runs locally in your browser, you won’t pass.
• It’s not a substitute for training: taking a class from an expert may not be sufficient, but it will really help (and we offer the most professional-grade courses available; did I mention Lucene Revolution has classes available, too?)
• It’s not a casual conceptual overview: expect to answer detailed questions on everything from Lucene fundamentals to Solr debug output.
• It’s not a simple checklist of facts: you’ll have to demonstrate judgement calls in identifying correct answers to topic areas tied to searching, indexing, deployment, data source types, etc.

Testing as a pedagogical method — a mechanism for driving learning — is not the be-all-end-all of education (you probably didn’t think highly  of classmates who asked the teacher, “Will this be on the test?”). But it turns out that tests can have a salutary impact on acquiring and retaining knowledge, according to a recent article in Science.

We expect that this test will help level the playing field for a broad range of application developers to acquire and prove their Solr/Lucene application development skills — and help employers who want to take full advantage of the best open-source search technology on the planet find the men and women who have the stuff to do it.

If you’re coming to Lucene Revolution, the exam will be offered there for free — a savings of $250 over the regular price. Details are here.

Here’s the release announcements as just sent to the mailing lists:

March 2011, Apache Lucene 3.1 available
The Lucene PMC is pleased to announce the release of Apache Lucene 3.1.

This release contains numerous bug fixes, optimizations, and
improvements, some of which are highlighted below.  The release
is available for immediate download at http://www.apache.org/dyn/closer.cgi/lucene/java.
See the CHANGES.txt
file included with the release for a full list of details.

Lucene 3.1 Release Highlights

* Numerous performance improvements: faster exact PhraseQuery; merging
 favors segments with deletions; primary key lookup is faster;
 IndexWriter.addIndexes(Directory[]) uses file copy instead of
 merging; various Directory performance improvements; compound file
 is dynamically turned off for large segments; fully deleted segments
 are dropped on commit; faster snowball analyzers (in contrib);
 ConcurrentMergeScheduler is more careful about setting priority of
 merge threads.

* ReusableAnalyzerBase makes it easier to reuse TokenStreams
 correctly.

* Improved Analysis capabilities: Improved Unicode support, including
 Unicode 4, more friendly term handling (CharTermAttribute), easier
 object reuse and better support for protected words in lossy token
 filters (e.g. stemmers).

* ConstantScoreQuery now allows directly wrapping a Query.

* IndexWriter is now configured with a new separate builder API,
 IndexWriterConfig.  You can now control IndexWriter's previously
 fixed internal thread limit by calling setMaxThreadStates.

* IndexWriter.getReader is replaced by IndexReader.open(IndexWriter).
 In addition you can now specify whether deletes should be resolved
 when you open an NRT reader.

* MultiSearcher is deprecated; ParallelMultiSearcher has been
 absorbed directly into IndexSearcher.

* On 64bit Windows and Solaris JVMs, MMapDirectory is now the
 default implementation (returned by FSDirectory.open).
 MMapDirectory also enables unmapping if the JVM supports it.

* New TotalHitCountCollector just counts total number of hits.

* ReaderFinishedListener API enables external caches to evict entries
 once a segment is finished.

March 2011, Apache Solr 3.1 available

The Lucene PMC is pleased to announce the release of Apache Solr 3.1.

This release contains numerous bug fixes, optimizations, and
improvements, some of which are highlighted below.  The release is
available for immediate download at http://www.apache.org/dyn/closer.cgi/lucene/solr.
See the CHANGES.txt file included with the release for a full list of
details as well as instructions on upgrading.

What's in a Version? 

The version number for Solr 3.1 was chosen to reflect the merge of
development with Lucene, which is currently also on 3.1.  Going
forward, we expect the Solr version to be the same as the Lucene
version.  Solr 3.1 contains Lucene 3.1 and is the release after Solr 1.4.1.

Solr 3.1 Release Highlights

* Numeric range facets (similar to date faceting).

* New spatial search, including spatial filtering, boosting and sorting capabilities.

* Example Velocity driven search UI at http://localhost:8983/solr/browse

* A new termvector-based highlighter

* Extend dismax (edismax) query parser which addresses some
 missing features in the dismax query parser along with some
 extensions.

* Several more components now support distributed mode:
 TermsComponent, SpellCheckComponent.

* A new Auto Suggest component.

* Ability to sort by functions.

* JSON document indexing

* CSV response format

* Apache UIMA integration for metadata extraction

* Leverages Lucene 3.1 and it's inherent optimizations and bug fixes
 as well as new analysis capabilities.

* Numerous improvements, bug fixes, and optimizations.

1. The Solr example is just that, an example.  If you haven’t cleaned up the example fields and other stuff it tells me you haven’t thought much about your domain and how best to represent it in search.  I’d say it’s safe to say a good chunk of our consulting work is about helping people better understand how to represent their domain in the search engine.  Take the time to think about what field types you are using, what analysis that field type does and how best to represent that content for searching, faceting, sorting, etc.  For instance, Porter/Snowball stemming is often quite aggressive, is that really what you want?  In other words, know your analysis.
2. Be prepared to state what each field is and why it is indexed/stored/multivalued/etc.  See http://wiki.apache.org/solr/FieldOptionsByUseCase.  This holds true for Lucene, too.
3. Likewise in the solrconfig.xml, remove ununsed/example RequestHandlers, etc.  Also, some of the configurations of Request Handlers, specifically the Spelling one, are for demo purposes.  Read the comments in the file to make sure you understand how to use it.  Cleaning up the config and the schema will make it easier to maintain these as well as help getting new devs up to speed.
4. Lucene users: faceting by loading the values from stored fields and counting them for every search is a no-no performance-wise.
5. Really large values for the Solr cache sizes is (usually) considered harmful.  Be prepared to justify why you have a filterCache size of 500K items.  Bigger isn’t always better.
6. Ad hoc relevance testing is pretty close to having no relevance testing.  If you don’t know what queries work, how do you know if your users are succeeding?
7. You should always know your top X queries, top Y query terms and any queries that return zero results, at least on a daily or weekly basis.  LucidWorks Enterprise builds this into the dashboard if you’d like to save yourself some time writing the log analysis code and still want to leverage all the power of Lucene/Solr.   FWIW, By you, I mean anyone on your team who has a stake in search, including engineers, business owners and QA.  Not knowing what your users are asking is a recipe for poor search.
8. NFS = bad for search (usually).  It’s all right to store the index there, but you shouldn’t read/write from it (it will work, it just won’t perform, in my experience)
9. Buy Lucene in Action.  Read it.
10. In Solr, the fastest way to index is by using multiple client threads sending batches of documents at a time.  In Lucene, the IndexWriter is thread-safe, so share it across threads as well for writes.
11. Bonus: Don’t re-invent the wheel.  I’ve seen a lot of good Lucene applications and a fair number of poorly written ones.  I’ve also seen a lot of Lucene applications that look just like Solr, but aren’t nearly as well written or as well tested.  Ask yourself, do I really need to maintain low-level Lucene code for managing IndexReaders?  Do I really need to implement my own Faceting code?  My own Replication/distributed search?  If not, consider Solr.  It really isn’t that hard to switch.  If you answer yes, great, I’d love to hear your use case.  In either case, we’d be happy to review your situation and give you best practices, all the way down to the code level, if need be.

Like I said, nothing too earth shattering, but I do come across many of these issues on a regular basis.  Also, note these are based on my experience, your mileage may vary.  Naturally, I’d love to hear your tips as well!

Both releases fix bugs in the previous versions:

• 2.9.4 is a bugfix release for the Lucene Java 2.x series, based on Java 1.4.
• 3.0.3 has the same bug fix level but is for the Lucene Java 3.x series, based on Java 5.

New users of Lucene are advised to use version 3.0.3 for new developments, because it has a clean, type-safe API.

This release contains numerous bug fixes and improvements since 2.9.3 / 3.0.2, including:

• a memory leak in IndexWriter exacerbated by frequent commits
• a file handle leak in IndexWriter when near-real-time readers are opened with compound file format enabled
• a rare index corruption case on disk full
• NumericRangeQuery / NumericRangeFilter sometimes returning incorrect results with bounds near Long.MIN_VALUE and Long.MAX_VALUE
• various thread safety issues
• Lucene 2.9.4 can now also read indexes created by 3.0.x

Both releases are fully compatible with the corresponding previous versions. We strongly recommend upgrading to 2.9.4 if you are using 2.9.x; and to 3.0.3 if you are using 3.0.x.

See 3.0.3 CHANGES and 2.9.4 CHANGES for details. Binary and source distributions are available here. Maven artifacts are available here.

We are still in the very early stages of moving things over from Sourceforge, but do keep your eyes on the Apache Incubator site and the OpenNLP site for news of when the new site is live and when the code has officially moved.