My most recent article on Mahout is up at IBM developerWorks.  It is titled Apache Mahout: Scalable machine learning for everyone and is designed to walk you through using Mahout with a real email data set using Hadoop and EC2.  It also gets you up to speed on some of the new things in Mahout since I last wrote on the subject for developerWorks.

Note, I will also be giving a talk on the subject, at the San Francisco Bay Area Mahout User Meeting on Nov. 29th.  Hope to see you there!

For more details and to RSVP, please see http://sf-mahout-11-11.eventbrite.com/

For me personally, I’m on to Vancouver and ApacheCon for two primary things, besides of course the community bits that go with every ApacheCon:

1. Providing the ApacheCon’s first ever Apache Mahout training on Monday, November 7th.  It’s still not too late to sign up!
2. Giving a talk on alternative uses of Lucene/Solr other than traditional free text search (things like recommendation engines, classification, etc.)

For the 2nd item, I’m also interested in hearing from you, the user, about interesting things you’ve done with Lucene/Solr that fall outside the norm of free text search.  If you care to share, please leave a comment on this post.

I’d be remiss if I didn’t also plug several other Lucid Imagination employees who are speaking at ApacheCon as well:

1. Solr Flair by Erik Hatcher.  Erik will also be doing a 2 day Solr training class.  Registration is still open for this class as well.
2. Apache Solr: Out of the Box by Chris Hostetter

Lucid Imagination is also sponsoring the Lucene/Solr meetup on Wed. November 9th, so if you are in town, please feel free to drop by for a drink and a chat.

With that, I’ll simply say, I hope to see you in Vancouver in a few weeks!

-Grant

So, before I start into my review, let me first say congratulations to Sean, Robin, Ted, Ellen and Manning for producing such an excellent product.   The simplest praise I can give it is to put it on the same level as one of the best intro to technology books I know:  Lucene In Action.  In other words, it sets the standard by which all other Mahout books will be judged.

As for the actual book, it is broken down into 3 sections, which I like to call the “three C’s”:

1. Collaborative Filtering
2. Clustering
3. Classification

So, without further ado, let’s take a deeper look at the book in this context of the three C’s.

Collaborative Filtering

Collaborative Filtering is by far one of the most popular parts of Mahout, being used in places like Amazon and Foursquare and this section of the book, via 5 chapters,  walks you nicely through both the concepts and the practical aspects of collaborative filtering.   Chapter 2 starts by getting you up and running using the GroupLens dataset for movie recommendations.  For those unfamiliar with collaborative filtering, this makes for a nice entrance into the subject with data everyone can relate to easily.  Chapter 3 then discusses how to best model your data, while chapter 4 looks at the mechanics of actually generating recommendations from the data.

Chapters 5 and 6 then discuss the ins and outs of taking a recommendation engine into production, including details on how to scale it out using Apache Hadoop.  I found the explanation of the Hadoop based co-occurrence process (via RecommenderJob) especially useful, as I recently just committed MAHOUT-798, which uses it to build an example recommendation system based off of user interaction with email.  In fact, I relied heavily on all of the concepts in this part of the book, as I first had to extract and clean the data, then properly model it before finally running the recommendation task on EC2.

When I first got access to the MEAP for this book (quite some time ago), I did not have a lot of background in collaborative filtering and these chapters really helped fill in the practical details for me as well as provided a good foundation for the theoretical aspects behind collab. filtering.  I think this will serve others well who are looking to get started with collaborative filtering as well.

Clustering

Similar to collaborative filtering, the clustering section starts off by introducing the basic concepts and then quickly gets you up and running with an example clustering run.  Chapter 8 then gets into how best to do feature selection for clustering.  Feature selection is often one of the keys to successful clustering, so be sure to make sure you have a good grasp on the contents of the chapter before moving ahead into chapter 9, which gets into some of Mahout’s clustering algorithms.  That chapter primarily focuses on K-Means and Dirichlet, but also covers a few others.  Note, Mahout actually has a few other algorithms for clustering then the ones described, like spectral, canopy, meanshift and minhash.  Of course, some of these were added later in the book cycle, so it is hard to complain that they weren’t incorporated. Chapter 10 then covers, in my experience, one of the harder aspects of clustering, namely how to evaluate the results.  This chapter is a little bit thin, but it seems the overall field is the same, so this is not a put down on the chapter!  There simply isn’t a lot of great tools available for evaluating clustering.

Chapter 11 then adds some meat onto the bones of taking clustering to producti0n, including information on leveraging clustering in a Hadoop cluster.  Chapter 12 adds some nice concreteness to the sections by looking at clustering of real data sets from Twitter, Last.fm and Stack Overflow.  For those looking to kick the tires with some real data, be sure to check out that chapter.

Classification

Classification is very popular these days both in search and beyond, so it is great to see this set of chapters covering the topic so well in practical, accessible terms.  As you would expect, the first chapter (13) gets you up and running as well as introduces the concepts of classification.  This chapter has a great explanation of how classification works and a typical workflow for building a classifier.

Chapter 14 then delves into the details of actually training a classifier using Mahout’s Stochastic Gradient Descent algorithm as well as it’s Bayesian classifier.

The next chapter then takes a look at how best to evaluate a classifier as well as some insight into what happens when a classifier goes bad.  Be sure to check this out, as you will no doubt run into many of the issues covered.  As an aside, I couldn’t help thinking of the classic “Far Side” cartoon to the right upon reading that section heading.The penultimate classification chapter digs into the practical aspects of deploying a classifier in production, including details on working through your scale and speed requirements.  It finishes off with an example Apache Thrift based server which some may find as a useful starting point for their applications.  Finally, Mahout in Action finishes off with a Case Study of how Shop It To Me uses a Mahout classifier to provide recommendations of offers to customers.  As with any technical book, it is great to have some concrete discussion of how this stuff functions in the wild.

What’s Missing (i.e. When’s the 2nd edition coming out?)

Mahout has a number of other interesting things that are in various stages of development like frequent patternset mining, Singular Value Decomposition (feature reduction), evolutionary programming, integration libraries for input/output as well as tools for storing data in Cassandra and Mongo.  Since Mahout is developing pretty quickly, the lack of this being in the book is no fault of the authors, I’m just putting it up here so that people are aware that Mahout has more to offer, even if the three “C’s” are the most popular.

All in all, Mahout in Action is an excellent introduction to the project.  Naturally I’m biased, but, pun intended, I highly recommend the book!

Before I dig in too much, though, a special thanks to all our customers at Lucid Imagination as well as to my coworkers.  I’m coming up on 15 years out in the “real world” and I can honestly say I’ve never enjoyed what I do as much as I do here and that even accounts for the normal rough patches one goes through in any job.  As an engineer, there are few things as cool as getting to work with customers who are not only using, but pushing your work/project/product on a daily basis to do new and interesting things (I think this is a direct result of the project being Open Source, which I believe has an inherently lower cost of experimentation).  I’ve been fortunate enough to meet and talk with many people doing all kinds of things with Lucene and Solr ranging from the “mundane” of basic keyword search to those building next generation search capabilities at incredible scale.  Through it all, I’m constantly amazed at the flexibility and efficiency of Lucene and Solr.  For instance, I’ve been working with one customer now whose Solr-based solution (for the exact same content) will use ~50% less hardware and will have an index that is 1/6 the size of their FAST index all while saving them major dinero.

Speaking of Lucid, one of the highlights of the year for us that relates directly to Lucene and Solr is the launch of our enterprise version: LucidWorks Enterprise.   I like to think of it as Apache Solr with a whole lot of Lucid expertise on how to use Solr baked in and topped off with other features and functionality to make building search applications easier.

OK, time to move on to the open source projects…

1. Without a doubt, the biggest news of the year is the merging of the Lucene and Solr code base as well as the “graduation” of several subprojects to Apache Soft. Foundation Top Level Projects (TLP).  The graduating projects are Tika, Nutch, and Mahout.  We also spun Lucy (a C port) to the Incubator, where it is working on it’s own community.  These moves were primarily done to focus the project management on single code base, but they also demonstrate the project has reached a level of maturity at the ASF.  The move also has the side benefit of bringing each project higher visibility.
2. I’m particularly excited about the addition of OpenNLP to the Apache umbrella.  OpenNLP is a nice open source Java project for natural language processing that has lived at Source Forge for quite some time.  I would expect development to grow quite a bit under the ASF community based model.  Also, integrating OpenNLP with Solr and Lucene is pretty easy to do.  I would be remiss if I didn’t also give a nod to the addition of the ManifoldCF project to the ASF.  ManifoldCF will help unlock content in Sharepoint, Documentum and other repositories for users of Lucene and Solr.
3. Lucene’s trunk code base now implements our “Flex APIs”, which should allow users to have near total control over what goes in the index as well as alternate compression techniques, different scoring models, etc.  See Michael McCandless’ excellent talk at Lucene Revolution for more details.
4. With all the location aware devices and capabilities on the market, geo-spatial search is a hot topic and Lucene and Solr have been adding quite a bit of capabilities in this regard with the ability to filter, boost and sort results based on location information in documents.  See Solr’s Spatial Search Wiki page for more info as well as several of my past blog posts.
5. Of course, everyone was a buzz about the cloud this year.  For Solr, this translates into greater efforts to make Solr easier to scale to very large installations (100s to 1000s of nodes and billions and billions of documents) via the Solr Cloud project that Yonik Seeley and Mark Miller have been spearheading.
6. On the user side, one of the biggest pieces of buzz this year related to Lucene was the migration of Twitter search to Lucene.  At 1 billion queries per day and 50 million posts per day (all indexed and searchable in near real time), Twitter’s search system certainly has it’s work cut out for itself.  However, as Michael Busch outlined at Lucene Revolution, Apache Lucene was up to the task!  Naturally, there were lots of other companies that migrated to Solr and Lucene as well.  Have you shared your use case?

Well, I’ve no doubt missed a bunch of other things, but those items, to me, are some of the bigger highlights.  Looking forward, there are some other exciting things coming to Lucene and Solr.  In particular, I’m working on adding language identification, related searches and point in polygon filtering to Solr.  I would also expect we will release Lucene/Solr 3.1 fairly soon, too, but you can’t pin me down on a date just yet.

Here’s hoping you all have a Happy Holidays and a Happy New Year!

As for the talks, I gave the Keynote on using open source tools like Apache Solr and Mahout to deliver intelligent applications (slides — really should be a PPT so you can see the animations) on Monday first thing in the morning and I felt it went pretty well, but I’ll let others be the judge (videos should be online soon).  The rest of the day, I spent going in and out of the various tracks.  The Lucene track was very well done, with good talks by: Uwe Schindler and Simon Willnauer on the State of Lucene, Robert Muir on Finite State queries in Lucene; Michael Busch on Real Time Search at Twitter, Jukka Zitting on Tika and Andrzej Bialecki on Nutch. See Berlinbuzzwords: Links To Slides for all the slides (not all are available just yet).

I also went to a variety of the Hadoop and NoSQL talks.  Lots of people in the NoSQL talks making pitches on why their approach is best, which is very helpful in determining what tool to use at the appropriate time.  I still, however, can’t shake the feeling that one could take the new Solr Cloud stuff, a dead simple schema (id and one or two simple fields), and have a large scale distributed key-value storage that overcomes almost all of the limitations of many of the NoSQL technologies (ad-hoc queries, range queries, search within the values, extendability) with minimal overhead of indexing (which can be greatly reduced by using either literals or very simple analysis).  Not only that, Lucene/Solr already is “document-centric” and I’ve seen it scale to billions of documents with high availability and high QPS and that was using “real” documents (i.e. articles, etc.), not simple key-value pairs, so I can’t help but feel like simple key-value pairs would be even faster and more scalable.  In other words, Lucene isn’t just for text search.  Naturally, this is just a thought at this point, I haven’t tried testing it just yet. Also, once the new real time stuff is in Lucene, I think it will be even faster.

At any rate, the best thing about the conference was the fact that it shows the eagerness for new solutions to large scale solutions that cost less money than the sturdy old database.

Again, congrats to Isabel and team for a well executed conference in a great city and at a great venue.  If you are interested in more on the Lucene portion of the conference, make sure you come visit us in Boston for Lucene Revolution!

One of the things I really like about this agenda is it is a great mix of basics, use cases from all over the search map (CMS, news, social media, advertising), business decisions (see last list and next list) and advanced topics (NLP, collab filtering, machine learning, advanced visualization, multilingual).  Moreover, the content, even though it is centered in Lucene, goes well beyond just being about Lucene and is really about search, in all of it’s power and glory.  It’s about real users with real needs getting real problems solved using the Lucene ecosystem.  Oh, and by the way, those users are doing it at scale!  Big scale.

That’s powerful stuff, because, in case you hadn’t noticed (shh, it’s our little secret) there is a revolution going on in search.  (Funny how that line coincides with Lucid’s frontman, Eric Gries, giving a talk titled “The Search Revolution”)

Are you a part of the revolution?  See you in Prague in mid-May.

In the end, though, what does all this mean for the users of the Lucene ecosystem?  On one hand, some of the move is just shuffling around of domain names, mailing lists and SVN source trees, but on the other hand, the moves are symbolic and represent a project reaching a level of maturity and self determination, not to mention critical mass and brand awareness.  Thus, in my mind, all of these moves are good things for Lucene as well as the associated projects that are spinning out.  As far as the actual code, I think users will still see the same high quality contributions and products coming out of Apache (aside: Lucid Imagination will still be business as usual in regards to these moves) as well as much more focus within the Project Management Committee (PMC) on the specific project.

Which brings me to a bit more on my view of the merge of Lucene and Solr.  I think we are already seeing the fruits of the merge for both Lucene and Solr (I know my open source life is easier already).  For instance, much of the analyzer code is going to being combined from Solr and Lucene to provide a single coherent analyzer library.  This is great news for people who have been using Lucene and pulling in Solr analyzers and is good for Solr users because it now has many more people keeping an eye on Solr’s analyzers as well as new Lucene analyzers showing up sooner (things like the WordDelimiterFilter, etc.)  Another example is the spatial work that we’ve been working pretty heavily on (see SOLR-773, SOLR-1568 and LUCENE-2350).  With the combination of the two development projects, it is now much easier for us to make sure there is a single, integrated way of delivering spatial search across both the Java API and the Solr REST-like API.

Moreover, in the short run, existing Lucene and Solr users should notice no difference in terms of the products, user communities and the like.  In the long run, it should make for less repeated code, faster integration, more test coverage and a larger, cohesive development team as well as more of Solr’s capabilities available in pure library form as well as many of Lucene’s cutting edge capabilities appearing sooner in Solr (flexible indexing and scoring, etc.)

Wrapping up, congrats to Lucene and all of the new top level projects!

Apache Mahout <http://lucene.apache.org/mahout> 0.3 has been released and is
now available for public
download at http://www.apache.org/dyn/closer.cgi/lucene/mahout

Up-to-date maven artifacts can be found in the Apache repository at

https://repository.apache.org/content/repositories/releases/org/apache/mahout/

Apache Mahout is a subproject of Apache Lucene with the goal of
delivering scalable machine learning algorithm implementations under
the Apache license. http://www.apache.org/licenses/LICENSE-2.0

Mahout is a machine learning library meant to scale: Scale in terms of
community to support anyone interested in using machine learning.
Scale in terms of business by providing the library under a
commercially friendly, free software license. Scale in terms of
computation to the size of data we manage today.

Built on top of the powerful map/reduce paradigm of the Apache Hadoop
project, Mahout lets you solve popular machine learning problem
settings like clustering, collaborative filtering and classification
over Terabytes of data over thousands of computers.

Implemented with scalability in mind the latest release brings many
performance optimizations so that even in a single node setup the
library performs well.

The complete changelist can be found here:

http://issues.apache.org/jira/browse/MAHOUT/fixforversion/12314281

New Mahout 0.3 features include:

* New math and collections modules based on the high performance Colt
library.
* Faster Frequent Pattern Growth (FPGrowth) using FP-bonsai pruning
* Parallel Dirichlet process clustering (a model-based clustering
algorithm)
* Parallel co-occurrence based recommender
* Parallel text document to vector conversion using LLR based ngram
generation
* Parallel Lanczos SVD (Singular Value Decomposition) solver
* Shell scripts for easier running of algorithms, utilities and examples
* …and much much more: code cleanup, many bug fixes and
performance improvements

Getting started: New to Mahout?

* Download Mahout at http://www.apache.org/dyn/closer.cgi/lucene/mahout
* Check out the Quick start: http://cwiki.apache.org/MAHOUT
* Read the Mahout Wiki: http://cwiki.apache.org/MAHOUT
* Join the community by subscribing to mahout-user@lucene.apache.org
* Give back: http://www.apache.org/foundation/getinvolved.html
* Consider adding yourself to the power by Wiki
page:http://cwiki.apache.org/MAHOUT/poweredby.html

For more information on Apache Mahout, see http://lucene.apache.org/mahout