For more details and to RSVP, please see http://sf-mahout-11-11.eventbrite.com/

So, before I start into my review, let me first say congratulations to Sean, Robin, Ted, Ellen and Manning for producing such an excellent product.   The simplest praise I can give it is to put it on the same level as one of the best intro to technology books I know:  Lucene In Action.  In other words, it sets the standard by which all other Mahout books will be judged.

As for the actual book, it is broken down into 3 sections, which I like to call the “three C’s”:

1. Collaborative Filtering
2. Clustering
3. Classification

So, without further ado, let’s take a deeper look at the book in this context of the three C’s.

Collaborative Filtering

Collaborative Filtering is by far one of the most popular parts of Mahout, being used in places like Amazon and Foursquare and this section of the book, via 5 chapters,  walks you nicely through both the concepts and the practical aspects of collaborative filtering.   Chapter 2 starts by getting you up and running using the GroupLens dataset for movie recommendations.  For those unfamiliar with collaborative filtering, this makes for a nice entrance into the subject with data everyone can relate to easily.  Chapter 3 then discusses how to best model your data, while chapter 4 looks at the mechanics of actually generating recommendations from the data.

Chapters 5 and 6 then discuss the ins and outs of taking a recommendation engine into production, including details on how to scale it out using Apache Hadoop.  I found the explanation of the Hadoop based co-occurrence process (via RecommenderJob) especially useful, as I recently just committed MAHOUT-798, which uses it to build an example recommendation system based off of user interaction with email.  In fact, I relied heavily on all of the concepts in this part of the book, as I first had to extract and clean the data, then properly model it before finally running the recommendation task on EC2.

When I first got access to the MEAP for this book (quite some time ago), I did not have a lot of background in collaborative filtering and these chapters really helped fill in the practical details for me as well as provided a good foundation for the theoretical aspects behind collab. filtering.  I think this will serve others well who are looking to get started with collaborative filtering as well.

Clustering

Similar to collaborative filtering, the clustering section starts off by introducing the basic concepts and then quickly gets you up and running with an example clustering run.  Chapter 8 then gets into how best to do feature selection for clustering.  Feature selection is often one of the keys to successful clustering, so be sure to make sure you have a good grasp on the contents of the chapter before moving ahead into chapter 9, which gets into some of Mahout’s clustering algorithms.  That chapter primarily focuses on K-Means and Dirichlet, but also covers a few others.  Note, Mahout actually has a few other algorithms for clustering then the ones described, like spectral, canopy, meanshift and minhash.  Of course, some of these were added later in the book cycle, so it is hard to complain that they weren’t incorporated. Chapter 10 then covers, in my experience, one of the harder aspects of clustering, namely how to evaluate the results.  This chapter is a little bit thin, but it seems the overall field is the same, so this is not a put down on the chapter!  There simply isn’t a lot of great tools available for evaluating clustering.

Chapter 11 then adds some meat onto the bones of taking clustering to producti0n, including information on leveraging clustering in a Hadoop cluster.  Chapter 12 adds some nice concreteness to the sections by looking at clustering of real data sets from Twitter, Last.fm and Stack Overflow.  For those looking to kick the tires with some real data, be sure to check out that chapter.

Classification

Classification is very popular these days both in search and beyond, so it is great to see this set of chapters covering the topic so well in practical, accessible terms.  As you would expect, the first chapter (13) gets you up and running as well as introduces the concepts of classification.  This chapter has a great explanation of how classification works and a typical workflow for building a classifier.

Chapter 14 then delves into the details of actually training a classifier using Mahout’s Stochastic Gradient Descent algorithm as well as it’s Bayesian classifier.

The next chapter then takes a look at how best to evaluate a classifier as well as some insight into what happens when a classifier goes bad.  Be sure to check this out, as you will no doubt run into many of the issues covered.  As an aside, I couldn’t help thinking of the classic “Far Side” cartoon to the right upon reading that section heading.The penultimate classification chapter digs into the practical aspects of deploying a classifier in production, including details on working through your scale and speed requirements.  It finishes off with an example Apache Thrift based server which some may find as a useful starting point for their applications.  Finally, Mahout in Action finishes off with a Case Study of how Shop It To Me uses a Mahout classifier to provide recommendations of offers to customers.  As with any technical book, it is great to have some concrete discussion of how this stuff functions in the wild.

What’s Missing (i.e. When’s the 2nd edition coming out?)

Mahout has a number of other interesting things that are in various stages of development like frequent patternset mining, Singular Value Decomposition (feature reduction), evolutionary programming, integration libraries for input/output as well as tools for storing data in Cassandra and Mongo.  Since Mahout is developing pretty quickly, the lack of this being in the book is no fault of the authors, I’m just putting it up here so that people are aware that Mahout has more to offer, even if the three “C’s” are the most popular.

All in all, Mahout in Action is an excellent introduction to the project.  Naturally I’m biased, but, pun intended, I highly recommend the book!

Apache Mahout 0.2 has been released and is now available for public
download at http://www.apache.org/dyn/closer.cgi/lucene/mahout

Apache Mahout is a subproject of Apache Lucene with the goal
of delivering scalable machine learning algorithm implementations
under the Apache license. http://www.apache.org/licenses/LICENSE-2.0
Scale in terms of computation to the
size of data you manage today.  Scale in terms of community to support anyone
interested in using machine learning. Scale
in terms of business by providing the library under a commercially
friendly, free software license.

Built on top of the powerful map/reduce paradigm of the Apache Hadoop
project, Mahout’s goal is to solve popular machine learning problems
like clustering, collaborative filtering and classification
over extremely large data sets over thousands of computers.

Up to date maven artifacts can be found in the Apache repository at

https://repository.apache.org/content/repositories/releases/org/apache/mahout/

The complete changelist can be found here:

http://issues.apache.org/jira/browse/MAHOUT/fixforversion/12313278

New Mahout 0.2 features include

- Major performance enhancements in Collaborative Filtering,
Classification and Clustering
- New: Latent Dirichlet Allocation(LDA) implementation for topic
modelling
- New: Frequent Itemset Mining for mining top-k patterns from a list
of transactions
- New: Decision Forests implementation for Decision Tree classification
(In Memory & Partial Data)
- New: HBase storage support for Naive Bayes model building and
classification
- New: Generation of vectors from Text documents for use with Mahout
Algorithms
- Performance improvements in various Vector implementations
- Tons of bug fixes and code cleanup

Getting started: New to Mahout?

1) Download Mahout at http://www.apache.org/dyn/closer.cgi/lucene/mahout
2) Check out the Quick start:

http://cwiki.apache.org/MAHOUT/quickstart.html

3) Read the Mahout Wiki: http://cwiki.apache.org/MAHOUT
4) Join the community by subscribing to mahout-user@lucene.apache.org
5) Give back: http://www.apache.org/foundation/getinvolved.html (optional, but much appreciated!)
6) Consider adding yourself to the power by Wiki page:

http://cwiki.apache.org/MAHOUT/poweredby.html

For more information on Apache Mahout, see

http://lucene.apache.org/mahout

For those wanting to read more, check out my IBM developerWorks article on Apache Mahout, as well.

First, by choosing an open backbone like Lucene and Solr, you are free to plugin the best tool for the job; proprietary solutions often limit you to their own tools and their implementation.  Let’s face it, we can’t be good at everything, so it makes sense to be able to plug in the best of breed for something that isn’t a core competency.  For example, one could choose OpenNLP or Teragram or any other commercial vendor for these capabilities.  Solr, especially, makes it simple to plugin these capabilities through its well defined plugin architecture.  (By the way, for almost every capability out there in this realm, there is an open source alternative that warrants investigation.)

Second, intelligent search–in other words, search that goes beyond simple keyword capabilities–is the leading edge of the field and is being adopted in more and more products, just as Dr. Schabes recommends.  Whether it is intelligent query parsing, better faceting and discovery capabilities or integration with natural language processing (NLP) tools for NER (Named Entity Recognition), sentiment analysis and relationship discovery, the companies making a difference in search are those that intelligently bring together a variety of approaches to solve the problem at hand.  I believe Lucene,  Solr and open source are uniquely positioned to fuel intelligent search because they drive down the cost of experimentation simply because it takes effort to get this stuff right, much of it due to the need to understand your domain and how to translate it into a good user experience.  Furthermore, open source lets you cost effectively fill in your infrastructure and conserve your precious resources for your core competencies.  Why would you pay millions of dollars for a search engine that implements a vector space retrieval model (which most of the commercial vendors do) when you can get the same thing from Lucene for free?  If you suspect that you think Lucene isn’t as good, think again; there’s a reason it is used at the likes of Apple, AOL, Comcast, CNET, Viacom and thousands of others.  If you like bells and whistles and knowing there is a company behind your chosen solution, I’ll do you three better:  with Lucene and Solr you not only get 1) a company that offers support, training, professional services, and bells and whistles, you also get 2) the very large Apache community of users as well who constantly use/test/fix/improve the software and 3) all of the source code,  completely unencumbered, so you are free to change it as you see fit.

Finally, you get to choose whether you even need a particular capability.  On more than one occasion, I have been involved in replacement of a proprietary search package so bloated with unused add-ons that the Solr installation, containing only the required functions, needed an index that was a mere fraction of the size of the proprietary solution, resulting in:

• Less hardware to achieve the same throughput
• Less operations costs — more hardware = more hardware failures
• faster indexing, faster queries, etc.

In short, Lucene and Solr offer a cost effective and fully capable mechanism for improving the efficiency of search along the lines of the approach Dr. Shabes recommends, giving you the freedom to choose based on your idea of what works, not someone else’s.