As Apache Mahout is about to release its next version (0.3), I thought I would share some thoughts on how it might be integrated with Apache Lucene and Apache Solr.  For those who aren’t aware of Mahout, it is an ASF project building out a library of machine learning algorithms that are designed to be scalable (often via Apache Hadoop) and licensed under the Apache Software License (i.e., commercially friendly).  Mahout has a variety of algorithms already implemented, ranging from clustering to classification and collaborative filtering.  For more on Mahout, see my TriJUG talk or my developerWorks article.  Instead of going over the litany of things implemented in Mahout, I’ll give a quick recap of what the primary features of 0.3 are:

1. New math, collections modules based on the time tested Colt project
2. LLR (Log-likelihood ratio – See Lucid Imagination advisor Ted Dunning’s blog entry for more info) co-location implementation
3. Hadoop-based Lanczos SVD (Singular Value Decomposition) solver — good for feature reduction, which is a common requirement at scale
4. Shell scripts for easier running of algorithms, examples
5. Faster Frequent Pattern Growth (FPGrowth) using FP-bonsai pruning
6. Parallel Dirichlet process clustering (model-based clustering algorithm)
7. Parallel co-occurrence based recommender
8. Code cleanup, many bug fixes and performance improvements
9. A new Logo:



Enough of the background; let’s get to what we can do right now.  I’ll break it down into three groups:

1. Lucene/Solr as a Data Source for Mahout batch processing
2. Document/Results Augmentation (clustering, classification, recommendations)
3. Learning about your data and your users (log analysis with Apache Mahout)

In Part I (this post), I’m going to focus on #1 as a way for people to get started without having to do any coding.  In Part II, I’ll focus on #2 and finally, as you might guess, Part III will focus on #3.

Lucene/Solr as a Data Source for Mahout

Most Apache Mahout algorithms run off of Feature Vectors.  For those in the Lucene world, a feature vector should feel very familiar.  It is, more or less a document, or some subset of a document.  Specifically, a feature vector is a tuple of features that are useful for the algorithm.  It is up to you to determine what features work best.  In many cases for Mahout, a vector is simply a tuple of weights for each of the words in a document.  In other cases, they might be the values from the output of some manufacturing process.  Do note that the features for having good search capabilities are often different than those needed for good machine learning.  For instance, in my experiments with Mahout’s clustering capabilities, I need far more aggressive stopword removal to get good results than I do for search.  (In fact, for search these days, I often don’t even remove stopwords, but instead deal with them at query time, but that is a whole other post.)

There are two different ways for Mahout to use Lucene/Solr as a data source:

1. Utilize Lucene’s term vector capability to create Mahout feature vectors.
2. Programmatically access low level Lucene features like TermEnum, TermDocs, TermPositions, etc. to construct features.

For this post, I’m going to focus on #1, as I have yet to even have a need for #2, even though in theory it could be done.

Mahout Vectors from Lucene Term Vectors

In order for Mahout to create vectors from a Lucene index, the first and foremost thing that must be done is that the index must contain Term Vectors.  A term vector is a document centric view of the terms and their frequencies (as opposed to the inverted index, which is a term centric view) and is not on by default.

For this example, I’m going to use Solr’s example, located in <Solr Home>/example

In Solr, storing Term Vectors is as simple as setting termVectors=”true” on on the field in the schema, as in:

<field name=”text” type=”text” indexed=”true” stored=”true” termVectors=”true”/>

For pure Lucene, you will need to set the TermVector option on during Field creation, as in:

Field fld = new Field(“text”, “foo”, Field.Store.NO, Field.Index.ANALYZED, Field.TermVector.YES);

From here, it’s as simple as pointing Mahout’s new shell script (try running <MAHOUT HOME>/bin/mahout for a full listing of it’s capabilities) at the index and letting it rip:

<MAHOUT HOME>/bin/mahout lucene.vector –dir <PATH TO INDEX>/example/solr/data/index/ –output /tmp/foo/part-out.vec –field title-clustering –idField id –dictOut /tmp/foo/dict.out –norm 2

A few things to note about this command:

1. This outputs a single vector file, title part-out.vec to the target/foo directory
2. It uses the title-clustering field.  If you want a combination of fields, then you will have to create a single “merged” field containing those fields.  Solr’s <copyField> syntax can make this easy.
3. The idField is used to provide a label to the Mahout vector such that the output from Mahout’s algorithms can be traced back to the actual documents.
4. The –dictOut outputs the list of terms that are represented in the Mahout vectors.  Mahout uses an internal, sparse vector representation for text documents (dense vector representations are also available) so this file contains the “key” for making sense of the vectors later.  As an aside, if you ever have problems with Mahout, you can often share your vectors with the list and simply keep the dictionary to yourself, since it would be pretty difficult (not sure if it is impossible) to reverse engineer just the vectors.
5. The –norm tells Mahout how to normalize the vector.  For many Mahout applications, normalization is a necessary process for obtaining good results.  In this case, I am using the Euclidean distance (aka the 2-norm) to normalize the vector because I intend to cluster the documents using the Euclidean distance similarity.  Other approaches may require other norms.

Obviously, this script above can be run at any time, but I think it is even more interesting to hook it into Solr’s event system, with caveats.  For those who aren’t familiar, Solr provides an event call back system for events like commit and optimize (see also the LucidWorks Reference Guide).    Hooking into the event system is as simple as setting up the appropriate event listener.  For this example, I’m going to hook into the commit listener by having it call out to the Mahout script above:

<listener event=”postCommit”>
<str name=”exe”>/Volumes/User/grantingersoll/projects/lucene/mahout/clean/bin/mahout</str>
<str name=”dir”>.</str>
<bool name=”wait”>false</bool>
<arr name=”args”>
<str>lucene.vector</str>
<str>–dir</str>
<str>./solr/data/index/</str>
<str>–output</str>
<str>/tmp/mahout/vectors/part-out.vec</str>
<str>–field</str>
<str>text</str>
<str>–idField</str>
<str>id</str>
<str>–dictOut</str>
<str>/tmp/mahout/vectors/dict.dat</str>
<str>–norm</str>
<str>2</str>
<str>–maxDFPercent</str>
<str>90</str>
</arr>
</listener>

From here, one can easily extrapolate how a script could be written to then call Mahout’s other methods, namely things like clustering and Latent Dirichlet Allocation (LDA) for topic modeling.  Alternatively, one could set up a process to watch for changes to the vector and then spawn a process to go and run the appropriate Mahout tasks.

So, what are the caveats with the above approach?

1. If you are running in a commit heavy environment, you may not want to run Mahout on every commit.  Mahout is designed for batch processing (well, most of it is, anyway) and most of these jobs are designed to run on Hadoop clusters.  In order to do that, you would have to modify the above paths, etc. to have it output to Hadoop’s HDFS, which I’ll leave as an exercise to the reader (the Mathematician in me always enjoys saying that!)
2. If you are running Solr in a distributed environment, you’re going to have to set things up appropriately on each node.  Hopefully, as the Solr Cloud stuff matures, this will become even simpler and we should be able to do some really smart things to make Mahout and Solr work together in a distributed environment.  For now, you’re on your own.

In the next posting, I’ll look at how we can more closely hook in Mahout into the indexing and search process.  As a teaser, think about how you could use Mahout to classify and cluster large volumes of text and then have that information available for things like faceting, discovery and filtering on the search side.

As always, let me know if you have any questions or comments.

References

1. Mahout In Action by Owen and Anil.  Manning Publications.
2. Various Solr and Lucene books, all linked via Lucid Imagination.
3. http://lucene.apache.org/mahout
4. http://cwiki.apache.org/MAHOUT
5. Grant’s Blog has a number of articles on Mahout

Given a term match in a document, what’s the best way to get a window of words around that match?

Getting a window of words around a match can be useful for a lot of things, including, to name a few:

1. Highlighting (although I’d recommend using Lucene’s Highlighter package for that)
2. Co-occurrence analysis
3. Sentiment analysis
4. Question Answering

Unfortunately, given how inverted indexes are structured, retrieving content around a match isn’t efficient without doing some extra work during indexing.  In Lucene, this “extra work” involves creating and storing Term Vectors with position and offset information.

Storing Term Vector info can be done by adding in the appropriate code during Field construction, as in the following indexing example where I create an index from a few dummy documents (complete code is at the bottom of this post):

    RAMDirectory ramDir = new RAMDirectory();
    //Index some made up content
    IndexWriter writer = new IndexWriter(ramDir, new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED);
    for (int i = 0; i < DOCS.length; i++){
      Document doc = new Document();
      Field id = new Field("id", "doc_" + i, Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS);
      doc.add(id);
      //Store both position and offset information
      Field text = new Field("content", DOCS[i], Field.Store.NO, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS);
      doc.add(text);
      writer.addDocument(doc);
    }
    writer.close();

Notice the use of the Field.TermVector.WITH_POSITIONS_OFFSETS when constructing the text Field.  This tells Lucene to store term vector information on a per document basis (in other words, not inverted) with both Position and Offset information.  (Due note, other storage options are available, see Field.TermVector.  Also note, storing Term Vectors will cost you in disk space.)

For completeness, the DOCS array looks like:

public static String [] DOCS = {
        "The quick red fox jumped over the lazy brown dogs.",
        "Mary had a little lamb whose fleece was white as snow.",
        "Moby Dick is a story of a whale and a man obsessed.",
        "The robber wore a black fleece jacket and a baseball cap.",
        "The English Springer Spaniel is the best of all dogs."
    };

Now that we have an index created, we need to do a search.  In our case, we need to do a position-based search as opposed to the more traditional document-based search.  In other words, it is not good enough to simply know whether a term is in a document or not (think TermQuery), we need to know where in the document the match occurred.  Lucene enables position-based search through a series of Query classes collectively known as Span Queries.  (See SpanQuery and its derivitaves in the org.apache.lucene.search.spans package.)

Again, an example is warranted.  Assume we wanted to find where the term “fleece” occurs.  In this case, let’s start by doing a “normal” search, wherein we submit a query to the index and print out the Dcoument id and Score:

    IndexSearcher searcher = new IndexSearcher(ramDir);
    // Do a search using SpanQuery
    SpanTermQuery fleeceQ = new SpanTermQuery(new Term("content", "fleece"));
    TopDocs results = searcher.search(fleeceQ, 10);
    for (int i = 0; i < results.scoreDocs.length; i++) {
      ScoreDoc scoreDoc = results.scoreDocs[i];
      System.out.println("Score Doc: " + scoreDoc);
    }

That code looks pretty much like any basic search code with the exception that I substituted in a SpanTermQuery for what is often a TermQuery.  In fact, so far this isn’t all that interesting and it is likely to be slower than the comparable TermQuery too.

What does make it interesting?  If you look at the SpanQuery API, you will notice a method called getSpans().  The getSpans() method provides positional information about where a match occurred.  Thus, to print out the positional information, one might do:

    Spans spans = fleeceQ.getSpans(searcher.getIndexReader());
    while (spans.next() == true){
      System.out.println("Doc: " + spans.doc() + " Start: " + spans.start() + " End: " + spans.end());
    }

First off, notice getting the Spans is completely independent of running the actual query. In fact, you need not run the query first. Second, the start and end values are the positions of the tokens, not the offsets.

Now, given the position information, the question becomes how to get only those tokens around the match. To answer that, we need a few things:

1. The specification of a window in terms of positions.  For instance, I want the terms within two positions of the start and end of the span.
2. A TermVectorMapper implementation that is aware of both the window and the position.  Think of a TermVectorMapper as the equivalent of a SAX parser for Lucene’s Term Vectors.  Basically, instead of assuming the data structure (like DOM) it provides call backs and let’s you, the programmer, decide on the data structures.  See the PositionBasedTermVectorMapper for a useful implementation.

As a quick hack (and it is by no means production quality), I created the following code that modifies the printing code above:

    WindowTermVectorMapper tvm = new WindowTermVectorMapper();
    int window = 2;//get the words within two of the match, inclusive of the boundaries
    while (spans.next() == true) {
      System.out.println("Doc: " + spans.doc() + " Start: " + spans.start() + " End: " + spans.end());
      //build up the window
      tvm.start = spans.start() - window;
      tvm.end = spans.end() + window;
      reader.getTermFreqVector(spans.doc(), "content", tvm);
      for (WindowEntry entry : tvm.entries.values()) {
        System.out.println("Entry: " + entry);
      }
      //clear out the entries for the next round
      tvm.entries.clear();
    }

Now, in this chunk of code, I first create a WindowTermVectorMapper (WTVM, beautiful name, right?) and then in the Spans loop, I tell the WTVM what my window looks like. Next up, I ask Lucene’s IndexReader for the TermVector and pass in my TermVectorMapper.  Finally, I print out the entries.

Of course, the last bit of useful info is what does the WTVM look like.  Here’s the most useful snippet of code:

public void map(String term, int frequency, TermVectorOffsetInfo[] offsets, int[] positions) {
    for (int i = 0; i < positions.length; i++) {//unfortunately, we still have to loop over the positions
      //we'll make this inclusive of the boundaries
      if (positions[i] >= start && positions[i] < end){
        WindowEntry entry = entries.get(term);
        if (entry == null) {
          entry = new WindowEntry(term);
          entries.put(term, entry);
        }
        entry.positions.add(positions[i]);
      }
    }
  }

As you can see, I just look at the positions and check to see if the current term has an entry that is inside the start and end. Obviously, you can do more interesting things here, but I’ll leave that up to you. Also know that there are a few TermVectorMapper implementations in the Lucene distribution that you can use as examples.

That about wraps it up.  From here, one can easily imagine different ways to utilize the information returned from the Term Vector Mapper to process information about the terms in a window.

The full code is below. It is intended for demonstration purposes only. Please note the disclaimers, etc.

package com.lucidimagination.noodles;
/**
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
 
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.TermVectorMapper;
import org.apache.lucene.index.TermVectorOffsetInfo;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.spans.SpanTermQuery;
import org.apache.lucene.search.spans.Spans;
 
import java.io.IOException;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.ArrayList;
 
/**
 *  This class is for demonstration purposes only.  No warranty, guarantee, etc. is implied.
 *
 * This is not production quality code!
 *
 *
 **/
public class TermVectorFun {
  public static String[] DOCS = {
          "The quick red fox jumped over the lazy brown dogs.",
          "Mary had a little lamb whose fleece was white as snow.",
          "Moby Dick is a story of a whale and a man obsessed.",
          "The robber wore a black fleece jacket and a baseball cap.",
          "The English Springer Spaniel is the best of all dogs."
  };
 
  public static void main(String[] args) throws IOException {
    RAMDirectory ramDir = new RAMDirectory();
    //Index some made up content
    IndexWriter writer = new IndexWriter(ramDir, new StandardAnalyzer(), true, IndexWriter.MaxFieldLength.UNLIMITED);
    for (int i = 0; i < DOCS.length; i++) {
      Document doc = new Document();
      Field id = new Field("id", "doc_" + i, Field.Store.YES, Field.Index.NOT_ANALYZED_NO_NORMS);
      doc.add(id);
      //Store both position and offset information
      Field text = new Field("content", DOCS[i], Field.Store.NO, Field.Index.ANALYZED, Field.TermVector.WITH_POSITIONS_OFFSETS);
      doc.add(text);
      writer.addDocument(doc);
    }
    writer.close();
    //Get a searcher
    IndexSearcher searcher = new IndexSearcher(ramDir);
    // Do a search using SpanQuery
    SpanTermQuery fleeceQ = new SpanTermQuery(new Term("content", "fleece"));
    TopDocs results = searcher.search(fleeceQ, 10);
    for (int i = 0; i < results.scoreDocs.length; i++) {
      ScoreDoc scoreDoc = results.scoreDocs[i];
      System.out.println("Score Doc: " + scoreDoc);
    }
    IndexReader reader = searcher.getIndexReader();
    Spans spans = fleeceQ.getSpans(reader);
    WindowTermVectorMapper tvm = new WindowTermVectorMapper();
    int window = 2;//get the words within two of the match
    while (spans.next() == true) {
      System.out.println("Doc: " + spans.doc() + " Start: " + spans.start() + " End: " + spans.end());
      //build up the window
      tvm.start = spans.start() - window;
      tvm.end = spans.end() + window;
      reader.getTermFreqVector(spans.doc(), "content", tvm);
      for (WindowEntry entry : tvm.entries.values()) {
        System.out.println("Entry: " + entry);
      }
      //clear out the entries for the next round
      tvm.entries.clear();
    }
  }
 
}
 
//Not thread-safe
class WindowTermVectorMapper extends TermVectorMapper {
 
  int start;
  int end;
  LinkedHashMap entries = new LinkedHashMap();
 
  public void map(String term, int frequency, TermVectorOffsetInfo[] offsets, int[] positions) {
    for (int i = 0; i < positions.length; i++) {//unfortunately, we still have to loop over the positions
      //we'll make this inclusive of the boundaries
      if (positions[i] >= start && positions[i] < end){
        WindowEntry entry = entries.get(term);
        if (entry == null) {
          entry = new WindowEntry(term);
          entries.put(term, entry);
        }
        entry.positions.add(positions[i]);
      }
    }
  }
 
  public void setExpectations(String field, int numTerms, boolean storeOffsets, boolean storePositions) {
    // do nothing for this example
    //See also the PositionBasedTermVectorMapper.
  }
 
}
 
class WindowEntry{
  String term;
  List positions = new ArrayList();//a term could appear more than once w/in a position
 
  WindowEntry(String term) {
    this.term = term;
  }
 
  @Override
  public String toString() {
    return "WindowEntry{" +
            "term='" + term + '\'' +
            ", positions=" + positions +
            '}';
  }
}