Comments on: Integrating Apache Mahout with Apache Lucene and Solr – Part I (of 3)

By: Janki

Janki — Thu, 12 Jan 2012 09:34:44 +0000

Hi,

I am new to Mahout and Lucene. I want to do clustering of users. I have 7 dimensions (features) in data. I have tried kMeans clustering taking data from csv. Now I want to get data from Lucene. I have one question that while converting lucene documents to vectors, how will it consider dimensions? How should I generate Lucene documents if I want to generate vectors with n dimesions (features)?

By: Mark Rosenberg

Mark Rosenberg — Wed, 02 Nov 2011 21:24:30 +0000

Trying the cookbook example provided by the article with Mahout trunk and Solr 3.4.0. Looks like –field title-clustering doesn’t have enough term vectors so I may be running afoul of https://issues.apache.org/jira/browse/MAHOUT-675.

11/11/02 14:16:41 ERROR lucene.LuceneIterator: There are too many documents that do not have a term vector for title-clustering
Exception in thread “main” java.lang.IllegalStateException: There are too many documents that do not have a term vector for title-clustering.

If I use –field text then mahout completes normally and writes 17 vectors. The recommendation to use copyField to accumulate field contents in a new title-clustering field appears to be mandatory if the article’s mahout command line is to be used without modification.

By: Grant Ingersoll

Grant Ingersoll — Sat, 29 Oct 2011 12:26:43 +0000

Mahout trunk should now be on Lucene 3.4. In general, if you are replacing the jars, I think you need to make sure they are packaged in to Mahout’s Job jars correctly.

By: Bob Stewart

Bob Stewart — Thu, 27 Oct 2011 15:02:26 +0000

I have the same problem, seems that Lucene version is out of sync between Solr and Mahout. Question is how exactly do I make them in sync? I have mahout having lucene-core-3.1.0.jar in mahout/lib directory. I have Solr 3.4. I downloaded Lucene 3.4 jar files and replaced lucene jars inside mahout/lib but that did not work (doesnt seem that mahout loads those lucene jars at all). So how to I make sure they use the same lucene version? I am somewhat new to java/linux world.

By: Grant Ingersoll

Grant Ingersoll — Fri, 21 Oct 2011 06:36:32 +0000

You should be fine upgrading Mahout’s version. In fact, we should do it in Mahout. Feel free to open an issue there. Although, the Java 7 issue and the 3.4.0 issue are separate. The 3.4.0 issue was due to a fsync issue in Lucene 3.3.0

By: Mark Rosenberg

Mark Rosenberg — Thu, 20 Oct 2011 21:59:01 +0000

Hi Grant,

Thanks for the quick response! We seem to be in an awkward situation WRT Mahout and Solr Lucene version dependencies. I’m using Mahout 0.6 snapshot, which has a Lucene 3.3.0 dependency. Due to Oracle Java 7 sabotage, Lucene users are being advised to upgrade to 3.4.0. Do I have an alternative to using the Mahout 0.5 release?

By: Grant Ingersoll

Grant Ingersoll — Thu, 20 Oct 2011 20:24:30 +0000

Hi Mark,

the issue here is likely a version mismatch between the Lucene version in Mahout and the Lucene version you created your index with. If you sync those up, you should be fine.

By: Mark Rosenberg

Mark Rosenberg — Thu, 20 Oct 2011 18:14:08 +0000

I’m having some trouble getting this to work with my own data. I issue the following command:

mahout lucene.vector –dir /home/markr/shgs/apache-solr-3.4.0/example/solr/data/index/ –output /tmp/part-out.vec –field content_encoded –idField id –dictOut /tmp/dict.out –norm 2

My intent is to generate term vectors for the content_encoded field whose schema.xml entry has the termVectors=”true” attribute setting. There is also a field named ‘id’. My data was imported into a sqlite3 db, and id is ‘not null’, but content_encoded may be null. When I run, I get the SLF4J multiple binding warning (just a warning?), and then the following exception:

Exception in thread “main” org.apache.lucene.index.CorruptIndexException: unrecognized format -3 in file “_b.fnm”
at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:351)
at org.apache.lucene.index.FieldInfos.(FieldInfos.java:71)
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:72)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:114)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:113)
at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:29)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:288)
at org.apache.mahout.utils.vectors.lucene.Driver.dumpVectors(Driver.java:84)
at org.apache.mahout.utils.vectors.lucene.Driver.main(Driver.java:250)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)

Advise on how to debug this problem would be greatly appreciated.

Mark

By: Moshe Lichman

Moshe Lichman — Tue, 29 Mar 2011 17:41:47 +0000

Hi, great post.
Have a Q though – I’m running the MAHOUT through the Eclipse and I created the vector from my Lucene index. Two file were created:
1. The vector file.
2. The Dict file.

When running the FuzzyKMeans on the vector file – I got Exception while the job was parsing it – NotANumber Exception – for the vec file is a ‘compiled’ file. Any ideas how to make it work?

By: Matthew Sacks

Matthew Sacks — Thu, 10 Feb 2011 04:26:11 +0000

Hi Grant,
If you eventually wanted to dump results into Solritas (VelocityRepsonseWriter), what would the flow of data need to look like? Raw Data->Lucene->Mahout->Solr?

Thanks,
Matthew