So we decided to try to put together a matrix of major features and show in which versions they are available. Solr 1.4 is pretty old by now, so it naturally appears not to hold up well against Solr 3.5, Solr Trunk, or LucidWorks. Think of it as the base from which the later features in the list grow.

This was an interesting exercise to work through. It’s easy to read through the changes.txt for each release and try to include everything in a list such as this (and our Support guys are probably disappointed that I didn’t do that), but I tried to keep it to the major innovations or bug fixes so it stays somewhat readable. But there’s always the question of whether it’s too much or too little detail.

I hope it’s useful and we’d like to know what you think. Is it worthwhile? Should we go to deeper detail? Could the features use more explanation? Look it over at Feature Matrix for Solr and LucidWorks and share your suggestions in the comments.

I really dislike the so called “Boolean Operators” (“AND”, “OR”, and “NOT”) and generally discourage people from using them. It’s understandable that novice users may tend to think about the queries they want to run in those terms, but as you become more familiar with IR concepts in general, and what Solr specifically is capable of, I think it’s a good idea to try to “set aside childish things” and start thinking (and encouraging your users to think) in terms of the superior “Prefix Operators” (“+”, “-”).

Background: Boolean Logic Makes For Terrible Scores

Boolean Algebra is (as my father would put it) “pretty neat stuff” and the world as we know it most certainly wouldn’t exist with out it. But when it comes to building a search engine, boolean logic tends to not be very helpful. Depending on how you look at it, boolean logic is all about truth values and/or set intersections. In either case, there is no concept of “relevancy” — either something is true or it’s false; either it is in a set, or it is not in the set.

When a user is looking for “all documents that contain the word ‘Alligator’” they aren’t going to very be happy if a search system applied simple boolean logic to just identify the unordered set of all matching documents. Instead algorithms like TF/IDF are used to try and identify the ordered list of matching documents, such that the “best” matches come first. Likewise, if a user is looking for “all documents that contain the words ‘Alligator’ or ‘Crocodile’”, a simple boolean logic union of the sets of documents from the individual queries would not generate results as good as a query that took into the TF/IDF scores of the documents for the individual queries, as well as considering which documents matches both queries. (The user is probably more interested in a document that discusses the similarities and differences between Alligators to Crocodiles then in documents that only mention one or the other a great many times).

This brings us to the crux of why I think it’s a bad idea to use the “Boolean Operators” in query strings: because it’s not how the underlying query structures actually work, and it’s not as expressive as the alternative for describing what you want.

BooleanQuery: Great Class, Bad Name

To really understand why the boolean operators are inferior to the prefix operators, you have to start by considering the underlying implementation. The BooleanQuery class is probably one of the most misleading class names in the entire Lucene code base because it doesn’t model simple boolean logic query operations at all. The basic function of a BooleanQuery is:

1. A BooleanQuery consists of one or more BooleanClauses, each of which contains two pieces of information:
   • A nested Query
   • An Occur flag, which has one of three values
   • MUST – indicating that documents must match this nested Query in order for the document to match the BooleanQuery, and the score from this subquery should contribute to the score for the BooleanQuery
   • MUST_NOT – indicating that documents which match this nested Query are prohibited from matching the BooleanQuery
   • SHOULD – indicating that documents which match this nested Query should have their score from the nested query contribute to the score from the BooleanQuery, but documents can be a match for the BooleanQuery even if they do not match the nested query
2. If a BooleanQuery contains no MUST BooleanClauses, then a document is only considered a match against the BooleanQuery if one or more of the SHOULD BooleanClauses is a match.
3. The final score of a document which matches a BooleanQuery is based on the sum of the scores from all the matching MUST and SHOULD BooleanClauses, multiplied by a “coord factor” based on the ratio of the number of matching BooleanClauses to the total number of BooleanClauses in the BooleanQuery.

These rules are not exactly simple to understand. They are certainly more complex then boolean logic truth tables, but that’s because they are more powerful. The examples below show how easy it is to implement “pure” boolean logic with BooleanQuery objects, but they only scratch the surface of what is possible with the BooleanQuery class:

• Conjunction: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• Disjunction: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• Negation: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

Query Parser: Prefix Operators

In the Lucene QueryParser (and all of the other parsers that are based on it, like DisMax and EDisMax) the “prefix” operators “+” and “-” map directly to the Occur.MUST and Occur.MUST_NOT flags, while the absence of a prefix maps to the Occur.SHOULD flag by default. (If you have any suggestions for a one character prefix syntax that could be used to explicitly indicate Occur.SHOULD, please comment with your suggestions, I’ve been trying to think of a good one for years.) So using the prefix syntax, you can express all of the permutations that the BooleanQuery class supports — not just simple boolean logic:

• (+X +Y) … Conjunction, ie: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• (X Y) … Disjunction, ie: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• (+X -Z) … Negation, ie: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Z, Occur.MUST_NOT);
  

• ((X Y) -Z) … ((X ∨ Y) ¬ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.SHOULD);
  inner.add(Y, Occur.SHOULD);
  q.add(inner, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

• (X Y -Z) … Not expressible in simple boolean logic

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

Note in particular the differences between the last two examples. (X Y -Z) is a single BooleanQuery object containing three clauses, while ((X Y) -Z) is a BooleanQuery containing two clauses, one of which is a nested BooleanQuery containing two clauses. In both cases a document must match either “X” or “Y” and it can not match against “Z” (so the set of documents matched by each query will be the same) and in both cases the score of a document will be higher if it matches both the “X” and “Y” clauses; but because of the difference in their structures, the scores for these queries will be different for every document. In particular, the coord factor will cause documents matching only one of “X” or “Y” (but not both) to have extremely different scores from each of these queries. (This assumes that the DefaultSimilarity is being used; it would be possible to write a custom Similarity to force the scores to be equivalent)

Query Parser: “Boolean Operators”

The query parser also supports the so called “boolean operators” which can also be used to express boolean logic, as demonstrated in these examples:

• (X AND Y) … Conjunction, ie: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• (X OR Y) … Disjunction, ie: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• (X NOT Z) … Negation, ie: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

• ((X AND Y) OR Z) … ((X ∧ Y) ∨ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.MUST);
  inner.add(Y, Occur.MUST);
  q.add(inner, Occur.SHOULD);
  q.add(Z, Occur.SHOULD);
  

• ((X OR Y) AND Z) … ((X ∨ Y) ∧ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.SHOULD);
  inner.add(Y, Occur.SHOULD);
  q.add(inner, Occur.MUST);
  q.add(Z, Occur.MUST);
  

• (X AND (Y NOT Z)) … (X ∧ (Y ¬ Z))

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(Y, Occur.MUST);
  inner.add(Z, Occur.MUST_NOT);
  q.add(X, Occur.MUST);
  q.add(inner, Occur.MUST);
  

Please note how import it is to use parentheses to combine multiple operators in order in order to generate queries that correctly model boolean logic. As mentioned before, the BooleanQuery class supports an arbitrary number of clauses, so (X OR Y OR Z) is a single BooleanQuery with three clauses — it is not equivalent to either ((X OR Y) OR Z) or (X OR (Y OR Z)) because those result in a BooleanQuery with two clauses, one of which is a nested BooleanQuery. As mentioned above when discussing the prefix operators, the scores from each of those queries will all be different depending on which clauses are matched by each document.

Things definitely get very confusing when these “boolean operators” are used in ways other then those described above. In some cases this is because the query parser is trying to be forgiving about “natural language” style usage of operators that many boolean logic systems would consider a parse error. In other cases, the behavior is bizarrely esoteric:

• Queries are parsed left to right
• NOT sets the Occurs flag of the clause to it’s right to MUST_NOT
• AND will change the Occurs flag of the clause to it’s left to MUST unless it has already been set to MUST_NOT
• AND sets the Occurs flag of the clause to it’s right to MUST
• If the default operator of the query parser has been set to “And”: OR will change the Occurs flag of the clause to it’s left to SHOULD unless it has already been set to MUST_NOT
• OR sets the Occurs flag of the clause to it’s right to SHOULD

Practically speaking this means that NOT takes precedence over AND which takes precedence over OR — but only if the default operator for the query parser has not been changed from the default (“Or”). If the default operator is set to “And” then the behavior is just plain weird.

In Conclusion

I won’t try to defend or justify the way the query parser behaves when it encounters these “boolean operators”, because in many cases I don’t understand or agree with the behavior myself — but that’s not really the point of this article. My goal isn’t to convince you that the behavior of these operators makes sense, quite the contrary my goal today is to point out that regardless of how these operators are parsed, they aren’t a good representation of the underlying functionality available in the BooleanQuery class.

Do yourself a favor and start thinking about BooleanQuery as a container of arbitrary nested queries annotated with MUST, MUST_NOT, or SHOULD and discover the power that is available to you beyond simple boolean logic.

Many thanks to Bill Dueber whose recent related blog post reminded me that I had some draft notes on this subject floating around my laptop waiting to finished up and posted online.

There are two stages where documents can be boosted: At index time and at query time.

At Index Time

This is probably the simplest way, because there are not too many options. It is also the most static way of adding boosts, as changing the boost for a documents would require re-indexing it.
When updating documents using the XMLUpdateRequestHandler, the way to boost a document is to add the optional attribute “boost” to the doc element. When using SolrJ, the way to do it is by using the method

document.setDocumentBoost(x)

The default boost for a field is 1, so setting a value between 0 and 1 would down boost the document.

It is also possible to add different boosts to different fields of a document. The only requirement here is that the boosted fields must store the norms (“omitNorms” attribute in the schema must be set to “false”). The way of applying the boosts when using the XMLUpdateRequestHandler is similar to boosting the whole document, but instead of adding the “boost” attribute to the doc element, add it to the field element. When using SolrJ:

document.addField(“title”, “Foo Bar”, x);


It’s important to know that the boost (either for a document or for a field) will be considered when calculating the final score for a document given a search. It is not the final score of the document. Boosting documents is not the same as sorting documents.

At Query Time

Boosting at query time is a little bit different than index time. It is much more dynamic as it doesn’t require re-indexing and can be specified with every new request to Solr. Also, what gets boosted is not a document or a field, but a subquery on the search. The simplest way to achieve query time boosting is by using the ^ character plus the boost number on the query, for example:

foo^5 bar

Much more complex expressions can also be used for query time boosting, like:

title:(foo bar)^5 OR content:(foo bar)^2 OR foo OR bar

title:(foo bar)^5 OR title:”foo bar”^20 OR …

The syntax can be very simple for simple cases, but it will get more and more complex with more complex use cases.

The above syntax is Lucene’s query syntax, it is supported by the Lucene Query Parser and the Extended Dismax Query Parser but not by the Dismax Query Parser.
However, this syntax requires having an expert user who knows how to use it, or some application logic to inject it in the background after the user enters the query and before sending it to Solr. Dismax provides other alternatives for query time boosting, as dynamic as the previous one, but with a much easier syntax (all of them also supported by Extended Dismax).

Query Time Boosting with the Dismax Query Parser

Boosting Fields

The Dismax Query Parser (QP) will create a query that will be executed on many different fields, even if the user hasn’t specified any. This is one of the most important improvements of the Dismax QP over the Lucene QP. But sometimes, not all the fields have the same importance. Sometimes, a hit on the title field is more important than a hit on the content field, or a hit on the content can be more important than a hit on the comments field. The Dismax Query Parser provides the ability to consider some fields more important than others with the “qf” (named after “query fields”) parameter, the same that is used for specifying the different fields on which to execute the user query. A common value for this parameter could be:

qf=title^5 content^2 comments^0.5

This will translate a user query like “boo bar” into something similar to:

title:(foo bar)^5 OR content:(foo bar)^2 OR comments:(foo bar)^0.5

Boosting Phrases

The same as with query fields, Dismax Query Parser will execute the user query as a phrase query on the specified “phrase” fields. In this parameter, and in a similar way as in the qf parameter, a different boost for each of the phrase fields can be specified:

pf=title^20 content^10

This will translate a user query like foo bar into:

title:”foo bar”^20 OR content:”foo bar”^10

The last query will only be used for boosting the documents resulting from the original query.

Boost Queries

Sometimes it is necessary to boost some documents regardless of the user query. A typical example of boost queries is boosting sponsored documents. The user searches for “car rental”, but the application has some sponsored document that should be boosted. A good way of doing this is by using boost queries. A boost query is a query that will be executed on background after a user query, and that will boost the documents that matched it.

For this example, the boost query (specified by the “bq” parameter) would be something like:

bq=sponsored:true

The boost query won’t determine which documents are considered a hit an which are not, but it will just influence the score of the result.

Boost Functions

Boost Functions are very similar to boost queries; in fact, they can achieve the same goals. The difference between boost functions and boost queries is that the boost function is an arbitrary function instead of a query (see http://lucidworks.lucidimagination.com/display/solr/Function+Queries). A typical example of boost functions is boosting those documents that are more recent than others. Imagine a forum search application, where the user is searching for forum entries with the text “foo bar”. The application should display all the forum entries that talk about “foo bar” but usually the most recent entries are more important (most users will want to see updated entries, and not historical). The boost function will be executed on background after each user query, and will boost some documents in some way.

For this example, a boost function (specified by the “bf” parameter) could be something like:

bf=recip(ms(NOW,publicationDate),3.16e-11,1,1)

The same as with the boost queries, this function will not determine which documents are a hit and which are not, it will just add additional score to them.

A note on boost functions: boost functions can also be used with the Lucene QP by using the “_val_” special key inside the query.

Tie Breaker

The “tie” (tie breaker) parameter is very important, but not easy to understand. First it is important to understand what is a dismax (http://www.lucidimagination.com/blog/2010/05/23/whats-a-dismax/). With DisMax queries, the different terms of the user input are executed against different fields, if many of them hit (the term appears in different fields in the same document) the hit that scores higher is used, but what happens with the other sub-queries that hit in that document for the term? Well, that’s what the “tie” parameter defines. DisMax will calculate the score for a term query as:

score= [score of the top scoring subquery] + tie * (sum of other hitting subqueries)

In consequence, the “tie” parameter is a value between 0 and 1 that will define if the Dismax will only consider the max hit score for a term (setting tie=0), all the hits for a term (setting tie=1) or something between those two extremes.

The boost Parameter

The “boost” parameter is very similar to the “bf” parameter, but instead of adding its result to the final score, it will multiply it. This is only available in the “Extended Dismax Query Parser” or the “Lucid Query Parser”.

A note on the parameters

All the above parameters can be specified when configuring Solr (in the solrconfig.xml file) but they can also be changed on each request just by sending the parameter on the request with the new value.

Prior to the current 3x branch (which will be released as 3.6) and the trunk (4.0) Solr code, users have frequently been perplexed by wildcard searching being un-analyzed, often manifesting in case sensitivity. Say you have an analysis chain in your schema.xml file defined as follows and a field named lc_field of this type:

<fieldType name="lowercase" class="solr.TextField" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.LowercaseFilterFactory" />
</fieldType>

Now, you index the text “My Dog Has Fleas”. So far, so good. Searching on this field as
field_lc:fleas returns the document, as does field_lc:flea*.

But now you search on field_lc:Flea* and you don’t get any results. What?!?!?! Nearly everyone scratches their heads about this, and it’s a question that often comes up on the Solr user’s list. Users wonder why the analysis chain above isn’t applied to the wildcard queries. It turns out that it’s trickier than you might think at first. What happens when a single input term gets split up into multiple parts? For instance, for those of you familiar with WordDelimiterFilterFactory (WDDF) that can split on case change. What does it mean to parse ‘fleA*’? Applying WDDF might well give the two tokens ‘fle’ and ‘A’ and possibly ‘fleA’. If a wildcard is present, what tokens should be emitted?

1. ‘fleA*’
2. ‘fle*’, ‘A*’, ‘fleA*’
3. ‘fle*’, ‘A*’
4. <insert your solution here>

You can, I daresay, create any rule that suits your fancy. And it’ll be wrong in some situations. Of particular horror is anything that produces ‘A*’ as above, conceptually, you’d than have an enormous OR clause consisting of all the terms that started with ‘A’ in your index. Unless you had a rule like “only do this if the preceding fragment was 2 characters or more”. But then someone would say “I need three characters”, so can WDDF provide a “wildCardMin=#” parameter? I have trouble keeping all the parameters with WDDF and how they interact in my mind already, going down this path would be a nightmare. And I haven’t even considered some of the really interesting issues, like how proximity would be incorporated in all this.

Wildcards aren’t the only issue

The same issue occurs with accent folding, normalizations, and, really, any other component of an analysis chain that somehow changes the query terms. This behavior has mostly been ignored in releases past, it’s been up to the application programmer to manually “do the right thing” before sending the query to Solr. This often involves operations such as lower-casing and accent folding on the application side when a wildcard is encountered.

The new way of handling these cases

As of SOLR-2438 this behavior is no longer true for a number of the most common cases. A query analysis chain that contains any of the following components will automatically “do the right thing” and apply them for multi-term queries. If your analysis chain consists of any of these elements, and you want them applied to “multi-term” queries, you don’t have to do anything at all, it will “just work”. At query time, the indicated transformations are applied to the query terms and everyone is happy. Or should be. Do note that it’s an all-or-nothing operation. All of the elements below that are found in the query analysis chain are applied to the multi-term terms.

• ASCIIFoldingFilterFactory
• LowerCaseFilterFactory
• LowerCaseTokenizerFactory
• MappingCharFilterFactory
• PersianCharFilterFactory

Again, this effectively means you don’t need to care about these transformations any more. One note of explanation, though. I’ve talked about the “query analysis chain”. But what if you don’t have one? Remember that your <analyzer> tag can have several possible ‘type’ parameters; “index”, or “query”, or none. Well, if a ‘ type=”query” ‘ is found, that analysis chain is inspected and any of the above components are recorded to be used on multi-term queries. If no ‘ type=”query” ‘ is found, then the ‘ type=”index” ‘ is used. And if no ‘ type=”index” ‘ is found, than the one with no ‘type’ parameter is used.

What does “multi-term” mean anyway?

I’ve also sprinkled the phrase “mult-term” around, and sometimes “wildcard”. It turns out that the simple wildcard case is a specialization of a broader category of queries, including:

• wildcard
• range
• prefix

All of these are now handled as above.

Expert level schema possibilities

All of the above is automatic, but there are three immediate questions:

• what about some of the other components?
• what if I need the old behavior?
• what if I want something completely different?

It turns out that all three of these questions have the same answer. But before I outline it, I want to emphasize that you very probably don’t need to care about what follows! You might need to know about this in special cases, so I’ll mention it here.

In the above explanations, I wrote that “analysis chain is inspected and any of the above components are recorded to be used on multi-term queries”. Well, what actually happens is that there’s a new analysis chain in town that can be specified in the schema.xml file called, you guessed it, “multiterm”. You specify it like this as part of a <fieldType>:

<analyzer type="multiterm" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.ASCIIFoldingFilterFactory" />
  <filter class="solr.YourFavoriteFilterFactoryHere" />
</analyzer>

You can put any component that’s legal in a ‘type=”index” ‘ or ‘type=”query” ‘ analysis chain. If you wanted, for instance, to enforce the old-style behavior, you could specify

  <tokenizer class="solr.KeywordTokenizerFactory" />

as the entire “multiterm” analysis chain. It seems a bit odd to use KeywordTokenizerFactory here, but this applies to the individual terms, not the entire input. So it’s in effect saying “don’t analyze the terms at all”. Sound familiar? This is just what happened historically.

How does this relate to the automatic behavior?

Well, what really happens under the covers is that if you don’t define your own “multiterm” analysis chain, Solr constructs one for you from the analyzers you have defined as outlined above; query, index or default, in that order.

Waaaaay under the covers, down in the code

All this is accomplished by making components “multiterm aware”. This means implementing the “MultiTermAwareComponent” interface. Currently, the components listed above are the only ones that implement this interface, but others may be good candidates, and some of these are listed in JIRA SOLR-2921. By and large, implementing these in the code may be trivial. What’s not trivial is understanding what “the right thing” is. Some examples:

• stemmers
• various language-specific normalization filters
• various language-specific lowercase filters.
• various ICU filters

The reason these haven’t been made “multi term aware” is the usual open-source reason; “What we have is a good step forward, we shouldn’t delay everything in order to get the last use cases taken care of”. In other words the implementors (me in this case, with lots of help from others) are tired .

Anyone who really understands what the right thing to do in the cases of components that do not yet implement “MultiTermAwareComponent” and could provide use cases for them would be giving us a great help, especially by providing examples illustrating the correct inputs and outputs for wildcard cases. And some examples of what should not come out as well. Or even better, a draft JUnit test that would show the expected behavior. Or even better yet, a full patch!

Any modification that potentially produces more than one token needs to be handled with care, see the code for LowerCaseTokenizerFactory for a case in point. Consider that Solr will now throw an exception if the transformation produces more than one token, so tread cautiously!

This change should remove a long-standing point of confusion for solr users. We’d be very interested in any feedback from the community, and especially any problems that crop up. SOLR-2438 has patches for both the 3x and 4x code lines, but it’s probably easier just to get a current 3x or 4x branch (or nightly build) if you want to test this “in the wild”; the code has been committed and built. There remains some work to be done to incorporate this change for more analysis components, anyone want to volunteer?

Resources:

This page on the Solr Wiki has the Wiki documentation: Multi Term Query Analysis

Main JIRA (already in 3.6 and 4.0 code lines): SOLR-2438

JIRA for other components not yet “multi-term aware” that are possibilities in the future: SOLR-2921

I’ll be speaking at the upcoming Rich Web Experience conference in Ft. Lauderdale, presenting an “Introduction to Solr”, “Solr Recipes”, and “Lucene for Solr Developers”.  I’ll be tying all of these presentations together into a cohesive search/Solr track going from the introduction, to recipes for common tasks, through advanced customization of Solr.

Unfortunately, Java 6′s only included library is Rhino that converts the javascript code into JVM code and its performance is not good. For reasons like these, the scripting languages in general are experiencing something that Java felt itself in the past: the popular belief that they are slow.

The performance is certaintly not that bad but in a critical application, people would prefer to develop native (java) components to keep the performance before losing performance with a probably unnecesary script. This entry is about performance of scripting languages; in particular, other Javascript engines that you can attach to Java.

Google chrome has surprised everyone with their blazing fast javascript engine: V8.

I downloaded V8 and tested it against the regular Rhino option. Actually I didn’t implement the neccesary wrappers to add V8 to Java, nor the JNI C programming necessary as a glue to access V8 functions (which are native in the traditional sense, real machine code running on real processors, like in the old days). I used a library that I found in internet: http://code.google.com/p/jav8/.

For the first test I downloaded a benchmark from V8 that computes an RSA encryption of a text using Javascript. I know, you will feel that I’m cheating by using a benchmark that is not impartial, but the results are pretty conclusive anyway: http://v8.googlecode.com/svn/data/benchmarks/.

The code (download here) also shows the basic usage of the scripting functionality of Java.

public class main {

	public static void main(String[] args) throws FileNotFoundException {

		ScriptEngineManager sm = new ScriptEngineManager();
		FileReader file = new FileReader("test.js");
		ScriptEngine jsEngine = sm.getEngineByName("jav8");

		int iter = Integer.parseInt(args[0]);

		try {
			long acum = 0;
			for(int i=0; i<iter; i++){
				long start = System.currentTimeMillis();
				Object ob = jsEngine.eval(file);
				long end = System.currentTimeMillis();

				acum += end - start;
			}
			System.out.println(acum);
		} catch (ScriptException ex) {
			ex.printStackTrace();
		}
	}
}

I created a simple bash script to run this many times with different number of iterations and these are the results (X has the iterations and Y has the time, so less is better)

The performance difference is outstanding. Particularly interesting is the scalability of V8 when we add more iterations. V8 has an advanced cache system an surely that is helping to keep the performance as the iteration grows.

A real application that performs a set of repetitive tasks is the ScriptTransformer of the Data Import Handler of Solr. This transformer applies a function written in Javascript to each row of a table. This infamous component is something very useful but its performance has always been horrible.

To continue the tests I compared the standard Rhino script engine vs V8 applied to the ScriptTransformer of Solr. I had to modify the ScriptTransformer and remove its “reflection style” implementation (apparently to keep compatibility with 1.5, but quite ugly anyway) to make jav8 work with it. The test consisted in encrypting 5000 records of text from a database. (modified ScriptTransformer.java)

The results:

Again the results show that V8 wins by a big margin.

The conclusion that we extract here is that adding script engines to our systems does not imply that the performance will be damaged. If you accept the use of native libraries (V8 in this case) a script engine can make your systems much more flexible without slowing them down.

For me personally, I’m on to Vancouver and ApacheCon for two primary things, besides of course the community bits that go with every ApacheCon:

1. Providing the ApacheCon’s first ever Apache Mahout training on Monday, November 7th.  It’s still not too late to sign up!
2. Giving a talk on alternative uses of Lucene/Solr other than traditional free text search (things like recommendation engines, classification, etc.)

For the 2nd item, I’m also interested in hearing from you, the user, about interesting things you’ve done with Lucene/Solr that fall outside the norm of free text search.  If you care to share, please leave a comment on this post.

I’d be remiss if I didn’t also plug several other Lucid Imagination employees who are speaking at ApacheCon as well:

1. Solr Flair by Erik Hatcher.  Erik will also be doing a 2 day Solr training class.  Registration is still open for this class as well.
2. Apache Solr: Out of the Box by Chris Hostetter

Lucid Imagination is also sponsoring the Lucene/Solr meetup on Wed. November 9th, so if you are in town, please feel free to drop by for a drink and a chat.

With that, I’ll simply say, I hope to see you in Vancouver in a few weeks!

-Grant

There is special class of software programs called monitoring software that are widely used among system and network administrators. In our case we would like to monitor not only OS level metrics, but also Solr internal parameters and act accordingly. LucidWorks and Apache Solr provide lots of valuable information through a JMX interface, so you can hook that up into your monitoring tool.

Zabbix is one of the most popular open source monitoring tools. It has many features like an easy-to-use web interface, different ways to gather metrics data, an ability to keep this data in persistent storage, built-in graphing, notifications and alerting, flexible configuration and many more. One of the most compelling features of integrating it with Apache Solr is built-in JMX support (available only in Zabbix 2.0 beta release). Using this feature you can easily configure Zabbix server to pull JMX metrics out of any LucidWorks or Solr application. This is because all configuration settings (JMX attributes, graphs, triggers) are stored centrally on a Zabbix server, which means you can add a new attribute for all monitored servers or change the pulling frequency for servers with a single click.

Here are example graphs you can build in Zabbix:

1. Total number of documents in Solr index

2. Search activity – number of search requests, errors and timeouts 
Solr request handlers provide cumulative counter for number of requests,  but you are probably more interested in number of search requests per specific period of time, like per minute or per second. The trick here is that Zabbix provides a way to setup monitoring to store not the value as-is, but as a delta (simple change value or speed per second).

3. Solr document operations (adds, deletes by id or query)

4. Crawling activity
LucidWorks provides different connectors/crawlers which you can use to index documents into Solr. It also provides additional statistics about crawler behavior, like total number of documents, new and deleted documents, number of updated documents in iterative crawl, failures, etc.

5. Solr index operations (commits, optimizes, rollbacks)

6. Search Average Response Time
Solr search request handler provides cumulative avgTimePerRequest value. The problem with this attribute is that  when your applications is running in production for a significant amount of time, current short term performance problems won’t cause significant effect on this aggregate metric. The solution is to use a Zabbix calculated item on delta change for totalTime and requests attributes. Here’s math expression to calculate average search response time for the last 5 minutes:

sum("jmx[\"solr/collection1:type=/lucid,id=org.apache.solr.handler.StandardRequestHandler\",\"totalTime\"]",300)/sum("jmx[\"solr/collection1:type=/lucid,id=org.apache.solr.handler.StandardRequestHandler\",\"requests\"]",300)

7. Solr searcher warmup time
This is an important metric if you pursue fast commit rate (near real time indexing) and don’t want to sacrifice fast faceting performance. You can configure monitoring tool to send alert in case of warmup time exceeds some pre-defined threshold.

8. Filter, query results and documents caches statistics (cache size, hits, hitratio, evictions, etc)

9.  Java Heap Memory Usage

How would I know if my search server is down?

There are two options – the obvious one is to set up your monitoring tool to issue search requests and verify response status or specific text on a search results page. Another option is to check the last time your monitoring tool retrieved an arbitrary JMX attribute from your application and assume the server is down if it’s longer than expected. In Zabbix there’s special function nodata which you can use to achieve that.

How would I know if I’m reaching a limits of my server and pro-actively react on this?

This is a complex issue as there are many things that can go wrong (such as JVM heap memory, CPU load, disk space, file descriptors, etc.) and you should monitor them all. Zabbix has great example templates for OS and Java triggers that allow you to keep an eye on all those parameters.

For more information about Solr and LucidWorks JMX support, instructions how to configure Zabbix and Nagios, Zabbix configuration templates and other helpful tips please see the Integrating Monitoring Services section on Lucid documentation portal.

And while I wasn’t around in the beginning, I thought I would offer up some (little) known tidbits, links, etc. about Lucene as an ode to the search library that has significantly changed the search world, as well as my own career:

1. Lucene was Doug’s way of learning Java!  How’s that for a start?  It took him 3 months, working 2 days a week to crank out the first version.
2. At the time, some commercial search engines could not do incremental updates of the index, meaning you had to re-index all your documents anytime you had an update.  Lucene has always had an incremental model, all the way through to today’s Near Real Time features that power the likes of Twitter at 1 billion+ searches and 100M+ new documents per day.
3. Field myField = Field.Text(“foo”, “bar”); anyone?  Or how about Field myField = Field.UnIndexed(“foo”, “bar”);
4. Back then, Lucene had it’s own PorterStemmer, now we just use Snowball.
5. Only 1 of the original committers still remains somewhat active.
6. Read the old FAQ!  True as it ever was.  (Mostly)
7. Lucene 2.3 drastically improved indexing performance thanks to a thorough overhaul of the innards while barely affecting the API.  4.0 will blow the doors off of previous versions in terms of speed and efficiency.
8. Lucene is Doug’s wife’s middle name.
9. Lucene has evolved from offering a single vector space scoring model to one that now offers plug-n-play ranking (BM25 anyone?)
10. Lucene is ubiquitous.  It powers search on everything from mobile devices to web scale engines.  I’ve seen indexes as small as 15% of the original content.  I’ve also seen indexes grow to several billion documents in size.  Lucene has been used as a caching store, an ORM, a cross language search engine, the guts of the popular Solr search server, the retrieval engine for IBM’s Watson as well as several commercial search engines and pretty much everything in between.
11. Did you know Apache Hadoop started as a subproject of Lucene?  Doug Cutting and Mike Cafarella first built out Hadoop in order to scale out indexing for the Apache Nutch project.  From there it was spun out to be a top level ASF project and has gone on to be the de facto choice for large scale distributed processing, much like Lucene is the de facto choice for search!  Lucene has also spun out Mahout, Tika, Lucene.NET and Lucy!

As for how Lucene’s impacted me?  In 2004, I took a job at the Center for Natural Language Processing at Syracuse University working for Dr. Liz Liddy.  My job was to build an Arabic-English cross language search engine.  Within a day or two of starting, Ozgur Yilmazel (my boss at the time) said something to the effect of “we’ll be using Lucene for the implementation.  Go learn it.”  Digging in, I quickly needed a couple of features, the biggest one being Term Vectors, so I updated a patch from an earlier version of Lucene and managed to convince the committers at the time to commit it.  From there, I kept supplying patches.  Eventually, I was asked to be a committer.  Some time after that, Yonik Seeley and Marc Krellenstein approached a bunch of the committers about starting a company and here I am today at the company we (Erik, Yonik, Marc and I) founded back in 2007, Lucid Imagination.  I feel fortunate to have the opportunity to work on hard problems in an interesting field and for that, Lucene, in no small part, I thank  you.

But enough of my self-indulgence, how has Lucene impacted you?  When did you first start using it?  What’s your biggest index or fastest QPS?   What ways have you used Lucene beyond that of a search engine?  Leave a comment and let us know.

Happy 10th Anniversary, Lucene!