Prior to the current 3x branch (which will be released as 3.6) and the trunk (4.0) Solr code, users have frequently been perplexed by wildcard searching being un-analyzed, often manifesting in case sensitivity. Say you have an analysis chain in your schema.xml file defined as follows and a field named lc_field of this type:

<fieldType name="lowercase" class="solr.TextField" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.LowercaseFilterFactory" />
</fieldType>

Now, you index the text “My Dog Has Fleas”. So far, so good. Searching on this field as
field_lc:fleas returns the document, as does field_lc:flea*.

But now you search on field_lc:Flea* and you don’t get any results. What?!?!?! Nearly everyone scratches their heads about this, and it’s a question that often comes up on the Solr user’s list. Users wonder why the analysis chain above isn’t applied to the wildcard queries. It turns out that it’s trickier than you might think at first. What happens when a single input term gets split up into multiple parts? For instance, for those of you familiar with WordDelimiterFilterFactory (WDDF) that can split on case change. What does it mean to parse ‘fleA*’? Applying WDDF might well give the two tokens ‘fle’ and ‘A’ and possibly ‘fleA’. If a wildcard is present, what tokens should be emitted?

1. ‘fleA*’
2. ‘fle*’, ‘A*’, ‘fleA*’
3. ‘fle*’, ‘A*’
4. <insert your solution here>

You can, I daresay, create any rule that suits your fancy. And it’ll be wrong in some situations. Of particular horror is anything that produces ‘A*’ as above, conceptually, you’d than have an enormous OR clause consisting of all the terms that started with ‘A’ in your index. Unless you had a rule like “only do this if the preceding fragment was 2 characters or more”. But then someone would say “I need three characters”, so can WDDF provide a “wildCardMin=#” parameter? I have trouble keeping all the parameters with WDDF and how they interact in my mind already, going down this path would be a nightmare. And I haven’t even considered some of the really interesting issues, like how proximity would be incorporated in all this.

Wildcards aren’t the only issue

The same issue occurs with accent folding, normalizations, and, really, any other component of an analysis chain that somehow changes the query terms. This behavior has mostly been ignored in releases past, it’s been up to the application programmer to manually “do the right thing” before sending the query to Solr. This often involves operations such as lower-casing and accent folding on the application side when a wildcard is encountered.

The new way of handling these cases

As of SOLR-2438 this behavior is no longer true for a number of the most common cases. A query analysis chain that contains any of the following components will automatically “do the right thing” and apply them for multi-term queries. If your analysis chain consists of any of these elements, and you want them applied to “multi-term” queries, you don’t have to do anything at all, it will “just work”. At query time, the indicated transformations are applied to the query terms and everyone is happy. Or should be. Do note that it’s an all-or-nothing operation. All of the elements below that are found in the query analysis chain are applied to the multi-term terms.

• ASCIIFoldingFilterFactory
• LowerCaseFilterFactory
• LowerCaseTokenizerFactory
• MappingCharFilterFactory
• PersianCharFilterFactory

Again, this effectively means you don’t need to care about these transformations any more. One note of explanation, though. I’ve talked about the “query analysis chain”. But what if you don’t have one? Remember that your <analyzer> tag can have several possible ‘type’ parameters; “index”, or “query”, or none. Well, if a ‘ type=”query” ‘ is found, that analysis chain is inspected and any of the above components are recorded to be used on multi-term queries. If no ‘ type=”query” ‘ is found, then the ‘ type=”index” ‘ is used. And if no ‘ type=”index” ‘ is found, than the one with no ‘type’ parameter is used.

What does “multi-term” mean anyway?

I’ve also sprinkled the phrase “mult-term” around, and sometimes “wildcard”. It turns out that the simple wildcard case is a specialization of a broader category of queries, including:

• wildcard
• range
• prefix

All of these are now handled as above.

Expert level schema possibilities

All of the above is automatic, but there are three immediate questions:

• what about some of the other components?
• what if I need the old behavior?
• what if I want something completely different?

It turns out that all three of these questions have the same answer. But before I outline it, I want to emphasize that you very probably don’t need to care about what follows! You might need to know about this in special cases, so I’ll mention it here.

In the above explanations, I wrote that “analysis chain is inspected and any of the above components are recorded to be used on multi-term queries”. Well, what actually happens is that there’s a new analysis chain in town that can be specified in the schema.xml file called, you guessed it, “multiterm”. You specify it like this as part of a <fieldType>:

<analyzer type="multiterm" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.ASCIIFoldingFilterFactory" />
  <filter class="solr.YourFavoriteFilterFactoryHere" />
</analyzer>

You can put any component that’s legal in a ‘type=”index” ‘ or ‘type=”query” ‘ analysis chain. If you wanted, for instance, to enforce the old-style behavior, you could specify

  <tokenizer class="solr.KeywordTokenizerFactory" />

as the entire “multiterm” analysis chain. It seems a bit odd to use KeywordTokenizerFactory here, but this applies to the individual terms, not the entire input. So it’s in effect saying “don’t analyze the terms at all”. Sound familiar? This is just what happened historically.

How does this relate to the automatic behavior?

Well, what really happens under the covers is that if you don’t define your own “multiterm” analysis chain, Solr constructs one for you from the analyzers you have defined as outlined above; query, index or default, in that order.

Waaaaay under the covers, down in the code

All this is accomplished by making components “multiterm aware”. This means implementing the “MultiTermAwareComponent” interface. Currently, the components listed above are the only ones that implement this interface, but others may be good candidates, and some of these are listed in JIRA SOLR-2921. By and large, implementing these in the code may be trivial. What’s not trivial is understanding what “the right thing” is. Some examples:

• stemmers
• various language-specific normalization filters
• various language-specific lowercase filters.
• various ICU filters

The reason these haven’t been made “multi term aware” is the usual open-source reason; “What we have is a good step forward, we shouldn’t delay everything in order to get the last use cases taken care of”. In other words the implementors (me in this case, with lots of help from others) are tired .

Anyone who really understands what the right thing to do in the cases of components that do not yet implement “MultiTermAwareComponent” and could provide use cases for them would be giving us a great help, especially by providing examples illustrating the correct inputs and outputs for wildcard cases. And some examples of what should not come out as well. Or even better, a draft JUnit test that would show the expected behavior. Or even better yet, a full patch!

Any modification that potentially produces more than one token needs to be handled with care, see the code for LowerCaseTokenizerFactory for a case in point. Consider that Solr will now throw an exception if the transformation produces more than one token, so tread cautiously!

This change should remove a long-standing point of confusion for solr users. We’d be very interested in any feedback from the community, and especially any problems that crop up. SOLR-2438 has patches for both the 3x and 4x code lines, but it’s probably easier just to get a current 3x or 4x branch (or nightly build) if you want to test this “in the wild”; the code has been committed and built. There remains some work to be done to incorporate this change for more analysis components, anyone want to volunteer?

Resources:

This page on the Solr Wiki has the Wiki documentation: Multi Term Query Analysis

Main JIRA (already in 3.6 and 4.0 code lines): SOLR-2438

JIRA for other components not yet “multi-term aware” that are possibilities in the future: SOLR-2921

Lucene in Action, Second Edition, completely revises and updates the best-selling first edition and remains the authoritative book on Lucene. This book shows you how to index your documents, including types such as MS Word, PDF, HTML, and XML. It introduces you to searching, sorting, and filtering, and covers the numerous changes to Lucene since the first edition. All source code has been updated to latest APIs (2.4.x and aims to be current to the upcoming Lucene 2.9 and 3.0 APIs)

What’s been updated in the 2nd edition:

• Updating and deleting documents using IndexWriter
• Using the different LockFactory, DeletionPolicy, MergePolicy and MergeScheduler implementations that have been factored out.
• Using the new autoCommit option in IndexWriter
• Understanding simplifications to Lucene’s locking
• Adding payloads to your index and using them with BoostingTermQuery
• Using Function queries
• Using FieldSelector to speed up loading of stored fields
• Using IndexReader.reopen() to efficiently opening a new reader from an existing one
• Measuring performance using the “benchmark” contrib package
• Tuning the indexing or searching speed
• Using threads to gain concurrency
• Managing resources like memory, disk, and file descriptors usage
• Making a backup copy of your index without pausing indexing.
• Debugging common problems

Still to come (and if you purchase now, you will receive the added content through MEAP and in the final print book): a chapter on Solr and a case study on our very own LucidFind search system of the entire Lucene ecosystem.

To purchase, visit http://www.manning.com/hatcher3

As background, Solr provides a schema that describes the Fields and Field Types (FT) that are used by an application.  Field Types describe how Solr should handle the information contained in a Field.  For instance, the integer FT tells Solr to treat the contents of any Field of type integer as, you guessed it, an integer.  By integer here, I mean, good old fashioned Java ints.  Solr provides other FTs like long, double, float, string, date, as well as Text (which can be associated with Lucene’s analysis process).  Additionally, Solr provides several “sortable” FTs such as sint, slong, sdouble and sfloat.  Therein lies the confusion.  I think what happens is developers hear the word “sortable” and think they should use the sortable FT for any field they want to sort results by.  However, there is some subtlety here.  Namely, “sortable” FTs manipulate the content so that the lexicographic order is the same as the numeric order for use during search.  Sortables are thus really meant to be used when doing things like range queries (i.e. [price:2 TO 100]) and not for sorting as it relates to returning results.  Due to these required changes, sortables take up more space in the index (and in memory) then their non-sortable compadres.

What’s this got to do with schema design?  Well, this client had three fields, all defined as sortable integer FTs, as in:

1. fieldOriginal  -  The source of the content.  This was the main field used for sorting
2. fieldSearch – Copy field of Original, but rounded to the nearest 100.  This was the main field for searching.
3. fieldFacet – Copy field of Original, but rounded based on a percentage of the original value so as to provide a sliding scale for faceting.  This was the main field used for faceting.

In this case, the client was using the Original for sorting, Search for searching, and Facet for faceting.  They were not doing any range queries, so they did not need fieldSearch to be “sortable”.  Furthermore, the Original field had over 1 million unique terms, so sorting on it was taking up a good chunk of memory and disk space.  The other two fields were smaller, so the cost of sortables was not that big of a deal.  Finally, this field “pattern” was replicated for several other fields as well, some of which also had a significant number of unique terms.

Thus, simply by changing the Fields to use integers where appropriate, we significantly reduced the memory footprint and the disk space required in this client application.

So, as is always the case, play close attention to your schema design.  While the Solr example schema is pretty good out of the box, you shouldn’t just take it as gospel, either.  Spend some time thinking about your needs during design and it will likely save you much time later when debugging and testing your application.

**UPDATE**:  Note, making these changes will require you to re-index.