The 2012 conference will build on the success of last year’s Lucene Revolution in San Francisco. Sponsored by Lucid Imagination with additional support from community and other commercial co-sponsors, we’ll be adding new sessions, new speakers, and new training sessions to the agenda. Lucid Imagination is the commercial entity exclusively dedicated to Apache Lucene/Solr open source search technology.

Registration will begin shortly – so make sure to save-the-date. In the meantime, the Call For Participation (CFP) is now open for Lucene Revolution 2012. If you have a great Solr or Lucene talk, this is a fantastic opportunity to share it with the community.

To submit a proposal for a 45-minute presentation, please complete the form at: http://lucenerevolution.org/Call_for_Participation

Topics of interest include:
- Lucene and Solr in the Enterprise (case studies, implementation, return on investment, etc.)
- Use of LucidWorks Enterprise
- “How We Did It” development case studies
- Lucene/Solr technology deep dives: features, how to use, etc.
- Spatial/Geo/local search
- Lucene and Solr in the Cloud
- Scalability and performance tuning
- Large Scale Search
- Real Time Search (or NRT search)
- Data Integration/Data Management
- Lucene & Solr for Mobile Applications
- Associated technologies: Mahout, Nutch, NLP, etc.

All accepted speakers will get complimentary conference passes. Financial assistance is available for speakers that qualify.

Submissions must be received by March 9th , 2012 , 12 Midnight PST

KEY DATES
March 9, 2012: Call for Participation Closes
March 23, 2012: Speaker Acceptance/Rejection Notification
May 7-8, 2012: Training
May 9-10, 2012: Conference

If you have more than one topic that you would like to propose, please complete an additional online form. To be considered, proposals must be received by 12 Midnight PDT, March 9th, 2012.

Interested in registration or other conference news? Want to be added to the conference mailing list? Is your organization interested in sponsorship opportunities? Please send an email to: info@lucenerevolution.org

We look forward to seeing you in Boston!

Cross post from Lucene Revolution blog.

Once we get Phase2 into trunk we will work on hardening and finishing a couple missing features – then SolrCloud should be ready to be part of the upcoming Lucene/Solr 4.0 release.

If you want to read more about SolrCloud and where we are with Phase 2, check out the new wiki page that we are working on at http://wiki.apache.org/solr/SolrCloud - feedback appreciated!

• Support for the Hunspell stemmer
• The new langid UpdateProcessor
• Numeric types now support sortMissingFirst/Last
• New parameter hl.q for use with highlighting
• Field types supported by the StatsComponent now includes date and string fields

The Solr Reference Guide is available for free online or as a downloadable PDF.  The 3.4 version of the Guide is available in PDF only.

The latest Los Angeles/ OC Apache Lucene/Solr User group meeting was held at Shopzilla in LA. We had Grant Ingersoll from Lucid Imagination speaking at the event. In this talk, Grant spoke about some of the tools available (recommendations, faceting options, amongst others) in Solr and Mahout to aid in the discovery process and how these two open source projects can work together.

Session slides are now available here.

So we decided to try to put together a matrix of major features and show in which versions they are available. Solr 1.4 is pretty old by now, so it naturally appears not to hold up well against Solr 3.5, Solr Trunk, or LucidWorks. Think of it as the base from which the later features in the list grow.

This was an interesting exercise to work through. It’s easy to read through the changes.txt for each release and try to include everything in a list such as this (and our Support guys are probably disappointed that I didn’t do that), but I tried to keep it to the major innovations or bug fixes so it stays somewhat readable. But there’s always the question of whether it’s too much or too little detail.

I hope it’s useful and we’d like to know what you think. Is it worthwhile? Should we go to deeper detail? Could the features use more explanation? Look it over at Feature Matrix for Solr and LucidWorks and share your suggestions in the comments.

Download

You can download the latest version 2.0.1 here.

Install

If you are running LucidWorks Enterprise 1.7 or LucidWorks 1.8, you can use the upgrade scripts and move to version 2.0.1.

For those of you running LucidWorks Enterprise 2.0, you can now upgrade to LucidWorks Enterprise 2.0.1 by following the steps outlined here. You can also find this file with your 2.0.1 (.jar and .zip) package as well.

More Resources

Please visit our documentation page for getting started and in-depth review of the product functionality.

You can participate in our forums and share your experiences, questions, and issues. We actively monitor our forums and respond back to help you with using LucidWorks search platform.

Apache Solr serves search requests at enterprises and the largest companies around the world. Built on top of the top–notch Apache Lucene library, Solr makes indexing and searching integration into your applications straightforward.

Solr provides faceted navigation, spell checking, highlighting, clustering, grouping, and other search features. Solr also scales query volume with replication and collection size with distributed capabilities. Solr can index rich documents such as PDF, Word, HTML, and other file types.

For more details on time and location, visit http://javamug.org/.

Prior to the current 3x branch (which will be released as 3.6) and the trunk (4.0) Solr code, users have frequently been perplexed by wildcard searching being un-analyzed, often manifesting in case sensitivity. Say you have an analysis chain in your schema.xml file defined as follows and a field named lc_field of this type:

<fieldType name="lowercase" class="solr.TextField" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.LowercaseFilterFactory" />
</fieldType>

Now, you index the text “My Dog Has Fleas”. So far, so good. Searching on this field as
field_lc:fleas returns the document, as does field_lc:flea*.

But now you search on field_lc:Flea* and you don’t get any results. What?!?!?! Nearly everyone scratches their heads about this, and it’s a question that often comes up on the Solr user’s list. Users wonder why the analysis chain above isn’t applied to the wildcard queries. It turns out that it’s trickier than you might think at first. What happens when a single input term gets split up into multiple parts? For instance, for those of you familiar with WordDelimiterFilterFactory (WDDF) that can split on case change. What does it mean to parse ‘fleA*’? Applying WDDF might well give the two tokens ‘fle’ and ‘A’ and possibly ‘fleA’. If a wildcard is present, what tokens should be emitted?

1. ‘fleA*’
2. ‘fle*’, ‘A*’, ‘fleA*’
3. ‘fle*’, ‘A*’
4. <insert your solution here>

You can, I daresay, create any rule that suits your fancy. And it’ll be wrong in some situations. Of particular horror is anything that produces ‘A*’ as above, conceptually, you’d than have an enormous OR clause consisting of all the terms that started with ‘A’ in your index. Unless you had a rule like “only do this if the preceding fragment was 2 characters or more”. But then someone would say “I need three characters”, so can WDDF provide a “wildCardMin=#” parameter? I have trouble keeping all the parameters with WDDF and how they interact in my mind already, going down this path would be a nightmare. And I haven’t even considered some of the really interesting issues, like how proximity would be incorporated in all this.

Wildcards aren’t the only issue

The same issue occurs with accent folding, normalizations, and, really, any other component of an analysis chain that somehow changes the query terms. This behavior has mostly been ignored in releases past, it’s been up to the application programmer to manually “do the right thing” before sending the query to Solr. This often involves operations such as lower-casing and accent folding on the application side when a wildcard is encountered.

The new way of handling these cases

As of SOLR-2438 this behavior is no longer true for a number of the most common cases. A query analysis chain that contains any of the following components will automatically “do the right thing” and apply them for multi-term queries. If your analysis chain consists of any of these elements, and you want them applied to “multi-term” queries, you don’t have to do anything at all, it will “just work”. At query time, the indicated transformations are applied to the query terms and everyone is happy. Or should be. Do note that it’s an all-or-nothing operation. All of the elements below that are found in the query analysis chain are applied to the multi-term terms.

• ASCIIFoldingFilterFactory
• LowerCaseFilterFactory
• LowerCaseTokenizerFactory
• MappingCharFilterFactory
• PersianCharFilterFactory

Again, this effectively means you don’t need to care about these transformations any more. One note of explanation, though. I’ve talked about the “query analysis chain”. But what if you don’t have one? Remember that your <analyzer> tag can have several possible ‘type’ parameters; “index”, or “query”, or none. Well, if a ‘ type=”query” ‘ is found, that analysis chain is inspected and any of the above components are recorded to be used on multi-term queries. If no ‘ type=”query” ‘ is found, then the ‘ type=”index” ‘ is used. And if no ‘ type=”index” ‘ is found, than the one with no ‘type’ parameter is used.

What does “multi-term” mean anyway?

I’ve also sprinkled the phrase “mult-term” around, and sometimes “wildcard”. It turns out that the simple wildcard case is a specialization of a broader category of queries, including:

• wildcard
• range
• prefix

All of these are now handled as above.

Expert level schema possibilities

All of the above is automatic, but there are three immediate questions:

• what about some of the other components?
• what if I need the old behavior?
• what if I want something completely different?

It turns out that all three of these questions have the same answer. But before I outline it, I want to emphasize that you very probably don’t need to care about what follows! You might need to know about this in special cases, so I’ll mention it here.

In the above explanations, I wrote that “analysis chain is inspected and any of the above components are recorded to be used on multi-term queries”. Well, what actually happens is that there’s a new analysis chain in town that can be specified in the schema.xml file called, you guessed it, “multiterm”. You specify it like this as part of a <fieldType>:

<analyzer type="multiterm" >
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  <filter class="solr.ASCIIFoldingFilterFactory" />
  <filter class="solr.YourFavoriteFilterFactoryHere" />
</analyzer>

You can put any component that’s legal in a ‘type=”index” ‘ or ‘type=”query” ‘ analysis chain. If you wanted, for instance, to enforce the old-style behavior, you could specify

  <tokenizer class="solr.KeywordTokenizerFactory" />

as the entire “multiterm” analysis chain. It seems a bit odd to use KeywordTokenizerFactory here, but this applies to the individual terms, not the entire input. So it’s in effect saying “don’t analyze the terms at all”. Sound familiar? This is just what happened historically.

How does this relate to the automatic behavior?

Well, what really happens under the covers is that if you don’t define your own “multiterm” analysis chain, Solr constructs one for you from the analyzers you have defined as outlined above; query, index or default, in that order.

Waaaaay under the covers, down in the code

All this is accomplished by making components “multiterm aware”. This means implementing the “MultiTermAwareComponent” interface. Currently, the components listed above are the only ones that implement this interface, but others may be good candidates, and some of these are listed in JIRA SOLR-2921. By and large, implementing these in the code may be trivial. What’s not trivial is understanding what “the right thing” is. Some examples:

• stemmers
• various language-specific normalization filters
• various language-specific lowercase filters.
• various ICU filters

The reason these haven’t been made “multi term aware” is the usual open-source reason; “What we have is a good step forward, we shouldn’t delay everything in order to get the last use cases taken care of”. In other words the implementors (me in this case, with lots of help from others) are tired .

Anyone who really understands what the right thing to do in the cases of components that do not yet implement “MultiTermAwareComponent” and could provide use cases for them would be giving us a great help, especially by providing examples illustrating the correct inputs and outputs for wildcard cases. And some examples of what should not come out as well. Or even better, a draft JUnit test that would show the expected behavior. Or even better yet, a full patch!

Any modification that potentially produces more than one token needs to be handled with care, see the code for LowerCaseTokenizerFactory for a case in point. Consider that Solr will now throw an exception if the transformation produces more than one token, so tread cautiously!

This change should remove a long-standing point of confusion for solr users. We’d be very interested in any feedback from the community, and especially any problems that crop up. SOLR-2438 has patches for both the 3x and 4x code lines, but it’s probably easier just to get a current 3x or 4x branch (or nightly build) if you want to test this “in the wild”; the code has been committed and built. There remains some work to be done to incorporate this change for more analysis components, anyone want to volunteer?

Resources:

This page on the Solr Wiki has the Wiki documentation: Multi Term Query Analysis

Main JIRA (already in 3.6 and 4.0 code lines): SOLR-2438

JIRA for other components not yet “multi-term aware” that are possibilities in the future: SOLR-2921

November 27 2011, Apache Lucene™ 3.5.0 available

The Lucene PMC is pleased to announce the release of Apache Lucene 3.5.0.

Apache Lucene is a high-performance, full-featured text search engine

library written entirely in Java. It is a technology suitable for nearly

any application that requires full-text search, especially cross-platform.

This release contains numerous bug fixes, optimizations, and

improvements, some of which are highlighted below.  The release

is available for immediate download at:

http://www.apache.org/dyn/closer.cgi/lucene/java (see note below).

See the CHANGES.txt file included with the release for a full list of

details.

Lucene 3.5.0 Release Highlights:

* Added a very substantial (3-5X) RAM reduction required to hold the

terms index on opening an IndexReader. (LUCENE-2205)

* Added IndexSearcher.searchAfter which returns results after a

specified ScoreDoc (e.g. last document on the previous page) to

support deep paging use cases. (LUCENE-2215)

* Added SearcherManager to manage sharing and reopening IndexSearchers

across multiple search threads. Underlying IndexReader instances are

safely closed if not referenced anymore. (LUCENE-3445, LUCENE-3558)

* Added SearcherLifetimeManager which safely provides a consistent

view of the index across multiple requests (e.g. paging/drilldown).

(LUCENE-3558, LUCENE-3486)

* Renamed IndexWriter.optimize to forceMerge to discourage use of

this method since it is horribly costly and rarely justified

anymore. (LUCENE-3439)

* Added NGramPhraseQuery that speeds up phrase queries 30-50%

when n-gram analysis is used. (LUCENE-3426)

* Added a new reopen API (IndexReader.openIfChanged) that

returns null instead of the old reader if there are no changes

in the index. (LUCENE-3464)

* Improvements to vector highlighting: support for more queries

such as wildcards and boundary analysis for generated snippets

(LUCENE-1824, LUCENE-1889)

* IndexSearcher and IndexReader now perform additional checks to

throw AlreadyClosedExceptions if searches are performed on a

closed IndexReader. Performing searches on already closed reader

can cause JVM crashes when invalid memory mapped files are

referenced.

* Several bugfixes, including a bug where closing an NRT reader

after the writer was closed was incorrectly invoking the

DeletionPolicy. See CHANGES.txt entries for full details.

27 November 2011, Apache Solr™ 3.5.0 available

The Lucene PMC is pleased to announce the release of Apache Solr 3.5.0.

Solr is the popular, blazing fast open source enterprise search platform from

the Apache Lucene project. Its major features include powerful full-text

search, hit highlighting, faceted search, dynamic clustering, database

integration, rich document (e.g., Word, PDF) handling, and geospatial search.

Solr is highly scalable, providing distributed search and index replication,

and it powers the search and navigation features of many of the world’s

largest internet sites.

This release contains numerous bug fixes, optimizations, and

improvements, some of which are highlighted below.  The release

is available for immediate download at:

http://www.apache.org/dyn/closer.cgi/lucene/solr (see note below).

See the CHANGES.txt file included with the release for a full list of

details.

Solr 3.5.0 Release Highlights:

* Bug fixes and improvements from Apache Lucene 3.5.0, including a

very substantial (3-5X) RAM reduction required to hold the terms

index on opening an IndexReader. (LUCENE-2205)

* Added support for distributed result grouping. (SOLR-2066,

SOLR-2776)

* Added support for Hunspell stemmer TokenFilter supporting stemming

for 99 languages. (SOLR-2769)

* A new contrib module “langid” adds language identification

capabilities as an Update Processor, using Tika’s

LanguageIdentifier or Cybozu language-detection library (SOLR-1979)

* Numeric types including Trie and date types now support

sortMissingFirst/Last. (SOLR-2881)

* Added hl.q parameter. It is optional and if it is specified, it overrides

q parameter in Highlighter. (SOLR-1926)

* Several minor bugfixes like date parsing for years from 0001-1000, ignored

configurations when using QueryAnalyzer with SpellCheckComponent

and many more.

See CHANGES.txt entries for full details.

Note: The Apache Software Foundation uses an extensive mirroring network for

distributing releases.  It is possible that the mirror you are using may not

have replicated the release yet.  If that is the case, please try another

mirror.  This also goes for Maven access.

Happy searching,

Apache Lucene/Solr Developers