So we decided to try to put together a matrix of major features and show in which versions they are available. Solr 1.4 is pretty old by now, so it naturally appears not to hold up well against Solr 3.5, Solr Trunk, or LucidWorks. Think of it as the base from which the later features in the list grow.

This was an interesting exercise to work through. It’s easy to read through the changes.txt for each release and try to include everything in a list such as this (and our Support guys are probably disappointed that I didn’t do that), but I tried to keep it to the major innovations or bug fixes so it stays somewhat readable. But there’s always the question of whether it’s too much or too little detail.

I hope it’s useful and we’d like to know what you think. Is it worthwhile? Should we go to deeper detail? Could the features use more explanation? Look it over at Feature Matrix for Solr and LucidWorks and share your suggestions in the comments.

There is special class of software programs called monitoring software that are widely used among system and network administrators. In our case we would like to monitor not only OS level metrics, but also Solr internal parameters and act accordingly. LucidWorks and Apache Solr provide lots of valuable information through a JMX interface, so you can hook that up into your monitoring tool.

Zabbix is one of the most popular open source monitoring tools. It has many features like an easy-to-use web interface, different ways to gather metrics data, an ability to keep this data in persistent storage, built-in graphing, notifications and alerting, flexible configuration and many more. One of the most compelling features of integrating it with Apache Solr is built-in JMX support (available only in Zabbix 2.0 beta release). Using this feature you can easily configure Zabbix server to pull JMX metrics out of any LucidWorks or Solr application. This is because all configuration settings (JMX attributes, graphs, triggers) are stored centrally on a Zabbix server, which means you can add a new attribute for all monitored servers or change the pulling frequency for servers with a single click.

Here are example graphs you can build in Zabbix:

1. Total number of documents in Solr index

2. Search activity – number of search requests, errors and timeouts 
Solr request handlers provide cumulative counter for number of requests,  but you are probably more interested in number of search requests per specific period of time, like per minute or per second. The trick here is that Zabbix provides a way to setup monitoring to store not the value as-is, but as a delta (simple change value or speed per second).

3. Solr document operations (adds, deletes by id or query)

4. Crawling activity
LucidWorks provides different connectors/crawlers which you can use to index documents into Solr. It also provides additional statistics about crawler behavior, like total number of documents, new and deleted documents, number of updated documents in iterative crawl, failures, etc.

5. Solr index operations (commits, optimizes, rollbacks)

6. Search Average Response Time
Solr search request handler provides cumulative avgTimePerRequest value. The problem with this attribute is that  when your applications is running in production for a significant amount of time, current short term performance problems won’t cause significant effect on this aggregate metric. The solution is to use a Zabbix calculated item on delta change for totalTime and requests attributes. Here’s math expression to calculate average search response time for the last 5 minutes:

sum("jmx[\"solr/collection1:type=/lucid,id=org.apache.solr.handler.StandardRequestHandler\",\"totalTime\"]",300)/sum("jmx[\"solr/collection1:type=/lucid,id=org.apache.solr.handler.StandardRequestHandler\",\"requests\"]",300)

7. Solr searcher warmup time
This is an important metric if you pursue fast commit rate (near real time indexing) and don’t want to sacrifice fast faceting performance. You can configure monitoring tool to send alert in case of warmup time exceeds some pre-defined threshold.

8. Filter, query results and documents caches statistics (cache size, hits, hitratio, evictions, etc)

9.  Java Heap Memory Usage

How would I know if my search server is down?

There are two options – the obvious one is to set up your monitoring tool to issue search requests and verify response status or specific text on a search results page. Another option is to check the last time your monitoring tool retrieved an arbitrary JMX attribute from your application and assume the server is down if it’s longer than expected. In Zabbix there’s special function nodata which you can use to achieve that.

How would I know if I’m reaching a limits of my server and pro-actively react on this?

This is a complex issue as there are many things that can go wrong (such as JVM heap memory, CPU load, disk space, file descriptors, etc.) and you should monitor them all. Zabbix has great example templates for OS and Java triggers that allow you to keep an eye on all those parameters.

For more information about Solr and LucidWorks JMX support, instructions how to configure Zabbix and Nagios, Zabbix configuration templates and other helpful tips please see the Integrating Monitoring Services section on Lucid documentation portal.

Our main objective with the 1.7 release is to provide you with improved enterprise readiness as well as core search enhancements, in particular Search enhancements from Solr 4.x trunk

The rapid rate of innovation in the Solr/Lucene development community is a double-edged sword. The Apache Solr/Lucene 3.1 release just came out last week. However there are several bug fixes, performance improvements, and features already contributed by the Solr community in 4.x trunk. It is hard to keep up with the nightly builds to take advantage of the community contributions. Our objective at Lucid is to give you more frequent and certified versions of search functionality. LucidWorks Enterprise 1.7 contains a fully tested and certified version of Solr 4.x branch. You no longer have to try his at home, we do it for you. You can build your search applications leveraging some of the latest Solr search improvements with LucidWorks Enterprise.

Other highlights of this release include:

New in LucidWorks Enterprise 1.7

• Search result grouping / Field Collapsing improves user experience and simplifies development by grouping multiple similar results as a single entry ; for example as products within a price range, multiple emails in a single threaded conversation, parts within a category. Also provides the ability to group by query – retrieves the top documents that match the query, not just the count additional info  (http://www.lucidimagination.com/blog/2010/09/16/2446/)
• Integration with UIMA for metadata extraction improves versatility of metadata driven analytics and operations, such as using Pivot Faceting. It treats each facet as a dimension, creating facet counts for a multi-dimensional matrix. For example, for each category in a story inventory, it can show how many products are in stock within the facets controls, rather than just documents retrieved. (http://wiki.apache.org/solr/SolrUIMA)
• Sort result sets by functions, so you can tweak the search relevancy result set using functions to create relevancy algorithms within the result of a query, eliminating the need to write custom rankers for common scenarios. (http://yonik.wordpress.com/2011/03/10/solr-relevancy-function-queries/)
• Spell check against the existing index improves efficiency by saving the need to build a special dictionary for the spellcheck component, as it automatically checks entries without having to create explicit misspelled entries
• Numeric range facets (similar to date faceting) organize and analyze data and content based on quantitative parameters, such as size, dimensions, prices, etc.
• New spatial search, including spatial filtering, boosting and sorting capabilities better integrate  location-based information into search applications
• A new Auto Suggest component simplifies rapid retrieval of candidate results interactively as the user types
• Fine-grained control of data acquisition timing and sources, both for application development and ongoing data-management and indexing for production search applications, simplifying integrated crawling configuration and management
• SharePoint CMS Connector A new connector for Sharepoint, which lets you crawl and index the content of your Sharepoint server as easily as other conventional content management systems, web servers, and databases, right from the LucidWorks Enterprise console (for Microsoft Office SharePoint Server 2007, Microsoft Windows SharePoint Services 3.0, SharePoint 2010); it also seamlessly integrates sharepoint document and user level ACLs  directly from the LucidWorks Enterprise UI.
• Security Enhancements LucidWorks Enterprise 1.7 improves integration and validation of security into search application development and deployment, with streamlined control of user privileges, groups, and LDAP integration. LDAP configuration activation, configuration and validation can be done via UI, in addition to ReST API configuration to simplify development and streamline production integration.

LucidWorks Enterprise is free to download for unlimited development and test use; production deployment requires an enterprise subscription. Subscriptions include production deployment support, business hours or 24/7 SLA-based incident management support and ExpertLink on-demand consulting. The upgrade is free for current customers.

You can download the product here. For the full set of functionality refer the documentation here. Of course, we welcome your inputs — help us improve the product by providing your feedback and suggestions in our Forums.

During a past ecommerce webinar with Brian Doll of Sheetmusicplus.com, I posted a checklist of items that are commonly occurring in many ecommerce applications and then I waved my hands, due to time constraints, and said Solr (and now LucidWorks) can do almost all of them out of the box and left the rest as an exercise for the reader.  (Note, the slides are available here.  Registration required.)  Well, now I have some time, so let me fill in the blanks with some more concrete examples about how to do this.

Setup

For this example, I am using real estate data freely available from the NYC government.  The reason I am interested in this data is that it is:

1. Free.
2. It has product-like data in it, as in: name, description, a bunch of metadata and price
3. It’s mostly real (I embellished it with descriptions and a few other pieces and filled in some missing pieces of data, see the Indexer class in the source code.)  In fact, it’s so real, that when setting up the app, one quickly sees how noisy the data is in terms of things like missing values, etc.  For instance, 1804 records don’t have the year built specified.

I have setup a Solr schema for this data as well as some tools for indexing the data.    To run the demo, you will need:

1. Java 1.6
2. Ant 1.7.X
3. Download ecommerce.zip (73 MB)

Once you have the prerequisites in place, take the following steps:

1. Unzip the ecommerce.zip file into the directory of your choice
2. cd lucid_ecom
3. In a separate terminal window: cd solr
   1. java -jar start.jar (just as if you were running the Solr tutorial.  Note, I am running a relatively recent version of the Solr 3.x branch)
4. Point your web browser at http://localhost:8983/solr/nyc and take a moment to familiarize yourself with the interface.

Once you have completed step 4, you should see something like:

(NOTE: I’m not a graphic designer.  I tried to create a reasonable UI w/o spending a ton of time on every last piece of it.  Also, I used the VelocityResponseWriter built into Solr.  It’s nice for prototyping, but it “ain’t” for production use.)

A pre-built index is included in the Zip file, but if you wish to index it yourself, run:

1. ant delete-all (deletes the existing content)
2. ant index

With the working application in place, let’s take a look at how to implement the various checklist items.

Implementing the Checklist

I’ve broken out each checklist item below and will cover each of them in more detail in the following subsections.

Keyword search

There really isn’t much to be said here other than Solr has built in support for querying in all the “usual” ways that one would expect out of a search engine.  Keywords, phrases, wildcards, fielded search and much, much more.  For example, try:

1. http://localhost:8983/solr/nyc?q=tottenville or just type tottenville in the search box.
2. http://localhost:8983/solr/nyc?q=5+bedrooms+%22Staten+Island%22 (5 bedrooms “Staten Island”)
3. http://localhost:8983/solr/nyc?q=5+bedrooms+borough_display%3ABro* (5 bedrooms borough_display:Bro* — Should match all 5 bedrooms in either the Bronx or Brooklyn)

Take some time and try out your own queries.  In our example, we are using the extended Dismax Query Parser, in case you want to learn more about how it works.

High Quality relevance (precision @ < 10)

In many search applications, and ecommerce is no exception, users often abandon searches when the first page of results (often the top 10) are not relevant to their query.  Thus, it is important that a search engine return good results on the first page.  While some guidance (more on this in the coming sections) can help alleviate the abandonment problem, a strong first showing is often the quickest way to more clickthroughs.  Since Solr utilizes Lucene, which implements an industry standard vector space approach to search, results are often quite good out of the box.  Nevertheless, many ecommerce applications may need one or more of the tools that Solr/Lucene provide out of the box to tweak relevance, such as:

1. Document, field, token boosting (i.e. matches in the title field are more important than matches in the description.)
2. Query term boosting (provide weights for different terms, such as synonyms.)
3. Disjunction Maximum Query scoring (aka the “dismax” parser or the extended dismax parser) for dealing with cross field matches.
4. Automatic phrase generation from multiword queries even when the user did not explicitly quote the keywords.
5. The ability to override low-level scoring information such as term frequency, document frequency, document length normalization and coordination factors.
6. Function queries (more later) to allow values in fields (such as price) to be factors in scoring.
7. Editorial Boosting/Sponsored Results (in Solr-speak it’s called the QueryElevationComponent — more later) to place specific results at the top.

Relevance tuning is a complex subject and one that is best viewed in the light of your data.  In summary, make sure you are making decisions about relevancy based on the big picture and try to avoid any local minima (i.e. tuning a specific query to the detriment of breaking lots of other queries.)  In other words, make sure your top money making queries aren’t effected by you “fixing” a one or two bad queries.  To learn more, see my articles on Improving Findability and Debugging Relevance.  With the basics out of the way, it’s time to take a look at faceting and discovery tools

Faceting/Discovery

One of Solr’s most appealing features is its out of the box support for faceting (sometimes called navigators, parametric search, guided navigation) in a number of different ways (see http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Faceted-Search-Solr for a primer.  Also see http://wiki.apache.org/solr/SimpleFacetParameters)  In the example application, the left hand nav area shows facets for things like borough (field based faceting), sale price (numeric range faceting), sale date (date range faceting) and pet friendly (facet by query).   Solr also supports “multi-select” faceting (see http://search.lucidimagination.com for an example.)  And, while there isn’t support for true hierarchical faceting in Solr yet, there are ways to achieve it through intelligent modeling of your tokens.  Last, but not least, you may find https://issues.apache.org/jira/browse/SOLR-792 useful for doing grouped faceting (color: red, size: large).

Additionally, helping customers discover items of interest goes well beyond facets.  Features like Did you mean, Related Items/Searches, Collaborative Filtering/Recommenders (see Mahout for an open source solution), Auto Suggest and others can go a long way in increasing the user’s ability to purchase items from your store.  Many of these features I’ll cover below.

Flexible language analysis tools

Multilingual support

Solr contains support for most of the commonly spoken languages in the world, including English, Chinese, French, Spanish, Korean, German, Thai and many more.  Lucene and Solr are also UNICODE compliant.

Frequent Incremental Updates

Lucene, and thus, Solr has supported incremental updates from it’s inception without the need to re-index the whole collection.  It is also very fast at making new documents available for search.  Additionally, with the combination of recent and upcoming work in Lucene, real time search should be available soon.  The one piece that is still missing is individual field update, but for certain types of fields (ratings, for instance), there may be easy workarounds.

Ratings and Reviews

In working with many ecommerce customers on Solr, there are usually questions around how to incorporate ratings and reviews into search results without skewing results or introducing too much noise.   On the ratings side, app developers often want to incorporate the aggregate rating of an item as a boost factor in the overall score.  I will discuss how to do this in detail in the section titled Editorial Relevance Controls below.  Meanwhile, on the review side, it is often the case that too much noise is introduced by including reviews “on par” with matches in the product title or description.  For instance, if I’m selling “Widget X” and a review for a different product says something like “You should also check out Widget X”, bringing back a match on that second product really isn’t all that useful for a customer searching for “Widget X”.   To deal with this noise, people often take a couple of different approaches:

1. They weight review matches lower than product matches via boosting (either at query time or indexing time)
2. They only search reviews if they don’t feel they have high quality matches for the main product search

You could also do some type of post processing analysis (NLP) of the review to see if it is on topic, but this approach likely isn’t viable for most people in most situations due to the processing power and accuracy of such a solution.  As for #2 above, see my post on Fake and Invisible Queries for more insight.

Auto-suggest

Auto suggest (aka auto complete) is one of the cheapest (in terms of development costs) mechanisms available for enhancing the chance that users find what they are looking for.  I’ve heard of vendors adding auto-suggest and having it add millions to their bottom line.  Simply by providing a drop down list of ways of completing what a user has typed so far an application can do a number of things:

1. Reduce spelling errors thus leading to lower frustration and better results sooner rather than later
2. Seed the user with items that they may want but weren’t explicitly looking for.  After all, an intelligent auto-suggest box can very easily not only give completions, but it can also hook in related items too.
3. Short-circuit search all together and go directly to a landing page for a specific search

For the demo, I implemented auto-suggest using SOLR-1316, which should be committed to trunk soon.  Note, also, there are other ways of doing auto-suggest, too, including using the TermsComponent and Faceting.  Here are the steps I went through to make auto-suggest work:

1. Applied the SOLR-1316 patch to the 3.x branch.  This required a minor tweak to the HighFreqDictionary.java file.  See patch below
2. Add the necessary piece to the solrconfig.xml.  See the /autosuggest SearchComponent in the solrconfig.xml in the appendix.
3. Decide what fields to use in building the auto-suggest index (see schema.xml).  I then “copy fielded” these into a field named suggest.  Note that I used a non-stemming analyzer.  I also used Solr’s word-based n-gram filter with a shingle base of 5 so as to give phrase suggestions too.  Note, this is intended for demonstration purposes, as you may wish to not use shingles and append terms as the user types or you may want to use a different value for n.  Also note, I did not spend much time at all on evaluating what went into the suggest field that is used as a source.  You will want to validate it and make sure it is aligned with your business goals.
4. Build the auto-suggest data structures via the Spell Checker build command (see the next section)
5. Modified the jQuery script that is in the Solr VelocityResponseWriter example to use the SOLR-1316 output instead of the TermsComponent output.  See the autocomplete.vm file for details on the Javascript.  See the next section on Did You Mean on how to make requests to the the auto-suggest component, as it uses the same mechanism as the spell checker.

Hopefully, from here you will have enough information to build you your auto-suggest capabilities.  If not, see our search site for more info, including alternate approaches to the SOLR-1316 patch.

Did You Mean?

Just like auto-suggest, spell checking can be helpful to users in finding what they are looking for, especially given the propensity of manufacturers/product designers to use incorrectly spelled words in their product name in order to better “brand” the product.  Good spell checking goes beyond merely hooking up a dictionary of terms, it is also quite important to know when to suggest a term and when not suggest a term.  Lucene/Solr has the basics of setting up spell checking covered via the SpellCheckComponent, but a good spell checking application will need to go beyond merely setting up the component in order to achieve good results.  First things first, however, let’s take a look at getting spell checking setup and then we can examine what is needed to make it better.

First, we need to configure the SpellCheckComponent in the solrconfig.xml file.  There is an example of this in the Solr tutorial example, from which I changed the distance measure from the Levenstein edit distance to the Jaro-Winkler distance.  The reason I did this is based on past experience that users tend to misspell words towards the end of the word and not the beginning, which the Jaro-Winkler distance accounts for.  My configuration looks like:

<searchComponent name="spellcheck">
 <str name="queryAnalyzerFieldType">textSpell</str>
 <lst name="spellchecker">
 <str name="name">default</str>
 <str name="field">spell</str>
 <str name="spellcheckIndexDir">./spellchecker</str>
 <str name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance</str>
 </lst>
<!-- ... -->
 </searchComponent>

The whole point of a SearchComponent such as the SpellCheckComponent is to hook it into the main Solr request processing instead of having to make a separate call.  Thus, I hooked the SpellCheckComponent into the /nyc RequestHandler so that all queries that are submitted to the “main” RequestHandler will also be spell checked.  Once the configuration is setup, the spelling index must be built (and maintained.)  This is handled by issuing an &spellcheck.build=true command to the spell checker, as in:

http://localhost:8983/solr/autosuggest?q=man&spellcheck=true&wt=xml&rows=0&indent=true&spellcheck.build=true

(Note, the &q param can be anything.)

Once the configuration is hooked up and the spell checking data structure is built, the last piece is to hook it into the UI.  (Note, I setup the solrconfig.xml to automatically do spell checking on every query request.)  To hook into the UI, I co-opted the suggest.vm file and spruced it up a bit to provide links, etc.  Other than that, it is exactly the same, since both are just different implementations of spell checking.

See the Solr wiki on the SpellCheckComponent for more information.

Related Searches/Items

In many ecommerce applications, stores position related items next to a particular item so as to inspire the user to either buy an additional item or offer an alternative.  Naturally, the “relation” is determined by the store and might take on a variety of forms, such as: accessories, enhanced versions, cheaper versions, alternatives from different manufacturers or popular items based on other users.  Similarly, a store may wish to give users not only suggestions and spelling corrections, but they may also want to give users alternative search terms or other popular searches.  For instance, if a user searches for TVs, a store may want to suggest they search for “LCD TVs” or “HD TVs”, etc.

When it comes to related items, many Solr users rely on either hand-crafting a second query (given an original query and a particular item) by using the original terms of the query and some of the terms that describe the item.  For instance, an application might use the category of the item plus some of the keywords for that item to then craft the query, submit it to Solr and then display the first few results.  This approach can also be done automatically using Solr’s built in More Like This (MLT) capability, but you may need to do some tuning to get the results you desire.  For the sake of the example, I incorporated MLT into the application.  You can see it on the left hand side, just below the map, under the “Similar Properties” heading.  The configuration of MLT was done in the solrconfig.xml file as part of the /nyc RequestHandler.  Note, in a typical application you may not wish to generate MLT results for a search query, but instead only provide them once a user chooses a particular document, as MLT can add a fair amount of overhead to the process.  Other Solr applications will often calculate related items off line or through some type of collaborative filtering approach (see Apache Mahout’s recommender capability for an open source library to do this) and either add the information to the document and re-index or integrate it at the application level.  In these cases, it’s not hard to integrate, but it is beyond the scope of this article.

As for the functionality to add related searches, there is not currently support built into Solr, but there is a JIRA issue open to track the idea.  Related searches can often be determined through a combination of log analysis (look for patterns in a user session) and synonyms or via collaborative filtering/recommenders.  Also, have a look at Mahout’s Frequent Pattern Mining capabilities.  One could also index the queries into another index (Solr core) and simply issue fuzzy queries to it.

Editorial Relevance Controls

Administration

Administration means many things to many people.  To the IT department, it means easy setup, configuration, monitoring, maintenance, scalability, fault tolerance, etc. while to the business user it means tools for manipulating results, reporting search statistics and following through on business goals.  While the latter is important, I am going to focus on the IT dept. needs for the sake of this article.  Solr is very easy for an IT person to get setup and have a baseline configuration in place.   I’ve seen customers (without my help) be up and running and searching their data in non-trivial ways in as little as 30 minutes, sometimes less.  As for monitoring, Solr comes with web pages that report status as well as JMX integration.  I’ve also seen Solr integrated nicely with Nagios, Cactus and other tools.  Lucid Imagination also partners with New Relic to offer Solr specific monitoring tools.

As for the big questions about scalability and fault tolerance, the answer is an unequivocal yes.  High traffic ecommerce sites like Zappos, Netflix, CNET, AOL and many others use Solr to server their search needs.  Solr can be setup to both handle large indexes and high query volumes.  For more information on how to do this, see Mark Miller’s excellent article on scaling Solr.

Recommendations (See Mahout)

For both online and offline recommendation calculations, see the Apache Mahout project, which has an excellent collaborative filtering library.  While integration with Solr does not yet exist, Mahout does expose web services (as well as Java APIs) for its recommender engine, so it is feasible to integrate it within an application.

Analytics and other Business Tools

Analytics are probably Solr’s weakest area, but that being said, we find that many customers already have platforms in place (like Omniture) that they can easily integrate Solr into.  This often saves business users from having to learn yet another tool.  As for other business tools, Solr likely does not have them (for instance, merchandising tools), but again, many people find it straightforward to integrate Solr into existing tools.  Also, this is an area that LucidWorks, with it’s administrative UI really can help.  It has screens and tools for doing log analysis and seeing what popular queries are, as well as popular terms and zero result values.

Looking Forward

Solr is a very popular and capable search engine for ecommerce and, looking forward, it is only getting better.  With a focus on greater features (spatial search, for instance), the latest Lucene and easier scalability, the next version of Solr promises to be even better.

Appendix A

Items needed here: schema, solrconfig, SOLR-1316 3.x branch patch

Before I dig in too much, though, a special thanks to all our customers at Lucid Imagination as well as to my coworkers.  I’m coming up on 15 years out in the “real world” and I can honestly say I’ve never enjoyed what I do as much as I do here and that even accounts for the normal rough patches one goes through in any job.  As an engineer, there are few things as cool as getting to work with customers who are not only using, but pushing your work/project/product on a daily basis to do new and interesting things (I think this is a direct result of the project being Open Source, which I believe has an inherently lower cost of experimentation).  I’ve been fortunate enough to meet and talk with many people doing all kinds of things with Lucene and Solr ranging from the “mundane” of basic keyword search to those building next generation search capabilities at incredible scale.  Through it all, I’m constantly amazed at the flexibility and efficiency of Lucene and Solr.  For instance, I’ve been working with one customer now whose Solr-based solution (for the exact same content) will use ~50% less hardware and will have an index that is 1/6 the size of their FAST index all while saving them major dinero.

Speaking of Lucid, one of the highlights of the year for us that relates directly to Lucene and Solr is the launch of our enterprise version: LucidWorks Enterprise.   I like to think of it as Apache Solr with a whole lot of Lucid expertise on how to use Solr baked in and topped off with other features and functionality to make building search applications easier.

OK, time to move on to the open source projects…

1. Without a doubt, the biggest news of the year is the merging of the Lucene and Solr code base as well as the “graduation” of several subprojects to Apache Soft. Foundation Top Level Projects (TLP).  The graduating projects are Tika, Nutch, and Mahout.  We also spun Lucy (a C port) to the Incubator, where it is working on it’s own community.  These moves were primarily done to focus the project management on single code base, but they also demonstrate the project has reached a level of maturity at the ASF.  The move also has the side benefit of bringing each project higher visibility.
2. I’m particularly excited about the addition of OpenNLP to the Apache umbrella.  OpenNLP is a nice open source Java project for natural language processing that has lived at Source Forge for quite some time.  I would expect development to grow quite a bit under the ASF community based model.  Also, integrating OpenNLP with Solr and Lucene is pretty easy to do.  I would be remiss if I didn’t also give a nod to the addition of the ManifoldCF project to the ASF.  ManifoldCF will help unlock content in Sharepoint, Documentum and other repositories for users of Lucene and Solr.
3. Lucene’s trunk code base now implements our “Flex APIs”, which should allow users to have near total control over what goes in the index as well as alternate compression techniques, different scoring models, etc.  See Michael McCandless’ excellent talk at Lucene Revolution for more details.
4. With all the location aware devices and capabilities on the market, geo-spatial search is a hot topic and Lucene and Solr have been adding quite a bit of capabilities in this regard with the ability to filter, boost and sort results based on location information in documents.  See Solr’s Spatial Search Wiki page for more info as well as several of my past blog posts.
5. Of course, everyone was a buzz about the cloud this year.  For Solr, this translates into greater efforts to make Solr easier to scale to very large installations (100s to 1000s of nodes and billions and billions of documents) via the Solr Cloud project that Yonik Seeley and Mark Miller have been spearheading.
6. On the user side, one of the biggest pieces of buzz this year related to Lucene was the migration of Twitter search to Lucene.  At 1 billion queries per day and 50 million posts per day (all indexed and searchable in near real time), Twitter’s search system certainly has it’s work cut out for itself.  However, as Michael Busch outlined at Lucene Revolution, Apache Lucene was up to the task!  Naturally, there were lots of other companies that migrated to Solr and Lucene as well.  Have you shared your use case?

Well, I’ve no doubt missed a bunch of other things, but those items, to me, are some of the bigger highlights.  Looking forward, there are some other exciting things coming to Lucene and Solr.  In particular, I’m working on adding language identification, related searches and point in polygon filtering to Solr.  I would also expect we will release Lucene/Solr 3.1 fairly soon, too, but you can’t pin me down on a date just yet.

Here’s hoping you all have a Happy Holidays and a Happy New Year!

The Lucid Query Parser is distributed as part of LucidWorks Enterprise, and it enhances your users’ ability to pinpoint information to be retrieved from LucidWorks Enterprise. Some LucidWorks Enterprise developers have opted to allow their users the full power of the parser, which is a viable strategy, as the parser is completely resilient to syntax errors, and will never pop up an exception. Others have developed a pre-parser, which modifies the query, based on their application specific needs.

As for us, at Lucid Imagination, we opted to give our users at search.lucidimagination.com to benefit from the parser as is; looking at the query logs, we see some sophisticated queries being submitted. For those who fully utilize the parser’s capabilities, great job!

As always, during the demo, you guys presented me with questions about Lucid Query Parser. Here’s a recap of them.

Q: Which features (if any) will now go into open source Solr development and which features will go into LucidWorks Enterprise?
A: LucidWorks Enterprise uses a full version of Solr, without any modifications or changes. In fact, it is extremely important for us at Lucid Imagination to maintain the innovation and leadership of Apache Solr. As a rule of thumb, any code that touches core Solr, is developed in Apache Solr first, before it is introduced to LucidWorks Enterprise. Such were the cases of Solr Cloud and Field Collapsing, a Lucid Imagination contribution to Solr. For a non core features, such as the admin UI and the ReST API, we are evaluating these features on a case by case basis, and might open source them as we see fit.

Q: Do you have a write-up describing which features are in which layers: Lucene, Solr, LucidWorks Enterprise?
A: Lucene, Solr and LucidWorks Enterprise are complementary technologies that depend on each other, and offer very similar underlying capabilities. In choosing a search solution that is best suited for your requirements, key factors to consider are application scope, development environment, time to deployment factor, rate of change and growth, and software development preferences.

Lucene is a Java technology-based search library that offers speed, relevancy ranking, complete query capabilities, portability, scalability, and low overhead indexes and rapid incremental indexing. This is the technology at its raw form – fully exposed, tweak-able and customizable, but requires java application development to access all of the capabilities.

Solr is the Lucene Search Server. It presents a web service layer built atop Lucene, and extending it to provide developers with an easy to use search server. Solr brings with it operational and administrative capabilities like web services, faceting, configurable schema, caching, replication, and administrative tools for configuration, data loading, statistics, logging, cache management, and more.

LucidWorks Enterprise adds a management, configuration, and application integration atop of Solr, and extending it to provide application developers a ready to use search platform. LucidWorks Enterprise brings with it a management and configuration ReST API for rapid integrations, integrated crawler for data acquisition, admin and search UI, industry strength query parser, document security with connectivity to LDAP systems, and more.

Q: Is the query parser part of Solr release or a feature only for LucidWorks Enterprise?
A: Yes, LucidWorks Enterprise contains some additional value add components, allowing for speedier development process, broader developer skillsets, and lower cost of growth and change adaption.

Q: Can the query parser be used independently of the rest of LucidWords Enterprise?
A: Please consider the following points, before diving in to the individual LucidWorks Enterprise components – All LucidWorks Enterprise value-add components are licensed under the Lucid Imagination developer license (link HERE) allowing you to develop, test, evaluate and integrate for free, however, production use of LucidWorks Enterprise value-add components requires a subscription service with Lucid Imagination. In addition, Lucid Imagination do not recommend using the parser as a stand alone due to the strong integration points with LucidWorks Enterprise schema.

Q: What are open ended range queries?

A: To put the question in context, it came up during the presentation of range queries. Range queries can be enclosed with curly brackets, or square brackets. Curly bracket will tell the parser to use exclusive range (or as I referred to it, open ended range), while square brackets will instruct the parser to use inclusive range (or close ended range). Example:
[cat TO dog] All terms lexically between “cat” and “dog”, including “cat” and “dog”
{cat TO dog} All terms lexically between “cat” and “dog”, excluding “cat” and “dog”
{cat TO dog] All terms lexically between “cat” and “dog”, excluding “cat”, but including “dog”
[cat TO dog} All terms lexically between “cat” and “dog”, including “cat”, but excluding “dog”

Q: What is the use of adding question marks to the syntax? This part of the webcast was not clear to me.
A: Sometimes, we see queries that are formatted as natural language queries, such as “What is a CPA?”. This query, submitted with a common query parser, will result very poorly, as the question mark at the end will be mistakenly treated as wildcard, missing the term CPA, and trying to match CPAN. (There goes my taxes… ) Lucid Query Parser will detect the natural language question form, and will ignore the misleading question mark at the end. Additional natural language features are documented here

Q: Can you ask “sony~0.8 and not sony”?
A: Yes. The query, as typed is correct. A shorter format for this query will be “sony~0.8 -sony”, beware, however, that the shorter query has implicit OR, and not AND between the positive terms. In this case, the positive term count is one, making the two queries identical.

Q: Is there an “explanation of understanding the query”?
A: Yes. In addition to Solr’s built in debug query, which will expose you to what query was actually submitted to Lucene, you can prefix your query with debugLog: this will output debug data from Lucid Query Parser to the console, for further examination.

Q: how would you explain this to users? What’s the reception rate of such “complex” queries?
A: Great question. Few approaches may be valid here, and all involve UI that is smarter than “a text box”. One approach is to have an advanced mode, which allows the users to specify their queries to a series of widgets, i.e. date range queries can be submitted to UI date widgets, fuzzy queries can be automated by a checkbox saying “find misspelled content as well”, etc.
Another approach is to provide a UI widget showing popular examples to your users, very much like I did with this demo. Users will be able then to create queries to their liking, using your samples as a reference.
Another very effective approach is to empower your users with the field structure. Queries can be very effective when are targeted for specific fields, and field types.

Q: Do you support phrases as synonyms, anti-stem, or stop words?
A: LucidWorks Enterprise supports phrases as synonyms. It does not support stop word phrases (i.e. ignore the phrase “dead end”, but not “dead” and not “end”).
Regarding anti-stem, the out of the box LucidWorks Enterprise does not support anti-stem, but can easily be configured to do so, or to index both stemming and non stemming, using copy fields.

There were some questions that were not clear to me. If you still have more questions, email them to me, and I’ll post them here.

Wishing you all happy holiday season!

During the demo, you presented me with questions about LucidWorks Enterprise. I was able to answer some of them immediately following the demo; however, our timeslot for the demo caught up with us, so some were left unanswered. The team here at Lucid Imagination helped me capture your questions, so let’s dive in:

Q: I have multiple domains, each domain should have its own search content and result presentation layer. For example: car search and job search, how could LucidWorks Enterprise handle multiple domains?

A: For complete separation of schema, relevancy and client UIs, use multiple collections. LucidWorks Enterprise is able to handle unlimited numbers of collections. (Available Dec 2010)

Q: Will LucidWorks Enterprise search content within SWF files?

A: In order to search content of SWF, you’ll have to provide the SWF metadata and text to LucidWorks Enterprise. This can be done with SWF file filter, available via commercial packages. LucidWorks Enterprise does not contain one out of the box.

Q: What about language sensitive search?

A: Even though the LucidWorks Enterprise admin UI is currently English only, LucidWorks Enterprise can support multiple languages both for content and for Search UI. There are multiple ways to do so; my favorite one is to organize different language content under different fields, each field can be associated with its own field type and its own analysis chain.

Q: Is Lucid Works Solr (not the enterprise version) still under the Apache License?

A: Yes. The LucidWorks for Solr Certified Version is still licensed under ASF license v2.

Q: When LucidWorks Enterprise connects to DB how does it update the indexes if user has crud multiple records?

A: When indexing content from DB, you can consider the following options: 1. Use SQL logic for delta updates (manage dirty flag, last update field or other DB logic) 2. Use the delta SQL statement when running updates 3. In some cases, use unique IDs from your database to manage duplicates in LucidWorks Enterprise indexes.

Q: Does Solr support personalized search?

A: No. Solr does not support personalized search. Personalized search is implemented in LucidWorks Enterprise only.

Q: Is there an online demo site that allows us to try out?

A: Search.lucidimagination.com, which indexes the mailing lists and other relevant community content of Lucene, Solr, Tika and many other projects is based on LucidWorks Enterprise. You can apply for the developer access program of LucidWorks Enterprise here.

Q: Does this software support spatial searches?  e.g. ‘Show me records with 100 miles of zip code 90210′

A: The out of the box distribution of LucidWorks Enterprise does not include spatial search, this can be added on top of the Solr included with LucidWorks Enterprise.

Q: What kind of document type do you support?

A: Out of the box, we support HTML, XML, PPT, PPTX, XLS, XSLX, DOC, DOCX, ODF, PDF, Quattro, RTF, TXT, ZIP, TAR, MP3 and JPEG

Q: Can you show how to add users to roles?

A: Users can be added either via the ReST API (See: http://lucidworks.lucidimagination.com/display/LWEUG/Users) or via binding to an LDAP server (see: http://lucidworks.lucidimagination.com/display/LWEUG/LDAP+Integration)

Groups are only supported via LDAP integration.

Configuring users and groups to use the search filters (filtering documents for specific users and groups) can be configured via LWE UI, at admin->roles

Q: Has LucidWorks Enterprise been used on any Drupal CMS applications?

A: To the best of my knowledge, not in production yet.

Q: In a database import, how will the index be updated and in sync with database changes?

A: LucidWorks Enterprise supports 2 SQL queries for database importing – one for a full import, and one to be used for consecutive delta imports. The 2 SQL statements configuration is available only when accessing  the ReST API.

LucidWorks for Solr v1.4.1 is a bugfix release for LucidWorks for Solr, incorporating Apache Solr release 1.4.1, with few additional bugfixes that were introduced post Solr 1.4.1

LucidWorks for Solr v1.4.1 is tested to work with Windows, Linux and Mac.

Grab your copy from our downloads section

Changes and additions from Apache Solr 1.4

* SOLR-1902: Upgraded Tika to 0.8-SNAPSHOT (Tommaso Teofili, gsingers)

* SOLR-2036: Avoid expensive fieldCache ram estimation for the admin stats page. (yonik)

* SOLR-2100: The replication handler backup command didn’t save the commit point and hence could fail when a newer commit caused the older commit point to be removed before it was finished being copied.  This did not affect normal master/slave replication.  (Peter Sturge via yonik)

* SOLR-2180: It was possible for EmbeddedSolrServer to leave searchers open if a request threw an exception. (yonik)

* SOLR-2192: StreamingUpdateSolrServer.blockUntilFinished was not thread safe and could throw an exception. (yonik)

This time around, the data set I chose was Data.gov’s catalog of datasets, which fit with the ApacheCon open source aura, and Lucid Imagination’s support of Open Source for America, which helps to advocate for open source in the US Federal Government.  The prototype built includes faceting browsing, query term suggest, hit highlighting, result clustering, spell checking, document detail, and a bonus Venn diagram visualization.

The prototype was built with these steps:

1. Install LucidWorks for Solr
2. Grab the Data.gov catalog CSV file
3. Iterate a bit with Solr’s CSV update handler (the funnest way to get data into Solr) and a little Solr schema tinkering
4. Adjusted the Solr configuration and UI templates to get a nice look and feel, adding in a document detail page and a Venn diagram visualization comparing query overlaps

Voilà (click the images for large view):

This isn’t the first time we’ve toyed with Data.gov data… earlier this year, Hoss demonstrated Solr’s stats component on another of Data.gov’s data sets.

My ApacheCon slides are published at Slideshare and embedded here:

All the code and instructions for running the entire prototype yourself can be found here: https://github.com/erikhatcher/solr-rapid-prototyping/tree/master/ApacheCon2010

Gee.  That almost sounds like a marketing guy wrote it.  Let me try again: LucidWorks Enterprise is software that let’s you easily build great search applications.  You can install it, index some content, and search that content with just a few keystrokes and clicks of the mouse.  Or, you can build a full-blown, integrated search application with custom plugins and your own user interface, all running on a fifty-node cluster.  And if you really need it, the full flexibility of Solr is right there, always accessible.

LucidWorks Enterprise extends Solr to include some enterprise-grade features, just as Solr extends Lucene to provide server functionality.  By extending and embedding Solr, we get to use all of its powerful features without having to do what many folks do: fork the code base.  You see, we are really embedding Solr without changing it appreciably.  Where we do change it, we contribute our code back to the open source project.  A great example of that is SolrCloud, a feature we developed specifically for LucidWorks Enterprise.  The problem was that it was such a fundamental change to Solr that we would have to maintain a forked code-base forever, or simply contribute the code to the open source project.  We chose the latter, and that’s the choice we will continue to prefer for such fundamental and powerful features.

So, what’s in LucidWorks Enterprise?  Here’s a quick summary of the main features:

SolrCloud:

provides simplified configuration management, load balancing and failover in a distributed environment.  It’s the key feature if you have a cluster of search nodes.

User Interface:

we added a web-driven user interface to simplify the process of configuring the system, indexing content, and trying out searches.

CLICK Scoring:

ever wonder how to get better results on searches over your own content?  The CLICK scoring framework provides feedback from searchers back into the relevance ranking of documents, improving relevance for all.

ReST API:

all of the power of LucidWorks Enterprise is available programmatically, so you can build more thorough applications and automated administration.

Smarter Defaults:

LucidWorks Enterprise is configured out-of-the-box to be a great search application, with good relevance and a full list of features like spell checking, autocomplete, and unsupervised feedback.

Data Acquisition:

we have added a crawler and scheduler that simplifies the process of getting data into LucidWorks Enterprise.  Just point it at a Web site or set of files, or import data from a database.

So that’s LucidWorks Enterprise.  We’ll be talking more about it over the coming days.  This is the beginning of an exciting time for Lucid Imagination: we’re really excited to raise the bar with search and get our customers using the latest and greatest in search technology: LucidWorks Enterprise. It’s free for developers, so download LucidWorks Enterprise now! ]]> http://www.lucidimagination.com/blog/2010/10/14/introducing-lucidworks-enterprise/feed/ 1