Lucid Imagination has now packaged the latest version of our LucidWorks Certified Distribution for Solr 1.4 as an Amazon Machine Instance (AMI). It’s available for free to anyone with an Amazon Web Services account and Amazon EC2 access. In this post, I’ll outline what this is about, what it can do for you and instructions on how to get it started.

Our free AMI running LucidWorks for Solr offers the following benefits:

1. A very straightforward way to get a Solr instance up and running for those who are new to Solr and want to try it for the first time.
2. An easy way to test out the LucidWorks certified distribution of Solr.
3. A convenient starting point with a machine instance that can be used by anyone interested in running Solr in the cloud.

The AMI we’ve provided is a very basic Solr instance using the default example configuration files that Solr includes as a starting place for new users, along with a set of nineteen documents indexed and ready for searching. If you are interested in building Solr instances in the cloud this AMI is a good place to begin: simply edit the configuration files for your needs, set up persistent storage with Amazon EBS or S3 for your indexes, and you have a machine instance that can be bundled and registered for your own use.

Amazon EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. Cloud computing is a way to offer internet-based services, sometimes categorized into the following descriptions: Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS). Services are sold on demand and a consumer is charged by the hour for the services that they use. EC2 is one of the most popular cloud computing providers.

Running LucidWorks for Solr on Amazon EC2 requires that you have an account on Amazon Web Services, and that you are somewhat familiar with how to deploy an Amazon Machine Instance, but we’ll describe the process in detail, step-by-step to get you up and running easily.

These instructions assume you are already familiar with running machine instances in EC2. We provide detailed instructions on how to get the LucidWorks instance up and running, but this is not a tutorial on how to use EC2 if you are not already familiar with it. If you do not yet have an Amazon Web Services account, and are unfamiliar with EC2, here is a link to an introduction on how to get started. Once you have an account and are comfortable working with the EC2 tools, you’ll be ready to proceed to the next steps in this guide.

LucidWorks for Solr version 1.4, as you may already know, is a comprehensive, tested and release-stable certified distribution of Apache Solr, available for free. LucidWorks for Solr offers:

• A solid foundation of reliability and consistency for production-grade use of open source
• Fast, convenient access to documentation needed to build better search applications faster
• Quick, simplified setup and maintenance of Solr and its constituent components

Running LucidWorks for Solr on EC2 is possibly the easiest and fastest way to get a Solr instance up and running.

Steps to Run LucidWorks for Solr on Amazon EC2

OK, let’s create an instance of the LucidWorks for Solr AMI. We’ll show how to do this using the AWS Management Console. You could also use Elasticfox, a Mozilla Firefox extension for managing your Amazon EC2 account. If you are experienced with Elasticfox the steps will be similar to these and should be familiar to you.

AWS Management Console Instructions

Log into your AWS account. You should be at the start location for the Management Console:

Before we launch the LucidWorks AMI we’ll need to first set up some security settings to allow access to two ports on the instance: 8983 for Solr, and 22 for ssh.

From the AWS Management Console under the Network & Security Settings section click on the “Security Groups” link, and from the Security page click on the “Create Security Group” button, and then enter the following values:

At the bottom of the security page you will need to set up and save two connection methods, one for SSH, the other a “Custom” connection for the Solr port:

We’ll use that new security group when we launch an instance of the LucidWorks AMI.

Now click on the “AMIs” link under the Images section. In the “Viewing” drop down box select “Public Images”, and in the input box enter “lucidworks”. You should see one machine image avaiable:

Check the checkbox to the left and click on the “Launch” button above it.

Select the options for number of instances, machine size, and availability zone. The defaults are fine to get started.

On the second page just take the default values presented:

On the third page select your key pairs, which you should already have set up:

On the next page you’re asked to select the security groups to enable. Select the solr8983 group that we configured earlier. This will allow connections to port 8983 for Solr and port 22 for ssh:

Finally you’re asked to review your settings. If everything looks good go ahead and click the “Launch” button:

You should see a confirmation that the instance is launching:

Now you’re instance should be starting up. Back at the main AWS Management Console page click on the “Instances” link in the Instances section:

It will take a few minutes for the instance to launch. Reload the page until the Status changes from “pending” to “running”. Once the Status shows the instance is “running” you should be able to connect to the Solr Admin Console. You’ll need to know the public DNS of your instance. With the checkbox checked on this instance there will be a few lines of status output at the bottom of the AWS Management Console that look this:

Use the public DNS to build the URL to access the Solr Admin Console: http://ec2-174-129-150-181.compute-1.amazonaws.com:8983/solr/admin/

Now you’ve got a Solr instance up and running with nineteen sample documents in the index. If you’re new to Solr make sure to download Lucid Imagination’s free reference guide as a good start to learning about Solr.

Final Step: Logging into the Instance to Stop and Start LucidWorks for Solr

With the instance now up and running you may want to make changes, index more content, etc. You’ll need to ssh into the machine instance to do these tasks. From the AWS Management Console select the machine instance and find the “Instance Actions” list. From that list select the “Connect” option:

You should see instructions for connecting to your instance via ssh:

Once you have successfully logged into the instance you will find the LucidWorks for Solr home directory here: /opt/LucidWorks. Within that directory there is a “start” and a “stop” script for stopping and starting Solr if you make changes to the schema.xml or solrconfig.xml files. The Solr home directory is /opt/LucidWorks/lucidworks/solr and the index directory is /opt/LucidWorks/lucidworks/solr/data/index.

The instance is configured to start up with the following heap size settings: -Xms1g -Xmx3g. These are set in the Tomcat start script that manages LucidWorks: /opt/LucidWorks/lucidworks/tomcat/bin/catalina.sh

Note that the machine instance does not have any persistent storage, so if you want to start working with your own data and creating indexes that persist, you will need to use the LucidWorks AMI as a starting point, and you will have to add permanent storage, either Amazon EBS or Amazon S3.

Hopefully this has given you a chance to experience Solr running in the cloud, and shown some of the basics you’ll need to build on to implement cloud-hosted distributed search with LucidWorks for Solr.

• Analyze This! Tips and tricks on getting the Lucene/Solr Analyzer to index and search your content right – On-demand Webinar
• Apache Solr 1.4: Faster, Easier and More Versatile than Ever – On-demand Webinar
• Solr 1.4 Download
• Solr 1.4 Reference Guide

Sin number 1: Sloth



Let’s define sloth as laziness or indifference. This one bites most of us at some time or another. We just can’t resist the impulse to take a shortcut, or we simply refuse to acknowledge the amount of effort required to do a task properly. Ultimately we wind up paying the price, usually with interest. Here are some common examples of how laziness or indifference lead to Solr problems.

• A general lack of commitment either to Solr or to the search application project itself. Sometimes you see this when a company has decided to switch from a commercial search application to open-source alternatives like Lucene and Solr. The engineers involved in the project are used to the “old ways” and really don’t feel like mastering another search technology. So without making even the slightest effort they will claim that Solr is inefficient, difficult to learn, not worth the effort, etc. If you’re hungry it’s usually not productive to stand around waiting for a fried chicken to fly into your mouth – your time might be better spent being a little more active in your efforts to acquire some food. Open-source software is flexible, adaptable, and powerful, but the developers that become the experts at open-source solutions are those that are not afraid to roll up their sleeves and dive in to learn what they need. Participate in the mailing lists. Open the source code. Read the javadocs and the wiki. I’ve worked with customers that have embraced Solr and become experts in a fairly short amount of time, even contributing patches within a few weeks of starting their project. On the other hand I’ve seen problems fester and grow because a team just won’t put any effort into their Solr implementation. “There are none so blind as those who will not see.”
• Not reviewing, editing, or changing the default schema.xml and/or solrconfig.xml files. If I had a dollar for every production Solr instance that was statically warming the query “solr rocks” I could afford a years worth of support from a commercial search vendor. The default example configuration files are there to be used as, yes, examples, and as starting points. Take the time to learn about the configuration settings and field types, and make the best use of them. Remove anything that is not being used (how many times have you really queried that “partitioned” request handler…) Keep your configuration files lean and mean and maintainable and it will pay off in the long run.
• Ignoring the dismax query parser. I’ve seen cases where someone has written a custom query parser on their own when the work they needed to do could have easily been done with the dismax query parser. There are two different extremes to why folks sometimes avoid dismax. On one side there is the feeling that it is a “dumbed down” parser. I think part of the problem here is caused from the first line on the DismaxRequestHandler wiki page (and by the way, we still suffer from this unfortunate legacy nomenclature – it is a query parser, not a request handler) which says that dismax is “designed to process simple user entered phrases”. There is sometimes a feeling that it is merely an entry-level tool for those who don’t want to do any work crafting their queries. Au contraire! Dismax has an enormous amount of power and flexibility. Which leads to the second side of “dismax avoidance”, namely that it’s “too complicated”. Indeed, it is somewhat complicated. But the rewards of spending some time to get familiar with it are substantial.
• Not enough attention on JVM settings and garbage collection. One needn’t become a JVM Jedi to run a well-tuned Solr instance, but some time spent on learning the basics about different garbage collector types and monitoring the JVM with tools such as JConsole, will pay dividends. A good starting place is a blog by my colleague Mark Miller of Lucid Imagination. Another good resource is this document put out by Sun.

Sin number 2: Greed

Penny-wise and pound-foolish. This is a surprisingly all-too-common trap that some fall into. Obviously not everyone has an unbounded budget, but sometimes terrible decisions are made to constrain resources, decisions that will prove to be more costly over time. For example:

• Refusal to add the proper amount of RAM to a server. There have been occasions when I’ve had more RAM on my Mac laptop (4GB) than some of the production Solr implementations I’ve seen. Sometimes even Solr projects at large companies have been under-funded. There will be business requirements that make high memory demands (sorting on large String fields, lots of faceting on fields with huge numbers of distinct terms, etc.) but the expectation will be that this can somehow be “made to work” with an insufficient amount of RAM and some kind of wizardry. A friend of mine has a saying, “You can’t fit 15 pounds of rice into a 10 pound bag.” By all means commit to at least acquiring the minimum adequate amount of resources.
• Insisting on running indexing and searching on the same host. One of the first recommendations we at Lucid Imagination often make to customers is to separate the indexing and searching process to (at least) two separate nodes. There are several benefits to be gained by doing this. First, the indexing and searching processes are not competing for resources (cpu, memory, etc.). Second, the master and slave(s) can be configured slightly differently for optimum performance. Be sure to budget for adequate hardware based on your document count, index size, and expected query volume.

Sin number 3: Pride

Pride (for our purposes): failing to acknowledge the good work of others. An excessive love of self.

Engineers love to code. Sometimes to the point of wanting to create custom work that may have a solution in place already, just because: a) They believe they can do it better. b) They believe they can learn by going through the process. c) It “would be fun”. This is not meant to discourage new work to help out with an open-source project, to contribute bug fixes, or certainly to improve existing functionality. But be careful not to rush off and start coding before you know what options already exist. Measure twice, cut once.

• Don’t re-invent the wheel. I’ve seen developers almost look for excuses to write their own query parser or other custom component. Sometimes such effort is necessary, and luckily open-source software makes this doable in ways that would never be possible with commercial search software. But make sure you have a real need before writing custom code – at least while on the company’s dime. There is extra effort in maintaining a custom codebase and keeping it in sync with Solr, so make sure it really is the only option to solve a particular use case.
• Make use of the mailing lists and the list archives. This should be obvious, but there are still many who think that this is beneath them in some way, as if asking for help was somehow a flaw. On the other hand, when posting to the mailing lists, make efficient use of everyone’s time. Be sure to thoroughly search the list archives before posting. (LucidFind makes it a snap to search relevant mailing lists, wikis, blogs, javadoc, and other sources in one place.) If and when you do post a question provide a succinct description of the problem and make it clear to others what you need. Stay on topic throughout the thread. Lucene and Solr committers and Lucid Imagination staff are regular participants on the mailing lists, so take advantage of this resource when you have a real need.

Sin number 4: Lust

You’ll have to grant me artistic license on this one, or else we won’t be able to keep this blog G-rated. So let’s define lust as “an unnatural craving for something to the point of self-indulgence or lunacy”. OK.

• Setting the JVM Heap size too high, not leaving enough RAM for the OS. So we finally get the RAM allocation we’ve been pining for (see: Greed) and now what do we do? We’ve got 16GB of RAM on our machine now so we allocate 15GB to the heap where Solr is running. Whoa! Time for a cold shower! Solr may be the only object of your desire, but don’t neglect the operating system. Patience and attention come into play here. Monitor the JVM under load and determine what the real memory footprint of Solr is. You’ll want the operating system to be able to cache file system data (especially the Lucene indexes) so be sure to leave enough RAM for the OS.
• Too much attention on the JVM and garbage collection. On the other hand (and in direct contrast to our first bullet-point under Greed), don’t overdo it on the JVM. There are seemingly unending ways to tweak and tune a JVM. Don’t fall into the trap of trying every arcane JVM or GC setting unless you are a JVM expert. Once you have mastered the basics of the JVM and understand the differences between the different types of garbage collectors, for the most part you shouldn’t have to get too creative. Don’t just toss “-XX:CMSIncrementalDutyCycleMin=10″ into the mix out of curiosity.
• Trying to “push the envelope” on auto-warm counts. How warm is too warm? We all want the fastest search response times possible, and auto-warming Solr’s queryResultCache and filterCache are important tools to help keep responses for the most popular queries as fast as possible. But let’s not get carried away. Excessive auto-warm counts can cause excessive warm-up times for the caches, which in turn affect the warm-up time of new IndexSearchers after every commit. Ask yourself if you really need to auto-warm the top 5,000 queries every time a commit occurs. It’s very easy to get obsessed with this and find that the time to warm-up a new IndexSearcher is extending beyond your commit cycles, which can lead to all kinds of odd behavior, including OutOfMemory Exceptions. Make sure you know what your average warm-up times are for all of your caches and your new IndexSearchers. It’s actually best to start out with more modest auto-warm counts and work up if necessary rather than start out too high. Create reports or database records of user queries by parsing your production log files. Use that data to get a feel for what the most popular queries are. Sometimes just setting an auto-warm count to 100 is plenty. But it takes time and effort to find the sweet spot between caches that are too cool and caches that are “en fuego”.

Sin number 5: Envy

• Wanting features that other sites have, that you really don’t need. Stay focused on your business needs. Make sure you know what you really need from your search application. A common scenario on the mailing lists is one that Lucene/Solr committer Chris Hostetter calls the “XY” problem. From the Solr user mailing list: “You are dealing with ‘X’, you are assuming ‘Y’ will help you, and you are asking about ‘Y’ without giving more details about the ‘X’ so that we can understand the full issue.  Perhaps the best solution doesn’t involve ‘Y’ at all”. Know what you need and keep focused on the requirements.
• Wanting to have a bigger index than the other guy. The antithesis of the “greed” issue of not allocating enough resources. “Shooting for the moon” and trying to allow for possible growth over the next 20 years. A trap for those who believe their status is determined by the size of their server farm. By all means plan ahead, but don’t expect that you can see into the future to foresee every possible scenario. Plan smartly, but don’t overdo it.

Sin number 6: Gluttony

“Staying fit and trim” is usually good practice when designing and running Solr applications. A lot of these issues cross over into the “Sloth” category, and are generally cases where the extra effort to keep your configuration and data efficiently managed is not considered important.

• Lack of attention to field configuration in the schema. Storing fields that will never be retrieved. Indexing fields that will never be searched. Storing term vectors, positions and offsets when they will never be used. Unnecessary bloat. Understand your data and your users and design your schema and fields accordingly.
• Unexamined queries that are redundant or inefficient. I’ve seen cases where queries have been generated programatically with a lot of redundancy and nonsensical logic. Take advantage of filter queries whenever possible. For example, if you have a query like this – &q=content:solr AND datasource:wiki AND category:search AND language:en – use filter queries on fields where it makes sense: &q=content:solr&fq=datasource:wiki&fq=category:search&fq=language:en.

Sin number 7: Wrath

While wrath is usually considered to be synonymous with anger, let’s use an older definition here: “a vehement denial of the truth, both to others and in the form of self-denial, impatience.”

• Assuming you will never need to re-index your data. It’s easy to focus on schema design, configuration, deployment, scaling issues, performance, and relevance tuning, while neglecting to consider how to re-create your index in the event of a disaster, either major or minor. One step that should never be omitted from your planning is a step to consider how to re-create your index in the case of hardware failures. If you are replicating from a master to a slave or slaves, consider having an extra slave that might not be used for searching but can receive replications of the index to serve as a backup to the master. If feasible back up your index data to other storage media. At the very least, if you don’t have a large index and can re-create it without too much effort if it is deleted or lost, make sure you have a plan and procedures in place in preparation for quickly re-indexing.
• Rushing to production. Of course we all have deadlines, but you only get one chance to make a first impression. Years ago I was part of a project where we released our search application prematurely (ahead of schedule) because the business decided it was better to have something in place rather than not have a search option. We developers felt that, with another four weeks of work we could deliver a fully-ready system that would be an excellent search application. But we rushed to production with some major flaws. Customers of ours were furious when they searched for their products and couldn’t find them. We developed a bad reputation, angered some business partners, and lost money just because it was deemed necessary to have a search application up and running four weeks early.

So keep it simple, stay smart, stay up to date, and keep your search application on the straight-and-narrow. Seek (intelligently) and ye shall find.

• Analyze This! Tips and tricks on getting the Lucene/Solr Analyzer to index and search your content right – On-demand Webinar
• Apache Solr 1.4: Faster, Easier and More Versatile than Ever – On-demand Webinar
• Solr 1.4 Download
• Solr 1.4 Reference Guide

N-grams and Edge N-grams

An N-gram is an n-character substring of a longer sequence of characters. For example, the term “cash” is composed of the following n-grams:

• unigrams: “c”, “a”, “s”, “h”
• bigrams: “ca”, “as”, “sh”
• trigrams: “cas”, “ash”
• 4-grams: “cash”

N-grams can be useful when substrings of terms need to be searched. An Edge n-gram is an n-gram built from one side or edge of a term. Edge n-grams for the term “cash” would be:

• “c”, “ca”, “cas”, “cash”

It’s easy to see how edge n-grams could be used to suggest queries as a user types in a search query character by character.

An Overview of the Process

In order to provide query suggestions we will need to have typical queries entered by users available in a Solr index. It is a good practice to capture and analyze the queries that are being entered by the users of a search application. Ideally you might have a scheduled process to parse your Solr output logs to capture queries entered by users of the application. The queries might be stored in a relational database where they could be analyzed independently of your running production Solr instances. Another benefit of storing queries (and the number of times they have been entered) is that it is then possible to use Solr’s DataImportHandler to build an index of query information that can be used to power an auto-suggest feature. You might design an auto-auggest index as a separate core hosted in a single Solr instance along with a core for your main index. Note that it is probably not worth indexing queries that return zero results since we won’t want to suggest those to a user, so we’ll include a boolean field to let us know which queries contain results. A minimal table design for our needs might look like this:

create table autosuggest (
 query varchar(250),
 hasResults boolean,
 count int);

More metadata could certainly be added as needed for reporting and analysis – these are just the columns we will need to demonstrate how to build the auto-suggest feature. As queries are parsed from the log files the process will need to query the main Solr index to deterine whether the query has one or more results in order to populate the hasResults column.

Typically an AJAX front-end would be used to query the “auto-suggest” index. For the purposes of this article we won’t deal with how to build the AJAX component, nor will we go into details about the pre-processing involved in building the database table. Instead we will focus on how to configure Solr to index the queries and then search the index, with responses written in JSON format using Solr’s JSON response writer.

In general the steps we will take are:

1. Parse log files to get a list of queries entered by users and load those queries (and any other useful metadata) into a database table. This should be an ongoing scheduled process.
2. Configure schema.xml.
3. Configure a dih-config.xml file.
4. Do a full-import with the DataImportHandler.
5. Build queries that can be used by an AJAX client.

Configuring schema.xml

We will need to define a fieldType that doesn’t tokenize the query and does very minimal analysis. We can use the KeywordTokenizerFactory which will leave our queries intact as a single term. We’ll include the LowerCaseFilterFactory to simplify input, and then, most importantly, we will include the EdgeNgramFilterFactory. The EdgeNgramFilterFactory will break up our query sources into a series of EdgeNgrams. We need to specify a minGramSize and a maxGramSize. For this example we’ll set the minimum to “1″ and the maximum to “25″. Note that storing n-grams in an index will require some extra storage, but since our “auto-suggest” index will only contain two fields (and only one of them stored), and since we can probably expect a document count that is not too high, this should not be a problem.

Here is our definition for a fieldType to accomplish what we need:

<fieldType name="edgytext" class="solr.TextField" positionIncrementGap="100">
 <analyzer type="index">
   <tokenizer class="solr.KeywordTokenizerFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
   <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
 </analyzer>
 <analyzer type="query">
   <tokenizer class="solr.KeywordTokenizerFactory"/>
   <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
</fieldType>

We should only need two fields defined, “user_query” to hold the query and it’s n-grams, and “count” which is the number of times a query has been found in the logs, and is what we can sort on to present more popular queries higher in the suggested list:

<field name="user_query" type="edgytext"
  indexed="true" stored="true" omitNorms="true"
  omitTermFreqAndPositions="true" />
<field name="count" type="int" indexed="true"
  stored="false" omitNorms="true"
  omitTermFreqAndPositions="true" />

Configuring dih-config.xml

Solr’s DataImportHandler (DIH) is an extremely fast and efficient tool to use for indexing data in a relational database. There are two places where configuration needs to be done to enable the DIH. First, we need to set up a request handler in solrconfig.xml:

<requestHandler name="/indexer/autosuggest"
    class="org.apache.solr.handler.dataimport.DataImportHandler">
 <lst name="defaults">
 <str name="config">dih-config.xml</str>
 </lst>
 <lst name="invariants">
 <str name="optimize">false</str>
 </lst>
</requestHandler>

Then we need to create the dih-config.xml indicated in the request handler configuration. Create a file named “dih-config.xml” in the same directory as solrconfig.xml and schema.xml with thw following contents:

<?xml version="1.0"?>
<dataConfig>
 <dataSource
   type="JdbcDataSource"
   readOnly="true"
   driver="com.mysql.jdbc.Driver"
   url="jdbc:mysql://localhost:3306/myDatabase"
   user="user"
   password="password"/>

  <document name="autoSuggester">
    <entity name="main"
       query="select query, count from autosuggest where hasResults = true">
     <field column="query" name="user_query"/>
     <field column="count" name="count"/>
   </entity>
 </document>
</dataConfig>

The dataConfig is pretty straightforward. We set up a connection to our datasource and then run a simple select query to get all records that have results. It would also be possible to set up a delta-import query but depending on how many user-entered queries you are indexing it may not be necessary. The DIH is very fast, and my tests were able to index about 75,000 documents a minute (on a MacBook Pro with a 2GB heap size), so running a full index once or several times a day may be easier. If you want to set up delta queries you would have to add a date field to the table that could be included in the delta queries. (For more details on DIH configuration see the wiki: http://wiki.apache.org/solr/DataImportHandler)

Indexing The Data

Now we’re ready to index the data in the autosuggest table. This can be done in two different ways:

1. Using a utility like “curl” or “wget”: curl ‘http://localhost:8983/solr/indexer/autosuggest?command=full-import (note: when developing and debugging you can add the parameter “&rows=10″ or whatever number you would like to limit the import.)
2. From the dataimport.jsp page: http://localhost:8983/solr/admin/dataimport.jsp. You should see a link to the handler defined in solrconfig.xml (which we named /indexer/autosuggest). Click on that link. The dih-config.xml file is displayed along with various buttons for different operations. Clicking on “Full-import” will fire off the full import process.

Now we should have documents in our auto-suggest index. We can use the analysis page from the admin console (http://localhost:8983/solr/admin/analysis.jsp) to get a look at how the n-grams are created for a query. From the analysis page select “type” from the “Field” drop-down box and enter a value of “edgytext”, the fieldType we defined in schema.xml” Check “verbose output” in the “Field value” section and enter “i’m not down” in the text area. Click “Analyze” and observe how the EdgeNGramFilterFactory breaks up our query into n-grams:

Running Some Queries

Now we can do some testing to see if we get the results we expect. My test index was for a music site where users are often entering song titles as keyword searches. Let’s say I’m searching for the song “I’m Not Down”. The first character typed should trigger the following query sent from the AJAX front-end: http://localhost:8983/solr/select/?q=user_query:”i”&wt=json&fl=user_query&indent=on&echoParams=none&rows=10&sort=count desc

Note that we are using the JSON response writer (&wt=json), asking only for the field we’re interested in (&fl=user_query), turning off echoParams (&echoParams=none) to keep the response as small as possible, and sorting on the count field descending (&sort=count desc). (&indent=on is set for clean display, but this should be omitted for a production site.) Since we entered the count values for each query into the database we can sort on that to get the most popular searches at the top of our results. The response for this query with only the first character entered looks like this:

{
 "responseHeader":{
 "status":0,
 "QTime":1},
 "response":{"numFound":12,"start":0,"docs":[
 {
 "user_query":"i'm only sleeping"},
 {
 "user_query":"i'm glad"},
 {
 "user_query":"i'm not down"},
 {
 "user_query":"i'm a believer"},
 {
 "user_query":"i'm not your stepping stone"},
 {
 "user_query":"i'm not in love"},
 {
 "user_query":"i'm shakin'"},
 {
 "user_query":"i've been everywhere"},
 {
 "user_query":"i'm hurtin'"},
 {
 "user_query":"i'm in the mood for love"}]
 }}

Let’s jump ahead to the fifth character entered: “i’m n” – now the results are more restricted:

{
 "responseHeader":{
 "status":0,
 "QTime":4},
 "response":{"numFound":3,"start":0,"docs":[
 {
 "user_query":"i'm not down"},
 {
 "user_query":"i'm not your stepping stone"},
 {
 "user_query":"i'm not in love"}]
 }}

Note that it’s necessary to wrap the query in double-quotes as a phrase. Otherwise unpredictable and unwanted matches can occur.

In Conclusion

We’ve shown one way to implement an auto-suggest feature. But no matter which approach you take it’s important to know what kinds of queries your users are entering, and how to index that data in such a way that it improves the end-user experience for your search application. Parsing log files to capture queries and storing them in a database not only makes it easy to index your query data for auto-suggestions, but the data can also be useful for reporting and analysis, as well as building query lists for load testing.