Clear as Mudd, right?

Regardless of whether you use the DisMaxRequestHandler via the qt=dismax parameter, or use the SearchHandler with the DisMaxQParser via defType=dismax the end result is that your q parameter gets parsed by the DisjunctionMaxQueryParser.

The original goals of dismax (whichever meaning you might infer) have never changed:

… supports a simplified version of the Lucene QueryParser syntax. Quotes can be used to group phrases, and +/- can be used to denote mandatory and optional clauses … but all other Lucene query parser special characters are escaped to simplify the user experience. The handler takes responsibility for building a good query from the user’s input using BooleanQueries containing DisjunctionMaxQueries across fields and boosts you specify It also allows you to provide additional boosting queries, boosting functions, and filtering queries to artificially affect the outcome of all searches. These options can all be specified as default parameters for the handler in your solrconfig.xml or overridden the Solr query URL.

In short: You worry about what fields and boosts you want to use when you configure it, your users just give you words w/o worrying too much about syntax.

The magic of dismax (in my opinion) comes from the query structure it produces. What it essentially boils down to is matrix multiplication: a one column matrix of each “chunk” of your user’s input, multiplied by a one row matrix of the qf fields to produce a big matrix of every field:chunk permutation. The matrix is then turned into a BooleanQuery consisting of DisjunctionMaxQueries for each row in the matrix. DisjunctionMaxQuery is used because it’s score is determined by the maximum score of it’s subclauses — instead of the sum like a BooleanQuery — so no one word from the user input dominates the final score. The best way to explain this is with an example, so let’s consider the following input…

defType = dismax
     mm = 50%
     qf = features^2 name^3
      q = +"apache solr" search server

First off, we consider the “markup” characters of the parser that appear in this q string:

• white space – dividing input string into chunk
• quotes – makes a single phrase chunk
• + – makes a chunk mandatory

So we have 3 “chunks” of user input:

• “apache solr” (must match)
• “search” (should match)
• “server” (should match>

If we “multiply” that with our qf list (features, name) we get a matrix like this…

If we then factor in the mm param to determing the “minimum number of ‘ShouldMatch’ clauses that (ahem) must match” (50% of 2 == 1) we get the following query structure (in psuedo-code)…

q = BooleanQuery(
  minNumberShouldMatch => 1,
  booleanClauses => ClauseList(
    MustMatch(DisjunctionMaxQuery(
      PhraseQuery("features","apache solr")^2,
      PhraseQuery("name","apache solr")^3)
    ),
    ShouldMatch(DisjunctionMaxQuery(
      TermQuery("features","search")^2,
      TermQuery("name","search")^3)
    ),
    ShouldMatch(DisjunctionMaxQuery(
      TermQuery("features","server")^2,
      TermQuery("name","server")^3))
    ));

With me so far right?

Where people tend to get tripped up, is in thinking about how Solr’s per-field analysis configuration (in schema.xml) impacts all of this. Our example above was pretty straight forward, but lets consider for a moment what might happen if:

• The name field uses the WordDelimiterFilter at query time but features does not.
• The features field is configured so that “the” is a stopword, but name is not.

Now let’s look at what we get when our input parameters are structurally similar to what we had before, but just different enough to for WordDelimiterFilter and StopFilter to come into play…

defType = dismax
     mm = 50%
     qf = features^2 name^3
      q = +"apache solr" the search-server

Our resulting query is going to be something like…

q = BooleanQuery(
  minNumberShouldMatch => 1,
  booleanClauses => ClauseList(
    MustMatch(DisjunctionMaxQuery(
      PhraseQuery("features","apache solr")^2,
      PhraseQuery("name","apache solr")^3)
    ),
    ShouldMatch(DisjunctionMaxQuery(
      TermQuery("name","the")^3)
    ),
    ShouldMatch(DisjunctionMaxQuery(
      TermQuery("features","search-server")^2,
      PhraseQuery("name","search server")^3))
  ));

The use of WordDelimiterFilter hasn’t changed things very much: features is treating “search-server” as a single Term, while in the name field we are searching for the phrase “search server” — hopefully this shouldn’t surprise anyone given the use of WordDelimiterFilter for the name field (presumably that’s why it’s being used). This DisjunctionMaxQuery still “makes sense”, but other fields with odd analysis that produce less/more Tokens then a “typical” field for the same thunk might produce queries that aren’t as easily to understand. In particular consider what has happened in our example with the word “the”: Because “the” is a stop word in the features field, no Query object is produced for that field/chunk combination. But a Query is produced for the name field, which means the total number of “ShouldMatch” clauses in our top level query is still 2 so our minNumberShouldMatch is still 1 (50% of 2 == 1).

This type of situation tends to confuse a lot of people: since “the” is a stop word in one field, they don’t expect it to matter in the final query — but as long as at least one qf field produces a Token for it (name in our example) it will be included in the final query, and will contribute to the count of “ShouldMatch” clauses.

So, what’s the take away from all of this?

DisMax is a complicated creature. When using it, you need to consider all of it’s options carefully, and look at the debugQuery=true output while experimenting with different query strings and different analysis configurations to make really sure you understand how queries from your users will be parsed.

fq={!frange l=0 u=2.2}log(sum(user_ranking,editor_ranking))

The other interesting use for frange is to trade off memory for speed when doing range queries on any type of single-valued field. For example, one can use frange on a string field provided that there is only one value per field, and that numeric functions are avoided.

For example, here is a filter that only allows authors between martin and rowling, specified using a standard range query:
fq=author_last_name:[martin TO rowling]

And the same filter using a function range query (frange):
fq={!frange l=martin u=rowling}author_last_name

This can lead to significant performance improvements for range queries with many terms between the endpoints, at the cost of memory to hold the un-inverted form of the field in memory (i.e. a FieldCache entry – same as would be used for sorting). If the field in question is already being used for sorting or other function queries, there won’t be any additional memory overhead.

The following chart shows the results of a test of frange queries vs standard range queries on a string field with 200,000 unique values. For example, frange was 14 times faster when executing a range query / range filter that covered 20% of the terms in the field. For narrower ranges that matched less than 5% of the values, the traditional range query performed better.

Of course, Solr 1.4 also contains the new TrieRange functionality that will generally have the best time/space profile for range queries over numeric fields.