I really dislike the so called “Boolean Operators” (“AND”, “OR”, and “NOT”) and generally discourage people from using them. It’s understandable that novice users may tend to think about the queries they want to run in those terms, but as you become more familiar with IR concepts in general, and what Solr specifically is capable of, I think it’s a good idea to try to “set aside childish things” and start thinking (and encouraging your users to think) in terms of the superior “Prefix Operators” (“+”, “-”).

Background: Boolean Logic Makes For Terrible Scores

Boolean Algebra is (as my father would put it) “pretty neat stuff” and the world as we know it most certainly wouldn’t exist with out it. But when it comes to building a search engine, boolean logic tends to not be very helpful. Depending on how you look at it, boolean logic is all about truth values and/or set intersections. In either case, there is no concept of “relevancy” — either something is true or it’s false; either it is in a set, or it is not in the set.

When a user is looking for “all documents that contain the word ‘Alligator’” they aren’t going to very be happy if a search system applied simple boolean logic to just identify the unordered set of all matching documents. Instead algorithms like TF/IDF are used to try and identify the ordered list of matching documents, such that the “best” matches come first. Likewise, if a user is looking for “all documents that contain the words ‘Alligator’ or ‘Crocodile’”, a simple boolean logic union of the sets of documents from the individual queries would not generate results as good as a query that took into the TF/IDF scores of the documents for the individual queries, as well as considering which documents matches both queries. (The user is probably more interested in a document that discusses the similarities and differences between Alligators to Crocodiles then in documents that only mention one or the other a great many times).

This brings us to the crux of why I think it’s a bad idea to use the “Boolean Operators” in query strings: because it’s not how the underlying query structures actually work, and it’s not as expressive as the alternative for describing what you want.

BooleanQuery: Great Class, Bad Name

To really understand why the boolean operators are inferior to the prefix operators, you have to start by considering the underlying implementation. The BooleanQuery class is probably one of the most misleading class names in the entire Lucene code base because it doesn’t model simple boolean logic query operations at all. The basic function of a BooleanQuery is:

1. A BooleanQuery consists of one or more BooleanClauses, each of which contains two pieces of information:
   • A nested Query
   • An Occur flag, which has one of three values
   • MUST – indicating that documents must match this nested Query in order for the document to match the BooleanQuery, and the score from this subquery should contribute to the score for the BooleanQuery
   • MUST_NOT – indicating that documents which match this nested Query are prohibited from matching the BooleanQuery
   • SHOULD – indicating that documents which match this nested Query should have their score from the nested query contribute to the score from the BooleanQuery, but documents can be a match for the BooleanQuery even if they do not match the nested query
2. If a BooleanQuery contains no MUST BooleanClauses, then a document is only considered a match against the BooleanQuery if one or more of the SHOULD BooleanClauses is a match.
3. The final score of a document which matches a BooleanQuery is based on the sum of the scores from all the matching MUST and SHOULD BooleanClauses, multiplied by a “coord factor” based on the ratio of the number of matching BooleanClauses to the total number of BooleanClauses in the BooleanQuery.

These rules are not exactly simple to understand. They are certainly more complex then boolean logic truth tables, but that’s because they are more powerful. The examples below show how easy it is to implement “pure” boolean logic with BooleanQuery objects, but they only scratch the surface of what is possible with the BooleanQuery class:

• Conjunction: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• Disjunction: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• Negation: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

Query Parser: Prefix Operators

In the Lucene QueryParser (and all of the other parsers that are based on it, like DisMax and EDisMax) the “prefix” operators “+” and “-” map directly to the Occur.MUST and Occur.MUST_NOT flags, while the absence of a prefix maps to the Occur.SHOULD flag by default. (If you have any suggestions for a one character prefix syntax that could be used to explicitly indicate Occur.SHOULD, please comment with your suggestions, I’ve been trying to think of a good one for years.) So using the prefix syntax, you can express all of the permutations that the BooleanQuery class supports — not just simple boolean logic:

• (+X +Y) … Conjunction, ie: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• (X Y) … Disjunction, ie: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• (+X -Z) … Negation, ie: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Z, Occur.MUST_NOT);
  

• ((X Y) -Z) … ((X ∨ Y) ¬ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.SHOULD);
  inner.add(Y, Occur.SHOULD);
  q.add(inner, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

• (X Y -Z) … Not expressible in simple boolean logic

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

Note in particular the differences between the last two examples. (X Y -Z) is a single BooleanQuery object containing three clauses, while ((X Y) -Z) is a BooleanQuery containing two clauses, one of which is a nested BooleanQuery containing two clauses. In both cases a document must match either “X” or “Y” and it can not match against “Z” (so the set of documents matched by each query will be the same) and in both cases the score of a document will be higher if it matches both the “X” and “Y” clauses; but because of the difference in their structures, the scores for these queries will be different for every document. In particular, the coord factor will cause documents matching only one of “X” or “Y” (but not both) to have extremely different scores from each of these queries. (This assumes that the DefaultSimilarity is being used; it would be possible to write a custom Similarity to force the scores to be equivalent)

Query Parser: “Boolean Operators”

The query parser also supports the so called “boolean operators” which can also be used to express boolean logic, as demonstrated in these examples:

• (X AND Y) … Conjunction, ie: (X ∧ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.MUST);
  q.add(Y, Occur.MUST);
  

• (X OR Y) … Disjunction, ie: (X ∨ Y)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Y, Occur.SHOULD);
  

• (X NOT Z) … Negation, ie: (X ¬ Z)

  BooleanQuery q = new BooleanQuery();
  q.add(X, Occur.SHOULD);
  q.add(Z, Occur.MUST_NOT);
  

• ((X AND Y) OR Z) … ((X ∧ Y) ∨ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.MUST);
  inner.add(Y, Occur.MUST);
  q.add(inner, Occur.SHOULD);
  q.add(Z, Occur.SHOULD);
  

• ((X OR Y) AND Z) … ((X ∨ Y) ∧ Z)

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(X, Occur.SHOULD);
  inner.add(Y, Occur.SHOULD);
  q.add(inner, Occur.MUST);
  q.add(Z, Occur.MUST);
  

• (X AND (Y NOT Z)) … (X ∧ (Y ¬ Z))

  BooleanQuery q = new BooleanQuery();
  BooleanQuery inner = new BooleanQuery();
  inner.add(Y, Occur.MUST);
  inner.add(Z, Occur.MUST_NOT);
  q.add(X, Occur.MUST);
  q.add(inner, Occur.MUST);
  

Please note how import it is to use parentheses to combine multiple operators in order in order to generate queries that correctly model boolean logic. As mentioned before, the BooleanQuery class supports an arbitrary number of clauses, so (X OR Y OR Z) is a single BooleanQuery with three clauses — it is not equivalent to either ((X OR Y) OR Z) or (X OR (Y OR Z)) because those result in a BooleanQuery with two clauses, one of which is a nested BooleanQuery. As mentioned above when discussing the prefix operators, the scores from each of those queries will all be different depending on which clauses are matched by each document.

Things definitely get very confusing when these “boolean operators” are used in ways other then those described above. In some cases this is because the query parser is trying to be forgiving about “natural language” style usage of operators that many boolean logic systems would consider a parse error. In other cases, the behavior is bizarrely esoteric:

• Queries are parsed left to right
• NOT sets the Occurs flag of the clause to it’s right to MUST_NOT
• AND will change the Occurs flag of the clause to it’s left to MUST unless it has already been set to MUST_NOT
• AND sets the Occurs flag of the clause to it’s right to MUST
• If the default operator of the query parser has been set to “And”: OR will change the Occurs flag of the clause to it’s left to SHOULD unless it has already been set to MUST_NOT
• OR sets the Occurs flag of the clause to it’s right to SHOULD

Practically speaking this means that NOT takes precedence over AND which takes precedence over OR — but only if the default operator for the query parser has not been changed from the default (“Or”). If the default operator is set to “And” then the behavior is just plain weird.

In Conclusion

I won’t try to defend or justify the way the query parser behaves when it encounters these “boolean operators”, because in many cases I don’t understand or agree with the behavior myself — but that’s not really the point of this article. My goal isn’t to convince you that the behavior of these operators makes sense, quite the contrary my goal today is to point out that regardless of how these operators are parsed, they aren’t a good representation of the underlying functionality available in the BooleanQuery class.

Do yourself a favor and start thinking about BooleanQuery as a container of arbitrary nested queries annotated with MUST, MUST_NOT, or SHOULD and discover the power that is available to you beyond simple boolean logic.

Many thanks to Bill Dueber whose recent related blog post reminded me that I had some draft notes on this subject floating around my laptop waiting to finished up and posted online.

I’ll be speaking at the upcoming Rich Web Experience conference in Ft. Lauderdale, presenting an “Introduction to Solr”, “Solr Recipes”, and “Lucene for Solr Developers”.  I’ll be tying all of these presentations together into a cohesive search/Solr track going from the introduction, to recipes for common tasks, through advanced customization of Solr.

And while I wasn’t around in the beginning, I thought I would offer up some (little) known tidbits, links, etc. about Lucene as an ode to the search library that has significantly changed the search world, as well as my own career:

1. Lucene was Doug’s way of learning Java!  How’s that for a start?  It took him 3 months, working 2 days a week to crank out the first version.
2. At the time, some commercial search engines could not do incremental updates of the index, meaning you had to re-index all your documents anytime you had an update.  Lucene has always had an incremental model, all the way through to today’s Near Real Time features that power the likes of Twitter at 1 billion+ searches and 100M+ new documents per day.
3. Field myField = Field.Text(“foo”, “bar”); anyone?  Or how about Field myField = Field.UnIndexed(“foo”, “bar”);
4. Back then, Lucene had it’s own PorterStemmer, now we just use Snowball.
5. Only 1 of the original committers still remains somewhat active.
6. Read the old FAQ!  True as it ever was.  (Mostly)
7. Lucene 2.3 drastically improved indexing performance thanks to a thorough overhaul of the innards while barely affecting the API.  4.0 will blow the doors off of previous versions in terms of speed and efficiency.
8. Lucene is Doug’s wife’s middle name.
9. Lucene has evolved from offering a single vector space scoring model to one that now offers plug-n-play ranking (BM25 anyone?)
10. Lucene is ubiquitous.  It powers search on everything from mobile devices to web scale engines.  I’ve seen indexes as small as 15% of the original content.  I’ve also seen indexes grow to several billion documents in size.  Lucene has been used as a caching store, an ORM, a cross language search engine, the guts of the popular Solr search server, the retrieval engine for IBM’s Watson as well as several commercial search engines and pretty much everything in between.
11. Did you know Apache Hadoop started as a subproject of Lucene?  Doug Cutting and Mike Cafarella first built out Hadoop in order to scale out indexing for the Apache Nutch project.  From there it was spun out to be a top level ASF project and has gone on to be the de facto choice for large scale distributed processing, much like Lucene is the de facto choice for search!  Lucene has also spun out Mahout, Tika, Lucene.NET and Lucy!

As for how Lucene’s impacted me?  In 2004, I took a job at the Center for Natural Language Processing at Syracuse University working for Dr. Liz Liddy.  My job was to build an Arabic-English cross language search engine.  Within a day or two of starting, Ozgur Yilmazel (my boss at the time) said something to the effect of “we’ll be using Lucene for the implementation.  Go learn it.”  Digging in, I quickly needed a couple of features, the biggest one being Term Vectors, so I updated a patch from an earlier version of Lucene and managed to convince the committers at the time to commit it.  From there, I kept supplying patches.  Eventually, I was asked to be a committer.  Some time after that, Yonik Seeley and Marc Krellenstein approached a bunch of the committers about starting a company and here I am today at the company we (Erik, Yonik, Marc and I) founded back in 2007, Lucid Imagination.  I feel fortunate to have the opportunity to work on hard problems in an interesting field and for that, Lucene, in no small part, I thank  you.

But enough of my self-indulgence, how has Lucene impacted you?  When did you first start using it?  What’s your biggest index or fastest QPS?   What ways have you used Lucene beyond that of a search engine?  Leave a comment and let us know.

Happy 10th Anniversary, Lucene!

For instance, for testing memory, it’s pretty easy to set up a series of tests that start with a small heap size and successively grow it until no Out Of Memory Errors (OOME) occur. Then, to be on the safe side, add 1 GB of memory to the heap.  It works for the large majority of people. Ironically, for Solr at least, this usually ends up with a heap size somewhere between 6-12 GBs for a system doing “consumer search” with faceting, etc. and reasonably sized caches on an index in the 10-50 million docs range. Sure, there are systems that go beyond this or are significantly less (I just saw one the other day that has around 200M docs in less than 3 GB of RAM while handling decent search load), but the 6-12 GB seems to be a nice sweet spot for the application and the JVM, especially when it comes to garbage collection, while still giving the operating system enough room to do it’s job.  Too much heap and garbage may pile up and give you ohmygodstoptheworld full garbage collections at some point in the future.  Too little heap and you get the dreaded OOME.  Also too much heap relative to total RAM and you choke off the OS.  Besides, that range also has a nice business/operations side effect in that 16 GBs of RAM has a nice performance/cost benefit for many people.

Recently, however, I thought it would be good to get beyond the inherent hand waving above and attempt to come up with a theoretical (with a little bit of empiricism thrown in) model for estimating memory usage and disk space.   After a few discussions on IRC with McCandless and others, I put together a DRAFT Excel spreadsheet that allows people to model both memory and disk space (based on the formula in Lucene in Action 2nd ed. – LIA2), after filling in some assumptions about their applications (I put in defaults.)   First a few caveats:

1. This is just an estimate, don’t construe it for what you are actually seeing in your system.
2. It is a DRAFT.  It is likely missing a few things, but I am putting it up here and in Subversion as a means to gather feedback.  I reserve the right to have messed up the calculations.
3. I feel the values might be a little bit low for the memory estimator, especially the Lucene section.
4. It’s only good for trunk.  I don’t think it will be right for 3.3 or 3.4.
5. The goal is to try to establish a model for the “high water mark” of memory and disk, not necessarily the typical case.
6. It inherently assumes you are searching and indexing on the same machine, which is often not the case.
7. There are still a couple of TODOs in the model.  More to come later.

As for using the memory estimator, the primary things to fill in are the number of documents, number of unique terms and information on sorting and indexed fields, but you can also mess with all of the other assumptions.  For Solr, there are entries for estimating cache memory usage.  Keep in mind that the assumption for caching is that they are full, which often is not the case and not even feasible.  For instance, your system may only ever employ 5 or 6 different filters.

The disk space estimator is much more straightforward and based on LIA2′s fairly simple formula of:

disk space used(original) = 1/3 original for each indexed field + 1 * original for stored + 2 * original per field with term vectors

It will be interesting to see how some of the new flexible indexing capabilities in trunk effect the results of this equation.  Also note, I’ve seen some applications where the size of the indexed fields is as low as 20%.

Hopefully, people will find this useful as well as enhance it and fix any bugs in it.  In other words, feedback is welcome.  As with any model like this, YMMV!

These improvements are now committed to Lucene’s trunk: you can use these models in tandem with all of Lucene’s features (boosts, slops, explanations, etc) and queries (term, phrase, spans, etc). A JIRA issue has been created to make it easy to use these models from Solr’s schema.xml.

Relevance ranking is the heart of the search engine, and I hope the additional models and flexibility will improve the user experience for Lucene: whether you’ve been frustrated with tuning TF/IDF weights and find an alternative model works better for your case, found it difficult to integrate custom logic that your application needs, or just want to experiment.

I’ll be giving a talk about how you can practically apply some of the upcoming Lucene 4 search features at Lucene Eurocon in October, and at the SFBay Apache Lucene/Solr Meetup later this month.

Some bullet points of the new scoring features:

• New ranking algorithms, in addition to Lucene’s Vector Space Model:
  • Okapi BM25 Model
  • Language Models
  • Divergence from Randomness Models
  • Information-based Models
• Added key statistics to the index format to support additional scoring models.
  • Term- and field-level statistics for collection frequencies and deriving averages.
  • Additional document-level statistics for computing normalization factors.
• Decoupled matching from ranking in Lucene’s core search classes:
  • Customize scoring without digging into the “guts”.
  • Customize explanations: essential for debugging relevance issues.
• Powerful low-level Similarity API, supporting advanced use cases:
  • Incorporate per-document values from Column Stride Fields into the score.
  • Use different scoring parameters or algorithms for different fields.
  • Fuse multiple scoring algorithms into a combined score.
• Convenient high-level SimilarityBase for everything else:
  • Write your own scoring function in one Java method.
  • Easy access to available index statistics.

For more information about this GSOC project, take a look at its wiki page

Solr makes using Lucene easier – you can index a variety of data sources easily, pretty much out of the box, and you can easily integrate features such as faceting, highlighting, and spellchecking – all without writing Java code. And if that’s all you need and it works solidly for you, awesome! You can stop reading now and attend one of our other excellent training courses that fit your needs. But if you are a tinkerer and want to know what makes Solr shine, or if you need some new or improved feature read on…

Deeper down, Lucene is cranking – analyzing, buffering, and indexing your documents, merging segments, parsing queries, caching data structures, rapidly hopping around an inverted index, computing scores, navigating finite state machines, and much more.

So how do you go about learning Lucene deeper? You can start with our “Understanding Lucene” DZone refcard.  And let’s not forget Lucene in Action, as it’s the most polished, detailed, and well crafted documentation available on the Lucene library. And of course there’s the incredibly vibrant and helpful Lucene open source community. Those resources will serve you well, but there’s no substitute for live, interactive, personal training to get you up to speed fast with best practices.

I’m in the process of overhauling our Lucene training course, that I’ll personally be delivering at Lucene EuroCon 2011 in Barcelona next month. This new and improved course takes an activity-based approach to learning and using Lucene’s API, beginning with the common tasks in building solutions using Lucene, whether you’re building directly to Lucene’s API or you’re writing custom components for Solr.

One area that I’m particularly jazzed about teaching is “query parsing”, the process of taking a user (or machine’s) search request and turning it into the appropriate underlying Lucene Query object instance.  Many folks developing with Lucene are familiar with Lucene’s QueryParser.  But did you know there are a couple of other query parsers with special powers?  There’s the surround query parser, enabling sophisticated proximity SpanQuery clauses.  And there’s the mysterious “XML query parser” (don’t let the ugly sounding name dissuade you) that slots dynamic query parameters, such as coming from an “advanced search” request, into a tree structured query template.   There’s some more insight into the world of Lucene query parsers an “Exploring Query Parsers” blog post.

What about all the Lucene contrib modules activity in the Lucene 3.x releases?   Here’s a bit of the goodnesses: better Unicode handling with the ICU tokenizers and filters, improved stemming, and many other analysis improvements, field grouping/collapsing, and block join/query for handling particular parent/child relationships.

Come learn the latest about the amazing Lucene library at Lucene EuroCon!  You, your boss, and your projects will all be glad you did.

Looking forward to seeing you there. Everyone in attendance will not only get to hear about the latest greatest enhancements to these technologies, but may also walk away with a cool Lucid t-shirt!

Here are the slides for “Rapid Prototyping with Solr”:

My next talk, titled “Solr Recipes”, was a 3 hour workshop covering the common ways to get data into Solr, configure it, and leverage it within applications.

Here are the “Solr Recipes” slides:

And finally, to a few hardcore folks, I discussed “Lucene for Solr Developers”, which more broadly covered the various ways to extend Solr. One cool example (or so I think, at least) I built for this talk that I’ve put out there as food for thought is “auto-faceting”, having a way for Solr to determine the best facets to select for a given query. See SOLR-2641 for my initial proof-of-concept implementation.

Here are the “Lucene for Solr Developers” slides:

There was so much going on that I barely got to tap into the conference experience myself, with the best part being the conversations had during meals, and between and during the scheduled talks. I was able to reconnect with many long-time friends that I’ve made through No Fluff, Just Stuff, and made many new acquaintances – I won’t name names, as this list covers most of them.

Thank you, Überconf, Jay, and fellow speakers and attendees for a stellar technical event. If you missed it, don’t despair, No Fluff, Just Stuff brings many of the same speakers and topics to an area near you. I’ll be speaking at a few NFJS events during the second half of this year, including Raleigh, NC, Boston, MA, and, surely to rival the quality and size of Überconf, The Rich Web Experience in Ft. Lauderdale, FL.

Let’s talk.

How, then, can you know how much you know about search with Solr and Lucene? In the world of Apache open source, there’s a clear meritocracy of peer review: contributors, committers, and active membership in the PMC. In theory, it’s a distinction anyone of sufficient talent and single-minded focus can achieve — just like anyone of sufficient talent and single-minded focus can make it to the NBA, or win the Nobel prize, or join the New York Philharmonic.

So you probably know your stuff if you’ve won the Nobel prize, made the NBA, or played the solo for Mozart’s Clarinet Concerto at Carnegie Hall, or you’re a Lucene/Solr contributor-or-committer. But what if you have not done any of those things, how do you know you know? Equally important, how do your peers or potential employers know how well you know your open source search stuff?

While there are more professional basketball players than Lucene/Solr committers, there are many, many more capable, talented, experienced Solr/Lucene application developers who are not going to ‘go pro’ in the Apache meritocracy. And the demand for Solr application development skills is exploding as interest and uptake of the leading open source application development technology spread like wildfire through organizations large and small. (Lucene Revolution, May 25-26 in San Francisco, will be packed with these people — sign up today if you haven’t already. And read on for another special  opportunity at Lucene Revolution).

It’s exactly for the broad base of interested, committed search application developers that today we’re introducing the Lucid Certified Apache Solr/Lucene Developer Program; a certification exam designed to benchmark development skills and experience in building applications with Apache Solr.

Designed with Prometric and a team of subject matter experts comprised of Apache Lucene/Solr committers, developers, and trainers, the test is designed to rigorously assess a broad base of search skills and experience, and provide the closest reasonable approximation possible to a standard measure of  search skills and experience.  It’s delivered via Prometric.com, consists of multiple choice questions, and costs $250. The test reflects a carefully selected, broad range of topics intended to reflect the real-world challenges and landscape of search application development problems, which you can see here.

Eric Pugh, who wrote the book on Solr, says this:

“I expect that the Lucid Imagination certification will quickly become the gold standard benchmark for whether someone who claims Solr and Lucene expertise truly possesses it. Oftentimes, a buyer of services has to take the leap of faith from sales pitch to execution that the knowledge is truly there. This certification can show, without a doubt, that the holder truly has the knowledge required to deliver a successful Solr/Lucene implementation. In the open source world, there are very few marks of authenticity: committer status, published author, and now the Lucid Solr certification. Just as the CPA certification shows a high level of knowledge and ability in the accounting industry, the Lucid Imagination Solr certification demonstrates unquestionable knowledge and experience in successful Solr/Lucene search engine implementation.”

It’s important to be clear about what the certification is not:

• It’s not easy: don’t expect to take your first Solr course one day and pass the exam the next.
• It’s not a substitute for experience: if you’ve only built one Solr application, earlier this morning, using the wiki demo that runs locally in your browser, you won’t pass.
• It’s not a substitute for training: taking a class from an expert may not be sufficient, but it will really help (and we offer the most professional-grade courses available; did I mention Lucene Revolution has classes available, too?)
• It’s not a casual conceptual overview: expect to answer detailed questions on everything from Lucene fundamentals to Solr debug output.
• It’s not a simple checklist of facts: you’ll have to demonstrate judgement calls in identifying correct answers to topic areas tied to searching, indexing, deployment, data source types, etc.

Testing as a pedagogical method — a mechanism for driving learning — is not the be-all-end-all of education (you probably didn’t think highly  of classmates who asked the teacher, “Will this be on the test?”). But it turns out that tests can have a salutary impact on acquiring and retaining knowledge, according to a recent article in Science.

We expect that this test will help level the playing field for a broad range of application developers to acquire and prove their Solr/Lucene application development skills — and help employers who want to take full advantage of the best open-source search technology on the planet find the men and women who have the stuff to do it.

If you’re coming to Lucene Revolution, the exam will be offered there for free — a savings of $250 over the regular price. Details are here.