First download the latest version of Solr from (3.3 as of this writing): http://www.apache.org/dyn/closer.cgi/lucene/solr/

Extract the compressed zip or tgz file to where you would like Solr to live.

Currently, I like using runit to run Linux services. http://smarden.org/runit/

Install runit with: sudo apt-get install runit

Create a new service directory.

Create a new shell script called run in the new /etc/sv/solr directory. You will need to have root permission to work in these directories, so use sudo. In this case, I want to run Solr as the user ‘mark’.

Make the run script executable.

Let runit know about the new service.

Now Solr should be up and running. If it dies or you kill it, it will automatically be restarted. If the server is restarted, Solr will be launched on startup.

To stop the service: sudo sv stop solr

To start the service: sudo sv start solr

Great.

Logging

By default, Solr logs to STD ERROR. You likely want to add a log configuration file to have the most control over how Solr logs – see http://wiki.apache.org/solr/LoggingInDefaultJettySetup. To be lazy though (and perhaps safe), let’s make sure STD OUT and STD ERR are nicely logged for us by runit.

This method just logs STD OUT, so lets first edit our Solr run script to redirect STD ERR to STD OUT

Now create a new directory called log in the /etc/sv/solr service directory. Inside this, create another script called run. This script will start the log service, run it under the user mark, and put the log files in the log directory we just made (we use . for the current working directory).

As we are running as mark, change the owner of the log dir to mark so that the log files can be created: sudo chown mark log

Now make the new run script executable.

The next time runit starts, Solr logs will be logged to the /etc/sv/solr/log/current file and auto rolled for you.

• What is Garbage Collection
• Tuning Garbage Collection
• The Garbage Collectors
• Choosing a Collector

What is Garbage Collection

Garbage collection in Java is the processes of freeing the dynamic memory used by objects that are no longer being used by an application. In languages such as or C or C++, the developer is often responsible for managing dynamic memory (using malloc and free or new and delete). However, in Java, this task is left up to something known as the garbage collector. A garbage collector automatically frees unused memory, freeing the developer from much of this thankless memory juggling.

The most basic garbage collection algorithm works by starting at the root objects (ie objects on the thread stack, static objects, etc) that are live (live meaning currently in use) – and then iterating down over every reachable object. Any object that cannot be reached in this manner is garbage and can be collected. The application is paused while this process goes on. This is referred to as mark and sweep – first you mark the objects that are live, then you sweep those that are not. The time needed to do this is obviously proportional to the number of live objects (which can be quite a large number in modern Java applications), and so more efficient collection schemes have been devised.

One such scheme comes from the natural fact that you can divide up objects based on how long they live. Most applications create a lot of very short lived objects, and fewer objects that are around for a long time (I’ve seen estimates that for the average application, 85-98% of allocated objects are short lived). You can take advantage of this fact when doing collections. In Java, objects are allocated from a region of memory known as the heap. The Java heap is generally divided up into a few spaces (its usually the same across implementations, but there is the odd exception or two). The major spaces are the young generation, the tenured generation (also called the old generation), and the permanent generation. The young generation is then further sub divided into the eden space and two survivor spaces. The permanent generation is generally for objects that are around for the life of the application (interned Strings, class objects, etc) and doesn’t usually play much of a role in garbage collection. The permanent generation size is not part of the heap region defined with -Xms and -Xmx. Though a very unusual need, it is still worth noting that the permanent generation can actually be collected if needed using:

-XX:+CMSPermGenSweepingEnabled

When objects are first created, they are allocated within the eden space. When the eden space becomes full, the still live objects within it are copied into one of the survivor spaces (or if they don’t fit, into the tenured space). One survivor space is always left empty, and on each young generation collection (a minor collection), the live objects from the eden space and the non empty survivor space are copied into the empty survivor space. This leaves a newly emptied survivor space for the next round, as any still live objects in the formerly full survivor space will be copied into the tenured space.

As you can see, rather then running over every object for every collection now, you can collect the young generations more often, and the tenured generation (long lived objects), much less often. You can also optimize your collection for the characteristics of the space – ie usually, almost all of the objects in the young space will be garbage. In general, an object will have to survive a couple minor collections to make it to the tenured space (first making it into a survivor space and then the tenured space). A copying collector identifies garbage by copying live objects from one space to another – anything left over is by definition garbage. The Sun JDK uses copying collectors for the young space and mark and sweep type collectors for the tenured space.

Tuning Garbage Collection

Tuning for garbage collection means adjusting the sizes of the various spaces mentioned in the previous section, as well as the algorithms used to collect them. You can do this with various JVM command line options.

The amount of RAM available for the various spaces is dependent upon the size of the heap that the JVM has allocated. Defaults are chosen based on the hardware detected, but you can usually do better by specifying a good Xms, Xmx yourself. On a server machine, it can be a good idea to pin those two settings together so that the JVM does not waste any time resizing itself. You generally do not want to size the heap much larger than is needed – this can needlessly increase the cost of full garbage collections, and take RAM from other important activities, such as file system caching.

-Xms

-Xmx

A Note About JVM Cmd Line Options

• Boolean options –   On: -XX:+<option> Off: -XX:-<option>.
• Numeric options:  -XX:<option>=<number>. Numbers can include ‘m’ or ‘M’ for megabytes, ‘k’ or ‘K’ for kilobytes, and ‘g’ or ‘G’ for gigabytes (1M= 1048576). In the case of Xms and Xmx, only one X is used and no colon.
• String options: -XX:<option>=<string>

Sizing the individual spaces

You usually want to grant plenty of memory to the young generation – especially when you have multiple processors – as allocation can be parallelized and each thread will get its own private piece of the eden space to work with. You generally want the young generation to have less than half the space of the tenured generation though – especially when using the Serialized collector. About 33% is usually a good number to start from. The best size will vary from application to application depending on its distribution of young vs long lived objects. You don’t want the young space to be so small that many short lived objects are getting piled into the tenured space. You also usually don’t want it to be so large that the tenured space doesn’t have enough space available to it and/or young generation collections start taking too long to complete.

Other than sizing the total heap, sizing the new generation (another name for the young generation) can be the most important piece to good performance.

-XX:NewSize

-XX:MaxNewSize

-Xmn

-XX:NewRatio

-XX:SurvivorRatio

The best sizing is usually chosen by playing with the parameters and then testing the performance of your application. Often, the JVM uses good defaults, or depending on the garbage collector in use, resizes the spaces on it’s own based on historical statistics.

There are a few helpful tools that give you insight into the garbage collection process.

Getting a View into Garbage Collection

You can use the following command line options to generate information about the garbage collection process:

-verbose:gc

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-Xloggc:C:\whereever\gc.log

There are various tools to then help you decipher these logs. One is GCViewer – though it only knows how to read gc logs up to Java 5.0 (though it can partially read 6.0 files). Another nice option from IBM is PMAT, and it can read Java 6 gc logs.

There is also a very cool tool called VisualGC that you can use to visually watch how objects move between spaces in real time as your application is running. This is available as a standalone application, or as a plugin for both Netbeans and VisualVM.

The Garbage Collectors

The following applies to the Sun Java implementation as well as OpenJDK.

There are three main garbage collection schemes that you should concern yourself with (much of this applies to Java 1.4, but in general, I am targeting Java 1.5 and up). These schemes are often called collectors themselves, but generally each involves two collectors – one for the old space and one for the new space. These collector schemes are often referred to by their old space collector names: the Serialized Collector, the Throughput Collector, and the Concurrent Low Pause Collector.

There is also an older incremental collector (unsupported and also called the train collector), and an incremental collection mode for the concurrent low pause collector (that I touch on and is generally used when only one or two CPU’s are available), but I’ll leave those for you to explore on your own if you are interested.

The Serialized Collector

-XX:+UseSerialGC (Since 5.0)

With the serialized collector, a major collection is done when the tenured space is full. This is known as a “stop the world” collection, because all application threads will be paused while the collection occurs.

This collector is best used with small applications, applications run on a single CPU machine, or applications where pause times don’t matter. This collector is relatively efficient because it does not need to communicate between threads, but you have to be willing to accept its “stop the world” pauses. Minor collections will “stop the world” as well, but are generally fairly efficient and fast.

This collector is the only one that I have seen to respect -XX:MaxHeapFreeRatio - though that still only happens if a full collection is triggered. If you where trying to keep your RAM usage to a minimum, and always return as much memory as possible to the operating system, using the serialized collector and an aggressive -XX:MaxHeapFreeRatio can be a good strategy. You might want to occasionally force a full collection with System.gc() when your application is idle.

The Throughput Collector (also known as the Parallel Collector)

-XX:+UseParallelGC (Since 1.4.1)

The throughput collector uses a parallel version of the young generation collector, while the tenured generation will still use the serial collector. So while a single thread will still perform collections on the tenured space, multiple threads will work together collecting the young space.

A feature called parallel compaction was added in Java 1.5 update 6 – this feature allows the throughput collector to also perform major collections in parallel. You can enable this with -XX:+UseParallelOldGC. Using this should help a lot with scalability, as you sidestep the single collection thread bottleneck on very large heaps (multi gigabyte). I’ve read this can actually lower performance on smaller heaps due to lock contention.

The throughput collector should be the default collector chosen on server class machines (in Java 1.5 and up), but there are exceptions – for example, my MacbookPro defaults to the CMS collector. You can always override these defaults.

Throughput is usually most useful when your application has a large number of threads creating new objects, and you have more than one processor available (though more than two is best). Typically, when you have multiple threads allocating objects, you also want to increase the size of the young generation. The number of garbage collector threads will generally be equal to the number of processors you have, but you can control that number with -XX:ParallelGCThreads=n. Sometimes you will want to lower the number of threads because each will reserve a part of the tenured generation for promotions – this can cause a fragmentation effect and effectively lower the size of the tenured generation (this is generally only an issue if your application has access to many processors or cores).

The throughput collector also supports something called Ergonomics. As part of this support, you can specify various desired behaviors for your application, and the JVM will attempt to tune various settings to meet your goals.

-XX:MaxGCPauseMillis=n hint to the throughput collector that a max pause time of n milliseconds is desired. By default there is no hint. The collector will adjust the heap size and other collection parameters in an attempt to meet the hint – keep in mind that throughput may be sacrificed in the attempt to meet this goal. There is also no guarantee that the goal will be met.

You can also specify a target goal for how much time is spent in garbage collection in comparison to running your application using -XX:GCTimeRatio. By default this is set to 1% (keep in mind that these defaults tend to change from release to release).

With the serialzed garbage collector a generation is collected when it is full (i.e., when no further allocations can be done from that generation). This is also true of the throughput collector.

The Concurrent Low Pause Collector

-XX:+UseConcMarkSweepGC (Since 1.4.1)

Use the concurrent low pause collector when you can afford to share the processor resources with the garbage collector while the application is running. This is usually good for an application with a lot of long lived data – meaning you need a large tenured generation space. Obviously, having multiple processors is also helpful. This collector still pauses the application threads twice in a collection – once briefly at the start (when it marks objects directly accessible from root objects), and a slightly longer pause towards the middle (when it sweeps to find what it missed due to parallel marking) – the rest of the collection is done concurrently using one of the available processors (or one thread). If this collector cannot complete collecting the tenured space before it is full, all threads will be paused and a full collection performed – this is known as a concurrent mode failure and likely means you need to adjust the concurrent collection parameters.

This collector is used for the tenured generation, and does the collection concurrently with the execution of the application. This collector can also be paired with a parallel version of the young generation collector (-XX:+UseParNewGC).

Note that -XX:+UseParallelGC (the throughput collector) should not be used with -XX:+UseConcMarkSweepGC, and the JVM will fail on startup if you try this with most modern JVMs. Same with -XX:+UseParallelOldGC.

The concurrent low pause collector will keep statistics so that it can best guess when to start collecting (so that it finishes before the tenured space is full) – also though, it will start collecting when the tenured space hits a percentage of what’s available – You can manually set this with -XX:CMSInitiatingOccupancyFraction=n. The default for this setting varies across JVMs. I’ve read that the default for 1.5 was 68%, while the default for 1.6 is 92%. You can lower this if needed to ensure that the collection is kicked off sooner, and then you will be more likely to finish the collection before the tenured space is full.

The concurrent low pause collector can also be used in an incremental mode that I will not go into here. This mode causes the low pause collector to occasional yield the processor used for parallel collection back to the application, and thereby lessen its impact on application performance.

The Parallel Young Generation Collector

-XX:+UseParNewGC

This collector is much like the throughput collector in that it collects the young generation in parallel. Most of what applies to the throughput collector also applies to this collector, however a different implementation is used that allows this collector to work in conjunction with the concurrent low pause collector, unlike the throughput collector. Despite some Sun/Oracle literature indicating this is off by default, it does seem to be on by default when using CMS in at least Java 6. You can disable it with:

-XX:+UseConcMarkSweepGC -XX:-UseParNewGC

The flip side of that coin is that while the throughput garbage collector (-XX:+UseParallelGC) can be used with adaptive sizing (-XX:+UseAdaptiveSizePolicy), the parallel young generation collector (-XX:+UseParNewGC) cannot.

-XX:+UseAdaptiveSizePolicy records statistics about GC times, allocation rates, and free space, and then sizes the young and tenured generations to best fit those statistics. This is for use with the throughput collector and is on by default.

Choosing a Collector

Note: this article is biased towards server applications and the -server hotspot vm.

Usually you just want to start with the Parallel (throughput) collector. It’s the one that has ergonomics, and it will automatically adjust key settings so that most server apps will do just fine. This is the default collector on most server class systems. In general, you do not need to change any garbage collection settings until you have determined you have a garbage collection issue to solve.

When you have to confront very large heaps, the Parallel collector can start to break down – it collects the tenured space using a stop the world collection, meaning your app is frozen while the collections happens. So when you find that the Parallel collector is just not cutting it, even when using UseParallelOldGC, you might try the mostly Concurrent Low-Pause Collector. It will collect as your application is running using a thread on the side, with two much shorter stop-the-world pauses. Overall, the CMS collector is slower in terms of throughput – but your application will likely be frozen less often.

Ergonomics do not apply here, so you are on your own for coming up with good settings if the defaults don’t turn out to be a good fit – but you can often remove long “the world is stopped” pauses with this collector.

The hope is that it is just going to make sense to always use the G1 collector in the future – it attempts to offer the best of both worlds of the throughput and mostly concurrent low pause collectors.

The Garbage First (G1) Collector

The Garbage First Collector is a new garbage collector that intends to rule them all. It is available in Sun Java 6 update 14 as well as recent versions of OpenJDK6 and early versions of OpenJDK 7. Eventually I plan to write more about his collector. Briefly: the G1 collector should combine the best of both the throughput and mostly concurrent low pause collectors. It uses new strategies to minimize stop the world pauses and maintain high throughput on multiprocessor systems with very large heaps.

Try this collector with:

-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

Solr is the popular, blazing fast open source enterprise search
platform from the Apache Lucene project. Its major features include
powerful full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, and rich document (e.g., Word, PDF)
handling. Solr is highly scalable, providing distributed search and
index replication, and it powers the search and navigation features of
many of the world’s largest internet sites.

Solr is written in Java and runs as a standalone full-text search server
within a servlet container such as Tomcat. Solr uses the Lucene Java
search library at its core for full-text indexing and search, and has
REST-like HTTP/XML and JSON APIs that make it easy to use from virtually
any programming language. Solr’s powerful external configuration allows
it to be tailored to almost any type of application without Java coding,
and it has an extensive plugin architecture when more advanced
customization is required.

Solr 1.4.1 is a bug fix release for Solr 1.4 that includes many Solr bug
fixes as well as Lucene bug fixes from Lucene 2.9.3.

See all of the CHANGES here:
http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1/CHANGES.txt

- – Mark Miller on behalf of the Solr team

On behalf of the Lucene development community I would like to announce the
release of Lucene Java versions 3.0.2 and 2.9.3:

Both releases fix bugs in the previous versions:

- 2.9.3 is a bugfix release for the Lucene Java 2.x series, based on Java
1.4.
- 3.0.2 has the same bug fix level but is for the Lucene Java 3.x series,
based on Java 5.

New users of Lucene are advised to use version 3.0.2 for new developments,
because it has a clean, type-safe API.

Important improvements in these releases include:
- Fixed memory leaks in IndexWriter when large documents are indexed. It
also uses now shared memory pools for term vectors and stored fields.
IndexWriter now releases Fieldables and Readers on close.
- NativeFSLockFactory fixes and improvements. Release write lock if
exception occurs in IndexWriter ctors.
- FieldCacheImpl.getStringIndex() no longer throws an exception when term
count exceeds doc count.
- Improve concurrency of IndexReader, especially in the context of near
real-time readers.
- Near real-time readers, opened while addIndexes* is running, no longer
miss some segments.
- Performance improvements in ParallelMultiSearcher (3.0.2 only).
- IndexSearcher no longer throws NegativeArraySizeException if you pass
Integer.MAX_VALUE as nDocs to search methods.

Both releases are fully compatible with the corresponding previous versions.
We strongly recommend upgrading to 2.9.3 if you are using 2.9.x; and to
3.0.2 if you are using 3.0.x.

See core changes at
http://lucene.apache.org/java/3_0_2/changes/Changes.html
http://lucene.apache.org/java/2_9_3/changes/Changes.html

and contrib changes at
http://lucene.apache.org/java/3_0_2/changes/Contrib-Changes.html
http://lucene.apache.org/java/2_9_3/changes/Contrib-Changes.html

Binary and source distributions are available at
http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at
http://repo1.maven.org/maven2/org/apache/lucene/

—–
Uwe Schindler
uschindler@apache.org
Apache Lucene PMC Member / Committer
Bremen, Germany

http://lucene.apache.org/

So what is with the merge?

Because of the way things worked in the past, even with many overlapping committers, many features that could benefit Lucene have been placed in Solr. They arguably “belonged” in Lucene, but due to dev issues, it benefited Solr to keep certain features that were contributed by Solr devs under Solr’s control. Moving some of this code to Lucene would mean that some Solr committers would no longer have access to it – A Solr committer that wrote and committed the code might actually lose the ability to maintain it without the assistance of a Lucene committer – and if Solr wanted to be sure to run off a stable, released version of Lucene, Solr’s release could be tied to Lucene’s latest release when some of this code needed to be updated. With Solr planning to update Lucene libs less frequently (due to the complexities of releasing with a development version of Lucene), there would be long waits for bug fixes to be available in Solr trunk.

All and all, there would be both pluses and minuses to refactoring Solr code into Lucene without the merge, but the majority have felt the minuses outweighed the pluses. Attempts at doing this type of thing in the past have failed and resulted in diverging similar code in both code bases. With many committers overlapping both projects, this was a very odd situation. Fix a bug in one place, and then go and look for the same bug in similar, but different code in another place – perhaps only being able to commit in one of the two spots.

With merged dev, there is now a single set of committers across both projects. Everyone in both communities can now drive releases – so when Solr releases, Lucene will also release – easing concerns about releasing Solr on a development version of Lucene. So now, Solr will always be on the latest trunk version of Lucene and code can be easily shared between projects – Lucene will likely benefit from Analyzers and QueryParsers that were only available to Solr users in the past. Lucene will also benefit from greater test coverage, as now you can make a single change in Lucene and run tests for both projects – getting immediate feedback on the change by testing an application that extensively uses the Lucene libraries. Both projects will also gain from a wider development community, as this change will foster more cross pollination between Lucene and Solr devs (now just Lucene/Solr devs).

All and all, I think this merge is going to be a big boon for both projects. A tremendous amount of work has already been done to get Solr working with the latest Lucene API’s and allow for a seamless development experience with Lucene/Solr as a single code base (the Lucene/Solr tests are ridiculously faster than they were as well!). Look for some really fantastic releases from Lucene/Solr in the future.

A little preview from the proposed release announce:

Important improvements in these releases are a increased maximum number of unique terms in each index segment. They also add fixes in IndexWriter’s commit and lost document deletes in near real-time indexing. Also lots of bugs in Contrib’s Analyzers package were fixed. Additionally, the 3.0.1 release restored some public methods, that get lost during deprecation removal. If you are using Lucene in a web application environment, you will notice that the new Attribute-based TokenStream API now works correct with different class loaders.
Both releases are fully compatible with the corresponding previous versions. We strongly recommend upgrading to 2.9.2 if you are using 2.9.1 or 2.9.0; and to 3.0.1 if you are using 3.0.0.

To learn more about what’s new in the Lucene 2.9.1 release, check out these resources:

• White Paper: “What’s New in Apache Lucene 2.9″
• Recorded Webinar: “Apache Lucene 2.9: Discover the Powerful New Features”

The History…

The first Lucene Highlighter was written and contributed to Lucene by Mark Harwood, a longtime Lucene contrib Committer and PMC member. Mark came up with a nice, robust, extensible API and a handful of default implementations for the API. It was a very solid Highlighter implementation that has held up nicely in the face of a lot of complicated Analyzers and Filters. A variety of contributors have enriched the code over the years since then (squashing bugs and making improvements), and the Highlighter is currently fairly capable and heavily used.

The Lucene contrib Highlighter was created with a focus on generating text fragments. This allows you to easily generate ‘keywords in context’ type views (ie the results list from your favorite search engine). Eventually, the NullFragmenter was added, allowing you to highlight a full document as well (you could have used the API to write your own NullFragmenter before Lucene added it – one of the nice things about the Highlighter’s fairly pluggable API).

Scoring and Highlighting…

The Highlighter works with a TokenStream and a Query. A TokenStream is just as it sounds: a stream of tokens – terms even, if thats easier – terms with possibly additional meta-data attached (position, offsets in original text, etc). An Analyzer and a document create a TokenStream – apply the Analyzer to the documents text, and out pops the Tokens. By comparing the tokens from the query with the tokens from the document, the Highlighter can identify which tokens should be highlighted (termFromDoc==termFromQuery? Highlight!). The highlighter works by feeding tokens from the document one at a time to a Scorer. The Scorer assigns a score to the token. The QueryScorer assigns the score based on whether the token matches a token in the query. Fragments are then generated and scored based on the underlying token scores. Generally the token score might just be 0 or 1, but you can do gradient highlighting by expanding the range of the scores (if you pass an IndexReader to QueryScorer, it will use term index stats to modify the score based on those stats). Finally, a pluggable Formatter implementation will actually insert the highlight text (using the score to decide what, if any, text to insert).

Obtaining a TokenStream for a Document…

Unfortunately, the index does not store the TokenStream for a document, so when its time to highlight, its up to the user to get a valid TokenStream for the document text. Generally this means shoving the original text for the document through the Analyzer you used for indexing the document. However, if you stored term vectors in your index, the position and/or offset information can be used to reconstruct the TokenStream from info in the index. Especially for large documents, this can be much faster. The TokenSources class in the Highlighter package will build a TokenStream for you, using the best method based on whether term vectors are available or not.

SpanScorer – adding position sensitive highlighting…

A couple of years ago I became interested in adding positional support to the Highlighter. The QueryScorer implementation just checks that tokens from the query match tokens from the document, and it doesn’t take the position of the tokens into account. The result is that if you use a PhraseQuery, rather than just highlighting the phrase, each term in the phrase will be highlighted everywhere it occurs. Attempts had been made to support PhraseQuery highlighting in the past, but not in a way that took advantage of the current Highlighter framework, and not in a way that supported the other positional queries (SpanQuery, MultiPhraseQuery, etc). I wanted pretty much full Highlighting support, as well as all of the goodness that had been squeezed into the current Highlighter. The result of this desire was a new Scorer implementation called SpanScorer.

The new SpanScorer would put the TokenStream into a fast single doc MemoryIndex, convert the query to a SpanQuery approximation, and call getSpans on the MemoryIndex to get all of the position hits for the document. This info is then used in scoring to filter out query terms that match doc terms, but are not in the correct position. The SpanScorer now supports almost the entire range of Lucene queries, and is just as fast as the QueryScorer for Query clauses that are not position sensitive.

Lucene added the SpanScorer in release 2.4 and Solr has also added support for the SpanScorer in 1.3. To take advantage of the SpanScorer in Lucene, just use the SpanScorer rather than the QueryScorer. You can enable the SpanScorer in Solr by passing hl.usePhraseHighlighter=true with your request.

Other Highlighter Implementations…

In Lucene JIRA there are a couple of other Highlighter implementations as well. The most interesting ideas you will find in them come from the two implementations that require term vectors to be stored for the documents you want to highlight. If you can enforce that requirement (something we don’t yet want to do with the default Highlighter), you can use the approach of looking at just the terms in the query, rather than looking at each of the terms in the document. This can be a very large win on large documents. The downsides are, that its not easy (and it hasn’t been done that I know of) to highlight based on position (phrase/span queries), and the exposed API for custom hooks is less rich. And of course, you have to store TermVectors to use the Highlighter.

]]> http://www.lucidimagination.com/blog/2009/02/17/highlighting-highlighter-thoughts/feed/ 0 http://www.lucidimagination.com/blog/2009/02/09/investigating-oom-and-other-jvm-issues/ http://www.lucidimagination.com/blog/2009/02/09/investigating-oom-and-other-jvm-issues/#comments Mon, 09 Feb 2009 15:29:40 +0000 Mark Miller http://www.lucidimagination.com/blog/?p=66 When working with a large scale Lucene/Solr installation, users sometimes run into memory issues or garbage collection performance problems. Its not a frequent occurrence in my experience, but just like life, it happens. Whenever I run into this sort of thing, I’ve come to rely on a variety of free tools to investigate the problem. There are many other free tools I have tried, but over time, I have found the following most useful:

jmap…

SOLR-560: Improved Logging

This lets you work with Solr with whatever logging implementation you are most familiar with. Very cool.

Use SLF4J logging API rather then JDK logging. The packaged .war file is shipped with a JDK logging implementation, so logging configuration for the .war should be identical to solr 1.3. However, if you are using the .jar file, you can select which logging implementation to use by dropping a different binding. See: http://www.slf4j.org/

SOLR-561: Pure Java Index Replication

A new replication implementation written in Java. This is a great addition to Solr and brings the simple replication that Unix users have taken for granted to Windows. Good stuff. Its not as battle tested as the old scripting/rsync solution, but its been used in production by a few people and has essentially gone through a strong beta period already. Anyone using replication for horizontal scaling should check this out.

Added Replication implemented in Java as a request handler. Supports index replication as well as configuration replication and exposes detailed statistics and progress information on the Admin page. Works on all platforms.

SOLR-284: Content Detection/Extraction with Tika

Apache Tika is a new Lucene sub project, and its looking very promising. Tika is a great content detection and extraction library that supports many popular formats:http://lucene.apache.org/tika/formats.html. This makes it a lot easier to pump most popular file types easily into Solr.

Added support for extracting content from binary documents like MS Word and PDF using Apache Tika.

SOLR-911: Multi-Select Faceting Support

Multi-select faceting support. Awesome. I’ve been seeing it more it more every day. Solr’s facet support continues to be excellent. Check out our use of multi-select at at www.lucidimagination.com/search.

Add support for multi-select faceting by allowing filters to be tagged and facet commands to exclude certain filters. This patch also added the ability to change the output key for facets in the response, and optimized distributed faceting refinement by lowering parsing overhead and by making requests and responses smaller.

SOLR-906: Buffered Updates With Solrj Over Http

More efficient index construction over http with solrj. If your doing it, this is a fantastic performance improvement.

Adding a StreamingUpdateSolrServer that writes update commands to an open HTTP connection. If you are using solrj for bulk update requests you should consider switching to this implementation. However, note that the error handling is not immediate as it is with the standard SolrServer.

SOLR-374: Index Reopen

Index reopening came to Lucene some time ago, and now comes to Solr. This means that when you add a couple documents to Solr, rather than opening the whole index again, only the one small segment is opened (subject to segment merging). A lot of work has gone on in Lucene development with reopen recently, and its going to be cool to see how Solr is able to take advantage of it all. Progress towards core real-time Lucene/Solr index/search is building.

Use IndexReader.reopen to save resources by re-using parts of the index that haven’t changed.

SOLR-475: Faceting Performance Boost

I havn’t used this first hand, but the reviews have been stellar. This should be a fantastic performance boost from what I hear.

New faceting method with better performance and smaller memory usage for multi-valued fields with many unique values but relatively few values per document. Controllable via the facet.method parameter – “fc” is the new default method and “enum” is the original method.

SOLR-84: New Solr Logo

Check out the new Solr logo. This was the winner of a community contest, and I think we really got a nice logo out of it. There were plenty of options to choose from, so it was a really successful contest

Use new Solr logo in admin

And…

And then of course, there are a handful of goodies in the latest Lucene libraries that will affect Solr or lead to new Solr features shortly – and tons of other features, bug fixes, and performance improvements.

- Mark