Once we get Phase2 into trunk we will work on hardening and finishing a couple missing features – then SolrCloud should be ready to be part of the upcoming Lucene/Solr 4.0 release.

If you want to read more about SolrCloud and where we are with Phase 2, check out the new wiki page that we are working on at http://wiki.apache.org/solr/SolrCloud - feedback appreciated!

November 27 2011, Apache Lucene™ 3.5.0 available

The Lucene PMC is pleased to announce the release of Apache Lucene 3.5.0.

Apache Lucene is a high-performance, full-featured text search engine

library written entirely in Java. It is a technology suitable for nearly

any application that requires full-text search, especially cross-platform.

This release contains numerous bug fixes, optimizations, and

improvements, some of which are highlighted below.  The release

is available for immediate download at:

http://www.apache.org/dyn/closer.cgi/lucene/java (see note below).

See the CHANGES.txt file included with the release for a full list of

details.

Lucene 3.5.0 Release Highlights:

* Added a very substantial (3-5X) RAM reduction required to hold the

terms index on opening an IndexReader. (LUCENE-2205)

* Added IndexSearcher.searchAfter which returns results after a

specified ScoreDoc (e.g. last document on the previous page) to

support deep paging use cases. (LUCENE-2215)

* Added SearcherManager to manage sharing and reopening IndexSearchers

across multiple search threads. Underlying IndexReader instances are

safely closed if not referenced anymore. (LUCENE-3445, LUCENE-3558)

* Added SearcherLifetimeManager which safely provides a consistent

view of the index across multiple requests (e.g. paging/drilldown).

(LUCENE-3558, LUCENE-3486)

* Renamed IndexWriter.optimize to forceMerge to discourage use of

this method since it is horribly costly and rarely justified

anymore. (LUCENE-3439)

* Added NGramPhraseQuery that speeds up phrase queries 30-50%

when n-gram analysis is used. (LUCENE-3426)

* Added a new reopen API (IndexReader.openIfChanged) that

returns null instead of the old reader if there are no changes

in the index. (LUCENE-3464)

* Improvements to vector highlighting: support for more queries

such as wildcards and boundary analysis for generated snippets

(LUCENE-1824, LUCENE-1889)

* IndexSearcher and IndexReader now perform additional checks to

throw AlreadyClosedExceptions if searches are performed on a

closed IndexReader. Performing searches on already closed reader

can cause JVM crashes when invalid memory mapped files are

referenced.

* Several bugfixes, including a bug where closing an NRT reader

after the writer was closed was incorrectly invoking the

DeletionPolicy. See CHANGES.txt entries for full details.

27 November 2011, Apache Solr™ 3.5.0 available

The Lucene PMC is pleased to announce the release of Apache Solr 3.5.0.

Solr is the popular, blazing fast open source enterprise search platform from

the Apache Lucene project. Its major features include powerful full-text

search, hit highlighting, faceted search, dynamic clustering, database

integration, rich document (e.g., Word, PDF) handling, and geospatial search.

Solr is highly scalable, providing distributed search and index replication,

and it powers the search and navigation features of many of the world’s

largest internet sites.

This release contains numerous bug fixes, optimizations, and

improvements, some of which are highlighted below.  The release

is available for immediate download at:

http://www.apache.org/dyn/closer.cgi/lucene/solr (see note below).

See the CHANGES.txt file included with the release for a full list of

details.

Solr 3.5.0 Release Highlights:

* Bug fixes and improvements from Apache Lucene 3.5.0, including a

very substantial (3-5X) RAM reduction required to hold the terms

index on opening an IndexReader. (LUCENE-2205)

* Added support for distributed result grouping. (SOLR-2066,

SOLR-2776)

* Added support for Hunspell stemmer TokenFilter supporting stemming

for 99 languages. (SOLR-2769)

* A new contrib module “langid” adds language identification

capabilities as an Update Processor, using Tika’s

LanguageIdentifier or Cybozu language-detection library (SOLR-1979)

* Numeric types including Trie and date types now support

sortMissingFirst/Last. (SOLR-2881)

* Added hl.q parameter. It is optional and if it is specified, it overrides

q parameter in Highlighter. (SOLR-1926)

* Several minor bugfixes like date parsing for years from 0001-1000, ignored

configurations when using QueryAnalyzer with SpellCheckComponent

and many more.

See CHANGES.txt entries for full details.

Note: The Apache Software Foundation uses an extensive mirroring network for

distributing releases.  It is possible that the mirror you are using may not

have replicated the release yet.  If that is the case, please try another

mirror.  This also goes for Maven access.

Happy searching,

Apache Lucene/Solr Developers

First download the latest version of Solr from (3.3 as of this writing): http://www.apache.org/dyn/closer.cgi/lucene/solr/

Extract the compressed zip or tgz file to where you would like Solr to live.

Currently, I like using runit to run Linux services. http://smarden.org/runit/

Install runit with: sudo apt-get install runit

Create a new service directory.

Create a new shell script called run in the new /etc/sv/solr directory. You will need to have root permission to work in these directories, so use sudo. In this case, I want to run Solr as the user ‘mark’.

Make the run script executable.

Let runit know about the new service.

Now Solr should be up and running. If it dies or you kill it, it will automatically be restarted. If the server is restarted, Solr will be launched on startup.

To stop the service: sudo sv stop solr

To start the service: sudo sv start solr

Great.

Logging

By default, Solr logs to STD ERROR. You likely want to add a log configuration file to have the most control over how Solr logs – see http://wiki.apache.org/solr/LoggingInDefaultJettySetup. To be lazy though (and perhaps safe), let’s make sure STD OUT and STD ERR are nicely logged for us by runit.

This method just logs STD OUT, so lets first edit our Solr run script to redirect STD ERR to STD OUT

Now create a new directory called log in the /etc/sv/solr service directory. Inside this, create another script called run. This script will start the log service, run it under the user mark, and put the log files in the log directory we just made (we use . for the current working directory).

As we are running as mark, change the owner of the log dir to mark so that the log files can be created: sudo chown mark log

Now make the new run script executable.

The next time runit starts, Solr logs will be logged to the /etc/sv/solr/log/current file and auto rolled for you.

To see how the recent changes have affected Solr’s performance, I decided to compare the most recent version of Solr trunk with a version from right before the UpdateHandler changes went into Solr trunk. I took the algorithm for NRT testing in Lucene and started tweaking it for use with Solr. I used my Intel Core 2 Quad @ 2.66 GHz for my test. It’s getting old, but it can still move a few bits. The results of this investigation follow.

The Process

So the first thing I did was check out the version of Solr that I wanted (revision 1141518) as well as the latest version of Solr trunk (at the time, revision 1144942). I then applied my Solr benchmark patch to each checkout and put together the following benchmark algorithm:

{

 StartSolrServer

 SolrClearIndex

 [ "PreLoad" { SolrAddDoc > : 50000] : 4

 SolrCommit

 Wait(220)

 [ "WarmupSearches" { SolrSearch > : 4 ] : 1

 # Get a new near-real-time reader, sequentially as fast as possible:

 [ "UpdateIndexView" { SolrCommit > : *] : 1 &

 # Index with 2 threads, each adding 100 docs per sec

 [ "Indexing" { SolrAddDoc > : * : 100/sec ] : 2 &

 # Redline search (from queries.txt) with 4 threads

 [ "Searching" { SolrSearch > : * ] : 4 &

 # Wait 60 sec, then wrap up

 Wait(60)

}

StopSolrServer

RepSumByPref Indexing

RepSumByPref Searching

RepSumByPref UpdateIndexView

This algorithm will start up the Solr example server (I’m using out of the box settings for this test), clear the current index, and then load 200,000 wikipedia docs into the index. Not a large index by any stretch, but it will help let us see the time affects due to various commit and merge activities well enough to make some simple judgements. After committing, the algorithm then waits 220 seconds – this is required on the latest trunk  version because commits no longer wait for background merges to complete – so we wait long enough for those merges to complete and not interfere with the benchmark. This is not necessary on the older version – that commit call will wait until the background merges are finished to return.

Next we do 4 searches to warm up the index just a bit before starting a background thread that will continuously call commit sequentially, as fast as possible. Then we start two more background threads, each adding wikipedia documents at a target rate of 100 docs per second. Then we start 4 background threads that each query Solr as fast as possible. We continue this barrage for a minute and then look at the results.

The Before Picture

This is a graph of the “refresh” times – the time it took to perform each commit and open up a new view on the index. In this case, the index was refreshed 400 times in the minute we allowed the benchmark to run for. For the most part, the refresh time really does not look too bad. The average “refresh” time is actually just 150ms. Now that Lucene and Solr work mostly per segment, this process can naturally be pretty fast. And this is a pretty small index really. There is a troubling spike in this minute though – one “refresh” time took about 23 seconds! The reason for this is that the commit triggered background merges, and Solr waited for those background merges to finish before opening a new IndexSearcher and releasing the commit lock. It gets worse though – not only was the refresh time hurt, but while that commit lock was held, neither of our 2 indexing threads could get a document into the index! They were effectively stalled. Over that minute, we were only able to index the wikipedia documents at 13.91 documents per second. Far below our target hopes of 100 documents per second for each thread! Also, there was a very large block of time were no indexing happened at all. Less troubling, our 4 threads were able to query at a rate of 11.24 queries per second (this can likely vary wildly depending on the ‘challenge’ of the queries.txt file) [UPDATE 9/4/2011: the search rate is very low due to a problem with the initial benchmark - many queries ended up malformed - without so many errors, search performance jumps drastically]. But overall, this is not an optimal use of this desktop’s resources.

The After Picture

Now we try with the new UpdateHandler. The new UpdateHandler no longer blocks updates while a commit is in progress. Nor does it wait for background merges to complete before opening a new IndexSearcher and returning.

The results are not bad – a low average refresh time of 116.74 ms, but also no 23 second spike. There are still spikes, but they are not too frequent, and stay below 2.5 seconds at worst. Micro spikes.

Even better though, our indexing rate is now 125.48 documents per second (vs 13.91 before). This is a fantastic increase – and likely absent large gaps of no indexing activity. The search performance dropped to 2.8 queries per second (from 11.24), but no doubt this is largely because of all the additional indexing activity that was able to take place. There was a lot more work which the CPUs could now do that they couldn’t before; since the indexing threads soaked up more CPU resources, queries were allocated fewer resources.

The After Picture With Lucene NRT

While I was changing around the UpdateHandler, a simple natural extension was too allow the use of Lucene’s NRT feature when opening new views of the index. This feature allows you to skip certain steps that a full commit performs. The tradeoff is that nothing is guaranteed to be on stable storage, but the benefit is very fast “refresh” times.

To take advantage of this in Solr, we added a new concept that I called a ‘soft’ commit. A soft commit returns quickly, but does not commit documents durably to stable storage. You must also occasionally call ‘hard’ commits to commit to stable storage. A ‘soft’ commit will refresh a SolrCore’s view of the index however.

You can now also setup a ‘soft’ auto-commit in solrconfig.xml. So you might, for example, set up a soft commit to occur every second or so, and a standard commit to occur every 5 minutes.

To test the new ‘soft’ commit feature, I changed the background commit line in the algorithm to:

[ "UpdateIndexView" { SolrCommit(soft) > : *] : 1 &

and ran the benchmark again.

Looking at the graph, it looks like the micro spikes are perhaps a little less frequent – more importantly though, the average “refresh” time has dropped from about 117ms to just 49ms. In the old case we were able to refresh the view 6.67 times in a minute – in the new case without soft commit it was 8.56 times per minute – and in the new case *with* soft commit, we were able to refresh the index view 22.18 times in a minute. We were also able to still index at 130 documents per second while running 3.64 queries per second.

One More Benchmark

Let’s try one more interesting benchmark. We tried to index at 200 documents per second in the previous tests – which my poor machine could not deliver on. So really, we just maxed out on indexing. In this test, I set the indexing rate to something achievable, rather than something that completely swamps the cpu – and looked at the results. This benchmark again used ‘soft’ commits.

Thing’s are looking pretty nice – an average of only 7.5 ms per refresh. That is a refresh rate of 132.6 times per second. Queries per second have also risen to nearly 6 per second from a little over 3 and half. This is an interesting result, and shows that there is still some interesting investigating to do with various settings and algorithm changes.

Conclusion

It looks like NRT in Solr has taken a nice step forward and these improvements will be available in Solr 4.0. There is still more to do though – certain features, such as faceting and function queries (edit: when you use ord), do not all yet work per segment. This means that using them can require more time than you might like when ‘refreshing’ the index view. Eventually we hope to improve most of those cases even further – but even when using those features,  in many cases, these changes will still allow you to significantly decrease your indexing to search time latency without resorting to clever tricks like juggling SolrCore’s.

Hopefully this was a fun glimpse at some of the improvements. There is a lot left to look into.

There are two details about the current UpdateHandler implementation that are particularly limiting.

First, Solr uses it’s own lock’s on top of Lucene, adding a courser, unnecessary layer of locking on top of the IndexWriter. These locks had a reason to exist once upon a time, but really, they no longer do. There is no reason to block additional document adds while performing a commit, but currently this is what Solr does. Removing these locks will reduce complexity and maintenance costs by allowing us to ‘mostly’ just use Lucene’s locking.  Solr will also more easily simply inherit improvements from Lucene in this area.

Second, because of historical requirements, Solr will close and open a new IndexWriter on every commit. This means that every commit waits for all background Index merging threads to finish merging. This can be a non insignificant amount of time – and during this time you cannot add any documents to the index. You also cannot see the documents that have just been added to the index until the merges and commit are complete. Really, the UpdateHandler should simply commit and open a new SolrIndexSearcher – with the background threads happily merging *in the background*.

There are a few other things that bug me as well.

Well I’m going to fix them all now. Time to remove the crust and introduce Lucene near-real-time support to Solr. You should be able to open a new view on recently added content with Solr in a fraction of the time possible right now. It’s not right that you have to juggle SolrCore’s to attempt near real time index updates – it’s time to make things easier. Time to makes things faster.

And when Lucene finishes it’s real-time support and stops IndexWriter flushes from blocking document additions, Solr will be even more ready to take advantage where it can. There will still be more to do – not everything Solr does is yet per segment, and replication is not currently very near-real-time friendly – but we will keeping moving things in the right direction.

I’m tackling these changes here: https://issues.apache.org/jira/browse/SOLR-2193

- Mark

• What is Garbage Collection
• Tuning Garbage Collection
• The Garbage Collectors
• Choosing a Collector

What is Garbage Collection

Garbage collection in Java is the processes of freeing the dynamic memory used by objects that are no longer being used by an application. In languages such as or C or C++, the developer is often responsible for managing dynamic memory (using malloc and free or new and delete). However, in Java, this task is left up to something known as the garbage collector. A garbage collector automatically frees unused memory, freeing the developer from much of this thankless memory juggling.

The most basic garbage collection algorithm works by starting at the root objects (ie objects on the thread stack, static objects, etc) that are live (live meaning currently in use) – and then iterating down over every reachable object. Any object that cannot be reached in this manner is garbage and can be collected. The application is paused while this process goes on. This is referred to as mark and sweep – first you mark the objects that are live, then you sweep those that are not. The time needed to do this is obviously proportional to the number of live objects (which can be quite a large number in modern Java applications), and so more efficient collection schemes have been devised.

One such scheme comes from the natural fact that you can divide up objects based on how long they live. Most applications create a lot of very short lived objects, and fewer objects that are around for a long time (I’ve seen estimates that for the average application, 85-98% of allocated objects are short lived). You can take advantage of this fact when doing collections. In Java, objects are allocated from a region of memory known as the heap. The Java heap is generally divided up into a few spaces (its usually the same across implementations, but there is the odd exception or two). The major spaces are the young generation, the tenured generation (also called the old generation), and the permanent generation. The young generation is then further sub divided into the eden space and two survivor spaces. The permanent generation is generally for objects that are around for the life of the application (interned Strings, class objects, etc) and doesn’t usually play much of a role in garbage collection. The permanent generation size is not part of the heap region defined with -Xms and -Xmx. Though a very unusual need, it is still worth noting that the permanent generation can actually be collected if needed using:

-XX:+CMSPermGenSweepingEnabled

When objects are first created, they are allocated within the eden space. When the eden space becomes full, the still live objects within it are copied into one of the survivor spaces (or if they don’t fit, into the tenured space). One survivor space is always left empty, and on each young generation collection (a minor collection), the live objects from the eden space and the non empty survivor space are copied into the empty survivor space. This leaves a newly emptied survivor space for the next round, as any still live objects in the formerly full survivor space will be copied into the tenured space.

As you can see, rather then running over every object for every collection now, you can collect the young generations more often, and the tenured generation (long lived objects), much less often. You can also optimize your collection for the characteristics of the space – ie usually, almost all of the objects in the young space will be garbage. In general, an object will have to survive a couple minor collections to make it to the tenured space (first making it into a survivor space and then the tenured space). A copying collector identifies garbage by copying live objects from one space to another – anything left over is by definition garbage. The Sun JDK uses copying collectors for the young space and mark and sweep type collectors for the tenured space.

Tuning Garbage Collection

Tuning for garbage collection means adjusting the sizes of the various spaces mentioned in the previous section, as well as the algorithms used to collect them. You can do this with various JVM command line options.

The amount of RAM available for the various spaces is dependent upon the size of the heap that the JVM has allocated. Defaults are chosen based on the hardware detected, but you can usually do better by specifying a good Xms, Xmx yourself. On a server machine, it can be a good idea to pin those two settings together so that the JVM does not waste any time resizing itself. You generally do not want to size the heap much larger than is needed – this can needlessly increase the cost of full garbage collections, and take RAM from other important activities, such as file system caching.

-Xms

-Xmx

A Note About JVM Cmd Line Options

• Boolean options –   On: -XX:+<option> Off: -XX:-<option>.
• Numeric options:  -XX:<option>=<number>. Numbers can include ‘m’ or ‘M’ for megabytes, ‘k’ or ‘K’ for kilobytes, and ‘g’ or ‘G’ for gigabytes (1M= 1048576). In the case of Xms and Xmx, only one X is used and no colon.
• String options: -XX:<option>=<string>

Sizing the individual spaces

You usually want to grant plenty of memory to the young generation – especially when you have multiple processors – as allocation can be parallelized and each thread will get its own private piece of the eden space to work with. You generally want the young generation to have less than half the space of the tenured generation though – especially when using the Serialized collector. About 33% is usually a good number to start from. The best size will vary from application to application depending on its distribution of young vs long lived objects. You don’t want the young space to be so small that many short lived objects are getting piled into the tenured space. You also usually don’t want it to be so large that the tenured space doesn’t have enough space available to it and/or young generation collections start taking too long to complete.

Other than sizing the total heap, sizing the new generation (another name for the young generation) can be the most important piece to good performance.

-XX:NewSize

-XX:MaxNewSize

-Xmn

-XX:NewRatio

-XX:SurvivorRatio

The best sizing is usually chosen by playing with the parameters and then testing the performance of your application. Often, the JVM uses good defaults, or depending on the garbage collector in use, resizes the spaces on it’s own based on historical statistics.

There are a few helpful tools that give you insight into the garbage collection process.

Getting a View into Garbage Collection

You can use the following command line options to generate information about the garbage collection process:

-verbose:gc

-XX:+PrintGCDetails

-XX:+PrintGCTimeStamps

-Xloggc:C:\whereever\gc.log

There are various tools to then help you decipher these logs. One is GCViewer – though it only knows how to read gc logs up to Java 5.0 (though it can partially read 6.0 files). Another nice option from IBM is PMAT, and it can read Java 6 gc logs.

There is also a very cool tool called VisualGC that you can use to visually watch how objects move between spaces in real time as your application is running. This is available as a standalone application, or as a plugin for both Netbeans and VisualVM.

The Garbage Collectors

The following applies to the Sun Java implementation as well as OpenJDK.

There are three main garbage collection schemes that you should concern yourself with (much of this applies to Java 1.4, but in general, I am targeting Java 1.5 and up). These schemes are often called collectors themselves, but generally each involves two collectors – one for the old space and one for the new space. These collector schemes are often referred to by their old space collector names: the Serialized Collector, the Throughput Collector, and the Concurrent Low Pause Collector.

There is also an older incremental collector (unsupported and also called the train collector), and an incremental collection mode for the concurrent low pause collector (that I touch on and is generally used when only one or two CPU’s are available), but I’ll leave those for you to explore on your own if you are interested.

The Serialized Collector

-XX:+UseSerialGC (Since 5.0)

With the serialized collector, a major collection is done when the tenured space is full. This is known as a “stop the world” collection, because all application threads will be paused while the collection occurs.

This collector is best used with small applications, applications run on a single CPU machine, or applications where pause times don’t matter. This collector is relatively efficient because it does not need to communicate between threads, but you have to be willing to accept its “stop the world” pauses. Minor collections will “stop the world” as well, but are generally fairly efficient and fast.

This collector is the only one that I have seen to respect -XX:MaxHeapFreeRatio - though that still only happens if a full collection is triggered. If you where trying to keep your RAM usage to a minimum, and always return as much memory as possible to the operating system, using the serialized collector and an aggressive -XX:MaxHeapFreeRatio can be a good strategy. You might want to occasionally force a full collection with System.gc() when your application is idle.

The Throughput Collector (also known as the Parallel Collector)

-XX:+UseParallelGC (Since 1.4.1)

The throughput collector uses a parallel version of the young generation collector, while the tenured generation will still use the serial collector. So while a single thread will still perform collections on the tenured space, multiple threads will work together collecting the young space.

A feature called parallel compaction was added in Java 1.5 update 6 – this feature allows the throughput collector to also perform major collections in parallel. You can enable this with -XX:+UseParallelOldGC. Using this should help a lot with scalability, as you sidestep the single collection thread bottleneck on very large heaps (multi gigabyte). I’ve read this can actually lower performance on smaller heaps due to lock contention.

The throughput collector should be the default collector chosen on server class machines (in Java 1.5 and up), but there are exceptions – for example, my MacbookPro defaults to the CMS collector. You can always override these defaults.

Throughput is usually most useful when your application has a large number of threads creating new objects, and you have more than one processor available (though more than two is best). Typically, when you have multiple threads allocating objects, you also want to increase the size of the young generation. The number of garbage collector threads will generally be equal to the number of processors you have, but you can control that number with -XX:ParallelGCThreads=n. Sometimes you will want to lower the number of threads because each will reserve a part of the tenured generation for promotions – this can cause a fragmentation effect and effectively lower the size of the tenured generation (this is generally only an issue if your application has access to many processors or cores).

The throughput collector also supports something called Ergonomics. As part of this support, you can specify various desired behaviors for your application, and the JVM will attempt to tune various settings to meet your goals.

-XX:MaxGCPauseMillis=n hint to the throughput collector that a max pause time of n milliseconds is desired. By default there is no hint. The collector will adjust the heap size and other collection parameters in an attempt to meet the hint – keep in mind that throughput may be sacrificed in the attempt to meet this goal. There is also no guarantee that the goal will be met.

You can also specify a target goal for how much time is spent in garbage collection in comparison to running your application using -XX:GCTimeRatio. By default this is set to 1% (keep in mind that these defaults tend to change from release to release).

With the serialzed garbage collector a generation is collected when it is full (i.e., when no further allocations can be done from that generation). This is also true of the throughput collector.

The Concurrent Low Pause Collector

-XX:+UseConcMarkSweepGC (Since 1.4.1)

Use the concurrent low pause collector when you can afford to share the processor resources with the garbage collector while the application is running. This is usually good for an application with a lot of long lived data – meaning you need a large tenured generation space. Obviously, having multiple processors is also helpful. This collector still pauses the application threads twice in a collection – once briefly at the start (when it marks objects directly accessible from root objects), and a slightly longer pause towards the middle (when it sweeps to find what it missed due to parallel marking) – the rest of the collection is done concurrently using one of the available processors (or one thread). If this collector cannot complete collecting the tenured space before it is full, all threads will be paused and a full collection performed – this is known as a concurrent mode failure and likely means you need to adjust the concurrent collection parameters.

This collector is used for the tenured generation, and does the collection concurrently with the execution of the application. This collector can also be paired with a parallel version of the young generation collector (-XX:+UseParNewGC).

Note that -XX:+UseParallelGC (the throughput collector) should not be used with -XX:+UseConcMarkSweepGC, and the JVM will fail on startup if you try this with most modern JVMs. Same with -XX:+UseParallelOldGC.

The concurrent low pause collector will keep statistics so that it can best guess when to start collecting (so that it finishes before the tenured space is full) – also though, it will start collecting when the tenured space hits a percentage of what’s available – You can manually set this with -XX:CMSInitiatingOccupancyFraction=n. The default for this setting varies across JVMs. I’ve read that the default for 1.5 was 68%, while the default for 1.6 is 92%. You can lower this if needed to ensure that the collection is kicked off sooner, and then you will be more likely to finish the collection before the tenured space is full.

The concurrent low pause collector can also be used in an incremental mode that I will not go into here. This mode causes the low pause collector to occasional yield the processor used for parallel collection back to the application, and thereby lessen its impact on application performance.

The Parallel Young Generation Collector

-XX:+UseParNewGC

This collector is much like the throughput collector in that it collects the young generation in parallel. Most of what applies to the throughput collector also applies to this collector, however a different implementation is used that allows this collector to work in conjunction with the concurrent low pause collector, unlike the throughput collector. Despite some Sun/Oracle literature indicating this is off by default, it does seem to be on by default when using CMS in at least Java 6. You can disable it with:

-XX:+UseConcMarkSweepGC -XX:-UseParNewGC

The flip side of that coin is that while the throughput garbage collector (-XX:+UseParallelGC) can be used with adaptive sizing (-XX:+UseAdaptiveSizePolicy), the parallel young generation collector (-XX:+UseParNewGC) cannot.

-XX:+UseAdaptiveSizePolicy records statistics about GC times, allocation rates, and free space, and then sizes the young and tenured generations to best fit those statistics. This is for use with the throughput collector and is on by default.

Choosing a Collector

Note: this article is biased towards server applications and the -server hotspot vm.

Usually you just want to start with the Parallel (throughput) collector. It’s the one that has ergonomics, and it will automatically adjust key settings so that most server apps will do just fine. This is the default collector on most server class systems. In general, you do not need to change any garbage collection settings until you have determined you have a garbage collection issue to solve.

When you have to confront very large heaps, the Parallel collector can start to break down – it collects the tenured space using a stop the world collection, meaning your app is frozen while the collections happens. So when you find that the Parallel collector is just not cutting it, even when using UseParallelOldGC, you might try the mostly Concurrent Low-Pause Collector. It will collect as your application is running using a thread on the side, with two much shorter stop-the-world pauses. Overall, the CMS collector is slower in terms of throughput – but your application will likely be frozen less often.

Ergonomics do not apply here, so you are on your own for coming up with good settings if the defaults don’t turn out to be a good fit – but you can often remove long “the world is stopped” pauses with this collector.

The hope is that it is just going to make sense to always use the G1 collector in the future – it attempts to offer the best of both worlds of the throughput and mostly concurrent low pause collectors.

The Garbage First (G1) Collector

The Garbage First Collector is a new garbage collector that intends to rule them all. It is available in Sun Java 6 update 14 as well as recent versions of OpenJDK6 and early versions of OpenJDK 7. Eventually I plan to write more about his collector. Briefly: the G1 collector should combine the best of both the throughput and mostly concurrent low pause collectors. It uses new strategies to minimize stop the world pauses and maintain high throughput on multiprocessor systems with very large heaps.

Try this collector with:

-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

Solr is the popular, blazing fast open source enterprise search
platform from the Apache Lucene project. Its major features include
powerful full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, and rich document (e.g., Word, PDF)
handling. Solr is highly scalable, providing distributed search and
index replication, and it powers the search and navigation features of
many of the world’s largest internet sites.

Solr is written in Java and runs as a standalone full-text search server
within a servlet container such as Tomcat. Solr uses the Lucene Java
search library at its core for full-text indexing and search, and has
REST-like HTTP/XML and JSON APIs that make it easy to use from virtually
any programming language. Solr’s powerful external configuration allows
it to be tailored to almost any type of application without Java coding,
and it has an extensive plugin architecture when more advanced
customization is required.

Solr 1.4.1 is a bug fix release for Solr 1.4 that includes many Solr bug
fixes as well as Lucene bug fixes from Lucene 2.9.3.

See all of the CHANGES here:
http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1/CHANGES.txt

- – Mark Miller on behalf of the Solr team

On behalf of the Lucene development community I would like to announce the
release of Lucene Java versions 3.0.2 and 2.9.3:

Both releases fix bugs in the previous versions:

- 2.9.3 is a bugfix release for the Lucene Java 2.x series, based on Java
1.4.
- 3.0.2 has the same bug fix level but is for the Lucene Java 3.x series,
based on Java 5.

New users of Lucene are advised to use version 3.0.2 for new developments,
because it has a clean, type-safe API.

Important improvements in these releases include:
- Fixed memory leaks in IndexWriter when large documents are indexed. It
also uses now shared memory pools for term vectors and stored fields.
IndexWriter now releases Fieldables and Readers on close.
- NativeFSLockFactory fixes and improvements. Release write lock if
exception occurs in IndexWriter ctors.
- FieldCacheImpl.getStringIndex() no longer throws an exception when term
count exceeds doc count.
- Improve concurrency of IndexReader, especially in the context of near
real-time readers.
- Near real-time readers, opened while addIndexes* is running, no longer
miss some segments.
- Performance improvements in ParallelMultiSearcher (3.0.2 only).
- IndexSearcher no longer throws NegativeArraySizeException if you pass
Integer.MAX_VALUE as nDocs to search methods.

Both releases are fully compatible with the corresponding previous versions.
We strongly recommend upgrading to 2.9.3 if you are using 2.9.x; and to
3.0.2 if you are using 3.0.x.

See core changes at
http://lucene.apache.org/java/3_0_2/changes/Changes.html
http://lucene.apache.org/java/2_9_3/changes/Changes.html

and contrib changes at
http://lucene.apache.org/java/3_0_2/changes/Contrib-Changes.html
http://lucene.apache.org/java/2_9_3/changes/Contrib-Changes.html

Binary and source distributions are available at
http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at
http://repo1.maven.org/maven2/org/apache/lucene/

—–
Uwe Schindler
uschindler@apache.org
Apache Lucene PMC Member / Committer
Bremen, Germany

http://lucene.apache.org/

So what is with the merge?

Because of the way things worked in the past, even with many overlapping committers, many features that could benefit Lucene have been placed in Solr. They arguably “belonged” in Lucene, but due to dev issues, it benefited Solr to keep certain features that were contributed by Solr devs under Solr’s control. Moving some of this code to Lucene would mean that some Solr committers would no longer have access to it – A Solr committer that wrote and committed the code might actually lose the ability to maintain it without the assistance of a Lucene committer – and if Solr wanted to be sure to run off a stable, released version of Lucene, Solr’s release could be tied to Lucene’s latest release when some of this code needed to be updated. With Solr planning to update Lucene libs less frequently (due to the complexities of releasing with a development version of Lucene), there would be long waits for bug fixes to be available in Solr trunk.

All and all, there would be both pluses and minuses to refactoring Solr code into Lucene without the merge, but the majority have felt the minuses outweighed the pluses. Attempts at doing this type of thing in the past have failed and resulted in diverging similar code in both code bases. With many committers overlapping both projects, this was a very odd situation. Fix a bug in one place, and then go and look for the same bug in similar, but different code in another place – perhaps only being able to commit in one of the two spots.

With merged dev, there is now a single set of committers across both projects. Everyone in both communities can now drive releases – so when Solr releases, Lucene will also release – easing concerns about releasing Solr on a development version of Lucene. So now, Solr will always be on the latest trunk version of Lucene and code can be easily shared between projects – Lucene will likely benefit from Analyzers and QueryParsers that were only available to Solr users in the past. Lucene will also benefit from greater test coverage, as now you can make a single change in Lucene and run tests for both projects – getting immediate feedback on the change by testing an application that extensively uses the Lucene libraries. Both projects will also gain from a wider development community, as this change will foster more cross pollination between Lucene and Solr devs (now just Lucene/Solr devs).

All and all, I think this merge is going to be a big boon for both projects. A tremendous amount of work has already been done to get Solr working with the latest Lucene API’s and allow for a seamless development experience with Lucene/Solr as a single code base (the Lucene/Solr tests are ridiculously faster than they were as well!). Look for some really fantastic releases from Lucene/Solr in the future.