Lucid Imagination » Mark Miller

[ANNOUNCE] Solr 1.4.1 Released

Mark Miller — Tue, 29 Jun 2010 01:23:11 +0000

Apache Solr 1.4.1 has been released and is now available for public
download!
http://www.apache.org/dyn/closer.cgi/lucene/solr/

Solr is the popular, blazing fast open source enterprise search
platform from the Apache Lucene project. Its major features include
powerful full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, and rich document (e.g., Word, PDF)
handling. Solr is highly scalable, providing distributed search and
index replication, and it powers the search and navigation features of
many of the world’s largest internet sites.

Solr is written in Java and runs as a standalone full-text search server
within a servlet container such as Tomcat. Solr uses the Lucene Java
search library at its core for full-text indexing and search, and has
REST-like HTTP/XML and JSON APIs that make it easy to use from virtually
any programming language. Solr’s powerful external configuration allows
it to be tailored to almost any type of application without Java coding,
and it has an extensive plugin architecture when more advanced
customization is required.

Solr 1.4.1 is a bug fix release for Solr 1.4 that includes many Solr bug
fixes as well as Lucene bug fixes from Lucene 2.9.3.

See all of the CHANGES here:
http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1/CHANGES.txt

- – Mark Miller on behalf of the Solr team

[ANNOUNCE] Release of Lucene Java 3.0.2 and 2.9.3

Mark Miller — Fri, 18 Jun 2010 17:01:29 +0000

Hello Lucene users,

On behalf of the Lucene development community I would like to announce the
release of Lucene Java versions 3.0.2 and 2.9.3:

Both releases fix bugs in the previous versions:

- 2.9.3 is a bugfix release for the Lucene Java 2.x series, based on Java
1.4.
- 3.0.2 has the same bug fix level but is for the Lucene Java 3.x series,
based on Java 5.

New users of Lucene are advised to use version 3.0.2 for new developments,
because it has a clean, type-safe API.

Important improvements in these releases include:
- Fixed memory leaks in IndexWriter when large documents are indexed. It
also uses now shared memory pools for term vectors and stored fields.
IndexWriter now releases Fieldables and Readers on close.
- NativeFSLockFactory fixes and improvements. Release write lock if
exception occurs in IndexWriter ctors.
- FieldCacheImpl.getStringIndex() no longer throws an exception when term
count exceeds doc count.
- Improve concurrency of IndexReader, especially in the context of near
real-time readers.
- Near real-time readers, opened while addIndexes* is running, no longer
miss some segments.
- Performance improvements in ParallelMultiSearcher (3.0.2 only).
- IndexSearcher no longer throws NegativeArraySizeException if you pass
Integer.MAX_VALUE as nDocs to search methods.

Both releases are fully compatible with the corresponding previous versions.
We strongly recommend upgrading to 2.9.3 if you are using 2.9.x; and to
3.0.2 if you are using 3.0.x.

See core changes at
http://lucene.apache.org/java/3_0_2/changes/Changes.html
http://lucene.apache.org/java/2_9_3/changes/Changes.html

and contrib changes at
http://lucene.apache.org/java/3_0_2/changes/Contrib-Changes.html
http://lucene.apache.org/java/2_9_3/changes/Contrib-Changes.html

Binary and source distributions are available at
http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at
http://repo1.maven.org/maven2/org/apache/lucene/

—–
Uwe Schindler
uschindler@apache.org
Apache Lucene PMC Member / Committer
Bremen, Germany

http://lucene.apache.org/

Lucene and Solr Development Have Merged

Mark Miller — Fri, 26 Mar 2010 23:06:52 +0000

The Lucene community has recently decided to merge the development of two of its sub-projects – Lucene->Java and Lucene->Solr. Both code bases now sit under the same trunk in svn and Solr actually runs straight off the latest Lucene code at all times. This is just a merge of development though. Release artifacts will remain separate: Lucene will remain a core search engine Java library and Solr will remain a search server built on top of Lucene. From a user perspective, things will be much the same as they were – just better.

So what is with the merge?

Because of the way things worked in the past, even with many overlapping committers, many features that could benefit Lucene have been placed in Solr. They arguably “belonged” in Lucene, but due to dev issues, it benefited Solr to keep certain features that were contributed by Solr devs under Solr’s control. Moving some of this code to Lucene would mean that some Solr committers would no longer have access to it – A Solr committer that wrote and committed the code might actually lose the ability to maintain it without the assistance of a Lucene committer – and if Solr wanted to be sure to run off a stable, released version of Lucene, Solr’s release could be tied to Lucene’s latest release when some of this code needed to be updated. With Solr planning to update Lucene libs less frequently (due to the complexities of releasing with a development version of Lucene), there would be long waits for bug fixes to be available in Solr trunk.

All and all, there would be both pluses and minuses to refactoring Solr code into Lucene without the merge, but the majority have felt the minuses outweighed the pluses. Attempts at doing this type of thing in the past have failed and resulted in diverging similar code in both code bases. With many committers overlapping both projects, this was a very odd situation. Fix a bug in one place, and then go and look for the same bug in similar, but different code in another place – perhaps only being able to commit in one of the two spots.

With merged dev, there is now a single set of committers across both projects. Everyone in both communities can now drive releases – so when Solr releases, Lucene will also release – easing concerns about releasing Solr on a development version of Lucene. So now, Solr will always be on the latest trunk version of Lucene and code can be easily shared between projects – Lucene will likely benefit from Analyzers and QueryParsers that were only available to Solr users in the past. Lucene will also benefit from greater test coverage, as now you can make a single change in Lucene and run tests for both projects – getting immediate feedback on the change by testing an application that extensively uses the Lucene libraries. Both projects will also gain from a wider development community, as this change will foster more cross pollination between Lucene and Solr devs (now just Lucene/Solr devs).

All and all, I think this merge is going to be a big boon for both projects. A tremendous amount of work has already been done to get Solr working with the latest Lucene API’s and allow for a seamless development experience with Lucene/Solr as a single code base (the Lucene/Solr tests are ridiculously faster than they were as well!). Look for some really fantastic releases from Lucene/Solr in the future.

Lucene 2.9.2 and 3.0.1

Mark Miller — Wed, 17 Feb 2010 14:02:39 +0000

The vote is on for what I think is a Lucene first – two simultaneous bug fix releases. Because the Lucene 2 series is the last to support Java 1.4, we are doing a bug fix release for for 2.9 as well as the recently released Java 1.5 required 3.0 release.

A little preview from the proposed release announce:

Important improvements in these releases are a increased maximum number of unique terms in each index segment. They also add fixes in IndexWriter’s commit and lost document deletes in near real-time indexing. Also lots of bugs in Contrib’s Analyzers package were fixed. Additionally, the 3.0.1 release restored some public methods, that get lost during deprecation removal. If you are using Lucene in a web application environment, you will notice that the new Attribute-based TokenStream API now works correct with different class loaders.
Both releases are fully compatible with the corresponding previous versions. We strongly recommend upgrading to 2.9.2 if you are using 2.9.1 or 2.9.0; and to 3.0.1 if you are using 3.0.0.

Lucene 2.9.1 Released

Mark Miller — Tue, 10 Nov 2009 01:30:04 +0000

We were all so caught up in the fun at ApacheCon that no one announced the Lucene 2.9.1 release. Its out, and its highly recommended if you are currently on 2.9.0. Check it out: http://lucene.apache.org/java/docs/#6+November+2009+-+Lucene+Java+2.9.1+available

To learn more about what’s new in the Lucene 2.9.1 release, check out these resources:

White Paper: “What’s New in Apache Lucene 2.9″
Recorded Webinar: “Apache Lucene 2.9: Discover the Powerful New Features”

Solr 1.4 Available on Some Mirrors Already

Mark Miller — Tue, 10 Nov 2009 01:26:49 +0000

Solr 1.4 won’t officially be released until tomorrow when the announcement goes out, but the official dist has already found its way onto some of the mirrors. Try your luck if your antsy

Lucene 2.9.1 about to be released

Mark Miller — Sun, 25 Oct 2009 16:32:15 +0000

Lucene 2.9.1 should hit the streets this week. 2.9.1 will be a bug fix release and includes a number of important fixes to the Lucene 2.9 release. You can see a list of the current 2.9.1 issues in JIRA here: JIRA 2.9.1 Issues. A couple of these bugs are quite nasty, so this is a highly recommended upgrade.

Why Lucene 2.9.1 rather than simply releasing the almost finished Lucene 3.0 with these bug fixes? After all, 3.0 is simply Lucene 2.9 with deprecations removed and tons of code converted to Java 1.5 (generics and what not). [though there is currently a bit of debate about adding a couple features to 3.0 as well] Well, there are two reasons. One: 2.9 was the last Java 1.4 release, and so anyone stuck on that platform would be without these fixes if we only put them in 3.0. Thats just about enough reason right there. Also though, 2: many users will not have completely moved off the deprecated API’s yet – which you will be required to do to move to 3.0. These “lazy” users will want to get these bug fixes too – without the requirement that they first finish moving to all of the new APIs. Hence Lucene 2.9.1 – coming any day now.

Lucene 2.9 is released

Mark Miller — Fri, 25 Sep 2009 16:50:51 +0000

Hello Lucene users,

On behalf of the Lucene dev community (a growing community far larger

than just the committers) I would like to announce the release of

Lucene 2.9.

While we generally try and maintain full backwards compatibility

between major versions, Lucene 2.9 has a variety of breaks that are

spelled out in the ‘Changes in backwards compatibility policy’ section

of CHANGES.txt.

We recommend that you recompile your application with Lucene 2.9

rather than attempting to “drop” it in. This will alert you to any

issues you may have to fix if you are affected by one of the backward

compatibility breaks. As always, its a really good idea to thoroughly

read CHANGES.txt before upgrading.

Lucene 2.9 comes with a bevy of new features, including:

* Per segment searching and caching (can lead to much faster reopen

among other things)

* Near real-time search capabilities added to IndexWriter

* New Query types

* Smarter, more scalable multi-term queries (wildcard, range, etc)

* A freshly optimized Collector/Scorer API

* Improved Unicode support and the addition of Collation contrib

* A new Attribute based TokenStream API

* A new QueryParser framework in contrib with a core QueryParser

replacement impl included.

* Scoring is now optional when sorting by Field, or using a custom

Collector, gaining sizable performance when scores are not

required.

* New analyzers (PersianAnalyzer, ArabicAnalyzer,

SmartChineseAnalyzer)

* New fast-vector-highlighter for large documents

* Lucene now includes high-performance handling of numeric fields.

Such fields are indexed with a trie structure, enabling simple to

use and much faster numeric range searching without having to

externally pre-process numeric values into textual values.

LucidWorks for Lucene 2.9

The certified distribution for Lucene 2.9 will be available in the coming weeks. Click on the link below to be notified when it is available

http://forms.lucidimagination.com/go/lucidimagination/notifyme

—

And many, many more features, bug fixes, optimizations, and various

improvements. You can find the full list of changes here:

http://lucene.apache.org/java/2_9_0/changes/Changes.html

Many changes have also occurred in Lucene’s Contrib area:

http://lucene.apache.org/java/2_9_0/changes/Contrib-Changes.html

Binary and source distributions are available at

http://www.apache.org/dyn/closer.cgi/lucene/java/

Lucene artifacts are also available in the Maven2 repository at

http://repo1.maven.org/maven2/org/apache/lucene/

The Next Release:

The next release will be Lucene 3.0. This should come along shortly,

and will remove all of the deprecated code in Lucene 2.9. Lucene 3.0 will also

be the first release to move from Java 1.4 to Java 1.5 as a requirement.

Thanks,

Mark Miller

Contrived FieldCache Load Test: Lucene 2.4 VS Lucene 2.9

Mark Miller — Tue, 22 Sep 2009 20:07:35 +0000

*edit* Sorry – jumped the gun with my original test code here – need to close the IndexWriter after the optimize! The gains are only with multi segment indexes. Corrected entry follows:

Lets do a little test. We will load up a FieldCache with 5,000,000 unique strings and see how long it takes Lucene 2.4 in comparison to Lucene 2.9.

Lets use my quad core laptop and the following test code:

public class ContrivedFCTest extends TestCase {
  public void testLoadTime() throws Exception {
    Directory dir = FSDirectory.getDirectory(System.getProperty("java.io.tmpdir") + File.separator + "test");
    IndexWriter writer = new IndexWriter (dir, new SimpleAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
    writer.setMergeFactor(37);
    writer.setUseCompoundFile(false);
    for(int i = 0; i &lt; 5000000; i++) {
      Document doc = new Document();
      doc.add (new Field ("field",  "String" + i, Field.Store.NO, Field.Index.NOT_ANALYZED));
      writer.addDocument(doc);
    }
    writer.close();
 
    IndexReader reader = IndexReader.open(dir);
    long start = System.currentTimeMillis();
    FieldCache.DEFAULT.getStrings(reader, "field");
    long end = System.currentTimeMillis();
    System.out.println("load time:" + (end - start)/1000.0f + "s");
  }
}

The results?

Lucene 2.4: 150.726s
Lucene 2.9: 9.695s

We discovered early this year that in the past, Lucene has been terribly inefficient when loading FieldCaches over multiple segments. Lucene 2.9 addresses this at the MultiReader level (thank you Yonik!). Also, internal FieldCache usage is now per segment, which sidesteps loading FieldCaches over mutiple segments all together – each segment has its own FieldCache.

Lucene 2.9 Release Vote Has Begun

Mark Miller — Mon, 21 Sep 2009 22:51:01 +0000

It took a couple more RC’s than I guessed (5 total), but the final vote candidate is up, and unless something critical is found during the 3 day vote process, Lucene 2.9, almost a year in the making, will be available by the end of the week.

http://search.lucidimagination.com/search/document/f15d32710b70ca6b/vote_release_lucene_2_9_0