Unfortunately, Java 6′s only included library is Rhino that converts the javascript code into JVM code and its performance is not good. For reasons like these, the scripting languages in general are experiencing something that Java felt itself in the past: the popular belief that they are slow.

The performance is certaintly not that bad but in a critical application, people would prefer to develop native (java) components to keep the performance before losing performance with a probably unnecesary script. This entry is about performance of scripting languages; in particular, other Javascript engines that you can attach to Java.

Google chrome has surprised everyone with their blazing fast javascript engine: V8.

I downloaded V8 and tested it against the regular Rhino option. Actually I didn’t implement the neccesary wrappers to add V8 to Java, nor the JNI C programming necessary as a glue to access V8 functions (which are native in the traditional sense, real machine code running on real processors, like in the old days). I used a library that I found in internet: http://code.google.com/p/jav8/.

For the first test I downloaded a benchmark from V8 that computes an RSA encryption of a text using Javascript. I know, you will feel that I’m cheating by using a benchmark that is not impartial, but the results are pretty conclusive anyway: http://v8.googlecode.com/svn/data/benchmarks/.

The code (download here) also shows the basic usage of the scripting functionality of Java.

public class main {

	public static void main(String[] args) throws FileNotFoundException {

		ScriptEngineManager sm = new ScriptEngineManager();
		FileReader file = new FileReader("test.js");
		ScriptEngine jsEngine = sm.getEngineByName("jav8");

		int iter = Integer.parseInt(args[0]);

		try {
			long acum = 0;
			for(int i=0; i<iter; i++){
				long start = System.currentTimeMillis();
				Object ob = jsEngine.eval(file);
				long end = System.currentTimeMillis();

				acum += end - start;
			}
			System.out.println(acum);
		} catch (ScriptException ex) {
			ex.printStackTrace();
		}
	}
}

I created a simple bash script to run this many times with different number of iterations and these are the results (X has the iterations and Y has the time, so less is better)

The performance difference is outstanding. Particularly interesting is the scalability of V8 when we add more iterations. V8 has an advanced cache system an surely that is helping to keep the performance as the iteration grows.

A real application that performs a set of repetitive tasks is the ScriptTransformer of the Data Import Handler of Solr. This transformer applies a function written in Javascript to each row of a table. This infamous component is something very useful but its performance has always been horrible.

To continue the tests I compared the standard Rhino script engine vs V8 applied to the ScriptTransformer of Solr. I had to modify the ScriptTransformer and remove its “reflection style” implementation (apparently to keep compatibility with 1.5, but quite ugly anyway) to make jav8 work with it. The test consisted in encrypting 5000 records of text from a database. (modified ScriptTransformer.java)

The results:

Again the results show that V8 wins by a big margin.

The conclusion that we extract here is that adding script engines to our systems does not imply that the performance will be damaged. If you accept the use of native libraries (V8 in this case) a script engine can make your systems much more flexible without slowing them down.

Several Solr-powered open source faceted navigation search systems for libraries have been built with various technologies:  Blacklight (Ruby on Rails), VUFind (PHP), Kochief (Django), MULtifacet (Drupal). The question is, how general purpose are these user interfaces for non-library uses?  In theory they could all be purposed in this way, as every library really has a need to customize the UI.  Blacklight, for example, is written up in the Solr 1.4 book (by Smiley and Pugh) with a showcase that works on their MusicBrainz example.

The tough part of generalizing a search UI is that what we all really want is a custom-for-us UI, one that is as flexible as our imagination.  And it must fit pragmatically into the technology constraints of our operation.  For some, Ruby on Rails is the ONLY way to go, for others a Java-based UI tier is the only technology that fits.

Here are some pointers to various other UI technologies on top of Solr:

• Solritas – Apache Velocity templating.  Available, with some config, in Solr 1.4.
• Solr Flare - a proof of concept RoR UI plugin, does Ajax suggest, faceted navigation, saved (session-based) searches, and more.
• AJAX Solr – JavaScript, purely client-side widgets

These are covered a bit with screenshots in my EdUI presentation “Solr Flair: Search User Interfaces Powered by Apache Solr”.

What other open source UI frameworks live on top of Solr?  Add a comment with a pointer!

VUFind and Blacklight both share a common indexer, SolrMarc.  SolrMarc provides a flexible, extensible tool for mapping the complex standard library MARC format into Solr documents.

Recently it was reported that that SolrMarc indexing performance needed help (Stanford reported 12 hours to index 6M records).  I couldn’t help but want to help.  So I grabbed the latest SolrMarc (version 2.1, in development), a publicly available MARC file containing 5.7M records, and gave it a try.  First I ran SolrMarc against the file, and I killed the job after 9 hours.  Rather than looking too deep into the code to see what might be wrong, I decided to get a baseline on how fast indexing MARC could be using the simplest thing that could possibly work.  I created a custom MarcEntityProcessor, a hook into Solr’s DataImportHandler.  Using the MARC4J library directly, only indexing the id and a toString() of the entire MARC record, I was able to index the same dataset in 55 minutes.  It went from 22 docs/s to 1,745 docs/s! To be fair, my implementation didn’t do the fancy mappings needed in the real world, and there is still a bit of work to do in order to fully flesh out a DataImportHandler refactoring, but hopefully this new approach will be embraced by the library Solr community.

This was a long-winded way of saying… I’m devoting a chunk of my next couple of months to the needs of the Solr using library community.  My favorite conference of all time is coming up, code4lib conference, and I’m getting ramped up.   Naomi Dushay (of Stanford) and I are leading a Solr Blackbelt preconference where we’ll be going through heavy topics like query parsing and improving relevancy.

Stay tuned for lots more light being shed on Solr in the Library!