Session topics -

• “Using Mahout to cluster, classify and recommend, plus a demonstration of using scripts packaged with Mahout” by Grant Ingersoll from Lucid Imagination.
• “How using random projection in Machine learning can benefit performance with out sacrificing quality” by Ted Dunning from MapR Technologies.

For those who missed this event the session slides and videos are now available. If you are interested in attending the next meet up session, we have it listed here,

I’ll be speaking at the upcoming Rich Web Experience conference in Ft. Lauderdale, presenting an “Introduction to Solr”, “Solr Recipes”, and “Lucene for Solr Developers”.  I’ll be tying all of these presentations together into a cohesive search/Solr track going from the introduction, to recipes for common tasks, through advanced customization of Solr.

During the session, moderator and former “Chump” Grant Ingersoll will present me with tough Lucene/Solr questions submitted by users, to see what kind of solutions I can come up with on the spot. A panel of judges will award prizes for questions that “stump” me the most….

• 1st Prize: $100 (USD) gift certificate or cash prize
• 2nd Prize: $50 (USD) gift certificate or cash prize
• 3rd Prize: $25 (USD) gift certificate or cash prize

So if you’ve got a tough Solr problem — send a description to “stump@lucene-eurocon.org” or comment on the Stump The Chump page of the Eurocon website. Even if you’ve already solved the problem yourself: submit it (along with your solution) to see if I can solve it on the spot.

If you can’t make it to Barcelona you can still participate: submit your question (along with a method to contact you) and you can watch me squirm later when we post video of the session on online.

PS: If you are coming to Eurocon, don’t forget to subscribe to the community RSS feed so you can stay informed on all the conference updates.

Solr makes using Lucene easier – you can index a variety of data sources easily, pretty much out of the box, and you can easily integrate features such as faceting, highlighting, and spellchecking – all without writing Java code. And if that’s all you need and it works solidly for you, awesome! You can stop reading now and attend one of our other excellent training courses that fit your needs. But if you are a tinkerer and want to know what makes Solr shine, or if you need some new or improved feature read on…

Deeper down, Lucene is cranking – analyzing, buffering, and indexing your documents, merging segments, parsing queries, caching data structures, rapidly hopping around an inverted index, computing scores, navigating finite state machines, and much more.

So how do you go about learning Lucene deeper? You can start with our “Understanding Lucene” DZone refcard.  And let’s not forget Lucene in Action, as it’s the most polished, detailed, and well crafted documentation available on the Lucene library. And of course there’s the incredibly vibrant and helpful Lucene open source community. Those resources will serve you well, but there’s no substitute for live, interactive, personal training to get you up to speed fast with best practices.

I’m in the process of overhauling our Lucene training course, that I’ll personally be delivering at Lucene EuroCon 2011 in Barcelona next month. This new and improved course takes an activity-based approach to learning and using Lucene’s API, beginning with the common tasks in building solutions using Lucene, whether you’re building directly to Lucene’s API or you’re writing custom components for Solr.

One area that I’m particularly jazzed about teaching is “query parsing”, the process of taking a user (or machine’s) search request and turning it into the appropriate underlying Lucene Query object instance.  Many folks developing with Lucene are familiar with Lucene’s QueryParser.  But did you know there are a couple of other query parsers with special powers?  There’s the surround query parser, enabling sophisticated proximity SpanQuery clauses.  And there’s the mysterious “XML query parser” (don’t let the ugly sounding name dissuade you) that slots dynamic query parameters, such as coming from an “advanced search” request, into a tree structured query template.   There’s some more insight into the world of Lucene query parsers an “Exploring Query Parsers” blog post.

What about all the Lucene contrib modules activity in the Lucene 3.x releases?   Here’s a bit of the goodnesses: better Unicode handling with the ICU tokenizers and filters, improved stemming, and many other analysis improvements, field grouping/collapsing, and block join/query for handling particular parent/child relationships.

Come learn the latest about the amazing Lucene library at Lucene EuroCon!  You, your boss, and your projects will all be glad you did.

Looking forward to seeing you there. Everyone in attendance will not only get to hear about the latest greatest enhancements to these technologies, but may also walk away with a cool Lucid t-shirt!

This time around, the data set I chose was Data.gov’s catalog of datasets, which fit with the ApacheCon open source aura, and Lucid Imagination’s support of Open Source for America, which helps to advocate for open source in the US Federal Government.  The prototype built includes faceting browsing, query term suggest, hit highlighting, result clustering, spell checking, document detail, and a bonus Venn diagram visualization.

The prototype was built with these steps:

1. Install LucidWorks for Solr
2. Grab the Data.gov catalog CSV file
3. Iterate a bit with Solr’s CSV update handler (the funnest way to get data into Solr) and a little Solr schema tinkering
4. Adjusted the Solr configuration and UI templates to get a nice look and feel, adding in a document detail page and a Venn diagram visualization comparing query overlaps

Voilà (click the images for large view):

This isn’t the first time we’ve toyed with Data.gov data… earlier this year, Hoss demonstrated Solr’s stats component on another of Data.gov’s data sets.

My ApacheCon slides are published at Slideshare and embedded here:

All the code and instructions for running the entire prototype yourself can be found here: https://github.com/erikhatcher/solr-rapid-prototyping/tree/master/ApacheCon2010

First off, hats off to Lulu for the excellent hosting, coordination and marketing of the event.  You could definitely see the evidence of Lulu’s “Be Remarkable” philosophy in the event. I’d say we had roughly 30-40 people for the first time event, with a good mix of developers, technical managers and a few recruiters.  There was even a “competitor” from an unnamed proprietary vendor present.  On the application front, there was a large mix of usages represented: ecommerce, publishing, video search, procurement, biopharma, etc.

After some socialization, we kicked off the night with Lulu CEO Bob Young, who gave a short intro to Lulu as well as a warm welcome to all.  Next up, I gave a talk (slides) on what’s coming in Lucene/Solr 3.x and beyond as well as answered some questions about features and functionality.  After me, Tarun Jain of The ABB Group, one of Lucid’s first customers and the world’s largest producer of industrial robots as well as a global leader in power and industrial automation with revenues around $33B USD, gave a presentation titled “Extreme Faceting Using Solr” (slides) on their move from a legacy proprietary vendor to Solr for searching all of their customer facing (and internal) product catalog (420K SKUs with 20+ million attributes and over 6M hits per month).   After setting the stage about the content to be searched and faceted, Tarun detailed how they went from wanting to “do everything in the DB” to doing nearly everything in Solr because it was that easy.  Moreover, slide 8 details the comparison they did between Solr and a very large proprietary search vendor (one of the so called top 3).  Here are the bullet points:

1. Stress test results in Proof of concept
   1. SOLR 35 req/sec vs 2 req/sec
   2. Average response times 200 ms vs 1-7 secs
   3. CPU usage 2-3% vs 100%
2. Sadly matchup was not even close (at least for the scenarios we tested for)
3. Conclusion .. Performance of SOLR is inversely proportional to the cost
4. Winner – SOLR by a KO

After Tarun’s talk, Paul Oakes from Lulu gave an excellent technical presentation (slides) on implementing auto-suggest in Solr using jQuery.  Just for grins, he also showed how trivial it was to add Google’s much hyped “Instant” search capability to Solr as well simply by making an extra jQuery call.  Naturally, the real work behind “Instant” is in capacity planning at scale, not in the programming of a few lines of Javascript.

As for the RTP meetup in general, I would suspect we will try to meet once a quarter, but maybe more often if the group so desires.

All in all, an excellent night, in my opinion.  Best of all it was a “home” event for me, so I didn’t have to fly anywhere or bum a ride back to a hotel!

The Bad News is that in spite of my best efforts, I won’t be sitting in a Dunk Tank during the session. Evidently, the hotel isn’t keen on the idea of having a giant sloshing water tank in one of their conference rooms.

The Good News is that there will be some great prizes for the top three people who “Stump The Chump”

• 1st: $100 Zappos Gift Card
• 2nd: $50 Gift Card
• 3rd: Your choice, a copy of “Taming Text” or “Lucene in Action“

The prizes will be awarded at the discretion of the judges, based on their combined (subjective) opinions about which questions they think stumped me the most.

It’s not to late to submit your questions!

If you’ve got a tough Solr problem, send a description to “stump@lucenerevolution.org” or reply to this crowdvine post by Mark Miller (our Head Judge). Even if you’ve already solved the problem yourself: submit it to see if I can come up with a solution on the spot. Even if you can’t make it to the conference: submit your question anyway and you can watch me squirm later when we post video of the session on online.

 More than just theory and academic hypothesis, Lucene Revolution will leave you with actionable information that you can take back to your organization to immediately hone competitive advantage and reap the cost and crowd-sourced innovation benefits of open source enterprise search. Expand your network, your knowledge, and your horizons all at one event.

Here’s  what attendees of Apache Lucene Eurocon, Lucene Revolution’s European predecessor, had to say about that groundbreaking event.

“My favorite part was hearing from the great range of use cases and learning from them”

“For me it was very helpful and valuable because I got a lot of inputs which was useful as we want to change from FAST ESP to Solr”

“Great speakers, excellent topics and talks”

“Very well planned and packed with many interesting talks and topics”

“ …Got a deeper view of Solr”

“I enjoyed learning new information about Solr – and seeing how other people are using it”

“Excellent conference”

“Meeting people struggling with the same issues was very valuable”

“Very interesting and focused talks”

“Well-organized conference”

“I liked putting a face to a name (from people in the Lucene/Solr community) – and the hallway chats”

“I’m looking forward to the next one!”

We’ve planned Lucene Revolution to offer these same benefits – and more.

 Join speakers from organizations like Twitter, LinkedIn, IBM, Cisco, Salesforce.com, eHarmony, Yale University, and Sears – and attendees from companies like Nokia, Monster.com, Verizon, Novell, SAIC, GE, Lockheed Martin, Comcast, Qualcomm, The Ladders, MTV, and MIT.

We hope to see you in Boston!

• Company A buys a software license from Company B, a startup.
• Company A crosses its fingers that Company B doesn’t go bankrupt and disappear, along with the source code for Company A’s mission-critical software.
• Company B goes kaput.
• Company A is left with some machine-readable binary code that it is powerless to develop or use.

Source code escrow has changed the outcome of this sticky situation for the better, and here’s how: Countless software companies go out of business every year, and either their code disappears entirely or goes to another company that doesn’t do any development or maintenance on it. The concept of escrow is one way in which open source gives companies a chance to continue their contribution and innovation, because the code they wrote can outlive them and continue to be evolved by the community.  I covered this topic in my most recent post on the Network World open source subnet. I invite your feedback: what’s your experience with source code or open source escrow? Any best practices or cautionary tales to share? Looking forward to hearing from you.