For more details and to RSVP, please see http://sf-mahout-11-11.eventbrite.com/

Here’s the steps I took:

1. Download and “install” (aka unzip) Apache Solr 3.3.0
2. Launch Solr (cd example; java -jar start.jar)
3. Index files

That’s it.  Here’s the indexing script I used:

require 'net/http'
 
@dir = Dir.new("/Users/erikhatcher/apache-solr-3.3.0/docs")
 
@url = URI.parse("http://localhost:8983/solr")
@connection = Net::HTTP.new(@url.host, @url.port)
 
def index(filename)
@connection.get(@url.path + "/update/extract?stream.file=#{filename}&literal.id=#{filename}")
end
 
def commit
@connection.get(@url.path + "/update?commit=true")
end
 
@dir.each {|name|
  f = "#{@dir.path}/#{name}"
  if File.file?(f)
    puts "Indexing #{f}..."
    index(f)
  end
}
 
puts "Committing..."
commit
 
puts "Done!"

To make it look prettier, only a little dabbling with the templates is needed – add your company logo, customize the colors. And a change to the example (/browse handler) configuration to facet on content_type will allow you to easily search just within documents of specific types through the included UI.  The example code above indexed the docs that ship with Apache Solr 3.3.0; just change the path to a directory of yours to index your own content.

Looking forward to seeing you there. Everyone in attendance will not only get to hear about the latest greatest enhancements to these technologies, but may also walk away with a cool Lucid t-shirt!

Here are the slides for “Rapid Prototyping with Solr”:

My next talk, titled “Solr Recipes”, was a 3 hour workshop covering the common ways to get data into Solr, configure it, and leverage it within applications.

Here are the “Solr Recipes” slides:

And finally, to a few hardcore folks, I discussed “Lucene for Solr Developers”, which more broadly covered the various ways to extend Solr. One cool example (or so I think, at least) I built for this talk that I’ve put out there as food for thought is “auto-faceting”, having a way for Solr to determine the best facets to select for a given query. See SOLR-2641 for my initial proof-of-concept implementation.

Here are the “Lucene for Solr Developers” slides:

There was so much going on that I barely got to tap into the conference experience myself, with the best part being the conversations had during meals, and between and during the scheduled talks. I was able to reconnect with many long-time friends that I’ve made through No Fluff, Just Stuff, and made many new acquaintances – I won’t name names, as this list covers most of them.

Thank you, Überconf, Jay, and fellow speakers and attendees for a stellar technical event. If you missed it, don’t despair, No Fluff, Just Stuff brings many of the same speakers and topics to an area near you. I’ll be speaking at a few NFJS events during the second half of this year, including Raleigh, NC, Boston, MA, and, surely to rival the quality and size of Überconf, The Rich Web Experience in Ft. Lauderdale, FL.

Let’s talk.

How, then, can you know how much you know about search with Solr and Lucene? In the world of Apache open source, there’s a clear meritocracy of peer review: contributors, committers, and active membership in the PMC. In theory, it’s a distinction anyone of sufficient talent and single-minded focus can achieve — just like anyone of sufficient talent and single-minded focus can make it to the NBA, or win the Nobel prize, or join the New York Philharmonic.

So you probably know your stuff if you’ve won the Nobel prize, made the NBA, or played the solo for Mozart’s Clarinet Concerto at Carnegie Hall, or you’re a Lucene/Solr contributor-or-committer. But what if you have not done any of those things, how do you know you know? Equally important, how do your peers or potential employers know how well you know your open source search stuff?

While there are more professional basketball players than Lucene/Solr committers, there are many, many more capable, talented, experienced Solr/Lucene application developers who are not going to ‘go pro’ in the Apache meritocracy. And the demand for Solr application development skills is exploding as interest and uptake of the leading open source application development technology spread like wildfire through organizations large and small. (Lucene Revolution, May 25-26 in San Francisco, will be packed with these people — sign up today if you haven’t already. And read on for another special  opportunity at Lucene Revolution).

It’s exactly for the broad base of interested, committed search application developers that today we’re introducing the Lucid Certified Apache Solr/Lucene Developer Program; a certification exam designed to benchmark development skills and experience in building applications with Apache Solr.

Designed with Prometric and a team of subject matter experts comprised of Apache Lucene/Solr committers, developers, and trainers, the test is designed to rigorously assess a broad base of search skills and experience, and provide the closest reasonable approximation possible to a standard measure of  search skills and experience.  It’s delivered via Prometric.com, consists of multiple choice questions, and costs $250. The test reflects a carefully selected, broad range of topics intended to reflect the real-world challenges and landscape of search application development problems, which you can see here.

Eric Pugh, who wrote the book on Solr, says this:

“I expect that the Lucid Imagination certification will quickly become the gold standard benchmark for whether someone who claims Solr and Lucene expertise truly possesses it. Oftentimes, a buyer of services has to take the leap of faith from sales pitch to execution that the knowledge is truly there. This certification can show, without a doubt, that the holder truly has the knowledge required to deliver a successful Solr/Lucene implementation. In the open source world, there are very few marks of authenticity: committer status, published author, and now the Lucid Solr certification. Just as the CPA certification shows a high level of knowledge and ability in the accounting industry, the Lucid Imagination Solr certification demonstrates unquestionable knowledge and experience in successful Solr/Lucene search engine implementation.”

It’s important to be clear about what the certification is not:

• It’s not easy: don’t expect to take your first Solr course one day and pass the exam the next.
• It’s not a substitute for experience: if you’ve only built one Solr application, earlier this morning, using the wiki demo that runs locally in your browser, you won’t pass.
• It’s not a substitute for training: taking a class from an expert may not be sufficient, but it will really help (and we offer the most professional-grade courses available; did I mention Lucene Revolution has classes available, too?)
• It’s not a casual conceptual overview: expect to answer detailed questions on everything from Lucene fundamentals to Solr debug output.
• It’s not a simple checklist of facts: you’ll have to demonstrate judgement calls in identifying correct answers to topic areas tied to searching, indexing, deployment, data source types, etc.

Testing as a pedagogical method — a mechanism for driving learning — is not the be-all-end-all of education (you probably didn’t think highly  of classmates who asked the teacher, “Will this be on the test?”). But it turns out that tests can have a salutary impact on acquiring and retaining knowledge, according to a recent article in Science.

We expect that this test will help level the playing field for a broad range of application developers to acquire and prove their Solr/Lucene application development skills — and help employers who want to take full advantage of the best open-source search technology on the planet find the men and women who have the stuff to do it.

If you’re coming to Lucene Revolution, the exam will be offered there for free — a savings of $250 over the regular price. Details are here.

Before I dig in too much, though, a special thanks to all our customers at Lucid Imagination as well as to my coworkers.  I’m coming up on 15 years out in the “real world” and I can honestly say I’ve never enjoyed what I do as much as I do here and that even accounts for the normal rough patches one goes through in any job.  As an engineer, there are few things as cool as getting to work with customers who are not only using, but pushing your work/project/product on a daily basis to do new and interesting things (I think this is a direct result of the project being Open Source, which I believe has an inherently lower cost of experimentation).  I’ve been fortunate enough to meet and talk with many people doing all kinds of things with Lucene and Solr ranging from the “mundane” of basic keyword search to those building next generation search capabilities at incredible scale.  Through it all, I’m constantly amazed at the flexibility and efficiency of Lucene and Solr.  For instance, I’ve been working with one customer now whose Solr-based solution (for the exact same content) will use ~50% less hardware and will have an index that is 1/6 the size of their FAST index all while saving them major dinero.

Speaking of Lucid, one of the highlights of the year for us that relates directly to Lucene and Solr is the launch of our enterprise version: LucidWorks Enterprise.   I like to think of it as Apache Solr with a whole lot of Lucid expertise on how to use Solr baked in and topped off with other features and functionality to make building search applications easier.

OK, time to move on to the open source projects…

1. Without a doubt, the biggest news of the year is the merging of the Lucene and Solr code base as well as the “graduation” of several subprojects to Apache Soft. Foundation Top Level Projects (TLP).  The graduating projects are Tika, Nutch, and Mahout.  We also spun Lucy (a C port) to the Incubator, where it is working on it’s own community.  These moves were primarily done to focus the project management on single code base, but they also demonstrate the project has reached a level of maturity at the ASF.  The move also has the side benefit of bringing each project higher visibility.
2. I’m particularly excited about the addition of OpenNLP to the Apache umbrella.  OpenNLP is a nice open source Java project for natural language processing that has lived at Source Forge for quite some time.  I would expect development to grow quite a bit under the ASF community based model.  Also, integrating OpenNLP with Solr and Lucene is pretty easy to do.  I would be remiss if I didn’t also give a nod to the addition of the ManifoldCF project to the ASF.  ManifoldCF will help unlock content in Sharepoint, Documentum and other repositories for users of Lucene and Solr.
3. Lucene’s trunk code base now implements our “Flex APIs”, which should allow users to have near total control over what goes in the index as well as alternate compression techniques, different scoring models, etc.  See Michael McCandless’ excellent talk at Lucene Revolution for more details.
4. With all the location aware devices and capabilities on the market, geo-spatial search is a hot topic and Lucene and Solr have been adding quite a bit of capabilities in this regard with the ability to filter, boost and sort results based on location information in documents.  See Solr’s Spatial Search Wiki page for more info as well as several of my past blog posts.
5. Of course, everyone was a buzz about the cloud this year.  For Solr, this translates into greater efforts to make Solr easier to scale to very large installations (100s to 1000s of nodes and billions and billions of documents) via the Solr Cloud project that Yonik Seeley and Mark Miller have been spearheading.
6. On the user side, one of the biggest pieces of buzz this year related to Lucene was the migration of Twitter search to Lucene.  At 1 billion queries per day and 50 million posts per day (all indexed and searchable in near real time), Twitter’s search system certainly has it’s work cut out for itself.  However, as Michael Busch outlined at Lucene Revolution, Apache Lucene was up to the task!  Naturally, there were lots of other companies that migrated to Solr and Lucene as well.  Have you shared your use case?

Well, I’ve no doubt missed a bunch of other things, but those items, to me, are some of the bigger highlights.  Looking forward, there are some other exciting things coming to Lucene and Solr.  In particular, I’m working on adding language identification, related searches and point in polygon filtering to Solr.  I would also expect we will release Lucene/Solr 3.1 fairly soon, too, but you can’t pin me down on a date just yet.

Here’s hoping you all have a Happy Holidays and a Happy New Year!

Gee.  That almost sounds like a marketing guy wrote it.  Let me try again: LucidWorks Enterprise is software that let’s you easily build great search applications.  You can install it, index some content, and search that content with just a few keystrokes and clicks of the mouse.  Or, you can build a full-blown, integrated search application with custom plugins and your own user interface, all running on a fifty-node cluster.  And if you really need it, the full flexibility of Solr is right there, always accessible.

LucidWorks Enterprise extends Solr to include some enterprise-grade features, just as Solr extends Lucene to provide server functionality.  By extending and embedding Solr, we get to use all of its powerful features without having to do what many folks do: fork the code base.  You see, we are really embedding Solr without changing it appreciably.  Where we do change it, we contribute our code back to the open source project.  A great example of that is SolrCloud, a feature we developed specifically for LucidWorks Enterprise.  The problem was that it was such a fundamental change to Solr that we would have to maintain a forked code-base forever, or simply contribute the code to the open source project.  We chose the latter, and that’s the choice we will continue to prefer for such fundamental and powerful features.

So, what’s in LucidWorks Enterprise?  Here’s a quick summary of the main features:

SolrCloud:

provides simplified configuration management, load balancing and failover in a distributed environment.  It’s the key feature if you have a cluster of search nodes.

User Interface:

we added a web-driven user interface to simplify the process of configuring the system, indexing content, and trying out searches.

CLICK Scoring:

ever wonder how to get better results on searches over your own content?  The CLICK scoring framework provides feedback from searchers back into the relevance ranking of documents, improving relevance for all.

ReST API:

all of the power of LucidWorks Enterprise is available programmatically, so you can build more thorough applications and automated administration.

Smarter Defaults:

LucidWorks Enterprise is configured out-of-the-box to be a great search application, with good relevance and a full list of features like spell checking, autocomplete, and unsupervised feedback.

Data Acquisition:

we have added a crawler and scheduler that simplifies the process of getting data into LucidWorks Enterprise.  Just point it at a Web site or set of files, or import data from a database.

So that’s LucidWorks Enterprise.  We’ll be talking more about it over the coming days.  This is the beginning of an exciting time for Lucid Imagination: we’re really excited to raise the bar with search and get our customers using the latest and greatest in search technology: LucidWorks Enterprise. It’s free for developers, so download LucidWorks Enterprise now! ]]> http://www.lucidimagination.com/blog/2010/10/14/introducing-lucidworks-enterprise/feed/ 1 http://www.lucidimagination.com/blog/2010/09/29/summary-of-first-ever-rtp-raleighchapel-hilldurham-apache-lucenesolr-meetup/ http://www.lucidimagination.com/blog/2010/09/29/summary-of-first-ever-rtp-raleighchapel-hilldurham-apache-lucenesolr-meetup/#comments Wed, 29 Sep 2010 12:55:42 +0000 Grant Ingersoll http://www.lucidimagination.com/blog/?p=2499 A week and a day later, I’ve finally got a chance to put up my thoughts/notes on the first ever RTP Apache Lucene/Solr Meetup hosted by Lulu Press and co-sponsored by Lucid Imagination.

First off, hats off to Lulu for the excellent hosting, coordination and marketing of the event.  You could definitely see the evidence of Lulu’s “Be Remarkable” philosophy in the event. I’d say we had roughly 30-40 people for the first time event, …

 More than just theory and academic hypothesis, Lucene Revolution will leave you with actionable information that you can take back to your organization to immediately hone competitive advantage and reap the cost and crowd-sourced innovation benefits of open source enterprise search. Expand your network, your knowledge, and your horizons all at one event.

Here’s  what attendees of Apache Lucene Eurocon, Lucene Revolution’s European predecessor, had to say about that groundbreaking event.

“My favorite part was hearing from the great range of use cases and learning from them”

“For me it was very helpful and valuable because I got a lot of inputs which was useful as we want to change from FAST ESP to Solr”

“Great speakers, excellent topics and talks”

“Very well planned and packed with many interesting talks and topics”

“ …Got a deeper view of Solr”

“I enjoyed learning new information about Solr – and seeing how other people are using it”

“Excellent conference”

“Meeting people struggling with the same issues was very valuable”

“Very interesting and focused talks”

“Well-organized conference”

“I liked putting a face to a name (from people in the Lucene/Solr community) – and the hallway chats”

“I’m looking forward to the next one!”

We’ve planned Lucene Revolution to offer these same benefits – and more.

 Join speakers from organizations like Twitter, LinkedIn, IBM, Cisco, Salesforce.com, eHarmony, Yale University, and Sears – and attendees from companies like Nokia, Monster.com, Verizon, Novell, SAIC, GE, Lockheed Martin, Comcast, Qualcomm, The Ladders, MTV, and MIT.

We hope to see you in Boston!

I talk about my experience in Brazil further in my “Logical Expression” post this week.  It was great to learn that all the participants at the event had growth and expansion plans, and open source is being widely used and embraced, from Lucene/Solr to Linux and beyond. From conversations with the executives in attendance, I learned that their main motivations for using open source were quality, easy access, a short learning curve, and lower cost. I was left with a very positive impression and reminded yet again about the power of the open source community — in a country with a growing economy and a growing appetite for technology.