How, then, can you know how much you know about search with Solr and Lucene? In the world of Apache open source, there’s a clear meritocracy of peer review: contributors, committers, and active membership in the PMC. In theory, it’s a distinction anyone of sufficient talent and single-minded focus can achieve — just like anyone of sufficient talent and single-minded focus can make it to the NBA, or win the Nobel prize, or join the New York Philharmonic.

So you probably know your stuff if you’ve won the Nobel prize, made the NBA, or played the solo for Mozart’s Clarinet Concerto at Carnegie Hall, or you’re a Lucene/Solr contributor-or-committer. But what if you have not done any of those things, how do you know you know? Equally important, how do your peers or potential employers know how well you know your open source search stuff?

While there are more professional basketball players than Lucene/Solr committers, there are many, many more capable, talented, experienced Solr/Lucene application developers who are not going to ‘go pro’ in the Apache meritocracy. And the demand for Solr application development skills is exploding as interest and uptake of the leading open source application development technology spread like wildfire through organizations large and small. (Lucene Revolution, May 25-26 in San Francisco, will be packed with these people — sign up today if you haven’t already. And read on for another special  opportunity at Lucene Revolution).

It’s exactly for the broad base of interested, committed search application developers that today we’re introducing the Lucid Certified Apache Solr/Lucene Developer Program; a certification exam designed to benchmark development skills and experience in building applications with Apache Solr.

Designed with Prometric and a team of subject matter experts comprised of Apache Lucene/Solr committers, developers, and trainers, the test is designed to rigorously assess a broad base of search skills and experience, and provide the closest reasonable approximation possible to a standard measure of  search skills and experience.  It’s delivered via Prometric.com, consists of multiple choice questions, and costs $250. The test reflects a carefully selected, broad range of topics intended to reflect the real-world challenges and landscape of search application development problems, which you can see here.

Eric Pugh, who wrote the book on Solr, says this:

“I expect that the Lucid Imagination certification will quickly become the gold standard benchmark for whether someone who claims Solr and Lucene expertise truly possesses it. Oftentimes, a buyer of services has to take the leap of faith from sales pitch to execution that the knowledge is truly there. This certification can show, without a doubt, that the holder truly has the knowledge required to deliver a successful Solr/Lucene implementation. In the open source world, there are very few marks of authenticity: committer status, published author, and now the Lucid Solr certification. Just as the CPA certification shows a high level of knowledge and ability in the accounting industry, the Lucid Imagination Solr certification demonstrates unquestionable knowledge and experience in successful Solr/Lucene search engine implementation.”

It’s important to be clear about what the certification is not:

• It’s not easy: don’t expect to take your first Solr course one day and pass the exam the next.
• It’s not a substitute for experience: if you’ve only built one Solr application, earlier this morning, using the wiki demo that runs locally in your browser, you won’t pass.
• It’s not a substitute for training: taking a class from an expert may not be sufficient, but it will really help (and we offer the most professional-grade courses available; did I mention Lucene Revolution has classes available, too?)
• It’s not a casual conceptual overview: expect to answer detailed questions on everything from Lucene fundamentals to Solr debug output.
• It’s not a simple checklist of facts: you’ll have to demonstrate judgement calls in identifying correct answers to topic areas tied to searching, indexing, deployment, data source types, etc.

Testing as a pedagogical method — a mechanism for driving learning — is not the be-all-end-all of education (you probably didn’t think highly  of classmates who asked the teacher, “Will this be on the test?”). But it turns out that tests can have a salutary impact on acquiring and retaining knowledge, according to a recent article in Science.

We expect that this test will help level the playing field for a broad range of application developers to acquire and prove their Solr/Lucene application development skills — and help employers who want to take full advantage of the best open-source search technology on the planet find the men and women who have the stuff to do it.

If you’re coming to Lucene Revolution, the exam will be offered there for free — a savings of $250 over the regular price. Details are here.

Our main objective with the 1.7 release is to provide you with improved enterprise readiness as well as core search enhancements, in particular Search enhancements from Solr 4.x trunk

The rapid rate of innovation in the Solr/Lucene development community is a double-edged sword. The Apache Solr/Lucene 3.1 release just came out last week. However there are several bug fixes, performance improvements, and features already contributed by the Solr community in 4.x trunk. It is hard to keep up with the nightly builds to take advantage of the community contributions. Our objective at Lucid is to give you more frequent and certified versions of search functionality. LucidWorks Enterprise 1.7 contains a fully tested and certified version of Solr 4.x branch. You no longer have to try his at home, we do it for you. You can build your search applications leveraging some of the latest Solr search improvements with LucidWorks Enterprise.

Other highlights of this release include:

New in LucidWorks Enterprise 1.7

• Search result grouping / Field Collapsing improves user experience and simplifies development by grouping multiple similar results as a single entry ; for example as products within a price range, multiple emails in a single threaded conversation, parts within a category. Also provides the ability to group by query – retrieves the top documents that match the query, not just the count additional info  (http://www.lucidimagination.com/blog/2010/09/16/2446/)
• Integration with UIMA for metadata extraction improves versatility of metadata driven analytics and operations, such as using Pivot Faceting. It treats each facet as a dimension, creating facet counts for a multi-dimensional matrix. For example, for each category in a story inventory, it can show how many products are in stock within the facets controls, rather than just documents retrieved. (http://wiki.apache.org/solr/SolrUIMA)
• Sort result sets by functions, so you can tweak the search relevancy result set using functions to create relevancy algorithms within the result of a query, eliminating the need to write custom rankers for common scenarios. (http://yonik.wordpress.com/2011/03/10/solr-relevancy-function-queries/)
• Spell check against the existing index improves efficiency by saving the need to build a special dictionary for the spellcheck component, as it automatically checks entries without having to create explicit misspelled entries
• Numeric range facets (similar to date faceting) organize and analyze data and content based on quantitative parameters, such as size, dimensions, prices, etc.
• New spatial search, including spatial filtering, boosting and sorting capabilities better integrate  location-based information into search applications
• A new Auto Suggest component simplifies rapid retrieval of candidate results interactively as the user types
• Fine-grained control of data acquisition timing and sources, both for application development and ongoing data-management and indexing for production search applications, simplifying integrated crawling configuration and management
• SharePoint CMS Connector A new connector for Sharepoint, which lets you crawl and index the content of your Sharepoint server as easily as other conventional content management systems, web servers, and databases, right from the LucidWorks Enterprise console (for Microsoft Office SharePoint Server 2007, Microsoft Windows SharePoint Services 3.0, SharePoint 2010); it also seamlessly integrates sharepoint document and user level ACLs  directly from the LucidWorks Enterprise UI.
• Security Enhancements LucidWorks Enterprise 1.7 improves integration and validation of security into search application development and deployment, with streamlined control of user privileges, groups, and LDAP integration. LDAP configuration activation, configuration and validation can be done via UI, in addition to ReST API configuration to simplify development and streamline production integration.

LucidWorks Enterprise is free to download for unlimited development and test use; production deployment requires an enterprise subscription. Subscriptions include production deployment support, business hours or 24/7 SLA-based incident management support and ExpertLink on-demand consulting. The upgrade is free for current customers.

You can download the product here. For the full set of functionality refer the documentation here. Of course, we welcome your inputs — help us improve the product by providing your feedback and suggestions in our Forums.

Gee.  That almost sounds like a marketing guy wrote it.  Let me try again: LucidWorks Enterprise is software that let’s you easily build great search applications.  You can install it, index some content, and search that content with just a few keystrokes and clicks of the mouse.  Or, you can build a full-blown, integrated search application with custom plugins and your own user interface, all running on a fifty-node cluster.  And if you really need it, the full flexibility of Solr is right there, always accessible.

LucidWorks Enterprise extends Solr to include some enterprise-grade features, just as Solr extends Lucene to provide server functionality.  By extending and embedding Solr, we get to use all of its powerful features without having to do what many folks do: fork the code base.  You see, we are really embedding Solr without changing it appreciably.  Where we do change it, we contribute our code back to the open source project.  A great example of that is SolrCloud, a feature we developed specifically for LucidWorks Enterprise.  The problem was that it was such a fundamental change to Solr that we would have to maintain a forked code-base forever, or simply contribute the code to the open source project.  We chose the latter, and that’s the choice we will continue to prefer for such fundamental and powerful features.

So, what’s in LucidWorks Enterprise?  Here’s a quick summary of the main features:

SolrCloud:

provides simplified configuration management, load balancing and failover in a distributed environment.  It’s the key feature if you have a cluster of search nodes.

User Interface:

we added a web-driven user interface to simplify the process of configuring the system, indexing content, and trying out searches.

CLICK Scoring:

ever wonder how to get better results on searches over your own content?  The CLICK scoring framework provides feedback from searchers back into the relevance ranking of documents, improving relevance for all.

ReST API:

all of the power of LucidWorks Enterprise is available programmatically, so you can build more thorough applications and automated administration.

Smarter Defaults:

LucidWorks Enterprise is configured out-of-the-box to be a great search application, with good relevance and a full list of features like spell checking, autocomplete, and unsupervised feedback.

Data Acquisition:

we have added a crawler and scheduler that simplifies the process of getting data into LucidWorks Enterprise.  Just point it at a Web site or set of files, or import data from a database.

So that’s LucidWorks Enterprise.  We’ll be talking more about it over the coming days.  This is the beginning of an exciting time for Lucid Imagination: we’re really excited to raise the bar with search and get our customers using the latest and greatest in search technology: LucidWorks Enterprise. It’s free for developers, so download LucidWorks Enterprise now! ]]> http://www.lucidimagination.com/blog/2010/10/14/introducing-lucidworks-enterprise/feed/ 1 http://www.lucidimagination.com/blog/2010/09/16/2446/ http://www.lucidimagination.com/blog/2010/09/16/2446/#comments Fri, 17 Sep 2010 01:52:26 +0000 yonik http://www.lucidimagination.com/blog/?p=2446 Result Grouping, also called Field Collapsing, has been committed to Solr!
This functionality limits the number of documents for each “group”, usually defined by the unique values in a field (just like field faceting).

You can think of it like faceted search, except instead of just getting a count, you get the top documents for that constraint or category. There are tons of potential use cases:

• For web search, only show 1 or
…

 More than just theory and academic hypothesis, Lucene Revolution will leave you with actionable information that you can take back to your organization to immediately hone competitive advantage and reap the cost and crowd-sourced innovation benefits of open source enterprise search. Expand your network, your knowledge, and your horizons all at one event.

Here’s  what attendees of Apache Lucene Eurocon, Lucene Revolution’s European predecessor, had to say about that groundbreaking event.

“My favorite part was hearing from the great range of use cases and learning from them”

“For me it was very helpful and valuable because I got a lot of inputs which was useful as we want to change from FAST ESP to Solr”

“Great speakers, excellent topics and talks”

“Very well planned and packed with many interesting talks and topics”

“ …Got a deeper view of Solr”

“I enjoyed learning new information about Solr – and seeing how other people are using it”

“Excellent conference”

“Meeting people struggling with the same issues was very valuable”

“Very interesting and focused talks”

“Well-organized conference”

“I liked putting a face to a name (from people in the Lucene/Solr community) – and the hallway chats”

“I’m looking forward to the next one!”

We’ve planned Lucene Revolution to offer these same benefits – and more.

 Join speakers from organizations like Twitter, LinkedIn, IBM, Cisco, Salesforce.com, eHarmony, Yale University, and Sears – and attendees from companies like Nokia, Monster.com, Verizon, Novell, SAIC, GE, Lockheed Martin, Comcast, Qualcomm, The Ladders, MTV, and MIT.

We hope to see you in Boston!

I talk about my experience in Brazil further in my “Logical Expression” post this week.  It was great to learn that all the participants at the event had growth and expansion plans, and open source is being widely used and embraced, from Lucene/Solr to Linux and beyond. From conversations with the executives in attendance, I learned that their main motivations for using open source were quality, easy access, a short learning curve, and lower cost. I was left with a very positive impression and reminded yet again about the power of the open source community — in a country with a growing economy and a growing appetite for technology.

• Company A buys a software license from Company B, a startup.
• Company A crosses its fingers that Company B doesn’t go bankrupt and disappear, along with the source code for Company A’s mission-critical software.
• Company B goes kaput.
• Company A is left with some machine-readable binary code that it is powerless to develop or use.

Source code escrow has changed the outcome of this sticky situation for the better, and here’s how: Countless software companies go out of business every year, and either their code disappears entirely or goes to another company that doesn’t do any development or maintenance on it. The concept of escrow is one way in which open source gives companies a chance to continue their contribution and innovation, because the code they wrote can outlive them and continue to be evolved by the community.  I covered this topic in my most recent post on the Network World open source subnet. I invite your feedback: what’s your experience with source code or open source escrow? Any best practices or cautionary tales to share? Looking forward to hearing from you.

With this post, I have the pleasure to announce availability of New Relic RPM integrated into the Lucid Imagination performance portal.  This is great news for the people who care about Solr or Lucene performance.

What is RPM?

RPM is a SaaS performance monitoring and troubleshooting tool for Ruby and Java applications that features the simplest deployment available in the market, highly intuitive performance feedback, customizable dashboards, transaction breakdown, and debugging tools.

RPM is able to monitor any Java and Ruby applications.  Additionally, the RPM monitor will report specific Solr native metrics if installed in a Solr JVM: cache sizes, cache hits, Solr requests broken by request handler, cores and update operations. In fact, RPM is now monitoring our LucidFind search portal, at search.lucidimagination.com, as seen in the following screenshot. (I snapped it this weekend)

Starting today, the Lucene community can use their Lucid Imagination usernames to log in to RPM, manage their accounts and monitor Lucene, Solr and other Java, Ruby or JRuby applications

Check out our new performance portal at http://www.lucidimagination.com/performanceportal. Once you’ve created your account, download the monitoring package from the link in the upper right corner, and drop it on your JVM. You’ll have your first result within two minutes. That’s it!

How much does it cost?

RPM is available in four different account levels: Lite, Bronze, Silver, and Gold.

RPM Lite is the most widely used solution for basic web application monitoring. RPM Lite provides application monitoring for unlimited Java, Ruby or Jruby applications, for unlimited users, for an unlimited time.

Price: Free

Server limit: None!

Time limit: None!

RPM Bronze is a step beyond monitoring to help you with problem troubleshooting. RPM Bronze shows you the slowest, most time-consuming, highest throughput, and most CPU-intensive transaction types. RPM Bronze also shows background tasks and their effects on application performance, as well as database calls with the slowest response time, highest time consumption, and highest throughput. Performance data is retained for a week allowing day to day comparisons of key performance metrics. Includes all the features found in Lite.

Price: $75/mo, $50/mo on a yearly plan

RPM Silver takes you to the code level to solve performance problems. RPM Silver provides application monitoring and troubleshooting features found in Lite and Bronze. It also enables in-depth tracing of individual transactions, including background tasks—down to the SQL and Explain plan. RPM Silver also captures application errors your users see and identifies the root cause of the error. It provides details about each new deployment of each application including before and after comparisons. Performance data is retained for 30 days. RPM Silver is for any team serious about application performance.

Price: $150/mo, $100/mo on a yearly plan

RPM Gold is for application teams that want to move beyond firefighting into performance management and optimization. RPM Gold provides application monitoring, troubleshooting and diagnostic features found in other levels. RPM Gold also provides a series of analyses focused on understanding and shaping the application’s performance over time: Scalability Analysis, Capacity Analysis, Long-term SLA Trending, Database Analysis, Transaction Report, and Deployment History. Performance data is retained for 90 days. With RPM Gold you can see if all your development efforts are making meaningful improvements in application performance. Go beyond firefighting with RPM Gold.

Price: $200/mo, $150/mo on a yearly plan

And remember – the lite version is FREE for an UNLIMITED time with UNLIMITED servers!

Hosted by KMWorld and featuring speakers from The Motley Fool

Click here to view the webinar or download the slides

As today’s business landscape changes, more and more companies want control over their search to take on the rapid acceleration in scale and diversity of their data and documents. But legacy commercial search applications can add hidden costs and constraints, and make it difficult to innovate and adapt.

Now, thousands of organizations of all shapes and sizes are turning to Apache Lucene and Solr open source search, gaining competitive advantage by building Enterprise Search applications that deliver more speed, versatility, and more relevant results.

Finding the right information at the right time is critical to a winning investment strategy. Multimedia financial-services company The Motley Fool worked with Lucid Imagination to successfully migrate from a closed commercial solution to the versatile power of Apache Lucene and Solr open source search.

Join Motley Fool Director of Search Danny Hsia and Motley Fool Search VP Tech Operation Chad Wolfsheimer, along with Lucid Imagination’s Erik Hatcher, to hear how The Motley Fool took control of their search technology and migrated to Lucene/Solr Open Source Search – and dramatically improved search relevancy, speed, versatility and costs.

SPEAKERS:

Danny Hsia, Director of Search
The Motley Fool

Chad Wolfsheimer, VP, Tech Operations
The Motley Fool

Erik Hatcher, Co-Founder, Member Technical Staff
Lucid Imagination

Location: Marriott Oakland City Center, Rooms 1&2

For more information about the meetup, visit
http://wiki.apache.org/lucene-java/LuceneAtApacheConUs2009