Enterprise Search

Lucene: It's coming from inside the firewall!

We've done a number of projects helping large clients with search roadmap planning, including an audit of  their various data sources. This is often an early step in implementing an enterprise search solution that will integrate diverse content across multiple sources.

On a number of our recent projects, an interesting thing has happened. As we've spoken with content owners, we've found an increasing number of Lucene implementations that no one knew about. This has often been a surprise to the people who brought us in, usually corporate IT.  Much like PCs infiltrated into corporations in the early days, it looks like Lucene is making its way into companies under the radar, often hacked in by a creative employee who just wants to get a simple search capability working, and who doesn't have time for a formal selection process or budget to purchase a commercial solution.

As we're written before, Lucene/Solr is getting to be a pretty decent search solution, although it's still a bit rough round the edges. This can't be a good sign for companies that market premium-priced search products.

Consider:

• IBM offers the free IBM/Yahoo! search for up to 500,000 documents
• Microsoft offers free Search Server Express as well as a higher-capacity Search Server
• Google Site Search and Google Custom Search are free and low-cost hosted solutions that provide search to your site - or a group of sites - and not spend much money

Finally, as Microsoft subsidiary FAST moves into the mid-range price sector with high end capabilities, the price of enterprise search is dropping for many companies that might have had to license six-figure deals for licensing alone. Add to this the free and low costs supporting technologies - consider clustering engine Carrot^2, for example - and you've got a movement.

To paraphrase our long-time friend Deep Search, you can spend a bunch of money on a commercial search, then spend much more implementing it well; or you can find a free or low-cost solution and spend a bunch of money implementing it. Your call.

Lucene/Solr Meet-Up in New York City 7/22

MTV is hosting a New York City Lucene/Solr meet up featuring the guys from Lucid Imagination.
This seems to be similar to the meet-up in San Francisco last month which we've written about in the blog. Unique to the New York event will be a talk by MTV's Michael Rosencrantz will talk about how they are using Solr and the benefits they are seeing.

Visit the meet-up page to learn more and to sign up for the event. If you want to go, register now because the number of available seats is dropping fast.

We've used Solr and Lucene on a number of large projects, and it's certainly become a darned good search engine. In fact, the internal structures of the 1.4 release are almost identical to the internal structure of Verity back in its early days. The catch is that the project technology is really enterprise scale; the packaging still leaves something to be desired to really succeed in the corporate environment. It's getting to be better and better all the time, though, so stay tuned.

We found the event really informative, although the crowd was not a typical corporate IT gathering. If you want to learn about what's new in Solr 1.4, and how a large media company uses open source search to their advantage, head for MTV. You'll also be able to brag about when you "were down at MTV..."

Is everyone running behind?

Charlie Hull of Lemur Consulting Ltd recently suggested on the LinkedIn Enterprise Search Forum that we all post a calendar of upcoming search events, which strikes me as a good idea (you'll need to log in to see the thread). The excellent shows and conferences are few and far between.

I've always thought the best search show is the Enterprise Search Summit - East run every May in New York by Information Today. Historically it's always been a much more useful conference than their ESS-West in San Jose every November, and far more relevant than the KMWorld conferences they co-locate with ESS in both cities. Search Engine Meeting, run by Infonortics in the Spring in Boston, is another good show, although it's much more appropriate for 'search geeks' who are interested in the bleeding edge of research.

It's been a tough year for shows everywhere though, with attendence way off from previous years. Let's hope next year is a better one, that more people can travel to conferences, and that the shows can rebound.

An odd note: I just looked for ESS East 2010 on the web and it's not been announced yet. Usually they have information on the venue and a call for papers (and for sponsor and exhibitors) by now. Maybe they realize that everyone knows it'll be at the Hilton, and the spring-like weather in New York has them all running behind!

SharePoint = Search?

Recently, Microsoft announced the availability of Exam 74-676, Enterprise Search Platform Developing - the first Microsoft exam for the ESP product acquired along with FAST Search last year. That's good news for those of us who work with enterprise-class search technology and want to establish a Microsoft competency as part of the extensive Microsoft partner program skill set. Well.. sort of.

First, Microsoft considers this exam one on Microsoft Office SharePoint Server 2007. Granted there is a special price on 'ESP for SharePoint'; but ESP has been a pretty powerful technology all on its own for a number of years. And I can assure you no SharePoint knowledge is required for this exam.

Secondly, it turns out that this exam may not count as a 'Search Competency" in the grand scheme of things. If you check the exams which qualify for the Search Competency (itself part of the Information Worker Solutions), you'll see that Exam 74-676 is not included on the list of qualifying exams. What tests do qualify? SharePoint and SharePoint Services application development and configuring.

Finally, I'm worried about customers who need an enterprise class search solution like FAST ESP and who hire a Microsoft partner whose qualifications include a Microsoft Search competency. They're going to find themselves knee-deep in a mess, which can't be good for Microsoft, FAST, or for anyone else in the business.

I know it sometimes takes a while for large companies to synchronize all of their web content, and I know the role of search is evolving at Microsoft. And I know SharePoint is an important part of the overall Microsoft Search Strategy going forward. But it would have been a nice touch if, after a $1.2B acquisition, the folks who really know FAST ESP could qualify for search competency without having to learn SharePoint.

I'd expect some changes on all of this in the coming months. But it seems a shame to have lost this opportunity to confirm that Microsoft understands the importance of enterprise search even when there's no SharePoint involved.