A paradigm shift in enterprise search

I've been involved in enterprise search since before the 'earthquake World Series' between the Giants and the A's in 1989. While our former company became part of LucidWorks last December, we still keep abreast of the market. But being a LucidWorks employee has brought me to a new realization: commercial enterprise search is pretty much dead.

Think back a few years: FAST ESP, Autonomy IDOL (including the then-recently acquired Verity), Exalead, and Endeca were the market. Now, every one of those companies has become part of a larger business. Some of the FAST technology lives on, buried in SharePoint 2013; Autonomy has suffered as part of HP because - well, because HP isn't what it was when Bill and Dave ran it. Current management doesn't know what they have in IDOL, and the awful deal they cut was probably based on optimistic sales numbers that may or may not have existed. Exalead, the engine I hoped would take the place of FAST ESP in the search market is now part of Dassault and is rarely heard of in search. And Endeca, the gem of a search platform optimized for the lucrative eCommerce market, has become one of three or four search-related companies in the Oracle stable. 

Microsoft is finally taking advantage of the technology acquired in the FAST acquisition for SharePoint 2013, but as long as it's tied to SharePoint - even with the ability to index external content - it's not going to be an enterprise-wide distribution - or a 'big data' solution. SharePoint Hadoop? Aslongf as you bring SQL Server. Mahout? Pig? I don't think so. There are too many companies that want or need Linux for their servers rather than Windows.

Then there is Google, the ultimate closed-box solution. As long as you use the Google search button/icon, users are happy – at least at first. If you have sixty guys named Sarah? Maybe not.

So what do we have? A few good options generally from small companies that tend to focus on hosted eCommerce - SLI Systems and Dieselpoint; and there’s Coveo, a strong Windows platform offering.

Solr is the enterprise search market now. My employer, LucidWorks, was the first, and remains the primary commercial driver to the open source Apache project. What's interesting is the number of commercial products based on Solr and it's underlying platform, Lucene.

Years ago, commercial search software was the 'safe choice'. Now I think things have changed: open source search is the safe choice for companies where search is mission. Do you agree?

I'll be writing more about why I believe this to be the case over the coming weeks and months: stay tuned.

/s/Miles

 

Comments

[Luis García Reyes]

Charlie you are right.

"Search is *hard* and no matter which license type you choose you're going to have to commit to a significant period of requirement analysis, development, integration and support from whoever you decide to work with."

The rest always is commercial bullsh**

In Martin words...

"You can tell a sales rep search his lips move Because When He talks"

That´s all.

//
Thanks Luis for your comments, always welcomed. I may have a word with Martin about the quote *I* used at ESS a few years back about sales reps - although great minds do sometimes create the same truism independently!
/s/Miles

[Charlie Hull]

It's certainly not a de-facto safe bet - and I write that as someone who's been consulting on open source search for over a decade. However, I don't think closed source search is a safe bet either! Search is *hard* and no matter which license type you choose you're going to have to commit to a significant period of requirement analysis, development, integration and support from whoever you decide to work with.

One problem we've seen is that many companies decide to try building something in-house with zero experience of search - of course this is much easier to get off the ground with open source. Thus we're seeing a lot of business either fixing or re-architecting search projects based on open source. In retrospect even half a day spent with our team on planning would have saved a lot of time.

Also, Miles, it's entirely possible to provide good service even if you don't have a product as such!
//
So true Charlie - there are many companies that build a good business on service - and many are excellent at it!
/s/Miles

[Martin White]

It may be but not yet. The main reason is that companies have no idea about what they need from a search application. For better or worse the pre-sales teams of commercial vendors arrive with checklists and return to the office with some ideas about what might work. Most of the open-source search developers I have come across do not have a background in business and requirements analysis because most of their work has been on web site search. Just getting a handle on legacy file formats in even a medium-sized operation is a challenge.

A further challenge is that there is no understanding of the requirements for a search support team, and this team could be even more important in open-source than in commercial just because there could be more options available to optimise the implementation and get the best possible return on the development investment.

So at present I don't think that open-source is de facto a safe bet, and in any case there are many flavours of open source search. I have no concerns about the stability and roadmap of the software, just the ability of the customer and the development team to be talking even approximately the same language. It will get better but before it does there may well be some disaster stories in which the software takes the blame. Bad news travels fast, as HP now know

==
Martin, well said (as usual). There I go assuming that folks are dealing with an open source vendor who can provide support for business users - not just a 'download and good luck' open source project. Believe it or not, there are some companies with products based on open source where the sales guy shows up with a smart systems engineer; and the provide the same kind of service you might have received from (some) commercial vendors. In any case, Im reminded of the old quote which I've used at workshops: "You can tell a search sales rep because his lips move when he talks":)