Enterprise Search

A paradigm shift in enterprise search

I've been involved in enterprise search since before the 'earthquake World Series' between the Giants and the A's in 1989. While our former company became part of LucidWorks last December, we still keep abreast of the market. But being a LucidWorks employee has brought me to a new realization: commercial enterprise search is pretty much dead.

Think back a few years: FAST ESP, Autonomy IDOL (including the then-recently acquired Verity), Exalead, and Endeca were the market. Now, every one of those companies has become part of a larger business. Some of the FAST technology lives on, buried in SharePoint 2013; Autonomy has suffered as part of HP because - well, because HP isn't what it was when Bill and Dave ran it. Current management doesn't know what they have in IDOL, and the awful deal they cut was probably based on optimistic sales numbers that may or may not have existed. Exalead, the engine I hoped would take the place of FAST ESP in the search market is now part of Dassault and is rarely heard of in search. And Endeca, the gem of a search platform optimized for the lucrative eCommerce market, has become one of three or four search-related companies in the Oracle stable. 

Microsoft is finally taking advantage of the technology acquired in the FAST acquisition for SharePoint 2013, but as long as it's tied to SharePoint - even with the ability to index external content - it's not going to be an enterprise-wide distribution - or a 'big data' solution. SharePoint Hadoop? Aslongf as you bring SQL Server. Mahout? Pig? I don't think so. There are too many companies that want or need Linux for their servers rather than Windows.

Then there is Google, the ultimate closed-box solution. As long as you use the Google search button/icon, users are happy – at least at first. If you have sixty guys named Sarah? Maybe not.

So what do we have? A few good options generally from small companies that tend to focus on hosted eCommerce - SLI Systems and Dieselpoint; and there’s Coveo, a strong Windows platform offering.

Solr is the enterprise search market now. My employer, LucidWorks, was the first, and remains the primary commercial driver to the open source Apache project. What's interesting is the number of commercial products based on Solr and it's underlying platform, Lucene.

Years ago, commercial search software was the 'safe choice'. Now I think things have changed: open source search is the safe choice for companies where search is mission. Do you agree?

I'll be writing more about why I believe this to be the case over the coming weeks and months: stay tuned.

/s/Miles

 

Mind the gap

A few weeks ago, a former client asked me about the 'lay of the land' in enterprise search - which companies were the one to be considered for evaluation. It's something I'm frequently asked, and one big reason why I strive to stay current with all of the leading commercial and open source vendors in the market.

As I pulled together the list, it occurred to me that recent consolidation has led to an odd situation: there is no longer a 'mid-market' in enterprise search.

Under $25,000(US), there are a number of options from free and low-cost open source (SearchBlox and my employer LucidWorks come to mind). 

Google has discontinued its low cost (blue) search appliance, and raised the cost of its regular (yellow) one to apparently be well above $25K.

We also have the old-school major commercial vendors - like FAST (now Microsoft SharePoint Search); Autonomy (now HP); Endeca (now Oracle), and finally Vivisimo (now IBM). Trend or not, these enterprise search products command high initial outlay, often significant implementation costs, and high ongoing 'support' once you've rolled it out. Looks like the mid-market is gone.

So now the question is: What do you get for the difference in price? I'd suggest not much in the way of capability; nothing in terms of scalability; and very very little in the way of flexibility.  I guess it's 'caveat emptor' - buyer beware!

What about some products/projects I haven't mentioned? Well, the focus of my article here is on enterprise search. Great candidates like Coveo are 'windows only' which disqualifies them from my list. I suppose you could consider the GSA as not enterprise ready, but I think appliances make the OS issue irrelevant. I've also omitted mentioning other projects because they have not yet shipped a 'Version 1.0' release - that's testware, no matter who it's from. And I'm sure there are open source projects where a single person is making all the calls - I don't consider that enterprise ready either.

I’ll be looking for the day when the big guys start value pricing their software licenses and help bring the market into line with today’s reality.

If you think I've unfairly represented the market, let me know - I'm not shy about posting comments that differ with my viewpoint.

 

s/Miles

 

Want search 'just like Google'?

This weekend I read a post by John Hefferman on Seth Earley's blog, which was related to a discussion going at the LinkedIn Enterprise Search Engine Professionals group. The similarity may not jump right out at you, but let me try to piece together the logic in my brain that makes them one in the same.

If asked, most enterprise search users will tell you they want search "just like Google'. In fact web search is different than enterprise search - but it is possible to deliver a somewhat psychic 'Google-like' experience in the enterprise. It takes time, effort, and understanding of what people mean by "just like Google". And yes, these are often hard to come by. (If you have a few minutes, see our recent webinar from earlier this year  "What enterprise users want from search".)

The LinkedIn discussion was started by Phil Lloyd of Standard Life in the UK where he asked about the effort required to keep an enterprise search system running reasonably well (in fact, he asked what the minimum effort to implement FAST Search for SharePoint; but the discussion quickly became one of ongoing effort). 

The answer, as valid for FAST as it is for any enterprise product is, of course "it depends". 

You've just spend a large sum to license the platform. If you want it to work well, you'll staff it to run well and provide a return on the investment. If you don't care how well it works, it can be arbitrarily inexpensive. Heck, if it doesn't need to work well, you may never need to touch it again. After a few years, a new VP will be willing to toss it out and spend a large sum on a new and improved platform. Of course, without attention, it, too, will be doomed to failure. 

How does this relate to the Google article? One reason Google on the public web is so good is that it has a huge data set to work with. But - and this is what most enterprise search owners don't get - Google also has armies of engineers and bots looking at search activity every day... in fact, perhaps even every second or two, 24 hours a day, 365 days a year, worldwide.

When you search Google for your FedEx tracking number, do you think that is in the search index? When Google recognizes a 12 digit number that matches the algorithm for FedEx tracking codes, it bypasses the search index and offers to link the user to FedEx. Did you know that if you change a single digit in that code there's an excellent chance you'll get a rare 'no hits' from Google?

How did they know this? Well, by noting a bunch of odd 12 digit numbers as searches... all with no hits and almost every one unique - some bot noticed that the subsequent search was for FedEx, and alerted a human engineer to answer the riddle and take action. In this case, the action was likely to contact FedEx to understand it's algorithm in order to recognize a valid tracking code - and to insert logic ahead of the actual search to suggest a specific page on the FedEx site - the one that tracks that package. 

If you want search that works just like Google, you need to invest some time. Maybe it's an average of one or two FTEs each year; maybe it's less. One thing is certain; when you first roll out search, it will take more effort than when it's been running for a year. But of you let search roll out and never look at it again - well, then you've managed to out in the minimum effort for search. And you've probably doomed your company to buying a new platform out of frustration in a few years. The choice is yours.

Great NEW SharePoint FAST Search Book

I've just finished scanning (and started reading) the new book on FAST Search for SharePoint from Mikael Swenson, Robert Piddocke, and Marcus Johansson. It's packed with information on the acchitecture & deployment of FS4SP, as well as in deptch chapters on content and query processing. The book provdes lots of code examples in C# for Powershell. Really a great resource to have.

Congrats on the book guys!

SharePoint Saturday Silicon Valley June 2 2012

I'm happy to announce a 'first ever' event: SharePoint Saturday will finally come to Silicon Valley in less than two months. We're been working behind the scenes for a few months now with some really enthusiastic people to secure sponsors, a facility, and some great speakers with valuable insights on SharePoint.

For those of you who follow search but may not know about SharePoint Saturday, it is an organization that holds events held in dozens of cities around the world. It brings together SharePoint experts in every aspect of SharePoint, from development to administration to power users - and yes, even to search. And, for attendees, it's always free to attend. To find an event near you, check the main SharePoint Saturday web site for updates and background.

I'd like to thank the people who we have been working with to get this first-ever event for Silicon Valley, all of whom are volunteering their time to make it possible. Christian Buckley of Axceler has provided great suggestions drawing on his deep experience with SharePoint and in organizing SharePoint Saturday events; Ken Lo from Kattelo has helped us locally organize a team and gather great speakers from around the country; Kelley Tyburski of Avepoint who ran last year's SharePoint Saturday in Sacramento; Jerry Pierre of SLAC, who has volunteered to help in selection of topics and in keeping us on track; and our own Chris Fernandez who we asked to get the event organized and manage all those pesky details. And of course I'd like to thank our sponsors, without whom this event could not be happening.

Between now and the June 2nd event, I'm sure you'll be hearing more about the event; you'll find #spssv on Twitter. And if you're in the area, register now and plan on attending. 

/s/ Miles

 

SharePoint 2010 Search Center Improvement

Our friends over at Arcovis are hosting a SharePoint Shoptalk webinar Thursday 3/29 that SP search admins and business line managers should attend. 

"5 Little things you can do to make a big impact on your SharePoint Search Center" will address steps that you can take to improve search in SharePint. Paul Olenick is the speaker, and he certainly knows his stuff.

Register here.

Are you spending too much on enterprise search?

If your organization uses enterprise search, or if you are in the market for a new search platform, you may want to attend our webinar next week "Are you spending too much for search?". The one hour session will address:

• What do users expect?
• Why not just use Google?
• How much search do you need?
• Is an RFI a waste of time?   

Date: Wednesday, November 16 2011

Time: 11AM Pacific Standard Time / 1900 UTC

Register today!

What search platform is best? Workshop at KMWorld

Next week in Washington DC, InfoToday runs their Fall enterprise search conferences - KM World, Enterprise Search Summit, SharePoint Symposium, and Taxonomy Boot Camp.. whew! Monday - Halloween Day! - I am giving a workshop at the conferences with the somewhat vague title 'Enterprise Search Technologies'.

What I'll be talking about is an overview of the platform vendors, with some detail on strengths and weaknesses of the vendors; and a drill down into what you need to do before you call the vendors (if you value your time).

You can still sign up for the workshop for $295US or the entire conference for a bit more; see you in DC in a week!

/s/Miles

Searching for Sarah at SharePoint Conference 2011

Just noticed one of the most interesting sessions at last May's Enterprise Search Summit is coming to the October Microsoft SharePoint Conference! We blogged about it back in May.

Basically, Booz & Company did an evaluation of SharePoint 2010 search - FAST Search for SharePoint as I recall - versus the Google Search Appliance they had been using. At one point, the search business owner was trying to find the last name of a woman she had met in the firm; and when she searched for 'Sarah', hoping to find her in the directory, the GSA returned 60 men in the result list. Can you guess why? A hint: metadata (check the earlier article, or come to SPC 2011 to find out).

Now in fact, we think the GSA could have been tuned to emulate this OOB behavior by SharePoint; but this is a reminder that not every search platform works great in every environment. Buyer beware!

Ever had a similar experience? Let us know about it!