Is Elasticsearch really enterprise search?

Not too long ago, Gartner released it's the 2014 Magic Quadrant which I’ve written about here and which has generated a lively discussion on the Enterprise Search Engine Professionals group over on LinkedIn.

Much of the discussions I’ve seen about this year's MQ deals with the omission of several platforms that most people think of as 'enterprise search’. Consider that MQ alumni Endeca, Exalead, Vivisimo, Microsoft FAST, and others don’t even appear this year. Over the last few years larger companies acquired most of these players, but in the MQ it's as if they simply ceased to exist.

The name I've heard mentioned besides these previous MQ alumni is Elasticsearch, a relatively new start-up. Elasticsearch, based on Apache Lucene, recently had a huge round of investment by some A-List VCs. What's the deal, Gartner?

Before I share my opinion, I have to reiterate that, until recently, I was an employee of Lucidworks, which many people see as a competitor to Elasticsearch. I believe my opinions are valid here, and I believe I’m known for being vendor-neutral. I think the best search platform for a given environment is a function of the platform and the environment – what data source, security, management and budget apply for any given company or department. “Search engine mismatch’ is a real problem and we’ve written about it for years.

Given that caveat, I believe I’m accurately describing the situation, and I encourage you to leave a comment if you think I've lost my objectivity!

OK, here goes. I don't believe Elasticsearch is in the enterprise search space. For that reason, if for no other, it doesn’t belong on the Gartner Magic Quadrant for search.

You heard it here. It's not that I don't think Elasticsearch isn’t a powerful, cool, and valuable tool. It is all that, and more. As I mentioned, it’s based on Apache Lucene, a fantastic embedded search tool. In fact, it's the same tool Solr (and therefore Lucidworks' commercial products) are based on.  But Lucene by itself is a tool more than a solution for enterprise search.

Let me start by addressing what I think Elasticsearch is great for: search-enabled data visualization. The first time I attended an Elasticsearch meet-up, they were showing the product in conjunction with two other open source projects: Logstash and Kibana. The total effect was great and made for a fantastic demo! I was fully and completely impressed, and saw the value immediately - search driving a visualization tool that was engaging, interactive, and exciting! 

Since then, Elasticsearch has apparently hired the guys who created those two respective open source projects, and has now morphed into a log analytics company - more like Splunk with great presentation capability, and less like traditional enterprise search. Their product is ELK - Elasticsearch Logstash Kibana. You can download all of these from GitHub, by the way.

(Lucidworks has also seen the value of Kibana to enterprise search, and has released their own version of Logstash and Kibana integrated with Solr called SiLK (Solr-Integrated Logstash and Kibana).

Now let me tell you why I do not think of Elasticsearch as an enterprise search solution. First, in my time at Lucid, I'm not aware of any enterprise opportunities that Lucidworks lost to ELK. I could be wrong, and maybe the Elastic guys know of many deals we never saw at Lucid. But with no crawler and other components I consider ‘required’ as part of an enterprise search product, I'm not sure they're interested - yet, at least.

Next, check the title of their home page: "Open Source Distributed Real Time Search". Doesn't scream 'Google Search Appliance replacement', does it? Read Elasticsearch founder Shay Banon on the GSA.

Finally, Wired Magazine has an even more interesting quote: Shay Banon on SharePoint. “We're not doing enterprise search in the traditional sense. We're not going to index SharePoint documents”.

Now, with the growth and the money Elasticsearch has, they may change their tune. But with over $100M in venture capital now, I think their investors are valuing Elasticsearch as a Splunk competitor, and perhaps a NoSQL search product for Hadoop - not a traditional enterprise search engine. 

So the real question is: which space are you in? Enterprise Search with SharePoint and other legacy data sources? Web content and file shares you need a crawler for? Is LDAP or Active Directory security important to you? Well - I won't say 'no way' - but I'd want to see it before I buy.

Do you use Elasticsearch for your enterprise search? Let me hear from you!

 

 

 

Comments

[Steve]

Both Solr and ElasticSearch are tools that you can use the way you want. They are very flexible and have a lot of features that offer solutions for Enterprise Search.

Whether you use Google Search Appliance, Solr, or ElasticSearch, you will always have to do customize your solution. If your project is complex, the customization of GSA can be very heavy and costly.

BTW, Google Search Appliance has been discontinued: www.bigwisdom.net/blog/2016/03/05/best-alternatives-to-replace-google-search-appliance/

[Ihsan Turkmen]

A wild?ard searc* into a 3 million records

We are a news agency and have a huge text,video and image archive which goes back to 20 years. Ok, not very huge maybe, but huge enough to wait several minutes to get the result of a fulltext search query from the SQL server. Recently, we got a quote from a Google partner for a GSA solution, and we noticed that searching a needle in the haystack by GSA would cost us many coins as we have several millions of items in our digital archive. GSA cost is based on the number of items enclosed by your archive and not feasible at all compared to the situation in an e-commerce datastore that has limited number of items held in its database.

So, as a solution we encountered Elasticsearch. It is suprisingly a very flexible solution for a free product. Elasticsearch along with the Kibana plugin installation can also be done in minutes. I don't even mention Logstash as I did not test it on a real application. To make the long story short, we completed our integration in 3 days and decided to drop Ms Indexing Server as it is also a product discontinued. If you don't want to get scammed, and waste your money, I strongly suggest you look into Elasticsearch as it is free as in beer. Don't forget the fact that no one can compete with free. Those who claim that it is not an enterprise search tool should convince me what feature is missing in Elasticsearch if it can produce the same result as the others would do. One more thing, we did not use a physical machine to install the Elasticsearch, yet two virtual machines were more than enough to activate a clustered solution. It is absolutely an enterprise solution just because it gives the redundancy and linear expansion opportunity as your capacity demand in time. Responsiveness? Well, is that a good answer if I say you get your results in milliseconds when you make a "wildcard search" into 3 million records?.

Ihsan Turkmen

[Timo Selvaraj]

Good discussion on the subject Miles. Elasticsearch is not a full enterprise search solution, yet, which is why organizations prefer to go with SearchBlox http://www.searchblox.com/ (Disclosure: I am a cofounder at SearchBlox) and prefer a solution that incorporates Elasticsearch + SearchBlox as it gives them a platform as well as all the features the Google Search Appliance provides for enterprise search. Customers have switched from Google to SearchBlox + Elasticsearch as they see the value.
==
Timo, thanks for the comment! And it's good to know that SearchBlox, which I'm familiar with, is providing the additional layers. Let's stay in touch: I want to know more!
/s/Miles