Enterprise Search

Gartner Magic Quadrant 2008 Now Available

If you have not seen it, the new Gartner Magic Quadrant for Information Access - their name for intranet and customer facing search - has been published and is available for viewing on the Gartner web site thanks to a pointer from Microsoft's Analyst Relations page.

The big story, one which must have them fuming in England, is that Autonomy has dropped down a bit, and the combined Microsoft-FAST offerings have moved up a bit. This puts Autonomy a bit higher up on the 'Completeness of Vision' scale - by a few pixels - but a decent quarter-inch below Microsoft on the 'Ability to Execute' scale. Endeca, IBM, ZyLAB and Vivisimo squeaked into the upper right quadrant, while Google moved right to the link splitting the 'Challengers' from the 'Leaders', but ever so close - one could say the Google dot is on the line. It's odd that Google is not higher on the 'Ability to Execute' scale, since that usually means how well funded the company is. Perhaps they are looking at the budget/sales for only the Google appliance; but even then, Steve Arnold's numbers put them above the others on the scale.

Some excellent search products fell off the list this year, as Gartner has changed their methodology. The products we feel still qualify for the report include Dieselpoint, SLI Systems, and X1 Technologies, as well as newcomer Attivio. The article has more details. And as the con artist Fagan said in the play base don Dicken's Oliver Twist, '...if you happen to pass the Tower of London, have a look at the Crown Jewels'.

The Future of Search is Simpler

Gary Szukalski, VP of Field Marketing at Autonomy,  gave a keynote speech at ESS West 2008 entitled "Meaning Based Computing" notable for its lack of vision. He was a replacement for Stouffer Egan, Autonomy's CEO, whose slides he likely used. The talk detailed the difficulties inherent in enterprise search, especially how challenging it can be to understand the full meaning of a word like "dog" or "shred" without context. We live in this world, we understand the difficulty.  While his talk outlined the direction of automatic categorization, alerts, profiles, dynamic and real-time clustering and schemas, it left me wanting real vision and a roadmap from the industry leader. He never rose above the complexity of information processing. 

In a sharp contrast, Google provided more vision in their 10-minute lunch pitch than Autonomy's same old key note speech. Google provided a clear vision: you can be up and operational in one day and search everything.  Simple to use and simple to administer.  In 2008, Google is not the naive implementation that we saw 6 years ago: they have made real progress toward their vision. While other companies are marching to that vision. Dieselpoint's OpenPipeline, Endeca's simple administrative controls, Fast's navigators, Autonomy's categorization, Google is providing the vision. The future is simpler and usable by everyone on the enterprise. We have along way to go - but we can change the business world.  We are moving closer to the vision of many sources of data providing insight and increasing the pace of business decisions.

Search Quality: You Can't Improve What You Don't Measure

In our latest survey of new newsletter subscribers we found that 29% had no formal metrics for measuring quality of search results.  Search metrics allow you to keep search on the right track and can be a powerful tool for managing your systems.  They are a wonderful source for insights and trends.  We thought we would share a couple that we think work well. Many of these are covered in greater depth in Interpreting Your Search Activity Reports in the Enterprise Search newsletter.

• Count the number of people who use search  
• Count the total number of searches  
• Count the number of zero search results  
• User feedback on top 100 searches  
• Track email complaints about search  
• Measure number of clicks on navigators (navigation menu items)  
• Business Goals  
•    
  • Reduce call volume (normallized for growth in customer base) by enabling self-service from search: results are good enough to reduce calls.
  • Reduce e-mail volume (again adjusted for growth in customer base) by enabling self-service from search: results are good enough to reduce e-mails. 
  • Revenue       
  • Add-on revenue       

Some interesting Enterprise search events the week of June 2nd

There are two really interesting events happening next week that might be of interest.

First, Leslie Owens of Forrester is presenting a the Forrester Wave Enterprise Search platform webinar  on Monday morning, June 2 at 8AM. There is a nominal fee, but I think you will find it interesting.

Then, Leslie and several other interesting speakers will be at a free one day seminar hosted by FAST on Wednesday the 4^th in Redwood Shores California at the Sofitel Hotel. In addition to Leslie Owens' presentation on 'Technology Populism', speakers will include Jeff Spataro of Microsoft; Hadley Reynolds of FAST; and senior IT managers from Cisco and National Instruments.  Hadley, by the way, speaks and writes on Search Centers of Excellence and other innovations in the application of enterprise search. Be sure to register for the free FAST Search event.

A proposed standard for enterprise search

Dieselpoint has announced support for a technology it calls OpenPipeline, which can enhance the task virtually every enterprise search technology uses to get documents into the search index. They will be showing the pipeline at the upcoming Enterprise Search Summit on May 20-21 integrated with their new Dieselpoint Search 4.0, also on display.

The Dieselpoint press release claims:

OpenPipeline provides a common architecture for connectors to data sources, file filters, text analyzers and modules to distribute documents across a network. It is fully functional out of the box and includes an installer, a job scheduler, file scanner and crawlers, doc filters, and point and click interface with drag and drop module installation.

OpenPipeline is compatible with IBM's UIMA (Unstructured Information Management Architecture), and is designed to connect UIMA annotators to other systems.

Document processing can be centralized or parallelized as needed. The transport mechanism is simple, web-services XML over HTTP. RSS/Atom feeds are also possible.

The development philosophy behind OpenPipeline stresses simple, elegant design, and massive scalability. Minimal external dependencies and straightforward plug-in implementation ensure that the learning curve is low.

OpenPipeline can be downloaded without charge from http://www.OpenPipeline.org. It's available under the Apache License.

Making this technology open source makes sense. The core technology for an enterprise search company, their 'secret sauce', is optimizing the index and making search great, not creating new code to parse the latest version of Microsoft Office or of Documentum. By embracing OpenPipeline, presumably we will start to see pipeline stages created by a number of smaller companies and individuals, easing the burden on enterprise search companies. And companies that provide possible sources of data like Content Management Systems, can create a single pipeline stage for their product that could work for every search technology, and be done with it.

To create a searchable index, all search technologies need to create a stream of text. If the source document is a binary file - Microsoft Word, for example - search vendors need to provide some way to read the format and convert it to text. The same is true of content stored in a relational database: each row represents a virtual document which needs to be extracted from the database and turned into a stream of text. This conversion is typically done as one stage of a pipeline. Other stages may include adding metadata, performing entity or sentiment extraction, or even enhanced language processing.

The concept of a 'pipeline' applies directly to many existing search technologies, each with a proprietary method of accessing content. On top of that, no search technology companies have cooperated with competitors to create standards. In the relational database world, standards have made life much better: consider ODBC and JDBC. Because of these standards, developers can write code that can connect to just about any relational database. Not so in search. Maybe this effort will help break the ice. Stay tuned...

As enterprise search users, are you glad to see an open source solution for part of the search puzzle?

The problem with alerts - Google or otherwise

I use Google alerts to keep an eye on current events. Over the weekend I got an alert: "AMEC uses Verity's K2" - Now, since Verity is part of former competitor Autonomy, and because K2 is generally not being actively marketed, I decided to read the article. Sure enough, the content is dated January 2004, but Google Alerts thinks it is brand new. So I have to conclude that either the publisher just changed something on the page, or Google is just finding that document - either way, Google thinks this is news and in reality, it isn't.

Not long after we started SearchButton.com, we met the Google founders Sergey and Larry. Mark Bennett, my co-founder at SearchButton and here at New Idea Engineering, asked about the then-young Google's handling of dates and recency, and the Google guys took the position that date wasn't that important. This has led to a couple of energetic email exchanges over the last few years, but my recent alert illustrates the problem Google - and most other search technologies have - in generating really useful alerts. In fact, this subject was of such relevance to enterprise search owners, we had an article about the importance of dates in the first issue of our enterprise search newsletter in April of 2003.

Deep Web proposes federation resource site

Sol Ledeman of Deep Web Technologies wants to create a one-stop demo center for federation technology and has invited all of the major vendors to participate.

Federated search is becoming increasingly popular as more corporate customers are looking for ways to delivery results from multiple enterprise search installations, often from many different vendors. Sometimes the issue is technical, sometimes political, but nearly all companies have three or more search vendor technologies running somewhere behind the firewall.

The one thing we'd like to have seen in Sol's challenge is security, since that's what we think separates the winners from the also-rans in federation. It's not always easy, but it is 'real world' in companies. Nonetheless, a demo site where users can compare vendor solutions 'apples to apples' on the same data sources would be nice.

By the way, we've seen some confusion among our customers and prospects on the subject, so we've taken a shot at defining 'federated search' in our Enterprise Search newsletter. We hope that helps some.

Is hosted / managed search behind the Microsoft FAST acquisition?

What does Microsoft the acquisition of FAST mean for the industry?

We've been sifting through the information available around the web and from our contacts in the enterprise search space, and we are beginning to see some signals come through the noise.

First, this is still an early story in the consolidation that will continue to take place over the next few years. Microsoft will operate FAST as a 'wholly owned subsidiary', which may well be a first for Microsoft - I'm not sure.  In addition, FAST seems to be heads-down on their release schedule, with some cool stuff rumored to be on the way.

In the Microsoft-FAST conference call (which will be available at that link through June 9th, 2008 - scroll to the Teleconference link), Microsoft's Jeff Raikes had little to say about how the integration would go forward, but I thought he dropped an interesting hint while answering an question from a JP Morgan analyst. He was talking about how the two technologies might fit when he said:

"Obviously, we feel one of our great strengths is that we'll bring to customers the power of on-premise software with software services; that combination can bring  customers greater capability plus we can give customers the power of choice  in terms of deployment models. So without going in greater detail which I wouldn't be able to do today ... I can just simply say that part of what we will look at ... will be to marry the strengths that we have with our software plus services with what FAST is doing in Enterprise Search."

Now, taken by itself it all sounds pretty generic. But it was his emphasis on the word "with" above - and on the parts about software as a service. Could it be Microsoft  wants FAST for the hosted/managed enterprise search solution that FAST can offer its customers? No enterprise data center; no need for in-house expertise; no pesky updates to install; no load-balancing to manage. And FAST can offer this data center service either fully hosted or remotely by connecting into the enterprise and providing only search management services.

Could Jeff be admitting that Microsoft wants to look towards more enterprise services using a hosted model - say something like his friends in Mountain View offer? Only time will tell!

It sure makes the FASTForward'08 user conference a must-see event.

Updated 2008 Enterprise Search Vendor Roundup

Jan. 10, 2008 - San Jose, CA, USA 

Microsoft announced they were acquiring FAST Search on January 8, forcing New Idea Engineering to amend our January 4th article "2008 Enterprise Search Vendors:  The new 'Fab4 ... and 1/2" (http://www.ideaeng.com/pub/entsrch/2008/number_01/article01.html). The announcement validates our original assessment and reinforces that search is mission critical for corporations, driving Microsoft to invest in a better search technology.

Some Highlights from NIE's 2008 Enterprise Search Vendor Roundup
 
Autonomy IDOL and FAST Search continue to hold the high end. K2 and Ultraseek are finally retiring.
Google's new version 5 appliance has arrived in the enterprise search mainstream.
Endeca is moving from the ecommerce side and had one of the most impressive search demos at ESS West 2007.
Lucene/ Nutch/ Solr (LNS) open source search engines continue to gain customer mindshare.
Microsoft with its acquistion moves in as Tier 1.
IBM and Oracle still not there.
 
Autonomy IDOL and FAST Search continue to hold the high end, evolving into "search platforms" that go beyond traditional drop in applications. The two leaders from earlier this decade, K2 and Ultraseek, are fading.

Google's new version 5 appliance has arrived in the enterprise search mainstream. While the new version won't satisfy every requirement, it addresses many of the earlier integration issues that had held it back. Expect to see the Google logo on a lot more enterprise portals.

Endeca has created some slick administration tools, doing very well in a head-to-head comparison with Autonomy and FAST despite their continued progress in this area.  As the importance of administration continues to increase, we are more enthusiastic about them in the Enterprise space.

Open source tools based on Lucene, including Nutch and Solr (LNS) are increasingly considered by companies, especially in niches that need to micromanage document relevancy and rating. Lucene and its derivatives are increasingly embedded in other software packages and services, to the point that many users won't even realize they're using it.

We had expected IBM to be the next entrant into the "Tier 1" lineup, based on their iPhrase acquisition. To our surprise, when we saw IBM at ESS East 2007, they were featuring one of their older engines, the OmniFind Enterprise Edition. IBM OmniFind is still not one of our new Fab 4 and an 1/2.

Dieselpoint, Intellisearch, Reccomind, ISYS, ZyLAN, Vivisimo, Siderean and Exalead have strong presences in niche markets.
 
To read the full article ... 2008 Enterprise Search Vendors: The New Fab 4 ... and 1/2. http://www.ideaeng.com/pub/entsrch/2008/number_01/article01.html

Microsoft is acquiring FAST Search for $1.2 billion

Obviously if this goes through, and we expect it will, it moves Microsoft into the Tier 1 position and creates an interesting situation for Autonomy, and to a lesser extent the Google search appliance.

Links to ZDNet, Yahoo!, Reuters -- many others online.

This continues the trend of search engine acquisitions.  Could Endeca or Autonomy be next?  If so, would Oracle be the logical choice?  Google, IBM and now Microsoft already have well established enterprise packages.  Although Oracle has its own private brand of search, so did Microsoft, but that didn't stop them.