August 11, 2013

Where do actual your Autocomplete / Autosuggest terms come from?

Most sites with search either already offer autocomplete search as you type into the search box, or are planning to implement it soon. And most search vendors now offer at least a passing support for this feature.

There can be technical issues with latency and how the page gets rendered in the browser, but search engines are addressing these too, and technical aren't likely to be totally blocked on this.

However, one area that should be thought about more is where these search suggestions will come from. There are many ideas in the industry, and sites may decide to do a combination of things.

Example sources for Autosuggest / Autocomplete suggestions:

  • Dictionary based – this is labor intensive and is best used only for specific occasional overrides
  • Specific URL suggestion based
  • Word index based – the search engine taps its own index of document phrases – implementations vary by vendor
  • Search Logs – the main issue here is to filter out any potentially objectionable language
  • Taxonomy matches – suggest specific nodes in the taxonomy
  • Page Title matches – suggest the titles of specific pages
  • Synonyms
  • A hybrid combination of these

Other technical / design items:

  • How to match – just the beginning or more advanced matching
  • What happens when the user clicks or selects a suggestion – do you show search results, or go directly to a page.
  • Check that suggestion being offered will actually bring back results

May 14, 2013

Open Source Search Myth 5 - Total Cost of Ownership

This is part of a series addressing the misconception that open sounce search is too risky for companies to use. You can find the introduction to the series here; and Part 4, Features and Capabilities, here.

Part 5: Total Cost of Ownership

Total cost of ownership, TCO, is a big deal to large users of search technology. Usually, the component of TCO with respect to search is the license fee; enterprise search was historically an expensive proposition. But in fact there are other major components of TCO including implementation/operations, hardware cost, and ongoing support come to mind.

Walter Underwood, one of the key developers at Ultraseek and later the guy who did the Netflix relevancy contest, once explained the difference between commercial and open source search. Let me paraphrase: 

"With commercial search, you spend a lot of money to license it; then you spend a lot of money to implement it.

With open source search, you download the software for free; then you spend alot of money implementing it."

But there is another big element: how much iron do you need? A few years ago we helped a company switch search platform. Their business was search enabling small-town newspaper archives going back to the 1890s, via OCR'd content. They add tens of thousands of documents - historical newspaper articles - every day. 

The commercial platform they replaced required major expense in new servers as they content grew. Every year.

As it turns out, the ROI for swapping out their old search engine was easy: they needed less new hardware every year than with the old engine. And so much less that the ROI period was less than a year.

A different project we did when we were still doing business as New Idea Engineering involved a comparison between Microsoft SharePoint 2010 and search with Solr. Our customer wanted to know if the switch would, indeed, require fewer servers to do the job. It turns out that it was quite reasonable to replace the 12 servers Microsoft FAST required with 6 or fewer servers running Solr. Half the cost of servers; half the cost of energy; half the cost of maintenance. Like the concept?

Now, I'll agree that LucidWorks - my employer - markets a proprietary search platform based on Solr. And we do not license the product for free. But compared to most commercial platforms, LucidWorks Search is pretty darned reasonable. And you still get the cost savings in energy, iron, and scalability.

Less hardware. Better search. How is the TCO of open source a liability compared to most commercial search platforms?

 

 

April 23, 2013

Open Source Search Myth 4 - Features and Capabilities Lag

This is part of a series addressing the misconception that open source search is too risky for companies to use. You can find the introduction to the series here; and Part 3, Skills required In-House, here.

Part 4: Features and Capabilities Lag

Keeping up with the latest and greatest technology is important, especially when there is a great deal of innovation in a field. Enterprise search is one such field.

In this post I'll address the claim that "Production functionality may trail in specific features relative to commercial search firms".

First, let me remind you that many of the coolest advanced capabilities in modern search platforms is delivered using third party products integrated into the actual search product. Examples:

Entity extraction: Cool stuff, and part of many search platforms. Often implemented using technology from companies like Basis Technology, Pingar, and others.

Non-English support: Required for any large-scale enterprise. Think Basis Technology again; or pretty darned good open source filters.

Document format support: Leaders here were smaller companies that were eventually purchased by larger search companies: Keyview (not Autonomy); Stellent (now Oracle); ISYS (now IBM). Open source Tika.

Sentiment Analysis: Identify 'positive' versus 'negative' sentiment, using products from Lexalytics, Attensity, SAS, LingPipe and others. 

My point is not that large enterprise search platform companies do not include some cool new technologies in their products: it's just that the 'cool' usually comes from a third party that can be licensed for use in any platform, not just "commercial" ones. 

And, when you use open source platforms, you always have the option of doing a feature yourself - either in-house, or using a consulting firm.

And you might not be aware of capabilities where open source Solr is ahead of many commercial vendors. For example, consider Geo search, which lets you easily search for 'documents' relevant to a particular location.  And it can even be used to answer questions like "what managers are on-duty on Saturday night at the LA store".

I will say that Microsoft, in its SharePoint 2013, has implemented a very nice query boosting tool that, as far as I can tell, was created in-house - I doubt it was in the FAST pipeline at the acquisition. 

But give that caveat, I'd ask, what with all of recent acquisition and mergers, whether any 'enterprise search' company implemented major new capability like pivot facets, entity extraction and more - without licensing the technology from an outside company?

 

 

 

 

 

March 20, 2013

Open Source Search Myth 3: Skills Required In-House

This is part of a series addressing the misconception that open source search is too risky for companies to use. You can find the introduction to the series here; this is Part 3 of the series; for Part 2 click Potentially Expensive Customizations.

Part 3: Skills Required In-House

One of the hallmarks of enterprise software in general is that it is complex. People in large organizations who manage instances of enterprise search as no less likely than their non-technical peers to believe that "if Google can make search so good on the internet, enterprise search must be trivial". Sadly, that is the killer myth of search.

Google on the internet - or Bing or Baidu or whichever site you use and love - is good because of the supporting technology, NOT simply because of search. I'd wager that most of what people like about Google et al has very little to do with search and a great deal to do constant monitoring and tweaking of the platform.

Consider: at the Google 'command line' (the search box), you can type in an arithmetic equation such as "2+3" get 5. You can enter a FedEx tracking number and get a suggestion to link to FedEx for information. It's cool that Google provides those capabilities and others; but those features are there because Google has programs looking at search behavior for all of its users every day in order to understand user intent. When something unusual comes up, humans get involved and make judgments. When it makes sense, Google implements another capability - in front of the search engine, not within it.

Enterprise search is the same - except that very few companies invest money in managing and running their search; so no matter how well you tune it at the beginning, quality deteriorates over time. Enterprise search is not 'fire and forget'.

 Any company that rolls out a mission critical application and does NOT have their own skilled team in house is going to pay a consulting form thousands of dollars a day forever. 'Nuff said.

 

March 18, 2013

Solr 4 Training 3/27 in Northern Virginia/DC area

Interrupting my series on whether open source search is a good idea in the enterprise to tell you about an opportunity to attend LucidWorks' Solr Bootcamp in Reston, Virginia on Wednesday March 27. Lucid staff and Lucene/Solr committers Erick Erickson and Erik Hatcher will be there, along with Solr pro Joel Bernstein. Heck, I'll even be there!

The link is here; for readers of our blog, use discount code SOLR4VA-5OFF for a discount.

Course Outline:

  • What's new in Solr 4
  • Solr 4 Functional Overview
  • Solr Cloud Deep Dive
  • Solr 4 Expert Panel Case Studies
  • Workshop and Open lab

And ask the guys how you can get involved in Solr as a contributor or committer!