Enterprise Search

Open Source Search Myth 2: Potentially Expensive Customizations

This is part of a series addressing the misconception that open source search is too risky for companies to use. You can find the introduction to the series here; this is Part 2 of the series; for Part 3 click Skills Required In House.

Part 2: Potentially Expensive Customization

Which is more expensive: open source or proprietary search platforms?

Commercial enterprise search vendors often quote man-years of effort to create and deploy what, in many cases, should be relatively straightforward site search.  Sure, there are tough issues: unusual security; the need to mark-up content as part of indexing; multi-language issues; and vaguely defined user requirements.

Not to single them out, but Autonomy implementations were legend for taking years. Granted, this was usually eDiscovery search, so the sponsor - often a Chief Risk Officer - had no worries about budget. Anything that would keep the CRO and his/her fellow executives out of jail was reasonable. But even with easier tasks such as search-enabling an intranet site, took more time and effort than it needed because no one scoped out the work. This is one reason so many IDOL projects hire large numbers of IDOL contractors for such long projects.

FAST was also famous for lengthy engagements. 

FAST once quoted a company we later worked with a one year $500K project to assist in moving from ESP Version 4.x to ESP Version 5.x. These were two versions that were, for all purposes, the same user interface, the same API, the same command line tools. Really? One year?

True story: I joked with one of the sales guy that FAST even wanted 6 months to roll out a web search for a small intranet; I thought two weeks was more like it. He put me on the spot a year later and challenged me to help one of his customers, and sure enough, we took almost a month to bring up search! But we had a constraint: the new FAST search had to be callable from the existing custom CMS, which had hard-coded calls to Verity K2 - the customer did not have time to re-write the CMS.

Thus, part of our SOW was to write a front-end that would accept search requests using the Verity K2 DLL; intercept the call; and perform the search in FAST ESP. Then, intercepting the K2 results list processing calls, deliver the FAST results to the CMS that thought it was talking with Verity. And we did it in less that 20% of the time FAST wanted to index a generic HTML-bases web site.

On the other hand, at LucidWorks we frequently have 5-day engagements to set up the Solr and LucidWorks Search; index the user's content; and integrate results in the end user application. I think for most engagements, other Solr and open source implementations are comparable. 

Let me ask: which was the more "expensive" implementation?

Open Source Search Myth 1 - Enhancements by Committee

This is part of a series addressing the misconception that open source search is too risky for companies to use. You can find the introduction to the series here; this is Part 1 of the series; for Part 2, click Potentially Expensive Customizations.

Part 1: Enhancements Subject to Committee

The original article back on LinkedIn called out one 'flaw' of open source search the belief that updates and improvements were made only on a timetable selected by the community - presumably the committers.  

One of the hallmarks of Apache open code projects is that, when you make a change or make an enhancement to the code, you submit the changes back to the Apache project. 

My employer, LucidWorks, enhances Solr, and we push back changes we make for consideration of the entire Solr community. Many of these changes are accepted and become part of Solr - almost all because of demand/need we see in our commercial customers, which helps everyone. 

Occasionally, a customer has a specific need and asks us to develop capabilities that are not part of the standard release. Sometimes the enhancements are on the Apache project plan; and sometimes they are unique. In any case, we create the enhancement and submit them for consideration in the standard Solr trunk. Once we’ve done so, anyone can download our enhancements and use them. And, as we do at LucidWorks, anyone can write enhancements for themselves and make them available.

Compare this to commercial search vendors who update on their own (typically unpublished) schedule; and no one can add a feature on their own. The vendor decides, and the consumer can only hope. And you pay upward of 20% of the list price every year in anticipation of the change you hope for but cannot add on your own.

And no matter what happens to Solr, our customers have the source code to self-support forever - no involuntary forced conversion. 

Why LucidWorks? And Why Now?

Big news for us here at New Idea Engineering. After 16 years as an independent search technology consulting company, we've become part of LucidWorks effective December 1, 2012.

For years we've focused on both the business and technology of search.  We've provided vendor neutral consulting services to large and small organizations. We've worked with search platform companies to help tune their product capabilities and their message; we've helped companies implement enterprise search from 'the usual (vendor) suspects'. We've provided business best practices, data audits, and implementation overview for dozens of companies for most of our time as an independent company.

As you know, the market for enterprise search has changed over the last several years. Verity then Autonomy, FAST, Endeca, Exalead, ISYS, and more have been acquired by large companies with varying levels of success. With these acquisitions the products have morphed to fit into the new owners' world view, we've politely referred to shift in focus as being "distracted". Google, one of the few non-acquired engines, got into the market with a low-cost entry which has enjoyed great acceptance; but as the market changed, Google has started raising its prices for the nifty yellow box.   And while they pursue laudable offerings like phones, tables, Google Glass and self driving cars, cloud computer and simultaneously retool their ad model for the mobile world, it's fair to say that even their enterprise offerings are potentially distracted at times.

Sure, if your company has a typical use case for search, there's an engine or appliance for you.  But so many complex projects we've seen are atypical, almost by definition.  These high-end projects are no longer efficiently served by commercial sector.  Many projects have turned to Open Source offerings, but not out of cost savings as you might think, but out of a desire to have extreme control and flexibility, and not be tied down by vendor meddling and license nit-picking.

Over the same period, more and more people have realized that the need to understand and manage 'big data' is taking off. In fact, search is the interface of choice to find content in big data repositories. 

It's been about 10 years since we did our first project based on Lucene, the basis for nearly all modern open source search engines today. Since then, the capabilities of open source search have increased to the point where we honestly think Solr may be the best search platform available on the market today. 

We didn't call what we did with Lucene back then 'big data', but that's really what it was. Scalable, controllable, flexible, powerful... and open! And free for the taking - and modifying. Just add programmers.

A few years back, Lucid Imagination was started to provide that support, along with training and an easy to use interface that lets business owners - not just developers - use Solr search.  We've called them "the RedHat of Open Source Search".  Now, Lucid Imagination has become LucidWorks, and it is set to be the best way to search web, file, and database content, with extreme control, and of course with big data.

A few months ago we spoke with Lucid CEO Paul Doscher about upping our contract with them, and about where they were going, and it just made sense to us at that time to join a bigger team.

While we're committed to success at LucidWorks, we'll continue to use our blog to discuss all aspects of enterprise search – vendors, tools, technologies, events, and trends.  Unlike our days at past search companies, this one is based on an open platform so we'll be able to share a lot more as we move forward.

We hope you'll find our posts interesting, helpful, and engaging. Let us know how we're doing.

 

Deep Solr in London and New York

Last week I had the pleasure of conducting a workshop at the recent Enterprise Search Summit on open source tools including Solr, Lucene, and some of the commercial products based on these tools. To a lesser extent we also covered ElasticSearch, SearchBlox, Alcove9, and a few other platforms, as well as a number of open source and commercial tools that support enterprise search.

One thing many of the attendees had in common was that they had been experimenting with Lucene/Solr for a while, but many were skeptical that they were ready do dive into a deep project on their own.

While that sentiment is no problem for me - after all, we provide services around implementing both open source and commercial search to our customers - I know many companies want to have expertise in-house

For those of you who are looking for those skills, you might be interested in a post I just saw from LucidWorks, They are offering a developer course titled  'Everything you always wanted to know about Solr' in both New York City and in London England during November. If you've been experimenting with Solr in-house (or on your own) for a while now, and you're ready to move to the next level, you might give some thought to registering for one of these classes.

It will cover the usual Solr topics, but also replication, sharding, and all the things you need to know to really use Solr in production search. Take a look and see if it's right for you.

 

 

 

 

Changes at Lucid Imagination

This morning Lucid Imagination announced that it is changing its name and simplifying its product family.

Effective today, Lucid Imagination becomes LucidWorks. The company remains the place to go for the highest quality open source search platforms, with their product offerings falling under the moniker LucidWorks Product Suite. The product suite is structured in to areas - LucidWorks Search and LucidWorks Big Data. 

LucidWorks Search is available on premises on in the cloud (Amazon and Azure); while LucidWorks Big Data is currently on premises or Amazon. LucidWorks strives to provide a more advanced feature set than the open source version of Solr. For example, the LucidWorks Search product release includes a user interface for creating and managing collections; click-through relevancy boosting; and better security implementation than that found in the open source release. Dealing with LucidWorks provides you features that you'd expect in an enterprise search platform well ahead of the availability in the open source release. LucidWorks continues to sell services and training for Solr for those of you who don't need the advanced capabilities in LucidWorks' products.

Finally, LucidWorks is launching a new support site SearchHub, which will take the place of the existing devZone Community Portal. It will be up in the near future, but you can sign up to be notified as soon as logins are available.

LucidWorks is the way to go for fully supported Solr.

 

 

Search appliance 'blues'

Over the US Independence day holiday many of us learned that Google is dropping its entry-level search box, the Google Mini. This comes as part of 'summer cleaning', the Mini being dropped with a number of other services and products that are just not hot enough to support the effort. (The one I'll really miss? iGoogle.) Google hasn't provided much information on how successful the GSA 'Blue' has been, but with a price point between $3K US and $10K US I imagine they moved a bunch of them to customers with simple search requirements. 

I think it may have Steve Arnold who said recently that the Google pubic web search and its advertising sales accounting for something like 96% of the company's revenue, so I don't think too many Googlers are upset about losing a small slice of a small slice of revenue. Heck, Mini proficts probably don't even pay the fuel bills for a weekend flight to Europe for the Google 767.

The impact? Well, back then the Mini was new and it was big news. Heck, the bigger  models were even better at not too much more money. Still, enterprise search was an expensive proposition then. Lucene was pretty new and quite rough around the edges; FAST, Exalead and Endeca were selling for upwards of $250K, and needed at least that amount of money to actually get them to work. Google Site Search was there; but not many other enterprise search products were around for that price.

A funny thing happened in the new century. Now enterprise customers are more demanding about search. The GSA - even the larger models - is generally well-received at first. At least as long as the 'Powered by Google' icon is visible. We had one customer tell us that just licensing the Google icon would solve most of his user complaints. And Verity's Andy Feit proved it statistically a year or two later. (Have a look at our post last year 'It's not Google unless it says it's Google'.)

But over time, even when content and user query activity remains about the same, people become increasingly frustrated using the GSA. But will Google abandon the color yellow too? Steve Arnold has wondered on LinkedIn whether the larger Google appliances are going to see the same fate soon. 

The problem isn't that it's an appliance. It's the closed system that people are turning away from. In the enterprise, you can't use the cool techniques that Google uses to generate psychic results on the internet. In the enterprise, managers know what content to boost; Metadata? Fielded search? Boost based on content? Not in the blue (or yellow) world. 

Still, I think Google and the GSA provide pretty darned good value for a certain part of the market. If your data is pretty decent; if you're serving highly interliked web and PDF content; if your data needs are not too demanding - GSA may be the solution you want. But before you spend money blindly, do what you do with any product you buy - verify it works in your environment. And as with any enterprise search platform, allocate a budget to run it properly after roll-out.

Yes, search has changed. Really good low-cost options are available. Where? Well, in addition to Google's site search offerings, there's Lucid Imagination's cloud and on-premises solutions; and some other darned good offerings based on open source: Flax - SearchBlox - and more.

What do you think? Is the loss of the Mini giving you the blues? 

 /s/Miles

(With thanks to Karan!)

Microsoft picks Lucid Imagination for Azure Search

Today Microsoft and Lucid Imagination announced an agreement to provide search on Azure. 

[Update on availability and pricing]

EE Magazine is reporting that the Microsoft Azure Marketplace is now offering cloud search from Lucid for its Azure customers. This extends the relationship between Microsoft and Lucid we wrote about last month, and brings the offer of Solr-based search to Microsoft's customers. Lucid has some details about the service on its site, with an invite to come back at 2PM Pacific Time for more details. 

The product, LucidWorks Cloud for Microsoft Azure, looks and feels like the LucidWorks Enterprise product, but without any need to have local servers to host the search service. It includes the REST API; but runs without any need to access the various configuration files normally associated with Solr.

We're quite familiar with the concept of hosted search, having started SearchButton.Com in 1998. Nevertheless, this is a big move forward for Lucid Imagination and a big move for Microsoft's Azure customers.

What does this say about Microsoft's own search products? It seems pretty clear that enterprise search from Microsoft means SharePoint; and search outside of SharePoint is something other companies can provide. Do you suppose this endorsement of Lucid's Solr-based product offering is the beginning of a more open Microsoft?

What do you think?

 

 

 

 

 

 

Lucene Revolution: MS talks of being more open

Lucene Revolution: MS talks of being more open

At yesterday’s kickoff of Lucene Revolution 2012, Lucid CEO Paul Doscher introduced Gianugo Rabellino, Microsoft's Director of Open Source Communities. Gianugo said little about search per se, but he did confess to having been a fan of Lucene and Solr for a while now. In his talk, he told the audience that Microsoft has changed with respect to open source, and he went on to tell everyone how they have become more involved in open standard like HTML5, CSS3; and in hardware specifications like USB. He went so far as to say 'Microsoft's survival depends on open source software'.

News to me, and perhaps to others in the room, was the extent to which Microsoft is supporting a number of open source products and languages. Gianugo reported that Linux is now a 'first-class guest operating system' on Microsoft HyperV; and provides support for PHP, Ruby on Rails, node.js and other projects on Azure (and presumably for 'on premises' systems).

A number of folks from large commercial organizations seemed to appreciate the news about Microsoft's shift towards supporting open source; but a number of the open-source folks in the room felt this offered little new, and some even felt it was an unrelated 'sales pitch'. Even though we are Microsoft partners, I'm glad to see more support for open source products like PHP and Linux.

The finniest part of the talk came as Gianugo was describing how SharePoint data was easily accessible to other non-Microsoft' search platforms. An attendee asked if he felt there was a role for other platforms to be used as the primary engine for search in SharePoint; as he paused to craft a reply, Paul Doscher (loudly) pronounced his belief that there was, much to the pleasure of the crown.

There was not much else in the way of Microsoft news; but it was interesting to see how many people and how much effort Microsoft is putting into open source projects.

 

 

Is Microsoft joining the Lucene/Solr dance?

Lucene Revolution is only 10 days away, and if you're not already planning on being in Boston, today's a great time to register.

Why be at the 3rd annual Lucene Revolution, Lucid Imagination's open source conference? Several reasons:

• Open source search is hot, and Lucene/Solr is better than ever;
• Lucid Imagination is just introducing their LucidWorks Enterprise 2.1 release;
• Paul Doscher, recently of Exalead, is the new CEO and keynote speaker; and
• Microsoft's Gianugo Rabellino is speaking about Lucene, Azure, and OSS.

Yes, you saw it here. A Microsoft Azure guy is speaking right after Paul Dorscher Wednesday moring at Lucene Revolution. Has Microsoft caught the drift of the market towards Lucene/Solr in search, big data, and the cloud? Even search pundit Steven Arnold posted a few days back about Microsoft and Linux. Strange bedfellows perhaps, but there it is. 

So yes, I think if you can find any way to get to Boston in a week, I'd say do it. See you there!

 

Lucid positioning for success in open source search

Lucid Imagination is the Redwood Shores company whose charter is to market advanced products based on the open source Lucene/Solr project. With a large number of the Apache project committers in its employ, they have the technical wherewithal to succeed, but they never really screamed 'business success' - until recently.

In December of last year, Lucid's board hired Paul Doscher as CEO, presumably to make Lucid's premier product, LucidWorks Enterprise, a success in the marketplace. He seems to have been a good choice: he was the guy at Exalead who built a first-class organization in the US; and who was as responsible as anyone in making Exalead an attractive acquisition last summer for Dassault. 

Now, just a few months later, Paul has hired Mike Moody, formerly of Spigit, to be Lucid's EVP of Development. I had the opportunity last year to work with Mike at Spigit, an up-and-coming product in its own right, and I suspect having Mike on-board will have a positive impact on Lucid's products and services in the coming years.

It's tough to break into the commercial search market, but it seems to me that Lucid is serious about being a leader in the space - soon.