81 posts categorized "FAST Search & Transfer"

November 08, 2011

Are you spending too much on enterprise search?

If your organization uses enterprise search, or if you are in the market for a new search platform, you may want to attend our webinar next week "Are you spending too much for search?". The one hour session will address:

  • What do users expect?
  • Why not just use Google?
  • How much search do you need?
  • Is an RFI a waste of time?   

Date: Wednesday, November 16 2011

Time: 11AM Pacific Standard Time / 1900 UTC

Register today!

October 25, 2011

What search platform is best? Workshop at KMWorld

Next week in Washington DC, InfoToday runs their Fall enterprise search conferences - KM World, Enterprise Search Summit, SharePoint Symposium, and Taxonomy Boot Camp.. whew! Monday - Halloween Day! - I am giving a workshop at the conferences with the somewhat vague title 'Enterprise Search Technologies'.

What I'll be talking about is an overview of the platform vendors, with some detail on strengths and weaknesses of the vendors; and a drill down into what you need to do before you call the vendors (if you value your time).

You can still sign up for the workshop for $295US or the entire conference for a bit more; see you in DC in a week!


August 19, 2011

7 Reasons Why the Autonomy Acquisition Makes Sense for HP

With HP's annual meeting coming up this week CEO Leo Apotheker is looking for a way to put some lipstick on his new baby: he wants to acquire Autonomy. PC sales are declining and Apple is eating their lunch in pads AND buying the old HP Cupertino campus. Apparently they will keep the server divisions.Bill and Dave must be beside themselves.

But wait! The knight in shining armor from England is here for the rescue. Autonomy announced just last month they were 'likely to beat expectations' for their fourth quarter. They have enjoyed huge success in eDiscovery.. they claim to 'own search'.. and sponsors its own football team, Tottenham Hotspur FC.

At first blush, the two seem like strange bedfellows. HP, the leader in PCs worldwide, with a well acknowledged reputation for management style - the HP Way. Autonomy has a rep like that of a great professional footballer: skilled and aggressive, known to occasionally feint an injury for his benefit after a rough hit; and perhaps booked for a card a few times every season. But still, a champ and damn good at what he does.

When you think about it, the deal isn't as odd as it may seem. Consider:

1. PC sales are down; they are being replaced by smart devices like Android and iOS devices. WebOS? Too little, too late, too expensive.

2. HP acquired EDS a couple of years back to compete with IBM in consulting services. Among enterprise class search engines, 'consulting services' certainly comes to mind - bring lots.

3.  IT managers, not C-level execs, buy PCs. Chief Risk Officers buy eDiscovery and compliance solutions so he and his fellow executives can sleep well at night. Rarely is there a spending freeze on compliance tools.

4. HP is rumored to be spinning out its PC products, but it seems the server business is staying in Palo Alto. IDOL likes lots of really big servers: HP wins on all three: servers, software, and consulting.

5. Iron Mountain: records management. See (3) above.

6. We used to speculate that SAP might be a buyer for Autonomy at some point in time. Now, the HP chief is Leo Apotheker, who came to HP from... SAP. Coincidence?

7. 'The cloud': HP needs one, HP gets one - enough said.

Still, it may not be a 'made in heaven' match.

1. You may recall that when Microsoft acquired FAST, they soon found some odd accounting issues - something about numbers overly optimistic, and booking revenues prior to firm orders were received. Until you dig deep into the details, there may be no way to know for sure until the deal is sealed.

2. And as mentioned earlier, personnel and policies seem relatively incompatible.

3. HP is along time Microsoft partner, uses SharePoint extensively, and just rolled our FAST search on its public facing web site. That should be interesting.

4. Finally, I seem to recall that when Autonomy bought their larger competitor Verity, part of the rationale for Autonomy being the surviving company was the cost of annual SarBox compliance if they were a US company. HP must be willing to pick up the tab, because I sure can't see Palo Alto moving to Tottenham.

Stay tuned, and let us hear what you think!


Full disclosure: I was an HP employee for 10 years, and during my time there was fortunate enough to be the PC support guy for Bill Hewlett, Dave Packard, and Dave Packard Jr. HP is not the same company as it was when they were running the show. I recommend Mike Malone's "Bill and Dave" as a great explanation of what made HP great for 40 years, and successful for 72 years so far.



August 18, 2011

HP looking at acquiring Autonomy

The consolidation continues. Since the FAST acquisition by Microsoft a few years back, people have asked who we thought might acquire Autonomy; and we always decided they were almost too big to be a target, that Autonomy would be the surviving company.

It seems that HP, which apparently just rolled out its FAST implementation to its public web site, is the surprising answer.

Quick take: in a way, it makes sense in a few ways:

1. Hardware is getting cheaper and cheaper; being the largest vendor in a shrinking market isn't where you really want to be.

2. HP has long been moving to a services delivery company - look at the 2008 acquisition of EDS. If there is a search technology that typically has huge implementation projects with teams of expensive consultants, it's Autonomy.

3. Company have Chief Risk Officers, who handle compliance, have the budget to buy software to keep fellow executives out of trouble with the government. What better prospects to have?

Wow.. I'm still a bit shocked; I feel like I must have overslept and that I'm having a strange dream. But my dog seems pretty sure it's real.

Stay tuned, and let us know what you think!


August 09, 2011

So how many machines does *your* vendor suggest for 100,000,000+ document dataset?

We've been chatting with folks lately about really large data sets.  Clients who have a problem, and vendors who claim they can help.

But a basic question keeps coming up - not licensing - but "how many machines will we need?"  And not everybody can put their data on a public cloud, and private clouds can't always spit out a dozen virtual machines to play with, plus duplicates of that for dev and staging, so not quite as trivial as some folks thing.

The Tier-1 vendors can handle hundreds of millions of dcs, sure, but usually on quite a few machines, plus of course their premium licensing, and some non trivial setup at that point.

And as much as we love Lucene, Solr, Nutch and Hadoop, our tests show you need a fair number of machines if you're going to turn around a half billion docs in less than a week.

And beyond indexing time, once you start doing 3 or 4 facet filters, you also hit another performance knee.

We've got 4 Tier-2 vendors on our "short list" that might be able to reduce machine counts by a factor of 10 or more over the Tier-1 and open source guys.  But we'd love to hear your experiences.

August 02, 2011

Connecting Google to SharePoint 2010: White Paper

Ba_insight_logo NIE partner BA Insight will soon be releasing a white paper highlighting key differences between SharePoint 2010 search and the Google Search Appliance.

The early draft we've seen of Google & Microsoft Enterprise Search Product Comparison, and can talk about some of the discussion points.

Update: The white paper is now available! Get it now.

Generally, the research paper discusses the following topics:

1. Content: Crawling, indexing, security, and connectors

2. Query processing: Manual and automatic relevance tuning; actionable results; and layout design

3. Vendor 'intangibles': Maintenance, support, vendor stability, licensing and the partner eco-system

BA-Insight is a large Microsoft partner, and their DNA reflects it. But many customers use Google Search Appliances with SharePoint, and this comparison is nicely done. Keep an eye on their site or stay tuned for updates here when the research paper is available.

And let us know what's on your mind with respect to enterprise search - leave a comment!



July 15, 2011

Musings on FAST and Microsoft

I started the draft for this post a week ago yesterday morning, but between interruptions and such, had not finished it by late afternoon, so I put it away with the intent to finish it later. At 530PM or so, I got a text message from a very good friend of mine who happens to work for FAST back east, telling me about the layoff. When we spoke a few minutes later, I heard that just about all the former FAST folks we knew at Microsoft – sales folks, partner reps, and systems engineers, East Coast, Midwest, and west, were gone. I didn’t confirm it, but I’d imagine some of the same sorts of folks in Europe are in the same boat.  I know some other folks really well, and messaged them to give them my support and offer to do whatever I could to help them. Out of a dozen good friends we’ve worked with, only one seems to have his job today.

Layoffs are vicious on the people and on the company; but honestly this event doesn’t really change the future I had imagined for FAST and Microsoft. It accelerates the timeline of course; but no matter how great these people are, sales folks are not the heart of the product. And Microsoft still has good sales guys who can fill in and solve problems for customers.

A few weeks back, one of the FAST folks I know told me how different it was selling FAST as part of a Microsoft sales team. Apparently, the Microsoft lead sets the agenda, decided what products to present, and was basically in charge of the sale. That makes business sense, of course; but the former FAST sales guy said selling with Microsoft was not like the old FAST days – taking on the world, with a product that was great for so many demanding customers, if not the price/performance leader for many kinds of ‘average’ search applications. FAST Search for SharePoint is going into markets that need modern features and capabilities, but Microsoft just wasn’t going into accounts that needed heavy duty, industrial strength search.

So, is this layoff an indication that Microsoft has decided to write off its investment in FAST Search and Transfer?

I’d have to say the answer is ‘no way’. I’m guessing that Microsoft saw that MOSS 2007 was not industrial strength search; customers were unhappy, and Microsoft were seeing companies like FAST, Google, Exalead, Attivio and Lucid Imagination selling successfully on the SharePoint 2007 platform. Ironically, I don’t think they saw much of Autonomy, which has such a tight laser-like focus on eDiscovery.

You can say what you will about Microsoft, but when their customers start to complain, Microsoft tends to get moving. They are not always successful out of the gate, but they stick with it. And Microsoft went shopping.

FAST had some real stand-out capabilities. The first one we usually talk to customers about is predictable and massive scalability. I don’t think we’ve ever seen a data problem too big for FAST ESP. It may take a bunch of servers, but FAST could tell you just how many you needed for a given data size and query volume. That is important to Microsoft, because their vision for SharePoint 2010 and beyond is as the content management repository of choice for some really large companies with really big data.

Another useful feature is a visible and well-defined indexing pipeline which makes it really easy to fix up really nasty data. If you needed to augment metadata with call-outs to external sources, or just needed to do some custom markup, the pipeline made it easy. In many technologies – I’m thinking here of the Old Verity K2 approach – you had to process “bulk insert files” after the crawler was done, but before the index was built. Sometimes that was very messy and hard to debug.

The FAST pipeline was never very well documented – but there were examples, and simple pipeline stages were pretty easy to write. We’ve even documented the use of a pipeline stage written as a Windows BAT file – although we don’t recommend it for production environments because of performance issues! A powerful pipeline architecture is a great way to consolidate social content with enterprise search.

FAST also had a powerful filtering capability in FQL and a structure that really enables personalization (see the console, below). Facets, tagging, filtering, and relevance were all built deep into the technology; and while it may not be very easy to implement, it was great to behold when a project was complete. All of this really facilitates the inclusion of ‘social elements with search.

The next capability that we really liked was not unique to FAST – in fact, I think it was first implemented by Endeca: interactive consoles for both IT staff and for the business line owner. No longer did you need a developer to tweak the relevancy – the business line owner could do it interactively. Add a new data source? An IT person logs into the console, point and click, and bingo. You did have to plan in advance for some of this; but it was possible to do without scripts and hacks. In SharePoint, IT and business management is console driven; so FAST was a win here as well.

One final thing Microsoft got from FAST is engineers who understand the challenges of search over really big data. I don’t think it’s a coincidence that Microsoft located its search technology research center in Norway. No doubt these folks are working on all aspects of search and usability, from the desktop all the way to Bing. And what these folks create will migrate into Microsoft’s key platform: SharePoint.

When Microsoft acquired FAST, they got all of these bits of technology and more: a vision of future search, including the capability to manage both the indexing process and the search process with powerful graphical tools. The delivery of these capabilities comes in Content Transformation Services (CTS) and the Interaction Management Studio (IMS). Together, these provide even finer control for IT/developers and business-line owners to manage search using graphic, interactive tools – think of Visio. Add a new data source? Drag and drop it into the system. Federate content from the web? Drop in the federation tool, set a few parameters and it’s done. It was darned near a web part already!

As of the acquisition, SharePoint 2010 was well into development, so today Microsoft currently offers three products: Microsoft Search Server – the MOSS 2007 replacement; FAST ESP (packaged as FSIS and FSIA); and FAST Search for SharePoint. The technology in this hybrid offers a clue for the direction Microsoft is going with the FAST technology. In fact, the next SharePoint may have only ‘basic search’ and what we’ll recognize as FAST.

So while it’s sad to see good folks lose their jobs; and while it may look like a series of poor decisions form the outside, I expect Microsoft is going to benefit from all that money spent up in Norway a few years ago. FAST, the huge, complex, expensive, and quirky search engine may not ever be seen again. That’s not where the mass market is, the mass market is where Microsoft makes its bread and butter.

But search to Microsoft will be SharePoint; they will meet the needs to a majority of the companies that need to find content; and they will push the envelope in search for the masses. There will also be ‘search’ companies that will serve the needs of companies that have demanding search needs. Lucene/Solr is very well positioned for that. But there are others. And Microsoft will take the middle masses, the sweet spot that needs flexible, pretty darned good search that can be managed easily for diverse corporate content repositories.

July 12, 2011

Inopportune time for a page not found: Microsoft

There has been an active discussion on the LinkedIn Enterprise Search Professionals Group since the news last week that Microsoft has laid off a number of sales and system engineering folks, at least in North America. Many are wondering if this marks the end of the FAST products, or whether Microsoft is writing off their $1.2B investment to acquire FAST a few years back.

I think the answer to both of these is 'no way'. Much of the FAST technology found its way into FAST Search for SharePoint, a hybrid between Microsoft search and FAST ESP. Microsoft have released two exciting new capabilities in FSIS (CTS and IMS), and they continue to run the Microsoft Research Center in Norway, where many of the FAST engineers are active participants.

All of that said, this morning I was surfing the interweb for some Microsoft proficiency tests, so I went to Google and searched

microsoft enterprise search proficiency

The first organic result looked promising:

But when I click on the link, I got this on my browser:

Not_found Oops!

Now we've all had bad links on our web sites, and even occasional outages - but given the concern some people have about the layoffs of FAST sales reps, this is a bit of an inopportune time, yes?

All of my browsers produced the error, and using the old telnet trick to port 80 or the WGET utility showed the page really did exist with no obvious re-directs or errors. So I drafted our CTO Mark Bennett to help, and after some analysis, he found the problem - in the scripts on the page: bad scripting is the culprit.

Yep - with apologies to the Buggles, JavaScript killed the Microsoft page. Disabling scripts in the browser lets the link work just fine. It just shows you that, even with all the resources in the world, lack of attention to detail will get you!

I'll be in LA at the Microsoft Worldwide Partner Conference tomorrow; it should be interesting!




July 06, 2011

Breaking news: FAST folks laid off from Microsoft

Just learned that most of the FAST people we work with here in California and across the country have been laid off by Microsoft, apparently effective immediately. This is the team that was responsible for selling the FAST ESP products - FSIS and FSIA - as well as working with the Microsoft sales teams on Fast Search for SharePoint (FS4SP). Funny, I was just drafting a blog post today on 'the future of FAST' and I'm glad I hadn't finished; I never would have guessed this at all.

Microsoft Worldwide Partner Conference next week; be interesting to see what they have to say. Stay tuned..


May 31, 2011

It's not Google unless it says it's Google

A few years back, one of our customers told us that, if he could just license the 'Powered by Google' icon, he was sure most of the users would stop complaining. Not long after that, we heard that our friend Andy Feit, who was VP of search at Verity, hired a marketing research team to compare the quality of search engines when one was "Powered by Verity" and the other was 'Powered by Google'. Andy found that people thought the Google results were significantly better - even though both test cases were, in fact, powered by Verity. The mere presence of the Google icon seemed to make people think the results were better.

At the recent ESS, a woman from Booz & Company talked about their previous enterprise search experience involving Google. A few years back, Booz used FAST ESP on SharePoint 2003 and it simply sucked. Users asked for Google by name. When they upgraded to SharePoint 2007, Booz gave the users what they wanted: they went with a Google Search Appliance. The trouble was that they built a custom interface with a generic search button. Users' responses? "Search still sucks - why don't we just use Google?" even as they were using Google!

This can teach us a number of lessons:

1. Analyze what you need search to do for you before you buy it.

2. Understand how your content and search platform play together.

3, It ain't Google unless you tell your users it's Google.

By the way: in 2010 Booz rolled out FAST Search for SharePoint, and it seems that the results are a bit better now that they understand their search requirements and the nature of their content and metadata.