I started the draft for this post a week ago yesterday morning, but between interruptions and such, had not finished it by late afternoon, so I put it away with the intent to finish it later. At 530PM or so, I got a text message from a very good friend of mine who happens to work for FAST back east, telling me about the layoff. When we spoke a few minutes later, I heard that just about all the former FAST folks we knew at Microsoft – sales folks, partner reps, and systems engineers, East Coast, Midwest, and west, were gone. I didn’t confirm it, but I’d imagine some of the same sorts of folks in Europe are in the same boat. I know some other folks really well, and messaged them to give them my support and offer to do whatever I could to help them. Out of a dozen good friends we’ve worked with, only one seems to have his job today.
Layoffs are vicious on the people and on the company; but honestly this event doesn’t really change the future I had imagined for FAST and Microsoft. It accelerates the timeline of course; but no matter how great these people are, sales folks are not the heart of the product. And Microsoft still has good sales guys who can fill in and solve problems for customers.
A few weeks back, one of the FAST folks I know told me how different it was selling FAST as part of a Microsoft sales team. Apparently, the Microsoft lead sets the agenda, decided what products to present, and was basically in charge of the sale. That makes business sense, of course; but the former FAST sales guy said selling with Microsoft was not like the old FAST days – taking on the world, with a product that was great for so many demanding customers, if not the price/performance leader for many kinds of ‘average’ search applications. FAST Search for SharePoint is going into markets that need modern features and capabilities, but Microsoft just wasn’t going into accounts that needed heavy duty, industrial strength search.
So, is this layoff an indication that Microsoft has decided to write off its investment in FAST Search and Transfer?
I’d have to say the answer is ‘no way’. I’m guessing that Microsoft saw that MOSS 2007 was not industrial strength search; customers were unhappy, and Microsoft were seeing companies like FAST, Google, Exalead, Attivio and Lucid Imagination selling successfully on the SharePoint 2007 platform. Ironically, I don’t think they saw much of Autonomy, which has such a tight laser-like focus on eDiscovery.
You can say what you will about Microsoft, but when their customers start to complain, Microsoft tends to get moving. They are not always successful out of the gate, but they stick with it. And Microsoft went shopping.
FAST had some real stand-out capabilities. The first one we usually talk to customers about is predictable and massive scalability. I don’t think we’ve ever seen a data problem too big for FAST ESP. It may take a bunch of servers, but FAST could tell you just how many you needed for a given data size and query volume. That is important to Microsoft, because their vision for SharePoint 2010 and beyond is as the content management repository of choice for some really large companies with really big data.
Another useful feature is a visible and well-defined indexing pipeline which makes it really easy to fix up really nasty data. If you needed to augment metadata with call-outs to external sources, or just needed to do some custom markup, the pipeline made it easy. In many technologies – I’m thinking here of the Old Verity K2 approach – you had to process “bulk insert files” after the crawler was done, but before the index was built. Sometimes that was very messy and hard to debug.
The FAST pipeline was never very well documented – but there were examples, and simple pipeline stages were pretty easy to write. We’ve even documented the use of a pipeline stage written as a Windows BAT file – although we don’t recommend it for production environments because of performance issues! A powerful pipeline architecture is a great way to consolidate social content with enterprise search.
FAST also had a powerful filtering capability in FQL and a structure that really enables personalization (see the console, below). Facets, tagging, filtering, and relevance were all built deep into the technology; and while it may not be very easy to implement, it was great to behold when a project was complete. All of this really facilitates the inclusion of ‘social elements with search.
The next capability that we really liked was not unique to FAST – in fact, I think it was first implemented by Endeca: interactive consoles for both IT staff and for the business line owner. No longer did you need a developer to tweak the relevancy – the business line owner could do it interactively. Add a new data source? An IT person logs into the console, point and click, and bingo. You did have to plan in advance for some of this; but it was possible to do without scripts and hacks. In SharePoint, IT and business management is console driven; so FAST was a win here as well.
One final thing Microsoft got from FAST is engineers who understand the challenges of search over really big data. I don’t think it’s a coincidence that Microsoft located its search technology research center in Norway. No doubt these folks are working on all aspects of search and usability, from the desktop all the way to Bing. And what these folks create will migrate into Microsoft’s key platform: SharePoint.
When Microsoft acquired FAST, they got all of these bits of technology and more: a vision of future search, including the capability to manage both the indexing process and the search process with powerful graphical tools. The delivery of these capabilities comes in Content Transformation Services (CTS) and the Interaction Management Studio (IMS). Together, these provide even finer control for IT/developers and business-line owners to manage search using graphic, interactive tools – think of Visio. Add a new data source? Drag and drop it into the system. Federate content from the web? Drop in the federation tool, set a few parameters and it’s done. It was darned near a web part already!
As of the acquisition, SharePoint 2010 was well into development, so today Microsoft currently offers three products: Microsoft Search Server – the MOSS 2007 replacement; FAST ESP (packaged as FSIS and FSIA); and FAST Search for SharePoint. The technology in this hybrid offers a clue for the direction Microsoft is going with the FAST technology. In fact, the next SharePoint may have only ‘basic search’ and what we’ll recognize as FAST.
So while it’s sad to see good folks lose their jobs; and while it may look like a series of poor decisions form the outside, I expect Microsoft is going to benefit from all that money spent up in Norway a few years ago. FAST, the huge, complex, expensive, and quirky search engine may not ever be seen again. That’s not where the mass market is, the mass market is where Microsoft makes its bread and butter.
But search to Microsoft will be SharePoint; they will meet the needs to a majority of the companies that need to find content; and they will push the envelope in search for the masses. There will also be ‘search’ companies that will serve the needs of companies that have demanding search needs. Lucene/Solr is very well positioned for that. But there are others. And Microsoft will take the middle masses, the sweet spot that needs flexible, pretty darned good search that can be managed easily for diverse corporate content repositories.
