« Infonortics Search Engine Meeting in Boston April 27 | Main | Attivio sponsoring SearchDev dinner at ESS NY »

April 10, 2009

80Legs - The Mercenary Spider

By Carl Grimm, New Idea Engineering

80Legs, a Houston based company that runs a spider-for-hire launched its private beta yesterday.  80legs runs on a grid computing system provided by Plura Processing, a sister company.

80legs is slated to charge $2 per million pages crawled and $.03 per CPU hour used for analysis. They will soon bring online the ability to write custom code for page specific processes you wish to execute by means of their pageProcess() function.  Simply upload a .jar file and their system will run your code on the retrieved data.

While this launch falls slightly in the shadow of Amazon’s recent announcement of the availability of Elastic MapReduce, it serves as another confirmation of the impact and looming changes grid and cloud computing will have on search, both Internet and enterprise.

In a world where the width of train tracks is the same with as the wheels on Imperial Roman war chariots, I cannot help but think how search technology, who some may consider a mature technology, some who even consider a dead technology, has grown up constrained and shackled to the past.

I hope enterprise search vendors are taking note and rethinking their architecture and capabilities in the newfound land of massive computing power. I know that enterprise data security poses a daunting barrier to leveraging these capabilities. I am confident the tide will come as sentiments change and new generations bring their attitudes into IT and the business world as a whole.

Learn more about 80legs at http://www.80legs.com/


TrackBack URL for this entry:

Listed below are links to weblogs that reference 80Legs - The Mercenary Spider:


Don't count on enterprise search vendors are taking note and rethinking their architecture.
We can all hope.
True - but architectures that use easily customized indexing pipelines (like Open Pipeline, FAST, etc) are probably more easily tweaked by customers to take advantage of things like 80Legs... heck, if they've got the content, why not use it? Miles

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated, and will not appear until the author has approved them.