80Legs - The Mercenary Spider

By Carl Grimm, New Idea Engineering

80Legs, a Houston based company that runs a spider-for-hire launched its private beta yesterday.  80legs runs on a grid computing system provided by Plura Processing, a sister company.

80legs is slated to charge $2 per million pages crawled and $.03 per CPU hour used for analysis. They will soon bring online the ability to write custom code for page specific processes you wish to execute by means of their pageProcess() function.  Simply upload a .jar file and their system will run your code on the retrieved data.

While this launch falls slightly in the shadow of Amazon’s recent announcement of the availability of Elastic MapReduce, it serves as another confirmation of the impact and looming changes grid and cloud computing will have on search, both Internet and enterprise.

In a world where the width of train tracks is the same with as the wheels on Imperial Roman war chariots, I cannot help but think how search technology, who some may consider a mature technology, some who even consider a dead technology, has grown up constrained and shackled to the past.

I hope enterprise search vendors are taking note and rethinking their architecture and capabilities in the newfound land of massive computing power. I know that enterprise data security poses a daunting barrier to leveraging these capabilities. I am confident the tide will come as sentiments change and new generations bring their attitudes into IT and the business world as a whole.

Learn more about 80legs at http://www.80legs.com/

Comments

[Michael Christensen]

Don't count on enterprise search vendors are taking note and rethinking their architecture.
We can all hope.
==
True - but architectures that use easily customized indexing pipelines (like Open Pipeline, FAST, etc) are probably more easily tweaked by customers to take advantage of things like 80Legs... heck, if they've got the content, why not use it? Miles