« October 2017 | Main

1 posts from December 2017

December 04, 2017

Search Indices are Not Content Repositories

Recently on Quora, someone asked for help with a corrupt Elasticsearch index. A number of folks responded, all recommending that he simply rebuild the search index and move on.

The bad news turns out that this person didn't have any source documents: he was so impressed with what Elasticsearch did that he had been using it as his primary storage for content. When it crashed, his content was gone. This is not an indictment of Elasticsearch: it can happen to any complex software product whether Elastic, Solr or SharePoint.

In my reply, I told him how sorry I was for his loss, and suggested he get to work restoring or recreating his content. I even offered to call and tell him how sorry I was for his loss. 

Then I launched into what I really felt I needed to say - there for his behalf, and here for yours.  I suggested - no, actually I insisted - that you NEVER use ANY search index as your primary store for content. Let me be more specific: NEVER. EVER.

Some commercial platforms such as Solr and commercial software based on Solr (ie, Lucidworks) have a reasonably robust ability to replicate the index over multiple servers or nodes which provides some safety (I’m thinking SOLR Cloud here); others do not. But the replication is a copy of the INDEX, which is NOT your documents.

The search index is optimized for retrieval. Databases, CMS, file systems and other tech are for storage.

For one, I’m not sure any search engine stores the entire document of any type. Conceptually, most search indices have two ‘logical’ (if not physical) files 

One of these files you can think of as a database table with one row per document, with field values (Title, Author, etc). This file generally stores the URL, file name, database row as well, basically ‘where do I go to find this full document?’ - and maybe a few other field values.

The second file is a list of all the (non-stopwords) in all of your documents. The word itself is stored once, along with a list of byte offsets in the document where the word appears (multiple byte offsets, one for each instance of the word). It also has a pointer to all docs which have that word. Again: Stop words are generally NOT indexed, so they are usually not in the index.

(There is more detail in an older article on my website Relational Databases vs. Full-Text Search Engines - New Idea Engineering)

COULD you rebuild the full document? Well, depends on the search platform. In most platforms I've seen, it would be difficult because stop words are not even stored. Recreating a document that omits ‘the’, ’a’, ‘an’, ‘and’ etc. MIGHT be human readable but it is NOT the original document.

Secondly, not all search engine indices are replicated for redundancy. The assumption is that if you lose the file system where the content lives, you can still search; you just can't retrieve any documents until you restore the original content.

And some platforms do not give you a way to access the index, short of searching. And a search index is an index, not a repository.

Finally, some platforms are better at redundant failover of indices than others. If the platform you use is one of those that do not have redundancy BY DEFAULT.. like some very popular platforms - and you use that index as the primary data store for your doc and the index dies.. you’re what we used to call SOL - ‘sure outta luck’.

The moral of the story? DO NOT USE A SEARCH INDEX AS THE PRIMARY DATASTORE. Specific enough?