« Search Terms | Main | Today's Search Term: hybrid search »

August 30, 2010

Today's Search Term: Stemming

stemming

Related Terms:  lemmatization, normalize
Search engines use stemming as a means to
determine the root of a given written word. Using a program or algorithm all of the affixes to a word (prefix and /or suffix in the English language) are removed, leaving the root word. By implementing the rules of the given language obstacles such as third- person singular present (as cries is of the verb cry) in the English language can be accurately indexed.


Stemmers become harder to design as the rules of the target language becomes more complex. For example, some languages have more verb and pronoun forms. Other languages do not always have clear word breaks between each word, and you can't do stemming until you've isolated the words!

 


TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c84cf53ef0133f2ee2358970b

Listed below are links to weblogs that reference Today's Search Term: Stemming:

Comments

Also normalization to increase search matching using de-pluralization - find the singular form of plural words step, and normalization of gender forms to a single form.

S-removal is one useful technique in de-pluralization. Where stemming might take accountant and accounts to the same root word, in the HR field this would be semantically incorrect causing false positive matches.

In some European languages, word endings change to determine a 'male' or 'female' job title.
===
Excellent points Jon... stemming properly is much more than just removing the trailing "S"! Makes you glad that really good search engines handle all that for you!
s/Miles

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.