#soundex

I have recently been tempted to go "all in" into semantic analysis. I have played around with the Sphinx search engine, even wrote a plugin for it, checked out the logic for infixes (suffixes, prefixes), word forms, spelling, stemming, etc. Have to say, the code they have in there is a mess, but still fast. It's hard to make heads or tails out of a 28 000 lines cpp file. (yes, it's not a mistake 28k lines). Maybe they make it this way to stop others from playing around with the code?! What's the meaning of open source then. Anyway, for the curious minds, here are the results of my research for the past couple of days:

spelldump - great utility, comes with sphinx. Have a read here: http://sphinxsearch.com/docs/1.10/ref-spelldump.html

snowball - Stemming library with multiple steamers for romantic, germanic, northern and russian languages http://snowball.tartarus.org/ 

php-stemmer - http://code.google.com/p/php-stemmer/

word lists - All the english words dictionary (not really, but close enough) http://wordlist.sourceforge.net

soundex - a phonetic algorithm http://en.wikipedia.org/wiki/Soundex

metaphone - a better phonetic algorithm http://en.wikipedia.org/wiki/Metaphone

double metaphone - an even better (but way slower) phonetic algorithm http://en.wikipedia.org/wiki/Metaphone

levenshtein distance - the distance between two words http://en.wikipedia.org/wiki/Levenshtein_distance

ispell - Non-GNU, but part of GNU spelling and typographical error correctors. http://www.gnu.org/software/ispell/ispell.html

Soundex implementation in Java

  Application of Soundex

Soundexis a phonetic algorithm for indexing names by sound, as pronounced in English. The goal is for homophones to be encoded to the same representation so that they can be matched despite minor differences in spelling. A homophone is a word that is pronounced the same as another word but differs in meaning and spelling. For example-carat, caret, and carrot, or to, two,…

Most genealogy researchers are familiar with a standard census form. But some census records appear in other forms as well. Some of the United States Census years have been posted to Archive.org using Soundex cards. 

What is Soundex? You've likely seen it as an option in search. Soundex is a means of phonetically categorizing like-sounding names.  This can be useful in helping to find ancestors whose names have alternate spellings or mis-spellings in census records. The presumption is, if a name is mis-spelled or possibly a spelling has been chosen because the individual cannot spell; soundex will help place it in the same category as the correct spelling.

I won't go into the process, but you can read about it for yourself by clicking here. The easiest way to convert the surname you are looking for into soundex is this handy website here:

http://resources.rootsweb.ancestry.com/cgi-bin/soundexconverter

You will end up with a soundex number starting with a letter followed by three numbers, for example: Holman would equal H455.  You can use this code to search through soundex cards on Archive.org.

When you visit Archive.org, in the search box, type in Census Soundex and a year (1880, 1900, 1910, or 1920).  You can also enter the name of a state (full name--not abbreviated). You will be presented with a page of links to individual books of soundex cards.  Using the above example, lets say we want to find Michael Helman and his wife Margaret in the 1900 census.  I know he lived in Waukesha, Wisconsin, but I've tried searching and he doesn't show up (this is just an example).  

So I set my browser to Archive.org and type Census Soundex 1900 Wisconsin into the search box (keep in mind, not all states for all years are digitized yet). Now we need to find the right link. We know the soundex code for Helman is H455.  Archive.org typically titles the individual books using a dash after the soundex letter.  So I will hit "CTRL+F" on the PC or "CMD+F" on the mac and search for H-.  

When doing so, I see G-655 H. P.-H-142 and P-400 Elizabeth-P-425 Fredrick.  These are ranges of codes in alpha-numeric order. I need to find a range that includes H-455. At the bottom of the list (if there are many documents), are links to additional pages.  I'll keep clicking to the next page until my search finds the link I am looking for.

Finally I find it, the link is titled: "[Index (soundex) to the population schedules of the twelfth census of the United States, 1900, Wisconsin] [microform] (Volume WISCONSIN – TI079 - H-451 Lambert-H-514 David) - National Archives (U.S.)."  The H-415 and H-514 covers the range I need.

Click on the link and you will be taken to the details page. On the left-sidebar will be a link "Read Online."  There is often a link to view as a PDF, but trust me, online is easier.  Once you click the Read Online link, you'll be taken into the viewer.  At the bottom are controls to navigate.  

Use the slider at the bottom-left and drag it partially to the right.  Typically on the top-left will be the soundex code.  Use the current code to determine if you need to move to the left or right for higher and lower numbers.  Once you have found the cards marked with the soundex code you are looking for, pay attention to the Given (first) names on the cards.  They are often alphabetized using those names.