Top Posts Tagged with #tokenizers

Programming Exercise - III Tokenizer in Java.

Hello Friends, Today I have got one use case I was working on. Lets say we have a big textfile. I want to do a wordcount on that file, wich counts number of times each word in that file exists. Immediately I can think of hadoop classic wordcount example and write a simple map-reduce program to do that stuff for me. In a regular map-reduce process, we tokenize all the words and then pass those key-value pairs to any mapper and reducer to calculate number of times each word is in the file. For that, we need to have a proper understanding of tokenizing.

Tokenizer API in Java provides multiple methods like hasMoreElements(), hasMoreTokens(),nextElement(), nextToken(), countTokens().

We get the instance of tokenizer with following declaration in line 34.

StringTokenizer st = new StringTokenizer(readFile,"., ");

Delimiters I have used are 'period(.)' and 'Space( )'. Our Strings are tokenized by using above delimiters. We can use any lists or maps to store these individual tokens.

Huge files processing became significant in todays programming practices. In Java, we either use 'Split' or 'tokenize' for it. I want to give a simple tokenizer program where I am giving two arguments filename and word I want to count in the file.

Hope I am clear.

package com.myCode.topCoder;

import java.io.*;

import java.util.*;

public class Tokenizer{

public static void main(String[] args) throws IOException

{

int wordcount = 0;

String word = "";

//list<String> readFile = new list<String>();

String readFile = "";

if(args.length == 0) {

System.out.println("Please specify a filename");

System.exit(1);

}

InputStreamReader reader = new InputStreamReader(new FileInputStream(args[0]));

BufferedReader br = new BufferedReader(reader);

String line = br.readLine();

//Read file in a String or list depends on size of file. Here I am taking String.

while(line!=null)

{

readFile+=line;

line = br.readLine();

}

Scanner scan = new Scanner(args[1]);

word = scan.next(args[1]);

StringTokenizer st = new StringTokenizer(readFile,", ");

System.out.println(word);

// Tokenizing readFile which has all words from file.

while(st.hasMoreTokens())

{

String s = st.nextToken();

System.out.println(s);

if(word.equals(s.toLowerCase()))

wordcount+=1;

}

System.out.println(wordcount);

}

Inputs are most welcome..

#Java #API #Tokenizers #Wordcount

•18+ Adults Only

Watch Anya Live on Cam

Anya is live and ready to show you everything. Watch her strip, dance, and perform exclusive shows just for you. Interact in real-time and make your fantasies come true.

✓ Live Streaming✓ Interactive Chat✓ Private Shows✓ HD Quality✓ Free Actions

Free to watch • No registration required • HD streaming

Solr Tokenizers

You configure the tokenizer for a text field type in schema.xml with a <tokenizer> element, as a child of <analyzer>:

The class attribute names a factory class that will instantiate a tokenizer object when needed. Tokenizer factory classes implement theorg.apache.solr.analysis.TokenizerFactory. A TokenizerFactory's create() method accepts a Reader and returns a TokenStream. When Solr creates the tokenizer it passes a Reader object that provides the content of the text field.

Arguments may be passed to tokenizer factories by setting attributes on the <tokenizer> element.

The following sections describe the tokenizer factory classes included in this release of Solr.

For more information about Solr's tokenizers, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

Standard Tokenizer

This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:

Periods (dots) that are not followed by whitespace are kept as part of the token.

Words are split at hyphens, unless there is a number in the word, in which case the token is not split and the numbers and hyphen(s) are preserved.

Recognizes Internet domain names and email addresses and preserves them as a single token.

The Standard Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following token types: <ALPHANUM>, <NUM>, <SOUTHEAST_ASIAN>, <IDEOGRAPHIC>, and <HIRAGANA>.

Factory class: solr.StandardTokenizerFactory

Arguments:

maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by maxTokenLength.

Example:

In: "Please, email [email protected] by 03-09, re: m37-xq."

Out: "Please", "email", "[email protected]", "by", "03-09", "re", "m37-xq"

Classic Tokenizer

The Classic Tokenizer preserves the same behavior as the Standard Tokenizer of Solr versions 3.1 and previous. It does not use the Unicode standard annex UAX#29 word boundary rules that the Standard Tokenizer uses. This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:

Periods (dots) that are not followed by whitespace are kept as part of the token.

Words are split at hyphens, unless there is a number in the word, in which case the token is not split and the numbers and hyphen(s) are preserved.

Recognizes Internet domain names and email addresses and preserves them as a single token.

Factory class: solr.ClassicTokenizerFactory

Arguments:

maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by maxTokenLength.

Example:

In: "Please, email [email protected] by 03-09, re: m37-xq."

Out: "Please", "email", "[email protected]", "by", "03-09", "re", "m37-xq"

Keyword Tokenizer

This tokenizer treats the entire text field as a single token.

Factory class: solr.KeywordTokenizerFactory

Arguments: None

Example:

In: "Please, email [email protected] by 03-09, re: m37-xq."

Out: "Please, email [email protected] by 03-09, re: m37-xq."

Letter Tokenizer

This tokenizer creates tokens from strings of contiguous letters, discarding all non-letter characters.

Factory class: solr.LetterTokenizerFactory

Arguments: None

Example:

In: "I can't."

Out: "I", "can", "t"

Lower Case Tokenizer

Tokenizes the input stream by delimiting at non-letters and then converting all letters to lowercase. Whitespace and non-letters are discarded.

Factory class: solr.LowerCaseTokenizerFactory

Arguments: None

Example:

In: "I just LOVE my iPhone!"

Out: "i", "just", "love", "my", "iphone"

N-Gram Tokenizer

Reads the field text and generates n-gram tokens of sizes in the given range.

Factory class: solr.NGramTokenizerFactory

Arguments:

minGramSize: (integer, default 1) The minimum n-gram size, must be > 0.

maxGramSize: (integer, default 2) The maximum n-gram size, must be >= minGramSize.

Example:

Default behavior. Note that this tokenizer operates over the whole field. It does not break the field at whitespace. As a result, the space character is included in the encoding.

In: "hey man"

Out: "h", "e", "y", " ", "m", "a", "n", "he", "ey", "y ", " m", "ma", "an"

Example:

With an n-gram size range of 4 to 5:

In: "bicycle"

Out: "bicy", "icyc", "cycl", "ycle", "bicyc", "icycl", "cycle"

Edge N-Gram Tokenizer

Reads the field text and generates edge n-gram tokens of sizes in the given range.

Factory class: solr.EdgeNGramTokenizerFactory

Arguments:

minGramSize: (integer, default 1) The minimum n-gram size, must be > 0.

maxGramSize: (integer, default 1) The maximum n-gram size, must be >= minGramSize.

side: ("front" or "back", default "front") Whether to compute the n-grams from the beginning (front) of the text or from the end (back).

Example:

Default behavior (min and max default to 1):

In: "babaloo"

Out: "b"

Example:

Edge n-gram range of 2 to 5

In: "babaloo"

Out:"ba", "bab", "baba", "babal"

Example:

Edge n-gram range of 2 to 5, from the back side:

In: "babaloo"

Out: "oo", "loo", "aloo", "baloo"

ICU Tokenizer

This tokenizer processes multilingual text and tokenizes it appropriately based on its script attribute.

Factory class: solr.ICUTokenizerFactory

Arguments: None

Example:

In: "Testing บริษัทชื่อ נאסק"ר"

Out: "Testing", "บริษัท", "ชื่อ", "נאסק"ר "

Path Hierarchy Tokenizer

This tokenizer creates synonyms from file path hierarchies.

Factory class: solr.PathHierarchyTokenizerFactory

Arguments:

delimiter: (character, no default) You can specify the file path delimiter and replace it with a delimiter you provide. This can be useful for working with backslash delimiters.

replace: (character, no default) Specifies the delimiter character Solr uses in the tokenized output.

Example:

In: "c:\usr\local\apache"

Out: "c:", "c:/usr", "c:/usr/local", "c:/usr/local/apache"

Regular Expression Pattern Tokenizer

This tokenizer uses a Java regular expression to break the input text stream into tokens. The expression provided by the pattern argument can be interpreted either as a delimiter that separates tokens, or to match patterns that should be extracted from the text as tokens.

See the Javadocs for java.util.regex.Pattern for more information on Java regular expression syntax.

Factory class: solr.PatternTokenizerFactory

Arguments:

pattern: (Required) The regular expression, as defined by in java.util.regex.Pattern.

group: (Optional, default -1) Specifies which regex group to extract as the token(s).The value -1 means the regex should be treated as a delimiter that separates tokens.Non-negative group numbers (>= 0) indicate that character sequences matching that regex group should be converted to tokens. Group zero refers to the entire regex, groups greater than zero refer to parenthesized sub-expressions of the regex, counted from left to right.

Example:

A comma separated list. Tokens are separated by a sequence of zero or more spaces, a comma, and zero or more spaces.

In: "fee,fie, foe , fum, foo"

Out: "fee", "fie", "foe", "fum", "foo"

Example:

Extract simple, capitalized words. A sequence of at least one capital letter followed by zero or more letters of either case is extracted as a token.

In: "Hello. My name is Inigo Montoya. You killed my father. Prepare to die."

Out: "Hello", "My", "Inigo", "Montoya". "You", "Prepare"

Example:

Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case sensitive, with an optional semi-colon separator. Part numbers must be all numeric digits, with an optional hyphen. Regex capture groups are numbered by counting left parenthesis from left to right. Group 3 is the subexpression "[0-9-]+", which matches one or more digits or hyphens.

In: "SKU: 1234, Part Number 5678, Part: 126-987"

Out: "1234", "5678", "126-987"

UAX29 URL Email Tokenizer

This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:

Periods (dots) that are not followed by whitespace are kept as part of the token.

Words are split at hyphens, unless there is a number in the word, in which case the token is not split and the numbers and hyphen(s) are preserved.

Recognizes top-level (.com) Internet domain names; email addresses; file:://, http(s)://, and ftp:// addresses; IPv4 and IPv6 addresses; and preserves them as a single token.

You configure the tokenizer for a text field type in schema.xml with a <tokenizer> element, as a child of <analyzer>:

Arguments may be passed to tokenizer factories by setting attributes on the <tokenizer> element.

The following sections describe the tokenizer factory classes included in this release of Solr.

For more information about Solr's tokenizers, see http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

The UAX29 URL Email Tokenizer supports Unicode standard annex UAX#29 word boundaries with the following token types: <ALPHANUM>, <NUM>, URL, EMAIL, <SOUTHEAST_ASIAN>, <IDEOGRAPHIC>, and <HIRAGANA>.

Factory class: solr.UAX29URLEmailTokenizerFactory

Arguments:

maxTokenLength: (integer, default 255) Solr ignores tokens that exceed the number of characters specified by maxTokenLength.

Example:

In: "Visit http://accarol.com/contact.htm?from=external&a=10 or e-mail [email protected]"

Out: "Visit", "http://accarol.com/contact.htm?from=external&a=10", "or", "email", "[email protected]"

White Space Tokenizer

Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters as tokens. Note that any punctuation will be included in the tokenization.

Factory class: solr.WhitespaceTokenizerFactory

Arguments: None

Example:

In: "To be, or what?"

Out: "To", "be,", "or", "what?"

source: http://lucidworks.lucidimagination.com/display/solr/Tokenizers

#Tokenizers #solr