iterate through tokens in standardtokenizer

[Karthik K <ka...@gmail.com>]

Hi,
I am trying to use standardTokenizer in a non-lucene project to generate
tokens. The previous versions i used supported token.next , getToken to
iterate over and retrieve the tokens continuously. 3.0.2 doesnt have that
and i cant figure out how to iterate. Can get number of tokens with
increment token, but cant get the exact terms. Is this the right place to
ask?

Thanks

Re: iterate through tokens in standardtokenizer

[Karthik K <ka...@gmail.com>]

Thanks a lot for the help.

RE: iterate through tokens in standardtokenizer

[Uwe Schindler <uw...@thetaphi.de>]

http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/analysis/pack
age-summary.html

Read the example for "Using the new TokenStream API"!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Karthik K [mailto:karthikkatooru@gmail.com]
> Sent: Sunday, September 12, 2010 7:12 AM
> To: java-user@lucene.apache.org
> Subject: iterate through tokens in standardtokenizer
> 
> Hi,
> I am trying to use standardTokenizer in a non-lucene project to generate
> tokens. The previous versions i used supported token.next , getToken to
iterate
> over and retrieve the tokens continuously. 3.0.2 doesnt have that and i
cant
> figure out how to iterate. Can get number of tokens with increment token,
but
> cant get the exact terms. Is this the right place to ask?
> 
> Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org