You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by zzT <zi...@gmail.com> on 2013/07/11 09:31:04 UTC
Lucene 4.0 tokenstream logic
Hi all,
I'm migrating from Lucene 3.6.1 to 4.3.1 and there seems to be a major
change in how analyzers work....
Given the code example below (which is almost copied from
http://lucene.apache.org/core/4_3_1/core/index.html)
@Test
public void testAnalysis() throws IOException {
final String[] texts = {"demo", "TokenStream", "API"};
CustomAnalyzer analyzer = new CustomAnalyzer(IndexLocale.ENGLISH,
false);
for (String text : texts) {
TokenStream stream = analyzer.tokenStream("field", new
StringReader(text));
CharTermAttribute termAtt =
stream.addAttribute(CharTermAttribute.class);
try {
stream.reset();
while (stream.incrementToken()) {
System.out.println("Token : " + termAtt.toString());
}
stream.end();
} finally {
stream.close();
}
}
}
The output is the following
in 3.6.1 :
Token : demo
Token : Tokenstream
Token : API
while in
4.3.1 :
Token : demo
This is happening because of the ReuseStrategy that is now embedded inside
Analyzer.TokenStream which caches the 1st token ("demo") and reuses this one
afterwards.
CustomAnalyzer is a custom analyzer :) and its implementation is irrelevant
to the question (apart from the fact that in 3.6.1 it overrides
tokenStream() while in 4.3.1 it overrides createComponents() ). I'm pretty
sure the same is happening with Lucene's analyzers too.
The question is : Do I need to change something in my logic to make it work
as in 3.6.1? The only way to get the same output is by initializing
CustomAnalyzer before calling tokenstream().
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-4-0-tokenstream-logic-tp4077203.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene 4.0 tokenstream logic
Posted by zzT <zi...@gmail.com>.
Hi Uwe,
Thanks for your immediate response and sorry for my late reply. I managed to
solve my problem. Your comment was enough to "guide" me in the right
direction.
The problem was indeed inside my custom Analyzers/Tokenizers. The key point
here is that createComponents() is called only once while the old
tokenStream was called for each token, right? It seems that I had code that
was executed only for the 1st token so refactoring was needed.
I'll provide an example in case it might help someone else out there. Below
is the design evolution till finally making it work properly in 4.3
*
3.6 WORKING*
-> CustomAnalyzer.tokenStream()
return new CustomTokenizer(tokenize(input))
(tokenize function performs some analysis basically on the input)
*4.3 NOT WORKING*
-> CustomAnalyzer.createComponents()
return new TokenStreamComponents(new CustomTokenizer(tokenize(input)))
(tokenize function actually called only once)
*4.3 WORKING*
-> CustomAnalyzer.createComponents()
return new TokenStreamComponents(new CustomTokenizer(), new
CustomFilter())
where all logic from tokenize(input) has been moved into a separate filter
(CustomFilter) inside
incrementToken()
Hope it makes sense!
And yes, of course and lucene analyzers are working just fine, shouldn't
even mention that especially without actually testing against them.
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-4-0-tokenstream-logic-tp4077203p4078427.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene 4.0 tokenstream logic
Posted by Uwe Schindler <uw...@thetaphi.de>.
Please post the code of your TokenStream(s) thats behind this analyzer? The bug is there (and it is a bug, if it is not working correctly). The Lucene internal analyzers don't have this problem, as the TokenStreams behind that are implemented and tested. In most cases those problems appear, if the underlying TokenStream does not correctly implement token caching (with captureState or cloneAttributes) or fails to implement reset() correctly.
The problem you have is: In Lucene 4.x it is *required* for a TokenStream to support reusing, so reset() must be implemented and restore a consistent state.
If you want to test your custom TokenStreams analyzers, you should use BaseTokenStreamTestCase from the Lucene test-framework. It wil shouw you all misuse of APIs inside TokenStream implementations (like incorrectly caching tokens not using captureState/restoreState and so on).
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: zzT [mailto:zis.tax@gmail.com]
> Sent: Thursday, July 11, 2013 9:31 AM
> To: java-user@lucene.apache.org
> Subject: Lucene 4.0 tokenstream logic
>
> Hi all,
>
> I'm migrating from Lucene 3.6.1 to 4.3.1 and there seems to be a major
> change in how analyzers work....
> Given the code example below (which is almost copied from
> http://lucene.apache.org/core/4_3_1/core/index.html)
>
> @Test
> public void testAnalysis() throws IOException {
> final String[] texts = {"demo", "TokenStream", "API"};
> CustomAnalyzer analyzer = new CustomAnalyzer(IndexLocale.ENGLISH,
> false);
>
> for (String text : texts) {
> TokenStream stream = analyzer.tokenStream("field", new
> StringReader(text));
> CharTermAttribute termAtt =
> stream.addAttribute(CharTermAttribute.class);
>
> try {
> stream.reset();
> while (stream.incrementToken()) {
> System.out.println("Token : " + termAtt.toString());
> }
> stream.end();
> } finally {
> stream.close();
> }
> }
> }
>
> The output is the following
> in 3.6.1 :
> Token : demo
> Token : Tokenstream
> Token : API
>
> while in
> 4.3.1 :
> Token : demo
>
> This is happening because of the ReuseStrategy that is now embedded
> inside Analyzer.TokenStream which caches the 1st token ("demo") and
> reuses this one afterwards.
>
> CustomAnalyzer is a custom analyzer :) and its implementation is irrelevant to
> the question (apart from the fact that in 3.6.1 it overrides
> tokenStream() while in 4.3.1 it overrides createComponents() ). I'm pretty
> sure the same is happening with Lucene's analyzers too.
>
> The question is : Do I need to change something in my logic to make it work
> as in 3.6.1? The only way to get the same output is by initializing
> CustomAnalyzer before calling tokenstream().
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Lucene-
> 4-0-tokenstream-logic-tp4077203.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 4.0 tokenstream logic
Posted by Falakh TE <mf...@gmail.com>.
Hi,
I have the same problem
for(String str2:split)
{
TokenStream stream = analyzer.tokenStream("field", new
StringReader(str2));
CharTermAttribute termAtt =
stream.addAttribute(CharTermAttribute.class);
stream.reset();
while (stream.incrementToken()) {
str=termAtt.toString().toCharArray();
System.out.println(termAtt.toString());
}
stream.end();
}
The out put only prints tokens of first element in the string array ...
I am first in here, so please help me to get tokens of each element in the
string array
--
View this message in context: http://lucene.472066.n3.nabble.com/Lucene-4-0-tokenstream-logic-tp4077203p4146458.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org