You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Konstantyn Smirnov <in...@yahoo.com> on 2013/02/27 18:25:21 UTC
Confusion with Analyzer.tokenStream() re-use in 4.1
Dear all,
I'm using the following test-code:
Document doc = new Document()
Analyzer a = new SimpleAnalyzer( Version.LUCENE_41 )
TokenStream inputTS = a.tokenStream( 'name1', new StringReader( 'aaa bbb
ccc' ) )
Field f = new TextField( 'name1', inputTS )
doc.add f
TokenStream ts = doc.getField( 'name1' ).tokenStreamValue()
ts.reset()
String sb = ''
while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) +
'|'
assert 'aaa|bbb|ccc|' == sb
inputTS = a.tokenStream( 'name2', new StringReader( 'xxx zzz' ) )
f = new TextField( 'name2', inputTS )
doc.add f
TokenStream ts = doc.getField( 'name2' ).tokenStreamValue()
ts.reset()
sb = ''
while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) +
'|'
assert 'xxx|zzz|' == sb // << FAILS! -> sb == '' and ts.incrementTokent() ==
false
The 1st added field lets read it's tokentStreamValue() tokens, all
subsequent calls bring nothing, unless I re-instantiate the analyzer.
Another strange thing is, that just before adding a new field to the
document, the tokenStream is filled..
What am I doing wrong?
TIA
--
View this message in context: http://lucene.472066.n3.nabble.com/Confusion-with-Analyzer-tokenStream-re-use-in-4-1-tp4043427.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Confusion with Analyzer.tokenStream() re-use in 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
In addition, in your first field you are using StringReader to feed in the data which can only be consumed once. This has nothing to do with TokenStream reuse.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, February 27, 2013 8:03 PM
> To: 'java-user@lucene.apache.org'
> Subject: RE: Confusion with Analyzer.tokenStream() re-use in 4.1
>
> The problem here is that the tokenstream is instantiated in the same thread
> from 2 different code paths and consumed later. If you add fields, the
> indexer will fetch a new reused TokenStream one after each other and
> consume them directly after getting. It will not interleave this. In your case,
> the second field is instantiated using a TokenStream, which is already
> initialized. Unfortunately, if you ask the analyzer for another TokenStream
> later, the already opened one gets invalid (the second field).
>
> Don't use new Field(name, TokenStream) with TokenStreams from
> Analyzers, because they are only "valid" for a very short time. If you need to
> do this, use a second Analyzer instance. If you add Fields with a String value,
> the TokenStream is created on they fly and is be consumed by the
> DocumentsWriter directly after getting it.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Konstantyn Smirnov [mailto:injecteer@yahoo.com]
> > Sent: Wednesday, February 27, 2013 6:25 PM
> > To: java-user@lucene.apache.org
> > Subject: Confusion with Analyzer.tokenStream() re-use in 4.1
> >
> > Dear all,
> >
> > I'm using the following test-code:
> >
> > Document doc = new Document()
> > Analyzer a = new SimpleAnalyzer( Version.LUCENE_41 )
> >
> > TokenStream inputTS = a.tokenStream( 'name1', new StringReader( 'aaa
> > bbb ccc' ) ) Field f = new TextField( 'name1', inputTS ) doc.add f
> >
> > TokenStream ts = doc.getField( 'name1' ).tokenStreamValue()
> > ts.reset()
> >
> > String sb = ''
> > while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) +
> '|'
> > assert 'aaa|bbb|ccc|' == sb
> >
> > inputTS = a.tokenStream( 'name2', new StringReader( 'xxx zzz' ) ) f =
> > new TextField( 'name2', inputTS ) doc.add f
> >
> > TokenStream ts = doc.getField( 'name2' ).tokenStreamValue()
> > ts.reset()
> >
> > sb = ''
> > while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) +
> '|'
> > assert 'xxx|zzz|' == sb // << FAILS! -> sb == '' and
> > ts.incrementTokent() == false
> >
> > The 1st added field lets read it's tokentStreamValue() tokens, all
> > subsequent calls bring nothing, unless I re-instantiate the analyzer.
> >
> > Another strange thing is, that just before adding a new field to the
> > document, the tokenStream is filled..
> >
> > What am I doing wrong?
> >
> > TIA
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Confusion-with-Analyzer-
> > tokenStream-re-use-in-4-1-tp4043427.html
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Confusion with Analyzer.tokenStream() re-use in 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
The problem is how you use the Document/Field/Analyzer in your test code (see my mail that explains it). The second problem is that you use new Field(...,TokenStream), which instantiates the TokenStream at the time of calling, so it is "in use", which violates the general call-order of Analyzers: Anylzers are fetched by IndexWriter consumed and given back to Analyzer. Here you get the analyzer to early, confusing the other TokenStreams.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Konstantyn Smirnov [mailto:injecteer@yahoo.com]
> Sent: Thursday, February 28, 2013 12:18 AM
> To: java-user@lucene.apache.org
> Subject: RE: Confusion with Analyzer.tokenStream() re-use in 4.1
>
> Thanks for the answer Uwe!
>
> so the behavior has changed since the 3.6, hasn't it?
>
> Now I need to instantiate the analyzer each time I feed the field with the
> tokenStream, or it happens behind the scenes if I use new (String name,
> String value, Field.Store store).
>
> Another question then... Now I try my best to reuse the Document and Field
> instances when indexing more than 1 document. Is the instantiation of an
> analyzer heavy enough and shouldn't it be also re-used?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Confusion-with-Analyzer-
> tokenStream-re-use-in-4-1-tp4043427p4043508.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Confusion with Analyzer.tokenStream() re-use in 4.1
Posted by Konstantyn Smirnov <in...@yahoo.com>.
Thanks for the answer Uwe!
so the behavior has changed since the 3.6, hasn't it?
Now I need to instantiate the analyzer each time I feed the field with the
tokenStream, or it happens behind the scenes if I use new (String name,
String value, Field.Store store).
Another question then... Now I try my best to reuse the Document and Field
instances when indexing more than 1 document. Is the instantiation of an
analyzer heavy enough and shouldn't it be also re-used?
--
View this message in context: http://lucene.472066.n3.nabble.com/Confusion-with-Analyzer-tokenStream-re-use-in-4-1-tp4043427p4043508.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Confusion with Analyzer.tokenStream() re-use in 4.1
Posted by Uwe Schindler <uw...@thetaphi.de>.
The problem here is that the tokenstream is instantiated in the same thread from 2 different code paths and consumed later. If you add fields, the indexer will fetch a new reused TokenStream one after each other and consume them directly after getting. It will not interleave this. In your case, the second field is instantiated using a TokenStream, which is already initialized. Unfortunately, if you ask the analyzer for another TokenStream later, the already opened one gets invalid (the second field).
Don't use new Field(name, TokenStream) with TokenStreams from Analyzers, because they are only "valid" for a very short time. If you need to do this, use a second Analyzer instance. If you add Fields with a String value, the TokenStream is created on they fly and is be consumed by the DocumentsWriter directly after getting it.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Konstantyn Smirnov [mailto:injecteer@yahoo.com]
> Sent: Wednesday, February 27, 2013 6:25 PM
> To: java-user@lucene.apache.org
> Subject: Confusion with Analyzer.tokenStream() re-use in 4.1
>
> Dear all,
>
> I'm using the following test-code:
>
> Document doc = new Document()
> Analyzer a = new SimpleAnalyzer( Version.LUCENE_41 )
>
> TokenStream inputTS = a.tokenStream( 'name1', new StringReader( 'aaa bbb
> ccc' ) ) Field f = new TextField( 'name1', inputTS ) doc.add f
>
> TokenStream ts = doc.getField( 'name1' ).tokenStreamValue()
> ts.reset()
>
> String sb = ''
> while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) + '|'
> assert 'aaa|bbb|ccc|' == sb
>
> inputTS = a.tokenStream( 'name2', new StringReader( 'xxx zzz' ) ) f = new
> TextField( 'name2', inputTS ) doc.add f
>
> TokenStream ts = doc.getField( 'name2' ).tokenStreamValue()
> ts.reset()
>
> sb = ''
> while( ts.incrementToken() ) sb += ts.getAttribute( CharTermAttribute ) + '|'
> assert 'xxx|zzz|' == sb // << FAILS! -> sb == '' and ts.incrementTokent() ==
> false
>
> The 1st added field lets read it's tokentStreamValue() tokens, all subsequent
> calls bring nothing, unless I re-instantiate the analyzer.
>
> Another strange thing is, that just before adding a new field to the
> document, the tokenStream is filled..
>
> What am I doing wrong?
>
> TIA
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Confusion-with-Analyzer-
> tokenStream-re-use-in-4-1-tp4043427.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org