You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by sumitj25 <su...@gmail.com> on 2017/02/07 08:35:13 UTC
java.lang.IllegalStateException: TokenStream contract violation:
close() call missing
Hi,
I am getting IllegalStateException even though I am following the
TokenStream guidelines:
public List<String> tokenize(String text){
List<String> tokens = new ArrayList<>();
try (TokenStream ts = analyzer.tokenStream("",text)){ //note that this
is implicit finally with close
ts.reset();
while (ts.incrementToken()) {
tokens.add(ts.getAttribute(CharTermAttribute.class).toString());
}
ts.end();
return tokens;
}catch(Exception e){
throw new RuntimeException(e);
}
}
Here's my analyzer creation code:
TokenizerFactory tokenizerFactory = (TokenizerFactory)
obj(config.getTokenizer());
Tokenizer tokenizer = tokenizerFactory.create();
TokenStream stream = new StandardFilter(tokenizer);
return new Analyzer() {
protected TokenStreamComponents createComponents(String fieldName) {
return new TokenStreamComponents(tokenizer, stream);
}
};
--
View this message in context: http://lucene.472066.n3.nabble.com/java-lang-IllegalStateException-TokenStream-contract-violation-close-call-missing-tp4319109.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: java.lang.IllegalStateException: TokenStream contract violation: close() call missing
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
you still have to create it. That is the contract and there is no way around.
Keep in mind, the Analyzer instance is internally caching the components, so the factory is only called if needed. Don't try to further optimize this, it will fail! The TokenStreamComponents class is made for this caching, so why do you care? Implement it like the API and all docs define and it will work as effective as possible.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: sumitj25 [mailto:sumitjain91@gmail.com]
> Sent: Tuesday, February 7, 2017 10:33 AM
> To: java-user@lucene.apache.org
> Subject: RE: java.lang.IllegalStateException: TokenStream contract violation:
> close() call missing
>
>
> Thanks Uwe, but I am using the same TokenStream so that I don't have to
> create the filter chain every time. This is my filter chain construction:
>
> Tokenizer tokenizer = tokenizerFactory.create();
> TokenStream stream = new StandardFilter(tokenizer);
> for (LuceneFactoryConfig filterConfig :
> config.getFilters()) {
> if(filterConfig.getClass()!=null){
> stream = stream(config, filterConfig,
> stream);
> }else {
> LOG.warn("Skipping filter {}, class not
> found!",filterConfig.getName());
> }
> }
>
> Does this mean, we have to create filter chain every time?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/java-
> lang-IllegalStateException-TokenStream-contract-violation-close-call-missing-
> tp4319109p4319123.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: java.lang.IllegalStateException: TokenStream contract
violation: close() call missing
Posted by sumitj25 <su...@gmail.com>.
Thanks Uwe, but I am using the same TokenStream so that I don't have to
create the filter chain every time. This is my filter chain construction:
Tokenizer tokenizer = tokenizerFactory.create();
TokenStream stream = new StandardFilter(tokenizer);
for (LuceneFactoryConfig filterConfig : config.getFilters()) {
if(filterConfig.getClass()!=null){
stream = stream(config, filterConfig, stream);
}else {
LOG.warn("Skipping filter {}, class not
found!",filterConfig.getName());
}
}
Does this mean, we have to create filter chain every time?
--
View this message in context: http://lucene.472066.n3.nabble.com/java-lang-IllegalStateException-TokenStream-contract-violation-close-call-missing-tp4319109p4319123.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: java.lang.IllegalStateException: TokenStream contract violation: close() call missing
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
The issue is: Your Analyzer's createComponents method returns the same instance on every call, this method has to _create_ a new instance on every call. Change your code to build the TokenStream *inside* createComponents, not outside.
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: sumitj25 [mailto:sumitjain91@gmail.com]
> Sent: Tuesday, February 7, 2017 9:35 AM
> To: java-user@lucene.apache.org
> Subject: java.lang.IllegalStateException: TokenStream contract violation:
> close() call missing
>
> Hi,
>
> I am getting IllegalStateException even though I am following the
> TokenStream guidelines:
>
> public List<String> tokenize(String text){
> List<String> tokens = new ArrayList<>();
> try (TokenStream ts = analyzer.tokenStream("",text)){ //note
> that this
> is implicit finally with close
> ts.reset();
> while (ts.incrementToken()) {
>
> tokens.add(ts.getAttribute(CharTermAttribute.class).toString());
> }
> ts.end();
> return tokens;
> }catch(Exception e){
> throw new RuntimeException(e);
> }
> }
>
> Here's my analyzer creation code:
>
> TokenizerFactory tokenizerFactory = (TokenizerFactory)
> obj(config.getTokenizer());
> Tokenizer tokenizer = tokenizerFactory.create();
> TokenStream stream = new StandardFilter(tokenizer);
> return new Analyzer() {
> protected TokenStreamComponents
> createComponents(String fieldName) {
> return new
> TokenStreamComponents(tokenizer, stream);
> }
> };
>
>
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/java-
> lang-IllegalStateException-TokenStream-contract-violation-close-call-missing-
> tp4319109.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org