You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "David Smiley (JIRA)" <ji...@apache.org> on 2016/01/15 22:50:40 UTC
[jira] [Commented] (LUCENE-6979) Tokenizer input state detection
should reset state before throwing
[ https://issues.apache.org/jira/browse/LUCENE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102524#comment-15102524 ]
David Smiley commented on LUCENE-6979:
--------------------------------------
One proposal I have is that this helpful checking only occur when assertions are enabled, since this seems like a good fit for that java feature.
Second, not mutually exclusive with the first, is that when TokenStreamComponents catches this problem, it is in a position to close the stream. This closes the entire stream, although an alternative might be to localize this change to just the Tokenizer.setReader and maybe it could reset itself, at the risk of not closing other possible filters that might need closing.
> Tokenizer input state detection should reset state before throwing
> ------------------------------------------------------------------
>
> Key: LUCENE-6979
> URL: https://issues.apache.org/jira/browse/LUCENE-6979
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: David Smiley
>
> The Tokenizer will helpfully let you know that you're using it wrong in certain cases, like if you forget to close it. However it does this in a way that happens lazily (after the fact) but worse it keeps the state of the Tokenizer in a cranky state (i.e. if you try to use it again, you'll get an exception again). What makes this issue insidious is that Tokenizers are re-used via a ReuseStrategy in a ThreadLocal. So once you hit this bug, you're thread is, in a word, "poisoned". And what makes the stack trace a real head-scratcher is that it is not of the original "guilty" party that didn't close; it's likely some other caller, perhaps an indexing thread who isn't going to misuse the TokenStream, or at least hasn't yet. The error message could make that clearer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org