You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2013/04/12 18:56:16 UTC
[jira] [Commented] (LUCENE-4931) Make oal.document.Field reuse its
internal StringTokenStream
[ https://issues.apache.org/jira/browse/LUCENE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630309#comment-13630309 ]
Robert Muir commented on LUCENE-4931:
-------------------------------------
I'm trying to understand the problem, its only if:
# you aren't reusing field instances
# you aren't just using KeywordTokenizer in your Analyzer for "string fields" (so they also work with queryparser etc)
# you have an ancient JVM
I dont like the idea of adding the private analyzer back to documentsinverter if we can avoid it.
I also don't understand how its not an issue for e.g. IntField too, shouldnt it be the same there?
> Make oal.document.Field reuse its internal StringTokenStream
> ------------------------------------------------------------
>
> Key: LUCENE-4931
> URL: https://issues.apache.org/jira/browse/LUCENE-4931
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0, 4.1, 4.2, 4.2.1
> Reporter: Uwe Schindler
>
> Followup from LUCENE-4930:
> Field.java has a private StringTokenStream which is used as TokenStream implementation for StringField (single value String tokens). Unfortunately this TokenStream is created on every new document/field while indexing, making the cost of creating the TS a significant time. With very old Java versions this also involves a lock in ReferenceQueue.poll() when called from addAttribute().
> In Lucene 3.x, DocInverterPerThread has a private thread-local AttributeSource for reusing, but because this was factored out to Field.java, we can no longer use CloseableThreadLocal (because Field are not Closeable). We should maybe move the special One-Token TokenStream back to DocInverterPerThread and just let Field.java delegate there. I know this would let us move back to 3.x where we had special handling of single token Fields in the indexer....
> Another approach would be to make Field.java use a static KeywordAnalyzer (it needs then be moved to core) or we add a ThreadLocal to Field.java (which may be expensive). Unfortunately this makes it hard to maintain, as the thread-localness is also needed to be bound to the IndexWriter instance. Because you could have 2 IndexWriters open at same time and add documents to both of them from one thread... This brings us back to my previous solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org