You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2013/04/12 21:14:16 UTC
[jira] [Resolved] (LUCENE-4931) Make oal.document.Field reuse its
internal StringTokenStream
[ https://issues.apache.org/jira/browse/LUCENE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler resolved LUCENE-4931.
-----------------------------------
Resolution: Not A Problem
Assignee: Uwe Schindler
Sorry,
this issue was already solved by an earlier commit by me: LUCENE-4317 and LUCENE-4315
If you still see the problems, you should reuse your StringField instances. If you recreate them, the TokenStream is not reused!
Uwe
> Make oal.document.Field reuse its internal StringTokenStream
> ------------------------------------------------------------
>
> Key: LUCENE-4931
> URL: https://issues.apache.org/jira/browse/LUCENE-4931
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0, 4.1, 4.2, 4.2.1
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
>
> Followup from LUCENE-4930:
> Field.java has a private StringTokenStream which is used as TokenStream implementation for StringField (single value String tokens). Unfortunately this TokenStream is created on every new document/field while indexing, making the cost of creating the TS a significant time. With very old Java versions this also involves a lock in ReferenceQueue.poll() when called from addAttribute().
> In Lucene 3.x, DocInverterPerThread has a private thread-local AttributeSource for reusing, but because this was factored out to Field.java, we can no longer use CloseableThreadLocal (because Field are not Closeable). We should maybe move the special One-Token TokenStream back to DocInverterPerThread and just let Field.java delegate there. I know this would let us move back to 3.x where we had special handling of single token Fields in the indexer....
> Another approach would be to make Field.java use a static KeywordAnalyzer (it needs then be moved to core) or we add a ThreadLocal to Field.java (which may be expensive). Unfortunately this makes it hard to maintain, as the thread-localness is also needed to be bound to the IndexWriter instance. Because you could have 2 IndexWriters open at same time and add documents to both of them from one thread... This brings us back to my previous solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org