You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael Gibney (JIRA)" <ji...@apache.org> on 2018/12/17 18:42:00 UTC
[jira] [Updated] (SOLR-13077) PreAnalyzedField
TokenStreamComponents should be reusable
[ https://issues.apache.org/jira/browse/SOLR-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Gibney updated SOLR-13077:
----------------------------------
Description:
{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from scratch for every field value.
This is necessary at the moment because the current implementation has no a priori knowledge about the schema/TokenStream that it's deserializing – Attributes are implicit in the serialized token stream, and token Attributes are lazily instantiated in {{incrementToken()}}.
Reuse of {{TokenStreamComponents}} with the current implementation would at a minimum cause problems at index time, when Attributes are cached in indexing components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For instance, if the first field encountered has no value specified for {{PayloadAttribute}}, a {{null}} value would be cached for that {{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that {{AttributeSource}} were to be reused for a field that _does_ specify a {{PayloadAttribute}}, indexing components would "consult" the cached {{null}} value, and the payload (and all subsequent payloads) would be silently ignored (not indexed).
This is not exactly _broken_ currently, but I gather it's an unorthodox implementation of {{TokenStream}}, and the current workaround of disabling {{TokenStreamComponents}} reuse necessarily adds to object creation and GC overhead.
For reference (and see LUCENE-8610), the [TokenStream API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html] says:
{quote}To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation.
{quote}
was:
{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from scratch for every field value.
This is necessary at the moment because the current implementation has no a priori knowledge about the schema/TokenStream that it's deserializing -- Attributes are implicit in the serialized token stream, and token Attributes are lazily instantiated in {{incrementToken()}}.
Reuse of {{TokenStreamComponents}} with the current implementation would at a minimum cause problems at index time, when Attributes are cached in indexing components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For instance, if the first field encountered has no value specified for {{PayloadAttribute}}, a {{null}} value would be cached for that {{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that {{AttributeSource}} were to be reused for a field that _does_ specify a {{PayloadAttribute}}, indexing components would "consult" the cached {{null}} value, and the payload (and all subsequent payloads) would be silently ignored (not indexed).
This is not exactly _broken_ currently, but I gather it's an unorthodox implementation of {{TokenStream}}, and the current workaround of disabling {{TokenStreamComponents}} reuse necessarily adds to object creation and GC overhead.
For reference, the [TokenStream API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html] says:
bq.To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation.
> PreAnalyzedField TokenStreamComponents should be reusable
> ---------------------------------------------------------
>
> Key: SOLR-13077
> URL: https://issues.apache.org/jira/browse/SOLR-13077
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Schema and Analysis
> Reporter: Michael Gibney
> Priority: Minor
>
> {{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from scratch for every field value.
> This is necessary at the moment because the current implementation has no a priori knowledge about the schema/TokenStream that it's deserializing – Attributes are implicit in the serialized token stream, and token Attributes are lazily instantiated in {{incrementToken()}}.
> Reuse of {{TokenStreamComponents}} with the current implementation would at a minimum cause problems at index time, when Attributes are cached in indexing components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For instance, if the first field encountered has no value specified for {{PayloadAttribute}}, a {{null}} value would be cached for that {{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that {{AttributeSource}} were to be reused for a field that _does_ specify a {{PayloadAttribute}}, indexing components would "consult" the cached {{null}} value, and the payload (and all subsequent payloads) would be silently ignored (not indexed).
> This is not exactly _broken_ currently, but I gather it's an unorthodox implementation of {{TokenStream}}, and the current workaround of disabling {{TokenStreamComponents}} reuse necessarily adds to object creation and GC overhead.
> For reference (and see LUCENE-8610), the [TokenStream API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html] says:
> {quote}To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org