[jira] [Created] (SOLR-13077) PreAnalyzedField TokenStreamComponents should be reusable

["Michael Gibney (JIRA)" <ji...@apache.org>]

Michael Gibney created SOLR-13077:
-------------------------------------

             Summary: PreAnalyzedField TokenStreamComponents should be reusable
                 Key: SOLR-13077
                 URL: https://issues.apache.org/jira/browse/SOLR-13077
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
          Components: Schema and Analysis
            Reporter: Michael Gibney


{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from scratch for every field value.

This is necessary at the moment because the current implementation has no a priori knowledge about the schema/TokenStream that it's deserializing -- Attributes are implicit in the serialized token stream, and token Attributes are lazily instantiated in {{incrementToken()}}.

Reuse of {{TokenStreamComponents}} with the current implementation would at a minimum cause problems at index time, when Attributes are cached in indexing components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For instance, if the first field encountered has no value specified for {{PayloadAttribute}}, a {{null}} value would be cached for that {{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that {{AttributeSource}} were to be reused for a field that _does_ specify a {{PayloadAttribute}}, indexing components would "consult" the cached {{null}} value, and the payload (and all subsequent payloads) would be silently ignored (not indexed).

This is not exactly _broken_ currently, but I gather it's an unorthodox implementation of {{TokenStream}}, and the current workaround of disabling {{TokenStreamComponents}} reuse necessarily adds to object creation and GC overhead.

For reference, the [TokenStream API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html] says:
bq.To make sure that filters and consumers know which attributes are available, the attributes must be added during instantiation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org