[jira] [Commented] (SOLR-4619) Improve PreAnalyzedField query analysis

["Robert Muir (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-4619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097440#comment-15097440 ] 

Robert Muir commented on SOLR-4619:
-----------------------------------

{quote}
 But PreAnalyzedTokenizer doesn't have any attributes defined until the input stream is consumed, in reset()
{quote}

Right, thats a bug really. According to TokenStream's class javadocs:

{quote}
 The workflow of the new TokenStream API is as follows:

 1.  Instantiation of TokenStream/TokenFilters which add/get attributes to/from the AttributeSource.
 2.  The consumer calls reset().
 3.  The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access. 
{quote}

So we have consumers (such as QueryBuilder) doing stuff out of order: if they do step 3 before they do step 2.

My question is, can we detect this in tests? If MockAnalyzer can enforce it, it is easier to fix it consistently everywhere. One idea is if MockTokenizer deferred initializing its attributes until reset()? Its not going to be the best (we need to tie it into its state machine logic somehow for that), but it might be an easy step.


> Improve PreAnalyzedField query analysis
> ---------------------------------------
>
>                 Key: SOLR-4619
>                 URL: https://issues.apache.org/jira/browse/SOLR-4619
>             Project: Solr
>          Issue Type: Bug
>          Components: Schema and Analysis
>    Affects Versions: 4.0, 4.1, 4.2, 4.2.1, Trunk
>            Reporter: Andrzej Bialecki 
>            Assignee: Andrzej Bialecki 
>             Fix For: Trunk
>
>         Attachments: SOLR-4619.patch, SOLR-4619.patch
>
>
> PreAnalyzed field extends plain FieldType and mistakenly uses the DefaultAnalyzer as query analyzer, and doesn't allow for customization via <analyzer> schema elements.
> Instead it should extend TextField and support all query analysis supported by that type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org