[jira] [Created] (SOLR-12518) PreAnalyzedField fails to index documents without tokens

["Yuki Yano (JIRA)" <ji...@apache.org>]

Yuki Yano created SOLR-12518:
--------------------------------

             Summary: PreAnalyzedField fails to index documents without tokens
                 Key: SOLR-12518
                 URL: https://issues.apache.org/jira/browse/SOLR-12518
             Project: Solr
          Issue Type: Bug
      Security Level: Public (Default Security Level. Issues are Public)
          Components: update
            Reporter: Yuki Yano
         Attachments: SOLR-12518.patch

h1. Overview

{{PreAnalyzedField}} fails to index documents without tokens like the following data:
{code:java}
{
  "v": "1",
  "str": "foo",
  "tokens": []
}
{code}
h1. Details

{{PreAnalyzedField}} consumes field values which have been pre-analyzed in advance. The format of pre-analyzed value is like follows:
{code:java}
{
  "v":"1",
  "str":"test",
  "tokens": [
    {"t":"one","s":123,"e":128,"i":22,"p":"DQ4KDQsODg8=","y":"word"},
    {"t":"two","s":5,"e":8,"i":1,"y":"word"},
    {"t":"three","s":20,"e":22,"i":1,"y":"foobar"}
  ]
}
{code}
As [the document|https://lucene.apache.org/solr/guide/7_3/working-with-external-files-and-processes.html#WorkingwithExternalFilesandProcesses] mensions, {{"str"}} and {{"tokens"}} are optional, i.e., both an empty value and no key are allowed. However, when {{"tokens"}} is empty or not defined, {{PreAnalyzedField}} throws IOException and fails to index the document.

This error is related to the behavior of {{Field#tokenStream}}. This method tries to create {{TokenStream}} by following steps (NOTE: assume {{indexed=true}}):
 * If the field has {{tokenStream}} value, returns it.
 * Otherwise, creates {{tokenStream}} by parsing the stored value.

If pre-analyzed value doesn't have tokens, the second step will be executed. Unfortunately, since {{PreAnalyzedField}} always returns {{PreAnalyzedAnalyzer}} as the index analyzer and the stored value (i.e., the value of {{"str"}}) is not the pre-analyzed format, this step will fail due to the pre-analyzed format error (i.e., IOException).
h1. How to reproduce

1. Download latest solr package and prepare solr server according to [Solr Tutorial|http://lucene.apache.org/solr/guide/7_3/solr-tutorial.html].
 2. Add following fieldType and field to the schema.
{code:xml}
    <fieldType name="preanalyzed-with-analyzer" class="solr.PreAnalyzedField">
      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      </analyzer>
    </fieldType>
    <field name="pre_with_analyzer" type="preanalyzed-with-analyzer" indexed="true" stored="true" multiValued="false"/>
{code}
3. Index following documents and Solr will throw IOException.
{code:java}
// This is OK
{"id": 1, "pre_with_analyzer": "{\'v\':\'1\',\'str\':\'document one\',\'tokens\':[{\'t\':\'one\'},{\'t\':\'two\'},{\'t\':\'three\',\'i\':100}]}"}

// Solr throws IOException
{"id": 2, "pre_with_analyzer": "{\'v\':\'1\',\'str\':\'document two\', \'tokens\':[]}"}

// Solr throws IOException
{"id": 3, "pre_with_analyzer": "{\'v\':\'1\',\'str\':\'document three\'}"}
{code}
h1. How to fix

Because we don't need to analyze again if {{"tokens"}} is empty or not set, we can avoid this error by setting {{EmptyTokenStream}} as {{tokenStream}} instead like the following code:
{code:java}
parse.hasTokenStream() ? parse : new EmptyTokenStream()
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org