Searching for Chinese characters is much slower

[Zheng Lin Edwin Yeo <ed...@gmail.com>]

Hi,

I'm using Solr 5.4.0, and the HMMChineseTokenizerFactory for my content
indexed from rich-text documents.

I found that during my search, the search for Chinese characters is much
longer than English characters. The English characters usually can be
returned in less than 200ms, but Chinese characters usually need at least 2
or 3 seconds for it to return.

I have about 3 million documents in my index, with an index size of 230GB,

Below is my setting in schema.xml.


<fieldType name="text_chinese" class="solr.TextField"
positionIncrementGap="100" autoGeneratePhraseQueries="false">
 <analyzer type="index">
<tokenizer class="solr.HMMChineseTokenizerFactory"/>
<filter class="solr.CJKWidthFilterFactory"/>
<filter class="solr.CJKBigramFilterFactory"/>
<filter class="solr.StopFilterFactory"
words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="false"/>
<filter class="solr.KStemFilterFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="1"
maxGramSize="15"/>
 </analyzer>
 <analyzer type="query">
<tokenizer class="solr.HMMChineseTokenizerFactory"/>
<filter class="solr.CJKWidthFilterFactory"/>
<filter class="solr.CJKBigramFilterFactory"/>
<filter class="solr.StopFilterFactory"
words="org/apache/lucene/analysis/cn/smart/stopwords.txt"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="0"
generateNumberParts="0" catenateWords="0" catenateNumbers="0"
catenateAll="0" splitOnCaseChange="0"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="false"/>
<filter class="solr.KStemFilterFactory"/>
      </analyzer>
</fieldType>



Regards,
Edwin