You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chirag Garg <cg...@sapient.com> on 2017/11/20 14:58:54 UTC
Issue facing with spell text field containing hyphen
Hi Team,
I am facing issue for string containing hyphen when searched in spell field.
My solr core is solr-6.6.0
Points to reproduce:-
Eg:- 1. My search string is "spider-man".
2. When I do a search in solr with query spell:*spider-*. It shows numDocs=0 even though content is present.
3 . But working fine when searched spell:*spider*.
My config for solr in schema.xml is:-
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<!-- Case insensitive stop word removal. -->
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
/>
<filter class="solr.WordDelimiterFilterFactory"
protected="protwords.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="1"
catenateNumbers="1"
catenateAll="0"
splitOnCaseChange="0"
preserveOriginal="1"/>
<filter class="solr.LengthFilterFactory" min="2" max="100" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
/>
<filter class="solr.WordDelimiterFilterFactory"
protected="protwords.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
splitOnCaseChange="0"
preserveOriginal="1"/>
<filter class="solr.LengthFilterFactory" min="2" max="100" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="multiterm">
<charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
/>
<filter class="solr.WordDelimiterFilterFactory"
protected="protwords.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
splitOnCaseChange="1"
preserveOriginal="1"/>
<filter class="solr.LengthFilterFactory" min="2" max="100" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory" />
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
<filter class="solr.LengthFilterFactory" min="2" max="20" />
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.RemoveDuplicatesTokenFilterFactory" />
</analyzer>
</fieldType>
Re: Issue facing with spell text field containing hyphen
Posted by Atita Arora <at...@gmail.com>.
I was about to suggest the same , Analysis Panel is the savior in such
cases of doubts.
-Atita
On Tue, Nov 21, 2017 at 7:26 AM, Rick Leir <rl...@leirtech.com> wrote:
> Chirag
> Look in Sor Admin, the Analysis panel. Put spider-man in the left and
> right text inputs, and see how it gets analysed. Cheers -- Rick
>
> On November 20, 2017 10:00:49 PM EST, Chirag garg <cg...@sapient.com>
> wrote:
> >Hi Rick,
> >
> >Actually my spell field also contains text with hyphen i.e. it contains
> >"spider-man" even then also i am not able to search it.
> >
> >Regards,
> >Chirag
> >
> >
> >
> >--
> >Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>
Re: Issue facing with spell text field containing hyphen
Posted by Rick Leir <rl...@leirtech.com>.
Chirag
Look in Sor Admin, the Analysis panel. Put spider-man in the left and right text inputs, and see how it gets analysed. Cheers -- Rick
On November 20, 2017 10:00:49 PM EST, Chirag garg <cg...@sapient.com> wrote:
>Hi Rick,
>
>Actually my spell field also contains text with hyphen i.e. it contains
>"spider-man" even then also i am not able to search it.
>
>Regards,
>Chirag
>
>
>
>--
>Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com
Re: Issue facing with spell text field containing hyphen
Posted by Chirag garg <cg...@sapient.com>.
Hi Rick,
Actually my spell field also contains text with hyphen i.e. it contains
"spider-man" even then also i am not able to search it.
Regards,
Chirag
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Issue facing with spell text field containing hyphen
Posted by Rick Leir <rl...@leirtech.com>.
Chirag
Some scattered clues:
StandardTokenizer splits on punctuation, so your spell field might not contain spider-man.
When you do a wildcard search, the analysis chain can be different from what you expected.
Cheers -- Rick
On November 20, 2017 9:58:54 AM EST, Chirag Garg <cg...@sapient.com> wrote:
>Hi Team,
>
>I am facing issue for string containing hyphen when searched in spell
>field.
>My solr core is solr-6.6.0
>
>Points to reproduce:-
>Eg:- 1. My search string is "spider-man".
>2. When I do a search in solr with query spell:*spider-*. It shows
>numDocs=0 even though content is present.
>3 . But working fine when searched spell:*spider*.
>
>My config for solr in schema.xml is:-
>
><fieldType name="text" class="solr.TextField"
>positionIncrementGap="100">
> <analyzer type="index">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
> <!-- in this example, we will only use synonyms at query time
><filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt"
>ignoreCase="true" expand="false"/>
> -->
> <!-- Case insensitive stop word removal. -->
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> protected="protwords.txt"
> generateWordParts="1"
> generateNumberParts="1"
> catenateWords="1"
> catenateNumbers="1"
> catenateAll="0"
> splitOnCaseChange="0"
> preserveOriginal="1"/>
> <filter class="solr.LengthFilterFactory" min="2" max="100" />
> <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> <analyzer type="query">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
><filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> protected="protwords.txt"
> generateWordParts="1"
> generateNumberParts="1"
> catenateWords="0"
> catenateNumbers="0"
> catenateAll="0"
> splitOnCaseChange="0"
> preserveOriginal="1"/>
> <filter class="solr.LengthFilterFactory" min="2" max="100" />
> <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> <analyzer type="multiterm">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
><filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> protected="protwords.txt"
> generateWordParts="1"
> generateNumberParts="1"
> catenateWords="0"
> catenateNumbers="0"
> catenateAll="0"
> splitOnCaseChange="1"
> preserveOriginal="1"/>
> <filter class="solr.LengthFilterFactory" min="2" max="100" />
> <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
> </analyzer>
> </fieldType>
>
><fieldType name="textSpell" class="solr.TextField"
>positionIncrementGap="100">
> <analyzer>
> <tokenizer class="solr.StandardTokenizerFactory" />
><filter class="solr.StopFilterFactory" ignoreCase="true"
>words="stopwords.txt"/>
> <filter class="solr.LengthFilterFactory" min="2" max="20" />
> <filter class="solr.LowerCaseFilterFactory" />
> <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
> </analyzer>
> </fieldType>
--
Sorry for being brief. Alternate email is rickleir at yahoo dot com