You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chirag Garg <cg...@sapient.com> on 2017/11/20 14:58:54 UTC

Issue facing with spell text field containing hyphen

Hi Team,

I am facing issue for string containing hyphen when searched in spell field.
My solr core is solr-6.6.0

Points to reproduce:-
Eg:- 1. My search string is "spider-man".
2. When I do a search in solr with query spell:*spider-*. It shows numDocs=0 even though content is present.
3 . But working fine when searched spell:*spider*.

My config for solr in schema.xml is:-

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
       -->
        <!-- Case insensitive stop word removal. -->
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="1"
                catenateNumbers="1"
                catenateAll="0"
                splitOnCaseChange="0"
                preserveOriginal="1"/>
        <filter class="solr.LengthFilterFactory" min="2" max="100" />
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="0"
                catenateNumbers="0"
                catenateAll="0"
                splitOnCaseChange="0"
                preserveOriginal="1"/>
        <filter class="solr.LengthFilterFactory" min="2" max="100" />
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
   <analyzer type="multiterm">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="stopwords.txt"
                />
        <filter class="solr.WordDelimiterFilterFactory"
                protected="protwords.txt"
                generateWordParts="1"
                generateNumberParts="1"
                catenateWords="0"
                catenateNumbers="0"
                catenateAll="0"
                splitOnCaseChange="1"
                preserveOriginal="1"/>
        <filter class="solr.LengthFilterFactory" min="2" max="100" />
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
    </fieldType>

<fieldType name="textSpell" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.StandardTokenizerFactory" />
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
        <filter class="solr.LengthFilterFactory" min="2" max="20" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
      </analyzer>
    </fieldType>


Re: Issue facing with spell text field containing hyphen

Posted by Atita Arora <at...@gmail.com>.
I was about to suggest the same , Analysis  Panel is the savior in such
cases of doubts.

-Atita

On Tue, Nov 21, 2017 at 7:26 AM, Rick Leir <rl...@leirtech.com> wrote:

> Chirag
> Look in Sor Admin, the Analysis panel. Put spider-man in the left and
> right text inputs, and see how it gets analysed. Cheers -- Rick
>
> On November 20, 2017 10:00:49 PM EST, Chirag garg <cg...@sapient.com>
> wrote:
> >Hi Rick,
> >
> >Actually my spell field also contains text with hyphen i.e. it contains
> >"spider-man" even then also i am not able to search it.
> >
> >Regards,
> >Chirag
> >
> >
> >
> >--
> >Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com
>

Re: Issue facing with spell text field containing hyphen

Posted by Rick Leir <rl...@leirtech.com>.
Chirag
Look in Sor Admin, the Analysis panel. Put spider-man in the left and right text inputs, and see how it gets analysed. Cheers -- Rick

On November 20, 2017 10:00:49 PM EST, Chirag garg <cg...@sapient.com> wrote:
>Hi Rick,
>
>Actually my spell field also contains text with hyphen i.e. it contains
>"spider-man" even then also i am not able to search it.
>
>Regards,
>Chirag
>
>
>
>--
>Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Issue facing with spell text field containing hyphen

Posted by Chirag garg <cg...@sapient.com>.
Hi Rick,

Actually my spell field also contains text with hyphen i.e. it contains
"spider-man" even then also i am not able to search it.

Regards,
Chirag



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Issue facing with spell text field containing hyphen

Posted by Rick Leir <rl...@leirtech.com>.
Chirag
Some scattered clues:
StandardTokenizer splits on punctuation, so your spell field might not contain spider-man.

When you do a wildcard search, the analysis chain can be different from what you expected.
Cheers -- Rick

On November 20, 2017 9:58:54 AM EST, Chirag Garg <cg...@sapient.com> wrote:
>Hi Team,
>
>I am facing issue for string containing hyphen when searched in spell
>field.
>My solr core is solr-6.6.0
>
>Points to reproduce:-
>Eg:- 1. My search string is "spider-man".
>2. When I do a search in solr with query spell:*spider-*. It shows
>numDocs=0 even though content is present.
>3 . But working fine when searched spell:*spider*.
>
>My config for solr in schema.xml is:-
>
><fieldType name="text" class="solr.TextField"
>positionIncrementGap="100">
>      <analyzer type="index">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>        <!-- in this example, we will only use synonyms at query time
><filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt"
>ignoreCase="true" expand="false"/>
>       -->
>        <!-- Case insensitive stop word removal. -->
>        <filter class="solr.StopFilterFactory"
>                ignoreCase="true"
>                words="stopwords.txt"
>                />
>        <filter class="solr.WordDelimiterFilterFactory"
>                protected="protwords.txt"
>                generateWordParts="1"
>                generateNumberParts="1"
>                catenateWords="1"
>                catenateNumbers="1"
>                catenateAll="0"
>                splitOnCaseChange="0"
>                preserveOriginal="1"/>
>        <filter class="solr.LengthFilterFactory" min="2" max="100" />
>        <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
>        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>      </analyzer>
>      <analyzer type="query">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
><filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>ignoreCase="true" expand="true"/>
>        <filter class="solr.StopFilterFactory"
>                ignoreCase="true"
>                words="stopwords.txt"
>                />
>        <filter class="solr.WordDelimiterFilterFactory"
>                protected="protwords.txt"
>                generateWordParts="1"
>                generateNumberParts="1"
>                catenateWords="0"
>                catenateNumbers="0"
>                catenateAll="0"
>                splitOnCaseChange="0"
>                preserveOriginal="1"/>
>        <filter class="solr.LengthFilterFactory" min="2" max="100" />
>        <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
>        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>      </analyzer>
>   <analyzer type="multiterm">
><charFilter class="solr.MappingCharFilterFactory"
>mapping="mapping-ISOLatin1Accent.txt"/>
>        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
><filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>ignoreCase="true" expand="true"/>
>        <filter class="solr.StopFilterFactory"
>                ignoreCase="true"
>                words="stopwords.txt"
>                />
>        <filter class="solr.WordDelimiterFilterFactory"
>                protected="protwords.txt"
>                generateWordParts="1"
>                generateNumberParts="1"
>                catenateWords="0"
>                catenateNumbers="0"
>                catenateAll="0"
>                splitOnCaseChange="1"
>                preserveOriginal="1"/>
>        <filter class="solr.LengthFilterFactory" min="2" max="100" />
>        <filter class="solr.LowerCaseFilterFactory"/>
><filter class="solr.SnowballPorterFilterFactory" language="English"
>protected="protwords.txt"/>
>        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>      </analyzer>
>    </fieldType>
>
><fieldType name="textSpell" class="solr.TextField"
>positionIncrementGap="100">
>      <analyzer>
>        <tokenizer class="solr.StandardTokenizerFactory" />
><filter class="solr.StopFilterFactory" ignoreCase="true"
>words="stopwords.txt"/>
>        <filter class="solr.LengthFilterFactory" min="2" max="20" />
>        <filter class="solr.LowerCaseFilterFactory" />
>        <filter class="solr.RemoveDuplicatesTokenFilterFactory" />
>      </analyzer>
>    </fieldType>

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com