You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by mahendra mahendra <ma...@yahoo.com> on 2009/02/02 08:27:13 UTC

Exact match search problem

Hi,

I have indexed my data as "custom123, customer, custom" for the "UserName" field.
I need to search the records for exact match, when I am trying to search with UserName:"customer" I am finding the records where UserName is custom123 and custom.

As per my understanding solr splits the AlphaNumeric words into sub words
custom123 => "custom","123"

As per above the above logic when I search for UserName:"customer", it shouldn't display the custom123 and custom.

Could you please tell me why it is behaving like that or how I can search for exact match.

I am using the following declaration for text field in schema file

   <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
        -->
        <!-- Case insensitive stop word removal.
             enablePositionIncrements=true ensures that a 'gap' is left to
             allow for accurate phrase queries.
        -->
        <filter class="solr.StopFilterFactory"
                ignoreCase="true"
                words="englishstopwords.txt"
                enablePositionIncrements="true"
                />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.EnglishPorterFilterFactory" protected="englishprotwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="englishsynonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="englishstopwords.txt"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.EnglishPorterFilterFactory" protected="englishprotwords.txt"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
    </fieldType>
 
Thanks in advance!!

- Mahendra




      

Re: Exact match search problem

Posted by Stephen Weiss <sw...@stylesight.com>.
Try using fieldtype "string"  instead of "text" for the UserName  
field.  Then it will not be tokenized so it should only give exact  
matches.

--
Steve


On Feb 2, 2009, at 2:27 AM, mahendra mahendra wrote:

> Hi,
>
> I have indexed my data as "custom123, customer, custom" for the  
> "UserName" field.
> I need to search the records for exact match, when I am trying to  
> search with UserName:"customer" I am finding the records where  
> UserName is custom123 and custom.
>
> As per my understanding solr splits the AlphaNumeric words into sub  
> words
> custom123 => "custom","123"
>
> As per above the above logic when I search for UserName:"customer",  
> it shouldn't display the custom123 and custom.
>
> Could you please tell me why it is behaving like that or how I can  
> search for exact match.
>
> I am using the following declaration for text field in schema file
>
>    <fieldType name="text" class="solr.TextField"  
> positionIncrementGap="100">
>       <analyzer type="index">
>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>         <!-- in this example, we will only use synonyms at query time
>         <filter class="solr.SynonymFilterFactory"  
> synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
>         -->
>         <!-- Case insensitive stop word removal.
>              enablePositionIncrements=true ensures that a 'gap' is  
> left to
>              allow for accurate phrase queries.
>         -->
>         <filter class="solr.StopFilterFactory"
>                 ignoreCase="true"
>                 words="englishstopwords.txt"
>                 enablePositionIncrements="true"
>                 />
>         <filter class="solr.WordDelimiterFilterFactory"  
> generateWordParts="1" generateNumberParts="1" catenateWords="1"  
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="solr.EnglishPorterFilterFactory"  
> protected="englishprotwords.txt"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>       <analyzer type="query">
>         <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>         <filter class="solr.SynonymFilterFactory"  
> synonyms="englishsynonyms.txt" ignoreCase="true" expand="true"/>
>         <filter class="solr.StopFilterFactory" ignoreCase="true"  
> words="englishstopwords.txt"/>
>         <filter class="solr.WordDelimiterFilterFactory"  
> generateWordParts="1" generateNumberParts="1" catenateWords="0"  
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
>         <filter class="solr.LowerCaseFilterFactory"/>
>         <filter class="solr.EnglishPorterFilterFactory"  
> protected="englishprotwords.txt"/>
>         <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>       </analyzer>
>     </fieldType>
>
> Thanks in advance!!
>
> - Mahendra
>
>
>
>


Re: Exact match search problem

Posted by Chris Hostetter <ho...@fucit.org>.
: I have indexed my data as "custom123, customer, custom" for the 
: "UserName" field. I need to search the records for exact match, when I 
: am trying to search with UserName:"customer" I am finding the records 
: where UserName is custom123 and custom.
: 
: As per my understanding solr splits the AlphaNumeric words into sub words
: custom123 => "custom","123"
: 
: As per above the above logic when I search for UserName:"customer", it 
: shouldn't display the custom123 and custom.
: 
: Could you please tell me why it is behaving like that or how I can search for exact match.

"exact match" means differnet things to different people ... based on your 
example it seems that you want tokenization on whitespace and punctuation, 
but no word-splitting.  removing the WordDelimiterFilterFactory from your 
field type will probably help ... but you might also want to get rid of 
the EnglishPorter and stopwords list too ... it all depends on what your 
goal is.

in general, this is what the Analysis Tool (linked to from the admin 
screen) is for -- it helps you see what exactly all of your existing 
filters are doing so you can decidewhich ones to remove(or alter the 
configuration of)







-Hoss