You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lukas Kahwe Smith <ml...@pooteeweet.org> on 2010/02/12 10:02:40 UTC

inconsistency between analysis.jsp and actual search

Hi

I am indexing the name "FC St. Gallen" using the following type:
    <fieldType name="prefix_token" class="solr.TextField" positionIncrementGap="1">
      <analyzer type="index">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt" />
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
        <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="20" />
      </analyzer>
      <analyzer type="query">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt" />
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
      </analyzer>
    </fieldType>

Which according to analysis.jsp gets split into:
f | fc | s | st | g | ga | gal | gall | galle | gallen

So far so good.

Now if I search for "fc st.gallen" according to analysis.jsp it will search for:
fc | st | gallen

But when I do a dismax search using the following handler:
  <requestHandler name="auto" class="solr.SearchHandler" default="true">
    <lst name="defaults">
     <str name="defType">dismax</str>
     <str name="echoParams">explicit</str>
     <int name="rows">10</int>
     <str name="qf">name firstname email^0.5 telefon^0.5 city^0.6 street^0.6</str>
     <str name="fl">id,type,name,firstname,zipcode,city,street,urlizedname</str>
    </lst>
  </requestHandler>

I do not get a match.
Looking at the debug of the query I can see that its actually splitting the query into "fc" and "st gallen":
<str name="rawquerystring">fc st.gallen</str>
<str name="querystring">fc st.gallen</str>
<str name="parsedquery">
+((DisjunctionMaxQuery((telefon:fc^0.5 | firstname:fc | email:fc^0.5 | street:fc^0.6 | city:fc^0.6 | name:fc)) DisjunctionMaxQuery((telefon:"st gallen"^0.5 | firstname:"st gallen" | email:"st gallen"^0.5 | street:"st gallen"^0.6 | city:"st gallen"^0.6 | name:"st gallen")))~2) ()
</str>
<str name="parsedquery_toString">
+(((telefon:fc^0.5 | firstname:fc | email:fc^0.5 | street:fc^0.6 | city:fc^0.6 | name:fc) (telefon:"st gallen"^0.5 | firstname:"st gallen" | email:"st gallen"^0.5 | street:"st gallen"^0.6 | city:"st gallen"^0.6 | name:"st gallen"))~2) ()
</str>

Whats going on there?

regards,
Lukas Kahwe Smith
mls@pooteeweet.org

Re: inconsistency between analysis.jsp and actual search

Posted by Ahmet Arslan <io...@yahoo.com>.
> Anyways so how can I get "st.gallen" split into two terms
> at query time?

As you mentioned in your first mail, query st.gallen is already broken into two terms/words. But query parser constructs a phrase query.

There was an disscussion about this behaviour earlier.
http://www.lucidimagination.com/search/document/d41bc0ef422b9238/understanding_the_query_parser#85db37e69ef29dba

Re: inconsistency between analysis.jsp and actual search

Posted by Lukas Kahwe Smith <ml...@pooteeweet.org>.
On 12.02.2010, at 11:17, Ahmet Arslan wrote:
> analysis.jsp does not do actual query parsing. just shows produced tokens step by step in analysis (charfilter, tokenizer, tokenfilter) phase.
> "admin/analysis.jsp page will show you how your field is processed while indexing and while querying, and if a particular query matches." [1]
> 
> [1]http://wiki.apache.org/solr/FAQ#My_search_returns_too_many_.2BAC8_too_little_.2BAC8_unexpected_results.2C_how_to_debug.3F


I see, thats good to know. Maybe even something that should be noted in the analysis.jsp page itself.

Anyways so how can I get "st.gallen" split into two terms at query time?

    <fieldType name="prefix_token" class="solr.TextField" positionIncrementGap="1">
      <analyzer type="index">
        ...
      </analyzer>
      <analyzer type="query">
        <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt" />
        <tokenizer class="solr.WhitespaceTokenizerFactory" />
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
      </analyzer>
    </fieldType>

It seems I should probably use the solr.StandardTokenizerFactory anyways, but for this case it wouldnt help either.

regards,
Lukas Kahwe Smith
mls@pooteeweet.org

Re: inconsistency between analysis.jsp and actual search

Posted by Ahmet Arslan <io...@yahoo.com>.
> Which according to analysis.jsp gets split into:
> f | fc | s | st | g | ga | gal | gall | galle | gallen
> 
> So far so good.
> 
> Now if I search for "fc st.gallen" according to
> analysis.jsp it will search for:
> fc | st | gallen
> 
> But when I do a dismax search using the following handler:
>   <requestHandler name="auto"
> class="solr.SearchHandler" default="true">
>     <lst name="defaults">
>      <str
> name="defType">dismax</str>
>      <str
> name="echoParams">explicit</str>
>      <int
> name="rows">10</int>
>      <str name="qf">name
> firstname email^0.5 telefon^0.5 city^0.6
> street^0.6</str>
>      <str
> name="fl">id,type,name,firstname,zipcode,city,street,urlizedname</str>
>     </lst>
>   </requestHandler>
> 
> I do not get a match.
> Looking at the debug of the query I can see that its
> actually splitting the query into "fc" and "st gallen":
> <str name="rawquerystring">fc st.gallen</str>
> <str name="querystring">fc st.gallen</str>
> <str name="parsedquery">
> +((DisjunctionMaxQuery((telefon:fc^0.5 | firstname:fc |
> email:fc^0.5 | street:fc^0.6 | city:fc^0.6 | name:fc))
> DisjunctionMaxQuery((telefon:"st gallen"^0.5 | firstname:"st
> gallen" | email:"st gallen"^0.5 | street:"st gallen"^0.6 |
> city:"st gallen"^0.6 | name:"st gallen")))~2) ()
> </str>
> <str name="parsedquery_toString">
> +(((telefon:fc^0.5 | firstname:fc | email:fc^0.5 |
> street:fc^0.6 | city:fc^0.6 | name:fc) (telefon:"st
> gallen"^0.5 | firstname:"st gallen" | email:"st gallen"^0.5
> | street:"st gallen"^0.6 | city:"st gallen"^0.6 | name:"st
> gallen"))~2) ()
> </str>
> 
> Whats going on there?

analysis.jsp does not do actual query parsing. just shows produced tokens step by step in analysis (charfilter, tokenizer, tokenfilter) phase.
"admin/analysis.jsp page will show you how your field is processed while indexing and while querying, and if a particular query matches." [1]

[1]http://wiki.apache.org/solr/FAQ#My_search_returns_too_many_.2BAC8_too_little_.2BAC8_unexpected_results.2C_how_to_debug.3F