You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by darniz <rn...@edmunds.com> on 2009/08/25 02:58:28 UTC

solr.StopFilterFactory not filtering words

HI
i dont understand whats wrong here.
i am trying to implement solr.StopFilterFactory
here is my field type definition
    <fieldtype name="teststop" class="solr.TextField"> 
        <analyzer type="index"> 
            <tokenizer class="solr.LowerCaseTokenizerFactory"/> 
                <filter class="solr.StopFilterFactory" words="stopwords.txt"
ignoreCase="true"/> 
        </analyzer>
    </fieldtype> 

and i have field which is
   <field name="onlynoun" type="teststop" indexed="true" stored="true"
required="false"/> 

when i try to insert a doc like
<field name="onlynoun">an elphant is an animal</field>
it does not remove an and is from the phrase.

Any idea why 

Thanks
-- 
View this message in context: http://www.nabble.com/solr.StopFilterFactory-not-filtering-words-tp25123903p25123903.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr.StopFilterFactory not filtering words

Posted by darniz <rn...@edmunds.com>.

Thanks Yonik
So its basically how the field is indexed and not stored.
So i give "the elephant is an animal" and try to get back the document it
should see the entire string, only the index is done on elephant and animal.
i was of the impression that when solr loads that document it strips out
those words in order to check that i did a q on "the elephant" and when i 
look at the output

 <str name="rawquerystring">"the elephant"</str> 
  <str name="querystring">"the elephant"</str> 
  <str name="parsedquery">+DisjunctionMaxQuery((onlynoun:elephant)) ()</str> 

Is this the correct way of evaluating your index are correctly done.
One last doubt about this is that when i declared the field i gave analyser
as index. what is the difference if i give the analyzer as query. 
<fieldtype name="teststop" class="solr.TextField"> 
        <analyzer type="index"> 
            <tokenizer class="solr.LowerCaseTokenizerFactory"/> 
                <filter class="solr.StopFilterFactory" words="stopwords.txt"
ignoreCase="true"/> 
        </analyzer>
    </fieldtype> 


-- 
View this message in context: http://www.nabble.com/solr.StopFilterFactory-not-filtering-words-tp25123903p25139099.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr.StopFilterFactory not filtering words

Posted by Yonik Seeley <yo...@lucidimagination.com>.

On Mon, Aug 24, 2009 at 8:58 PM, darniz<rn...@edmunds.com> wrote:
>
> HI
> i dont understand whats wrong here.
> i am trying to implement solr.StopFilterFactory
> here is my field type definition
>    <fieldtype name="teststop" class="solr.TextField">
>        <analyzer type="index">
>            <tokenizer class="solr.LowerCaseTokenizerFactory"/>
>                <filter class="solr.StopFilterFactory" words="stopwords.txt"
> ignoreCase="true"/>
>        </analyzer>
>    </fieldtype>
>
> and i have field which is
>   <field name="onlynoun" type="teststop" indexed="true" stored="true"
> required="false"/>
>
> when i try to insert a doc like
> <field name="onlynoun">an elphant is an animal</field>
> it does not remove an and is from the phrase.

How can you tell?
The stored field value you get back will always be what you put in.
Analysis only affects what is indexed.
Try searching for "the animal" and it will probably match your
document since "the" will be removed as a stopword.

-Yonik
http://www.lucidimagination.com

Re: solr.StopFilterFactory not filtering words

Posted by darniz <rn...@edmunds.com>.

Thanks Yonik
So the stopFilter works is that if i give a string like "the elephant is an
animal", and when  i retrieve the document the stored value will always be
the same, only the index will be done on "elephant" and  "animal".
I was of the impression that Solr automatically takes out that words when it
is even storing the value.
How can i confirm that those are the only two terms indexex. 
AS you said i gave the following query to check is that the way it was
indexed so if i give q on "animal" it should return me that document, but i
did a q on "the animal" it should not return me that document since "the"
was removed while indexing.
Please let me know my assumption is correct, if this is the case then even i
did a q on "the elephant" it still returns me that document.

q="the%20elphant"&version=2.2&start=0&rows=10&indent=on&qf=onlynoun^5

And this the field how entered in to Solr
<field name="onlynoun">an elphant is an animal</field>

Thanks
-- 
View this message in context: http://www.nabble.com/solr.StopFilterFactory-not-filtering-words-tp25123903p25138502.html
Sent from the Solr - User mailing list archive at Nabble.com.