You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kolodziej Christian <Ko...@huberverlag.de> on 2008/08/28 16:45:33 UTC

Problems using SnowballPorterFilterFactory for german

Hello everyone,

I'm quite new in using solr but I managed to create some indexes and I wrote some nice working PHP classes for adding, inserting, querying and deleting records. But when I perform a search there have been some crazy phenomenons. If a word is searched exactly in the right spelling, it was found (okay, I didn't except anything else). But when I searched the same correct spelled word in the fuzzy way (by adding ~0.5), it wasn't found any more. Hello?!

It costs some hours of today to find out the problem, it's the german snowball porter filter (both German and German2). When using the EnglishPorterFilterFactory everything is working well (but it isn't very commendable if nearly all contents are german I think) and if no porter filter is used the result is as expected, too.

Therefore my question to you: Does everyone know about this problem and is there a solution? My schema.xml looks as follows:
(...)
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer type="index">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <!-- in this example, we will only use synonyms at query time
        <filter class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
        -->
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
      <analyzer type="query">
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"/>
        <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"/>
        <filter class="solr.LowerCaseFilterFactory"/>
        <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
      </analyzer>
    </fieldType>
(...)

I'm pretty happy now when I leave off work soon, but maybe there is an error on mine? Thank for any answer and have a nice evening now!

Best regards from Karlsruhe, Germany
Christian

AW: Problems using SnowballPorterFilterFactory for german

Posted by Kolodziej Christian <Ko...@huberverlag.de>.
>On Thu, Aug 28, 2008 at 10:45 AM, Kolodziej Christian
><Ko...@huberverlag.de> wrote:
>> I'm quite new in using solr but I managed to create some indexes and I
>wrote some nice working PHP classes for adding, inserting, querying and
>deleting records. But when I perform a search there have been some crazy
>phenomenons. If a word is searched exactly in the right spelling, it was
>found (okay, I didn't except anything else). But when I searched the
>same correct spelled word in the fuzzy way (by adding ~0.5), it wasn't
>found any more. Hello?!
>>
>> It costs some hours of today to find out the problem, it's the german
>snowball porter filter
>
>Yes, stemmers don't work well in conjunction with prefix queries,
>fuzzy queries, etc, because partial words can't be accurately stemmed.

Thanks for this quick answer, you were so fast that I'm still at work ;-) But your answer make me asure that my schema.xml is as good as it could be for the moment.

Best regards
Christian

Re: Problems using SnowballPorterFilterFactory for german

Posted by Yonik Seeley <yo...@apache.org>.
On Thu, Aug 28, 2008 at 10:45 AM, Kolodziej Christian
<Ko...@huberverlag.de> wrote:
> I'm quite new in using solr but I managed to create some indexes and I wrote some nice working PHP classes for adding, inserting, querying and deleting records. But when I perform a search there have been some crazy phenomenons. If a word is searched exactly in the right spelling, it was found (okay, I didn't except anything else). But when I searched the same correct spelled word in the fuzzy way (by adding ~0.5), it wasn't found any more. Hello?!
>
> It costs some hours of today to find out the problem, it's the german snowball porter filter

Yes, stemmers don't work well in conjunction with prefix queries,
fuzzy queries, etc, because partial words can't be accurately stemmed.

-Yonik