You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Robert Haschart <rh...@virginia.edu> on 2011/02/17 23:25:29 UTC
solr.KeepWordsFilterFactory confusion
I have a solr index where certain facet fields should only contain one
or more items from a limited list of values. To enforce this
restriction at index time I have been looking at using a
KeepWordFilterFactory. It seems it ought to work as I have it
implamented, and actually seems to work when tested through the admin
analysis page, but when I index a document with that filter in place
values that ought to be filtered out aren't. (I am running the solr 1.4
release)
I've added a new field type in schema.xml:
<fieldType name="formatFacet" class="solr.StrField"
sortMissingLast="true" omitNorms="true">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.KeepWordFilterFactory"
words="format_facet.txt" ignoreCase="false" />
</analyzer>
</fieldType>
placed a file format_facet.txt in the conf directory containing:
Book
Online
Microform
Journal/Magazine
Musical Score
Musical Recording
Thesis/Dissertation
Video
Streaming Video
Software/Multimedia
Photographs
Cassette
referenced this new field type with a field declaration in schema.xml
<field name="format_facet" type="formatFacet" indexed="true"
stored="true" multiValued="true" />
also have this dynamic field, but this seems irrelevant:
<dynamicField name="*_facet" type="string" indexed="true"
stored="true" multiValued="true" omitNorms="true" />
restarted the jetty server running the solr server.
and submitted a solr add document containing
format_facet=format_facet(1.0)={[Video, Streaming Video, Online,
Gooberhead, Book of the Month]}
Of these values only Video, Streaming Video and Online ought to end up
in the index, however all five values end up as format_facet values for
the solr item in question.
<arr name="format_facet">
<str>Video</str>
<str>Streaming Video</str>
<str>Online</str>
<str>Gooberhead</str>
<str>Book of the Month</str>
</arr>
I must be missing something fairly basic, since this doesn't seem
especially complicated.
Thanks in advance for any assistance,
-Bob Haschart
Re: solr.KeepWordsFilterFactory confusion
Posted by Ahmet Arslan <io...@yahoo.com>.
--- On Fri, 2/18/11, Robert Haschart <rh...@virginia.edu> wrote:
> From: Robert Haschart <rh...@virginia.edu>
> Subject: Re: solr.KeepWordsFilterFactory confusion
> To: solr-user@lucene.apache.org
> Date: Friday, February 18, 2011, 10:19 PM
> Thanks for your response. After
> making that change it seemed at first like it made no
> difference, after restarting the jetty server, and
> reindexing the test object, the display still shows:
>
> <arr name="format_facet">
> <str>Video</str>
> <str>Streaming Video</str>
> <str>Online</str>
> <str>Gooberhead</str>
> <str>Book of the Month</str>
> </arr>
>
> But it turns out that I had been making an incorrect
> assumption. I was looking at the retruned stored
> values for the solr document, and seeing the "Gooberhead"
> entry listed, and thinking that the analyzer wasn't
> running. However as I have subsequently figured out,
> the analyzers are not run on the data that is to be stored,
> only on the data that is to being indexed.
> So after making your change to that field type statement,
> if I search
> for format_facet:Gooberhead I
> get results = 0 which is what I'd expect. But seeing
> that the unexpected values are still stored with the solr
> document, it seems that I will have to take a different
> approach.
Facets are populated from indexed values. However deleted documents (and their terms) are not really deleted until an optimize. Issuing an optimize may help in your case.
Re: solr.KeepWordsFilterFactory confusion
Posted by Robert Haschart <rh...@virginia.edu>.
Thanks for your response. After making that change it seemed at first
like it made no difference, after restarting the jetty server, and
reindexing the test object, the display still shows:
<arr name="format_facet">
<str>Video</str>
<str>Streaming Video</str>
<str>Online</str>
<str>Gooberhead</str>
<str>Book of the Month</str>
</arr>
But it turns out that I had been making an incorrect assumption. I was
looking at the retruned stored values for the solr document, and seeing
the "Gooberhead" entry listed, and thinking that the analyzer wasn't
running. However as I have subsequently figured out, the analyzers are
not run on the data that is to be stored, only on the data that is to
being indexed.
So after making your change to that field type statement, if I search
for format_facet:Gooberhead I get results = 0 which is what I'd
expect. But seeing that the unexpected values are still stored with the
solr document, it seems that I will have to take a different approach.
Thanks again.
-Bob Haschart
Ahmet Arslan wrote:
>>I've added a new field type in schema.xml:
>>
>> <fieldType name="formatFacet"
>>class="solr.StrField" sortMissingLast="true"
>>omitNorms="true">
>> <analyzer type="index">
>> <tokenizer
>>class="solr.KeywordTokenizerFactory"/>
>> <filter
>>class="solr.KeepWordFilterFactory" words="format_facet.txt"
>>ignoreCase="false" />
>> </analyzer>
>> </fieldType>
>>
>>
>>
>
>class="solr.StrField" should be class="solr.TextField"
>
>
>
>
>
Re: solr.KeepWordsFilterFactory confusion
Posted by Ahmet Arslan <io...@yahoo.com>.
> I've added a new field type in schema.xml:
>
> <fieldType name="formatFacet"
> class="solr.StrField" sortMissingLast="true"
> omitNorms="true">
> <analyzer type="index">
> <tokenizer
> class="solr.KeywordTokenizerFactory"/>
> <filter
> class="solr.KeepWordFilterFactory" words="format_facet.txt"
> ignoreCase="false" />
> </analyzer>
> </fieldType>
>
class="solr.StrField" should be class="solr.TextField"