You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Reece <li...@gmail.com> on 2008/02/18 21:56:57 UTC
Questions about filters and scoring
Hello Everyone,
First off, sorry about the thread hijack earlier, it was not intentional.
Back to the point though, I'm having some issues getting
SOLR to work with our dataset. I'm using it to index ticket data for
our technical support department. Below are a few of the problems
I've been having, and the wiki hasn't had much to say about them.
1) As an example, searching for "binarydata_groupdocument_fk" returns
nothing, while searching for "BinaryData_GroupDocument_FK" returns
results. I have the lowercasefilterfactory applied to both the index
and query analyzers. Does this not actually set everything to lower
case? From the wiki at
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters, it says
"Creates tokens by lowercasing all letters and dropping non-letters"
but that does not seem to be happening here. Am I forgetting to
configure something?
2) Some of our data is one sentence. Some is over 5 MB of text. When
searching for a term, it's returning the one sentence data first
because the fieldNorm is so different (0.4 for one, 0.002 for others).
Is there a way to disable using the fieldnorm in the score
calculation? An alternative I tried was posting parts of the data in
as different values of the field (so having multiple tags of that
field-name in the add xml post), but that appeared to have zero effect
on the results - even the querydebugger showed the exact same
calculation for the search. Does anyone know how to disable the
fieldnorm, or have the score created from adding the scores from each
value of a multivalued field?
3) I discovered that searching for '"certificate not found"' (using
the double quotes for a phrase here) did not return any results, even
though the phrase did exist (and was lower case originally too, so
different than my first issue). I discovered it was because of the
stopword "not", but the same stopfilterfactory was applied to both the
index and query analyzers. Am I doing something wrong there? As a
workaround I'm having php manually removing stopwords from the
querystring, which is a real pain. I'm thinking my filters aren't being
applied correctly since this is similar to issue #1 but with a different
filter.
Here is my fieldtype I do the actual searches on:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- in this example, we will only use synonyms at query time
<filter class="solr.SynonymFilterFactory"
synonyms="index_synonyms.txt" ignoreCase="true" expand="false"/>
-->
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory"
synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
Any help or advice would be greatly appreciated, thanks!
-Reece
Re: Questions about filters and scoring
Posted by Reece <li...@gmail.com>.
The omitNorms="true" attribute worked perfectly, thanks Yonik!
Also, the stopword issue isn't happening anymore, go figure. I
probably had a mistype or something as well, Thanks for the help!
-Reece
On Feb 18, 2008 7:17 PM, Reece <li...@gmail.com> wrote:
> For #1, I just testing again and found the problem.
> WordDelimiterFilterFactory was splitting the words up because it had
> capitals in the middle of the word, so a lower case version was seen
> as a different set of tokens.
>
> For #2, I'll try using that attribute for the fieldtype and let you
> know how it goes, but that looks like exactly what I needed.
>
> For #3, I'll test it again tomorrow and make sure I didn't have a
> mistype or something.
>
> Thanks for the help!
>
> -Reece
>
>
>
>
> On Feb 18, 2008 5:11 PM, Yonik Seeley <yo...@apache.org> wrote:
> >
> > On Feb 18, 2008 5:05 PM, Yonik Seeley <yo...@apache.org> wrote:
> > > On Feb 18, 2008 4:42 PM, Yonik Seeley <yo...@apache.org> wrote:
> > > > Hmmm, looks like a recent change in lucene probably causes this bug.
> > >
> > > Nope... I just checked Solr 1.2, and it shows the same behavior.
> > > With the example data, a query of
> > > optimized for high
> > > finds the solr document, but
> > > "optimized for high"
> > > does not.
> >
> > Scratch that... both Solr 1.2 and trunk seem to work fine for me.
> > My test was flawed because I was searching for "optimized for high"
> > while the solr document had a misspelling: Optimizied
> >
> > -Yonik
> >
>
Re: Questions about filters and scoring
Posted by Reece <li...@gmail.com>.
For #1, I just testing again and found the problem.
WordDelimiterFilterFactory was splitting the words up because it had
capitals in the middle of the word, so a lower case version was seen
as a different set of tokens.
For #2, I'll try using that attribute for the fieldtype and let you
know how it goes, but that looks like exactly what I needed.
For #3, I'll test it again tomorrow and make sure I didn't have a
mistype or something.
Thanks for the help!
-Reece
On Feb 18, 2008 5:11 PM, Yonik Seeley <yo...@apache.org> wrote:
>
> On Feb 18, 2008 5:05 PM, Yonik Seeley <yo...@apache.org> wrote:
> > On Feb 18, 2008 4:42 PM, Yonik Seeley <yo...@apache.org> wrote:
> > > Hmmm, looks like a recent change in lucene probably causes this bug.
> >
> > Nope... I just checked Solr 1.2, and it shows the same behavior.
> > With the example data, a query of
> > optimized for high
> > finds the solr document, but
> > "optimized for high"
> > does not.
>
> Scratch that... both Solr 1.2 and trunk seem to work fine for me.
> My test was flawed because I was searching for "optimized for high"
> while the solr document had a misspelling: Optimizied
>
> -Yonik
>
Re: Questions about filters and scoring
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 18, 2008 5:05 PM, Yonik Seeley <yo...@apache.org> wrote:
> On Feb 18, 2008 4:42 PM, Yonik Seeley <yo...@apache.org> wrote:
> > Hmmm, looks like a recent change in lucene probably causes this bug.
>
> Nope... I just checked Solr 1.2, and it shows the same behavior.
> With the example data, a query of
> optimized for high
> finds the solr document, but
> "optimized for high"
> does not.
Scratch that... both Solr 1.2 and trunk seem to work fine for me.
My test was flawed because I was searching for "optimized for high"
while the solr document had a misspelling: Optimizied
-Yonik
Re: Questions about filters and scoring
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 18, 2008 4:42 PM, Yonik Seeley <yo...@apache.org> wrote:
> Hmmm, looks like a recent change in lucene probably causes this bug.
Nope... I just checked Solr 1.2, and it shows the same behavior.
With the example data, a query of
optimized for high
finds the solr document, but
"optimized for high"
does not.
-Yonik
Re: Questions about filters and scoring
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 18, 2008 4:50 PM, Lance Norskog <go...@gmail.com> wrote:
> But then would not 'certificate anystopword found' match your phrase?
Yes... stopwords are ignored, so that's what it should do in general.
-Yonik
RE: Questions about filters and scoring
Posted by Lance Norskog <go...@gmail.com>.
> 3)
But then would not 'certificate anystopword found' match your phrase? I
wound up making a separate index without stopwords just so that my phrase
lookups would work. (I do not have the luxury of re-indexing, so now I'm
stuck with this design even if there is a better one.)
I also made one with the phonetic DoubleMetaphone analyzer. This is really
useful, especially for spell checking.
Cheers,
Lance
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, February 18, 2008 1:43 PM
To: solr-user@lucene.apache.org; Reece
Subject: Re: Questions about filters and scoring
On Feb 18, 2008 3:56 PM, Reece <li...@gmail.com> wrote:
> Hello Everyone,
>
> First off, sorry about the thread hijack earlier, it was not intentional.
>
> Back to the point though, I'm having some issues getting SOLR to work
> with our dataset. I'm using it to index ticket data for our technical
> support department. Below are a few of the problems I've been having,
> and the wiki hasn't had much to say about them.
>
> 1) As an example, searching for "binarydata_groupdocument_fk" returns
> nothing, while searching for "BinaryData_GroupDocument_FK" returns
> results. I have the lowercasefilterfactory applied to both the index
> and query analyzers. Does this not actually set everything to lower
> case? From the wiki at
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters, it says
> "Creates tokens by lowercasing all letters and dropping non-letters"
> but that does not seem to be happening here. Am I forgetting to
> configure something?
Did you re-index?
> 2) Some of our data is one sentence. Some is over 5 MB of text. When
> searching for a term, it's returning the one sentence data first
> because the fieldNorm is so different (0.4 for one, 0.002 for others).
> Is there a way to disable using the fieldnorm in the score
> calculation?
It's probably Lucene's default length normalization over-emphasizing short
fields.
You could use a better similarity for your data, or turn off length
normalization by setting omitNorms="true" for that field in the schema and
then re-indexing (make sure to delete the old index entirely first).
> An alternative I tried was posting parts of the data in as different
> values of the field (so having multiple tags of that field-name in the
> add xml post), but that appeared to have zero effect on the results -
> even the querydebugger showed the exact same calculation for the
> search. Does anyone know how to disable the fieldnorm, or have the
> score created from adding the scores from each value of a multivalued
> field?
>
> 3) I discovered that searching for '"certificate not found"' (using
> the double quotes for a phrase here) did not return any results, even
> though the phrase did exist (and was lower case originally too, so
> different than my first issue). I discovered it was because of the
> stopword "not", but the same stopfilterfactory was applied to both the
> index and query analyzers. Am I doing something wrong there? As a
> workaround I'm having php manually removing stopwords from the
> querystring, which is a real pain. I'm thinking my filters aren't
> being applied correctly since this is similar to issue #1 but with a
> different filter.
Hmmm, looks like a recent change in lucene probably causes this bug.
Could you open a new Solr JIRA issue to report this bug?
-Yonik
Re: Questions about filters and scoring
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 18, 2008 3:56 PM, Reece <li...@gmail.com> wrote:
> Hello Everyone,
>
> First off, sorry about the thread hijack earlier, it was not intentional.
>
> Back to the point though, I'm having some issues getting
> SOLR to work with our dataset. I'm using it to index ticket data for
> our technical support department. Below are a few of the problems
> I've been having, and the wiki hasn't had much to say about them.
>
> 1) As an example, searching for "binarydata_groupdocument_fk" returns
> nothing, while searching for "BinaryData_GroupDocument_FK" returns
> results. I have the lowercasefilterfactory applied to both the index
> and query analyzers. Does this not actually set everything to lower
> case? From the wiki at
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters, it says
> "Creates tokens by lowercasing all letters and dropping non-letters"
> but that does not seem to be happening here. Am I forgetting to
> configure something?
Did you re-index?
> 2) Some of our data is one sentence. Some is over 5 MB of text. When
> searching for a term, it's returning the one sentence data first
> because the fieldNorm is so different (0.4 for one, 0.002 for others).
> Is there a way to disable using the fieldnorm in the score
> calculation?
It's probably Lucene's default length normalization over-emphasizing
short fields.
You could use a better similarity for your data, or turn off length
normalization by
setting omitNorms="true" for that field in the schema and then
re-indexing (make sure to delete the old index entirely first).
> An alternative I tried was posting parts of the data in
> as different values of the field (so having multiple tags of that
> field-name in the add xml post), but that appeared to have zero effect
> on the results - even the querydebugger showed the exact same
> calculation for the search. Does anyone know how to disable the
> fieldnorm, or have the score created from adding the scores from each
> value of a multivalued field?
>
> 3) I discovered that searching for '"certificate not found"' (using
> the double quotes for a phrase here) did not return any results, even
> though the phrase did exist (and was lower case originally too, so
> different than my first issue). I discovered it was because of the
> stopword "not", but the same stopfilterfactory was applied to both the
> index and query analyzers. Am I doing something wrong there? As a
> workaround I'm having php manually removing stopwords from the
> querystring, which is a real pain. I'm thinking my filters aren't being
> applied correctly since this is similar to issue #1 but with a different
> filter.
Hmmm, looks like a recent change in lucene probably causes this bug.
Could you open a new Solr JIRA issue to report this bug?
-Yonik