You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Itamar Syn-Hershko <it...@code972.com> on 2010/07/16 17:28:10 UTC
Scoring exact matches higher in a stemmed field
Hi all,
Consider the following string: "the buffalo buffaloes" [1].
When passed through a stemming analyzer, the resulting token would be
"buffalo buffalo" (assuming a good stemmer).
To enable exact searches, say I mark the original term and index it at
the same term position. So "the buffalo buffaloes" -> (buffalo buffalo$)
(buffalo buffaloes$) - now exact searches are allowed on the same field
without having 2 different fields [2].
However, with this approach default scoring isn't working well. What is
my best option at upgrading a match for an exact match of this sort,
also when using the same stemming analyzer, without using payloads on
the marked token?
In other words - how do I make documents containing "the buffalo
buffaloes" considered more relevant than docs containing the word
"buffalo" only once?
The trick here is to boost the marked token if found at search time.
While this sounds easy to do, I can't find the best approach on
implementing this - esp. since Similarity.float Idf(Index.Term term,
Searcher searcher) seem to have been deprecated for some reason.
Itamar.
[1]
http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo
:)
[2] Rationale:
http://www.code972.com/blog/2010/07/more-flexible-hebrew-indexing-hebmorph/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scoring exact matches higher in a stemmed field
Posted by Itamar Syn-Hershko <it...@code972.com>.
On 22/7/2010 9:20 PM, Shai Erera wrote:
> How is that different than extending QP?
>
Mainly because the problem I'm having isn't there, and doing it from
there doesn't feel right, and definitely not like solving the issue. I
want to explore what other options there are before doing anything, and
I started this thread because I hit a dead end after seeing Similarity
can no more be of help.
> About the "song of songs" example -- the result you describe is already what
> will happen. A document which contains just the word 'song' will score lower
> than a document containing "song of songs".
Incorrect, and I have a sample app to show that (this is how I thought
of this example for the first place).
Since while indexing the 2 words will be saved into index as 1:(song
song$) 2:(song songs$), short documents with one word "song" will score
higher than longer documents with "song of songs". This is a product of
Lucene's default tf/idf implementation which cares about a field's
length, and at this stage I want to avoid replacing it (with BM25 for
example).
> Also, what I'd do in such a case
> is search for the phrase (in addition to the rest), 'cause documents
> containing the word "songs" 100 times will score higher than the single
> document that will contain "song of songs" once ...
>
In one of my applications I am providing an "as typed" capability, which
does exactly what you are suggesting (looking for the $-ed terms only),
but I want my original analyzer (the one that also looks for non $-ed
terms) to do better scoring. Without this the implementation is somewhat
broken...
> If you just want a query "abc def" to rank higher if a document contains the
> exact words, then I'd go w/ the QP extension approach, or do other
> sophistication like searching for 'abc' '\"abc\"' etc. or something like
> that. There are many tricks you can do on your end, w/o overriding much in
> Lucene. Still, IMO extending QP is the easiest and gives you the control you
> need.
>
I am overriding stuff in Lucene either way. I also don't want an exact
match of a phrase to rank higher; I want an original term (saved as-is
with a $ marker) to score higher than a stemmed / lemmatized one
(without the marker). Sorry if the thread's title is misleading.
I'd have used payloads if it wasn't costly. So my question is: where do
I have control over boosting (or scoring), and also have access to the
term's text?
Thanks,
Itamar.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scoring exact matches higher in a stemmed field
Posted by Shai Erera <se...@gmail.com>.
>
> Ideally, that would be through a class or a function I can override or
> extend
>
How is that different than extending QP?
About the "song of songs" example -- the result you describe is already what
will happen. A document which contains just the word 'song' will score lower
than a document containing "song of songs". Also, what I'd do in such a case
is search for the phrase (in addition to the rest), 'cause documents
containing the word "songs" 100 times will score higher than the single
document that will contain "song of songs" once ...
If you just want a query "abc def" to rank higher if a document contains the
exact words, then I'd go w/ the QP extension approach, or do other
sophistication like searching for 'abc' '\"abc\"' etc. or something like
that. There are many tricks you can do on your end, w/o overriding much in
Lucene. Still, IMO extending QP is the easiest and gives you the control you
need.
Shai
On Mon, Jul 19, 2010 at 9:24 PM, Itamar Syn-Hershko <it...@code972.com>wrote:
> On 19/7/2010 5:50 PM, Shai Erera wrote:
>
>> If your analyzer outputs b and b$ in the same position, then the below
>> query
>> will already be what the QP output today If you want to incorporate
>> boosting, I can suggest that you extend QP, override newTermQuery for
>> example, and if the term is a stemmed term, then set the query's boost
>> (Query.setBoost) accordingly. Would that work for you?
>>
>>
> I want to avoid overriding the QP, and do this as a pluggable extension.
> What other options do I have other than what you've suggested?
>
> Ideally, that would be through a class or a function I can override or
> extend, so each term hit while searching will be examined. By checking its
> type and text (for suffix), that interface could double its weight (or
> boost). The similarity functions I mentioned could have provided this
> ability (see below). How can this be done without them?
>
> You'll need to check whether you want to boost terms inside phrases, or
>> entire phrases, and then override more methods from QP. But that approach
>> will get you the native product of the engine, I think.
>>
> Just to make sure we are on the same page here, here's an example (assuming
> the default tf/idf implementation in Lucene).
>
> I want to make sure anyone searching for "song of songs" will find texts
> discussing the biblical book, and have them ranked the highest, instead of
> having short texts containing one word "song" score higher.
>
> So what I do is have my stemming analyzer save the string "song of songs"
> like this, where each parenthesis represents a token position: (song song$)
> (song songs$).
>
> The part I'm missing is how to score terms with suffixes higher. The best
> approach seem to be looking at the term read by IndexReader and boost this
> finding somehow. The assumption is if IndexReader has read the term songs$
> it has been looked for, and therefore this is the exact word that has been
> queried for.
>
> Which is the best Lucene part to hijack for this mission?
>
> Alternatively, you
>> can set a payload on the stemmed terms and incorporate that into
>> Similarity,
>> but that's more costly.
>>
>>
> I had mentioned Payloads - this will get me exactly what I want but as you
> say are quite costly when used for almost every term in the index. If I
> could replace the suffix with Payloads I would have done this (byte vs.
> byte), but I'm using the suffix for one other thing.
>
> I don't follow that's been deprecated on Sim that you cannot use anymore?
>> All I see are 3 deprecated static methods which are related to norms ...
>>
>>
> In 2.3.2 there were these functions:
>
> public float idf(Term term, Searcher searcher)
>
> public float idf(Collection terms, Searcher searcher)
>
> These have been deprecated somewhere between that version and 2.9.2, and it
> seems like I could have used those for what I'm trying to do.
>
> Thanks,
>
> Itamar.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Scoring exact matches higher in a stemmed field
Posted by Itamar Syn-Hershko <it...@code972.com>.
On 19/7/2010 5:50 PM, Shai Erera wrote:
> If your analyzer outputs b and b$ in the same position, then the below query
> will already be what the QP output today If you want to incorporate
> boosting, I can suggest that you extend QP, override newTermQuery for
> example, and if the term is a stemmed term, then set the query's boost
> (Query.setBoost) accordingly. Would that work for you?
>
I want to avoid overriding the QP, and do this as a pluggable extension.
What other options do I have other than what you've suggested?
Ideally, that would be through a class or a function I can override or
extend, so each term hit while searching will be examined. By checking
its type and text (for suffix), that interface could double its weight
(or boost). The similarity functions I mentioned could have provided
this ability (see below). How can this be done without them?
> You'll need to check whether you want to boost terms inside phrases, or
> entire phrases, and then override more methods from QP. But that approach
> will get you the native product of the engine, I think.
Just to make sure we are on the same page here, here's an example
(assuming the default tf/idf implementation in Lucene).
I want to make sure anyone searching for "song of songs" will find texts
discussing the biblical book, and have them ranked the highest, instead
of having short texts containing one word "song" score higher.
So what I do is have my stemming analyzer save the string "song of
songs" like this, where each parenthesis represents a token position:
(song song$) (song songs$).
The part I'm missing is how to score terms with suffixes higher. The
best approach seem to be looking at the term read by IndexReader and
boost this finding somehow. The assumption is if IndexReader has read
the term songs$ it has been looked for, and therefore this is the exact
word that has been queried for.
Which is the best Lucene part to hijack for this mission?
> Alternatively, you
> can set a payload on the stemmed terms and incorporate that into Similarity,
> but that's more costly.
>
I had mentioned Payloads - this will get me exactly what I want but as
you say are quite costly when used for almost every term in the index.
If I could replace the suffix with Payloads I would have done this (byte
vs. byte), but I'm using the suffix for one other thing.
> I don't follow that's been deprecated on Sim that you cannot use anymore?
> All I see are 3 deprecated static methods which are related to norms ...
>
In 2.3.2 there were these functions:
public float idf(Term term, Searcher searcher)
public float idf(Collection terms, Searcher searcher)
These have been deprecated somewhere between that version and 2.9.2, and
it seems like I could have used those for what I'm trying to do.
Thanks,
Itamar.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scoring exact matches higher in a stemmed field
Posted by Shai Erera <se...@gmail.com>.
If your analyzer outputs b and b$ in the same position, then the below query
will already be what the QP output today If you want to incorporate
boosting, I can suggest that you extend QP, override newTermQuery for
example, and if the term is a stemmed term, then set the query's boost
(Query.setBoost) accordingly. Would that work for you?
You'll need to check whether you want to boost terms inside phrases, or
entire phrases, and then override more methods from QP. But that approach
will get you the native product of the engine, I think. Alternatively, you
can set a payload on the stemmed terms and incorporate that into Similarity,
but that's more costly.
I don't follow that's been deprecated on Sim that you cannot use anymore?
All I see are 3 deprecated static methods which are related to norms ...
Shai
On Sat, Jul 17, 2010 at 9:04 PM, Itamar Syn-Hershko <it...@code972.com>wrote:
> Shai, you got it right. I want to be able to send "b bb" through the QP
> with my custom analyzer, and get back "(b b$) (b bb$)" -- 2 terms with 2
> tokens in the same position for each.
>
> I want this to be a native product of the engine, as opposed to forcing
> this from the query end. I'm using different types of queries (Bool,
> DisMax), and I'm actually interested in using the QP itself. Instead of
> going through all sub-queries post-parsing and boosting terms ending with $,
> I want some sort of a plugin mechanism to do this for me per result. The
> easiest path would be subcalssing Similarity, if only the relevant functions
> wouldn't have been deprecated...
>
> Are there any other ways to do so? For example, is this doable with
> function queries (since access to the actual term is required)?
>
> Itamar.
>
> On 16/7/2010 8:01 PM, Shai Erera wrote:
>
>> Depends for which query no? ;)
>>
>> Sounds like you want to simulate the QP behavior
>> http://lucene.apache.org/java/2_4_0/queryparsersyntax.html for
>> boosting. Meaning, if for the query "b" you want to simulate the query
>> "b OR b$^2" and have matches of b$ count more than b, then I'd follow
>> how QP does it - create the query programmatically or something (I'm
>> not near the code at the moment so I cannot give a more concrete
>> approach).
>>
>> If you want b and b$ to count the same, then that's already the
>> behavior - i.e., docs containing both will score higher.
>>
>> If I misunderstood your question, then plea correct me.
>>
>> Shai
>>
>> On Friday, July 16, 2010, Itamar Syn-Hershko<it...@code972.com> wrote:
>>
>>
>>> Hi all,
>>>
>>>
>>> Consider the following string: "the buffalo buffaloes" [1].
>>>
>>>
>>> When passed through a stemming analyzer, the resulting token would be
>>> "buffalo buffalo" (assuming a good stemmer).
>>>
>>>
>>> To enable exact searches, say I mark the original term and index it at
>>> the same term position. So "the buffalo buffaloes" -> (buffalo buffalo$)
>>> (buffalo buffaloes$) - now exact searches are allowed on the same field
>>> without having 2 different fields [2].
>>>
>>>
>>> However, with this approach default scoring isn't working well. What is
>>> my best option at upgrading a match for an exact match of this sort, also
>>> when using the same stemming analyzer, without using payloads on the marked
>>> token?
>>>
>>>
>>> In other words - how do I make documents containing "the buffalo
>>> buffaloes" considered more relevant than docs containing the word "buffalo"
>>> only once?
>>>
>>>
>>> The trick here is to boost the marked token if found at search time.
>>> While this sounds easy to do, I can't find the best approach on implementing
>>> this - esp. since Similarity.float Idf(Index.Term term, Searcher searcher)
>>> seem to have been deprecated for some reason.
>>>
>>>
>>> Itamar.
>>>
>>>
>>> [1]
>>> http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo:)
>>>
>>> [2] Rationale:
>>> http://www.code972.com/blog/2010/07/more-flexible-hebrew-indexing-hebmorph/
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Scoring exact matches higher in a stemmed field
Posted by Itamar Syn-Hershko <it...@code972.com>.
Shai, you got it right. I want to be able to send "b bb" through the QP
with my custom analyzer, and get back "(b b$) (b bb$)" -- 2 terms with 2
tokens in the same position for each.
I want this to be a native product of the engine, as opposed to forcing
this from the query end. I'm using different types of queries (Bool,
DisMax), and I'm actually interested in using the QP itself. Instead of
going through all sub-queries post-parsing and boosting terms ending
with $, I want some sort of a plugin mechanism to do this for me per
result. The easiest path would be subcalssing Similarity, if only the
relevant functions wouldn't have been deprecated...
Are there any other ways to do so? For example, is this doable with
function queries (since access to the actual term is required)?
Itamar.
On 16/7/2010 8:01 PM, Shai Erera wrote:
> Depends for which query no? ;)
>
> Sounds like you want to simulate the QP behavior
> http://lucene.apache.org/java/2_4_0/queryparsersyntax.html for
> boosting. Meaning, if for the query "b" you want to simulate the query
> "b OR b$^2" and have matches of b$ count more than b, then I'd follow
> how QP does it - create the query programmatically or something (I'm
> not near the code at the moment so I cannot give a more concrete
> approach).
>
> If you want b and b$ to count the same, then that's already the
> behavior - i.e., docs containing both will score higher.
>
> If I misunderstood your question, then plea correct me.
>
> Shai
>
> On Friday, July 16, 2010, Itamar Syn-Hershko<it...@code972.com> wrote:
>
>> Hi all,
>>
>>
>> Consider the following string: "the buffalo buffaloes" [1].
>>
>>
>> When passed through a stemming analyzer, the resulting token would be "buffalo buffalo" (assuming a good stemmer).
>>
>>
>> To enable exact searches, say I mark the original term and index it at the same term position. So "the buffalo buffaloes" -> (buffalo buffalo$) (buffalo buffaloes$) - now exact searches are allowed on the same field without having 2 different fields [2].
>>
>>
>> However, with this approach default scoring isn't working well. What is my best option at upgrading a match for an exact match of this sort, also when using the same stemming analyzer, without using payloads on the marked token?
>>
>>
>> In other words - how do I make documents containing "the buffalo buffaloes" considered more relevant than docs containing the word "buffalo" only once?
>>
>>
>> The trick here is to boost the marked token if found at search time. While this sounds easy to do, I can't find the best approach on implementing this - esp. since Similarity.float Idf(Index.Term term, Searcher searcher) seem to have been deprecated for some reason.
>>
>>
>> Itamar.
>>
>>
>> [1] http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo :)
>>
>> [2] Rationale: http://www.code972.com/blog/2010/07/more-flexible-hebrew-indexing-hebmorph/
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Scoring exact matches higher in a stemmed field
Posted by Shai Erera <se...@gmail.com>.
Depends for which query no? ;)
Sounds like you want to simulate the QP behavior
http://lucene.apache.org/java/2_4_0/queryparsersyntax.html for
boosting. Meaning, if for the query "b" you want to simulate the query
"b OR b$^2" and have matches of b$ count more than b, then I'd follow
how QP does it - create the query programmatically or something (I'm
not near the code at the moment so I cannot give a more concrete
approach).
If you want b and b$ to count the same, then that's already the
behavior - i.e., docs containing both will score higher.
If I misunderstood your question, then plea correct me.
Shai
On Friday, July 16, 2010, Itamar Syn-Hershko <it...@code972.com> wrote:
> Hi all,
>
>
> Consider the following string: "the buffalo buffaloes" [1].
>
>
> When passed through a stemming analyzer, the resulting token would be "buffalo buffalo" (assuming a good stemmer).
>
>
> To enable exact searches, say I mark the original term and index it at the same term position. So "the buffalo buffaloes" -> (buffalo buffalo$) (buffalo buffaloes$) - now exact searches are allowed on the same field without having 2 different fields [2].
>
>
> However, with this approach default scoring isn't working well. What is my best option at upgrading a match for an exact match of this sort, also when using the same stemming analyzer, without using payloads on the marked token?
>
>
> In other words - how do I make documents containing "the buffalo buffaloes" considered more relevant than docs containing the word "buffalo" only once?
>
>
> The trick here is to boost the marked token if found at search time. While this sounds easy to do, I can't find the best approach on implementing this - esp. since Similarity.float Idf(Index.Term term, Searcher searcher) seem to have been deprecated for some reason.
>
>
> Itamar.
>
>
> [1] http://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo :)
>
> [2] Rationale: http://www.code972.com/blog/2010/07/more-flexible-hebrew-indexing-hebmorph/
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org