You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cedric Ho <ce...@gmail.com> on 2008/02/13 10:31:41 UTC
How to pass additional information into Similarity.scorePayload(...)
Hi all,
My problem is I have some additional weighting info that come with
each search. And I need to take both the weighting info and the
payload to calculate scores.
So how do I access the weighting info in
Similarity.scorePayload(String,byte[],int,int) ? I've thought about
using a ThreadLocal, but this seems like a hack, are there better
options ?
Regards,
Cedric
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Grant Ingersoll <gs...@apache.org>.
The only Query that currently utilizes the scorePayload functionality
is the BoostingTermQuery. I guess I would have a look at that as a
starting point.
On Feb 13, 2008, at 8:11 PM, Cedric Ho wrote:
> I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> WildcardQuery, BooleanQuery and many of the SpanQueries.
>
> The info I am going to pass in is just some weightings for different
> part of the indexed contents. For example if the payload indicate that
> a term is in the 2nd paragraph, then I'll take the weighting for the
> 2nd paragraph and multiply it by the score.
>
> So it seems without writing my own query there's no way to do it ?
>
> Cedric
>
>
> On Feb 13, 2008 7:53 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> Are you writing your own Query? What kind of info did you have in
>> mind? scorePayload is called from the query scoring class, so I am
>> not sure how you would pass in info to it unless you were writing
>> your
>> own Query class.
>>
>> -Grant
>>
>>
>> On Feb 13, 2008, at 4:31 AM, Cedric Ho wrote:
>>
>>> Hi all,
>>>
>>> My problem is I have some additional weighting info that come with
>>> each search. And I need to take both the weighting info and the
>>> payload to calculate scores.
>>>
>>> So how do I access the weighting info in
>>> Similarity.scorePayload(String,byte[],int,int) ? I've thought about
>>> using a ThreadLocal, but this seems like a hack, are there better
>>> options ?
>>>
>>>
>>> Regards,
>>> Cedric
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com
http://www.lucenebootcamp.com
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Cedric Ho <ce...@gmail.com>.
Thanks ~ Yes it seems this would be quite difficult to achieve with
Lucene. Nevermind, I'll try to figure out a workaround for it.
Thanks for helping =)
Cedric
On Feb 16, 2008 5:30 AM, Paul Elschot <pa...@xs4all.nl> wrote:
> Hi Cedric,
>
> I think I'm beginning to get the point of the [10/5/2],
> and why you called that requirement a bit strange, see below.
>
> To use both normal position info and paragraph position info
> you'll need two separate, one normal, and one for the paragraphs.
>
> To use the normal field to determine the matches, and the
> paragraph field to determine the weightings of these matches
> the TermPositions of both fields will have to be advanced
> completely in sync. That is possible, but not really nice to do.
> If Lucene had multiple positions for an indexed term, it
> would be straightforward.
> But as long as that is not the case, you'll either have to advance
> the two TermPositions in sync, or use payloads with the
> paragraph numbers.
>
> Or you could relax the paragraph numbering requirement
> into a positional requirement, and use the modified SpanFirstQuery.
> That could be done by using an avarage paragraph length to
> determine the weight at the matching position.
> As this is easy to implement, I'd first implement this and try to sell
> it to the users :)
>
> At that marketing moment you might as well ask the users
> what they think of matches that cross paragraph borders.
> Do you already have a firm requirement for that case?
>
> SpanNotQuery can be used to prevent matches over paragraph
> borders when these are indexed as such, but I would not expect
> that you would need those, given the fuzzyness of the [10/5/2].
>
> Regards,
> Paul Elschot
>
>
> Op Friday 15 February 2008 09:45:58 schreef Cedric Ho:
>
> > Hi Paul,
> >
> > Do you mean the following?
> >
> > e.g. to index this: "first second third <paragraphBorder> forth fifth six"
> >
> > originally it would be indexed as:
> > (first,0) (second,1) (third,2) (forth,3) (fifth,4) (six,5)
> >
> > now it will be:
> > (first,0) (second,0) (third,0) (forth,1) (fifth,1) (six,1)
> >
> > Then those Query classes that depends on the positional information
> > (PhraseQuery, SpanQueries) won't work then? unfortunately I'll need
> > those Query classes as well.
> >
> > Cedric
> >
> >
> > > For each word in the input stream make sure that the position
> > > at which it is indexed in an extra field is the same as the paragraph
> > > number. That will involve only allowing a position increment at
> > > a paragraph border during indexing.
> > > Call this extra field the paragraph field if you will.
> > >
> > > Then, during search, search for a Term in paragraph field, and
> > > use the position from that field, i.e. the paragraph number
> > > to find a weight for the found term.
> > > Have a look at PhraseQuery on how to use term positions during
> > > search. It computes relative positions, but it works on the absolute
> > > positions that it gets from the index.
> > >
> > > SpanFirstQuery also allows to do that, it's a bit more involved, but
> > > in the end it works from the same absolute positions from the index.
> > > The version at the jira issue will even allow to use the length of the
> > > matching spans as the absolute paragraph number, which, in turn,
> > > allows the use of a Similarity for the paragraph weights [10/5/2].
> > >
> > > There is nothing special about indexed term positions; any term can
> > > be indexed at any position in a field. Lucene will take advantage of
> > > the incremental nature of positions by storing only compressed
> > > differences of positions in the index, but during search the original
> > > positions are directly available, You can do the same with payloads,
> > > but why reimplement something that is already available?
> > >
> > > Payloads have better uses than positional info, for one they are
> > > great to avoid disjunctions. For example for verbs, one could
> > > index only the stem and use a payload for the actual inflected
> > > form (singular/plural, past/present, first/second/third person, etc).
> > >
> > > Regards,
> > > Paul Elschot
> > >
> >
>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Paul Elschot <pa...@xs4all.nl>.
Hi Cedric,
I think I'm beginning to get the point of the [10/5/2],
and why you called that requirement a bit strange, see below.
To use both normal position info and paragraph position info
you'll need two separate, one normal, and one for the paragraphs.
To use the normal field to determine the matches, and the
paragraph field to determine the weightings of these matches
the TermPositions of both fields will have to be advanced
completely in sync. That is possible, but not really nice to do.
If Lucene had multiple positions for an indexed term, it
would be straightforward.
But as long as that is not the case, you'll either have to advance
the two TermPositions in sync, or use payloads with the
paragraph numbers.
Or you could relax the paragraph numbering requirement
into a positional requirement, and use the modified SpanFirstQuery.
That could be done by using an avarage paragraph length to
determine the weight at the matching position.
As this is easy to implement, I'd first implement this and try to sell
it to the users :)
At that marketing moment you might as well ask the users
what they think of matches that cross paragraph borders.
Do you already have a firm requirement for that case?
SpanNotQuery can be used to prevent matches over paragraph
borders when these are indexed as such, but I would not expect
that you would need those, given the fuzzyness of the [10/5/2].
Regards,
Paul Elschot
Op Friday 15 February 2008 09:45:58 schreef Cedric Ho:
> Hi Paul,
>
> Do you mean the following?
>
> e.g. to index this: "first second third <paragraphBorder> forth fifth six"
>
> originally it would be indexed as:
> (first,0) (second,1) (third,2) (forth,3) (fifth,4) (six,5)
>
> now it will be:
> (first,0) (second,0) (third,0) (forth,1) (fifth,1) (six,1)
>
> Then those Query classes that depends on the positional information
> (PhraseQuery, SpanQueries) won't work then? unfortunately I'll need
> those Query classes as well.
>
> Cedric
>
>
> > For each word in the input stream make sure that the position
> > at which it is indexed in an extra field is the same as the paragraph
> > number. That will involve only allowing a position increment at
> > a paragraph border during indexing.
> > Call this extra field the paragraph field if you will.
> >
> > Then, during search, search for a Term in paragraph field, and
> > use the position from that field, i.e. the paragraph number
> > to find a weight for the found term.
> > Have a look at PhraseQuery on how to use term positions during
> > search. It computes relative positions, but it works on the absolute
> > positions that it gets from the index.
> >
> > SpanFirstQuery also allows to do that, it's a bit more involved, but
> > in the end it works from the same absolute positions from the index.
> > The version at the jira issue will even allow to use the length of the
> > matching spans as the absolute paragraph number, which, in turn,
> > allows the use of a Similarity for the paragraph weights [10/5/2].
> >
> > There is nothing special about indexed term positions; any term can
> > be indexed at any position in a field. Lucene will take advantage of
> > the incremental nature of positions by storing only compressed
> > differences of positions in the index, but during search the original
> > positions are directly available, You can do the same with payloads,
> > but why reimplement something that is already available?
> >
> > Payloads have better uses than positional info, for one they are
> > great to avoid disjunctions. For example for verbs, one could
> > index only the stem and use a payload for the actual inflected
> > form (singular/plural, past/present, first/second/third person, etc).
> >
> > Regards,
> > Paul Elschot
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Cedric Ho <ce...@gmail.com>.
Hi Paul,
Do you mean the following?
e.g. to index this: "first second third <paragraphBorder> forth fifth six"
originally it would be indexed as:
(first,0) (second,1) (third,2) (forth,3) (fifth,4) (six,5)
now it will be:
(first,0) (second,0) (third,0) (forth,1) (fifth,1) (six,1)
Then those Query classes that depends on the positional information
(PhraseQuery, SpanQueries) won't work then? unfortunately I'll need
those Query classes as well.
Cedric
> For each word in the input stream make sure that the position
> at which it is indexed in an extra field is the same as the paragraph
> number. That will involve only allowing a position increment at
> a paragraph border during indexing.
> Call this extra field the paragraph field if you will.
>
> Then, during search, search for a Term in paragraph field, and
> use the position from that field, i.e. the paragraph number
> to find a weight for the found term.
> Have a look at PhraseQuery on how to use term positions during
> search. It computes relative positions, but it works on the absolute
> positions that it gets from the index.
>
> SpanFirstQuery also allows to do that, it's a bit more involved, but
> in the end it works from the same absolute positions from the index.
> The version at the jira issue will even allow to use the length of the
> matching spans as the absolute paragraph number, which, in turn,
> allows the use of a Similarity for the paragraph weights [10/5/2].
>
> There is nothing special about indexed term positions; any term can
> be indexed at any position in a field. Lucene will take advantage of
> the incremental nature of positions by storing only compressed
> differences of positions in the index, but during search the original
> positions are directly available, You can do the same with payloads,
> but why reimplement something that is already available?
>
> Payloads have better uses than positional info, for one they are
> great to avoid disjunctions. For example for verbs, one could
> index only the stem and use a payload for the actual inflected
> form (singular/plural, past/present, first/second/third person, etc).
>
> Regards,
> Paul Elschot
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Paul Elschot <pa...@xs4all.nl>.
Op Friday 15 February 2008 02:47:14 schreef Cedric Ho:
> Sorry that I didn't make myself clear.
>
> [10/5/2] means for terms found in the 1st paragraph, give it score*10,
> for terms in the 2nd, give it score*5, etc.
>
> So I don't know how to do this scoring if the position (paragraph)
> information is in a separate field.
For each word in the input stream make sure that the position
at which it is indexed in an extra field is the same as the paragraph
number. That will involve only allowing a position increment at
a paragraph border during indexing.
Call this extra field the paragraph field if you will.
Then, during search, search for a Term in paragraph field, and
use the position from that field, i.e. the paragraph number
to find a weight for the found term.
Have a look at PhraseQuery on how to use term positions during
search. It computes relative positions, but it works on the absolute
positions that it gets from the index.
SpanFirstQuery also allows to do that, it's a bit more involved, but
in the end it works from the same absolute positions from the index.
The version at the jira issue will even allow to use the length of the
matching spans as the absolute paragraph number, which, in turn,
allows the use of a Similarity for the paragraph weights [10/5/2].
There is nothing special about indexed term positions; any term can
be indexed at any position in a field. Lucene will take advantage of
the incremental nature of positions by storing only compressed
differences of positions in the index, but during search the original
positions are directly available, You can do the same with payloads,
but why reimplement something that is already available?
Payloads have better uses than positional info, for one they are
great to avoid disjunctions. For example for verbs, one could
index only the stem and use a payload for the actual inflected
form (singular/plural, past/present, first/second/third person, etc).
Regards,
Paul Elschot
>
> Cedric
>
>
> On Fri, Feb 15, 2008 at 7:15 AM, Paul Elschot <pa...@xs4all.nl> wrote:
> > I have no idea what the [10/5/2] means, so I can't comment on that.
> > In case I have missed it previously I'm sorry.
> >
> > My point was that payloads need not be used for different position info.
> > It's possible to do that, and it may be good for performance in some cases,
> > but one can revert to using another field for different position info.
> >
> > Regards,
> > Paul Elschot
> >
> >
> > Op Thursday 14 February 2008 09:44:40 schreef Cedric Ho:
> >
> >
> > > Hi Paul,
> > >
> > > Sorry I am not sure I understand your solution.
> > >
> > > Because I would need to apply this scoring logic to all the different
> > > types of Queries. A search may consists of something like:
> > >
> > > +(term1 phrase2 wildcard*) +spanNear(term3 term4) [10/5/2]
> > >
> > > And this [10/5/2] ratio have to be applied to the whole search query
> > > before it. So I am not sure how would using just SpanFirstQuery with a
> > > separate field work in this situation.
> > >
> > > Anyway, I know my requirement is a bit strange, so it's ok if I can't
> > > do this in Lucene. I'll settle with using a ThreadLocal to store the
> > > [10/5/2] weighting and retrieve it in the Similarity.scorePayload(...)
> > > function.
> > >
> > >
> > > BTW, this problem I am facing now is different from the last one I
> > > asked here, which you have proposed with the Modified SpanFirstQuery
> > > solution =)
> > >
> > > But I am really grateful with all the helps I get here. Keep up the good work!
> > >
> > > Cheers,
> > > Cedric
> > >
> > >
> > > On Thu, Feb 14, 2008 at 2:58 PM, Paul Elschot <pa...@xs4all.nl> wrote:
> > > > Op Thursday 14 February 2008 02:11:24 schreef Cedric Ho:
> > > >
> > > > > I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> > > > > WildcardQuery, BooleanQuery and many of the SpanQueries.
> > > > >
> > > > > The info I am going to pass in is just some weightings for different
> > > > > part of the indexed contents. For example if the payload indicate that
> > > > > a term is in the 2nd paragraph, then I'll take the weighting for the
> > > > > 2nd paragraph and multiply it by the score.
> > > > >
> > > > > So it seems without writing my own query there's no way to do it ?
> > > >
> > > > In case it is only positional information that is stored in the payload
> > > > (i.e. some integer number that does not decrease when tokenizing the
> > > > document), it is also possible to use an extra field and make sure the
> > > > position increment for that field is only positive when the number
> > > > (currently your payload) increases.
> > > > A SpanFirstQuery on this extra field would almost do, and you will
> > > > probably need https://issues.apache.org/jira/browse/LUCENE-1093 .
> > > > This will be somewhat slower than using a payload, because the search
> > > > will be done in two separate fields, but it will work.
> > > >
> > > > Regards,
> > > > Paul Elschot
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Cedric Ho <ce...@gmail.com>.
Sorry that I didn't make myself clear.
[10/5/2] means for terms found in the 1st paragraph, give it score*10,
for terms in the 2nd, give it score*5, etc.
So I don't know how to do this scoring if the position (paragraph)
information is in a separate field.
Cedric
On Fri, Feb 15, 2008 at 7:15 AM, Paul Elschot <pa...@xs4all.nl> wrote:
> I have no idea what the [10/5/2] means, so I can't comment on that.
> In case I have missed it previously I'm sorry.
>
> My point was that payloads need not be used for different position info.
> It's possible to do that, and it may be good for performance in some cases,
> but one can revert to using another field for different position info.
>
> Regards,
> Paul Elschot
>
>
> Op Thursday 14 February 2008 09:44:40 schreef Cedric Ho:
>
>
> > Hi Paul,
> >
> > Sorry I am not sure I understand your solution.
> >
> > Because I would need to apply this scoring logic to all the different
> > types of Queries. A search may consists of something like:
> >
> > +(term1 phrase2 wildcard*) +spanNear(term3 term4) [10/5/2]
> >
> > And this [10/5/2] ratio have to be applied to the whole search query
> > before it. So I am not sure how would using just SpanFirstQuery with a
> > separate field work in this situation.
> >
> > Anyway, I know my requirement is a bit strange, so it's ok if I can't
> > do this in Lucene. I'll settle with using a ThreadLocal to store the
> > [10/5/2] weighting and retrieve it in the Similarity.scorePayload(...)
> > function.
> >
> >
> > BTW, this problem I am facing now is different from the last one I
> > asked here, which you have proposed with the Modified SpanFirstQuery
> > solution =)
> >
> > But I am really grateful with all the helps I get here. Keep up the good work!
> >
> > Cheers,
> > Cedric
> >
> >
> > On Thu, Feb 14, 2008 at 2:58 PM, Paul Elschot <pa...@xs4all.nl> wrote:
> > > Op Thursday 14 February 2008 02:11:24 schreef Cedric Ho:
> > >
> > > > I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> > > > WildcardQuery, BooleanQuery and many of the SpanQueries.
> > > >
> > > > The info I am going to pass in is just some weightings for different
> > > > part of the indexed contents. For example if the payload indicate that
> > > > a term is in the 2nd paragraph, then I'll take the weighting for the
> > > > 2nd paragraph and multiply it by the score.
> > > >
> > > > So it seems without writing my own query there's no way to do it ?
> > >
> > > In case it is only positional information that is stored in the payload
> > > (i.e. some integer number that does not decrease when tokenizing the
> > > document), it is also possible to use an extra field and make sure the
> > > position increment for that field is only positive when the number
> > > (currently your payload) increases.
> > > A SpanFirstQuery on this extra field would almost do, and you will
> > > probably need https://issues.apache.org/jira/browse/LUCENE-1093 .
> > > This will be somewhat slower than using a payload, because the search
> > > will be done in two separate fields, but it will work.
> > >
> > > Regards,
> > > Paul Elschot
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Paul Elschot <pa...@xs4all.nl>.
I have no idea what the [10/5/2] means, so I can't comment on that.
In case I have missed it previously I'm sorry.
My point was that payloads need not be used for different position info.
It's possible to do that, and it may be good for performance in some cases,
but one can revert to using another field for different position info.
Regards,
Paul Elschot
Op Thursday 14 February 2008 09:44:40 schreef Cedric Ho:
> Hi Paul,
>
> Sorry I am not sure I understand your solution.
>
> Because I would need to apply this scoring logic to all the different
> types of Queries. A search may consists of something like:
>
> +(term1 phrase2 wildcard*) +spanNear(term3 term4) [10/5/2]
>
> And this [10/5/2] ratio have to be applied to the whole search query
> before it. So I am not sure how would using just SpanFirstQuery with a
> separate field work in this situation.
>
> Anyway, I know my requirement is a bit strange, so it's ok if I can't
> do this in Lucene. I'll settle with using a ThreadLocal to store the
> [10/5/2] weighting and retrieve it in the Similarity.scorePayload(...)
> function.
>
>
> BTW, this problem I am facing now is different from the last one I
> asked here, which you have proposed with the Modified SpanFirstQuery
> solution =)
>
> But I am really grateful with all the helps I get here. Keep up the good work!
>
> Cheers,
> Cedric
>
>
> On Thu, Feb 14, 2008 at 2:58 PM, Paul Elschot <pa...@xs4all.nl> wrote:
> > Op Thursday 14 February 2008 02:11:24 schreef Cedric Ho:
> >
> > > I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> > > WildcardQuery, BooleanQuery and many of the SpanQueries.
> > >
> > > The info I am going to pass in is just some weightings for different
> > > part of the indexed contents. For example if the payload indicate that
> > > a term is in the 2nd paragraph, then I'll take the weighting for the
> > > 2nd paragraph and multiply it by the score.
> > >
> > > So it seems without writing my own query there's no way to do it ?
> >
> > In case it is only positional information that is stored in the payload
> > (i.e. some integer number that does not decrease when tokenizing the
> > document), it is also possible to use an extra field and make sure the
> > position increment for that field is only positive when the number
> > (currently your payload) increases.
> > A SpanFirstQuery on this extra field would almost do, and you will
> > probably need https://issues.apache.org/jira/browse/LUCENE-1093 .
> > This will be somewhat slower than using a payload, because the search
> > will be done in two separate fields, but it will work.
> >
> > Regards,
> > Paul Elschot
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Grant Ingersoll <gs...@apache.org>.
On Feb 14, 2008, at 3:44 AM, Cedric Ho wrote:
>
> Anyway, I know my requirement is a bit strange, so it's ok if I can't
> do this in Lucene. I'll settle with using a ThreadLocal to store the
> [10/5/2] weighting and retrieve it in the Similarity.scorePayload(...)
> function.
>
I don't think you are still considering this, but I wanted to note
that Similarity.scorePayload() is a call back function, just like the
other methods on Similarity. The only query that currently makes that
callback is the BoostingTermQuery. Essentially, the only queries that
can use payloads are those that use position information when scoring,
i.e. the SpanQuerys, of which the BoostingTQ is one.
The takeaway here, being, you can't retrieve the payload from the
Similarity class. You can only get payloads via the TermPositions
object.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Cedric Ho <ce...@gmail.com>.
Hi Paul,
Sorry I am not sure I understand your solution.
Because I would need to apply this scoring logic to all the different
types of Queries. A search may consists of something like:
+(term1 phrase2 wildcard*) +spanNear(term3 term4) [10/5/2]
And this [10/5/2] ratio have to be applied to the whole search query
before it. So I am not sure how would using just SpanFirstQuery with a
separate field work in this situation.
Anyway, I know my requirement is a bit strange, so it's ok if I can't
do this in Lucene. I'll settle with using a ThreadLocal to store the
[10/5/2] weighting and retrieve it in the Similarity.scorePayload(...)
function.
BTW, this problem I am facing now is different from the last one I
asked here, which you have proposed with the Modified SpanFirstQuery
solution =)
But I am really grateful with all the helps I get here. Keep up the good work!
Cheers,
Cedric
On Thu, Feb 14, 2008 at 2:58 PM, Paul Elschot <pa...@xs4all.nl> wrote:
> Op Thursday 14 February 2008 02:11:24 schreef Cedric Ho:
>
> > I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> > WildcardQuery, BooleanQuery and many of the SpanQueries.
> >
> > The info I am going to pass in is just some weightings for different
> > part of the indexed contents. For example if the payload indicate that
> > a term is in the 2nd paragraph, then I'll take the weighting for the
> > 2nd paragraph and multiply it by the score.
> >
> > So it seems without writing my own query there's no way to do it ?
>
> In case it is only positional information that is stored in the payload
> (i.e. some integer number that does not decrease when tokenizing the
> document), it is also possible to use an extra field and make sure the
> position increment for that field is only positive when the number
> (currently your payload) increases.
> A SpanFirstQuery on this extra field would almost do, and you will
> probably need https://issues.apache.org/jira/browse/LUCENE-1093 .
> This will be somewhat slower than using a payload, because the search
> will be done in two separate fields, but it will work.
>
> Regards,
> Paul Elschot
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Paul Elschot <pa...@xs4all.nl>.
Op Thursday 14 February 2008 02:11:24 schreef Cedric Ho:
> I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
> WildcardQuery, BooleanQuery and many of the SpanQueries.
>
> The info I am going to pass in is just some weightings for different
> part of the indexed contents. For example if the payload indicate that
> a term is in the 2nd paragraph, then I'll take the weighting for the
> 2nd paragraph and multiply it by the score.
>
> So it seems without writing my own query there's no way to do it ?
In case it is only positional information that is stored in the payload
(i.e. some integer number that does not decrease when tokenizing the
document), it is also possible to use an extra field and make sure the
position increment for that field is only positive when the number
(currently your payload) increases.
A SpanFirstQuery on this extra field would almost do, and you will
probably need https://issues.apache.org/jira/browse/LUCENE-1093 .
This will be somewhat slower than using a payload, because the search
will be done in two separate fields, but it will work.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Cedric Ho <ce...@gmail.com>.
I am using Lucene's Built-in query classes: TernQuery, PhraseQuery,
WildcardQuery, BooleanQuery and many of the SpanQueries.
The info I am going to pass in is just some weightings for different
part of the indexed contents. For example if the payload indicate that
a term is in the 2nd paragraph, then I'll take the weighting for the
2nd paragraph and multiply it by the score.
So it seems without writing my own query there's no way to do it ?
Cedric
On Feb 13, 2008 7:53 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Are you writing your own Query? What kind of info did you have in
> mind? scorePayload is called from the query scoring class, so I am
> not sure how you would pass in info to it unless you were writing your
> own Query class.
>
> -Grant
>
>
> On Feb 13, 2008, at 4:31 AM, Cedric Ho wrote:
>
> > Hi all,
> >
> > My problem is I have some additional weighting info that come with
> > each search. And I need to take both the weighting info and the
> > payload to calculate scores.
> >
> > So how do I access the weighting info in
> > Similarity.scorePayload(String,byte[],int,int) ? I've thought about
> > using a ThreadLocal, but this seems like a hack, are there better
> > options ?
> >
> >
> > Regards,
> > Cedric
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to pass additional information into Similarity.scorePayload(...)
Posted by Grant Ingersoll <gs...@apache.org>.
Are you writing your own Query? What kind of info did you have in
mind? scorePayload is called from the query scoring class, so I am
not sure how you would pass in info to it unless you were writing your
own Query class.
-Grant
On Feb 13, 2008, at 4:31 AM, Cedric Ho wrote:
> Hi all,
>
> My problem is I have some additional weighting info that come with
> each search. And I need to take both the weighting info and the
> payload to calculate scores.
>
> So how do I access the weighting info in
> Similarity.scorePayload(String,byte[],int,int) ? I've thought about
> using a ThreadLocal, but this seems like a hack, are there better
> options ?
>
>
> Regards,
> Cedric
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org