You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by HG1212 <fu...@yahoo.com> on 2007/04/19 23:07:16 UTC
Document Boost
Hi there,
I am new to Lucene and would appreciate any help on this. Thank you in
advance.
I want the order of the search results based on the keywords mentioned in
the meta information of the document. For example, if I have two very
similar documents first.htm and second.htm, first.htm has keywords
("security") mentioned in its meta and the second.htm does not mention those
keywords in its meta information, but both documents contain "security" in
its content, then I would want to see the first.htm at the top of the search
results lists.
I tried to boost these keywords (added new fields to the document and
setting the boost for these fields) before adding to the document, but when
I do a search, the results are not as expected.
Any ideas how to accomplish this.
Thanks
HG
--
View this message in context: http://www.nabble.com/Document-Boost-tf3609748.html#a10086883
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document Boost
Posted by Erick Erickson <er...@gmail.com>.
I hate to ask this (actually, I don't hate it, but...) "what behavior
of the scoring are you actually finding doesn't fit your needs?".
The reason I ask is that I've been asked to change the scoring,
that is, set boosts, based on some vague notion of "how
things should work" that is often just the fevered imagination
of someone rather than an actual set of search results that are
unsatisfactory.
Of course, a geek's response is "Sure, I can make it do that."
without the concomitant analysis of whether the users would
ever notice.
Anyway, just a rant on implementing features that do no real
good and delay delivery for no good reason.
You may very well have very good reasons to want to fiddle with
boosts, but I thought I'd mention it <G>....
Erick
On 4/19/07, Les Fletcher <le...@affinitycircles.com> wrote:
>
> Oooooo.... I like the BAR_significant field idea. It seems that you'd
> have to have one of those for every different level of boosting in your
> document. But that is significantly easier than reforming a query for
> 30-odd fields.
>
> The next quersion would be should you omit the boosted field words from
> the default searchg field to get the "appropriate" score for that
> document. In other words, would matching the query in the
> BAR_significant and the default search field return a different score
> than just matching the BAR_significant field for the query
> "BAR_significant:queryword queryword"? They should return the same set
> of results, just curious about how the ranking order might differ. Not
> overly important, more for my own personal curiosity.
>
> Thanks Chris for the quick response to the previous question, it helped
> a lot.
>
> Les
>
>
> Chris Hostetter wrote:
>
> >The full post Erick alluded too may be helpful...
> >
> >
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/200609.mbox/%3CPine.LNX.4.58.0609271134380.32280@hal.rescomp.berkeley.edu%3E
> >
> >in general, if your goal is that words in the "metadata" of a document
> >should be worth more then words in the "body" then you should have two
> >fields: "metadata" and "body", you shoudl query for the same word in both
> >fields, and your query should have a query boost on the "metadata" part.
> >
> >: way it seems to work, is that if you boost a field then you have to
> >: actually specify that field in your query to benefit from that field
> >
> >correct ... you are saying "this documents field_X is better then other
> >document's field_X" .. but that info only comes into play if you actually
> >use "field_X"
> >
> >: ignored. I hacked around this by just adding the field's text to the
> >: default search field n times where n is the boost for that field. I
> >: seriously doubt that this the ideal way to do it, but I couldn't figure
> >: out how to do it other than reforming all of my queries to search all
> >: the fields.
> >
> >that is a feasible solution to the use case where "this doc's use of the
> >word FOO in field BAR is more significant then the use of the word FOO in
> >field BAR for any other docs" but 90% of the time those use cases can be
> >better solved by making a BAR_significant field that only contains the
> >significant words in BAR and querying both.
> >
> >for the other 10% of the time, you'll have to wait for Payload based
> >queries to come along, so when you add FOO to BAR you can give it a
> >payload that says how important it is.
> >
> >(or so the Query Payload people say ... i choose to believe them even
> >though i don't understand them)
> >
> >
> >
> >-Hoss
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Document Boost
Posted by Les Fletcher <le...@affinitycircles.com>.
Oooooo.... I like the BAR_significant field idea. It seems that you'd
have to have one of those for every different level of boosting in your
document. But that is significantly easier than reforming a query for
30-odd fields.
The next quersion would be should you omit the boosted field words from
the default searchg field to get the "appropriate" score for that
document. In other words, would matching the query in the
BAR_significant and the default search field return a different score
than just matching the BAR_significant field for the query
"BAR_significant:queryword queryword"? They should return the same set
of results, just curious about how the ranking order might differ. Not
overly important, more for my own personal curiosity.
Thanks Chris for the quick response to the previous question, it helped
a lot.
Les
Chris Hostetter wrote:
>The full post Erick alluded too may be helpful...
>
>http://mail-archives.apache.org/mod_mbox/lucene-java-user/200609.mbox/%3CPine.LNX.4.58.0609271134380.32280@hal.rescomp.berkeley.edu%3E
>
>in general, if your goal is that words in the "metadata" of a document
>should be worth more then words in the "body" then you should have two
>fields: "metadata" and "body", you shoudl query for the same word in both
>fields, and your query should have a query boost on the "metadata" part.
>
>: way it seems to work, is that if you boost a field then you have to
>: actually specify that field in your query to benefit from that field
>
>correct ... you are saying "this documents field_X is better then other
>document's field_X" .. but that info only comes into play if you actually
>use "field_X"
>
>: ignored. I hacked around this by just adding the field's text to the
>: default search field n times where n is the boost for that field. I
>: seriously doubt that this the ideal way to do it, but I couldn't figure
>: out how to do it other than reforming all of my queries to search all
>: the fields.
>
>that is a feasible solution to the use case where "this doc's use of the
>word FOO in field BAR is more significant then the use of the word FOO in
>field BAR for any other docs" but 90% of the time those use cases can be
>better solved by making a BAR_significant field that only contains the
>significant words in BAR and querying both.
>
>for the other 10% of the time, you'll have to wait for Payload based
>queries to come along, so when you add FOO to BAR you can give it a
>payload that says how important it is.
>
>(or so the Query Payload people say ... i choose to believe them even
>though i don't understand them)
>
>
>
>-Hoss
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document Boost
Posted by Chris Hostetter <ho...@fucit.org>.
The full post Erick alluded too may be helpful...
http://mail-archives.apache.org/mod_mbox/lucene-java-user/200609.mbox/%3CPine.LNX.4.58.0609271134380.32280@hal.rescomp.berkeley.edu%3E
in general, if your goal is that words in the "metadata" of a document
should be worth more then words in the "body" then you should have two
fields: "metadata" and "body", you shoudl query for the same word in both
fields, and your query should have a query boost on the "metadata" part.
: way it seems to work, is that if you boost a field then you have to
: actually specify that field in your query to benefit from that field
correct ... you are saying "this documents field_X is better then other
document's field_X" .. but that info only comes into play if you actually
use "field_X"
: ignored. I hacked around this by just adding the field's text to the
: default search field n times where n is the boost for that field. I
: seriously doubt that this the ideal way to do it, but I couldn't figure
: out how to do it other than reforming all of my queries to search all
: the fields.
that is a feasible solution to the use case where "this doc's use of the
word FOO in field BAR is more significant then the use of the word FOO in
field BAR for any other docs" but 90% of the time those use cases can be
better solved by making a BAR_significant field that only contains the
significant words in BAR and querying both.
for the other 10% of the time, you'll have to wait for Payload based
queries to come along, so when you add FOO to BAR you can give it a
payload that says how important it is.
(or so the Query Payload people say ... i choose to believe them even
though i don't understand them)
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document Boost
Posted by Les Fletcher <le...@affinitycircles.com>.
I am also releatively new to lucene and was wondering about this. The
way it seems to work, is that if you boost a field then you have to
actually specify that field in your query to benefit from that field
boost. Otherwise you'll search the default field and the boost will be
ignored. I hacked around this by just adding the field's text to the
default search field n times where n is the boost for that field. I
seriously doubt that this the ideal way to do it, but I couldn't figure
out how to do it other than reforming all of my queries to search all
the fields.
If someone has a real solution to this or could shed some more light on
the practical use of field boosts, I would appreciate it.
Les
HG1212 wrote:
>I am setting the boost at index time.
>
>Thanks
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document Boost
Posted by HG1212 <fu...@yahoo.com>.
I am setting the boost at index time.
Thanks
--
View this message in context: http://www.nabble.com/Document-Boost-tf3609748.html#a10088201
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Document Boost
Posted by Erick Erickson <er...@gmail.com>.
Were you setting the boosts at index or search time? From an old
e-mail from Chris H.
"index time field boosts are a way to express things like
"this documents title is worth twice as much as the title of
most documents" query time boosts are a way to express
"I care about matches on this clause of my query twice as
much as i do about matches to other clauses of my query".
Erick
On 4/19/07, HG1212 <fu...@yahoo.com> wrote:
>
>
> Hi there,
> I am new to Lucene and would appreciate any help on this. Thank you in
> advance.
> I want the order of the search results based on the keywords mentioned in
> the meta information of the document. For example, if I have two very
> similar documents first.htm and second.htm, first.htm has keywords
> ("security") mentioned in its meta and the second.htm does not mention
> those
> keywords in its meta information, but both documents contain "security" in
> its content, then I would want to see the first.htm at the top of the
> search
> results lists.
> I tried to boost these keywords (added new fields to the document and
> setting the boost for these fields) before adding to the document, but
> when
> I do a search, the results are not as expected.
>
> Any ideas how to accomplish this.
>
> Thanks
> HG
> --
> View this message in context:
> http://www.nabble.com/Document-Boost-tf3609748.html#a10086883
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>