You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Scott Davies <sc...@gmail.com> on 2006/06/02 22:14:41 UTC
Per-token weighting / attribute data in index
Hi...reasonably experienced web search programmer but total Lucene newbie here.
After poking through Lucene for a while, I still haven't figured out a
decent way to tweak the scoring based on per-token data. For example,
as far as I can tell so far, the only reasonable way to have words in
the titles or headers of HTML documents be "worth more" for scoring
purposes than ordinary body text is to make "title" and "header"
fields and apply appropriate field boosts across all documents. That
works OK if you only have a few special fields you want to boost by
some consistent amount each, but falls down if, say, you wanted to
include some sort of "tags" or anchortext in the scoring of documents
where there's a high degree of variability in how much any given tag
or anchor should be "trusted" and thus influence the score. (I could
conceivably discretize the boosts and, say, put all the anchortext
with boost 2.5 in a special "anchortext-boost2.5" field, but that
would be extremely awkward and presumably cause major performance
issues as the number of fields increases.)
Have I just failed to notice the right way to do this, or is there
really no decent way to do it in Lucene at this time? If the latter,
are there any plans to add this feature at some point semi-soon? This
seems to me like a major scoring limitation for applications not just
indexing and searching over plain text documents...
Thanks,
-- Scott
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Andrzej Bialecki <ab...@getopt.org>.
Marvin Humphrey wrote:
> Enabling individual positions to be boosted is indeed one of the main targets
> of the current discussion. A slightly easier to understand application would
> be boosting individual tokens according to relative font size. For instance,
> we might assume that text between <h1> tags is more important than text
> between <p> tags and boost it. There's no good way to handle that right now.
>
.. except for adding separate fields with <h1>, <h2> ... content.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Fri, Jun 02, 2006 at 03:47:10PM -0700, Chris Hostetter wrote:
> You may want to check out the java-dev list ... there's been some talk
> among the people who really unerstand the low levels of lucene's file
> formats about adding arbitrary "payload" data with each term/doc pair .. a
> proposal that started (as far as i can tell) from a desire to have
> individual term/doc boosting...
It's funny that you don't seem to include yourself in that group yet, Hoss. I
imagine it won't be long. If you haven't read that Brin/Page paper from 1998
yet, you should check it out.
Enabling individual positions to be boosted is indeed one of the main targets
of the current discussion. A slightly easier to understand application would
be boosting individual tokens according to relative font size. For instance,
we might assume that text between <h1> tags is more important than text
between <p> tags and boost it. There's no good way to handle that right now.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Scott Davies <sc...@gmail.com>.
Dang, that's what I was afraid of. Good to hear they're actively
considering extensions that'd fix the issue, though. In the meantime
I guess I'll try limping along without 'em. Thanks!
-- Scott
On 6/2/06, Chris Hostetter <ho...@fucit.org> wrote:
> :
> : A simple example would be indexing and scoring the hyperlink text from
> : other web pages that point to the page P that I'm indexing/scoring. I
> : might have some metric saying how much I "trust" each of the pages or
> : sites with hyperlinks to P, and want to use that metric to increase or
>
> Hmmm... yes, other then having a "trustedAnchorText" field and an
> "untrustedAnchorText" field i don't know any way way to achieve your goal
> at the moment.
>
> You may want to check out the java-dev list ... there's been some talk
> among the people who really unerstand the low levels of lucene's file
> formats about adding arbitrary "payload" data with each term/doc pair .. a
> proposal that started (as far as i can tell) from a desire to have
> individual term/doc boosting...
>
> http://wiki.apache.org/jakarta-lucene/FlexibleIndexing
> http://www.nabble.com/Lucene-Planning-t1694958.html#a4646533
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Chris Hostetter <ho...@fucit.org>.
:
: A simple example would be indexing and scoring the hyperlink text from
: other web pages that point to the page P that I'm indexing/scoring. I
: might have some metric saying how much I "trust" each of the pages or
: sites with hyperlinks to P, and want to use that metric to increase or
Hmmm... yes, other then having a "trustedAnchorText" field and an
"untrustedAnchorText" field i don't know any way way to achieve your goal
at the moment.
You may want to check out the java-dev list ... there's been some talk
among the people who really unerstand the low levels of lucene's file
formats about adding arbitrary "payload" data with each term/doc pair .. a
proposal that started (as far as i can tell) from a desire to have
individual term/doc boosting...
http://wiki.apache.org/jakarta-lucene/FlexibleIndexing
http://www.nabble.com/Lucene-Planning-t1694958.html#a4646533
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Scott Davies <sc...@gmail.com>.
A simple example would be indexing and scoring the hyperlink text from
other web pages that point to the page P that I'm indexing/scoring. I
might have some metric saying how much I "trust" each of the pages or
sites with hyperlinks to P, and want to use that metric to increase or
decrease how much the text in those hyperlinks increases the score of
P for queries containing that anchortext. Since each incoming
hyperlink is from a different site with a different trustworthiness,
I'd obviously want to be able to vary that boost independently for
every different hyperlink pointing at page P.
On 6/2/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> i may be missunderstanding your goal .. it sounds like what you want to do
> is say thta for certain documents (which you trust) matching on the title
> is "worth more" then matching on the title of other documents (which you
> don't trust)
>
> if that' the case, then at index time you can add field boost on the
> title just for hte documents you trust, and add no boost for hte documents
> you don't trust.
>
> I've i've missunderstood your question, could you provide a use case
> describing your goal, and where lucene fails to meet it?
>
>
>
> : Date: Fri, 2 Jun 2006 13:14:41 -0700
> : From: Scott Davies <sc...@gmail.com>
> : Reply-To: java-user@lucene.apache.org
> : To: java-user@lucene.apache.org
> : Subject: Per-token weighting / attribute data in index
> :
> : Hi...reasonably experienced web search programmer but total Lucene newbie here.
> :
> : After poking through Lucene for a while, I still haven't figured out a
> : decent way to tweak the scoring based on per-token data. For example,
> : as far as I can tell so far, the only reasonable way to have words in
> : the titles or headers of HTML documents be "worth more" for scoring
> : purposes than ordinary body text is to make "title" and "header"
> : fields and apply appropriate field boosts across all documents. That
> : works OK if you only have a few special fields you want to boost by
> : some consistent amount each, but falls down if, say, you wanted to
> : include some sort of "tags" or anchortext in the scoring of documents
> : where there's a high degree of variability in how much any given tag
> : or anchor should be "trusted" and thus influence the score. (I could
> : conceivably discretize the boosts and, say, put all the anchortext
> : with boost 2.5 in a special "anchortext-boost2.5" field, but that
> : would be extremely awkward and presumably cause major performance
> : issues as the number of fields increases.)
> :
> : Have I just failed to notice the right way to do this, or is there
> : really no decent way to do it in Lucene at this time? If the latter,
> : are there any plans to add this feature at some point semi-soon? This
> : seems to me like a major scoring limitation for applications not just
> : indexing and searching over plain text documents...
> :
> : Thanks,
> :
> : -- Scott
> :
> : ---------------------------------------------------------------------
> : To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> : For additional commands, e-mail: java-user-help@lucene.apache.org
> :
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Per-token weighting / attribute data in index
Posted by Chris Hostetter <ho...@fucit.org>.
i may be missunderstanding your goal .. it sounds like what you want to do
is say thta for certain documents (which you trust) matching on the title
is "worth more" then matching on the title of other documents (which you
don't trust)
if that' the case, then at index time you can add field boost on the
title just for hte documents you trust, and add no boost for hte documents
you don't trust.
I've i've missunderstood your question, could you provide a use case
describing your goal, and where lucene fails to meet it?
: Date: Fri, 2 Jun 2006 13:14:41 -0700
: From: Scott Davies <sc...@gmail.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Per-token weighting / attribute data in index
:
: Hi...reasonably experienced web search programmer but total Lucene newbie here.
:
: After poking through Lucene for a while, I still haven't figured out a
: decent way to tweak the scoring based on per-token data. For example,
: as far as I can tell so far, the only reasonable way to have words in
: the titles or headers of HTML documents be "worth more" for scoring
: purposes than ordinary body text is to make "title" and "header"
: fields and apply appropriate field boosts across all documents. That
: works OK if you only have a few special fields you want to boost by
: some consistent amount each, but falls down if, say, you wanted to
: include some sort of "tags" or anchortext in the scoring of documents
: where there's a high degree of variability in how much any given tag
: or anchor should be "trusted" and thus influence the score. (I could
: conceivably discretize the boosts and, say, put all the anchortext
: with boost 2.5 in a special "anchortext-boost2.5" field, but that
: would be extremely awkward and presumably cause major performance
: issues as the number of fields increases.)
:
: Have I just failed to notice the right way to do this, or is there
: really no decent way to do it in Lucene at this time? If the latter,
: are there any plans to add this feature at some point semi-soon? This
: seems to me like a major scoring limitation for applications not just
: indexing and searching over plain text documents...
:
: Thanks,
:
: -- Scott
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org