You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Doug Cutting <cu...@lucene.com> on 2003/12/01 19:10:42 UTC
Re: Dates and others
Dion Almaer wrote:
> The only real item that I still want to tweak more is getting recent results higher in the list.
>
> I was wondering if something like this could work (or if there is a better solution)
>
> At index time, I have the date of the content. I could do some math where the higher the date
> (based on the time_t version or whatever) the more of a setBoost(metric). Or, for every month in the
> past, create a larger negative number to setBoost()... or something like that.
>
> Would something like this make sense?
The problem with this approach is that eventually you'll exhaust the
range of the boost. So this will only work if you re-index things from
scratch periodically, with a boost of something like 1/days-ago.
If you're adding documents to the index in date order, then you could
use a HitCollector which adjusts scores according to the document
number, since document numbers increase as you add to the index.
If you're not adding things in date order, then you can, when you open
the index, build an array mapping document numbers to integer dates.
Then your hit collector can use this to either boost or sort hits by date.
Or you could add a "month" or "week" field to documents, then add it as
a clause to your queries with a boost. Then documents matching the most
recent week(s) and/or month(s) would get the boost.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Dates and others
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Monday, December 1, 2003, at 11:55 PM, Tatu Saloranta wrote:
> On a related note, it would also be nice if there was a way to start
> categorizing general "hot topics" for Lucene developers; it seems like
> there
> are about half a dozen areas where there's lots of interest for
> improvements
> (most of them related to ranking). If so, perhaps there could be more
> specific discussion groups, and also perhaps web pages summarizing
> some of
> discussions, consensus achieved, even if there's no code to show for
> it?
I agree.
Sounds like the perfect solution is a wiki! Just happens we have one:
http://nagoya.apache.org/wiki/apachewiki.cgi?LuceneProjectPages
Have at it!
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Dates and others
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Monday 01 December 2003 15:13, Dion Almaer wrote:
...
> Interesting. I implemented an approach which boosted based on the number
> of months in the past, and after tweaking the boost amounts, it seems to do
> the job. I do a fresh reindex every night (since the indexing process takes
> no time at all... unlike our old search solution!)
This sounds interesting, as I have been thinking of what's the best way
to boost newer documents. Can you share some of your experience regarding
boost values that seemed to make sense? In my case, CMS I'm working on stores
support documentation for software/hardware, meaning that content is highly
time-sensitive (ie. documents "decay" pretty quickly).
Since the system is already doing both incremental reindexing, and nightly
full reindexing (latter to make sure that even if temporarily some changed
content was not [fully] reindexed, it eventually gets indexed properly), I
can fairly easily add boosting I think.
On a related note, it would also be nice if there was a way to start
categorizing general "hot topics" for Lucene developers; it seems like there
are about half a dozen areas where there's lots of interest for improvements
(most of them related to ranking). If so, perhaps there could be more
specific discussion groups, and also perhaps web pages summarizing some of
discussions, consensus achieved, even if there's no code to show for it?
-+ Tatu +-
>
> I read content for the index from different sources. Sometimes the source
> gives me documents loosely in date order, but not all of them. So, it seems
> that one of the other approaches should be taken (adding a month/week field
> etc). I should look more into the HitCollector and see how it can help me.
>
> The other issue I have is that I would like to prioritize the title field.
> At the moment I am lazy and add the title to the body (contents = title +
> body) which seems to be OK... however sometimes something that mentions the
> search term in the title should appear higher up in the pecking order.
>
> I am using the QueryParser (subclassed to disallow wildcards etc) to do the
> dirty work for me. Should I get away from this and manage the queries
> myself (and run a Multi against the title field as well as the contents?
>
> Thanks for the great feedback,
>
> Dion
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Dates and others
Posted by Doug Cutting <cu...@lucene.com>.
Dion Almaer wrote:
> Interesting. I implemented an approach which boosted based on the number of months in the past, and
> after tweaking the boost amounts, it seems to do the job. I do a fresh reindex every night (since
> the indexing process takes no time at all... unlike our old search solution!)
If you're reindexing every night, then document boosting should work
well. The other approaches I mentioned would only be required if you
can't afford to re-index frequently.
> I read content for the index from different sources. Sometimes the source gives me documents loosely
> in date order, but not all of them. So, it seems that one of the other approaches should be taken
> (adding a month/week field etc). I should look more into the HitCollector and see how it can help
> me.
I wouldn't bother to pursue these approaches. Document boosting should
work well for you, since you're reindexing.
> The other issue I have is that I would like to prioritize the title field. At the moment I am lazy
> and add the title to the body (contents = title + body) which seems to be OK... however sometimes
> something that mentions the search term in the title should appear higher up in the pecking order.
>
> I am using the QueryParser (subclassed to disallow wildcards etc) to do the dirty work for me.
> Should I get away from this and manage the queries myself (and run a Multi against the title field
> as well as the contents?
A separate title field will solve this for you. You can, as you
suggest, boost title clauses at query time. Alternately, you could
boost title fields at index time, although that's less flexible.
Note that if you put titles in a separate field and search both the
title and the body field then title matches will tend to be naturally
boosted, since titles tend to be shorter than bodies and hence are less
normalized.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Dates and others
Posted by Dion Almaer <di...@almaer.com>.
> -----Original Message-----
> From: Doug Cutting [mailto:cutting@lucene.com]
> Sent: Monday, December 01, 2003 1:11 PM
> To: Lucene Users List
> Subject: Re: Dates and others
>
> Dion Almaer wrote:
> > The only real item that I still want to tweak more is
> getting recent results higher in the list.
> >
> > I was wondering if something like this could work (or if there is a
> > better solution)
> >
> > At index time, I have the date of the content. I could do
> some math
> > where the higher the date (based on the time_t version or whatever)
> > the more of a setBoost(metric). Or, for every month in the
> past, create a larger negative number to setBoost()... or
> something like that.
> >
> > Would something like this make sense?
>
> The problem with this approach is that eventually you'll
> exhaust the range of the boost. So this will only work if
> you re-index things from scratch periodically, with a boost
> of something like 1/days-ago.
>
> If you're adding documents to the index in date order, then
> you could use a HitCollector which adjusts scores according
> to the document number, since document numbers increase as
> you add to the index.
>
> If you're not adding things in date order, then you can, when
> you open the index, build an array mapping document numbers
> to integer dates.
> Then your hit collector can use this to either boost or sort
> hits by date.
>
> Or you could add a "month" or "week" field to documents, then
> add it as a clause to your queries with a boost. Then
> documents matching the most recent week(s) and/or month(s)
> would get the boost.
>
> Doug
Interesting. I implemented an approach which boosted based on the number of months in the past, and
after tweaking the boost amounts, it seems to do the job. I do a fresh reindex every night (since
the indexing process takes no time at all... unlike our old search solution!)
I read content for the index from different sources. Sometimes the source gives me documents loosely
in date order, but not all of them. So, it seems that one of the other approaches should be taken
(adding a month/week field etc). I should look more into the HitCollector and see how it can help
me.
The other issue I have is that I would like to prioritize the title field. At the moment I am lazy
and add the title to the body (contents = title + body) which seems to be OK... however sometimes
something that mentions the search term in the title should appear higher up in the pecking order.
I am using the QueryParser (subclassed to disallow wildcards etc) to do the dirty work for me.
Should I get away from this and manage the queries myself (and run a Multi against the title field
as well as the contents?
Thanks for the great feedback,
Dion
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org