"diversity" of search results?

[Paul Libbrecht <pa...@hoplahup.net>]

Hello SOLR expert,

yesterday in our group we realized that a danger we may need to face is that a search result includes very similar results.
Of course, one would expect skimming so that duplicates that show almost the same results in a search result would be avoided but we fear that this is not possible.

I was wondering if some technology, plugin, or even research was existing that would enable a search result to be partially reordered so that "diversity" is ensured for a first page of results at least.

I suppose that might be doable by processing the result page and the next (and the five next?) and pushing down some results if they are "too" similar to previous ones.

Hope I am being clear.

Paul

Re: "diversity" of search results?

[dirk <ku...@googlemail.com>]

Hi Paul,

yes that`s a typical problem in configuring a search engine. A solution
depends on your data. Sometimes you can overcome this problem by fine tuning
your search engine on boosting level. Thats not easy and always based on
trail and error tests.

Another thing you can do is to try to realize a data pre-processing which
compensate the reasons of similar content in certain fields, e.g. in a title
field. 
For example if you have products with very similar titles and you boost such
a field. The result is, that you always will found all documents in the
result list. But if you go on and add some informations (perhaps out of
other search fields) in this title field you perhaps can reduce the
similarity. (typical example in my branch: Book titles in different volumes,
then I add the volumn  number and der year to the title field.) 

Perhaps it is also necessary to cape with a pre-processed deduplication.
Here you can find an entry point:
http://wiki.apache.org/solr/Deduplication

Dirk

   



-----
my developer logs 
--
View this message in context: http://lucene.472066.n3.nabble.com/diversity-of-search-results-tp4014692p4014696.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: "diversity" of search results?

[Otis Gospodnetic <ot...@gmail.com>]

Hi Paul,

We've done this for a client in the past via a custom SearchComponent
and it worked well.  Yes, it involved some post-processing, but on the
server, not client.  I *think* we saw 10% performance degradation.

Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html


On Fri, Oct 19, 2012 at 3:26 AM, Paul Libbrecht <pa...@hoplahup.net> wrote:
> Hello SOLR expert,
>
> yesterday in our group we realized that a danger we may need to face is that a search result includes very similar results.
> Of course, one would expect skimming so that duplicates that show almost the same results in a search result would be avoided but we fear that this is not possible.
>
> I was wondering if some technology, plugin, or even research was existing that would enable a search result to be partially reordered so that "diversity" is ensured for a first page of results at least.
>
> I suppose that might be doable by processing the result page and the next (and the five next?) and pushing down some results if they are "too" similar to previous ones.
>
> Hope I am being clear.
>
> Paul