You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nils Höller <nu...@nhoeller.de> on 2006/12/01 11:31:53 UTC

Incremental Index and Comparing different Scores from different Index

Hi,

I have some questions about the scoring function and about how different
scores can be compared.

I use Lucene for indexing an archive of the web.

/archive/day1/differentsites => indexday1
/archive/day2/differentsites => indexday2
/archive/day3/differentsites => indexday3
/archive/day4/differentsites => indexday4
/archive/day5/differentsites => indexday5

No I have a continious query for a certain topic.

Querying indexday1 gives me some Hits with the best having score a
Querying indexday2 gives me some Hits with the best having score b
and so on .....

Now how can I compare those scores, since the regular idf will be
different in the formular of every index?! Am I right here?

I ve read about the incremental idf. Is there a possibility to normalize
the scores so that different indexes/scores can be compared

My Goal is to plot the movement of the score a query has over the time. 
So when I ll look at the Hits on a certain day, I ll must be sure that
all the Hits are out of the day's archive.

I hope you understand my problem and can help me.

Thanks 

Nils


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Incremental Index and Comparing different Scores from different Index

Posted by Soeren Pekrul <so...@gmx.de>.
Hello Nils,

how about having one index for all documents with two fields "date" and 
"content"? You can search documents for a specific date and the score 
uses the global idf of all documents.

Sören

Nils Höller schrieb:
> I thought of making the idf function a NOOP, since this is somehow one
> of the Ways Y. Yang wrote about in "A System For New Event Detection".
> 
> The final idf estimate can be found by using a training set, which i
> can use.
> 
> But this is not the best way, as you said, due to the score quality.
> 
> The second way is: Doing an incremental idf, by updating the index
> everytime a new (day) archive has arrived. Then I use the whole Index
> for getting the idf. But for search results I WANT ONLY results of the
> current day to arrive.
> 
> As I said, this all deals with Continous Queries and Online
> classification.
> 
> Situation: I ve to find the best document in a period of 60 days.
> I m using optimal stopping theory, but this doesnt matter here.
> I want to query for  documents for a special day, but have the idf vor
> all days.
> 
> An Example: 
> 
> I ve got an Index of day 1. The idf can be calculated based on it.
> My Query System doesn't want to mark a document as relevant.
> So now all documents are marked unrelevant and can't be taken as
> relevant ones anymore.
> 
> Now on the second day ive got a new version of my web archive.
> I will "merge" Index of day 1 with the new index/documents to get a
> global idf (called incremental idf).
> For my classification I'm only allowed to classify documents of day2.
> 
> Now how can I query the index (the query is everyday the same) and get
> only documents of day 2, but using the score function (with idf and so
> number of document = all documents day 1 + 2).
> 
> I hope you understand my problem.
> 
> Is it possible to search a lucene index and get only documents of a
> certain day (or the last added documents) but using a global idf the
> global index.
> 
> This would be also the solution of my problem.
> 
> Thanks Nils

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Incremental Index and Comparing different Scores from different Index

Posted by Nils Höller <nu...@nhoeller.de>.
Am Freitag, den 01.12.2006, 11:54 -0800 schrieb Chris Hostetter:
....
> the short answer is you can't, not with the DefaultSimilarity, but you
> might be able to write a custom Similarity that makes the scores
> comparable by making the idf function a NOOP (of course, then your scores
> won't be as "good" -- for some definition of good, but that may not matter
> to you)

I thought of making the idf function a NOOP, since this is somehow one
of the Ways Y. Yang wrote about in "A System For New Event Detection".

The final idf estimate can be found by using a training set, which i
can use.

But this is not the best way, as you said, due to the score quality.

The second way is: Doing an incremental idf, by updating the index
everytime a new (day) archive has arrived. Then I use the whole Index
for getting the idf. But for search results I WANT ONLY results of the
current day to arrive.

As I said, this all deals with Continous Queries and Online
classification.

Situation: I ve to find the best document in a period of 60 days.
I m using optimal stopping theory, but this doesnt matter here.
I want to query for  documents for a special day, but have the idf vor
all days.

An Example: 

I ve got an Index of day 1. The idf can be calculated based on it.
My Query System doesn't want to mark a document as relevant.
So now all documents are marked unrelevant and can't be taken as
relevant ones anymore.

Now on the second day ive got a new version of my web archive.
I will "merge" Index of day 1 with the new index/documents to get a
global idf (called incremental idf).
For my classification I'm only allowed to classify documents of day2.

Now how can I query the index (the query is everyday the same) and get
only documents of day 2, but using the score function (with idf and so
number of document = all documents day 1 + 2).

I hope you understand my problem.

Is it possible to search a lucene index and get only documents of a
certain day (or the last added documents) but using a global idf the
global index.

This would be also the solution of my problem.

Thanks Nils

> 
> this FAQ entry, and in particular the thread from teh second link in it
> have a *lot* of more info on this...
> http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-912c1f237bb00259185353182948e5935f0c2f03
> 
> 
> -Hoss
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Incremental Index and Comparing different Scores from different Index

Posted by Chris Hostetter <ho...@fucit.org>.
: I have some questions about the scoring function and about how different
: scores can be compared.
	...
: Querying indexday1 gives me some Hits with the best having score a
: Querying indexday2 gives me some Hits with the best having score b
: and so on .....
:
: Now how can I compare those scores, since the regular idf will be
: different in the formular of every index?! Am I right here?

the short answer is you can't, not with the DefaultSimilarity, but you
might be able to write a custom Similarity that makes the scores
comparable by making the idf function a NOOP (of course, then your scores
won't be as "good" -- for some definition of good, but that may not matter
to you)

this FAQ entry, and in particular the thread from teh second link in it
have a *lot* of more info on this...
http://wiki.apache.org/jakarta-lucene/LuceneFAQ#head-912c1f237bb00259185353182948e5935f0c2f03


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: an alternative to optimize?

Posted by Yonik Seeley <yo...@apache.org>.
On 12/1/06, Grant Ingersoll <gs...@apache.org> wrote:
> Are the search statistics the same for the MultiReader?  That is,
> would a search on a MultiReader over several small indexes
> necessarily return the same ranking as a single IndexReader on an
> optimized reader?  Would they return the same actual scores?
>
> Just curious, I haven't tried MultiReader, but the code seems like it
> should, at least based on my quick glance at it.

You use a MultiReader whenever you open a multi-segment index.
idf factors can change in the presence of deleted documents.


-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: an alternative to optimize?

Posted by Grant Ingersoll <gs...@apache.org>.
Are the search statistics the same for the MultiReader?  That is,  
would a search on a MultiReader over several small indexes  
necessarily return the same ranking as a single IndexReader on an  
optimized reader?  Would they return the same actual scores?

Just curious, I haven't tried MultiReader, but the code seems like it  
should, at least based on my quick glance at it.

-Grant

On Dec 1, 2006, at 2:59 PM, Chris Hostetter wrote:

>
> : I haven't tried it, but according to http://lucene.apache.org/java/
> : docs/fileformats.html, each segment is a complete sub index.  I
> : _wonder_ if you couldn't manage your own merges by using
> : IndexWriter.addIndexes() where you load each segment in separately
> : (this may mean copying the segments to other directories, but I am
>
> more specificly: opening an IndexReader on a an index with multiple
> segments is functionally the same as opening a MultiReader on several
> indexes with single segments.
>
> so instead of having one index, and carefully controlling  
> maxMergeDocs,
> you could manually create many small indexes, all of which you
> continiously optimize (both to remove deletions, and to keep the  
> index at
> a single segment) which you then explicly open a multi reader on.
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org

Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/ 
LuceneFAQ



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: an alternative to optimize?

Posted by Chris Hostetter <ho...@fucit.org>.
: I haven't tried it, but according to http://lucene.apache.org/java/
: docs/fileformats.html, each segment is a complete sub index.  I
: _wonder_ if you couldn't manage your own merges by using
: IndexWriter.addIndexes() where you load each segment in separately
: (this may mean copying the segments to other directories, but I am

more specificly: opening an IndexReader on a an index with multiple
segments is functionally the same as opening a MultiReader on several
indexes with single segments.

so instead of having one index, and carefully controlling maxMergeDocs,
you could manually create many small indexes, all of which you
continiously optimize (both to remove deletions, and to keep the index at
a single segment) which you then explicly open a multi reader on.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: an alternative to optimize?

Posted by Grant Ingersoll <gs...@apache.org>.
I haven't tried it, but according to http://lucene.apache.org/java/ 
docs/fileformats.html, each segment is a complete sub index.  I  
_wonder_ if you couldn't manage your own merges by using  
IndexWriter.addIndexes() where you load each segment in separately  
(this may mean copying the segments to other directories, but I am  
not sure).  Another option would be to modify Lucene to expose the  
merge functionality.

This is pure speculation at this point, but I know the capabilities  
exist (as all optimize does is merge segments until there is one  
segment) so it seems like it should be possible.

-Grant

On Dec 1, 2006, at 8:11 AM, Stanislav Jordanov wrote:

> Guys,
>
> I've already asked this question but nobody answered:
>
> Suppose we have a relatively big index which is continuously  
> updated - i.e. new docs get added while some of the old docs get  
> deleted.
> For pragmatic reasons we have a restriction on maxMergeDocs so that  
> segment files don't get enormously big.
> Consider now a segment of max size (i.e. containing maxMergeDocs  
> docs hence not eligible for a merge)
> It is possible that (as time passes) this segment will have more  
> and more of its docs deleted.
> But as it is not merge-able it will remain the same size and with  
> lots of "wholes" in it which is bad for performance.
> The only way that I am aware of to correct this problem is to  
> invoke index optimization, which has several drawbacks:
> 1. it takes a while to optimize a big index.
> 2. the optimization process always produces a index comprising of a  
> single (extremely) large segment.
> We can live with 1.
> But 2 is undesirable.
> Is there a way to "optimize" (in terms of purging its deleted docs)  
> an index or a single segment
> without ending up with a single segment index?
>
> Best,
> Stanislav
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org

Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/ 
LuceneFAQ



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


an alternative to optimize?

Posted by Stanislav Jordanov <st...@sirma.bg>.
Guys,

I've already asked this question but nobody answered:

Suppose we have a relatively big index which is continuously updated - 
i.e. new docs get added while some of the old docs get deleted.
For pragmatic reasons we have a restriction on maxMergeDocs so that 
segment files don't get enormously big.
Consider now a segment of max size (i.e. containing maxMergeDocs docs 
hence not eligible for a merge)
It is possible that (as time passes) this segment will have more and 
more of its docs deleted.
But as it is not merge-able it will remain the same size and with lots 
of "wholes" in it which is bad for performance.
The only way that I am aware of to correct this problem is to invoke 
index optimization, which has several drawbacks:
1. it takes a while to optimize a big index.
2. the optimization process always produces a index comprising of a 
single (extremely) large segment.
We can live with 1.
But 2 is undesirable.
Is there a way to "optimize" (in terms of purging its deleted docs) an 
index or a single segment
without ending up with a single segment index?

Best,
Stanislav

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org