You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 01:00:59 UTC
[jira] [Created] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
make similarities/term/collectionstats take long (for > 2B docs)
----------------------------------------------------------------
Key: LUCENE-3722
URL: https://issues.apache.org/jira/browse/LUCENE-3722
Project: Lucene - Java
Issue Type: Improvement
Affects Versions: 4.0
Reporter: Robert Muir
Attachments: LUCENE-3722.patch
As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192792#comment-13192792 ]
Robert Muir commented on LUCENE-3722:
-------------------------------------
{quote}
Ok, it makes sense in local (multi reader) situations but in distributed scenario it may be still acceptable to lose just a part of the statistics from one shard while keeping the stats from other shards.
{quote}
I really think we should not do this: it can result in NaN/Inf/negative scores to have 'invalid'
statistics (this causes serious problems!), but we have (and test) that all the sims fallback
gracefully for the -1 case.
-1 means 'preflex codec does not support the stat'. Once we no longer have to worry about 3.x
indexes, we no longer need to worry about -1.
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Andrzej Bialecki (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192790#comment-13192790 ]
Andrzej Bialecki commented on LUCENE-3722:
-------------------------------------------
bq. maybe we should not have add() here then at all and let the consumer take care of this themselves.
Fair point, I'd rather remove it.
bq. i think it must not be treated as 0
Ok, it makes sense in local (multi reader) situations but in distributed scenario it may be still acceptable to lose just a part of the statistics from one shard while keeping the stats from other shards.
Which would be yet another reason to drop the add() methods :)
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: (was: LUCENE-3722.patch)
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: LUCENE-3722.patch
updated patch with improvements for
the preflex case (when they return -1).
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Andrzej Bialecki (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192774#comment-13192774 ]
Andrzej Bialecki commented on LUCENE-3722:
-------------------------------------------
I think the add() methods as implemented in this patch are of limited usefulness... The reason I created a pair of similar classes in SOLR-1632 was to avoid creating new objects by allowing modification of int/long members in place, which is useful when aggregating partial stats. So I think a more useful implementation of add() could look like this:
{code}
public void add(CollectionStatistics other) {
assert this.field.equals(other.field);
this.maxDoc += other.maxDoc;
this.docCount += other.docCount;
this.sumTotalTermFreq += other.sumTotalTermFreq;
this.sumDocFreq += other.sumDocFreq;
}
{code}
Regarding the handling of -1: this code doesn't handle the case when only one stats is -1, which may happen in distributed scenario. I think in such case a -1 value should be treated as 0, i.e. it should not "reset" the accumulated value from other shards to -1, right?
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192779#comment-13192779 ]
Robert Muir commented on LUCENE-3722:
-------------------------------------
{quote}
So I think a more useful implementation of add() could look like this:
{quote}
But I don't think we should just immediately make the stats classes mutable without thinking things through more, I'm not sure there is really any problem creating new objects (we should benchmark this, before making things mutable).
maybe we should not have add() here then at all and let the consumer take care of this themselves.
{quote}
should be treated as 0, i.e. it should not "reset" the accumulated value from other shards to -1, right?
{quote}
i think it must not be treated as 0, the same way its handled by default in TermContext if some segments are 3.x: it means this statistic is simply not supported.
In other words, if any segment across any shard returns -1, the cumulative stat should always be -1: it means its not supported by the codec.
The similarities that use these new statistics already look for -1 and have fallback mechanisms for this case.
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir resolved LUCENE-3722.
---------------------------------
Resolution: Fixed
Fix Version/s: 4.0
Committed the switch of int->long.
For now the summation stuff is still up to the caller... seems unfriendly, but maybe it should stay that way, but at least we have asserts that the stats aren't out of bounds.
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Fix For: 4.0
>
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: LUCENE-3722.patch
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: LUCENE-3722.patch
updated patch: this is just int->long conversion... at least we should be able to agree on this :)
we should still figure out how to make this summation easier and safer (especially for the "preflex-returns-1" case)
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192796#comment-13192796 ]
Robert Muir commented on LUCENE-3722:
-------------------------------------
just a simple example for what i meant about the -1 case:
Lets assume you have two shards, and one returns -1 for totalTermFreq().
If you were using BasicModelIF, which scores as:
{code}
tf_norm * log(1 + (maxdoc + 1)/(totalTermFreq + 0.5))
{code}
its far better to actually use -1 than a 'partial/incorrect' totalTermFreq,
because in that case the formula will fall back to totalTermFreq=docFreq...
it also must do this in case frequencies are omitted (omitTF), and for that
case the formula is still correct: but either way its falling back nicely to
IDF:
{code}
tf_norm * log(1 + (maxdoc + 1)/(docFreq + 0.5))
{code}
Yeah, i totally forgot about this being -1 in the omitTF case, so we should
still really think this summation through and make it easy to prevent mistakes,
because i gather omitTF isn't going anywhere... grrr
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: LUCENE-3722.patch
Patch synced up to trunk.
as far as the unrelated -1/add()/summation stuff i added assertions to do basic bounds checking of these stats (found a bug in simpletext), so I think thats a good step for now about concerns of summation: at least we will probably catch any bad bugs.
But this patch just changes int->long and I think we can move forward with it.
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3722) make
similarities/term/collectionstats take long (for > 2B docs)
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3722:
--------------------------------
Attachment: LUCENE-3722.patch
> make similarities/term/collectionstats take long (for > 2B docs)
> ----------------------------------------------------------------
>
> Key: LUCENE-3722
> URL: https://issues.apache.org/jira/browse/LUCENE-3722
> Project: Lucene - Java
> Issue Type: Improvement
> Affects Versions: 4.0
> Reporter: Robert Muir
> Attachments: LUCENE-3722.patch, LUCENE-3722.patch
>
>
> As noted by Yonik and Andrzej on SOLR-1632, this would be useful for distributed scoring.
> we can also add a sugar method add() to both of these to make it easier to sum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org