You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (Created) (JIRA)" <ji...@apache.org> on 2011/12/11 21:22:40 UTC
[jira] [Created] (LUCENE-3639) Add test case support for shard
searching
Add test case support for shard searching
-----------------------------------------
Key: LUCENE-3639
URL: https://issues.apache.org/jira/browse/LUCENE-3639
Project: Lucene - Java
Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 4.0, 3.5
New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3639) Add test case support for shard
searching
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169662#comment-13169662 ]
Robert Muir commented on LUCENE-3639:
-------------------------------------
starting to look good, i like testing of the distributed stats!
I think, looking at this test, that unfortunately that distributed scoring is still too difficult with lucene.
Its nice how you dont do any extra rewrites or anything like that, but I don't like this:
{noformat}
+ // TODO: nothing evicts from here!!! Somehow, on searcher
+ // expiration on remote nodes we must evict from our
+ // local cache...?
{noformat}
There are two problems here we should separate:
1. the searcher should be able to get the stats, and ensure that they are available for scorers. this is separate from:
2. the searcher doing some caching of stats to prevent network traffic.
Currently your cache handles both 1 and 2, but i think a cache should be a cache.
Maybe we can improve the scoring api, here is where the problem is:
{noformat}
public Weight createNormalizedWeight(Query query) throws IOException {
query = rewrite(query);
// right here is where you want to extractTerms and get stats (from cache or remotely)
Weight weight = query.createWeight(this); // right here, in the weights ctor, is where its going to callback to your IS to ask for those stats
{noformat}
I don't like the fact that you need to handle this crazy state here... there has to be somethign we can do to simplify this.
> Add test case support for shard searching
> -----------------------------------------
>
> Key: LUCENE-3639
> URL: https://issues.apache.org/jira/browse/LUCENE-3639
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3639.patch, LUCENE-3639.patch
>
>
> New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3639) Add test case support for shard
searching
Posted by "Michael McCandless (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3639.
----------------------------------------
Resolution: Fixed
> Add test case support for shard searching
> -----------------------------------------
>
> Key: LUCENE-3639
> URL: https://issues.apache.org/jira/browse/LUCENE-3639
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 4.0, 3.5
>
> Attachments: LUCENE-3639.patch, LUCENE-3639.patch
>
>
> New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3639) Add test case support for shard
searching
Posted by "Michael McCandless (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3639:
---------------------------------------
Attachment: LUCENE-3639.patch
New patch, beefing up the test some more. I think it's ready.
I improved SLM's age calculation to use double precision
and to compute age by how long ago the searcher was replaced with a
new searcher (not how long ago the searcher was first enrolled).
I also fixed a bug TopDocs.merge when you use searchAfter with shards:
it was incorrectly assuming that topDocs.scoreDocs.length == 0 meant
topDocs.totalHits == 0, which is not necessarily true if you use
searchAfter.
> Add test case support for shard searching
> -----------------------------------------
>
> Key: LUCENE-3639
> URL: https://issues.apache.org/jira/browse/LUCENE-3639
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3639.patch, LUCENE-3639.patch
>
>
> New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3639) Add test case support for shard
searching
Posted by "Michael McCandless (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3639:
---------------------------------------
Attachment: LUCENE-3639.patch
Initial patch; I think it's close.
It only tests random TermQuery, verifying the hits match across a IS(MR) and searching as shards... we can improve over time.
I also found a minor bug in TopDocs.merge, where it sets maxScore to Float.MIN_VALUE instead of Float.NaN when there are 0 hits.
> Add test case support for shard searching
> -----------------------------------------
>
> Key: LUCENE-3639
> URL: https://issues.apache.org/jira/browse/LUCENE-3639
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3639.patch
>
>
> New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3639) Add test case support for shard
searching
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169664#comment-13169664 ]
Robert Muir commented on LUCENE-3639:
-------------------------------------
By the way, the test is great... i actually think we should just commit it as-is.
Otherwise, how can we improve this stuff? we need this test!
> Add test case support for shard searching
> -----------------------------------------
>
> Key: LUCENE-3639
> URL: https://issues.apache.org/jira/browse/LUCENE-3639
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3639.patch, LUCENE-3639.patch
>
>
> New test case that helps stress test the APIs to support sharding....
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org