You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2009/08/07 08:31:14 UTC
[jira] Created: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
-------------------------------------------------------------------------------------------
Key: LUCENE-1791
URL: https://issues.apache.org/jira/browse/LUCENE-1791
Project: Lucene - Java
Issue Type: Test
Reporter: Hoss Man
methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-1791) Enhance QueryUtils and
CheckHIts to wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742586#action_12742586 ]
Mark Miller edited comment on LUCENE-1791 at 8/12/09 2:35 PM:
--------------------------------------------------------------
I'm guess the NAN failures are not a problem - looks like they fail because NAN != NAN ? Havn't looked closer.
I don't think the fieldcache insanity is multi-reader related - it looks to me like some entries have a parser, and some null for the parser, even though the default parser is being used in both cases. The FieldSource types grab a FieldCache and may pass null as the parser, which ends up putting null in the cache entry - but if you specifically ask for the default parser, that puts the default parser in the fieldcache entry - same stuff now, doubled entry.
As for the out of bounds - havn't look at that one yet - odd one ...
... interesting - it alternates between nullpointer and out of bounds exceptions ...
was (Author: markrmiller@gmail.com):
I'm guess the NAN failures are not a problem - looks like they fail because NAN != NAN ? Havn't looked closer.
I don't think the fieldcache insanity is multi-reader related - it looks to me like some entries have a parser, and some null for the parser, even though the default parser is being used in both cases. The FieldSource types grab a FieldCache and may pass null as the parser, which ends up putting null in the cache entry - but if you specifically ask for the default parser, that puts the default parser in the fieldcache entry - same stuff now, doubled entry.
As for the out of bounds - havn't look at that one yet - odd one ...
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742868#action_12742868 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
I guess the out of bounds exception is still there:
[junit] Testsuite: org.apache.lucene.search.TestBoolean2
[junit] Tests run: 11, Failures: 0, Errors: 1, Time elapsed: 0.723 sec
[junit] ------------- Standard Output ---------------
[junit] failed query: field:yy field:zzz
[junit] NOTE: random seed of testcase 'testRandomQueries' was: -2696660952902803926
[junit] ------------- ---------------- ---------------
[junit] Testcase: testRandomQueries(org.apache.lucene.search.TestBoolean2): Caused an ERROR
[junit] -1
[junit] java.lang.ArrayIndexOutOfBoundsException: -1
No sure what to do about the ItemizedFilter - its not really a legal way to check things ...
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742813#action_12742813 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
Have you started working on the ItemizedFilter at all? It can't use outside doc ids like that. Saying to match doc 3 will match every doc 3 in every reader.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-1791:
--------------------------------
Attachment: LUCENE-1791.patch
One more minor entry from me:
minor javadoc/spelling fixes
removed unused imports
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Simon Willnauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742758#action_12742758 ]
Simon Willnauer commented on LUCENE-1791:
-----------------------------------------
I just looked at the patch - one little minor thing. Would it make sense to randomized reader / searcher indexes in wrapUnderlyingReader / wrapSearchers to catch possible array index related issues? This could also be done for the num of deleted terms and types of searcher / readers.
simon
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744670#action_12744670 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
You going to resolve this Hoss? Or was that commit just a tease?
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742600#action_12742600 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
bq. I'm guess the NAN failures are not a problem - looks like they fail because NAN != NAN?
right -- but why would the scores be NaN when wrapped in a MultiReader? when it's *not* wrapped in a MultiReader the test passes, so the scores must not be NaN in that case.
bq. I don't think the fieldcache insanity is multi-reader related [...] same stuff now, doubled entry.
The sanity checker ignores when two CacheEntries differ only by parser (precisely because of the the null/default parser issue) and the resulting value object is the same. but it does include all related CacheEntry objects in an Insanity object so that you have them all for debugging.
Looking at TestCustomScoreQuery.testCustomScoreByte (for example)...
{code}
*** BEGIN org.apache.lucene.search.function.TestCustomScoreQuery.testCustomScoreByte: Insane FieldCache usage(s) ***
SUBREADER: Found caches for decendents of org.apache.lucene.index.DirectoryReader@88d2ae+iii
'org.apache.lucene.index.DirectoryReader@88d2ae'=>'iii',byte,null=>[B#841343 (size =~ 33 bytes)
'org.apache.lucene.index.DirectoryReader@88d2ae'=>'iii',byte,org.apache.lucene.search.FieldCache.DEFAULT_BYTE_PARSER=>[B#841343 (size =~ 33 bytes)
'org.apache.lucene.index.CompoundFileReader$CSIndexInput@77daaa'=>'iii',byte,org.apache.lucene.search.FieldCache.DEFAULT_BYTE_PARSER=>[B#981898 (size =~ 33 bytes)
'org.apache.lucene.index.CompoundFileReader$CSIndexInput@77daaa'=>'iii',byte,null=>[B#981898 (size =~ 33 bytes)
*** END org.apache.lucene.search.function.TestCustomScoreQuery.testCustomScoreByte: Insane FieldCache usage(s) ***
{code}
The insanity type is "SUBREADER", so it's specificly identified a problem with that type of relationship. There are 4 CacheEntries listed in the error all from the same field, but from two different readers. If you note the value identity hashcodes (just before the size estimate) each reader has only one value cached for that field (with different parsers) which is why there isn't a seperate error about the multiple values.
as the first line of hte Instanity.toString() states: what it found is that DirectoryReader@88d2ae and at least one of it's decendents both have cached entires for the same field.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-1791:
--------------------------------
Attachment: LUCENE-1791.patch
every so slight cleaned up
removed a couple unused imports
added a comment about whats going on with the ItemizedFilter
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742606#action_12742606 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
midair collision (x2) ... i think i see what you mean in your revised patch ... the tests don't need changed, it's just the test utility methods that were trying to recurse the readers that weren't doing the entire job.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742900#action_12742900 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
Okay, I've got all tests passing.
I rigged up a smarted ItemizedFilter that works for these tests.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743061#action_12743061 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
{quote}sweet. i'll review the patch for real tomorow or sat, and maybe rip out the ItemizedFilter and replace it with something more sensical (it's a bad example that people might stumble upon, that seems even more confusing now, and i feel responsible since i'm the one that wrote it for those explanation tests anyway)
{quote}
Cool - it is very confusing now, and not super clean really - its kind of a hack - I just wasn't sure how else to handle it.
Essentially, every time it sees a reader, it assumes its a reader thats part of a sequence of sub readers and adds maxDoc to a base to figure out what doc ids it should really set - and then if it sees the top level reader, or the the first sub reader, it resets that base - so it can start over if the set of sub readers need to come through again. It works for these tests, but it makes no sense to do such a thing in the real world.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-1791:
-----------------------------
Attachment: LUCENE-1791.patch
one other thing i experimented with earlier today was making the "wrapped" MultiReaders and MultiSearchers a little more complicated (deeper structures, and adding some deleted docs to the other subreaders)
at first i ran into problems because the deleted docs would through off the docIds of real docs (and some tests expected certain docs to have specific IDs) but even when the deleted docs only appear "after" the real docs, there are still some test failures with this latest patch.
TestSimpleExplanations and TestComplexExplanations are the ones i've been looking at but there may be others (my laptop gets to hot if i try to run the full test suite too often)
my head hurts trying to figure out why the deeper nested structures and deleted docs might be causing these new failures ... i'll try to look at it tomorow.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744694#action_12744694 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
I would have been all over it ;) But who knows if you have your reasons for not resolving yet.
Thanks for all your work on this ! Brilliant issue that caught a lot of great stuff.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742552#action_12742552 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
bq. (grr.... it uses it's own random so no seed was logged)
Correction: it does log the seed, i was just looking for stderr when i should have been looking for stdout...
{code}
failed query: +field:w2 field:w3 field:xx field:w4 field:w2
NOTE: random seed of testcase 'testRandomQueries' was: 5695251427490718890
{code}
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742603#action_12742603 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
{quote}
Well that explains half the output anyway - even if thats fixed there is still a fail. Its because the tests doesn't expand fully into the subreaders - just needed the top level before - with this test, we need to recursively grab them.
{quote}
You lost me there... are you saying the _tests_ needs to be changed? ... why?
For this patch to trigger an error in an existing test, that test must either be using CheckHits or QueryUtils to execute a query against a seracher and validate the results are ok ... why would the test be responsible for any subreader expansion in this case?
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743036#action_12743036 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
bq. Sweet! ... Even the random boolean query test? (i see changes to ReqExclScorer in your patch, but i don't relaly see how that could solve the problem)
Yes - the issue was again how it was dealing with the sub readers - my original fix was not fully correct - it just happened to work with the previous tests.
By the way - any changes to non tests should have been pulled - sorry - I'll put up another patch.
bq. I'm clearly missing something ... why does ItemizedFilter need to change? I know it's an abomination and you would never want to use something like that in real code, but in this test the docIds passed to it on construction shouldn't change - even when we wrap the reader/searcher in other searchers, i specificly only put empty indexes (with no deletions) in front of the original index when bulding the multireader/searcher so the docIds will be the same. why doesn't the existing implementation work?
Because it uses external ids - that only works if you have a single reader. The filter is applied per sub-reader - so if you use a filter that just sets doc 0 - every sub reader will match its first doc. You could do this in the past, because it just worked on the top level reader - in that case, external ids match completely against the reader - but with the sub-reader approach, each sub-reader has an id space that starts at 0.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man reassigned LUCENE-1791:
--------------------------------
Assignee: Hoss Man
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-1791:
--------------------------------
Attachment: LUCENE-1791.patch
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742608#action_12742608 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
Okay - so the first the original Parser issue:
The output looked odd - it showed multiple entries with the same type, field, and reader, but null and DefaultParser.
When I quickly switched the code to use DefaultParser instead of null, those extra entries went away, and it just showed the segmentreader and directory reader that were doubled up. Thats what had me thinking that was involved. I'm not sure I understand why that was happening now though.
Anyway, all of the issues appear to be because the test code was written expecting all of the readers to be top level.
The tests, in certain cases, use a Reader to grab from a fieldcache - that reader has to be the right subreader and not the top level reader -
the tests were just using getSequentialSubReaders - they need use gatherSubReaders instead - because you introduced the multi level reader stuff. They should have used gatherSubReaders from the start, but because it wasn't needed at the time (for the tests to pass), it didn't even occur to me.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-1791:
-----------------------------
Attachment: LUCENE-1791.patch
Patch showing what i have in mind.
Current patch causes 14 failures in TestComplexExplanations, all of which stem from calling checkFirstSkipTo() on IndexSearcher created by wrapUnderlyingReader().
The failure is QueryUtils.java:305: "unstable skipTo(0) score! expected:<NaN> but was:<NaN>" which seems like it could easily be a bug introduced by the patch, except...
* Why only cause 14/22 in TestComplexExplanations to fail?
* Why wouldn't the original IndexSearcher have produced a NaN score?
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-1791:
--------------------------------
Attachment: LUCENE-1791.patch
just fully rolling out to all of the subreaders makes the test pass I believe.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742586#action_12742586 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
I'm guess the NAN failures are not a problem - looks like they fail because NAN != NAN ? Havn't looked closer.
I don't think the fieldcache insanity is multi-reader related - it looks to me like some entries have a parser, and some null for the parser, even though the default parser is being used in both cases. The FieldSource types grab a FieldCache and may pass null as the parser, which ends up putting null in the cache entry - but if you specifically ask for the default parser, that puts the default parser in the fieldcache entry - same stuff now, doubled entry.
As for the out of bounds - havn't look at that one yet - odd one ...
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742598#action_12742598 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
{quote}I don't think the fieldcache insanity is multi-reader related - it looks to me like some entries have a parser, and some null for the parser, even though the default parser is being used in both cases. The FieldSource types grab a FieldCache and may pass null as the parser, which ends up putting null in the cache entry - but if you specifically ask for the default parser, that puts the default parser in the fieldcache entry - same stuff now, doubled entry.{quote}
Well that explains half the output anyway - even if thats fixed there is still a fail. Its because the tests doesn't expand fully into the subreaders - just needed the top level before - with this test, we need to recursively grab them.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-1791:
-----------------------------
Attachment: LUCENE-1791.patch
i put the doc "ids" into a KEY field and refactored ItemizedFilter to be a trivial subclass of FieldCacheTermsFilter.
I also added more wrap permutations to address some of the possible edge cases Simon pointed out (good catch SImon) but didn't introduce any randomization for hte reasons mentioned before (even with the change to not rely on consistent docIds in ItemizedFilter, we can't allow deletions before the wrapped searcher/reader because CheckHIts does it magic based on docIds.
(hmm... i suppose the wrap functions could return some metadata about what offset the old ids have in the new search/reader and CheckHits could use that .... hmmm ... seems kludgy so i'm not going to worry about it)
I think we're good to go here unless anyone has any objections
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742614#action_12742614 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
FYI: with mark's updated path, we're back to just the NaN failures from TestComplexExplanations. (and possibly TestBoolean2.testRandomQueries, but i can't confirm that)
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742634#action_12742634 ]
Mark Miller commented on LUCENE-1791:
-------------------------------------
I only get the NAN issue showing up now.
I don't know why its returning NAN as the score at the moment, but as far as the test is concerned, that particular failure appears to be false. Its just checking that two calls to score return the same things - but if it returns NAN, they are not considered equal.
So I guess the question is - does it make sense that the score is NAN?
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743039#action_12743039 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
bq. The filter is applied per sub-reader
Doh! .. right, that's the part i was forgetting -- filters are another thing that happen per reader now.
sweet. i'll review the patch for real tomorow or sat, and maybe rip out the ItemizedFilter and replace it with something more sensical (it's a bad example that people might stumble upon, that seems even more confusing now, and i feel responsible since i'm the one that wrote it for those explanation tests anyway)
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-1791:
-----------------------------
Attachment: LUCENE-1791.patch
I figured out the problem with TestComplexExplanations ... the test uses a searcher with a Custom Similarity, and the new code wasn't setting the same Similarity on the new Searcher & MultiSearcher being created.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated LUCENE-1791:
-----------------------------
Fix Version/s: 2.9
I just retried this patch against the trunk now that the FieldCacheSanityChecker and some other patches have been committed. In addition to the possibly false-negatives from TestComplexExplanation (NaN score) this is now surfacing FielCache sanity failures from TestCustomScoreQuery, TestFieldScoreQuery, and TestOrdValues (suggesting that there are code paths where those query types don't correctly use the subreaders to get the FieldCache) as well as checkFirstSkipTo() failures for TestSpansAdvanced2 and an ArrayIndexOutOfBoundsException from TestBoolean2.testRandomQueries (grr.... it uses it's own random so no seed was logged)
I don't pretend this patch is perfect, but i can't imagine these are all false-negatives.
We should get to the bottom of this before 2.9. I'll start trying to figure it out on the train tonight.
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated LUCENE-1791:
--------------------------------
Attachment: LUCENE-1791.patch
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743032#action_12743032 ]
Hoss Man commented on LUCENE-1791:
----------------------------------
bq. Okay, I've got all tests passing.
Sweet! ... Even the random boolean query test? (i see changes to ReqExclScorer in your patch, but i don't relaly see how that could solve the problem)
bq. I rigged up a smarted ItemizedFilter that works for these tests.
I'm clearly missing something ... why does ItemizedFilter need to change? I know it's an abomination and you would never want to use something like that in real code, but in this test the docIds passed to it on construction shouldn't change -- even when we wrap the reader/searcher in other searchers, i specificly only put empty indexes (with no deletions) in front of the original index when bulding the multireader/searcher so the docIds will be the same. why doesn't the existing implementation work?
bq. Would it make sense to randomized reader / searcher indexes
Simon: randomized wrapping would be nice, but i didn't try to do that for two reasons:
* the utility has no way to get the seed from LuceneTestCase, and i didn't want to introduce randomness unless it was predictable randomness that could be logged.
* i needed to be sure we didn't add an index with deletions prior to the original index (but based on Mark's comments, it looks like that's not really an issue since we have to deal with it anyway)
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1791) Enhance QueryUtils and CheckHIts to
wrap everything they check in MultiReader/MultiSearcher
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man resolved LUCENE-1791.
------------------------------
Resolution: Fixed
Geezz... you expect me to commit *and* resolve ... i thought this open source stuff was all about volunteering and cooperation, how come i gotta do _both_ ?!?!
> Enhance QueryUtils and CheckHIts to wrap everything they check in MultiReader/MultiSearcher
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1791
> URL: https://issues.apache.org/jira/browse/LUCENE-1791
> Project: Lucene - Java
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Hoss Man
> Fix For: 2.9
>
> Attachments: LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch, LUCENE-1791.patch
>
>
> methods in CheckHits & QueryUtils are in a good position to take any Searcher they are given and not only test it, but also test MultiReader & MultiSearcher constructs built around them
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org