You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2013/04/24 22:59:17 UTC
[jira] [Comment Edited] (LUCENE-4953) readerClosedListener is not
invoked for ParallelCompositeReader's leaves
[ https://issues.apache.org/jira/browse/LUCENE-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640943#comment-13640943 ]
Uwe Schindler edited comment on LUCENE-4953 at 4/24/13 8:58 PM:
----------------------------------------------------------------
As discussed in IRC today, like MultiReader/SlowComposite that warps another reader, we should no call close() on the subreaders. The problem here is: E.g., A wrapped slow reader still returns its delegate as core cache key, so the reader closed listener works as expected, when the child is closed. But PMR must not return the inner cache keys, as it wraps them with modifying index contents.
The problem here is that the leaves are atomic and the field cache does not get the reader closed event. The somehow best fix would be to call readerClosedListener for all childs (not leaves!) on doClose. Theoretically, all CompositeReaders that wrap ll childs of other readers should do this (only Parallel is currently doing this).
Another idea would be to use incRef and decRef, but that would not affect the wrapped atomic readers, so I prefer the previous solution.
was (Author: thetaphi):
As discussed in IRC today, like MultiReader/SlowComposite that warps another reader, we should no call close() on the subreaders. The problem here is: A wrapped slow reader still returns its delegate as core cache key, so the reader closed listener works as expected.
The problem here is that the leaves are atomic and the field cache does not get the reader closed event. The somehow best fix would be to call readerClosedListener for all childs (not leaves!) on close. Theoretically, all CompositeReaders that wrap ll childs of other readers should do this (only Parallel is currently doing this).
Another idea would be to use incRef and decRef, but that would not affect the wrapped atomic readers, so I prefer the previous solution.
> readerClosedListener is not invoked for ParallelCompositeReader's leaves
> ------------------------------------------------------------------------
>
> Key: LUCENE-4953
> URL: https://issues.apache.org/jira/browse/LUCENE-4953
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Fix For: 5.0, 4.4
>
> Attachments: LUCENE-4953.patch, LUCENE-4953.patch
>
>
> There was a test failure last night:
> {noformat}
> 1 tests failed.
> REGRESSION: org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest.testBasic
> Error Message:
> testBasic(org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest): Insane FieldCache usage(s) found expected:<0> but was:<2>
> Stack Trace:
> java.lang.AssertionError: testBasic(org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest): Insane FieldCache usage(s) found expected:<0> but was:<2>
> at __randomizedtesting.SeedInfo.seed([1F9C2A2AD23A8E02:B466373F0DE6082C]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at org.apache.lucene.util.LuceneTestCase.assertSaneFieldCaches(LuceneTestCase.java:592)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:55)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:722)
> Build Log:
> [...truncated 6904 lines...]
> [junit4:junit4] Suite: org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest
> [junit4:junit4] 2> *** BEGIN testBasic(org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest): Insane FieldCache usage(s) ***
> [junit4:junit4] 2> VALUEMISMATCH: Multiple distinct value objects for ParallelAtomicReader(_0(5.0):C3)+id
> [junit4:junit4] 2> 'ParallelAtomicReader(_0(5.0):C3)'=>'id',class org.apache.lucene.index.SortedDocValues,0.5=>org.apache.lucene.search.FieldCacheImpl$SortedDocValuesImpl#386041791 (size =~ 232 bytes)
> [junit4:junit4] 2> 'ParallelAtomicReader(_0(5.0):C3)'=>'id',int,org.apache.lucene.search.FieldCache.DEFAULT_INT_PARSER=>org.apache.lucene.search.FieldCacheImpl$IntsFromArray#140912913 (size =~ 48 bytes)
> [junit4:junit4] 2> 'ParallelAtomicReader(_0(5.0):C3)'=>'id',int,null=>org.apache.lucene.search.FieldCacheImpl$IntsFromArray#140912913 (size =~ 48 bytes)
> [junit4:junit4] 2>
> [junit4:junit4] 2> VALUEMISMATCH: Multiple distinct value objects for ParallelAtomicReader(_1(5.0):C5)+id
> [junit4:junit4] 2> 'ParallelAtomicReader(_1(5.0):C5)'=>'id',int,null=>org.apache.lucene.search.FieldCacheImpl$IntsFromArray#1105632232 (size =~ 56 bytes)
> [junit4:junit4] 2> 'ParallelAtomicReader(_1(5.0):C5)'=>'id',int,org.apache.lucene.search.FieldCache.DEFAULT_INT_PARSER=>org.apache.lucene.search.FieldCacheImpl$IntsFromArray#1105632232 (size =~ 56 bytes)
> [junit4:junit4] 2> 'ParallelAtomicReader(_1(5.0):C5)'=>'id',class org.apache.lucene.index.SortedDocValues,0.5=>org.apache.lucene.search.FieldCacheImpl$SortedDocValuesImpl#27148040 (size =~ 232 bytes)
> [junit4:junit4] 2>
> [junit4:junit4] 2> *** END testBasic(org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest): Insane FieldCache usage(s) ***
> [junit4:junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=AllGroupHeadsCollectorTest -Dtests.method=testBasic -Dtests.seed=1F9C2A2AD23A8E02 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=be_BY -Dtests.timezone=Asia/Manila -Dtests.file.encoding=US-ASCII
> [junit4:junit4] FAILURE 0.75s J1 | AllGroupHeadsCollectorTest.testBasic <<<
> [junit4:junit4] > Throwable #1: java.lang.AssertionError: testBasic(org.apache.lucene.search.grouping.AllGroupHeadsCollectorTest): Insane FieldCache usage(s) found expected:<0> but was:<2>
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([1F9C2A2AD23A8E02:B466373F0DE6082C]:0)
> [junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.assertSaneFieldCaches(LuceneTestCase.java:592)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {sort3=MockFixedIntBlock(blockSize=733), id=Pulsing41(freqCutoff=3 minBlockSize=50 maxBlockSize=177), content=MockFixedIntBlock(blockSize=733), author=Pulsing41(freqCutoff=3 minBlockSize=50 maxBlockSize=177), sort2=MockVariableIntBlock(baseBlockSize=71), sort1=Pulsing41(freqCutoff=3 minBlockSize=50 maxBlockSize=177), group=Pulsing41(freqCutoff=3 minBlockSize=50 maxBlockSize=177)}, docValues:{author_dv=DocValuesFormat(name=Disk), group_dv=DocValuesFormat(name=Disk)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {content=IB LL-L1, author=DFR GBZ(0.3)}, locale=be_BY, timezone=Asia/Manila
> [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_17 (64-bit)/cpus=16,threads=1,free=157973280,total=249626624
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [GroupFacetCollectorTest, AllGroupsCollectorTest, AllGroupHeadsCollectorTest]
> {noformat}
> It reproduces, and happens because ParallelCompositeReader isn't invoking the reader listeners on its .leaves() when everything is closed. I made a separate test case to show the issue ...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org