You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Ivan Godwin (Jira)" <ji...@apache.org> on 2020/02/04 23:56:04 UTC
[jira] [Resolved] (GEODE-7516)
RebalanceWithRedundancyDUnitTest.returnCorrectResultsWhenIndexUpdateHappensIntheMiddleofGII(PARTITION_REDUNDANT)
[ https://issues.apache.org/jira/browse/GEODE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Godwin resolved GEODE-7516.
--------------------------------
Fix Version/s: 1.12.0
Resolution: Fixed
Fixed by following commit:
commit 9f4d70b849d7f7094fc78e589587bd28be65f122
Author: Juan José Ramos <ju...@users.noreply.github.com>
Date: Fri Jan 17 14:24:35 2020 -0800
GEODE-7703: Catch IndexWriter Exceptions (#4597)
The IndexWriter initialization might fail when other threads are
updating the fileAndChunkRegion, which can be triggered by other normal
operations (query, event listener, close, reindex, etc.). This doesn't
happen often and, instead of propagating the exception to the caller
and failing, Geode now catches it and returns null to let the callers
retry.
- Added unit and distrbuted tests.
- Return null instead of re-throwing the IOException while building
the Lucene IndexWriter.
Co-authored-by: Xiaojian Zhou <ge...@users.noreply.github.com>
> RebalanceWithRedundancyDUnitTest.returnCorrectResultsWhenIndexUpdateHappensIntheMiddleofGII(PARTITION_REDUNDANT)
> -----------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-7516
> URL: https://issues.apache.org/jira/browse/GEODE-7516
> Project: Geode
> Issue Type: Bug
> Components: tests
> Reporter: Mark Hanson
> Assignee: Ivan Godwin
> Priority: Major
> Labels: flaky
> Fix For: 1.12.0
>
>
> Test failures in mass test runs
> {noformat}
> returnCorrectResultsWhenIndexUpdateHappensIntheMiddleofGII(PARTITION_REDUNDANT) https://concourse.apachegeode-ci.info/teams/main/pipelines/mhansonp-mhanson-mass-test-ru-main/jobs/DistributedTestOpenJDK8/builds/1879
> returnCorrectResultsWhenIndexUpdateHappensIntheMiddleofGII(PARTITION_REDUNDANT) https://concourse.apachegeode-ci.info/teams/main/pipelines/mhansonp-mhanson-mass-test-ru-main/jobs/DistributedTestOpenJDK8/builds/1808{noformat}
> {noformat}
> Stack trace from first test failure listed
> org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.cache.lucene.LuceneQueriesAccessorBase$$Lambda$44/1008073793.run in VM 3 running on Host 963109d98c34 with 4 VMs
> at org.apache.geode.test.dunit.VM.executeMethodOnObject(VM.java:610)
> at org.apache.geode.test.dunit.VM.invoke(VM.java:437)
> at org.apache.geode.cache.lucene.LuceneQueriesAccessorBase.executeTextSearch(LuceneQueriesAccessorBase.java:110)
> at org.apache.geode.cache.lucene.RebalanceWithRedundancyDUnitTest.returnCorrectResultsWhenIndexUpdateHappensIntheMiddleofGII(RebalanceWithRedundancyDUnitTest.java:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at junitparams.internal.InvokeParameterisedMethod.evaluate(InvokeParameterisedMethod.java:234)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at junitparams.internal.ParameterisedTestMethodRunner.runMethodInvoker(ParameterisedTestMethodRunner.java:47)
> at junitparams.internal.ParameterisedTestMethodRunner.runTestMethod(ParameterisedTestMethodRunner.java:40)
> at junitparams.internal.ParameterisedTestClassRunner.runParameterisedTest(ParameterisedTestClassRunner.java:146)
> at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:446)
> at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:393)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
> at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
> at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
> at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.InternalGemFireError: Unable to create index repository
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.lambda$computeRepository$1(PartitionedRepositoryManager.java:172)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.computeRepository(PartitionedRepositoryManager.java:162)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.getRepository(PartitionedRepositoryManager.java:139)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.getRepositories(PartitionedRepositoryManager.java:100)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:153)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:83)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2998)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:101)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:372)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:436)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:475)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:393)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.lang.Thread.run(Thread.java:748)
> at org.apache.geode.distributed.internal.ReplyException.handleCause(ReplyException.java:90)
> at org.apache.geode.internal.cache.execute.FunctionStreamingResultCollector.waitForCacheOrFunctionException(FunctionStreamingResultCollector.java:481)
> at org.apache.geode.internal.cache.partitioned.PRFunctionStreamingResultCollector.getResultInternal(PRFunctionStreamingResultCollector.java:100)
> at org.apache.geode.internal.cache.execute.ResultCollectorHolder.getResult(ResultCollectorHolder.java:53)
> at org.apache.geode.internal.cache.partitioned.PRFunctionStreamingResultCollector.getResult(PRFunctionStreamingResultCollector.java:88)
> at org.apache.geode.cache.lucene.internal.LuceneQueryImpl.findTopEntries(LuceneQueryImpl.java:122)
> at org.apache.geode.cache.lucene.internal.LuceneQueryImpl.findKeys(LuceneQueryImpl.java:70)
> at org.apache.geode.cache.lucene.LuceneQueriesAccessorBase.lambda$executeTextSearch$e67cf21a$1(LuceneQueriesAccessorBase.java:117)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
> at org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> Caused by: java.io.FileNotFoundException: _0.cfe
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.filesystem.FileSystem.getFile(FileSystem.java:101)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.directory.RegionDirectory.openInput(RegionDirectory.java:115)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.store.Directory.openChecksumInput(Directory.java:137)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:105)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1058)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1075)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:960)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.IndexRepositoryFactory.finishComputingRepository(IndexRepositoryFactory.java:123)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.IndexRepositoryFactory.computeIndexRepository(IndexRepositoryFactory.java:66)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.computeRepository(PartitionedRepositoryManager.java:151)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.lambda$computeRepository$1(PartitionedRepositoryManager.java:170)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.computeRepository(PartitionedRepositoryManager.java:162)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.getRepository(PartitionedRepositoryManager.java:139)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.getRepositories(PartitionedRepositoryManager.java:100)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:153)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:83)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2998)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:101)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:372)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:436)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:475)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:393)
> at Remote Member '172.17.0.4(246)<v30>:41003' in org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
> at Remote Member '172.17.0.4(246)<v30>:41003' in java.lang.Thread.run(Thread.java:748) {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)