You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/06/28 23:09:10 UTC
[jira] [Commented] (GEODE-1558)
LuceneQueriesPeerPRDUnitTest.returnCorrectResultsAfterRebalance failed on
flushing the queue
[ https://issues.apache.org/jira/browse/GEODE-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353895#comment-15353895 ]
ASF GitHub Bot commented on GEODE-1558:
---------------------------------------
GitHub user aparnard opened a pull request:
https://github.com/apache/incubator-geode/pull/176
GEODE-1558: Fixing failures due to rebalancing buckets during lucene …
…index updates
There are two failures we're fixing here:
1) Calling repository.create for create events was leaving duplicate events in the
index because when a bucket fails over, the event is dispatched again on the new primary.
Using the isPossibleDuplicate flag did not work because it was not consistently set to
true for duplicate events. Changed the code to call repository.update even for create events
2) The async event queue was repeatedly dispatching the same events event after
a bucket moved to another node. We changed the async event queue code to filter out
events for buckets that are no longer present on this dispatching member.
Cleaning up the rebalancing test and adding new tests to make these scenarios
more reproducable.
Signed-off-by: Aparna Dharmakkan <ad...@pivotal.io>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aparnard/incubator-geode feature/GEODE-1558
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-geode/pull/176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #176
----
commit d659a4ae97602e885638dd4081250aee02708a15
Author: Dan Smith <ds...@pivotal.io>
Date: 2016-06-28T22:33:51Z
GEODE-1558: Fixing failures due to rebalancing buckets during lucene index updates
There are two failures we're fixing here:
1) Calling repository.create for create events was leaving duplicate events in the
index because when a bucket fails over, the event is dispatched again on the new primary.
Using the isPossibleDuplicate flag did not work because it was not consistently set to
true for duplicate events. Changed the code to call repository.update even for create events
2) The async event queue was repeatedly dispatching the same events event after
a bucket moved to another node. We changed the async event queue code to filter out
events for buckets that are no longer present on this dispatching member.
Cleaning up the rebalancing test and adding new tests to make these scenarios
more reproducable.
Signed-off-by: Aparna Dharmakkan <ad...@pivotal.io>
----
> LuceneQueriesPeerPRDUnitTest.returnCorrectResultsAfterRebalance failed on flushing the queue
> --------------------------------------------------------------------------------------------
>
> Key: GEODE-1558
> URL: https://issues.apache.org/jira/browse/GEODE-1558
> Project: Geode
> Issue Type: Bug
> Components: lucene
> Reporter: Aparna Dharmakkan
>
> The error was:
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at com.gemstone.gemfire.cache.lucene.LuceneQueriesPRBase.returnCorrectResultsAfterRebalance(LuceneQueriesPRBase.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:112)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
> at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> and the log output:
> {noformat}
> [vm_0]49.448: [GC (Allocation Failure) [PSYoungGen: 150556K->11769K(162816K)] 165672K->37374K(230400K), 0.0693682 secs] [Times: user=0.06 sys=0.01, real=0.07 secs]
> [vm_3][warn 2016/06/16 16:08:16.915 PDT <Event Processor for GatewaySender_AsyncEventQueue_index#_region_0> tid=0x48] com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher@1215f139: Exception during processing batch 0
> [vm_3]com.gemstone.gemfire.internal.cache.wan.GatewaySenderException: com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher@1215f139: Exception during processing batch 0, caused by com.gemstone.gemfire.internal.cache.PrimaryBucketException: Bucket _B__index#__region.files_0 is not primary. Current primary holder is 192.168.1.17(12468)<ec><v51>:1027
> [vm_3] at com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:177)
> [vm_3] at com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:87)
> [vm_3] at com.gemstone.gemfire.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:613)
> [vm_3] at com.gemstone.gemfire.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1058)
> [vm_3]Caused by: com.gemstone.gemfire.internal.cache.PrimaryBucketException: Bucket _B__index#__region.files_0 is not primary. Current primary holder is 192.168.1.17(12468)<ec><v51>:1027
> [vm_3] at com.gemstone.gemfire.internal.cache.BucketRegion.checkForPrimary(BucketRegion.java:670)
> [vm_3] at com.gemstone.gemfire.internal.cache.BucketRegion.doLockForPrimary(BucketRegion.java:721)
> [vm_3] at com.gemstone.gemfire.internal.cache.BucketRegion.beginLocalWrite(BucketRegion.java:697)
> [vm_3] at com.gemstone.gemfire.internal.cache.BucketRegion.virtualPut(BucketRegion.java:476)
> [vm_3] at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:132)
> [vm_3] at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:5241)
> [vm_3] at com.gemstone.gemfire.internal.cache.LocalRegion.putIfAbsent(LocalRegion.java:12022)
> [vm_3] at com.gemstone.gemfire.internal.cache.LocalRegion.putIfAbsent(LocalRegion.java:12046)
> [vm_3] at com.gemstone.gemfire.cache.lucene.internal.filesystem.FileSystem.createFile(FileSystem.java:67)
> [vm_3] at com.gemstone.gemfire.cache.lucene.internal.directory.RegionDirectory.createOutput(RegionDirectory.java:84)
> [vm_3] at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
> [vm_3] at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> [vm_3] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
> [vm_3] at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
> [vm_3] at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
> [vm_3] at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:83)
> [vm_3] at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:331)
> [vm_3] at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:368)
> [vm_3] at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> [vm_3] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
> [vm_3] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
> [vm_3] at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
> [vm_3] at com.gemstone.gemfire.cache.lucene.internal.repository.IndexRepositoryImpl.create(IndexRepositoryImpl.java:70)
> [vm_3] at com.gemstone.gemfire.cache.lucene.internal.LuceneEventListener.processEvents(LuceneEventListener.java:78)
> [vm_3] at com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:168)
> [vm_3] ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)