You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Owen Nichols (Jira)" <ji...@apache.org> on 2022/06/22 20:47:05 UTC
[jira] [Closed] (GEODE-9311) It is possible that JedisCluster client may still retry if the Geode Redis server it is connected to shuts down
[ https://issues.apache.org/jira/browse/GEODE-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen Nichols closed GEODE-9311.
-------------------------------
> It is possible that JedisCluster client may still retry if the Geode Redis server it is connected to shuts down
> ---------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-9311
> URL: https://issues.apache.org/jira/browse/GEODE-9311
> Project: Geode
> Issue Type: Bug
> Components: redis
> Reporter: Eric Shu
> Priority: Major
> Labels: unreleased
> Fix For: 1.15.0
>
>
> Even after the issue (GEODE-9310) is addressed, the retry issue may still occur if the JedisCluster client is connected to the node is being shut down.
> Here is a test run result:
> vm2 gets the command:
> [vm2] [warn 2021/05/25 16:01:41.479 PDT server-2 <GeodeRedisServer-Command-1> tid=0x64] Executing Redis command: ZREM key member1-212 member1-211 member1-214 member1-213 member1-210 member1-209 member1-208 member1-205 member1-204
> *** command is executed on the primary ***
> [vm2] [warn 2021/05/25 16:01:41.503 PDT server-2 <Function Execution Processor2> tid=0x51] bucket region is BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
> [vm2] java.lang.Exception
> [vm2] at org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
> [vm2] at org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> [vm2] at org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
> [vm2] at org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
> [vm2] at org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
> [vm2] at org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> And distributed to replica (vm1):
> vm1] [warn 2021/05/25 16:01:41.518 PDT server-1 <P2P message reader for 192.168.0.14(server-2:83136)<v2>:41002 shared ordered sender uid=7 local port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
> [vm1] java.lang.Exception
> [vm1] at org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
> [vm1] at org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
> [vm1] at org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
> [vm1] at org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
> [vm1] at org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
> [vm1] at org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
> [vm1] at org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
> [vm1] at org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)
> vm2 is bounced:
> [vm2] [info 2021/05/25 16:01:41.526 PDT server-2 <RMI TCP Connection(1)-192.168.0.14> tid=0x14] Got result: 83136
> [vm2] from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0 args on object: org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a (took 0 ms)
> [info 2021/05/25 16:01:41.527 PDT <pool-1-thread-2> tid=0x36] Bouncing 2 old pid is 83136 and version is 10240.0.0
> The JedisCluster client did not get response back (possibly detects connection is gone), and it does retry again, as the test failed with following:
> java.util.concurrent.ExecutionException: redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more cluster attempts left.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.apache.geode.test.dunit.rules.ClusterStartupRule$1.evaluate(ClusterStartupRule.java:138)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:54)
> at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more cluster attempts left.
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:84)
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:122)
> at redis.clients.jedis.JedisClusterCommand.run(JedisClusterCommand.java:25)
> at redis.clients.jedis.JedisCluster.zrem(JedisCluster.java:984)
> at org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.doZRem(ZRemDUnitTest.java:157)
> at org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.lambda$zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed$0(ZRemDUnitTest.java:174)
> at org.apache.geode.test.junit.rules.ExecutorServiceRule.lambda$submit$2(ExecutorServiceRule.java:232)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)