You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Bruce J Schuchardt (Jira)" <ji...@apache.org> on 2020/02/26 21:35:00 UTC
[jira] [Comment Edited] (GEODE-7807)
PersistentColocatedPartitionedRegionDistributedTest.
testReplaceOfflineMemberAndRestart_WithMultipleDiskStores failed with a
communications error
[ https://issues.apache.org/jira/browse/GEODE-7807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045923#comment-17045923 ]
Bruce J Schuchardt edited comment on GEODE-7807 at 2/26/20 9:34 PM:
--------------------------------------------------------------------
I hit a similar error in feature/GEODE-7807 in test PdxQueryDUnitTest > testPdxReadSerializedForPRQuery
{noformat}
Found suspect string in log4j at line 2794
[fatal 2020/02/26 21:24:09.132 GMT <P2P message reader for 172.17.0.18(191)<v76>:41002 shared ordered uid=98 port=47004> tid=1491] While pushing message <ReplyMessage processorId=1876 from 172.17.0.18(187)<v75>:41001> to recipients: <172.17.0.18(191)<v76>:41002>
java.lang.IndexOutOfBoundsException
at java.base/java.nio.Buffer.checkIndex(Buffer.java:688)
at java.base/java.nio.DirectByteBuffer.putShort(DirectByteBuffer.java:692)
at org.apache.geode.internal.tcp.MsgStreamer.setMessageHeader(MsgStreamer.java:457)
at org.apache.geode.internal.tcp.MsgStreamer.realFlush(MsgStreamer.java:312)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:393)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:248)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
at org.apache.geode.distributed.internal.DistributionImpl.directChannelSend(DistributionImpl.java:348)
at org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:293)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2060)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1987)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2024)
at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1084)
at org.apache.geode.distributed.internal.ReplyMessage.send(ReplyMessage.java:176)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.sendReply(DistributedCacheOperation.java:1273)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1259)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1110)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:394)
at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:449)
at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2073)
at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1847)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:931)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:862)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1204)
at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:826)
at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:707)
at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:661)
at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3190)
at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:2937)
at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2744)
at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1632)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1472)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
This branch did not have the revert of GEODE-7727 at the time (0eb49ab9c1535a6ae5d56c63ce7f85ed33e090fb).
This code is setting the header of a message buffer to note the length of the message, etc. It should not cause an IOOB exception.
was (Author: bschuchardt):
I hit a similar error in feature/GEODE-7807 in test PdxQueryDUnitTest > testPdxReadSerializedForPRQuery
{noformat}
Found suspect string in log4j at line 2794
[fatal 2020/02/26 21:24:09.132 GMT <P2P message reader for 172.17.0.18(191)<v76>:41002 shared ordered uid=98 port=47004> tid=1491] While pushing message <ReplyMessage processorId=1876 from 172.17.0.18(187)<v75>:41001> to recipients: <172.17.0.18(191)<v76>:41002>
java.lang.IndexOutOfBoundsException
at java.base/java.nio.Buffer.checkIndex(Buffer.java:688)
at java.base/java.nio.DirectByteBuffer.putShort(DirectByteBuffer.java:692)
at org.apache.geode.internal.tcp.MsgStreamer.setMessageHeader(MsgStreamer.java:457)
at org.apache.geode.internal.tcp.MsgStreamer.realFlush(MsgStreamer.java:312)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:393)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:248)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
at org.apache.geode.distributed.internal.DistributionImpl.directChannelSend(DistributionImpl.java:348)
at org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:293)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2060)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1987)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2024)
at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1084)
at org.apache.geode.distributed.internal.ReplyMessage.send(ReplyMessage.java:176)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.sendReply(DistributedCacheOperation.java:1273)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1259)
at org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1110)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:394)
at org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:449)
at org.apache.geode.distributed.internal.ClusterDistributionManager.scheduleIncomingMessage(ClusterDistributionManager.java:2073)
at org.apache.geode.distributed.internal.ClusterDistributionManager.handleIncomingDMsg(ClusterDistributionManager.java:1847)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.dispatchMessage(GMSMembership.java:931)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.handleOrDeferMessage(GMSMembership.java:862)
at org.apache.geode.distributed.internal.membership.gms.GMSMembership.processMessage(GMSMembership.java:1204)
at org.apache.geode.distributed.internal.DistributionImpl$MyDCReceiver.messageReceived(DistributionImpl.java:826)
at org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:707)
at org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:661)
at org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:3190)
at org.apache.geode.internal.tcp.Connection.readMessage(Connection.java:2937)
at org.apache.geode.internal.tcp.Connection.processInputBuffer(Connection.java:2744)
at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1632)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1472)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
This branch did not have the revert of GEODE-7727 at the time (0eb49ab9c1535a6ae5d56c63ce7f85ed33e090fb).
> PersistentColocatedPartitionedRegionDistributedTest. testReplaceOfflineMemberAndRestart_WithMultipleDiskStores failed with a communications error
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-7807
> URL: https://issues.apache.org/jira/browse/GEODE-7807
> Project: Geode
> Issue Type: Bug
> Components: membership, messaging
> Reporter: Bruce J Schuchardt
> Priority: Major
>
> This test failed with a suspect string showing a possible message transmission problem. Is this possibly related to work done in early 2020 on BufferPool? See this revision:
> 418d929e3e03185cd6330c828c9b9ed395a76d4b
> {noformat}
> [fatal 2020/02/19 02:50:04.862 GMT <Pooled Waiting Message Processor 1>
> tid=8410] While pushing message
> <InitialImageOperation$RequestImageMessage(region
> path='/__PR/_B__region2_1'; sender=172.17.0.4(185)<v758>:41003;
> keysOnly=false; processorId=40462; waitForInit=false;
> checkTombstoneVersions=true;
> versionVector=RegionVersionVector[2ab5849689d446bd-a7da0400b0e718f7={rv0
> gc0 localVersion=0 local exceptions=[]} others={}, gc={}]; unfinished
> keys=[])> to recipients: <172.17.0.4(179)<v757>:41002>
> java.lang.IllegalArgumentException: newPosition > limit: (32768 > 90)
> at
> java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
> at java.base/java.nio.Buffer.position(Buffer.java:293)
> at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226)
> at
> java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67)
> at
> java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:116)
> at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:58)
> at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
> at
> java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:463)
> at
> org.apache.geode.internal.tcp.Connection.writeFully(Connection.java:2587)
> at
> org.apache.geode.internal.tcp.Connection.sendPreserialized(Connection.java:1867)
> at
> org.apache.geode.internal.tcp.MsgStreamer.realFlush(MsgStreamer.java:324)
> at
> org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:393)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:248)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
> at
> org.apache.geode.distributed.internal.DistributionImpl.directChannelSend(DistributionImpl.java:348)
> at
> org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:293)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2060)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1987)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2024)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1084)
> at
> org.apache.geode.internal.cache.InitialImageOperation.getFromOne(InitialImageOperation.java:514)
> at
> org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1222)
> at
> org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1082)
> at
> org.apache.geode.internal.cache.BucketRegion.initialize(BucketRegion.java:259)
> at
> org.apache.geode.internal.cache.LocalRegion.createSubregion(LocalRegion.java:983)
> at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.createBucketRegion(PartitionedRegionDataStore.java:785)
> at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucket(PartitionedRegionDataStore.java:460)
> at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucketRecursively(PartitionedRegionDataStore.java:319)
> at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.grabBucket(PartitionedRegionDataStore.java:2896)
> at
> org.apache.geode.internal.cache.partitioned.ManageBackupBucketMessage.operateOnPartitionedRegion(ManageBackupBucketMessage.java:159)
> at
> org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
> at
> org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:394)
> at
> org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:458)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:449)
> at
> org.apache.geode.distributed.internal.ClusterOperationExecutors.doWaitingThread(ClusterOperationExecutors.java:416)
> at
> org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)