You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Kirk Lund (Jira)" <ji...@apache.org> on 2020/02/26 00:13:00 UTC
[jira] [Commented] (GEODE-6448)
LocatorUDPSecurityDUnitTest.testStartTwoLocators has suspicious string for
javax.crypto.BadPaddingException
[ https://issues.apache.org/jira/browse/GEODE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045029#comment-17045029 ]
Kirk Lund commented on GEODE-6448:
----------------------------------
Note that many dunit tests are reproducing this failure (not frequently though).
While running PersistentColocatedPartitionedRegionDistributedTest a thousand times to verify that I've fixed a flaky issue in the test, it hit an interesting failure trying to send a RequestImageMessage. This generated a stack trace which caused the test to fail grep for suspect strings. I can easily suppress this failure, BUT it looks like a bug in message distribution which may have been introduced by the recent membership changes (modularization).
{noformat}
[fatal 2020/02/19 02:50:04.862 GMT <Pooled Waiting Message Processor 1> tid=8410] While pushing message <InitialImageOperation$RequestImageMessage(region path='/__PR/_B__region2_1'; sender=172.17.0.4(185)<v758>:41003; keysOnly=false; processorId=40462; waitForInit=false; checkTombstoneVersions=true; versionVector=RegionVersionVector[2ab5849689d446bd-a7da0400b0e718f7={rv0 gc0 localVersion=0 local exceptions=[]} others={}, gc={}]; unfinished keys=[])> to recipients: <172.17.0.4(179)<v757>:41002>
java.lang.IllegalArgumentException: newPosition > limit: (32768 > 90)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
at java.base/java.nio.Buffer.position(Buffer.java:293)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
at java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226)
at java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:116)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:58)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:463)
at org.apache.geode.internal.tcp.Connection.writeFully(Connection.java:2587)
at org.apache.geode.internal.tcp.Connection.sendPreserialized(Connection.java:1867)
at org.apache.geode.internal.tcp.MsgStreamer.realFlush(MsgStreamer.java:324)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:249)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:393)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:248)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
at org.apache.geode.distributed.internal.DistributionImpl.directChannelSend(DistributionImpl.java:348)
at org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:293)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2060)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1987)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2024)
at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1084)
at org.apache.geode.internal.cache.InitialImageOperation.getFromOne(InitialImageOperation.java:514)
at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1222)
at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1082)
at org.apache.geode.internal.cache.BucketRegion.initialize(BucketRegion.java:259)
at org.apache.geode.internal.cache.LocalRegion.createSubregion(LocalRegion.java:983)
at org.apache.geode.internal.cache.PartitionedRegionDataStore.createBucketRegion(PartitionedRegionDataStore.java:785)
at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucket(PartitionedRegionDataStore.java:460)
at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucketRecursively(PartitionedRegionDataStore.java:319)
at org.apache.geode.internal.cache.PartitionedRegionDataStore.grabBucket(PartitionedRegionDataStore.java:2896)
at org.apache.geode.internal.cache.partitioned.ManageBackupBucketMessage.operateOnPartitionedRegion(ManageBackupBucketMessage.java:159)
at org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:394)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:458)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:449)
at org.apache.geode.distributed.internal.ClusterOperationExecutors.doWaitingThread(ClusterOperationExecutors.java:416)
at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
> LocatorUDPSecurityDUnitTest.testStartTwoLocators has suspicious string for javax.crypto.BadPaddingException
> -----------------------------------------------------------------------------------------------------------
>
> Key: GEODE-6448
> URL: https://issues.apache.org/jira/browse/GEODE-6448
> Project: Geode
> Issue Type: Bug
> Components: locator, membership
> Reporter: Xiaojian Zhou
> Priority: Major
> Labels: CI
>
> {noformat}
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/427 failed with
> > Task :geode-core:distributedTest
> org.apache.geode.distributed.LocatorUDPSecurityDUnitTest > testStartTwoLocators FAILED
> java.lang.AssertionError: Suspicious strings were written to the log during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 1024
> [error 2019/02/22 23:02:43.432 UTC <unicast receiver,170b732f838e-11757> tid=603] Exception deserializing message payload: [dst: 170b732f838e:41002, src: 170b732f838e:41001 (2 headers), size=250 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
> java.lang.Exception: Message id is -143
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1138)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1042)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1296)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1265)
> at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
> at org.jgroups.JChannel.up(JChannel.java:741)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
> at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73)
> at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
> at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
> at org.jgroups.protocols.TP.receive(TP.java:1714)
> at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152)
> at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:991)
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:847)
> at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
> at javax.crypto.Cipher.doFinal(Cipher.java:2164)
> at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncryptionCipherPool.decryptBytes(GMSEncryptionCipherPool.java:68)
> at org.apache.geode.distributed.internal.membership.gms.messenger.GMSEncrypt.decryptData(GMSEncrypt.java:147)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1117)
> ... 21 more
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 1111
> [error 2019/02/22 23:02:43.737 UTC <unicast receiver,170b732f838e-11757> tid=603] Exception deserializing message payload: [dst: 170b732f838e<v1>:41002, src: 170b732f838e<v0>:41001 (2 headers), size=1324 bytes, flags=OOB|DONT_BUNDLE|NO_FC|SKIP_BARRIER]
> java.lang.Exception: Message id is -150
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1138)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readJGMessage(JGroupsMessenger.java:1042)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1296)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1265)
> at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
> at org.jgroups.JChannel.up(JChannel.java:741)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
> at org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73)
> at org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
> at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
> at org.jgroups.protocols.TP.receive(TP.java:1714)
> at org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152)
> at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: While reading an InetAddress
> at org.apache.geode.DataSerializer.readInetAddress(DataSerializer.java:478)
> at org.apache.geode.distributed.internal.membership.gms.GMSMember.readEssentialData(GMSMember.java:545)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.deserializeMessage(JGroupsMessenger.java:1148)
> at org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.readEncryptedMessage(JGroupsMessenger.java:1128)
> ... 21 more
> Caused by: java.net.UnknownHostException: addr is of illegal length
> at java.net.InetAddress.getByAddress(InetAddress.java:1043)
> at java.net.InetAddress.getByAddress(InetAddress.java:1440)
> at org.apache.geode.DataSerializer.readInetAddress(DataSerializer.java:472)
> ... 24 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)