You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Galen O'Sullivan (JIRA)" <ji...@apache.org> on 2018/02/07 21:41:00 UTC
[jira] [Updated] (GEODE-4614) PutMessage.toData throws NPE if cache
has started closing
[ https://issues.apache.org/jira/browse/GEODE-4614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Galen O'Sullivan updated GEODE-4614:
------------------------------------
Description:
This is from a failure in CI. I'm pretty sure it was caused by a PutMessage being sent after PartitionedRegion.prIdToPR was cleared on Cache close. This causes the message to throw a NullPointerException, which eventually results in a CacheClosedException.
One problem is that this message is getting access to the Region to increment some stats -- that should probably be done at a higher level. Another problem is that we use this static map rather than holding a reference to the Region itself.
{code:none}
[warning 2018/01/30 15:30:35.543 PST bridgegemfire6_rs-FullRegression-2018-01-29-17-49-20-client-19_2013 <ServerConnection on port 29379 Thread 3> tid=0xee] Server connection from [identity(rs-FullRegression-2018-01-29-17-49-20-client-19(edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055:2055:loner):42782:6bcd6549:edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055,connection=1; port=45552]: Unexpected Exception
org.apache.geode.cache.CacheClosedException: The cache is closed., caused by org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
at org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1527)
at org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
at org.apache.geode.internal.cache.tier.CachedRegionHelper.checkCancelInProgress(CachedRegionHelper.java:38)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.checkForInterrupt(BaseCommand.java:690)
at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:430)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
at org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:638)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2421)
at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1499)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:233)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:378)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:238)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1715)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1905)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:3224)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:3139)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:3177)
at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1706)
at org.apache.geode.internal.cache.partitioned.PutMessage.send(PutMessage.java:386)
at org.apache.geode.internal.cache.PartitionedRegion.putRemotely(PartitionedRegion.java:3185)
at org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2825)
at org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2018)
at org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
at org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5591)
at org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5227)
at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
... 8 more
Caused by: java.lang.NullPointerException
at org.apache.geode.internal.cache.partitioned.PutMessage.toData(PutMessage.java:621)
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2391)
... 27 more
{code}
was:
This is from a failure in CI. I'm pretty sure it was caused by a PutMessage being sent after PartitionedRegion.prIdToPR was cleared on Cache close. This causes the message to thrown a NullPointerException, which eventually results in a CacheClosedException.
One problem is that this message is getting access to the Region to increment some stats -- that should probably be done at a higher level. Another problem is that we use this static map rather than holding a reference to the Region itself.
{code:none}
[warning 2018/01/30 15:30:35.543 PST bridgegemfire6_rs-FullRegression-2018-01-29-17-49-20-client-19_2013 <ServerConnection on port 29379 Thread 3> tid=0xee] Server connection from [identity(rs-FullRegression-2018-01-29-17-49-20-client-19(edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055:2055:loner):42782:6bcd6549:edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055,connection=1; port=45552]: Unexpected Exception
org.apache.geode.cache.CacheClosedException: The cache is closed., caused by org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
at org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1527)
at org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
at org.apache.geode.internal.cache.tier.CachedRegionHelper.checkCancelInProgress(CachedRegionHelper.java:38)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.checkForInterrupt(BaseCommand.java:690)
at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:430)
at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
at org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:638)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2421)
at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1499)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:233)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:378)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:238)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1715)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1905)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:3224)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:3139)
at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:3177)
at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1706)
at org.apache.geode.internal.cache.partitioned.PutMessage.send(PutMessage.java:386)
at org.apache.geode.internal.cache.PartitionedRegion.putRemotely(PartitionedRegion.java:3185)
at org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2825)
at org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2018)
at org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
at org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5591)
at org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5227)
at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
... 8 more
Caused by: java.lang.NullPointerException
at org.apache.geode.internal.cache.partitioned.PutMessage.toData(PutMessage.java:621)
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2391)
... 27 more
{code}
> PutMessage.toData throws NPE if cache has started closing
> ---------------------------------------------------------
>
> Key: GEODE-4614
> URL: https://issues.apache.org/jira/browse/GEODE-4614
> Project: Geode
> Issue Type: Bug
> Components: messaging
> Reporter: Galen O'Sullivan
> Priority: Major
>
> This is from a failure in CI. I'm pretty sure it was caused by a PutMessage being sent after PartitionedRegion.prIdToPR was cleared on Cache close. This causes the message to throw a NullPointerException, which eventually results in a CacheClosedException.
> One problem is that this message is getting access to the Region to increment some stats -- that should probably be done at a higher level. Another problem is that we use this static map rather than holding a reference to the Region itself.
> {code:none}
> [warning 2018/01/30 15:30:35.543 PST bridgegemfire6_rs-FullRegression-2018-01-29-17-49-20-client-19_2013 <ServerConnection on port 29379 Thread 3> tid=0xee] Server connection from [identity(rs-FullRegression-2018-01-29-17-49-20-client-19(edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055:2055:loner):42782:6bcd6549:edgegemfire1_rs-FullRegression-2018-01-29-17-49-20-client-19_2055,connection=1; port=45552]: Unexpected Exception
> org.apache.geode.cache.CacheClosedException: The cache is closed., caused by org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
> at org.apache.geode.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1527)
> at org.apache.geode.CancelCriterion.checkCancelInProgress(CancelCriterion.java:83)
> at org.apache.geode.internal.cache.tier.CachedRegionHelper.checkCancelInProgress(CachedRegionHelper.java:38)
> at org.apache.geode.internal.cache.tier.sockets.BaseCommand.checkForInterrupt(BaseCommand.java:690)
> at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:430)
> at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
> at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
> at org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
> at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1177)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:638)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.partitioned.PutMessage
> at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2421)
> at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1499)
> at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:233)
> at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:378)
> at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:238)
> at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:604)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1715)
> at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1905)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:3224)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:3139)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:3177)
> at org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1706)
> at org.apache.geode.internal.cache.partitioned.PutMessage.send(PutMessage.java:386)
> at org.apache.geode.internal.cache.PartitionedRegion.putRemotely(PartitionedRegion.java:3185)
> at org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2825)
> at org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2018)
> at org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
> at org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5591)
> at org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5227)
> at org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
> ... 8 more
> Caused by: java.lang.NullPointerException
> at org.apache.geode.internal.cache.partitioned.PutMessage.toData(PutMessage.java:621)
> at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2391)
> ... 27 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)