You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by Vahram Aharonyan <va...@vmware.com> on 2019/01/30 15:12:32 UTC
Numerous SSLSocketImpl objects after NotSerializableException in
message sending flow
Hi All,
At some circumstance we have seen following exception while using geode-1.1.0 and trying to do Function Execution:
[severe 2018/05/22 01:32:31.964 EDT Node-7ccee253-7f00-4670-960a-3ba6e5a2365b <Function Execution Processor22> tid=0x58] ack read exception
org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.UpdateOperation$UpdateMessage
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436)
at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
at org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
at org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
at org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621)
at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: An Exception was thrown while serializing.
at org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930)
at org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305)
at org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124)
at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419)
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
... 61 more
Caused by: java.io.NotSerializableException: dataobject.PolicyInfo
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
At the same time, we have experienced OOM on our jvm and heapdump was generated. Heapdump analysis showed that nearly half of the heap (16 GB of 32 GB) is consumed by SSLSocketImpl objects. Dominator Tree was topped by 'Function Execution Processor22' thread, which is mentioned above in the stacktrace including serialization exception.
So the main question is whether having NotSerializableException exception in sendMessage() flow could result to leak of not closed connections. Connection closing calls are spread over the code and is hard to track them and I'm not sure If org.apache.geode.internal.tcp.MsgStreamer#close is the one that is responsible to close connections in this case. But if so, connections will not be closed here as due to abovementioned NotSerializableException flushedBytes will not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage.
Can someone assist us to understand if this exception can yield to the SSLSocketImpl leak or we should look into another direction?
Thanks,
Vahram.
RE: Numerous SSLSocketImpl objects after NotSerializableException in
message sending flow
Posted by Vahram Aharonyan <va...@vmware.com>.
Hi Bruce,
Thank you for the feedback. Will keep an eye on this after Geode version update in our end.
Best Regards,
Vahram.
From: Bruce Schuchardt <bs...@pivotal.io>
Sent: Thursday, January 31, 2019 3:02 AM
To: user@geode.apache.org
Subject: Re: Numerous SSLSocketImpl objects after NotSerializableException in message sending flow
Hi Vahram,
The socket leak was fixed in v1.6
https://issues.apache.org/jira/browse/GEODE-3563<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-3563&data=02%7C01%7Cvaharonyan%40vmware.com%7C04b42bd0388a4f8eb37a08d68706fd8a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636844861427040537&sdata=S6LuHtzICKzkMns6KMkgVaSl8UQzGs730mtU8N2aZJA%3D&reserved=0>
On 1/30/19 7:12 AM, Vahram Aharonyan wrote:
Hi All,
At some circumstance we have seen following exception while using geode-1.1.0 and trying to do Function Execution:
[severe 2018/05/22 01:32:31.964 EDT Node-7ccee253-7f00-4670-960a-3ba6e5a2365b <Function Execution Processor22> tid=0x58] ack read exception
org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.UpdateOperation$UpdateMessage
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436)
at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446)
at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377)
at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237)
at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684)
at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
at org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
at org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
at org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621)
at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: An Exception was thrown while serializing.
at org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930)
at org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305)
at org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124)
at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419)
at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
... 61 more
Caused by: java.io.NotSerializableException: dataobject.PolicyInfo
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
At the same time, we have experienced OOM on our jvm and heapdump was generated. Heapdump analysis showed that nearly half of the heap (16 GB of 32 GB) is consumed by SSLSocketImpl objects. Dominator Tree was topped by 'Function Execution Processor22' thread, which is mentioned above in the stacktrace including serialization exception.
So the main question is whether having NotSerializableException exception in sendMessage() flow could result to leak of not closed connections. Connection closing calls are spread over the code and is hard to track them and I'm not sure If org.apache.geode.internal.tcp.MsgStreamer#close is the one that is responsible to close connections in this case. But if so, connections will not be closed here as due to abovementioned NotSerializableException flushedBytes will not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage.
Can someone assist us to understand if this exception can yield to the SSLSocketImpl leak or we should look into another direction?
Thanks,
Vahram.
Re: Numerous SSLSocketImpl objects after NotSerializableException in
message sending flow
Posted by Bruce Schuchardt <bs...@pivotal.io>.
Hi Vahram,
The socket leak was fixed in v1.6
https://issues.apache.org/jira/browse/GEODE-3563
On 1/30/19 7:12 AM, Vahram Aharonyan wrote:
>
> Hi All,
>
> At some circumstance we have seen following exception while using
> geode-1.1.0 and trying to do Function Execution:
>
> [severe 2018/05/22 01:32:31.964 EDT
> Node-7ccee253-7f00-4670-960a-3ba6e5a2365b <Function Execution
> Processor22> tid=0x58] ack read exception
> org.apache.geode.ToDataException: toData failed on DataSerializable
> class org.apache.geode.internal.cache.UpdateOperation$UpdateMessage
> at
> org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436)
> at
> org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446)
> at
> org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
> at
> org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
> at
> org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
> at
> org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
> at
> org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
> at
> org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621)
> at
> org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: An Exception was thrown
> while serializing.
> at
> org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930)
> at
> org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305)
> at
> org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124)
> at
> org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419)
> at
> org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
> ... 61 more
> Caused by: java.io.NotSerializableException: dataobject.PolicyInfo
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>
> At the same time, we have experienced OOM on our jvm and heapdump was
> generated. Heapdump analysis showed that nearly half of the heap (16
> GB of 32 GB) is consumed by SSLSocketImpl objects. Dominator Tree was
> topped by 'Function Execution Processor22' thread, which is mentioned
> above in the stacktrace including serialization exception.
>
> So the main question is whether having NotSerializableException exception insendMessage() flow could result
> to leak of not closed connections. Connection closing calls are spread over the code and is hard to track
> them and I’m not sure If org.apache.geode.internal.tcp.MsgStreamer#closeis the one that is responsible to close connections in this case. But if so, connections will not be closed here as due to abovementioned NotSerializableException flushedByteswill not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage.
> Can someone assist us to understand if this exception can yield to the
> SSLSocketImpl leak or we should look into another direction?
> Thanks,
> Vahram.