You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mitch Claborn <mi...@claborn.net> on 2018/09/07 21:44:27 UTC
Error receiving all sessions in cluster
I'm using a cluster with the DeltaManager between two servers on Tomcat
9.0.11. When I restart my "A" server, I sometimes see the error below.
Peaking at the code in DeltaManager it appears that there is a problem
in the serialized data stream when receiving all of the session data
from the "B" server. Any ideas what could be causing this? It seems like
that if there was something in a Session object that was not
Serializable it would not even make it over the wire. Perhaps if
something in a session did not serialize properly?
(From the code, it looks like all the sessions in the message after the
error are lost, so this is kind of important.)
07-Sep-2018 16:09:53.220 SEVERE
[Tribes-Task-Receiver[Catalina-Channel]-6]
org.apache.catalina.ha.session.DeltaManager.deserializeSessions
IOException while loading persisted sessions:
[java.io.StreamCorruptedException: invalid type code: 00]
java.io.StreamCorruptedException: invalid type code: 00
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1381)
at java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1984)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1946)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
at
org.apache.catalina.ha.session.DeltaSession.doReadObject(DeltaSession.java:748)
at
org.apache.catalina.ha.session.DeltaSession.readObjectData(DeltaSession.java:569)
at
org.apache.catalina.ha.session.DeltaManager.deserializeSessions(DeltaManager.java:613)
at
org.apache.catalina.ha.session.DeltaManager.handleALL_SESSION_DATA(DeltaManager.java:1323)
at
org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1162)
at
org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:917)
at
org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:77)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:788)
at
org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:771)
at
org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:324)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:175)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:116)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor.messageReceived(StaticMembershipInterceptor.java:96)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.messageReceived(ThroughputInterceptor.java:86)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
at
org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:274)
at
org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:261)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:213)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:101)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
Mitch
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error receiving all sessions in cluster
Posted by Mitch Claborn <mi...@claborn.net>.
I apologize. I know see the "sendAllSessions" setting on DeltaManager.
Not sure how I missed that the first time. I'll experiment with that.
Still would like any insight into question #2.
Mitch
On 09/18/2018 08:51 AM, Mitch Claborn wrote:
> 1) Any time the DeltaManager runs into an error deserializing the "all
> sessions" message, all sessions from the one with the error and later
> are lost. I realize it must currently be this way because once there is
> an error in the serialized stream there is no realistic way to find
> where the next session object begins in the stream. I'd like to be able
> to configure the Delta Manager to send the "all sessions" message in
> chunks of N sessions per message, where N can be configured in the XML.
> Doing this would limit the impact of a problematic session to only those
> in the same chunk. The final chunk would need some kind of marker so
> that the receiver knows that all sessions have been received. Any
> thoughts on this? I don't think it is possible in the current code. How
> hard would it be to make this change. If I were to code it, would it be
> considered for acceptance into the base code?
>
> 2) Is it possible that a session object could be accessed and used while
> it is being deserialized in another thread for the "all sessions"
> message to another host? This might explain the corruption I'm seeing.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error receiving all sessions in cluster
Posted by Mitch Claborn <mi...@claborn.net>.
I continue to have problems of this nature, although not with the exact
symptom shown below. I've been able to identify some possible problems
in my code that might have been causing some objects in the session to
deserialize improperly. I've fixed those and have not seen those
specific errors crop up again. I have another error that I'm working on
now: some of my own objects serialize to/from JSON and the JSON string
in the serialized session object has obviously been corrupted. I can't
see how that could have happened, but I'm still working on it. So.... 2
big questions/ideas:
1) Any time the DeltaManager runs into an error deserializing the "all
sessions" message, all sessions from the one with the error and later
are lost. I realize it must currently be this way because once there is
an error in the serialized stream there is no realistic way to find
where the next session object begins in the stream. I'd like to be able
to configure the Delta Manager to send the "all sessions" message in
chunks of N sessions per message, where N can be configured in the XML.
Doing this would limit the impact of a problematic session to only those
in the same chunk. The final chunk would need some kind of marker so
that the receiver knows that all sessions have been received. Any
thoughts on this? I don't think it is possible in the current code. How
hard would it be to make this change. If I were to code it, would it be
considered for acceptance into the base code?
2) Is it possible that a session object could be accessed and used while
it is being deserialized in another thread for the "all sessions"
message to another host? This might explain the corruption I'm seeing.
PS - here's how I'm discovering the session objects that are in error. I
have a channel interceptor that watches for the receipt of the "all
sessions" message and writes it to disk then stops watching for that
message. I copy that disk file to my development machine where I run a
quick-and-dirty program that I wrote that emulates the processing of
Delta Manager with some debugging in place so that I can see exactly
which session had the problem and which attribute within that session.
I'd also like to propose some updates to Delta Manager that would
capture that information and put it into the error logging.
Mitch
On 09/07/2018 04:44 PM, Mitch Claborn wrote:
> I'm using a cluster with the DeltaManager between two servers on Tomcat
> 9.0.11. When I restart my "A" server, I sometimes see the error below.
> Peaking at the code in DeltaManager it appears that there is a problem
> in the serialized data stream when receiving all of the session data
> from the "B" server. Any ideas what could be causing this? It seems like
> that if there was something in a Session object that was not
> Serializable it would not even make it over the wire. Perhaps if
> something in a session did not serialize properly?
>
> (From the code, it looks like all the sessions in the message after the
> error are lost, so this is kind of important.)
>
> 07-Sep-2018 16:09:53.220 SEVERE
> [Tribes-Task-Receiver[Catalina-Channel]-6]
> org.apache.catalina.ha.session.DeltaManager.deserializeSessions
> IOException while loading persisted sessions:
> [java.io.StreamCorruptedException: invalid type code: 00]
> java.io.StreamCorruptedException: invalid type code: 00
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1381)
> at
> java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1984)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1946)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at
> org.apache.catalina.ha.session.DeltaSession.doReadObject(DeltaSession.java:748)
>
> at
> org.apache.catalina.ha.session.DeltaSession.readObjectData(DeltaSession.java:569)
>
> at
> org.apache.catalina.ha.session.DeltaManager.deserializeSessions(DeltaManager.java:613)
>
> at
> org.apache.catalina.ha.session.DeltaManager.handleALL_SESSION_DATA(DeltaManager.java:1323)
>
> at
> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1162)
>
> at
> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:917)
>
> at
> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:77)
>
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:788)
>
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:771)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:324)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:175)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:116)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor.messageReceived(StaticMembershipInterceptor.java:96)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.messageReceived(ThroughputInterceptor.java:86)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:274)
>
> at
> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:261)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:213)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:101)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org