You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Tim K <ti...@gmail.com> on 2019/02/11 16:27:57 UTC
Re: StaticMembers within Multiple Clusters
On Fri, Jan 18, 2019, 12:44 PM Tim K <ti...@gmail.com> wrote:
> On Fri, Jan 18, 2019 at 11:05 AM Christopher Schultz
> <ch...@christopherschultz.net> wrote:
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Tim,
> >
> > On 1/18/19 06:38, Tim K wrote:
> > > Thanks for this. The video helps explain it a bit better than the
> > > documentation. So I set it up with a backup manager instead of the
> > > delta manager, changing the channelSendOptions to 6 for the
> > > cluster.
> >
> > If you think you can help clarify the documentation, patches are of
> > course always welcome.
> >
> > > From a maintenance standpoint, what is the best way to stop/start
> > > the nodes without losing sessions; one at a time, letting it fully
> > > come up before moving on to the next one (like a ripple restart)?
> > > I presume you don't want too many nodes to be down at a single
> > > time.
> >
> > I definitely wouldn't bring two down simultaneously if your can avoid
> > it. The cluster needs time to re-stabalize after the loss of a member,
> > meaning that new backup nodes must be selected for each session and
> > then the sessions must be transmitted to those backups nodes. If you
> > have small amounts of data in the sessions, this will probably be
> > fairly fast. If you have lots of data or a very busy network, it will
> > take longer.
> >
> > I would recommend setting-up a scenario (even in production) where you
> > intentionally disable a node in the cluster and watch to see how long
> > the cluster takes to re-stabalize. I think you'll learn a lot from
> > that exercise and it will help you plan for scheduled maintenance and
> > downtime.
> >
> > - -chris
>
> Is there a way to tell which server was assigned as the primary and
> backup roles?
>
> When I stop a member, is it this line which would tell me how long it
> took to sync up the sessions?
> Relocation of map entries was complete in [X] ms.
>
> Another question, I'm using the StaticMembershipService; do I need to
> define a LocalMember for each of my nodes or is that optional/assumed?
>
> Also, I recall reading something about the uniqueId might not really
> be used? Do I need to set that for each member?
>
I'm noticing my SSO cookie is being removed when I force myself to another
node. Is this a bug?
>
Re: StaticMembers within Multiple Clusters
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Tim,
On 1/25/23 11:26, Tim K wrote:
>> Can you post the rest of that stack trace?
> Yes, here are 2 stack traces that were encountered. We basically had
> the cluster working for a few years. We introduced a new Valve for
> authentication purposes. Also, with this change we had to set a proxy
> in CATALINA_OPTS, I'm not sure if that affected local communication
> between the nodes? For now we commented out the cluster on each of
> our nodes in order to have it running.
>
> Exception in thread "Tribes-Task-Receiver[Catalina-Channel]-9"
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.catalina.tribes.ChannelException
>
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:110)
>
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
>
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
>
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:89)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
>
> at
> org.apache.catalina.tribes.group.interceptors.EncryptInterceptor.sendMessage(EncryptInterceptor.java:127)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
>
> at
> org.apache.catalina.tribes.group.RpcChannel.messageReceived(RpcChannel.java:171)
>
> at
> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:345)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
>
> at
> org.apache.catalina.tribes.group.interceptors.EncryptInterceptor.messageReceived(EncryptInterceptor.java:148)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:182)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
>
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:114)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
>
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
>
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:288)
>
> at
> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:272)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:228)
>
> at
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:103)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:750)
Hmm... no "caused by"? That's disappointing. NCDFE usually means "you
done broke your install" but it says that the initialization of the
class failed, not that the class wasn't actually found.
Any way you can attach a debugger and find out what's causing that to be
thrown?
>> What was your previous version of Tomcat?
> We were always on version 9. We keep it pretty much up to date with
> the latest available. I am not sure of the sub-version we started at
> where it was working. I'm guessing it was whatever version was the
> latest around Jan-2019.
>
>> Did you upgrade all nodes at the same time, or are you upgrading a single node in the cluster?
> All are at the same version, we have 4, they all get updated at the same time.
>
>> How did you upgrade (e.g. installer, unzip/untar/etc.)?
> untar
Yeah, that definitely smells weird. I have no specific advice (other
than "try a debugger") but I do want to at least validate that it
doesn't look like you are doing anything wrong, or misinterpreting
something obvious :)
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: StaticMembers within Multiple Clusters
Posted by Tim K <ti...@gmail.com>.
> Can you post the rest of that stack trace?
Yes, here are 2 stack traces that were encountered. We basically had
the cluster working for a few years. We introduced a new Valve for
authentication purposes. Also, with this change we had to set a proxy
in CATALINA_OPTS, I'm not sure if that affected local communication
between the nodes? For now we commented out the cluster on each of
our nodes in order to have it running.
Exception in thread "Tribes-Task-Receiver[Catalina-Channel]-9"
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.catalina.tribes.ChannelException
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:110)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:89)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.EncryptInterceptor.sendMessage(EncryptInterceptor.java:127)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
at
org.apache.catalina.tribes.group.RpcChannel.messageReceived(RpcChannel.java:171)
at
org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:345)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
at
org.apache.catalina.tribes.group.interceptors.EncryptInterceptor.messageReceived(EncryptInterceptor.java:148)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
at
org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:182)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:114)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
at
org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:288)
at
org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:272)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:228)
at
org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
WARNING [https-jsse-nio-9443-exec-22]
org.apache.catalina.tribes.transport.nio.ParallelNioSender.keepalive
Error during keepalive test for
sender:[org.apache.catalina.tribes.transport.nio.NioSender@ee16415]
java.nio.channels.NotYetConnectedException
at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:258)
at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:299)
at
org.apache.catalina.tribes.transport.nio.NioSender.read(NioSender.java:175)
at
org.apache.catalina.tribes.transport.nio.ParallelNioSender.keepalive(ParallelNioSender.java:395)
at
org.apache.catalina.tribes.transport.PooledSender.returnSender(PooledSender.java:48)
at
org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:57)
at
org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
at
org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:89)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.interceptors.EncryptInterceptor.sendMessage(EncryptInterceptor.java:127)
at
org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
at
org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
at
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:189)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1177)
at
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1159)
at
org.apache.catalina.session.ManagerBase.add(ManagerBase.java:722)
at
org.apache.catalina.session.StandardSession.setId(StandardSession.java:359)
at
org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:327)
at
org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:345)
at
org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:763)
at
org.apache.catalina.connector.Request.doGetSession(Request.java:3104)
at
org.apache.catalina.connector.Request.getSessionInternal(Request.java:2757)
... remove a few lines ...
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:625)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at
org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at
org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:183)
at
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:750)
> What was your previous version of Tomcat?
We were always on version 9. We keep it pretty much up to date with
the latest available. I am not sure of the sub-version we started at
where it was working. I'm guessing it was whatever version was the
latest around Jan-2019.
> Did you upgrade all nodes at the same time, or are you upgrading a single node in the cluster?
All are at the same version, we have 4, they all get updated at the same time.
> How did you upgrade (e.g. installer, unzip/untar/etc.)?
untar
Thanks,
Tim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: StaticMembers within Multiple Clusters
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Tim,
On 1/15/23 12:26, Tim K wrote:
> I hate to bring back my original thread and I am probably not doing
> this correctly, but I've been seeing this message occur on my cluster.
> My tomcat is now at 9.0.70. Possibly there was a breaking change
> since I first started using the cluster?
>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.catalina.tribes.ChannelException
There was a new type of message introduced recently, but it should have
been both forward- and backward-compatible.
Can you post the rest of that stack trace? What was your previous
version of Tomcat? Did you upgrade all nodes at the same time, or are
you upgrading a single node in the cluster? How did you upgrade (e.g.
installer, unzip/untar/etc.)?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: StaticMembers within Multiple Clusters
Posted by Tim K <ti...@gmail.com>.
I hate to bring back my original thread and I am probably not doing
this correctly, but I've been seeing this message occur on my cluster.
My tomcat is now at 9.0.70. Possibly there was a breaking change
since I first started using the cluster?
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.catalina.tribes.ChannelException
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: StaticMembers within Multiple Clusters
Posted by Tim K <ti...@gmail.com>.
On Tue, Feb 12, 2019, 3:17 AM Keiichi Fujino <kf...@apache.org> wrote:
>
> Are you using SSO(org.apache.catalina.authenticator.SingleSignOn)?
> DeltaManager/BackupManager replicate sessions. They do not replicate SSO
> entries.
>
> If you want to replicate SSO Entry in cluster, you can use
> ClusterSingleSignOn.
>
>
> http://tomcat.apache.org/tomcat-9.0-doc/config/cluster-valve.html#org.apache.catalina.ha.authenticator.ClusterSingleSignOn
>
>
>
> --
> Keiichi.Fujino
>
Yes. I tried adding a Value element for ClusterSingleSignOn to my Cluster
(removed the JvmBinderValue I had) and upon login, I'm noticing that
subsequent calls to my app are removing my SSO cookie, but I don't
understand why. It successfully gets created but a subsequent call
immediately removes it.
>
Re: StaticMembers within Multiple Clusters
Posted by Keiichi Fujino <kf...@apache.org>.
2019年2月12日(火) 1:28 Tim K <ti...@gmail.com>:
> On Fri, Jan 18, 2019, 12:44 PM Tim K <ti...@gmail.com> wrote:
>
> > On Fri, Jan 18, 2019 at 11:05 AM Christopher Schultz
> > <ch...@christopherschultz.net> wrote:
> > >
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA256
> > >
> > > Tim,
> > >
> > > On 1/18/19 06:38, Tim K wrote:
> > > > Thanks for this. The video helps explain it a bit better than the
> > > > documentation. So I set it up with a backup manager instead of the
> > > > delta manager, changing the channelSendOptions to 6 for the
> > > > cluster.
> > >
> > > If you think you can help clarify the documentation, patches are of
> > > course always welcome.
> > >
> > > > From a maintenance standpoint, what is the best way to stop/start
> > > > the nodes without losing sessions; one at a time, letting it fully
> > > > come up before moving on to the next one (like a ripple restart)?
> > > > I presume you don't want too many nodes to be down at a single
> > > > time.
> > >
> > > I definitely wouldn't bring two down simultaneously if your can avoid
> > > it. The cluster needs time to re-stabalize after the loss of a member,
> > > meaning that new backup nodes must be selected for each session and
> > > then the sessions must be transmitted to those backups nodes. If you
> > > have small amounts of data in the sessions, this will probably be
> > > fairly fast. If you have lots of data or a very busy network, it will
> > > take longer.
> > >
> > > I would recommend setting-up a scenario (even in production) where you
> > > intentionally disable a node in the cluster and watch to see how long
> > > the cluster takes to re-stabalize. I think you'll learn a lot from
> > > that exercise and it will help you plan for scheduled maintenance and
> > > downtime.
> > >
> > > - -chris
> >
> > Is there a way to tell which server was assigned as the primary and
> > backup roles?
> >
> > When I stop a member, is it this line which would tell me how long it
> > took to sync up the sessions?
> > Relocation of map entries was complete in [X] ms.
> >
> > Another question, I'm using the StaticMembershipService; do I need to
> > define a LocalMember for each of my nodes or is that optional/assumed?
> >
> > Also, I recall reading something about the uniqueId might not really
> > be used? Do I need to set that for each member?
> >
>
>
> I'm noticing my SSO cookie is being removed when I force myself to another
> node. Is this a bug?
>
>
>
Are you using SSO(org.apache.catalina.authenticator.SingleSignOn)?
DeltaManager/BackupManager replicate sessions. They do not replicate SSO
entries.
If you want to replicate SSO Entry in cluster, you can use
ClusterSingleSignOn.
http://tomcat.apache.org/tomcat-9.0-doc/config/cluster-valve.html#org.apache.catalina.ha.authenticator.ClusterSingleSignOn
--
Keiichi.Fujino