Re: Tomcat Clustering

[Arun Kumar <ar...@gmail.com>]

Hi Chris

Appreciate your help.

On 12/28/14 5:59 PM, Arun Kumar wrote:
> Thanks a ton for reaching me out and i really appreciate your help. I am
> new to Tomcat clustering, if my questions doesn't make any sense, please
> forgive me :)
>
>> On 12/26/14 12:36 PM, Arun Kumar wrote:
>>> I have two Tomcat 7.0  installed on two different machines and
>>> edited the server.xml file on both the machines with the code
>>> below, i am not sure if the Tomcats are clustered.
>>
>> Which exact version of Tomcat 7.0?
>
> I have Tomcat 7.0.32 installed on both the machines.

You should definitely upgrade to 7.0.57 and continue with your testing.
There have been a lot of fixes in the intervening years between the two
releases. I haven't read the changelog, but I wouldn't be surprised if
something has been fixed between then and now.

The Tomcat versions that i have installed are by default by installing my
product SAP Business Objects, i haven't installed them separately. I cannot
update my tomcats, they will get updated once i update my Business Objects
product to higher versions.

>>> I tried to turn of one Tomcat manually to check if the clustering
>>> is implemented or not. But the failover doesn't work.
>>
>> Are you expecting that your sessions will be available to any node, or
>> are you using sticky sessions? The term "cluster" usually indicates
>> that you want distributable sessions that will be available on any
>> cluster node, but I just wanted to confirm that's what you want.
>
> _Two things_
>
> First i wanted to test if the first tomcat fails the second Tomcat has
> to start. I was not testing the session sticky session in the first
> scenario.

Just to be clear: both Tomcats have to be running at the same time. When
one fails, the other can't start-up and take-over. Instead, they share
the load at first and then if one node goes down, fail-over to the other
(still running node) occurs.

I used a wronging wording to explain, the Two Tomcats will be running all
the time

> Secondly i want to implement Sticky session. I guess Apache (mod_jk) has
> to be implemented in front of the Tomcat servers for sticky session.

If you want to use load-balancing *at all*, you'll need a network
component out front. mod_jk is not necessary, and neither is Apache
httpd: you can do this with other products as well.

> We
> have a hardware load balancer appliance, can we use this hardware load
> balancer for load balancing without installing Apache (mod_jk). I don't
> have the details of the hardware appliance that load balances the
> Tomcats. (My Manager is on leave, so i have to wait until he returns)

If you can rig the LB to use part of Tomcat's session id as a
differentiating factor, then use the part of the session id after the
period (.) to determine which "route" to use (set by the jvmRoute
attribute of your <Engine> in server.xml). If that doesn't work, you can
almost always use a LB's built-in session-stickiness, which typically
uses HTTP cookies to maintain identification of the back-end instance.

We have a Netscaler Hardware Loadbalance which we used to do loadbalance
our two Tomcat servers, implemented HTTP cookies for Sticky session.

>> If you can afford to have users re-login and re-start a workflow when
>> a node fails, using session-stickiness without any formal "clustering"
>> is a much easier configuration and will yield higher performance from
>> the whole setup.
>
> What is the process if the users can afford re-login and restart a
> workflow ? Could you please guide me how to implement it. Does the code
> which i sent will be helpful for this scenario.

If a user makes a request to a resource which requires authentication,
Tomcat will store the user's request, challenge the user for
authentication and then re-process the original request
post-authentication. So, as long as your workflows contain enough
information in the HTTP request to resume those workflows after a
failed-over login, then there's nothing else for you to do.

I logged in using a unique URL given by my loadbalancing team to test if
the session replication. Created an active session and started a workflow
while the two Tomcats were running, stopped Tomcat 1 and tested my session
to see if i got kicked out or not, in which i was not kicked out and
workflow was still remained, did the same procedure by turning on the
Tomcat 1 and stopping the Tomcat 2 and checked if the the workflow existed
or not. I got an error message saying

com.businessobjects.sdk.core.server.CommunicationException$UnexpectedServerException:
> HTTP transport error: java.net.ConnectException: Connection timed out:
> connect
> at
> com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilder.java:152)
> at
> com.businessobjects.sdk.core.exception.ExceptionBuilder.make(ExceptionBuilder.java:109)
> at
> com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:183)
> at
> com.businessobjects.sdk.core.server.internal.AbstractServer.process(AbstractServer.java:133)
> at
> com.businessobjects.sdk.core.server.internal.InstanceServer.process(InstanceServer.java:94)
> at
> com.sap.sl.sdk.parameter.service.processor.AbstractParameterProcessor.getParameters(AbstractParameterProcessor.java:336)
> at
> com.sap.sl.sdk.parameter.service.processor.WorkspaceParameterProcessor.getParameters(WorkspaceParameterProcessor.java:64)
> at
> com.sap.sl.sdk.parameter.service.processor.AbstractParameterProcessor.getNeededParameters(AbstractParameterProcessor.java:194)
> at
> com.sap.sl.sdk.parameter.service.ParameterServiceImpl.getNeededParameters(ParameterServiceImpl.java:52)
> at
> com.sap.webi.ui.tasks.workflows.PrepareWorkspaceTask.doIt(PrepareWorkspaceTask.java:161)
> at
> com.sap.webi.ui.tasks.workflows.PrepareWorkspaceTask.doIt(PrepareWorkspaceTask.java:20)
> at
> com.sap.webi.toolkit.ui.tasks.WebITask$PrivateWorker.doInBackground(WebITask.java:348)
> at javax.swing.SwingWorker$1.call(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at javax.swing.SwingWorker.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: com.sun.xml.internal.ws.client.ClientTransportException: HTTP
> transport error: java.net.ConnectException: Connection timed out: connect
> at
> com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(Unknown
> Source)
> at
> com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown
> Source)
> at
> com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown
> Source)
> at
> com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown
> Source)
> at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
> at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
> at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
> at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
> at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
> at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
> at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown
> Source)
> at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown
> Source)
> at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
> at com.sun.proxy.$Proxy34.processRequest(Unknown Source)
> at
> com.businessobjects.sdk.core.server.internal.http.HTTPServerImpl.doProcess(HTTPServerImpl.java:184)
> at
> com.businessobjects.sdk.core.server.internal.AbstractServer.processIt(AbstractServer.java:171)
> ... 15 more
> Caused by: java.net.ConnectException: Connection timed out: connect
> at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
> at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
> at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
> at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
> at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
> at java.net.PlainSocketImpl.connect(Unknown Source)
> at java.net.SocksSocketImpl.connect(Unknown Source)
> at java.net.Socket.connect(Unknown Source)
> at sun.net.NetworkClient.doConnect(Unknown Source)
> at sun.net.www.http.HttpClient.openServer(Unknown Source)
> at sun.net.www.http.HttpClient.openServer(Unknown Source)
> at sun.net.www.http.HttpClient.<init>(Unknown Source)
> at sun.net.www.http.HttpClient.New(Unknown Source)
> at sun.net.www.http.HttpClient.New(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown
> Source)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
> at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown
> Source)
>
>> ... 31 more
>
>
> By clicking OK after the error message, i still had the workflow and i
> wasn't kicked from my session. ( I mean i didn't need to relogin and start
> my entire workflow)
>

 I am confused if the session replication is happening or not with the code
that i put in the server.xml and the inlcuded the <distrubutable/> element
in the web.xml file as well.

>
>

> Anyway...
>
> Please let me know how to confirm if the servers are clustered or
> not. Appreciate your help.
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6">

I have this code in my server.xml file but not sure if the Tomcat's are
picking up the code to do the clustering or not, as i already explained you
the procedure that i tested to see if the clustering and session
replication are working or not. But not quite confident about it. My only
goal now is to get session replication working without loosing the workflow
and not getting kicked of the session. Could you please help me in detail
how to achieve it. I already have edited the web.xml (<distributable/> and
server.xml file with the following code, do i need to do something else


> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">

<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100" selectorTimeout="5000"
maxThreads="6"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

If you have set up <Cluster>, then you are using clustering. I was just
saying that if you don't *need* clustering, maybe you could abandon
trying to get it set up in the first place. Session distribution is
quite chatty, from a network perspective.

>> My first question is: do you have <distributable/> set in your
>> web.xml? That's the easiest thing to overlook because it's not in
>> server.xml and will silently prevent your sessions from being
>> distributed to the rest of the cluster.
>
> No, i haven't edited the web.xml file with the <distributable/> set, the
> "Tomcat - How to cluster" web page has that information, but they don't
> clearly explain the procedure where to add it in the web.xml file. Do i
> need to just add "<distributable/>" parameter at the end by opening it
> in the notepad. Please let me know.

Take a look at the XML Schema for web.xml and you should be able to find
the place where to put <distributable/>.

I have added the <distributable/> element in the web.xml file on both the
Tomcat servers. Path for the edited web.xml file is
 (E:\Program Files (x86)\SAP BusinessObjects\tomcat\webapps\BOE\WEB-INF)
and there is another web.xml file which is located at
 (E:\Program Files (x86)\SAP BusinessObjects\tomcat\conf) for which i
haven't done any changes, assuming that i have edited the right web.xml
file under Web-INF file.

<?xml version="1.0" encoding="UTF-8"?>
>
> <web-app id="BOE" version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee"
>
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
> xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
>> http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
>
>
>
> *<distributable/>*
>
>
>> <display-name>BOE</display-name>
>
>
>>     <listener>
>
>
>> <listener-class>com.businessobjects.servletbridge.listener.ProxyApplicationLifeCycleListener</listener-class>
>
>     </listener>
>
>
>
>     <servlet id="bridge">
>
> <description>Equinox Bridge Servlet</description>
>
> <display-name>Equinox Bridge Servlet</display-name>
>
> <servlet-name>equinoxbridgeservlet</servlet-name>
>
>
>> <servlet-class>org.eclipse.equinox.servletbridge.BridgeServlet</servlet-class>
>
> <init-param>
>
> <param-name>commandline</param-name>
>
> <param-value></param-value><!-- To turn on console, add "-console"-->
>
> <!-- Syntax: optionally add port #, eg "-console 8081", to redirect to
>> given port, into which you can telnet -->
>
> </init-param>
>
> <init-param>
>
> <!-- Used to enable sp_* control URLs.  Do not enable -->
>
> <param-name>enableFrameworkControls</param-name>
>
> <param-value>false</param-value>
>
> </init-param>
>
> <!--
>
>   org.eclipse.equinox.servletbridge and the Servlet API are exported
>> automatically to the underlying OSGi framework.
>
>   The extendedFrameworkExports parameter allows the specification of
>> additional java package exports.
>
>   The format is a comma separated list of exports as specified by the
>> "Export-Package" bundle manifest header.
>
>   For example: com.mycompany.exports; version=1.0.0,
>> com.mycompany.otherexports; version=1.0.0
>
>   -->
>
> <init-param>
>
> <param-name>extendedFrameworkExports</param-name>
>
>
>> <param-value>com.businessobjects.servletbridge.listener,com.businessobjects.servletbridge.customconfig,com.businessobjects.servletbridge.external,com.businessobjects.servletbridge.session,com.businessobjects.resource,oracle.jdbc.pool,com.siebel.data,com.jdedwards.system.xml,org.ietf.jgss,com.sap.security.api</param-value>
>
> </init-param>
>
>
>> <load-on-startup>1</load-on-startup>
>
> </servlet>
>
>     <servlet-mapping>
>
> <servlet-name>equinoxbridgeservlet</servlet-name>
>
> <url-pattern>/*</url-pattern>
>
> </servlet-mapping>
>
> <!--
>
>   This is required if your application bundles expose JSPs.
>
> -->
>
> <servlet-mapping>
>
> <servlet-name>equinoxbridgeservlet</servlet-name>
>
> <url-pattern>*.jsp</url-pattern>
>
> </servlet-mapping>
>
>
>
>     <filter>
>
>         <filter-name>TimeoutManangerFilter</filter-name>
>
>
>> <filter-class>com.businessobjects.pinger.TimeoutManagerFilter</filter-class>
>
>     </filter>
>
>
>
>     <filter-mapping>
>
>         <filter-name>TimeoutManangerFilter</filter-name>
>
>         <url-pattern>/*</url-pattern>
>
>     </filter-mapping>
>
>
>
>     <session-config>
>
>         <session-timeout>60</session-timeout>
>
>     </session-config>
>
> </web-app>
>
>
Thanks
Arun Chanda



Regards !!
Arun Chanda

On Mon, Dec 29, 2014 at 4:40 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> Arun,
>
> On 12/28/14 5:59 PM, Arun Kumar wrote:
> > Thanks a ton for reaching me out and i really appreciate your help. I am
> > new to Tomcat clustering, if my questions doesn't make any sense, please
> > forgive me :)
> >
> >> On 12/26/14 12:36 PM, Arun Kumar wrote:
> >>> I have two Tomcat 7.0  installed on two different machines and
> >>> edited the server.xml file on both the machines with the code
> >>> below, i am not sure if the Tomcats are clustered.
> >>
> >> Which exact version of Tomcat 7.0?
> >
> > I have Tomcat 7.0.32 installed on both the machines.
>
> You should definitely upgrade to 7.0.57 and continue with your testing.
> There have been a lot of fixes in the intervening years between the two
> releases. I haven't read the changelog, but I wouldn't be surprised if
> something has been fixed between then and now.
>
> >>> I tried to turn of one Tomcat manually to check if the clustering
> >>> is implemented or not. But the failover doesn't work.
> >>
> >> Are you expecting that your sessions will be available to any node, or
> >> are you using sticky sessions? The term "cluster" usually indicates
> >> that you want distributable sessions that will be available on any
> >> cluster node, but I just wanted to confirm that's what you want.
> >
> > _Two things_
> >
> > First i wanted to test if the first tomcat fails the second Tomcat has
> > to start. I was not testing the session sticky session in the first
> > scenario.
>
> Just to be clear: both Tomcats have to be running at the same time. When
> one fails, the other can't start-up and take-over. Instead, they share
> the load at first and then if one node goes down, fail-over to the other
> (still running node) occurs.
>
> > Secondly i want to implement Sticky session. I guess Apache (mod_jk) has
> > to be implemented in front of the Tomcat servers for sticky session.
>
> If you want to use load-balancing *at all*, you'll need a network
> component out front. mod_jk is not necessary, and neither is Apache
> httpd: you can do this with other products as well.
>
> > We
> > have a hardware load balancer appliance, can we use this hardware load
> > balancer for load balancing without installing Apache (mod_jk). I don't
> > have the details of the hardware appliance that load balances the
> > Tomcats. (My Manager is on leave, so i have to wait until he returns)
>
> If you can rig the LB to use part of Tomcat's session id as a
> differentiating factor, then use the part of the session id after the
> period (.) to determine which "route" to use (set by the jvmRoute
> attribute of your <Engine> in server.xml). If that doesn't work, you can
> almost always use a LB's built-in session-stickiness, which typically
> uses HTTP cookies to maintain identification of the back-end instance.
>
> >> If you can afford to have users re-login and re-start a workflow when
> >> a node fails, using session-stickiness without any formal "clustering"
> >> is a much easier configuration and will yield higher performance from
> >> the whole setup.
> >
> > What is the process if the users can afford re-login and restart a
> > workflow ? Could you please guide me how to implement it. Does the code
> > which i sent will be helpful for this scenario.
>
> If a user makes a request to a resource which requires authentication,
> Tomcat will store the user's request, challenge the user for
> authentication and then re-process the original request
> post-authentication. So, as long as your workflows contain enough
> information in the HTTP request to resume those workflows after a
> failed-over login, then there's nothing else for you to do.
>
> > Anyway...
> >
> > Please let me know how to confirm if the servers are clustered or
> > not. Appreciate your help.
> >
> > <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> > channelSendOptions="6">
>
>
> If you have set up <Cluster>, then you are using clustering. I was just
> saying that if you don't *need* clustering, maybe you could abandon
> trying to get it set up in the first place. Session distribution is
> quite chatty, from a network perspective.
>
> >> My first question is: do you have <distributable/> set in your
> >> web.xml? That's the easiest thing to overlook because it's not in
> >> server.xml and will silently prevent your sessions from being
> >> distributed to the rest of the cluster.
> >
> > No, i haven't edited the web.xml file with the <distributable/> set, the
> > "Tomcat - How to cluster" web page has that information, but they don't
> > clearly explain the procedure where to add it in the web.xml file. Do i
> > need to just add "<distributable/>" parameter at the end by opening it
> > in the notepad. Please let me know.
>
> Take a look at the XML Schema for web.xml and you should be able to find
> the place where to put <distributable/>.
>
> -chris
>
>