You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Fady Haikal <fa...@gmail.com> on 2017/02/06 06:52:06 UTC
Operation has timed out
Guys, we are facing the below errors in Tomcat cluster, please advise
06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
process request in NioReceiver
java.io.IOException: A non-blocking socket operation could not be
completed immediately
at sun.nio.ch.SocketDispatcher.close0(Native Method)
at sun.nio.ch.SocketDispatcher.close(Unknown Source)
at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.select(Unknown Source)
at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
at java.lang.Thread.run(Unknown Source)
Regards,
Fady
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Ashwin,
I tried changing the port but unfortunately i'm still facing the same error
On Wed, Feb 8, 2017 at 4:44 PM, ashwin rajput <as...@gmail.com> wrote:
> Hi Fady,
>
> can you please try changing default Multicast port to something like :
> From port="45564"
> to port="45562" or port="45560".
>
> let me know the outcome will try to fix it and please make sure if you have
> firewall enable on system please disable it first and then give a try
> otherwise you have to change firewall configuration to allow
> multicast traffic.
>
> Regards,
> Ashwin
>
> On Wed, Feb 8, 2017 at 12:56 PM, Fady Haikal <fa...@gmail.com> wrote:
>
>> Hi All,
>> As u know we are using 224.0.0.4 multicast IP for tomcat cluster
>> (Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the
>> multicast IP i get reply from 10.114.43.51 IP!
>>
>> Also i have executed the nslookup for 224.0.0.4 i get the DC IP
>> (10.114.43.7) and mcast.net domain:
>>
>> C:\Users\Administrator>nslookup 224.0.0.4
>> Server: hq-dc02.albaraka.com.sd
>> Address: 10.114.43.7
>>
>> Name: dvmrp.mcast.net
>> Address: 224.0.0.4
>>
>> On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikal <fa...@gmail.com> wrote:
>> > Ashwin,
>> > I'm using the below configuration, please let me know how i can check
>> > if i'm using a unique multicast address and port
>> >
>> >
>> > <Channel className="org.apache.catalina.tribes.group.
>> GroupChannel">
>> > <Membership
>> > className="org.apache.catalina.tribes.membership.McastService"
>> > address="228.0.0.4"
>> > port="45564"
>> > frequency="500"
>> > dropTime="9000"/>
>> > <Receiver
>> > className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> > address="auto"
>> > port="4000"
>> > autoBind="100"
>> > selectorTimeout="5000"
>> > maxThreads="6"/>
>> >
>> > On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput <as...@gmail.com>
>> wrote:
>> >> I am not sure if anyone has verified below.
>> >>
>> >> Have you verifyed clustering is using unique multicast address and port.
>> >> Cluster multicast address should be unique and not used by any other
>> >> cluster.
>> >>
>> >> Regards,
>> >> Ashwin
>> >> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" <aw...@ice-sa.com>
>> wrote:
>> >>
>> >>> On 07.02.2017 17:20, Fady Haikal wrote:
>> >>>
>> >>>> Christopher,
>> >>>> For the first time
>> >>>>
>> >>>
>> >>> @Christopher : just to make sure you got that bit, buried below : the
>> >>> actual replication seems to work fine. The problem is only these
>> >>> "unsuccesful ping" messages in the log, which fill the log, and which
>> so
>> >>> far nobody has managed to find an explanation for.
>> >>>
>> >>>
>> >>>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
>> >>>> <ch...@christopherschultz.net> wrote:
>> >>>>
>> >>>>> -----BEGIN PGP SIGNED MESSAGE-----
>> >>>>> Hash: SHA256
>> >>>>>
>> >>>>> Fady,
>> >>>>>
>> >>>>> On 2/7/17 10:53 AM, Fady Haikal wrote:
>> >>>>>
>> >>>>>> Process PID Protocol local address local port
>> >>>>>> Remote Address State Tomcat8.exe 8160 TCP
>> >>>>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>> >>>>>>
>> >>>>>
>> >>>>> Stupid question: was this working in the past, and it stopped
>> working?
>> >>>>> Or are you trying to get this working for the first time?
>> >>>>>
>> >>>>> - -chris
>> >>>>>
>> >>>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Yes there is a ESTABLISHED connection, the replication of
>> >>>>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>> >>>>>>> also faced this error on the log file
>> >>>>>>>
>> >>>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>> >>>>>>> <aw...@ice-sa.com> wrote:
>> >>>>>>>
>> >>>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>> >>>>>>>>> network monitoring at the level of IP and Port and i didnt
>> >>>>>>>>> see any disconnection,
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
>> >>>>>>>> if you use the Windows "netstat" program, for example as
>> >>>>>>>> netstat -aon -p TCP you should see a list of connections in the
>> >>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>> >>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
>> >>>>>>>> node, this port :4000 should be in the "local" column, in
>> >>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>> >>>>>>>> they agree.)
>> >>>>>>>>
>> >>>>>>>> Is that the case ?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> and yes i'm sure that no firewall is enabled.
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> I saw some strange think on the server that I have tried to
>> >>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>> >>>>>>>>> different IPs in the network, i don't know why and how i get
>> >>>>>>>>> those IPs, after checking with the network team they told me
>> >>>>>>>>> that those IPs are related to the SAN storage taking into
>> >>>>>>>>> consideration that the Tomcat servers are not connected in
>> >>>>>>>>> anyway to that SUN storage.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>> >>>>>>>>> <aw...@ice-sa.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Hi.
>> >>>>>>>>>>
>> >>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>> >>>>>>>>>>
>> >>>>>>>>>> I know nothing of Tribes, but the on-line documentation
>> >>>>>>>>>> seems to say that the communication happens over TCP and
>> >>>>>>>>>> that the protocol used is not encrypted. Fady previously
>> >>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>> >>>>>>>>>> nodes, and that is the base for him mentioning that "there
>> >>>>>>>>>> is no network disconnection" between the nodes.
>> >>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>> >>>>>>>>>> times out without getting a response fom the target node.
>> >>>>>>>>>> There is evidently a contradiction there. So this could
>> >>>>>>>>>> still be some kind of network issue.
>> >>>>>>>>>>
>> >>>>>>>>>> Considering that the protocol command for this "ping"
>> >>>>>>>>>> should be known by someone here, would it not be possible
>> >>>>>>>>>> to imagine a little program in some scripting language (or
>> >>>>>>>>>> even java, God forbid), which would open a TCP channel with
>> >>>>>>>>>> the target node IP/port, send such a "ping" message, wait
>> >>>>>>>>>> for a response and report the result ? That would at least
>> >>>>>>>>>> confirm/deny that the problem is with the network.
>> >>>>>>>>>>
>> >>>>>>>>>> The log below does not for example say if the error happens
>> >>>>>>>>>> when opening the TCP communication channel, or after
>> >>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>> >>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>> >>>>>>>>>> if Fady tried this). Maybe tribes also already contains
>> >>>>>>>>>> some löw-level debugging options ? wireshark maybe another
>> >>>>>>>>>> option, but it has quite a learning curve. And this is on
>> >>>>>>>>>> Windows.
>> >>>>>>>>>>
>> >>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>> >>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>> >>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>> >>>>>>>>>> "network policies" there which can have wide-ranging
>> >>>>>>>>>> side-effects.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>> >>>>>>>>>>> not able to replicate the key from node to node, please
>> >>>>>>>>>>> find below the error, taking into consideration that
>> >>>>>>>>>>> there is no network disconnection between 2 nodes
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>> >>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.
>> publishEntryIn
>> >>>>>>>>>>>
>> >>>>>>>>>> fo
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Unable to replicate backup
>> >>>>>
>> >>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>> >>>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[
>> tcp://{10
>> >>>>>>>>>>>
>> >>>>>>>>>> ,
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>> >>>>>
>> >>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>> >>>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>> >>>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>> >>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>> >>>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>> >>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>> >>>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>> >>>>>>>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.transport.nio.
>> ParallelNioSender.sendMe
>> >>>>>>>>>>>
>> >>>>>>>>>> ssage(ParallelNioSender.java:108)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.transport.nio.
>> PooledParallelSender.sen
>> >>>>>>>>>>>
>> >>>>>>>>>> dMessage(PooledParallelSender.java:48)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.transport.
>> ReplicationTransmitter.sendM
>> >>>>>>>>>>>
>> >>>>>>>>>> essage(ReplicationTransmitter.java:54)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.
>> sendMessage(C
>> >>>>>>>>>>>
>> >>>>>>>>>> hannelCoordinator.java:82)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessa
>> >>>>>>>>>>>
>> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.
>> MessageDispatchInte
>> >>>>>>>>>>>
>> >>>>>>>>>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessa
>> >>>>>>>>>>>
>> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.
>> TcpFailureDetector.
>> >>>>>>>>>>>
>> >>>>>>>>>> sendMessage(TcpFailureDetector.java:93)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessa
>> >>>>>>>>>>>
>> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.j
>> >>>>>>>>>>>
>> >>>>>>>>>> ava:233)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.j
>> >>>>>>>>>>>
>> >>>>>>>>>> ava:186)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.
>> publishEntryIn
>> >>>>>>>>>>>
>> >>>>>>>>>> fo(LazyReplicatedMap.java:170)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> put(Abstra
>> >>>>>>>>>>>
>> >>>>>>>>>> ctReplicatedMap.java:1040)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> put(Abstra
>> >>>>>>>>>>>
>> >>>>>>>>>> ctReplicatedMap.java:1024)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.
>> java
>> >>>>> :647)
>> >>>>>
>> >>>>>> at
>> >>>>>>>>>>>
>> >>>>>>>>>>> org.apache.catalina.session.StandardSession.setId(
>> StandardSession
>> >>>>>>>>>>>
>> >>>>>>>>>> .java:374)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
>> >>>>>>>>>>>
>> >>>>>>>>>> va:279)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(
>> ManagerBase
>> >>>>>>>>>>>
>> >>>>>>>>>> .java:708)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at org.apache.catalina.connector.
>> Request.doGetSession(Request.j
>> >>>>> ava:2936)
>> >>>>>
>> >>>>>> at
>> >>>>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.
>> java:226
>> >>>>>>>>>>>
>> >>>>>>>>>> 0)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(
>> RequestFac
>> >>>>>>>>>>>
>> >>>>>>>>>> ade.java:895)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.
>> getSession(HttpServl
>> >>>>>>>>>>>
>> >>>>>>>>>> etRequestWrapper.java:231)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.
>> getSession(Applic
>> >>>>>>>>>>>
>> >>>>>>>>>> ationHttpRequest.java:568)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.
>> getSession(Applic
>> >>>>>>>>>>>
>> >>>>>>>>>> ationHttpRequest.java:513)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(
>> PageContextI
>> >>>>>>>>>>>
>> >>>>>>>>>> mpl.java:137)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.
>> internalGetPageContext(J
>> >>>>>>>>>>>
>> >>>>>>>>>> spFactoryImpl.java:109)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(
>> JspFactor
>> >>>>>>>>>>>
>> >>>>>>>>>> yImpl.java:60)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
>> >>>>>>>>>>>
>> >>>>>>>>>> :100)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.
>> >>>>> java:70)
>> >>>>>
>> >>>>>> at
>> >>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(
>> JspServletWra
>> >>>>>>>>>>>
>> >>>>>>>>>> pper.java:438)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
>> >>>>>>>>>>>
>> >>>>>>>>>> va:396)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.
>> java
>> >>>>> :340)
>> >>>>>
>> >>>>>> at
>> >>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.
>> internalDoFilter(
>> >>>>>>>>>>>
>> >>>>>>>>>> ApplicationFilterChain.java:291)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.
>> doFilter(Applicat
>> >>>>>>>>>>>
>> >>>>>>>>>> ionFilterChain.java:206)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(
>> Application
>> >>>>>>>>>>>
>> >>>>>>>>>> Dispatcher.java:719)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.
>> processRequest(App
>> >>>>>>>>>>>
>> >>>>>>>>>> licationDispatcher.java:467)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.
>> doForward(Applicat
>> >>>>>>>>>>>
>> >>>>>>>>>> ionDispatcher.java:390)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(
>> Applicatio
>> >>>>>>>>>>>
>> >>>>>>>>>> nDispatcher.java:317)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(
>> StandardHostVal
>> >>>>>>>>>>>
>> >>>>>>>>>> ve.java:445)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.status(
>> StandardHostVal
>> >>>>>>>>>>>
>> >>>>>>>>>> ve.java:304)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(
>> StandardHostVal
>> >>>>>>>>>>>
>> >>>>>>>>>> ve.java:181)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.
>> invoke(JvmRout
>> >>>>>>>>>>>
>> >>>>>>>>>> eBinderValve.java:194)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(
>> ReplicationVal
>> >>>>>>>>>>>
>> >>>>>>>>>> ve.java:318)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(
>> ErrorReportVal
>> >>>>>>>>>>>
>> >>>>>>>>>> ve.java:79)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.
>> invoke(Stuck
>> >>>>>>>>>>>
>> >>>>>>>>>> ThreadDetectionValve.java:208)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(
>> Abstract
>> >>>>>>>>>>>
>> >>>>>>>>>> AccessLogValve.java:616)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(
>> StandardEngin
>> >>>>>>>>>>>
>> >>>>>>>>>> eValve.java:88)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(
>> CoyoteAdapter
>> >>>>>>>>>>>
>> >>>>>>>>>> .java:521)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.
>> process(Abstract
>> >>>>>>>>>>>
>> >>>>>>>>>> Http11Processor.java:1096)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.coyote.AbstractProtocol$
>> AbstractConnectionHandler.proc
>> >>>>>>>>>>>
>> >>>>>>>>>> ess(AbstractProtocol.java:674)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
>> doRun(NioE
>> >>>>>>>>>>>
>> >>>>>>>>>> ndpoint.java:1500)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
>> run(NioEnd
>> >>>>>>>>>>>
>> >>>>>>>>>> point.java:1456)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> >>>>> Source)
>> >>>>>
>> >>>>>> at
>> >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> >>>>>>>>>>> Source) at
>> >>>>>>>>>>>
>> >>>>>>>>>>> org.apache.tomcat.util.threads.TaskThread$
>> WrappingRunnable.run(Ta
>> >>>>>>>>>>>
>> >>>>>>>>>> skThread.java:61)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>> >>>>>>>>>>> <pk...@airplus.com> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Fady,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Sorry for top posting.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> If I remember correctly, the Cluster Element goes into
>> >>>>>>>>>>>> the Container and not the Host. Plus I see in our
>> >>>>>>>>>>>> (working) case, a DeltaManager and a
>> >>>>>>>>>>>> JvmRouteSessionIDBinderListener
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> <Manager
>> >>>>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> expireSessionsOnShutdown="false"
>> >>>>>
>> >>>>>> notifyListenersOnReplication="true"/> ...
>> >>>>>>>>>>>> <ClusterListener
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> className="org.apache.catalina.ha.session.
>> JvmRouteSessionIDBinde
>> >>>>>>>>>>>>
>> >>>>>>>>>>> rListener"/>
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Besides this, only ports, limits and values are different.
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>> You may want to filter out the replication for static
>> >>>>>>>>>>>> resources as gifs jpg or css. <Valve
>> >>>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.
>> >>>>> css;.*\.txt;
>> >>>>> " />
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>> Best regards
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Peter
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> below is the server.xml configuration, as mentioened
>> >>>>>>>>>>>>> earlier the issue is related to the cluster
>> >>>>>>>>>>>>> configuration, and as per my research i can see that
>> >>>>>>>>>>>>> some users are facing the same issue but i didnt
>> >>>>>>>>>>>>> found the solution of it
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>> >>>>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>> >>>>>>>>>>>>> more contributor license agreements. See the NOTICE
>> >>>>>>>>>>>>> file distributed with this work for additional
>> >>>>>>>>>>>>> information regarding copyright ownership. The ASF
>> >>>>>>>>>>>>> licenses this file to You under the Apache License,
>> >>>>>>>>>>>>> Version 2.0 (the "License"); you may not use this
>> >>>>>>>>>>>>> file except in compliance with the License. You may
>> >>>>>>>>>>>>> obtain a copy of the License at
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Unless required by applicable law or agreed to in
>> >>>>>>>>>>>>> writing, software distributed under the License is
>> >>>>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>> >>>>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>> >>>>>>>>>>>>> implied. See the License for the specific language
>> >>>>>>>>>>>>> governing permissions and limitations under the
>> >>>>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>> >>>>>>>>>>>>> "Container", so you may not define subcomponents such
>> >>>>>>>>>>>>> as "Valves" at this level. Documentation at
>> >>>>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>> >>>>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>> >>>>>>>>>>>>> className="org.apache.catalina.startup.
>> VersionLoggerListener"
>> >>>>>>>>>>>>> /> <!-- Security listener. Documentation at
>> >>>>>>>>>>>>> /docs/config/listeners.html <Listener
>> >>>>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> />
>> >>>>>
>> >>>>>> --> <!--APR library loader. Documentation at
>> >>>>>>>>>>>>> /docs/apr.html --> <Listener
>> >>>>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> SSLEngine="on" />
>> >>>>>
>> >>>>>> <!-- Prevent memory leaks due to use of particular
>> >>>>>>>>>>>>> java/javax APIs--> <Listener
>> >>>>>>>>>>>>> className="org.apache.catalina.core.
>> JreMemoryLeakPreventionList
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> ener"
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> />
>> >>>>>
>> >>>>>> <Listener
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> className="org.apache.catalina.mbeans.
>> GlobalResourcesLifecycleL
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> istener"
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> />
>> >>>>>
>> >>>>>> <Listener
>> >>>>>>>>>>>>> className="org.apache.catalina.core.
>> ThreadLocalLeakPreventionLi
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> stener"
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> />
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>>> <!-- Global JNDI resources Documentation at
>> >>>>>>>>>>>>> /docs/jndi-resources-howto.html -->
>> >>>>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>> >>>>>>>>>>>>> that can also be used by UserDatabaseRealm to
>> >>>>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>> >>>>>>>>>>>>> auth="Container"
>> >>>>>>>>>>>>> type="org.apache.catalina.UserDatabase"
>> >>>>>>>>>>>>> description="User database that can be updated and
>> >>>>>>>>>>>>> saved"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> factory="org.apache.catalina.users.
>> MemoryUserDatabaseFactory"
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> pathname="conf/tomcat-users.xml" />
>> >>>>>
>> >>>>>> ...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by ashwin rajput <as...@gmail.com>.
Hi Fady,
can you please try changing default Multicast port to something like :
From port="45564"
to port="45562" or port="45560".
let me know the outcome will try to fix it and please make sure if you have
firewall enable on system please disable it first and then give a try
otherwise you have to change firewall configuration to allow
multicast traffic.
Regards,
Ashwin
On Wed, Feb 8, 2017 at 12:56 PM, Fady Haikal <fa...@gmail.com> wrote:
> Hi All,
> As u know we are using 224.0.0.4 multicast IP for tomcat cluster
> (Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the
> multicast IP i get reply from 10.114.43.51 IP!
>
> Also i have executed the nslookup for 224.0.0.4 i get the DC IP
> (10.114.43.7) and mcast.net domain:
>
> C:\Users\Administrator>nslookup 224.0.0.4
> Server: hq-dc02.albaraka.com.sd
> Address: 10.114.43.7
>
> Name: dvmrp.mcast.net
> Address: 224.0.0.4
>
> On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikal <fa...@gmail.com> wrote:
> > Ashwin,
> > I'm using the below configuration, please let me know how i can check
> > if i'm using a unique multicast address and port
> >
> >
> > <Channel className="org.apache.catalina.tribes.group.
> GroupChannel">
> > <Membership
> > className="org.apache.catalina.tribes.membership.McastService"
> > address="228.0.0.4"
> > port="45564"
> > frequency="500"
> > dropTime="9000"/>
> > <Receiver
> > className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> > address="auto"
> > port="4000"
> > autoBind="100"
> > selectorTimeout="5000"
> > maxThreads="6"/>
> >
> > On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput <as...@gmail.com>
> wrote:
> >> I am not sure if anyone has verified below.
> >>
> >> Have you verifyed clustering is using unique multicast address and port.
> >> Cluster multicast address should be unique and not used by any other
> >> cluster.
> >>
> >> Regards,
> >> Ashwin
> >> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" <aw...@ice-sa.com>
> wrote:
> >>
> >>> On 07.02.2017 17:20, Fady Haikal wrote:
> >>>
> >>>> Christopher,
> >>>> For the first time
> >>>>
> >>>
> >>> @Christopher : just to make sure you got that bit, buried below : the
> >>> actual replication seems to work fine. The problem is only these
> >>> "unsuccesful ping" messages in the log, which fill the log, and which
> so
> >>> far nobody has managed to find an explanation for.
> >>>
> >>>
> >>>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
> >>>> <ch...@christopherschultz.net> wrote:
> >>>>
> >>>>> -----BEGIN PGP SIGNED MESSAGE-----
> >>>>> Hash: SHA256
> >>>>>
> >>>>> Fady,
> >>>>>
> >>>>> On 2/7/17 10:53 AM, Fady Haikal wrote:
> >>>>>
> >>>>>> Process PID Protocol local address local port
> >>>>>> Remote Address State Tomcat8.exe 8160 TCP
> >>>>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
> >>>>>>
> >>>>>
> >>>>> Stupid question: was this working in the past, and it stopped
> working?
> >>>>> Or are you trying to get this working for the first time?
> >>>>>
> >>>>> - -chris
> >>>>>
> >>>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Yes there is a ESTABLISHED connection, the replication of
> >>>>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
> >>>>>>> also faced this error on the log file
> >>>>>>>
> >>>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
> >>>>>>> <aw...@ice-sa.com> wrote:
> >>>>>>>
> >>>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
> >>>>>>>>> network monitoring at the level of IP and Port and i didnt
> >>>>>>>>> see any disconnection,
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
> >>>>>>>> if you use the Windows "netstat" program, for example as
> >>>>>>>> netstat -aon -p TCP you should see a list of connections in the
> >>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
> >>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
> >>>>>>>> node, this port :4000 should be in the "local" column, in
> >>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
> >>>>>>>> they agree.)
> >>>>>>>>
> >>>>>>>> Is that the case ?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and yes i'm sure that no firewall is enabled.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I saw some strange think on the server that I have tried to
> >>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
> >>>>>>>>> different IPs in the network, i don't know why and how i get
> >>>>>>>>> those IPs, after checking with the network team they told me
> >>>>>>>>> that those IPs are related to the SAN storage taking into
> >>>>>>>>> consideration that the Tomcat servers are not connected in
> >>>>>>>>> anyway to that SUN storage.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
> >>>>>>>>> <aw...@ice-sa.com> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi.
> >>>>>>>>>>
> >>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
> >>>>>>>>>>
> >>>>>>>>>> I know nothing of Tribes, but the on-line documentation
> >>>>>>>>>> seems to say that the communication happens over TCP and
> >>>>>>>>>> that the protocol used is not encrypted. Fady previously
> >>>>>>>>>> tried a standard "ping" and a "telnet" between the two
> >>>>>>>>>> nodes, and that is the base for him mentioning that "there
> >>>>>>>>>> is no network disconnection" between the nodes.
> >>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
> >>>>>>>>>> times out without getting a response fom the target node.
> >>>>>>>>>> There is evidently a contradiction there. So this could
> >>>>>>>>>> still be some kind of network issue.
> >>>>>>>>>>
> >>>>>>>>>> Considering that the protocol command for this "ping"
> >>>>>>>>>> should be known by someone here, would it not be possible
> >>>>>>>>>> to imagine a little program in some scripting language (or
> >>>>>>>>>> even java, God forbid), which would open a TCP channel with
> >>>>>>>>>> the target node IP/port, send such a "ping" message, wait
> >>>>>>>>>> for a response and report the result ? That would at least
> >>>>>>>>>> confirm/deny that the problem is with the network.
> >>>>>>>>>>
> >>>>>>>>>> The log below does not for example say if the error happens
> >>>>>>>>>> when opening the TCP communication channel, or after
> >>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
> >>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
> >>>>>>>>>> if Fady tried this). Maybe tribes also already contains
> >>>>>>>>>> some löw-level debugging options ? wireshark maybe another
> >>>>>>>>>> option, but it has quite a learning curve. And this is on
> >>>>>>>>>> Windows.
> >>>>>>>>>>
> >>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
> >>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
> >>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
> >>>>>>>>>> "network policies" there which can have wide-ranging
> >>>>>>>>>> side-effects.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
> >>>>>>>>>>> not able to replicate the key from node to node, please
> >>>>>>>>>>> find below the error, taking into consideration that
> >>>>>>>>>>> there is no network disconnection between 2 nodes
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
> >>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.
> publishEntryIn
> >>>>>>>>>>>
> >>>>>>>>>> fo
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Unable to replicate backup
> >>>>>
> >>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
> >>>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[
> tcp://{10
> >>>>>>>>>>>
> >>>>>>>>>> ,
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
> >>>>>
> >>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
> >>>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
> >>>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
> >>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
> >>>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
> >>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
> >>>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
> >>>>>>>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.transport.nio.
> ParallelNioSender.sendMe
> >>>>>>>>>>>
> >>>>>>>>>> ssage(ParallelNioSender.java:108)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.transport.nio.
> PooledParallelSender.sen
> >>>>>>>>>>>
> >>>>>>>>>> dMessage(PooledParallelSender.java:48)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.transport.
> ReplicationTransmitter.sendM
> >>>>>>>>>>>
> >>>>>>>>>> essage(ReplicationTransmitter.java:54)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.
> sendMessage(C
> >>>>>>>>>>>
> >>>>>>>>>> hannelCoordinator.java:82)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessa
> >>>>>>>>>>>
> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.
> MessageDispatchInte
> >>>>>>>>>>>
> >>>>>>>>>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessa
> >>>>>>>>>>>
> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.
> TcpFailureDetector.
> >>>>>>>>>>>
> >>>>>>>>>> sendMessage(TcpFailureDetector.java:93)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessa
> >>>>>>>>>>>
> >>>>>>>>>> ge(ChannelInterceptorBase.java:76)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.j
> >>>>>>>>>>>
> >>>>>>>>>> ava:233)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.j
> >>>>>>>>>>>
> >>>>>>>>>> ava:186)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.
> publishEntryIn
> >>>>>>>>>>>
> >>>>>>>>>> fo(LazyReplicatedMap.java:170)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> put(Abstra
> >>>>>>>>>>>
> >>>>>>>>>> ctReplicatedMap.java:1040)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> put(Abstra
> >>>>>>>>>>>
> >>>>>>>>>> ctReplicatedMap.java:1024)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.
> java
> >>>>> :647)
> >>>>>
> >>>>>> at
> >>>>>>>>>>>
> >>>>>>>>>>> org.apache.catalina.session.StandardSession.setId(
> StandardSession
> >>>>>>>>>>>
> >>>>>>>>>> .java:374)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
> >>>>>>>>>>>
> >>>>>>>>>> va:279)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(
> ManagerBase
> >>>>>>>>>>>
> >>>>>>>>>> .java:708)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at org.apache.catalina.connector.
> Request.doGetSession(Request.j
> >>>>> ava:2936)
> >>>>>
> >>>>>> at
> >>>>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.
> java:226
> >>>>>>>>>>>
> >>>>>>>>>> 0)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(
> RequestFac
> >>>>>>>>>>>
> >>>>>>>>>> ade.java:895)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.
> getSession(HttpServl
> >>>>>>>>>>>
> >>>>>>>>>> etRequestWrapper.java:231)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.
> getSession(Applic
> >>>>>>>>>>>
> >>>>>>>>>> ationHttpRequest.java:568)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.
> getSession(Applic
> >>>>>>>>>>>
> >>>>>>>>>> ationHttpRequest.java:513)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(
> PageContextI
> >>>>>>>>>>>
> >>>>>>>>>> mpl.java:137)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.
> internalGetPageContext(J
> >>>>>>>>>>>
> >>>>>>>>>> spFactoryImpl.java:109)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(
> JspFactor
> >>>>>>>>>>>
> >>>>>>>>>> yImpl.java:60)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
> >>>>>>>>>>>
> >>>>>>>>>> :100)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.
> >>>>> java:70)
> >>>>>
> >>>>>> at
> >>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWra
> >>>>>>>>>>>
> >>>>>>>>>> pper.java:438)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
> >>>>>>>>>>>
> >>>>>>>>>> va:396)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.
> java
> >>>>> :340)
> >>>>>
> >>>>>> at
> >>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.
> internalDoFilter(
> >>>>>>>>>>>
> >>>>>>>>>> ApplicationFilterChain.java:291)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.
> doFilter(Applicat
> >>>>>>>>>>>
> >>>>>>>>>> ionFilterChain.java:206)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(
> Application
> >>>>>>>>>>>
> >>>>>>>>>> Dispatcher.java:719)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.
> processRequest(App
> >>>>>>>>>>>
> >>>>>>>>>> licationDispatcher.java:467)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.
> doForward(Applicat
> >>>>>>>>>>>
> >>>>>>>>>> ionDispatcher.java:390)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(
> Applicatio
> >>>>>>>>>>>
> >>>>>>>>>> nDispatcher.java:317)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(
> StandardHostVal
> >>>>>>>>>>>
> >>>>>>>>>> ve.java:445)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.status(
> StandardHostVal
> >>>>>>>>>>>
> >>>>>>>>>> ve.java:304)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(
> StandardHostVal
> >>>>>>>>>>>
> >>>>>>>>>> ve.java:181)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.
> invoke(JvmRout
> >>>>>>>>>>>
> >>>>>>>>>> eBinderValve.java:194)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(
> ReplicationVal
> >>>>>>>>>>>
> >>>>>>>>>> ve.java:318)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(
> ErrorReportVal
> >>>>>>>>>>>
> >>>>>>>>>> ve.java:79)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.
> invoke(Stuck
> >>>>>>>>>>>
> >>>>>>>>>> ThreadDetectionValve.java:208)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(
> Abstract
> >>>>>>>>>>>
> >>>>>>>>>> AccessLogValve.java:616)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(
> StandardEngin
> >>>>>>>>>>>
> >>>>>>>>>> eValve.java:88)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(
> CoyoteAdapter
> >>>>>>>>>>>
> >>>>>>>>>> .java:521)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.
> process(Abstract
> >>>>>>>>>>>
> >>>>>>>>>> Http11Processor.java:1096)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.coyote.AbstractProtocol$
> AbstractConnectionHandler.proc
> >>>>>>>>>>>
> >>>>>>>>>> ess(AbstractProtocol.java:674)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> doRun(NioE
> >>>>>>>>>>>
> >>>>>>>>>> ndpoint.java:1500)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at
> >>>>>
> >>>>>>
> >>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.
> run(NioEnd
> >>>>>>>>>>>
> >>>>>>>>>> point.java:1456)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> >>>>> Source)
> >>>>>
> >>>>>> at
> >>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> >>>>>>>>>>> Source) at
> >>>>>>>>>>>
> >>>>>>>>>>> org.apache.tomcat.util.threads.TaskThread$
> WrappingRunnable.run(Ta
> >>>>>>>>>>>
> >>>>>>>>>> skThread.java:61)
> >>>>>
> >>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
> >>>>>
> >>>>>>
> >>>>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
> >>>>>>>>>>> <pk...@airplus.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Fady,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sorry for top posting.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If I remember correctly, the Cluster Element goes into
> >>>>>>>>>>>> the Container and not the Host. Plus I see in our
> >>>>>>>>>>>> (working) case, a DeltaManager and a
> >>>>>>>>>>>> JvmRouteSessionIDBinderListener
> >>>>>>>>>>>>
> >>>>>>>>>>>> <Manager
> >>>>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> expireSessionsOnShutdown="false"
> >>>>>
> >>>>>> notifyListenersOnReplication="true"/> ...
> >>>>>>>>>>>> <ClusterListener
> >>>>>>>>>>>>
> >>>>>>>>>>>> className="org.apache.catalina.ha.session.
> JvmRouteSessionIDBinde
> >>>>>>>>>>>>
> >>>>>>>>>>> rListener"/>
> >>>>>
> >>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Besides this, only ports, limits and values are different.
> >>>>>
> >>>>>>
> >>>>>>>>>>>> You may want to filter out the replication for static
> >>>>>>>>>>>> resources as gifs jpg or css. <Valve
> >>>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.
> >>>>> css;.*\.txt;
> >>>>> " />
> >>>>>
> >>>>>>
> >>>>>>>>>>>> Best regards
> >>>>>>>>>>>>
> >>>>>>>>>>>> Peter
> >>>>>>>>>>>>
> >>>>>>>>>>>> below is the server.xml configuration, as mentioened
> >>>>>>>>>>>>> earlier the issue is related to the cluster
> >>>>>>>>>>>>> configuration, and as per my research i can see that
> >>>>>>>>>>>>> some users are facing the same issue but i didnt
> >>>>>>>>>>>>> found the solution of it
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
> >>>>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
> >>>>>>>>>>>>> more contributor license agreements. See the NOTICE
> >>>>>>>>>>>>> file distributed with this work for additional
> >>>>>>>>>>>>> information regarding copyright ownership. The ASF
> >>>>>>>>>>>>> licenses this file to You under the Apache License,
> >>>>>>>>>>>>> Version 2.0 (the "License"); you may not use this
> >>>>>>>>>>>>> file except in compliance with the License. You may
> >>>>>>>>>>>>> obtain a copy of the License at
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Unless required by applicable law or agreed to in
> >>>>>>>>>>>>> writing, software distributed under the License is
> >>>>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
> >>>>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
> >>>>>>>>>>>>> implied. See the License for the specific language
> >>>>>>>>>>>>> governing permissions and limitations under the
> >>>>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
> >>>>>>>>>>>>> "Container", so you may not define subcomponents such
> >>>>>>>>>>>>> as "Valves" at this level. Documentation at
> >>>>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
> >>>>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
> >>>>>>>>>>>>> className="org.apache.catalina.startup.
> VersionLoggerListener"
> >>>>>>>>>>>>> /> <!-- Security listener. Documentation at
> >>>>>>>>>>>>> /docs/config/listeners.html <Listener
> >>>>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> />
> >>>>>
> >>>>>> --> <!--APR library loader. Documentation at
> >>>>>>>>>>>>> /docs/apr.html --> <Listener
> >>>>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> SSLEngine="on" />
> >>>>>
> >>>>>> <!-- Prevent memory leaks due to use of particular
> >>>>>>>>>>>>> java/javax APIs--> <Listener
> >>>>>>>>>>>>> className="org.apache.catalina.core.
> JreMemoryLeakPreventionList
> >>>>>>>>>>>>>
> >>>>>>>>>>>> ener"
> >>>>>
> >>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> />
> >>>>>
> >>>>>> <Listener
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> className="org.apache.catalina.mbeans.
> GlobalResourcesLifecycleL
> >>>>>>>>>>>>>
> >>>>>>>>>>>> istener"
> >>>>>
> >>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> />
> >>>>>
> >>>>>> <Listener
> >>>>>>>>>>>>> className="org.apache.catalina.core.
> ThreadLocalLeakPreventionLi
> >>>>>>>>>>>>>
> >>>>>>>>>>>> stener"
> >>>>>
> >>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> />
> >>>>>
> >>>>>>
> >>>>>>>>>>>>> <!-- Global JNDI resources Documentation at
> >>>>>>>>>>>>> /docs/jndi-resources-howto.html -->
> >>>>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
> >>>>>>>>>>>>> that can also be used by UserDatabaseRealm to
> >>>>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
> >>>>>>>>>>>>> auth="Container"
> >>>>>>>>>>>>> type="org.apache.catalina.UserDatabase"
> >>>>>>>>>>>>> description="User database that can be updated and
> >>>>>>>>>>>>> saved"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> factory="org.apache.catalina.users.
> MemoryUserDatabaseFactory"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> pathname="conf/tomcat-users.xml" />
> >>>>>
> >>>>>> ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Hi All,
As u know we are using 224.0.0.4 multicast IP for tomcat cluster
(Node1: 10.114.43.102 / Node2: 10.114.43.103) i was trying to ping the
multicast IP i get reply from 10.114.43.51 IP!
Also i have executed the nslookup for 224.0.0.4 i get the DC IP
(10.114.43.7) and mcast.net domain:
C:\Users\Administrator>nslookup 224.0.0.4
Server: hq-dc02.albaraka.com.sd
Address: 10.114.43.7
Name: dvmrp.mcast.net
Address: 224.0.0.4
On Wed, Feb 8, 2017 at 8:59 AM, Fady Haikal <fa...@gmail.com> wrote:
> Ashwin,
> I'm using the below configuration, please let me know how i can check
> if i'm using a unique multicast address and port
>
>
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="500"
> dropTime="9000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput <as...@gmail.com> wrote:
>> I am not sure if anyone has verified below.
>>
>> Have you verifyed clustering is using unique multicast address and port.
>> Cluster multicast address should be unique and not used by any other
>> cluster.
>>
>> Regards,
>> Ashwin
>> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" <aw...@ice-sa.com> wrote:
>>
>>> On 07.02.2017 17:20, Fady Haikal wrote:
>>>
>>>> Christopher,
>>>> For the first time
>>>>
>>>
>>> @Christopher : just to make sure you got that bit, buried below : the
>>> actual replication seems to work fine. The problem is only these
>>> "unsuccesful ping" messages in the log, which fill the log, and which so
>>> far nobody has managed to find an explanation for.
>>>
>>>
>>>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
>>>> <ch...@christopherschultz.net> wrote:
>>>>
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA256
>>>>>
>>>>> Fady,
>>>>>
>>>>> On 2/7/17 10:53 AM, Fady Haikal wrote:
>>>>>
>>>>>> Process PID Protocol local address local port
>>>>>> Remote Address State Tomcat8.exe 8160 TCP
>>>>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>>>>>>
>>>>>
>>>>> Stupid question: was this working in the past, and it stopped working?
>>>>> Or are you trying to get this working for the first time?
>>>>>
>>>>> - -chris
>>>>>
>>>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes there is a ESTABLISHED connection, the replication of
>>>>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>>>>>>> also faced this error on the log file
>>>>>>>
>>>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>>
>>>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>>>>>>> network monitoring at the level of IP and Port and i didnt
>>>>>>>>> see any disconnection,
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
>>>>>>>> if you use the Windows "netstat" program, for example as
>>>>>>>> netstat -aon -p TCP you should see a list of connections in the
>>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
>>>>>>>> node, this port :4000 should be in the "local" column, in
>>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>>>>>>> they agree.)
>>>>>>>>
>>>>>>>> Is that the case ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> and yes i'm sure that no firewall is enabled.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I saw some strange think on the server that I have tried to
>>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>>>>>> different IPs in the network, i don't know why and how i get
>>>>>>>>> those IPs, after checking with the network team they told me
>>>>>>>>> that those IPs are related to the SAN storage taking into
>>>>>>>>> consideration that the Tomcat servers are not connected in
>>>>>>>>> anyway to that SUN storage.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>>
>>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>>>>>
>>>>>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>>>>>> seems to say that the communication happens over TCP and
>>>>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>>>>> is no network disconnection" between the nodes.
>>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>>>>> times out without getting a response fom the target node.
>>>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>>>> still be some kind of network issue.
>>>>>>>>>>
>>>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>>>> for a response and report the result ? That would at least
>>>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>>>
>>>>>>>>>> The log below does not for example say if the error happens
>>>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>>>> Windows.
>>>>>>>>>>
>>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>>>> side-effects.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>>>>> find below the error, taking into consideration that
>>>>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>>>
>>>>>>>>>> fo
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Unable to replicate backup
>>>>>
>>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>>>>>>>>>>>
>>>>>>>>>> ,
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>>
>>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
>>>>>>>>>>>
>>>>>>>>>> ssage(ParallelNioSender.java:108)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
>>>>>>>>>>>
>>>>>>>>>> dMessage(PooledParallelSender.java:48)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
>>>>>>>>>>>
>>>>>>>>>> essage(ReplicationTransmitter.java:54)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
>>>>>>>>>>>
>>>>>>>>>> hannelCoordinator.java:82)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>>
>>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
>>>>>>>>>>>
>>>>>>>>>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>>
>>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>>>>>>>>>>>
>>>>>>>>>> sendMessage(TcpFailureDetector.java:93)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>>
>>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>>>
>>>>>>>>>> ava:233)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>>>
>>>>>>>>>> ava:186)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>>>
>>>>>>>>>> fo(LazyReplicatedMap.java:170)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>>>
>>>>>>>>>> ctReplicatedMap.java:1040)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>>>
>>>>>>>>>> ctReplicatedMap.java:1024)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java
>>>>> :647)
>>>>>
>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
>>>>>>>>>>>
>>>>>>>>>> .java:374)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
>>>>>>>>>>>
>>>>>>>>>> va:279)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
>>>>>>>>>>>
>>>>>>>>>> .java:708)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at org.apache.catalina.connector.Request.doGetSession(Request.j
>>>>> ava:2936)
>>>>>
>>>>>> at
>>>>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
>>>>>>>>>>>
>>>>>>>>>> 0)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
>>>>>>>>>>>
>>>>>>>>>> ade.java:895)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
>>>>>>>>>>>
>>>>>>>>>> etRequestWrapper.java:231)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>>>
>>>>>>>>>> ationHttpRequest.java:568)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>>>
>>>>>>>>>> ationHttpRequest.java:513)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
>>>>>>>>>>>
>>>>>>>>>> mpl.java:137)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
>>>>>>>>>>>
>>>>>>>>>> spFactoryImpl.java:109)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
>>>>>>>>>>>
>>>>>>>>>> yImpl.java:60)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
>>>>>>>>>>>
>>>>>>>>>> :100)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.
>>>>> java:70)
>>>>>
>>>>>> at
>>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
>>>>>>>>>>>
>>>>>>>>>> pper.java:438)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
>>>>>>>>>>>
>>>>>>>>>> va:396)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java
>>>>> :340)
>>>>>
>>>>>> at
>>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>>>>>>>>>>>
>>>>>>>>>> ApplicationFilterChain.java:291)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
>>>>>>>>>>>
>>>>>>>>>> ionFilterChain.java:206)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
>>>>>>>>>>>
>>>>>>>>>> Dispatcher.java:719)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
>>>>>>>>>>>
>>>>>>>>>> licationDispatcher.java:467)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
>>>>>>>>>>>
>>>>>>>>>> ionDispatcher.java:390)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
>>>>>>>>>>>
>>>>>>>>>> nDispatcher.java:317)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
>>>>>>>>>>>
>>>>>>>>>> ve.java:445)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
>>>>>>>>>>>
>>>>>>>>>> ve.java:304)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
>>>>>>>>>>>
>>>>>>>>>> ve.java:181)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
>>>>>>>>>>>
>>>>>>>>>> eBinderValve.java:194)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
>>>>>>>>>>>
>>>>>>>>>> ve.java:318)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
>>>>>>>>>>>
>>>>>>>>>> ve.java:79)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
>>>>>>>>>>>
>>>>>>>>>> ThreadDetectionValve.java:208)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
>>>>>>>>>>>
>>>>>>>>>> AccessLogValve.java:616)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
>>>>>>>>>>>
>>>>>>>>>> eValve.java:88)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
>>>>>>>>>>>
>>>>>>>>>> .java:521)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
>>>>>>>>>>>
>>>>>>>>>> Http11Processor.java:1096)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
>>>>>>>>>>>
>>>>>>>>>> ess(AbstractProtocol.java:674)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
>>>>>>>>>>>
>>>>>>>>>> ndpoint.java:1500)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>
>>>>>>
>>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
>>>>>>>>>>>
>>>>>>>>>> point.java:1456)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>>> Source)
>>>>>
>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>>>>>> Source) at
>>>>>>>>>>>
>>>>>>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
>>>>>>>>>>>
>>>>>>>>>> skThread.java:61)
>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>
>>>>>>
>>>>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>>>>>>> <pk...@airplus.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Fady,
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry for top posting.
>>>>>>>>>>>>
>>>>>>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>>>>>>> (working) case, a DeltaManager and a
>>>>>>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>>>>>>
>>>>>>>>>>>> <Manager
>>>>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> expireSessionsOnShutdown="false"
>>>>>
>>>>>> notifyListenersOnReplication="true"/> ...
>>>>>>>>>>>> <ClusterListener
>>>>>>>>>>>>
>>>>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
>>>>>>>>>>>>
>>>>>>>>>>> rListener"/>
>>>>>
>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Besides this, only ports, limits and values are different.
>>>>>
>>>>>>
>>>>>>>>>>>> You may want to filter out the replication for static
>>>>>>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.
>>>>> css;.*\.txt;
>>>>> " />
>>>>>
>>>>>>
>>>>>>>>>>>> Best regards
>>>>>>>>>>>>
>>>>>>>>>>>> Peter
>>>>>>>>>>>>
>>>>>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>>>>>>> earlier the issue is related to the cluster
>>>>>>>>>>>>> configuration, and as per my research i can see that
>>>>>>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>>>>>>> found the solution of it
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>>>>>>> file distributed with this work for additional
>>>>>>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>>>>>>> file except in compliance with the License. You may
>>>>>>>>>>>>> obtain a copy of the License at
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>>>>>>> writing, software distributed under the License is
>>>>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>>>>>> implied. See the License for the specific language
>>>>>>>>>>>>> governing permissions and limitations under the
>>>>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> />
>>>>>
>>>>>> --> <!--APR library loader. Documentation at
>>>>>>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> SSLEngine="on" />
>>>>>
>>>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>>>>>>> java/javax APIs--> <Listener
>>>>>>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
>>>>>>>>>>>>>
>>>>>>>>>>>> ener"
>>>>>
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> />
>>>>>
>>>>>> <Listener
>>>>>>>>>>>>>
>>>>>>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
>>>>>>>>>>>>>
>>>>>>>>>>>> istener"
>>>>>
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> />
>>>>>
>>>>>> <Listener
>>>>>>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
>>>>>>>>>>>>>
>>>>>>>>>>>> stener"
>>>>>
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> />
>>>>>
>>>>>>
>>>>>>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>>>>>>> auth="Container"
>>>>>>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>>>>>>> description="User database that can be updated and
>>>>>>>>>>>>> saved"
>>>>>>>>>>>>>
>>>>>>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> pathname="conf/tomcat-users.xml" />
>>>>>
>>>>>> ...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Ashwin,
I'm using the below configuration, please let me know how i can check
if i'm using a unique multicast address and port
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="9000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
On Wed, Feb 8, 2017 at 6:39 AM, ashwin rajput <as...@gmail.com> wrote:
> I am not sure if anyone has verified below.
>
> Have you verifyed clustering is using unique multicast address and port.
> Cluster multicast address should be unique and not used by any other
> cluster.
>
> Regards,
> Ashwin
> On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" <aw...@ice-sa.com> wrote:
>
>> On 07.02.2017 17:20, Fady Haikal wrote:
>>
>>> Christopher,
>>> For the first time
>>>
>>
>> @Christopher : just to make sure you got that bit, buried below : the
>> actual replication seems to work fine. The problem is only these
>> "unsuccesful ping" messages in the log, which fill the log, and which so
>> far nobody has managed to find an explanation for.
>>
>>
>>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
>>> <ch...@christopherschultz.net> wrote:
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA256
>>>>
>>>> Fady,
>>>>
>>>> On 2/7/17 10:53 AM, Fady Haikal wrote:
>>>>
>>>>> Process PID Protocol local address local port
>>>>> Remote Address State Tomcat8.exe 8160 TCP
>>>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>>>>>
>>>>
>>>> Stupid question: was this working in the past, and it stopped working?
>>>> Or are you trying to get this working for the first time?
>>>>
>>>> - -chris
>>>>
>>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes there is a ESTABLISHED connection, the replication of
>>>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>>>>>> also faced this error on the log file
>>>>>>
>>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>
>>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>>>>>> network monitoring at the level of IP and Port and i didnt
>>>>>>>> see any disconnection,
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> but did you see a *connection* ? I mean, on the pinging node,
>>>>>>> if you use the Windows "netstat" program, for example as
>>>>>>> netstat -aon -p TCP you should see a list of connections in the
>>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>>>>>>> target IP:4000 (in the "remote" column). And on the pinged
>>>>>>> node, this port :4000 should be in the "local" column, in
>>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>>>>>> they agree.)
>>>>>>>
>>>>>>> Is that the case ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> and yes i'm sure that no firewall is enabled.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I saw some strange think on the server that I have tried to
>>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>>>>> different IPs in the network, i don't know why and how i get
>>>>>>>> those IPs, after checking with the network team they told me
>>>>>>>> that those IPs are related to the SAN storage taking into
>>>>>>>> consideration that the Tomcat servers are not connected in
>>>>>>>> anyway to that SUN storage.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>>>>
>>>>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>>>>> seems to say that the communication happens over TCP and
>>>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>>>> is no network disconnection" between the nodes.
>>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>>>> times out without getting a response fom the target node.
>>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>>> still be some kind of network issue.
>>>>>>>>>
>>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>>> for a response and report the result ? That would at least
>>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>>
>>>>>>>>> The log below does not for example say if the error happens
>>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>>> Windows.
>>>>>>>>>
>>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>>> side-effects.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>>>> find below the error, taking into consideration that
>>>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>>
>>>>>>>>> fo
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> Unable to replicate backup
>>>>
>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>>>>>>>>>>
>>>>>>>>> ,
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>
>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
>>>>>>>>>>
>>>>>>>>> ssage(ParallelNioSender.java:108)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
>>>>>>>>>>
>>>>>>>>> dMessage(PooledParallelSender.java:48)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
>>>>>>>>>>
>>>>>>>>> essage(ReplicationTransmitter.java:54)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
>>>>>>>>>>
>>>>>>>>> hannelCoordinator.java:82)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>
>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
>>>>>>>>>>
>>>>>>>>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>
>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>>>>>>>>>>
>>>>>>>>> sendMessage(TcpFailureDetector.java:93)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>>
>>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>>
>>>>>>>>> ava:233)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>>
>>>>>>>>> ava:186)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>>
>>>>>>>>> fo(LazyReplicatedMap.java:170)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>>
>>>>>>>>> ctReplicatedMap.java:1040)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>>
>>>>>>>>> ctReplicatedMap.java:1024)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java
>>>> :647)
>>>>
>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
>>>>>>>>>>
>>>>>>>>> .java:374)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
>>>>>>>>>>
>>>>>>>>> va:279)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
>>>>>>>>>>
>>>>>>>>> .java:708)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at org.apache.catalina.connector.Request.doGetSession(Request.j
>>>> ava:2936)
>>>>
>>>>> at
>>>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
>>>>>>>>>>
>>>>>>>>> 0)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
>>>>>>>>>>
>>>>>>>>> ade.java:895)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
>>>>>>>>>>
>>>>>>>>> etRequestWrapper.java:231)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>>
>>>>>>>>> ationHttpRequest.java:568)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>>
>>>>>>>>> ationHttpRequest.java:513)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
>>>>>>>>>>
>>>>>>>>> mpl.java:137)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
>>>>>>>>>>
>>>>>>>>> spFactoryImpl.java:109)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
>>>>>>>>>>
>>>>>>>>> yImpl.java:60)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
>>>>>>>>>>
>>>>>>>>> :100)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.
>>>> java:70)
>>>>
>>>>> at
>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
>>>>>>>>>>
>>>>>>>>> pper.java:438)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
>>>>>>>>>>
>>>>>>>>> va:396)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java
>>>> :340)
>>>>
>>>>> at
>>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>>>>>>>>>>
>>>>>>>>> ApplicationFilterChain.java:291)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
>>>>>>>>>>
>>>>>>>>> ionFilterChain.java:206)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
>>>>>>>>>>
>>>>>>>>> Dispatcher.java:719)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
>>>>>>>>>>
>>>>>>>>> licationDispatcher.java:467)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
>>>>>>>>>>
>>>>>>>>> ionDispatcher.java:390)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
>>>>>>>>>>
>>>>>>>>> nDispatcher.java:317)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
>>>>>>>>>>
>>>>>>>>> ve.java:445)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
>>>>>>>>>>
>>>>>>>>> ve.java:304)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
>>>>>>>>>>
>>>>>>>>> ve.java:181)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
>>>>>>>>>>
>>>>>>>>> eBinderValve.java:194)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
>>>>>>>>>>
>>>>>>>>> ve.java:318)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
>>>>>>>>>>
>>>>>>>>> ve.java:79)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
>>>>>>>>>>
>>>>>>>>> ThreadDetectionValve.java:208)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
>>>>>>>>>>
>>>>>>>>> AccessLogValve.java:616)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
>>>>>>>>>>
>>>>>>>>> eValve.java:88)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
>>>>>>>>>>
>>>>>>>>> .java:521)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
>>>>>>>>>>
>>>>>>>>> Http11Processor.java:1096)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
>>>>>>>>>>
>>>>>>>>> ess(AbstractProtocol.java:674)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
>>>>>>>>>>
>>>>>>>>> ndpoint.java:1500)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at
>>>>
>>>>>
>>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
>>>>>>>>>>
>>>>>>>>> point.java:1456)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>>> Source)
>>>>
>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>>>>> Source) at
>>>>>>>>>>
>>>>>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
>>>>>>>>>>
>>>>>>>>> skThread.java:61)
>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>
>>>>>
>>>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>>>>>> <pk...@airplus.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Fady,
>>>>>>>>>>>
>>>>>>>>>>> Sorry for top posting.
>>>>>>>>>>>
>>>>>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>>>>>> (working) case, a DeltaManager and a
>>>>>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>>>>>
>>>>>>>>>>> <Manager
>>>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> expireSessionsOnShutdown="false"
>>>>
>>>>> notifyListenersOnReplication="true"/> ...
>>>>>>>>>>> <ClusterListener
>>>>>>>>>>>
>>>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
>>>>>>>>>>>
>>>>>>>>>> rListener"/>
>>>>
>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Besides this, only ports, limits and values are different.
>>>>
>>>>>
>>>>>>>>>>> You may want to filter out the replication for static
>>>>>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.
>>>> css;.*\.txt;
>>>> " />
>>>>
>>>>>
>>>>>>>>>>> Best regards
>>>>>>>>>>>
>>>>>>>>>>> Peter
>>>>>>>>>>>
>>>>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>>>>>> earlier the issue is related to the cluster
>>>>>>>>>>>> configuration, and as per my research i can see that
>>>>>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>>>>>> found the solution of it
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>>>>>> file distributed with this work for additional
>>>>>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>>>>>> file except in compliance with the License. You may
>>>>>>>>>>>> obtain a copy of the License at
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>>>>
>>>>>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>>>>>> writing, software distributed under the License is
>>>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>>>>> implied. See the License for the specific language
>>>>>>>>>>>> governing permissions and limitations under the
>>>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> />
>>>>
>>>>> --> <!--APR library loader. Documentation at
>>>>>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> SSLEngine="on" />
>>>>
>>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>>>>>> java/javax APIs--> <Listener
>>>>>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
>>>>>>>>>>>>
>>>>>>>>>>> ener"
>>>>
>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> />
>>>>
>>>>> <Listener
>>>>>>>>>>>>
>>>>>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
>>>>>>>>>>>>
>>>>>>>>>>> istener"
>>>>
>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> />
>>>>
>>>>> <Listener
>>>>>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
>>>>>>>>>>>>
>>>>>>>>>>> stener"
>>>>
>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> />
>>>>
>>>>>
>>>>>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>>>>>> auth="Container"
>>>>>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>>>>>> description="User database that can be updated and
>>>>>>>>>>>> saved"
>>>>>>>>>>>>
>>>>>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> pathname="conf/tomcat-users.xml" />
>>>>
>>>>> ...
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by ashwin rajput <as...@gmail.com>.
I am not sure if anyone has verified below.
Have you verifyed clustering is using unique multicast address and port.
Cluster multicast address should be unique and not used by any other
cluster.
Regards,
Ashwin
On 07-Feb-2017 10:38 pm, "André Warnier (tomcat)" <aw...@ice-sa.com> wrote:
> On 07.02.2017 17:20, Fady Haikal wrote:
>
>> Christopher,
>> For the first time
>>
>
> @Christopher : just to make sure you got that bit, buried below : the
> actual replication seems to work fine. The problem is only these
> "unsuccesful ping" messages in the log, which fill the log, and which so
> far nobody has managed to find an explanation for.
>
>
>> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
>> <ch...@christopherschultz.net> wrote:
>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> Fady,
>>>
>>> On 2/7/17 10:53 AM, Fady Haikal wrote:
>>>
>>>> Process PID Protocol local address local port
>>>> Remote Address State Tomcat8.exe 8160 TCP
>>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>>>>
>>>
>>> Stupid question: was this working in the past, and it stopped working?
>>> Or are you trying to get this working for the first time?
>>>
>>> - -chris
>>>
>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Yes there is a ESTABLISHED connection, the replication of
>>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>>>>> also faced this error on the log file
>>>>>
>>>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>>>>> <aw...@ice-sa.com> wrote:
>>>>>
>>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>>>>> network monitoring at the level of IP and Port and i didnt
>>>>>>> see any disconnection,
>>>>>>>
>>>>>>
>>>>>>
>>>>>> but did you see a *connection* ? I mean, on the pinging node,
>>>>>> if you use the Windows "netstat" program, for example as
>>>>>> netstat -aon -p TCP you should see a list of connections in the
>>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>>>>>> target IP:4000 (in the "remote" column). And on the pinged
>>>>>> node, this port :4000 should be in the "local" column, in
>>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>>>>> they agree.)
>>>>>>
>>>>>> Is that the case ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> and yes i'm sure that no firewall is enabled.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I saw some strange think on the server that I have tried to
>>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>>>> different IPs in the network, i don't know why and how i get
>>>>>>> those IPs, after checking with the network team they told me
>>>>>>> that those IPs are related to the SAN storage taking into
>>>>>>> consideration that the Tomcat servers are not connected in
>>>>>>> anyway to that SUN storage.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi.
>>>>>>>>
>>>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>>>
>>>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>>>> seems to say that the communication happens over TCP and
>>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>>> is no network disconnection" between the nodes.
>>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>>> times out without getting a response fom the target node.
>>>>>>>> There is evidently a contradiction there. So this could
>>>>>>>> still be some kind of network issue.
>>>>>>>>
>>>>>>>> Considering that the protocol command for this "ping"
>>>>>>>> should be known by someone here, would it not be possible
>>>>>>>> to imagine a little program in some scripting language (or
>>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>>> for a response and report the result ? That would at least
>>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>>
>>>>>>>> The log below does not for example say if the error happens
>>>>>>>> when opening the TCP communication channel, or after
>>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>>> Windows.
>>>>>>>>
>>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>>> "network policies" there which can have wide-ranging
>>>>>>>> side-effects.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>>> find below the error, taking into consideration that
>>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>
>>>>>>>> fo
>>>
>>>>
>>>>>>>>>
>>>>>>>>> Unable to replicate backup
>>>
>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>>>>>>>>>
>>>>>>>> ,
>>>
>>>>
>>>>>>>>>
>>>>>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>
>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
>>>>>>>>>
>>>>>>>> ssage(ParallelNioSender.java:108)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
>>>>>>>>>
>>>>>>>> dMessage(PooledParallelSender.java:48)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
>>>>>>>>>
>>>>>>>> essage(ReplicationTransmitter.java:54)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
>>>>>>>>>
>>>>>>>> hannelCoordinator.java:82)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>
>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
>>>>>>>>>
>>>>>>>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>
>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>>>>>>>>>
>>>>>>>> sendMessage(TcpFailureDetector.java:93)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>>>>>>>>>
>>>>>>>> ge(ChannelInterceptorBase.java:76)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>
>>>>>>>> ava:233)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>>>>>>>>>
>>>>>>>> ava:186)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>>>>>>>>>
>>>>>>>> fo(LazyReplicatedMap.java:170)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>
>>>>>>>> ctReplicatedMap.java:1040)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>>>>>>>>>
>>>>>>>> ctReplicatedMap.java:1024)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java
>>> :647)
>>>
>>>> at
>>>>>>>>>
>>>>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
>>>>>>>>>
>>>>>>>> .java:374)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
>>>>>>>>>
>>>>>>>> va:279)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
>>>>>>>>>
>>>>>>>> .java:708)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at org.apache.catalina.connector.Request.doGetSession(Request.j
>>> ava:2936)
>>>
>>>> at
>>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
>>>>>>>>>
>>>>>>>> 0)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
>>>>>>>>>
>>>>>>>> ade.java:895)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
>>>>>>>>>
>>>>>>>> etRequestWrapper.java:231)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>
>>>>>>>> ationHttpRequest.java:568)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>>>>>>>>>
>>>>>>>> ationHttpRequest.java:513)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
>>>>>>>>>
>>>>>>>> mpl.java:137)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
>>>>>>>>>
>>>>>>>> spFactoryImpl.java:109)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
>>>>>>>>>
>>>>>>>> yImpl.java:60)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
>>>>>>>>>
>>>>>>>> :100)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.
>>> java:70)
>>>
>>>> at
>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
>>>>>>>>>
>>>>>>>> pper.java:438)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
>>>>>>>>>
>>>>>>>> va:396)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java
>>> :340)
>>>
>>>> at
>>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>>>>>>>>>
>>>>>>>> ApplicationFilterChain.java:291)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
>>>>>>>>>
>>>>>>>> ionFilterChain.java:206)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
>>>>>>>>>
>>>>>>>> Dispatcher.java:719)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
>>>>>>>>>
>>>>>>>> licationDispatcher.java:467)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
>>>>>>>>>
>>>>>>>> ionDispatcher.java:390)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
>>>>>>>>>
>>>>>>>> nDispatcher.java:317)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
>>>>>>>>>
>>>>>>>> ve.java:445)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
>>>>>>>>>
>>>>>>>> ve.java:304)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
>>>>>>>>>
>>>>>>>> ve.java:181)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
>>>>>>>>>
>>>>>>>> eBinderValve.java:194)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
>>>>>>>>>
>>>>>>>> ve.java:318)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
>>>>>>>>>
>>>>>>>> ve.java:79)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
>>>>>>>>>
>>>>>>>> ThreadDetectionValve.java:208)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
>>>>>>>>>
>>>>>>>> AccessLogValve.java:616)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
>>>>>>>>>
>>>>>>>> eValve.java:88)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
>>>>>>>>>
>>>>>>>> .java:521)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
>>>>>>>>>
>>>>>>>> Http11Processor.java:1096)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
>>>>>>>>>
>>>>>>>> ess(AbstractProtocol.java:674)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
>>>>>>>>>
>>>>>>>> ndpoint.java:1500)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at
>>>
>>>>
>>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
>>>>>>>>>
>>>>>>>> point.java:1456)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>>> Source)
>>>
>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>>>> Source) at
>>>>>>>>>
>>>>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
>>>>>>>>>
>>>>>>>> skThread.java:61)
>>>
>>>>
>>>>>>>>>
>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>
>>>>
>>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>>>>> <pk...@airplus.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Fady,
>>>>>>>>>>
>>>>>>>>>> Sorry for top posting.
>>>>>>>>>>
>>>>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>>>>> (working) case, a DeltaManager and a
>>>>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>>>>
>>>>>>>>>> <Manager
>>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> expireSessionsOnShutdown="false"
>>>
>>>> notifyListenersOnReplication="true"/> ...
>>>>>>>>>> <ClusterListener
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
>>>>>>>>>>
>>>>>>>>> rListener"/>
>>>
>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Besides this, only ports, limits and values are different.
>>>
>>>>
>>>>>>>>>> You may want to filter out the replication for static
>>>>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.
>>> css;.*\.txt;
>>> " />
>>>
>>>>
>>>>>>>>>> Best regards
>>>>>>>>>>
>>>>>>>>>> Peter
>>>>>>>>>>
>>>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>>>>> earlier the issue is related to the cluster
>>>>>>>>>>> configuration, and as per my research i can see that
>>>>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>>>>> found the solution of it
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>>>>> file distributed with this work for additional
>>>>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>>>>> file except in compliance with the License. You may
>>>>>>>>>>> obtain a copy of the License at
>>>>>>>>>>>
>>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>>>
>>>>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>>>>> writing, software distributed under the License is
>>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>>>> implied. See the License for the specific language
>>>>>>>>>>> governing permissions and limitations under the
>>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> />
>>>
>>>> --> <!--APR library loader. Documentation at
>>>>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> SSLEngine="on" />
>>>
>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>>>>> java/javax APIs--> <Listener
>>>>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
>>>>>>>>>>>
>>>>>>>>>> ener"
>>>
>>>>
>>>>>>>>>>>
>>>>>>>>>>> />
>>>
>>>> <Listener
>>>>>>>>>>>
>>>>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
>>>>>>>>>>>
>>>>>>>>>> istener"
>>>
>>>>
>>>>>>>>>>>
>>>>>>>>>>> />
>>>
>>>> <Listener
>>>>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
>>>>>>>>>>>
>>>>>>>>>> stener"
>>>
>>>>
>>>>>>>>>>>
>>>>>>>>>>> />
>>>
>>>>
>>>>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>>>>> auth="Container"
>>>>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>>>>> description="User database that can be updated and
>>>>>>>>>>> saved"
>>>>>>>>>>>
>>>>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> pathname="conf/tomcat-users.xml" />
>>>
>>>> ...
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 07.02.2017 17:20, Fady Haikal wrote:
> Christopher,
> For the first time
@Christopher : just to make sure you got that bit, buried below : the actual replication
seems to work fine. The problem is only these "unsuccesful ping" messages in the log,
which fill the log, and which so far nobody has managed to find an explanation for.
>
> On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
> <ch...@christopherschultz.net> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Fady,
>>
>> On 2/7/17 10:53 AM, Fady Haikal wrote:
>>> Process PID Protocol local address local port
>>> Remote Address State Tomcat8.exe 8160 TCP
>>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>>
>> Stupid question: was this working in the past, and it stopped working?
>> Or are you trying to get this working for the first time?
>>
>> - -chris
>>
>>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>>> wrote:
>>>> Yes there is a ESTABLISHED connection, the replication of
>>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>>>> also faced this error on the log file
>>>>
>>>> On Tue, Feb 7, 2017 at 5:44 PM, Andr� Warnier (tomcat)
>>>> <aw...@ice-sa.com> wrote:
>>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>>>
>>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>>>> network monitoring at the level of IP and Port and i didnt
>>>>>> see any disconnection,
>>>>>
>>>>>
>>>>> but did you see a *connection* ? I mean, on the pinging node,
>>>>> if you use the Windows "netstat" program, for example as
>>>>> netstat -aon -p TCP you should see a list of connections in the
>>>>> ESTABLISHED state, of which one of the IP/ports should be your
>>>>> target IP:4000 (in the "remote" column). And on the pinged
>>>>> node, this port :4000 should be in the "local" column, in
>>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>>>> they agree.)
>>>>>
>>>>> Is that the case ?
>>>>>
>>>>>
>>>>>
>>>>> and yes i'm sure that no firewall is enabled.
>>>>>>
>>>>>>
>>>>>> I saw some strange think on the server that I have tried to
>>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>>> different IPs in the network, i don't know why and how i get
>>>>>> those IPs, after checking with the network team they told me
>>>>>> that those IPs are related to the SAN storage taking into
>>>>>> consideration that the Tomcat servers are not connected in
>>>>>> anyway to that SUN storage.
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 7, 2017 at 4:51 PM, Andr� Warnier (tomcat)
>>>>>> <aw...@ice-sa.com> wrote:
>>>>>>>
>>>>>>> Hi.
>>>>>>>
>>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>>
>>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>>> seems to say that the communication happens over TCP and
>>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>>> is no network disconnection" between the nodes.
>>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>>> times out without getting a response fom the target node.
>>>>>>> There is evidently a contradiction there. So this could
>>>>>>> still be some kind of network issue.
>>>>>>>
>>>>>>> Considering that the protocol command for this "ping"
>>>>>>> should be known by someone here, would it not be possible
>>>>>>> to imagine a little program in some scripting language (or
>>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>>> for a response and report the result ? That would at least
>>>>>>> confirm/deny that the problem is with the network.
>>>>>>>
>>>>>>> The log below does not for example say if the error happens
>>>>>>> when opening the TCP communication channel, or after
>>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>>> some l�w-level debugging options ? wireshark maybe another
>>>>>>> option, but it has quite a learning curve. And this is on
>>>>>>> Windows.
>>>>>>>
>>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>>> "network policies" there which can have wide-ranging
>>>>>>> side-effects.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>>> not able to replicate the key from node to node, please
>>>>>>>> find below the error, taking into consideration that
>>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>>
>>>>>>>>
>>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>> fo
>>>>>>>>
>>>>>>>>
>> Unable to replicate backup
>>>>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
>> ,
>>>>>>>>
>>>>>>>>
>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
>> ssage(ParallelNioSender.java:108)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
>> dMessage(PooledParallelSender.java:48)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
>> essage(ReplicationTransmitter.java:54)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
>> hannelCoordinator.java:82)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>> ge(ChannelInterceptorBase.java:76)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
>> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>> ge(ChannelInterceptorBase.java:76)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> sendMessage(TcpFailureDetector.java:93)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
>> ge(ChannelInterceptorBase.java:76)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>> ava:233)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
>> ava:186)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
>> fo(LazyReplicatedMap.java:170)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>> ctReplicatedMap.java:1040)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
>> ctReplicatedMap.java:1024)
>>>>>>>>
>>>>>>>>
>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
>> .java:374)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
>> va:279)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
>> .java:708)
>>>>>>>>
>>>>>>>>
>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>>>>>> at
>>>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
>> 0)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
>> ade.java:895)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
>> etRequestWrapper.java:231)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>> ationHttpRequest.java:568)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
>> ationHttpRequest.java:513)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
>> mpl.java:137)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
>> spFactoryImpl.java:109)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
>> yImpl.java:60)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
>> :100)
>>>>>>>>
>>>>>>>>
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>>>>> at
>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
>> pper.java:438)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
>> va:396)
>>>>>>>>
>>>>>>>>
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>>>>>> at
>>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>> ApplicationFilterChain.java:291)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
>> ionFilterChain.java:206)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
>> Dispatcher.java:719)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
>> licationDispatcher.java:467)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
>> ionDispatcher.java:390)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
>> nDispatcher.java:317)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
>> ve.java:445)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
>> ve.java:304)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
>> ve.java:181)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
>> eBinderValve.java:194)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
>> ve.java:318)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
>> ve.java:79)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
>> ThreadDetectionValve.java:208)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
>> AccessLogValve.java:616)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
>> eValve.java:88)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
>> .java:521)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
>> Http11Processor.java:1096)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
>> ess(AbstractProtocol.java:674)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
>> ndpoint.java:1500)
>>>>>>>>
>>>>>>>>
>> at
>>>>>>>>
>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
>> point.java:1456)
>>>>>>>>
>>>>>>>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>>> Source) at
>>>>>>>>
>>>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
>> skThread.java:61)
>>>>>>>>
>>>>>>>>
>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>>>> <pk...@airplus.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Fady,
>>>>>>>>>
>>>>>>>>> Sorry for top posting.
>>>>>>>>>
>>>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>>>> (working) case, a DeltaManager and a
>>>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>>>
>>>>>>>>> <Manager
>>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>>>
>>>>>>>>>
>> expireSessionsOnShutdown="false"
>>>>>>>>> notifyListenersOnReplication="true"/> ...
>>>>>>>>> <ClusterListener
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
>> rListener"/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>> Besides this, only ports, limits and values are different.
>>>>>>>>>
>>>>>>>>> You may want to filter out the replication for static
>>>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;
>> " />
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> Peter
>>>>>>>>>
>>>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>>>> earlier the issue is related to the cluster
>>>>>>>>>> configuration, and as per my research i can see that
>>>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>>>> found the solution of it
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>>>> file distributed with this work for additional
>>>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>>>> file except in compliance with the License. You may
>>>>>>>>>> obtain a copy of the License at
>>>>>>>>>>
>>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>>
>>>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>>>> writing, software distributed under the License is
>>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>>> implied. See the License for the specific language
>>>>>>>>>> governing permissions and limitations under the
>>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>>>
>>>>>>>>>>
>> />
>>>>>>>>>> --> <!--APR library loader. Documentation at
>>>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>>>
>>>>>>>>>>
>> SSLEngine="on" />
>>>>>>>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>>>> java/javax APIs--> <Listener
>>>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
>> ener"
>>>>>>>>>>
>>>>>>>>>>
>> />
>>>>>>>>>> <Listener
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
>> istener"
>>>>>>>>>>
>>>>>>>>>>
>> />
>>>>>>>>>> <Listener
>>>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
>> stener"
>>>>>>>>>>
>>>>>>>>>>
>> />
>>>>>>>>>>
>>>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>>>> auth="Container"
>>>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>>>> description="User database that can be updated and
>>>>>>>>>> saved"
>>>>>>>>>>
>>>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>>>
>>>>>>>>>>
>> pathname="conf/tomcat-users.xml" />
>>>>>>>>>> </GlobalNamingResources>
>>>>>>>>>>
>>>>>>>>>> <!-- A "Service" is a collection of one or more
>>>>>>>>>> "Connectors" that share a single "Container" Note: A
>>>>>>>>>> "Service" is not itself a "Container", so you may not
>>>>>>>>>> define subcomponents such as "Valves" at this level.
>>>>>>>>>> Documentation at /docs/config/service.html -->
>>>>>>>>>> <Service name="Catalina">
>>>>>>>>>>
>>>>>>>>>> <!--The connectors can use a shared executor, you can
>>>>>>>>>> define one or more named thread pools--> <!--
>>>>>>>>>> <Executor name="tomcatThreadPool"
>>>>>>>>>> namePrefix="catalina-exec-" maxThreads="150"
>>>>>>>>>> minSpareThreads="4"/> -->
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <!-- A "Connector" represents an endpoint by which
>>>>>>>>>> requests are received and responses are returned.
>>>>>>>>>> Documentation at : Java HTTP Connector:
>>>>>>>>>> /docs/config/http.html (blocking & non-blocking) Java
>>>>>>>>>> AJP Connector: /docs/config/ajp.html APR (HTTP/AJP)
>>>>>>>>>> Connector: /docs/apr.html Define a non-SSL/TLS
>>>>>>>>>> HTTP/1.1 Connector on port 8080 --> <Connector
>>>>>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>>>> enableLookups="false" disableUploadTimeout="true"
>>>>>>>>>> acceptCount="100" redirectPort="8443" /> <!-- A
>>>>>>>>>> "Connector" using the shared thread pool--> <!--
>>>>>>>>>> <Connector executor="tomcatThreadPool" port="8080"
>>>>>>>>>> protocol="HTTP/1.1" connectionTimeout="60000"
>>>>>>>>>> redirectPort="8443" /> --> <!-- Define a SSL/TLS
>>>>>>>>>> HTTP/1.1 Connector on port 8443 This connector uses
>>>>>>>>>> the NIO implementation that requires the JSSE style
>>>>>>>>>> configuration. When using the APR/native
>>>>>>>>>> implementation, the OpenSSL style configuration is
>>>>>>>>>> required as described in the APR/native documentation
>>>>>>>>>> --> <!-- <Connector port="8443"
>>>>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>>>>>
>>>>>>>>>>
>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>>>>>> secure="true" clientAuth="false" sslProtocol="TLS"
>>>>>>>>>> /> -->
>>>>>>>>>>
>>>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>>>> <Connector port="8009" protocol="AJP/1.3"
>>>>>>>>>> redirectPort="8443" />
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <!-- An Engine represents the entry point (within
>>>>>>>>>> Catalina) that processes every request. The Engine
>>>>>>>>>> implementation for Tomcat stand alone analyzes the
>>>>>>>>>> HTTP headers included with the request, and passes
>>>>>>>>>> them on to the appropriate Host (virtual host).
>>>>>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>>>>>
>>>>>>>>>> <!-- You should set jvmRoute to support
>>>>>>>>>> load-balancing via AJP ie :
>>>>>>>>>>
>>>>>>>>>> --> <Engine name="Catalina" defaultHost="localhost"
>>>>>>>>>> jvmRoute="TomcatNode1"> <!-- Use the LockOutRealm to
>>>>>>>>>> prevent attempts to guess user passwords via a
>>>>>>>>>> brute-force attack --> <Realm
>>>>>>>>>> className="org.apache.catalina.realm.LockOutRealm">
>>>>>>>>>> <!-- This Realm uses the UserDatabase configured in
>>>>>>>>>> the global JNDI resources under the key
>>>>>>>>>> "UserDatabase". Any edits that are performed against
>>>>>>>>>> this UserDatabase are immediately available for use
>>>>>>>>>> by the Realm. --> <Realm
>>>>>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>>>>>>
>>>>>>>>>>
>> resourceName="UserDatabase"/>
>>>>>>>>>> </Realm>
>>>>>>>>>>
>>>>>>>>>> <Host name="localhost" appBase="webapps"
>>>>>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>>>>>> startStopThreads="0"> <!--For clustering, please take
>>>>>>>>>> a look at documentation at: /docs/cluster-howto.html
>>>>>>>>>> (simple how to) /docs/config/cluster.html (reference
>>>>>>>>>> documentation) --> <!-- <Cluster
>>>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>>>>>>
>>>>>>>>>>
>> - -->
>>>>>>>>>>
>>>>>>>>>> <Cluster
>>>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>>>>>>
>>>>>>>>>>
>> channelSendOptions="4">
>>>>>>>>>> <Channel
>>>>>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>>>>>>
>>>>>>>>>>
>> <Membership
>>>>>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>>>>>>
>>>>>>>>>>
>> address="228.0.0.4"
>>>>>>>>>> port="45564" frequency="500" dropTime="9000"/>
>>>>>>>>>> <Receiver
>>>>>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver
>> "
>>>>>>>>>>
>>>>>>>>>>
>> address="auto"
>>>>>>>>>> port="4000" autoBind="100" selectorTimeout="5000"
>>>>>>>>>> maxThreads="6"/>
>>>>>>>>>>
>>>>>>>>>> <Sender
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTran
>> smitter">
>>>>>>>>>>
>>>>>>>>>>
>> <Transport
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParal
>> lelSender"/>
>>>>>>>>>>
>>>>>>>>>>
>> </Sender>
>>>>>>>>>> <Interceptor
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFai
>> lureDetector"/>
>>>>>>>>>>
>>>>>>>>>>
>> <Interceptor
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.Messag
>> eDispatch15Interceptor"/>
>>>>>>>>>>
>>>>>>>>>>
>> </Channel>
>>>>>>>>>>
>>>>>>>>>> <Valve
>>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>>
>>>>>>>>>>
>> filter=""/>
>>>>>>>>>> <Valve
>>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/
>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>> <Deployer
>>>>>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>>>>>>
>>>>>>>>>>
>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>>>>>
>>>>>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>>>>>
>>>>>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>>>>>> watchEnabled="false"/>
>>>>>>>>>>
>>>>>>>>>> <ClusterListener
>>>>>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListene
>> r"/>
>>>>>>>>>>
>>>>>>>>>>
>> </Cluster>
>>>>>>>>>> <!-- SingleSignOn valve, share authentication between
>>>>>>>>>> web applications Documentation at:
>>>>>>>>>> /docs/config/valve.html --> <!-- <Valve
>>>>>>>>>> className="org.apache.catalina.authenticator.SingleSignOn"
>>>>>>>>>> /> -->
>>>>>>>>>>
>>>>>>>>>> <!-- Access log processes all example. Documentation
>>>>>>>>>> at: /docs/config/valve.html Note: The pattern used is
>>>>>>>>>> equivalent to using pattern="common" --> <Valve
>>>>>>>>>> className="org.apache.catalina.valves.AccessLogValve"
>>>>>>>>>>
>>>>>>>>>>
>> directory="logs"
>>>>>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>>>>>> pattern="%h %l %u %t "%r" %s %b" /> <Valve
>>>>>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve
>> "
>>>>>>>>>>
>>>>>>>>>>
>> threshold="900" />
>>>>>>>>>>
>>>>>>>>>> </Host> </Engine> </Service> </Server>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, Andr� Warnier
>>>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, What is the host OS ? Windows Server 2012
>>>>>>>>>>>> What is the Tomcat version ? Apache
>>>>>>>>>>>> Tomcat/8.0.30
>>>>>>>>>>>>
>>>>>>>>>>>> Is this problem new ? was this working before ?
>>>>>>>>>>>> how long ? Since cluster implementation
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I still don't know tribes, but then my non-educated
>>>>>>>>>>> guess at this point would be that there is
>>>>>>>>>>> something wrong in your configuration. Can you
>>>>>>>>>>> copy/paste it here ? (remove sensible things like
>>>>>>>>>>> passwords, public IP addresses etc..)(but not to
>>>>>>>>>>> the point of making it uncheckable).
>>>>>>>>>>>
>>>>>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Is there actually something listening on that
>>>>>>>>>>>> address/port ? Tomcat cluster
>>>>>>>>>>>>
>>>>>>>>>>>> the Port 4000 is listening and there is no
>>>>>>>>>>>> disconnection between 2 nodes ping and telnet are
>>>>>>>>>>>> OK
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, Andr� Warnier
>>>>>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Plz can i get some help here? This issue is
>>>>>>>>>>>>>> still occurring and it's filling the log file
>>>>>>>>>>>>>> in the Production server
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi. If you want quick answers, you should
>>>>>>>>>>>>> provide more information. What is the host OS
>>>>>>>>>>>>> ? What is the Tomcat version ? Is this problem
>>>>>>>>>>>>> new ? was this working before ? how long ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do not know tribes at all, but according to
>>>>>>>>>>>>> the logfile below, it seems that something is
>>>>>>>>>>>>> trying to "ping" the address 10.114.43.103,
>>>>>>>>>>>>> port 4000, and never getting a response (or at
>>>>>>>>>>>>> least not within 3000ms). Is there actually
>>>>>>>>>>>>> something listening on that address/port ? The
>>>>>>>>>>>>> "netstat" command (available both on Linux and
>>>>>>>>>>>>> Windows) can tell you. If there is something
>>>>>>>>>>>>> listening there, can it respond to whatever is
>>>>>>>>>>>>> ping-ing it ? (routing, firewall, ..)
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal
>>>>>>>>>>>>>> <fa...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Guys, we are facing the below errors in
>>>>>>>>>>>>>>> Tomcat cluster, please advise
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE
>>>>>>>>>>>>>>> [GroupChannel-Heartbeat-1]
>>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
>> rtbeat
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>>>>>> org.apache.catalina.tribes.ChannelException:
>>>>>>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>>>>>>>> members:tcp://{10, 114, 43, 103}:4000; at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender
>> .sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSen
>> der.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitte
>> r.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMe
>> ssage(ChannelCoordinator.java:82)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
>> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispa
>> tchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
>> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDe
>> tector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
>> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
>> annel.java:233)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
>> annel.java:186)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChanne
>> l.java:99)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.pin
>> g(AbstractReplicatedMap.java:267)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
>> rtbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(Gr
>> oupChannel.java:161)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThr
>> ead.run(GroupChannel.java:697)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
>> n
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> Unable
>>>>>>>>>>>>>>> to process request in NioReceiver
>>>>>>>>>>>>>>> java.io.IOException: A non-blocking socket
>>>>>>>>>>>>>>> operation could not be completed
>>>>>>>>>>>>>>> immediately at
>>>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close0(Native
>>>>>>>>>>>>>>> Method) at
>>>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.SocketChannelImpl.kill(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.select(Unknown
>>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
>> n(NioReceiver.java:272)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(N
>> ioReceiver.java:425)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----------------------------------------------------------
>> - ----------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------
>> - ---------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -------------------------------------------------------------
>> - --------
>>>>>>>>>>>>
>>>>>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --------------------------------------------------------------
>> - -------
>>>>>>>>>>>
>>>>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------
>> - ------
>>>>>>>>>>
>>>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> -----------------------------------------------------------------
>> - ----
>>>>>>>>
>>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail:
>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------
>> - ---
>>>>>>>
>>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail:
>>>>>>> users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------------------
>> - --
>>>>>>
>>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> --------------------------------------------------------------------
>> - -
>>>>>
>>>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJYmfN4AAoJEBzwKT+lPKRYZuIQAI2NXtOMNcNpd88y2rXRZFCK
>> p3YN8OBo07UMkT+N6OgmkCrWBGRqIk9SZSUugnWP7N1n5nOFpaio/I2n/2dGOXDB
>> oYfPA3KVkWaMQenOAujjyodi7Q7ABUurJ07V+D0zfBtPpqE7up9/edl94pq8znuR
>> KGp1eEKeMlMiZAS9vm5/0ifhhMIDGfE/Y3xjqf+Kb5gHXlA5BrOrlo7+UX26uuqr
>> qIIaobTCKYaQvYE5HP+9FyOsXCd1oicKz9JIPAoc8A7Yj//n+5Ogtoczyu+mSzxL
>> X7/JSRdywbyfFo+1bOaecxmx9Bh9UgyziEh2AuXXMGwf17a58ciN8VCqtJcJC3HO
>> rffZtc9sqm/Tyt5tvo5PgPbpTrvsjeZXMrxUx5QuOjWlkoVOPptACB4LATVmXaoM
>> lHGw0OBCQgNViI5LXMs59QSdj9/J7tzN+0AakFmO0GM2fHjiFzgHTLGa5QA9yO7b
>> wIDpCQj3cDFig+4z95ZFHcGTmcUd9u5a6dO/4BPzFvsR/r6kfmH7qV1mwVu1TkEH
>> 4B6EO5AwCwZJNM2asxFktswNoI4Wl2pP7ZjdxW8tIXchJco0iHn/VhnO1kPeNI8x
>> ws6KXUpZqPp4ylLWYKTLHg9fMc9Ffn6u6jR09o9amEIWnSlinA4R+n9VztWiBR45
>> h9q0x3f+aKuQp2IE0UBE
>> =WoKC
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Christopher,
For the first time
On Tue, Feb 7, 2017 at 6:19 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Fady,
>
> On 2/7/17 10:53 AM, Fady Haikal wrote:
>> Process PID Protocol local address local port
>> Remote Address State Tomcat8.exe 8160 TCP
>> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
>
> Stupid question: was this working in the past, and it stopped working?
> Or are you trying to get this working for the first time?
>
> - -chris
>
>> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
>> wrote:
>>> Yes there is a ESTABLISHED connection, the replication of
>>> sessions is working fine (port 4000 is for tomcat cluster) but we
>>> also faced this error on the log file
>>>
>>> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat)
>>> <aw...@ice-sa.com> wrote:
>>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>>
>>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>>> network monitoring at the level of IP and Port and i didnt
>>>>> see any disconnection,
>>>>
>>>>
>>>> but did you see a *connection* ? I mean, on the pinging node,
>>>> if you use the Windows "netstat" program, for example as
>>>> netstat -aon -p TCP you should see a list of connections in the
>>>> ESTABLISHED state, of which one of the IP/ports should be your
>>>> target IP:4000 (in the "remote" column). And on the pinged
>>>> node, this port :4000 should be in the "local" column, in
>>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>>> they agree.)
>>>>
>>>> Is that the case ?
>>>>
>>>>
>>>>
>>>> and yes i'm sure that no firewall is enabled.
>>>>>
>>>>>
>>>>> I saw some strange think on the server that I have tried to
>>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>>> different IPs in the network, i don't know why and how i get
>>>>> those IPs, after checking with the network team they told me
>>>>> that those IPs are related to the SAN storage taking into
>>>>> consideration that the Tomcat servers are not connected in
>>>>> anyway to that SUN storage.
>>>>>
>>>>>
>>>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat)
>>>>> <aw...@ice-sa.com> wrote:
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>>
>>>>>> I know nothing of Tribes, but the on-line documentation
>>>>>> seems to say that the communication happens over TCP and
>>>>>> that the protocol used is not encrypted. Fady previously
>>>>>> tried a standard "ping" and a "telnet" between the two
>>>>>> nodes, and that is the base for him mentioning that "there
>>>>>> is no network disconnection" between the nodes.
>>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>>> times out without getting a response fom the target node.
>>>>>> There is evidently a contradiction there. So this could
>>>>>> still be some kind of network issue.
>>>>>>
>>>>>> Considering that the protocol command for this "ping"
>>>>>> should be known by someone here, would it not be possible
>>>>>> to imagine a little program in some scripting language (or
>>>>>> even java, God forbid), which would open a TCP channel with
>>>>>> the target node IP/port, send such a "ping" message, wait
>>>>>> for a response and report the result ? That would at least
>>>>>> confirm/deny that the problem is with the network.
>>>>>>
>>>>>> The log below does not for example say if the error happens
>>>>>> when opening the TCP communication channel, or after
>>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>>> if Fady tried this). Maybe tribes also already contains
>>>>>> some löw-level debugging options ? wireshark maybe another
>>>>>> option, but it has quite a learning curve. And this is on
>>>>>> Windows.
>>>>>>
>>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>>> "network policies" there which can have wide-ranging
>>>>>> side-effects.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>>> not able to replicate the key from node to node, please
>>>>>>> find below the error, taking into consideration that
>>>>>>> there is no network disconnection between 2 nodes
>>>>>>>
>>>>>>>
>>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
> fo
>>>>>>>
>>>>>>>
> Unable to replicate backup
>>>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
> ,
>>>>>>>
>>>>>>>
> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
> ssage(ParallelNioSender.java:108)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
> dMessage(PooledParallelSender.java:48)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
> essage(ReplicationTransmitter.java:54)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
> hannelCoordinator.java:82)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
> rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> sendMessage(TcpFailureDetector.java:93)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
> ge(ChannelInterceptorBase.java:76)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
> ava:233)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
> ava:186)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
> fo(LazyReplicatedMap.java:170)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
> ctReplicatedMap.java:1040)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
> ctReplicatedMap.java:1024)
>>>>>>>
>>>>>>>
> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>>>>> at
>>>>>>>
>>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
> .java:374)
>>>>>>>
>>>>>>>
> at
>>>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
> va:279)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
> .java:708)
>>>>>>>
>>>>>>>
> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>>>>> at
>>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
> 0)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
> ade.java:895)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
> etRequestWrapper.java:231)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
> ationHttpRequest.java:568)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
> ationHttpRequest.java:513)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
> mpl.java:137)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
> spFactoryImpl.java:109)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
> yImpl.java:60)
>>>>>>>
>>>>>>>
> at
>>>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
> :100)
>>>>>>>
>>>>>>>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>>>> at
>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
> pper.java:438)
>>>>>>>
>>>>>>>
> at
>>>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
> va:396)
>>>>>>>
>>>>>>>
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>>>>> at
>>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
> ApplicationFilterChain.java:291)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
> ionFilterChain.java:206)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
> Dispatcher.java:719)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
> licationDispatcher.java:467)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
> ionDispatcher.java:390)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
> nDispatcher.java:317)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
> ve.java:445)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
> ve.java:304)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
> ve.java:181)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
> eBinderValve.java:194)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
> ve.java:318)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
> ve.java:79)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
> ThreadDetectionValve.java:208)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
> AccessLogValve.java:616)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
> eValve.java:88)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
> .java:521)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
> Http11Processor.java:1096)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
> ess(AbstractProtocol.java:674)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
> ndpoint.java:1500)
>>>>>>>
>>>>>>>
> at
>>>>>>>
>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
> point.java:1456)
>>>>>>>
>>>>>>>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>> Source) at
>>>>>>>
>>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
> skThread.java:61)
>>>>>>>
>>>>>>>
> at java.lang.Thread.run(Unknown Source)
>>>>>>>
>>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>>> <pk...@airplus.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Fady,
>>>>>>>>
>>>>>>>> Sorry for top posting.
>>>>>>>>
>>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>>> (working) case, a DeltaManager and a
>>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>>
>>>>>>>> <Manager
>>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>>
>>>>>>>>
> expireSessionsOnShutdown="false"
>>>>>>>> notifyListenersOnReplication="true"/> ...
>>>>>>>> <ClusterListener
>>>>>>>>
>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
> rListener"/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
> Besides this, only ports, limits and values are different.
>>>>>>>>
>>>>>>>> You may want to filter out the replication for static
>>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>
>>>>>>>>
>>>>>>>>
> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;
> " />
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> Peter
>>>>>>>>
>>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>>> earlier the issue is related to the cluster
>>>>>>>>> configuration, and as per my research i can see that
>>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>>> found the solution of it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>>> file distributed with this work for additional
>>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>>> file except in compliance with the License. You may
>>>>>>>>> obtain a copy of the License at
>>>>>>>>>
>>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>
>>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>>> writing, software distributed under the License is
>>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>> implied. See the License for the specific language
>>>>>>>>> governing permissions and limitations under the
>>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>>
>>>>>>>>>
> />
>>>>>>>>> --> <!--APR library loader. Documentation at
>>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>>
>>>>>>>>>
> SSLEngine="on" />
>>>>>>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>>> java/javax APIs--> <Listener
>>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
> ener"
>>>>>>>>>
>>>>>>>>>
> />
>>>>>>>>> <Listener
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
> istener"
>>>>>>>>>
>>>>>>>>>
> />
>>>>>>>>> <Listener
>>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
> stener"
>>>>>>>>>
>>>>>>>>>
> />
>>>>>>>>>
>>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>>> auth="Container"
>>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>>> description="User database that can be updated and
>>>>>>>>> saved"
>>>>>>>>>
>>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>>
>>>>>>>>>
> pathname="conf/tomcat-users.xml" />
>>>>>>>>> </GlobalNamingResources>
>>>>>>>>>
>>>>>>>>> <!-- A "Service" is a collection of one or more
>>>>>>>>> "Connectors" that share a single "Container" Note: A
>>>>>>>>> "Service" is not itself a "Container", so you may not
>>>>>>>>> define subcomponents such as "Valves" at this level.
>>>>>>>>> Documentation at /docs/config/service.html -->
>>>>>>>>> <Service name="Catalina">
>>>>>>>>>
>>>>>>>>> <!--The connectors can use a shared executor, you can
>>>>>>>>> define one or more named thread pools--> <!--
>>>>>>>>> <Executor name="tomcatThreadPool"
>>>>>>>>> namePrefix="catalina-exec-" maxThreads="150"
>>>>>>>>> minSpareThreads="4"/> -->
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <!-- A "Connector" represents an endpoint by which
>>>>>>>>> requests are received and responses are returned.
>>>>>>>>> Documentation at : Java HTTP Connector:
>>>>>>>>> /docs/config/http.html (blocking & non-blocking) Java
>>>>>>>>> AJP Connector: /docs/config/ajp.html APR (HTTP/AJP)
>>>>>>>>> Connector: /docs/apr.html Define a non-SSL/TLS
>>>>>>>>> HTTP/1.1 Connector on port 8080 --> <Connector
>>>>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>>> enableLookups="false" disableUploadTimeout="true"
>>>>>>>>> acceptCount="100" redirectPort="8443" /> <!-- A
>>>>>>>>> "Connector" using the shared thread pool--> <!--
>>>>>>>>> <Connector executor="tomcatThreadPool" port="8080"
>>>>>>>>> protocol="HTTP/1.1" connectionTimeout="60000"
>>>>>>>>> redirectPort="8443" /> --> <!-- Define a SSL/TLS
>>>>>>>>> HTTP/1.1 Connector on port 8443 This connector uses
>>>>>>>>> the NIO implementation that requires the JSSE style
>>>>>>>>> configuration. When using the APR/native
>>>>>>>>> implementation, the OpenSSL style configuration is
>>>>>>>>> required as described in the APR/native documentation
>>>>>>>>> --> <!-- <Connector port="8443"
>>>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>>>>
>>>>>>>>>
> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>>>>> secure="true" clientAuth="false" sslProtocol="TLS"
>>>>>>>>> /> -->
>>>>>>>>>
>>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>>> <Connector port="8009" protocol="AJP/1.3"
>>>>>>>>> redirectPort="8443" />
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <!-- An Engine represents the entry point (within
>>>>>>>>> Catalina) that processes every request. The Engine
>>>>>>>>> implementation for Tomcat stand alone analyzes the
>>>>>>>>> HTTP headers included with the request, and passes
>>>>>>>>> them on to the appropriate Host (virtual host).
>>>>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>>>>
>>>>>>>>> <!-- You should set jvmRoute to support
>>>>>>>>> load-balancing via AJP ie :
>>>>>>>>>
>>>>>>>>> --> <Engine name="Catalina" defaultHost="localhost"
>>>>>>>>> jvmRoute="TomcatNode1"> <!-- Use the LockOutRealm to
>>>>>>>>> prevent attempts to guess user passwords via a
>>>>>>>>> brute-force attack --> <Realm
>>>>>>>>> className="org.apache.catalina.realm.LockOutRealm">
>>>>>>>>> <!-- This Realm uses the UserDatabase configured in
>>>>>>>>> the global JNDI resources under the key
>>>>>>>>> "UserDatabase". Any edits that are performed against
>>>>>>>>> this UserDatabase are immediately available for use
>>>>>>>>> by the Realm. --> <Realm
>>>>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>>>>>
>>>>>>>>>
> resourceName="UserDatabase"/>
>>>>>>>>> </Realm>
>>>>>>>>>
>>>>>>>>> <Host name="localhost" appBase="webapps"
>>>>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>>>>> startStopThreads="0"> <!--For clustering, please take
>>>>>>>>> a look at documentation at: /docs/cluster-howto.html
>>>>>>>>> (simple how to) /docs/config/cluster.html (reference
>>>>>>>>> documentation) --> <!-- <Cluster
>>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>>>>>
>>>>>>>>>
> - -->
>>>>>>>>>
>>>>>>>>> <Cluster
>>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>>>>>
>>>>>>>>>
> channelSendOptions="4">
>>>>>>>>> <Channel
>>>>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>>>>>
>>>>>>>>>
> <Membership
>>>>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>>>>>
>>>>>>>>>
> address="228.0.0.4"
>>>>>>>>> port="45564" frequency="500" dropTime="9000"/>
>>>>>>>>> <Receiver
>>>>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver
> "
>>>>>>>>>
>>>>>>>>>
> address="auto"
>>>>>>>>> port="4000" autoBind="100" selectorTimeout="5000"
>>>>>>>>> maxThreads="6"/>
>>>>>>>>>
>>>>>>>>> <Sender
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTran
> smitter">
>>>>>>>>>
>>>>>>>>>
> <Transport
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParal
> lelSender"/>
>>>>>>>>>
>>>>>>>>>
> </Sender>
>>>>>>>>> <Interceptor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFai
> lureDetector"/>
>>>>>>>>>
>>>>>>>>>
> <Interceptor
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.Messag
> eDispatch15Interceptor"/>
>>>>>>>>>
>>>>>>>>>
> </Channel>
>>>>>>>>>
>>>>>>>>> <Valve
>>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>>
>>>>>>>>>
> filter=""/>
>>>>>>>>> <Valve
>>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/
>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
> <Deployer
>>>>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>>>>>
>>>>>>>>>
> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>>>>
>>>>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>>>>
>>>>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>>>>> watchEnabled="false"/>
>>>>>>>>>
>>>>>>>>> <ClusterListener
>>>>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListene
> r"/>
>>>>>>>>>
>>>>>>>>>
> </Cluster>
>>>>>>>>> <!-- SingleSignOn valve, share authentication between
>>>>>>>>> web applications Documentation at:
>>>>>>>>> /docs/config/valve.html --> <!-- <Valve
>>>>>>>>> className="org.apache.catalina.authenticator.SingleSignOn"
>>>>>>>>> /> -->
>>>>>>>>>
>>>>>>>>> <!-- Access log processes all example. Documentation
>>>>>>>>> at: /docs/config/valve.html Note: The pattern used is
>>>>>>>>> equivalent to using pattern="common" --> <Valve
>>>>>>>>> className="org.apache.catalina.valves.AccessLogValve"
>>>>>>>>>
>>>>>>>>>
> directory="logs"
>>>>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>>>>> pattern="%h %l %u %t "%r" %s %b" /> <Valve
>>>>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve
> "
>>>>>>>>>
>>>>>>>>>
> threshold="900" />
>>>>>>>>>
>>>>>>>>> </Host> </Engine> </Service> </Server>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier
>>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi, What is the host OS ? Windows Server 2012
>>>>>>>>>>> What is the Tomcat version ? Apache
>>>>>>>>>>> Tomcat/8.0.30
>>>>>>>>>>>
>>>>>>>>>>> Is this problem new ? was this working before ?
>>>>>>>>>>> how long ? Since cluster implementation
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I still don't know tribes, but then my non-educated
>>>>>>>>>> guess at this point would be that there is
>>>>>>>>>> something wrong in your configuration. Can you
>>>>>>>>>> copy/paste it here ? (remove sensible things like
>>>>>>>>>> passwords, public IP addresses etc..)(but not to
>>>>>>>>>> the point of making it uncheckable).
>>>>>>>>>>
>>>>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is there actually something listening on that
>>>>>>>>>>> address/port ? Tomcat cluster
>>>>>>>>>>>
>>>>>>>>>>> the Port 4000 is listening and there is no
>>>>>>>>>>> disconnection between 2 nodes ping and telnet are
>>>>>>>>>>> OK
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier
>>>>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Plz can i get some help here? This issue is
>>>>>>>>>>>>> still occurring and it's filling the log file
>>>>>>>>>>>>> in the Production server
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi. If you want quick answers, you should
>>>>>>>>>>>> provide more information. What is the host OS
>>>>>>>>>>>> ? What is the Tomcat version ? Is this problem
>>>>>>>>>>>> new ? was this working before ? how long ?
>>>>>>>>>>>>
>>>>>>>>>>>> I do not know tribes at all, but according to
>>>>>>>>>>>> the logfile below, it seems that something is
>>>>>>>>>>>> trying to "ping" the address 10.114.43.103,
>>>>>>>>>>>> port 4000, and never getting a response (or at
>>>>>>>>>>>> least not within 3000ms). Is there actually
>>>>>>>>>>>> something listening on that address/port ? The
>>>>>>>>>>>> "netstat" command (available both on Linux and
>>>>>>>>>>>> Windows) can tell you. If there is something
>>>>>>>>>>>> listening there, can it respond to whatever is
>>>>>>>>>>>> ping-ing it ? (routing, firewall, ..)
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal
>>>>>>>>>>>>> <fa...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guys, we are facing the below errors in
>>>>>>>>>>>>>> Tomcat cluster, please advise
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE
>>>>>>>>>>>>>> [GroupChannel-Heartbeat-1]
>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
> rtbeat
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>>>>> org.apache.catalina.tribes.ChannelException:
>>>>>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>>>>>>> members:tcp://{10, 114, 43, 103}:4000; at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender
> .sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSen
> der.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitte
> r.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMe
> ssage(ChannelCoordinator.java:82)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispa
> tchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDe
> tector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
> ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
> annel.java:233)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
> annel.java:186)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChanne
> l.java:99)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.pin
> g(AbstractReplicatedMap.java:267)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
> rtbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(Gr
> oupChannel.java:161)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThr
> ead.run(GroupChannel.java:697)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
> n
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> Unable
>>>>>>>>>>>>>> to process request in NioReceiver
>>>>>>>>>>>>>> java.io.IOException: A non-blocking socket
>>>>>>>>>>>>>> operation could not be completed
>>>>>>>>>>>>>> immediately at
>>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close0(Native
>>>>>>>>>>>>>> Method) at
>>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.SocketChannelImpl.kill(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.select(Unknown
>>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
> n(NioReceiver.java:272)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(N
> ioReceiver.java:425)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------------------------------------------------
> - ----------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------------------
> - ---------
>>>>>>>>>>>>
>>>>>>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -------------------------------------------------------------
> - --------
>>>>>>>>>>>
>>>>>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------
> - -------
>>>>>>>>>>
>>>>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------
> - ------
>>>>>>>>>
>>>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> -----------------------------------------------------------------
> - ----
>>>>>>>
>>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail:
>>>>>>> users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------
> - ---
>>>>>>
>>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>> -------------------------------------------------------------------
> - --
>>>>>
>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail:
>>>>> users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> --------------------------------------------------------------------
> - -
>>>>
>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>
>> ---------------------------------------------------------------------
>>
>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYmfN4AAoJEBzwKT+lPKRYZuIQAI2NXtOMNcNpd88y2rXRZFCK
> p3YN8OBo07UMkT+N6OgmkCrWBGRqIk9SZSUugnWP7N1n5nOFpaio/I2n/2dGOXDB
> oYfPA3KVkWaMQenOAujjyodi7Q7ABUurJ07V+D0zfBtPpqE7up9/edl94pq8znuR
> KGp1eEKeMlMiZAS9vm5/0ifhhMIDGfE/Y3xjqf+Kb5gHXlA5BrOrlo7+UX26uuqr
> qIIaobTCKYaQvYE5HP+9FyOsXCd1oicKz9JIPAoc8A7Yj//n+5Ogtoczyu+mSzxL
> X7/JSRdywbyfFo+1bOaecxmx9Bh9UgyziEh2AuXXMGwf17a58ciN8VCqtJcJC3HO
> rffZtc9sqm/Tyt5tvo5PgPbpTrvsjeZXMrxUx5QuOjWlkoVOPptACB4LATVmXaoM
> lHGw0OBCQgNViI5LXMs59QSdj9/J7tzN+0AakFmO0GM2fHjiFzgHTLGa5QA9yO7b
> wIDpCQj3cDFig+4z95ZFHcGTmcUd9u5a6dO/4BPzFvsR/r6kfmH7qV1mwVu1TkEH
> 4B6EO5AwCwZJNM2asxFktswNoI4Wl2pP7ZjdxW8tIXchJco0iHn/VhnO1kPeNI8x
> ws6KXUpZqPp4ylLWYKTLHg9fMc9Ffn6u6jR09o9amEIWnSlinA4R+n9VztWiBR45
> h9q0x3f+aKuQp2IE0UBE
> =WoKC
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Fady,
On 2/7/17 10:53 AM, Fady Haikal wrote:
> Process PID Protocol local address local port
> Remote Address State Tomcat8.exe 8160 TCP
> imal14-app2 4000 imal14-app1.albaraka.com.sd ESTABLISHED
Stupid question: was this working in the past, and it stopped working?
Or are you trying to get this working for the first time?
- -chris
> On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com>
> wrote:
>> Yes there is a ESTABLISHED connection, the replication of
>> sessions is working fine (port 4000 is for tomcat cluster) but we
>> also faced this error on the log file
>>
>> On Tue, Feb 7, 2017 at 5:44 PM, Andr� Warnier (tomcat)
>> <aw...@ice-sa.com> wrote:
>>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>>
>>>> Hi, telnet IP 4000 is working fine, i installed a tool for
>>>> network monitoring at the level of IP and Port and i didnt
>>>> see any disconnection,
>>>
>>>
>>> but did you see a *connection* ? I mean, on the pinging node,
>>> if you use the Windows "netstat" program, for example as
>>> netstat -aon -p TCP you should see a list of connections in the
>>> ESTABLISHED state, of which one of the IP/ports should be your
>>> target IP:4000 (in the "remote" column). And on the pinged
>>> node, this port :4000 should be in the "local" column, in
>>> LISTEN mode (and also probably one in the ESTABLISHED state, if
>>> they agree.)
>>>
>>> Is that the case ?
>>>
>>>
>>>
>>> and yes i'm sure that no firewall is enabled.
>>>>
>>>>
>>>> I saw some strange think on the server that I have tried to
>>>> ping the multicast IP (228.0.0.4) and i get reply from
>>>> different IPs in the network, i don't know why and how i get
>>>> those IPs, after checking with the network team they told me
>>>> that those IPs are related to the SAN storage taking into
>>>> consideration that the Tomcat servers are not connected in
>>>> anyway to that SUN storage.
>>>>
>>>>
>>>> On Tue, Feb 7, 2017 at 4:51 PM, Andr� Warnier (tomcat)
>>>> <aw...@ice-sa.com> wrote:
>>>>>
>>>>> Hi.
>>>>>
>>>>> This is for the Tomcat/Tribes experts on the list.
>>>>>
>>>>> I know nothing of Tribes, but the on-line documentation
>>>>> seems to say that the communication happens over TCP and
>>>>> that the protocol used is not encrypted. Fady previously
>>>>> tried a standard "ping" and a "telnet" between the two
>>>>> nodes, and that is the base for him mentioning that "there
>>>>> is no network disconnection" between the nodes.
>>>>> Nevertheless, the calling pinging node seems to say that it
>>>>> times out without getting a response fom the target node.
>>>>> There is evidently a contradiction there. So this could
>>>>> still be some kind of network issue.
>>>>>
>>>>> Considering that the protocol command for this "ping"
>>>>> should be known by someone here, would it not be possible
>>>>> to imagine a little program in some scripting language (or
>>>>> even java, God forbid), which would open a TCP channel with
>>>>> the target node IP/port, send such a "ping" message, wait
>>>>> for a response and report the result ? That would at least
>>>>> confirm/deny that the problem is with the network.
>>>>>
>>>>> The log below does not for example say if the error happens
>>>>> when opening the TCP communication channel, or after
>>>>> sending the ping message on it, (Of course, testing the TCP
>>>>> open could be done with "telnet IP 4000", but I don't know
>>>>> if Fady tried this). Maybe tribes also already contains
>>>>> some l�w-level debugging options ? wireshark maybe another
>>>>> option, but it has quite a learning curve. And this is on
>>>>> Windows.
>>>>>
>>>>> By the way Fady, are you sure that your "Windows Firewall
>>>>> with Enhanced Security" is not just dropping TCP packets
>>>>> to/from port 40xx (or from "java.exe") ? There are some
>>>>> "network policies" there which can have wide-ranging
>>>>> side-effects.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>>
>>>>>>
>>>>>> Hi, issue still not fixed. Tomcat session replication is
>>>>>> not able to replicate the key from node to node, please
>>>>>> find below the error, taking into consideration that
>>>>>> there is no network disconnection between 2 nodes
>>>>>>
>>>>>>
>>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
fo
>>>>>>
>>>>>>
Unable to replicate backup
>>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10
,
>>>>>>
>>>>>>
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64
>>>>>> 12 -102 -14 -85 -87 15 9 -51 -112 }, payload={},
>>>>>> command={}, domain={}, ]. Reason:Operation has timed
>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
>>>>>> 102}:4000; org.apache.catalina.tribes.ChannelException:
>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>> members:tcp://{10, 114, 43, 102}:4000; at
>>>>>>
>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMe
ssage(ParallelNioSender.java:108)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sen
dMessage(PooledParallelSender.java:48)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendM
essage(ReplicationTransmitter.java:54)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(C
hannelCoordinator.java:82)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
ge(ChannelInterceptorBase.java:76)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInte
rceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
ge(ChannelInterceptorBase.java:76)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
sendMessage(TcpFailureDetector.java:93)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessa
ge(ChannelInterceptorBase.java:76)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
ava:233)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.j
ava:186)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryIn
fo(LazyReplicatedMap.java:170)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
ctReplicatedMap.java:1040)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(Abstra
ctReplicatedMap.java:1024)
>>>>>>
>>>>>>
at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>>>> at
>>>>>>
>>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession
.java:374)
>>>>>>
>>>>>>
at
>>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.ja
va:279)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase
.java:708)
>>>>>>
>>>>>>
at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>>>> at
>>>>>> org.apache.catalina.connector.Request.getSession(Request.java:226
0)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFac
ade.java:895)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServl
etRequestWrapper.java:231)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
ationHttpRequest.java:568)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(Applic
ationHttpRequest.java:513)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextI
mpl.java:137)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(J
spFactoryImpl.java:109)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactor
yImpl.java:60)
>>>>>>
>>>>>>
at
>>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java
:100)
>>>>>>
>>>>>>
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>>> at
>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWra
pper.java:438)
>>>>>>
>>>>>>
at
>>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.ja
va:396)
>>>>>>
>>>>>>
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>>>> at
>>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:291)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicat
ionFilterChain.java:206)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(Application
Dispatcher.java:719)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(App
licationDispatcher.java:467)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(Applicat
ionDispatcher.java:390)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(Applicatio
nDispatcher.java:317)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostVal
ve.java:445)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostVal
ve.java:304)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostVal
ve.java:181)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRout
eBinderValve.java:194)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationVal
ve.java:318)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVal
ve.java:79)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(Stuck
ThreadDetectionValve.java:208)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(Abstract
AccessLogValve.java:616)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngin
eValve.java:88)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
.java:521)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
Http11Processor.java:1096)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proc
ess(AbstractProtocol.java:674)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioE
ndpoint.java:1500)
>>>>>>
>>>>>>
at
>>>>>>
>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEnd
point.java:1456)
>>>>>>
>>>>>>
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>> Source) at
>>>>>>
>>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta
skThread.java:61)
>>>>>>
>>>>>>
at java.lang.Thread.run(Unknown Source)
>>>>>>
>>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter
>>>>>> <pk...@airplus.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Fady,
>>>>>>>
>>>>>>> Sorry for top posting.
>>>>>>>
>>>>>>> If I remember correctly, the Cluster Element goes into
>>>>>>> the Container and not the Host. Plus I see in our
>>>>>>> (working) case, a DeltaManager and a
>>>>>>> JvmRouteSessionIDBinderListener
>>>>>>>
>>>>>>> <Manager
>>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>>>
>>>>>>>
expireSessionsOnShutdown="false"
>>>>>>> notifyListenersOnReplication="true"/> ...
>>>>>>> <ClusterListener
>>>>>>>
>>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinde
rListener"/>
>>>>>>>
>>>>>>>
>>>>>>>
Besides this, only ports, limits and values are different.
>>>>>>>
>>>>>>> You may want to filter out the replication for static
>>>>>>> resources as gifs jpg or css. <Valve
>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>
>>>>>>>
>>>>>>>
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;
" />
>>>>>>>
>>>>>>> Best regards
>>>>>>>
>>>>>>> Peter
>>>>>>>
>>>>>>>> below is the server.xml configuration, as mentioened
>>>>>>>> earlier the issue is related to the cluster
>>>>>>>> configuration, and as per my research i can see that
>>>>>>>> some users are facing the same issue but i didnt
>>>>>>>> found the solution of it
>>>>>>>>
>>>>>>>>
>>>>>>>> <?xml version='1.0' encoding='utf-8'?> <!-- Licensed
>>>>>>>> to the Apache Software Foundation (ASF) under one or
>>>>>>>> more contributor license agreements. See the NOTICE
>>>>>>>> file distributed with this work for additional
>>>>>>>> information regarding copyright ownership. The ASF
>>>>>>>> licenses this file to You under the Apache License,
>>>>>>>> Version 2.0 (the "License"); you may not use this
>>>>>>>> file except in compliance with the License. You may
>>>>>>>> obtain a copy of the License at
>>>>>>>>
>>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>
>>>>>>>> Unless required by applicable law or agreed to in
>>>>>>>> writing, software distributed under the License is
>>>>>>>> distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
>>>>>>>> OR CONDITIONS OF ANY KIND, either express or
>>>>>>>> implied. See the License for the specific language
>>>>>>>> governing permissions and limitations under the
>>>>>>>> License. --> <!-- Note: A "Server" is not itself a
>>>>>>>> "Container", so you may not define subcomponents such
>>>>>>>> as "Valves" at this level. Documentation at
>>>>>>>> /docs/config/server.html --> <Server port="8005"
>>>>>>>> shutdown="SHUTDOWN"> <Listener
>>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener"
>>>>>>>> /> <!-- Security listener. Documentation at
>>>>>>>> /docs/config/listeners.html <Listener
>>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>>>
>>>>>>>>
/>
>>>>>>>> --> <!--APR library loader. Documentation at
>>>>>>>> /docs/apr.html --> <Listener
>>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>>>
>>>>>>>>
SSLEngine="on" />
>>>>>>>> <!-- Prevent memory leaks due to use of particular
>>>>>>>> java/javax APIs--> <Listener
>>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionList
ener"
>>>>>>>>
>>>>>>>>
/>
>>>>>>>> <Listener
>>>>>>>>
>>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleL
istener"
>>>>>>>>
>>>>>>>>
/>
>>>>>>>> <Listener
>>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionLi
stener"
>>>>>>>>
>>>>>>>>
/>
>>>>>>>>
>>>>>>>> <!-- Global JNDI resources Documentation at
>>>>>>>> /docs/jndi-resources-howto.html -->
>>>>>>>> <GlobalNamingResources> <!-- Editable user database
>>>>>>>> that can also be used by UserDatabaseRealm to
>>>>>>>> authenticate users --> <Resource name="UserDatabase"
>>>>>>>> auth="Container"
>>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>>> description="User database that can be updated and
>>>>>>>> saved"
>>>>>>>>
>>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>>>
>>>>>>>>
pathname="conf/tomcat-users.xml" />
>>>>>>>> </GlobalNamingResources>
>>>>>>>>
>>>>>>>> <!-- A "Service" is a collection of one or more
>>>>>>>> "Connectors" that share a single "Container" Note: A
>>>>>>>> "Service" is not itself a "Container", so you may not
>>>>>>>> define subcomponents such as "Valves" at this level.
>>>>>>>> Documentation at /docs/config/service.html -->
>>>>>>>> <Service name="Catalina">
>>>>>>>>
>>>>>>>> <!--The connectors can use a shared executor, you can
>>>>>>>> define one or more named thread pools--> <!--
>>>>>>>> <Executor name="tomcatThreadPool"
>>>>>>>> namePrefix="catalina-exec-" maxThreads="150"
>>>>>>>> minSpareThreads="4"/> -->
>>>>>>>>
>>>>>>>>
>>>>>>>> <!-- A "Connector" represents an endpoint by which
>>>>>>>> requests are received and responses are returned.
>>>>>>>> Documentation at : Java HTTP Connector:
>>>>>>>> /docs/config/http.html (blocking & non-blocking) Java
>>>>>>>> AJP Connector: /docs/config/ajp.html APR (HTTP/AJP)
>>>>>>>> Connector: /docs/apr.html Define a non-SSL/TLS
>>>>>>>> HTTP/1.1 Connector on port 8080 --> <Connector
>>>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>>>> minSpareThreads="25" maxSpareThreads="75"
>>>>>>>> enableLookups="false" disableUploadTimeout="true"
>>>>>>>> acceptCount="100" redirectPort="8443" /> <!-- A
>>>>>>>> "Connector" using the shared thread pool--> <!--
>>>>>>>> <Connector executor="tomcatThreadPool" port="8080"
>>>>>>>> protocol="HTTP/1.1" connectionTimeout="60000"
>>>>>>>> redirectPort="8443" /> --> <!-- Define a SSL/TLS
>>>>>>>> HTTP/1.1 Connector on port 8443 This connector uses
>>>>>>>> the NIO implementation that requires the JSSE style
>>>>>>>> configuration. When using the APR/native
>>>>>>>> implementation, the OpenSSL style configuration is
>>>>>>>> required as described in the APR/native documentation
>>>>>>>> --> <!-- <Connector port="8443"
>>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>>>
>>>>>>>>
maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>>>> secure="true" clientAuth="false" sslProtocol="TLS"
>>>>>>>> /> -->
>>>>>>>>
>>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>>> <Connector port="8009" protocol="AJP/1.3"
>>>>>>>> redirectPort="8443" />
>>>>>>>>
>>>>>>>>
>>>>>>>> <!-- An Engine represents the entry point (within
>>>>>>>> Catalina) that processes every request. The Engine
>>>>>>>> implementation for Tomcat stand alone analyzes the
>>>>>>>> HTTP headers included with the request, and passes
>>>>>>>> them on to the appropriate Host (virtual host).
>>>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>>>
>>>>>>>> <!-- You should set jvmRoute to support
>>>>>>>> load-balancing via AJP ie :
>>>>>>>>
>>>>>>>> --> <Engine name="Catalina" defaultHost="localhost"
>>>>>>>> jvmRoute="TomcatNode1"> <!-- Use the LockOutRealm to
>>>>>>>> prevent attempts to guess user passwords via a
>>>>>>>> brute-force attack --> <Realm
>>>>>>>> className="org.apache.catalina.realm.LockOutRealm">
>>>>>>>> <!-- This Realm uses the UserDatabase configured in
>>>>>>>> the global JNDI resources under the key
>>>>>>>> "UserDatabase". Any edits that are performed against
>>>>>>>> this UserDatabase are immediately available for use
>>>>>>>> by the Realm. --> <Realm
>>>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>>>>
>>>>>>>>
resourceName="UserDatabase"/>
>>>>>>>> </Realm>
>>>>>>>>
>>>>>>>> <Host name="localhost" appBase="webapps"
>>>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>>>> startStopThreads="0"> <!--For clustering, please take
>>>>>>>> a look at documentation at: /docs/cluster-howto.html
>>>>>>>> (simple how to) /docs/config/cluster.html (reference
>>>>>>>> documentation) --> <!-- <Cluster
>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>>>>
>>>>>>>>
- -->
>>>>>>>>
>>>>>>>> <Cluster
>>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>>>>
>>>>>>>>
channelSendOptions="4">
>>>>>>>> <Channel
>>>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>>>>
>>>>>>>>
<Membership
>>>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>>>>
>>>>>>>>
address="228.0.0.4"
>>>>>>>> port="45564" frequency="500" dropTime="9000"/>
>>>>>>>> <Receiver
>>>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver
"
>>>>>>>>
>>>>>>>>
address="auto"
>>>>>>>> port="4000" autoBind="100" selectorTimeout="5000"
>>>>>>>> maxThreads="6"/>
>>>>>>>>
>>>>>>>> <Sender
>>>>>>>>
>>>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTran
smitter">
>>>>>>>>
>>>>>>>>
<Transport
>>>>>>>>
>>>>>>>>
>>>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParal
lelSender"/>
>>>>>>>>
>>>>>>>>
</Sender>
>>>>>>>> <Interceptor
>>>>>>>>
>>>>>>>>
>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFai
lureDetector"/>
>>>>>>>>
>>>>>>>>
<Interceptor
>>>>>>>>
>>>>>>>>
>>>>>>>> className="org.apache.catalina.tribes.group.interceptors.Messag
eDispatch15Interceptor"/>
>>>>>>>>
>>>>>>>>
</Channel>
>>>>>>>>
>>>>>>>> <Valve
>>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>>>
>>>>>>>>
filter=""/>
>>>>>>>> <Valve
>>>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/
>
>>>>>>>>
>>>>>>>>
>>>>>>>>
<Deployer
>>>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>>>>
>>>>>>>>
tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>>>
>>>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>>>
>>>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>>>> watchEnabled="false"/>
>>>>>>>>
>>>>>>>> <ClusterListener
>>>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListene
r"/>
>>>>>>>>
>>>>>>>>
</Cluster>
>>>>>>>> <!-- SingleSignOn valve, share authentication between
>>>>>>>> web applications Documentation at:
>>>>>>>> /docs/config/valve.html --> <!-- <Valve
>>>>>>>> className="org.apache.catalina.authenticator.SingleSignOn"
>>>>>>>> /> -->
>>>>>>>>
>>>>>>>> <!-- Access log processes all example. Documentation
>>>>>>>> at: /docs/config/valve.html Note: The pattern used is
>>>>>>>> equivalent to using pattern="common" --> <Valve
>>>>>>>> className="org.apache.catalina.valves.AccessLogValve"
>>>>>>>>
>>>>>>>>
directory="logs"
>>>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>>>> pattern="%h %l %u %t "%r" %s %b" /> <Valve
>>>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve
"
>>>>>>>>
>>>>>>>>
threshold="900" />
>>>>>>>>
>>>>>>>> </Host> </Engine> </Service> </Server>
>>>>>>>>
>>>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, Andr� Warnier
>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, What is the host OS ? Windows Server 2012
>>>>>>>>>> What is the Tomcat version ? Apache
>>>>>>>>>> Tomcat/8.0.30
>>>>>>>>>>
>>>>>>>>>> Is this problem new ? was this working before ?
>>>>>>>>>> how long ? Since cluster implementation
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I still don't know tribes, but then my non-educated
>>>>>>>>> guess at this point would be that there is
>>>>>>>>> something wrong in your configuration. Can you
>>>>>>>>> copy/paste it here ? (remove sensible things like
>>>>>>>>> passwords, public IP addresses etc..)(but not to
>>>>>>>>> the point of making it uncheckable).
>>>>>>>>>
>>>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Is there actually something listening on that
>>>>>>>>>> address/port ? Tomcat cluster
>>>>>>>>>>
>>>>>>>>>> the Port 4000 is listening and there is no
>>>>>>>>>> disconnection between 2 nodes ping and telnet are
>>>>>>>>>> OK
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, Andr� Warnier
>>>>>>>>>> (tomcat) <aw...@ice-sa.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Plz can i get some help here? This issue is
>>>>>>>>>>>> still occurring and it's filling the log file
>>>>>>>>>>>> in the Production server
>>>>>>>>>>>>
>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi. If you want quick answers, you should
>>>>>>>>>>> provide more information. What is the host OS
>>>>>>>>>>> ? What is the Tomcat version ? Is this problem
>>>>>>>>>>> new ? was this working before ? how long ?
>>>>>>>>>>>
>>>>>>>>>>> I do not know tribes at all, but according to
>>>>>>>>>>> the logfile below, it seems that something is
>>>>>>>>>>> trying to "ping" the address 10.114.43.103,
>>>>>>>>>>> port 4000, and never getting a response (or at
>>>>>>>>>>> least not within 3000ms). Is there actually
>>>>>>>>>>> something listening on that address/port ? The
>>>>>>>>>>> "netstat" command (available both on Linux and
>>>>>>>>>>> Windows) can tell you. If there is something
>>>>>>>>>>> listening there, can it respond to whatever is
>>>>>>>>>>> ping-ing it ? (routing, firewall, ..)
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal
>>>>>>>>>>>> <fa...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Guys, we are facing the below errors in
>>>>>>>>>>>>> Tomcat cluster, please advise
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE
>>>>>>>>>>>>> [GroupChannel-Heartbeat-1]
>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
rtbeat
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>>>> org.apache.catalina.tribes.ChannelException:
>>>>>>>>>>>>> Operation has timed out(3000 ms.).; Faulty
>>>>>>>>>>>>> members:tcp://{10, 114, 43, 103}:4000; at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender
.sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSen
der.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitte
r.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMe
ssage(ChannelCoordinator.java:82)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispa
tchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDe
tector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.se
ndMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
annel.java:233)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupCh
annel.java:186)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChanne
l.java:99)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.pin
g(AbstractReplicatedMap.java:267)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.hea
rtbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(Gr
oupChannel.java:161)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThr
ead.run(GroupChannel.java:697)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
n
>>>>>>>>>>>>>
>>>>>>>>>>>>>
Unable
>>>>>>>>>>>>> to process request in NioReceiver
>>>>>>>>>>>>> java.io.IOException: A non-blocking socket
>>>>>>>>>>>>> operation could not be completed
>>>>>>>>>>>>> immediately at
>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close0(Native
>>>>>>>>>>>>> Method) at
>>>>>>>>>>>>> sun.nio.ch.SocketDispatcher.close(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.SocketChannelImpl.kill(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.select(Unknown
>>>>>>>>>>>>> Source) at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.liste
n(NioReceiver.java:272)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(N
ioReceiver.java:425)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards, Fady
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----------------------------------------------------------
- ----------
>>>>>>>>>>>>
>>>>>>>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ------------------------------------------------------------
- ---------
>>>>>>>>>>>
>>>>>>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -------------------------------------------------------------
- --------
>>>>>>>>>>
>>>>>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------
- -------
>>>>>>>>>
>>>>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------
- ------
>>>>>>>>
>>>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail:
>>>>>>>> users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> -----------------------------------------------------------------
- ----
>>>>>>
>>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
- ---
>>>>>
>>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail:
>>>>> users-help@tomcat.apache.org
>>>>>
>>>>
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
- -
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYmfN4AAoJEBzwKT+lPKRYZuIQAI2NXtOMNcNpd88y2rXRZFCK
p3YN8OBo07UMkT+N6OgmkCrWBGRqIk9SZSUugnWP7N1n5nOFpaio/I2n/2dGOXDB
oYfPA3KVkWaMQenOAujjyodi7Q7ABUurJ07V+D0zfBtPpqE7up9/edl94pq8znuR
KGp1eEKeMlMiZAS9vm5/0ifhhMIDGfE/Y3xjqf+Kb5gHXlA5BrOrlo7+UX26uuqr
qIIaobTCKYaQvYE5HP+9FyOsXCd1oicKz9JIPAoc8A7Yj//n+5Ogtoczyu+mSzxL
X7/JSRdywbyfFo+1bOaecxmx9Bh9UgyziEh2AuXXMGwf17a58ciN8VCqtJcJC3HO
rffZtc9sqm/Tyt5tvo5PgPbpTrvsjeZXMrxUx5QuOjWlkoVOPptACB4LATVmXaoM
lHGw0OBCQgNViI5LXMs59QSdj9/J7tzN+0AakFmO0GM2fHjiFzgHTLGa5QA9yO7b
wIDpCQj3cDFig+4z95ZFHcGTmcUd9u5a6dO/4BPzFvsR/r6kfmH7qV1mwVu1TkEH
4B6EO5AwCwZJNM2asxFktswNoI4Wl2pP7ZjdxW8tIXchJco0iHn/VhnO1kPeNI8x
ws6KXUpZqPp4ylLWYKTLHg9fMc9Ffn6u6jR09o9amEIWnSlinA4R+n9VztWiBR45
h9q0x3f+aKuQp2IE0UBE
=WoKC
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Process PID Protocol local address local port Remote
Address State
Tomcat8.exe 8160 TCP imal14-app2 4000
imal14-app1.albaraka.com.sd ESTABLISHED
On Tue, Feb 7, 2017 at 5:46 PM, Fady Haikal <fa...@gmail.com> wrote:
> Yes there is a ESTABLISHED connection, the replication of sessions is
> working fine (port 4000 is for tomcat cluster) but we also faced this
> error on the log file
>
> On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
>> On 07.02.2017 16:24, Fady Haikal wrote:
>>>
>>> Hi,
>>> telnet IP 4000 is working fine, i installed a tool for network
>>> monitoring at the level of IP and Port and i didnt see any
>>> disconnection,
>>
>>
>> but did you see a *connection* ?
>> I mean, on the pinging node, if you use the Windows "netstat" program, for
>> example as
>> netstat -aon -p TCP
>> you should see a list of connections in the ESTABLISHED state, of which one
>> of the IP/ports should be your target IP:4000 (in the "remote" column).
>> And on the pinged node, this port :4000 should be in the "local" column, in
>> LISTEN mode
>> (and also probably one in the ESTABLISHED state, if they agree.)
>>
>> Is that the case ?
>>
>>
>>
>> and yes i'm sure that no firewall is enabled.
>>>
>>>
>>> I saw some strange think on the server that I have tried to ping the
>>> multicast IP (228.0.0.4) and i get reply from different IPs in the
>>> network, i don't know why and how i get those IPs, after checking with
>>> the network team they told me that those IPs are related to the SAN
>>> storage taking into consideration that the Tomcat servers are not
>>> connected in anyway to that SUN storage.
>>>
>>>
>>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>>> wrote:
>>>>
>>>> Hi.
>>>>
>>>> This is for the Tomcat/Tribes experts on the list.
>>>>
>>>> I know nothing of Tribes, but the on-line documentation seems to say that
>>>> the communication happens over TCP and that the protocol used is not
>>>> encrypted.
>>>> Fady previously tried a standard "ping" and a "telnet" between the two
>>>> nodes, and that is the base for him mentioning that "there is no network
>>>> disconnection" between the nodes.
>>>> Nevertheless, the calling pinging node seems to say that it times out
>>>> without getting a response fom the target node. There is evidently a
>>>> contradiction there.
>>>> So this could still be some kind of network issue.
>>>>
>>>> Considering that the protocol command for this "ping" should be known by
>>>> someone here, would it not be possible to imagine a little program in
>>>> some
>>>> scripting language (or even java, God forbid), which would open a TCP
>>>> channel with the target node IP/port, send such a "ping" message, wait
>>>> for a
>>>> response and report the result ?
>>>> That would at least confirm/deny that the problem is with the network.
>>>>
>>>> The log below does not for example say if the error happens when opening
>>>> the
>>>> TCP communication channel, or after sending the ping message on it,
>>>> (Of course, testing the TCP open could be done with "telnet IP 4000", but
>>>> I
>>>> don't know if Fady tried this).
>>>> Maybe tribes also already contains some löw-level debugging options ?
>>>> wireshark maybe another option, but it has quite a learning curve.
>>>> And this is on Windows.
>>>>
>>>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>>>> Security" is not just dropping TCP packets to/from port 40xx (or from
>>>> "java.exe") ? There are some "network policies" there which can have
>>>> wide-ranging side-effects.
>>>>
>>>>
>>>>
>>>>
>>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>>
>>>>>
>>>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>>>> replicate the key from node to node, please find below the error,
>>>>> taking into consideration that there is no network disconnection
>>>>> between 2 nodes
>>>>>
>>>>>
>>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>>>>> Unable to replicate backup
>>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>>>>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>>>>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>>>>> 114, 43, 102}:4000;
>>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
>>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>>> at
>>>>>
>>>>> org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
>>>>> at
>>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
>>>>> at
>>>>>
>>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
>>>>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>>> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
>>>>> at
>>>>>
>>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
>>>>> at
>>>>>
>>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
>>>>> at
>>>>>
>>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
>>>>> at
>>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
>>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>> at
>>>>>
>>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
>>>>> at
>>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
>>>>> at
>>>>>
>>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>>>>> at
>>>>>
>>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
>>>>> at
>>>>>
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>>>>> at
>>>>>
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
>>>>> at
>>>>>
>>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
>>>>> at
>>>>>
>>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>>> at
>>>>>
>>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>
>>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Fady,
>>>>>>
>>>>>> Sorry for top posting.
>>>>>>
>>>>>> If I remember correctly, the Cluster Element goes into the Container
>>>>>> and
>>>>>> not the Host.
>>>>>> Plus I see in our (working) case, a DeltaManager and a
>>>>>> JvmRouteSessionIDBinderListener
>>>>>>
>>>>>> <Manager
>>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>>> expireSessionsOnShutdown="false"
>>>>>> notifyListenersOnReplication="true"/>
>>>>>> ...
>>>>>> <ClusterListener
>>>>>>
>>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>>>>>
>>>>>> Besides this, only ports, limits and values are different.
>>>>>>
>>>>>> You may want to filter out the replication for static resources as gifs
>>>>>> jpg or css.
>>>>>> <Valve
>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>
>>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>> below is the server.xml configuration, as mentioened earlier the issue
>>>>>>> is related to the cluster configuration, and as per my research i can
>>>>>>> see that some users are facing the same issue but i didnt found the
>>>>>>> solution of it
>>>>>>>
>>>>>>>
>>>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>>>> <!--
>>>>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>>> contributor license agreements. See the NOTICE file distributed
>>>>>>> with
>>>>>>> this work for additional information regarding copyright
>>>>>>> ownership.
>>>>>>> The ASF licenses this file to You under the Apache License,
>>>>>>> Version
>>>>>>> 2.0
>>>>>>> (the "License"); you may not use this file except in compliance
>>>>>>> with
>>>>>>> the License. You may obtain a copy of the License at
>>>>>>>
>>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>
>>>>>>> Unless required by applicable law or agreed to in writing,
>>>>>>> software
>>>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>>> implied.
>>>>>>> See the License for the specific language governing permissions
>>>>>>> and
>>>>>>> limitations under the License.
>>>>>>> -->
>>>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>>>>> define subcomponents such as "Valves" at this level.
>>>>>>> Documentation at /docs/config/server.html
>>>>>>> -->
>>>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.startup.VersionLoggerListener" />
>>>>>>> <!-- Security listener. Documentation at
>>>>>>> /docs/config/listeners.html
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>>> />
>>>>>>> -->
>>>>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>>> SSLEngine="on" />
>>>>>>> <!-- Prevent memory leaks due to use of particular java/javax
>>>>>>> APIs-->
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>>>>>> />
>>>>>>> <Listener
>>>>>>>
>>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>>>>> />
>>>>>>> <Listener
>>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>>>>>> />
>>>>>>>
>>>>>>> <!-- Global JNDI resources
>>>>>>> Documentation at /docs/jndi-resources-howto.html
>>>>>>> -->
>>>>>>> <GlobalNamingResources>
>>>>>>> <!-- Editable user database that can also be used by
>>>>>>> UserDatabaseRealm to authenticate users
>>>>>>> -->
>>>>>>> <Resource name="UserDatabase" auth="Container"
>>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>>> description="User database that can be updated and
>>>>>>> saved"
>>>>>>>
>>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>>> pathname="conf/tomcat-users.xml" />
>>>>>>> </GlobalNamingResources>
>>>>>>>
>>>>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>>>>> share
>>>>>>> a single "Container" Note: A "Service" is not itself a
>>>>>>> "Container",
>>>>>>> so you may not define subcomponents such as "Valves" at this
>>>>>>> level.
>>>>>>> Documentation at /docs/config/service.html
>>>>>>> -->
>>>>>>> <Service name="Catalina">
>>>>>>>
>>>>>>> <!--The connectors can use a shared executor, you can define one
>>>>>>> or more named thread pools-->
>>>>>>> <!--
>>>>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>>>> maxThreads="150" minSpareThreads="4"/>
>>>>>>> -->
>>>>>>>
>>>>>>>
>>>>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>>>>> received
>>>>>>> and responses are returned. Documentation at :
>>>>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>>>>> non-blocking)
>>>>>>> Java AJP Connector: /docs/config/ajp.html
>>>>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>>>>> -->
>>>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>>>>> <!-- A "Connector" using the shared thread pool-->
>>>>>>> <!--
>>>>>>> <Connector executor="tomcatThreadPool"
>>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>>> connectionTimeout="60000"
>>>>>>> redirectPort="8443" />
>>>>>>> -->
>>>>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>>>>> This connector uses the NIO implementation that requires
>>>>>>> the
>>>>>>> JSSE
>>>>>>> style configuration. When using the APR/native
>>>>>>> implementation,
>>>>>>> the
>>>>>>> OpenSSL style configuration is required as described in the
>>>>>>> APR/native
>>>>>>> documentation -->
>>>>>>> <!--
>>>>>>> <Connector port="8443"
>>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>>> secure="true"
>>>>>>> clientAuth="false" sslProtocol="TLS" />
>>>>>>> -->
>>>>>>>
>>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>>>
>>>>>>>
>>>>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>>>>> processes
>>>>>>> every request. The Engine implementation for Tomcat stand
>>>>>>> alone
>>>>>>> analyzes the HTTP headers included with the request, and
>>>>>>> passes them
>>>>>>> on to the appropriate Host (virtual host).
>>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>>
>>>>>>> <!-- You should set jvmRoute to support load-balancing via AJP
>>>>>>> ie :
>>>>>>>
>>>>>>> -->
>>>>>>> <Engine name="Catalina" defaultHost="localhost"
>>>>>>> jvmRoute="TomcatNode1">
>>>>>>> <!-- Use the LockOutRealm to prevent attempts to guess user
>>>>>>> passwords
>>>>>>> via a brute-force attack -->
>>>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>>>>> <!-- This Realm uses the UserDatabase configured in the
>>>>>>> global
>>>>>>> JNDI
>>>>>>> resources under the key "UserDatabase". Any edits
>>>>>>> that are performed against this UserDatabase are
>>>>>>> immediately
>>>>>>> available for use by the Realm. -->
>>>>>>> <Realm
>>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>>> resourceName="UserDatabase"/>
>>>>>>> </Realm>
>>>>>>>
>>>>>>> <Host name="localhost" appBase="webapps"
>>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>>> startStopThreads="0">
>>>>>>> <!--For clustering, please take a look at documentation at:
>>>>>>> /docs/cluster-howto.html (simple how to)
>>>>>>> /docs/config/cluster.html (reference documentation) -->
>>>>>>> <!--
>>>>>>> <Cluster
>>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>>> -->
>>>>>>>
>>>>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>>> channelSendOptions="4">
>>>>>>> <Channel
>>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>>> <Membership
>>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>>> address="228.0.0.4"
>>>>>>> port="45564"
>>>>>>> frequency="500"
>>>>>>> dropTime="9000"/>
>>>>>>> <Receiver
>>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>>>>> address="auto"
>>>>>>> port="4000"
>>>>>>> autoBind="100"
>>>>>>> selectorTimeout="5000"
>>>>>>> maxThreads="6"/>
>>>>>>>
>>>>>>> <Sender
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>>>>>> <Transport
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>>>>>> </Sender>
>>>>>>> <Interceptor
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>>>>>> <Interceptor
>>>>>>>
>>>>>>>
>>>>>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>>>>>> </Channel>
>>>>>>>
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>>> filter=""/>
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>>>>
>>>>>>> <Deployer
>>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>>
>>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>>
>>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>>> watchEnabled="false"/>
>>>>>>>
>>>>>>> <ClusterListener
>>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>>>>> </Cluster>
>>>>>>> <!-- SingleSignOn valve, share authentication between web
>>>>>>> applications
>>>>>>> Documentation at: /docs/config/valve.html -->
>>>>>>> <!--
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.authenticator.SingleSignOn" />
>>>>>>> -->
>>>>>>>
>>>>>>> <!-- Access log processes all example.
>>>>>>> Documentation at: /docs/config/valve.html
>>>>>>> Note: The pattern used is equivalent to using
>>>>>>> pattern="common" -->
>>>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>>>> directory="logs"
>>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>>>>> <Valve
>>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>>>> threshold="900" />
>>>>>>>
>>>>>>> </Host>
>>>>>>> </Engine>
>>>>>>> </Service>
>>>>>>> </Server>
>>>>>>>
>>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> What is the host OS ? Windows Server 2012
>>>>>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>>>>>
>>>>>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>>>>>> cluster implementation
>>>>>>>>>
>>>>>>>>
>>>>>>>> I still don't know tribes, but then my non-educated guess at this
>>>>>>>> point
>>>>>>>> would be that there is something wrong in your configuration.
>>>>>>>> Can you copy/paste it here ? (remove sensible things like passwords,
>>>>>>>> public
>>>>>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>>>>>
>>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>>>>>> cluster
>>>>>>>>>
>>>>>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>>>>>> nodes ping and telnet are OK
>>>>>>>>>
>>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)
>>>>>>>>> <aw...@ice-sa.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Plz can i get some help here?
>>>>>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>>>>>> Production server
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Fady
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi.
>>>>>>>>>> If you want quick answers, you should provide more information.
>>>>>>>>>> What is the host OS ?
>>>>>>>>>> What is the Tomcat version ?
>>>>>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>>>>>
>>>>>>>>>> I do not know tribes at all, but according to the logfile below, it
>>>>>>>>>> seems
>>>>>>>>>> that something is trying to "ping" the address 10.114.43.103, port
>>>>>>>>>> 4000,
>>>>>>>>>> and
>>>>>>>>>> never getting a response (or at least not within 3000ms).
>>>>>>>>>> Is there actually something listening on that address/port ?
>>>>>>>>>> The "netstat" command (available both on Linux and Windows) can
>>>>>>>>>> tell
>>>>>>>>>> you.
>>>>>>>>>> If there is something listening there, can it respond to whatever
>>>>>>>>>> is
>>>>>>>>>> ping-ing it ?
>>>>>>>>>> (routing, firewall, ..)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please
>>>>>>>>>>>> advise
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has
>>>>>>>>>>>> timed
>>>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>>>>>>>>>>>> Unable
>>>>>>>>>>>> to
>>>>>>>>>>>> process request in NioReceiver
>>>>>>>>>>>> java.io.IOException: A non-blocking socket operation could
>>>>>>>>>>>> not
>>>>>>>>>>>> be
>>>>>>>>>>>> completed immediately
>>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Fady
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Yes there is a ESTABLISHED connection, the replication of sessions is
working fine (port 4000 is for tomcat cluster) but we also faced this
error on the log file
On Tue, Feb 7, 2017 at 5:44 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
> On 07.02.2017 16:24, Fady Haikal wrote:
>>
>> Hi,
>> telnet IP 4000 is working fine, i installed a tool for network
>> monitoring at the level of IP and Port and i didnt see any
>> disconnection,
>
>
> but did you see a *connection* ?
> I mean, on the pinging node, if you use the Windows "netstat" program, for
> example as
> netstat -aon -p TCP
> you should see a list of connections in the ESTABLISHED state, of which one
> of the IP/ports should be your target IP:4000 (in the "remote" column).
> And on the pinged node, this port :4000 should be in the "local" column, in
> LISTEN mode
> (and also probably one in the ESTABLISHED state, if they agree.)
>
> Is that the case ?
>
>
>
> and yes i'm sure that no firewall is enabled.
>>
>>
>> I saw some strange think on the server that I have tried to ping the
>> multicast IP (228.0.0.4) and i get reply from different IPs in the
>> network, i don't know why and how i get those IPs, after checking with
>> the network team they told me that those IPs are related to the SAN
>> storage taking into consideration that the Tomcat servers are not
>> connected in anyway to that SUN storage.
>>
>>
>> On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>> wrote:
>>>
>>> Hi.
>>>
>>> This is for the Tomcat/Tribes experts on the list.
>>>
>>> I know nothing of Tribes, but the on-line documentation seems to say that
>>> the communication happens over TCP and that the protocol used is not
>>> encrypted.
>>> Fady previously tried a standard "ping" and a "telnet" between the two
>>> nodes, and that is the base for him mentioning that "there is no network
>>> disconnection" between the nodes.
>>> Nevertheless, the calling pinging node seems to say that it times out
>>> without getting a response fom the target node. There is evidently a
>>> contradiction there.
>>> So this could still be some kind of network issue.
>>>
>>> Considering that the protocol command for this "ping" should be known by
>>> someone here, would it not be possible to imagine a little program in
>>> some
>>> scripting language (or even java, God forbid), which would open a TCP
>>> channel with the target node IP/port, send such a "ping" message, wait
>>> for a
>>> response and report the result ?
>>> That would at least confirm/deny that the problem is with the network.
>>>
>>> The log below does not for example say if the error happens when opening
>>> the
>>> TCP communication channel, or after sending the ping message on it,
>>> (Of course, testing the TCP open could be done with "telnet IP 4000", but
>>> I
>>> don't know if Fady tried this).
>>> Maybe tribes also already contains some löw-level debugging options ?
>>> wireshark maybe another option, but it has quite a learning curve.
>>> And this is on Windows.
>>>
>>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>>> Security" is not just dropping TCP packets to/from port 40xx (or from
>>> "java.exe") ? There are some "network policies" there which can have
>>> wide-ranging side-effects.
>>>
>>>
>>>
>>>
>>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>>
>>>>
>>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>>> replicate the key from node to node, please find below the error,
>>>> taking into consideration that there is no network disconnection
>>>> between 2 nodes
>>>>
>>>>
>>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>>>> Unable to replicate backup
>>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>>>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>>>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>>>> 114, 43, 102}:4000;
>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
>>>> at
>>>>
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
>>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>>> at
>>>>
>>>> org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
>>>> at
>>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
>>>> at
>>>>
>>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
>>>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>>> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
>>>> at
>>>>
>>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
>>>> at
>>>>
>>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
>>>> at
>>>>
>>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
>>>> at
>>>>
>>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
>>>> at
>>>>
>>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
>>>> at
>>>> org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
>>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>> at
>>>>
>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
>>>> at
>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
>>>> at
>>>>
>>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
>>>> at
>>>>
>>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
>>>> at
>>>>
>>>> org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
>>>> at
>>>>
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
>>>> at
>>>>
>>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
>>>> at
>>>>
>>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
>>>> at
>>>>
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>>>> at
>>>>
>>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>>>> at
>>>>
>>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
>>>> at
>>>>
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>>>> at
>>>>
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
>>>> at
>>>>
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
>>>> at
>>>>
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
>>>> at
>>>>
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
>>>> at
>>>>
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>>> at
>>>>
>>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>>> at java.lang.Thread.run(Unknown Source)
>>>>
>>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Fady,
>>>>>
>>>>> Sorry for top posting.
>>>>>
>>>>> If I remember correctly, the Cluster Element goes into the Container
>>>>> and
>>>>> not the Host.
>>>>> Plus I see in our (working) case, a DeltaManager and a
>>>>> JvmRouteSessionIDBinderListener
>>>>>
>>>>> <Manager
>>>>> className="org.apache.catalina.ha.session.DeltaManager"
>>>>> expireSessionsOnShutdown="false"
>>>>> notifyListenersOnReplication="true"/>
>>>>> ...
>>>>> <ClusterListener
>>>>>
>>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>>>>
>>>>> Besides this, only ports, limits and values are different.
>>>>>
>>>>> You may want to filter out the replication for static resources as gifs
>>>>> jpg or css.
>>>>> <Valve
>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>
>>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>>>>>
>>>>> Best regards
>>>>>
>>>>> Peter
>>>>>
>>>>>> below is the server.xml configuration, as mentioened earlier the issue
>>>>>> is related to the cluster configuration, and as per my research i can
>>>>>> see that some users are facing the same issue but i didnt found the
>>>>>> solution of it
>>>>>>
>>>>>>
>>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>>> <!--
>>>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>> contributor license agreements. See the NOTICE file distributed
>>>>>> with
>>>>>> this work for additional information regarding copyright
>>>>>> ownership.
>>>>>> The ASF licenses this file to You under the Apache License,
>>>>>> Version
>>>>>> 2.0
>>>>>> (the "License"); you may not use this file except in compliance
>>>>>> with
>>>>>> the License. You may obtain a copy of the License at
>>>>>>
>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>
>>>>>> Unless required by applicable law or agreed to in writing,
>>>>>> software
>>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>> implied.
>>>>>> See the License for the specific language governing permissions
>>>>>> and
>>>>>> limitations under the License.
>>>>>> -->
>>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>>>> define subcomponents such as "Valves" at this level.
>>>>>> Documentation at /docs/config/server.html
>>>>>> -->
>>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>>> <Listener
>>>>>> className="org.apache.catalina.startup.VersionLoggerListener" />
>>>>>> <!-- Security listener. Documentation at
>>>>>> /docs/config/listeners.html
>>>>>> <Listener
>>>>>> className="org.apache.catalina.security.SecurityListener"
>>>>>> />
>>>>>> -->
>>>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>>>> <Listener
>>>>>> className="org.apache.catalina.core.AprLifecycleListener"
>>>>>> SSLEngine="on" />
>>>>>> <!-- Prevent memory leaks due to use of particular java/javax
>>>>>> APIs-->
>>>>>> <Listener
>>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>>>>> />
>>>>>> <Listener
>>>>>>
>>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>>>> />
>>>>>> <Listener
>>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>>>>> />
>>>>>>
>>>>>> <!-- Global JNDI resources
>>>>>> Documentation at /docs/jndi-resources-howto.html
>>>>>> -->
>>>>>> <GlobalNamingResources>
>>>>>> <!-- Editable user database that can also be used by
>>>>>> UserDatabaseRealm to authenticate users
>>>>>> -->
>>>>>> <Resource name="UserDatabase" auth="Container"
>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>> description="User database that can be updated and
>>>>>> saved"
>>>>>>
>>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>>> pathname="conf/tomcat-users.xml" />
>>>>>> </GlobalNamingResources>
>>>>>>
>>>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>>>> share
>>>>>> a single "Container" Note: A "Service" is not itself a
>>>>>> "Container",
>>>>>> so you may not define subcomponents such as "Valves" at this
>>>>>> level.
>>>>>> Documentation at /docs/config/service.html
>>>>>> -->
>>>>>> <Service name="Catalina">
>>>>>>
>>>>>> <!--The connectors can use a shared executor, you can define one
>>>>>> or more named thread pools-->
>>>>>> <!--
>>>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>>> maxThreads="150" minSpareThreads="4"/>
>>>>>> -->
>>>>>>
>>>>>>
>>>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>>>> received
>>>>>> and responses are returned. Documentation at :
>>>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>>>> non-blocking)
>>>>>> Java AJP Connector: /docs/config/ajp.html
>>>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>>>> -->
>>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>>>> <!-- A "Connector" using the shared thread pool-->
>>>>>> <!--
>>>>>> <Connector executor="tomcatThreadPool"
>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>> connectionTimeout="60000"
>>>>>> redirectPort="8443" />
>>>>>> -->
>>>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>>>> This connector uses the NIO implementation that requires
>>>>>> the
>>>>>> JSSE
>>>>>> style configuration. When using the APR/native
>>>>>> implementation,
>>>>>> the
>>>>>> OpenSSL style configuration is required as described in the
>>>>>> APR/native
>>>>>> documentation -->
>>>>>> <!--
>>>>>> <Connector port="8443"
>>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>> secure="true"
>>>>>> clientAuth="false" sslProtocol="TLS" />
>>>>>> -->
>>>>>>
>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>>
>>>>>>
>>>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>>>> processes
>>>>>> every request. The Engine implementation for Tomcat stand
>>>>>> alone
>>>>>> analyzes the HTTP headers included with the request, and
>>>>>> passes them
>>>>>> on to the appropriate Host (virtual host).
>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>
>>>>>> <!-- You should set jvmRoute to support load-balancing via AJP
>>>>>> ie :
>>>>>>
>>>>>> -->
>>>>>> <Engine name="Catalina" defaultHost="localhost"
>>>>>> jvmRoute="TomcatNode1">
>>>>>> <!-- Use the LockOutRealm to prevent attempts to guess user
>>>>>> passwords
>>>>>> via a brute-force attack -->
>>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>>>> <!-- This Realm uses the UserDatabase configured in the
>>>>>> global
>>>>>> JNDI
>>>>>> resources under the key "UserDatabase". Any edits
>>>>>> that are performed against this UserDatabase are
>>>>>> immediately
>>>>>> available for use by the Realm. -->
>>>>>> <Realm
>>>>>> className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>> resourceName="UserDatabase"/>
>>>>>> </Realm>
>>>>>>
>>>>>> <Host name="localhost" appBase="webapps"
>>>>>> unpackWARs="true" autoDeploy="true"
>>>>>> startStopThreads="0">
>>>>>> <!--For clustering, please take a look at documentation at:
>>>>>> /docs/cluster-howto.html (simple how to)
>>>>>> /docs/config/cluster.html (reference documentation) -->
>>>>>> <!--
>>>>>> <Cluster
>>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>>> -->
>>>>>>
>>>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>> channelSendOptions="4">
>>>>>> <Channel
>>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>>> <Membership
>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>> address="228.0.0.4"
>>>>>> port="45564"
>>>>>> frequency="500"
>>>>>> dropTime="9000"/>
>>>>>> <Receiver
>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>>>> address="auto"
>>>>>> port="4000"
>>>>>> autoBind="100"
>>>>>> selectorTimeout="5000"
>>>>>> maxThreads="6"/>
>>>>>>
>>>>>> <Sender
>>>>>>
>>>>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>>>>> <Transport
>>>>>>
>>>>>>
>>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>>>>> </Sender>
>>>>>> <Interceptor
>>>>>>
>>>>>>
>>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>>>>> <Interceptor
>>>>>>
>>>>>>
>>>>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>>>>> </Channel>
>>>>>>
>>>>>> <Valve
>>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>>> filter=""/>
>>>>>> <Valve
>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>>>
>>>>>> <Deployer
>>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>>
>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>>
>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>> watchEnabled="false"/>
>>>>>>
>>>>>> <ClusterListener
>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>>>> </Cluster>
>>>>>> <!-- SingleSignOn valve, share authentication between web
>>>>>> applications
>>>>>> Documentation at: /docs/config/valve.html -->
>>>>>> <!--
>>>>>> <Valve
>>>>>> className="org.apache.catalina.authenticator.SingleSignOn" />
>>>>>> -->
>>>>>>
>>>>>> <!-- Access log processes all example.
>>>>>> Documentation at: /docs/config/valve.html
>>>>>> Note: The pattern used is equivalent to using
>>>>>> pattern="common" -->
>>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>>> directory="logs"
>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>>>> <Valve
>>>>>> className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>>> threshold="900" />
>>>>>>
>>>>>> </Host>
>>>>>> </Engine>
>>>>>> </Service>
>>>>>> </Server>
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> What is the host OS ? Windows Server 2012
>>>>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>>>>
>>>>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>>>>> cluster implementation
>>>>>>>>
>>>>>>>
>>>>>>> I still don't know tribes, but then my non-educated guess at this
>>>>>>> point
>>>>>>> would be that there is something wrong in your configuration.
>>>>>>> Can you copy/paste it here ? (remove sensible things like passwords,
>>>>>>> public
>>>>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>>>>
>>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>>>>> cluster
>>>>>>>>
>>>>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>>>>> nodes ping and telnet are OK
>>>>>>>>
>>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat)
>>>>>>>> <aw...@ice-sa.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Plz can i get some help here?
>>>>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>>>>> Production server
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Fady
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>> If you want quick answers, you should provide more information.
>>>>>>>>> What is the host OS ?
>>>>>>>>> What is the Tomcat version ?
>>>>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>>>>
>>>>>>>>> I do not know tribes at all, but according to the logfile below, it
>>>>>>>>> seems
>>>>>>>>> that something is trying to "ping" the address 10.114.43.103, port
>>>>>>>>> 4000,
>>>>>>>>> and
>>>>>>>>> never getting a response (or at least not within 3000ms).
>>>>>>>>> Is there actually something listening on that address/port ?
>>>>>>>>> The "netstat" command (available both on Linux and Windows) can
>>>>>>>>> tell
>>>>>>>>> you.
>>>>>>>>> If there is something listening there, can it respond to whatever
>>>>>>>>> is
>>>>>>>>> ping-ing it ?
>>>>>>>>> (routing, firewall, ..)
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please
>>>>>>>>>>> advise
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has
>>>>>>>>>>> timed
>>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>>>>>>>>>>> Unable
>>>>>>>>>>> to
>>>>>>>>>>> process request in NioReceiver
>>>>>>>>>>> java.io.IOException: A non-blocking socket operation could
>>>>>>>>>>> not
>>>>>>>>>>> be
>>>>>>>>>>> completed immediately
>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Fady
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 07.02.2017 16:24, Fady Haikal wrote:
> Hi,
> telnet IP 4000 is working fine, i installed a tool for network
> monitoring at the level of IP and Port and i didnt see any
> disconnection,
but did you see a *connection* ?
I mean, on the pinging node, if you use the Windows "netstat" program, for example as
netstat -aon -p TCP
you should see a list of connections in the ESTABLISHED state, of which one of the
IP/ports should be your target IP:4000 (in the "remote" column).
And on the pinged node, this port :4000 should be in the "local" column, in LISTEN mode
(and also probably one in the ESTABLISHED state, if they agree.)
Is that the case ?
and yes i'm sure that no firewall is enabled.
>
> I saw some strange think on the server that I have tried to ping the
> multicast IP (228.0.0.4) and i get reply from different IPs in the
> network, i don't know why and how i get those IPs, after checking with
> the network team they told me that those IPs are related to the SAN
> storage taking into consideration that the Tomcat servers are not
> connected in anyway to that SUN storage.
>
>
> On Tue, Feb 7, 2017 at 4:51 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com> wrote:
>> Hi.
>>
>> This is for the Tomcat/Tribes experts on the list.
>>
>> I know nothing of Tribes, but the on-line documentation seems to say that
>> the communication happens over TCP and that the protocol used is not
>> encrypted.
>> Fady previously tried a standard "ping" and a "telnet" between the two
>> nodes, and that is the base for him mentioning that "there is no network
>> disconnection" between the nodes.
>> Nevertheless, the calling pinging node seems to say that it times out
>> without getting a response fom the target node. There is evidently a
>> contradiction there.
>> So this could still be some kind of network issue.
>>
>> Considering that the protocol command for this "ping" should be known by
>> someone here, would it not be possible to imagine a little program in some
>> scripting language (or even java, God forbid), which would open a TCP
>> channel with the target node IP/port, send such a "ping" message, wait for a
>> response and report the result ?
>> That would at least confirm/deny that the problem is with the network.
>>
>> The log below does not for example say if the error happens when opening the
>> TCP communication channel, or after sending the ping message on it,
>> (Of course, testing the TCP open could be done with "telnet IP 4000", but I
>> don't know if Fady tried this).
>> Maybe tribes also already contains some l�w-level debugging options ?
>> wireshark maybe another option, but it has quite a learning curve.
>> And this is on Windows.
>>
>> By the way Fady, are you sure that your "Windows Firewall with Enhanced
>> Security" is not just dropping TCP packets to/from port 40xx (or from
>> "java.exe") ? There are some "network policies" there which can have
>> wide-ranging side-effects.
>>
>>
>>
>>
>> On 07.02.2017 14:42, Fady Haikal wrote:
>>>
>>> Hi, issue still not fixed. Tomcat session replication is not able to
>>> replicate the key from node to node, please find below the error,
>>> taking into consideration that there is no network disconnection
>>> between 2 nodes
>>>
>>>
>>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>>> Unable to replicate backup
>>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>>> 114, 43, 102}:4000;
>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>> at
>>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
>>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>>> at
>>> org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
>>> at
>>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
>>> at
>>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
>>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>>> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
>>> at
>>> org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
>>> at
>>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
>>> at
>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
>>> at
>>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
>>> at
>>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
>>> at
>>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
>>> at
>>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
>>> at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
>>> at
>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
>>> at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
>>> at
>>> org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
>>> at
>>> org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
>>> at
>>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
>>> at
>>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>>> at
>>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>>> at
>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
>>> at
>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
>>> at
>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>> at
>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com>
>>> wrote:
>>>>
>>>> Fady,
>>>>
>>>> Sorry for top posting.
>>>>
>>>> If I remember correctly, the Cluster Element goes into the Container and
>>>> not the Host.
>>>> Plus I see in our (working) case, a DeltaManager and a
>>>> JvmRouteSessionIDBinderListener
>>>>
>>>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>>>> expireSessionsOnShutdown="false"
>>>> notifyListenersOnReplication="true"/>
>>>> ...
>>>> <ClusterListener
>>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>>>
>>>> Besides this, only ports, limits and values are different.
>>>>
>>>> You may want to filter out the replication for static resources as gifs
>>>> jpg or css.
>>>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>>>>
>>>> Best regards
>>>>
>>>> Peter
>>>>
>>>>> below is the server.xml configuration, as mentioened earlier the issue
>>>>> is related to the cluster configuration, and as per my research i can
>>>>> see that some users are facing the same issue but i didnt found the
>>>>> solution of it
>>>>>
>>>>>
>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>> <!--
>>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>>> contributor license agreements. See the NOTICE file distributed with
>>>>> this work for additional information regarding copyright ownership.
>>>>> The ASF licenses this file to You under the Apache License, Version
>>>>> 2.0
>>>>> (the "License"); you may not use this file except in compliance with
>>>>> the License. You may obtain a copy of the License at
>>>>>
>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>
>>>>> Unless required by applicable law or agreed to in writing, software
>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>> implied.
>>>>> See the License for the specific language governing permissions and
>>>>> limitations under the License.
>>>>> -->
>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>>> define subcomponents such as "Valves" at this level.
>>>>> Documentation at /docs/config/server.html
>>>>> -->
>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>> <Listener
>>>>> className="org.apache.catalina.startup.VersionLoggerListener" />
>>>>> <!-- Security listener. Documentation at /docs/config/listeners.html
>>>>> <Listener className="org.apache.catalina.security.SecurityListener"
>>>>> />
>>>>> -->
>>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>>>>> SSLEngine="on" />
>>>>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>>>>> <Listener
>>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>>>> />
>>>>> <Listener
>>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>>> />
>>>>> <Listener
>>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>>>> />
>>>>>
>>>>> <!-- Global JNDI resources
>>>>> Documentation at /docs/jndi-resources-howto.html
>>>>> -->
>>>>> <GlobalNamingResources>
>>>>> <!-- Editable user database that can also be used by
>>>>> UserDatabaseRealm to authenticate users
>>>>> -->
>>>>> <Resource name="UserDatabase" auth="Container"
>>>>> type="org.apache.catalina.UserDatabase"
>>>>> description="User database that can be updated and saved"
>>>>>
>>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>>> pathname="conf/tomcat-users.xml" />
>>>>> </GlobalNamingResources>
>>>>>
>>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>>> share
>>>>> a single "Container" Note: A "Service" is not itself a
>>>>> "Container",
>>>>> so you may not define subcomponents such as "Valves" at this
>>>>> level.
>>>>> Documentation at /docs/config/service.html
>>>>> -->
>>>>> <Service name="Catalina">
>>>>>
>>>>> <!--The connectors can use a shared executor, you can define one
>>>>> or more named thread pools-->
>>>>> <!--
>>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>> maxThreads="150" minSpareThreads="4"/>
>>>>> -->
>>>>>
>>>>>
>>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>>> received
>>>>> and responses are returned. Documentation at :
>>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>>> non-blocking)
>>>>> Java AJP Connector: /docs/config/ajp.html
>>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>>> -->
>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>> connectionTimeout="60000" maxThreads="500"
>>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>>> <!-- A "Connector" using the shared thread pool-->
>>>>> <!--
>>>>> <Connector executor="tomcatThreadPool"
>>>>> port="8080" protocol="HTTP/1.1"
>>>>> connectionTimeout="60000"
>>>>> redirectPort="8443" />
>>>>> -->
>>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>>> This connector uses the NIO implementation that requires the
>>>>> JSSE
>>>>> style configuration. When using the APR/native implementation,
>>>>> the
>>>>> OpenSSL style configuration is required as described in the
>>>>> APR/native
>>>>> documentation -->
>>>>> <!--
>>>>> <Connector port="8443"
>>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>> secure="true"
>>>>> clientAuth="false" sslProtocol="TLS" />
>>>>> -->
>>>>>
>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>
>>>>>
>>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>>> processes
>>>>> every request. The Engine implementation for Tomcat stand
>>>>> alone
>>>>> analyzes the HTTP headers included with the request, and
>>>>> passes them
>>>>> on to the appropriate Host (virtual host).
>>>>> Documentation at /docs/config/engine.html -->
>>>>>
>>>>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>>>>
>>>>> -->
>>>>> <Engine name="Catalina" defaultHost="localhost"
>>>>> jvmRoute="TomcatNode1">
>>>>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>>>>> via a brute-force attack -->
>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>>> <!-- This Realm uses the UserDatabase configured in the global
>>>>> JNDI
>>>>> resources under the key "UserDatabase". Any edits
>>>>> that are performed against this UserDatabase are
>>>>> immediately
>>>>> available for use by the Realm. -->
>>>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>> resourceName="UserDatabase"/>
>>>>> </Realm>
>>>>>
>>>>> <Host name="localhost" appBase="webapps"
>>>>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>>>> <!--For clustering, please take a look at documentation at:
>>>>> /docs/cluster-howto.html (simple how to)
>>>>> /docs/config/cluster.html (reference documentation) -->
>>>>> <!--
>>>>> <Cluster
>>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>>> -->
>>>>>
>>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>> channelSendOptions="4">
>>>>> <Channel
>>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>>> <Membership
>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>> address="228.0.0.4"
>>>>> port="45564"
>>>>> frequency="500"
>>>>> dropTime="9000"/>
>>>>> <Receiver
>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>>> address="auto"
>>>>> port="4000"
>>>>> autoBind="100"
>>>>> selectorTimeout="5000"
>>>>> maxThreads="6"/>
>>>>>
>>>>> <Sender
>>>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>>>> <Transport
>>>>>
>>>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>>>> </Sender>
>>>>> <Interceptor
>>>>>
>>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>>>> <Interceptor
>>>>>
>>>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>>>> </Channel>
>>>>>
>>>>> <Valve
>>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>>> filter=""/>
>>>>> <Valve
>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>>
>>>>> <Deployer
>>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>> watchEnabled="false"/>
>>>>>
>>>>> <ClusterListener
>>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>>> </Cluster>
>>>>> <!-- SingleSignOn valve, share authentication between web
>>>>> applications
>>>>> Documentation at: /docs/config/valve.html -->
>>>>> <!--
>>>>> <Valve
>>>>> className="org.apache.catalina.authenticator.SingleSignOn" />
>>>>> -->
>>>>>
>>>>> <!-- Access log processes all example.
>>>>> Documentation at: /docs/config/valve.html
>>>>> Note: The pattern used is equivalent to using
>>>>> pattern="common" -->
>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>> directory="logs"
>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>> threshold="900" />
>>>>>
>>>>> </Host>
>>>>> </Engine>
>>>>> </Service>
>>>>> </Server>
>>>>>
>>>>> On Mon, Feb 6, 2017 at 6:51 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com>
>>>>> wrote:
>>>>>>
>>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>> What is the host OS ? Windows Server 2012
>>>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>>>
>>>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>>>> cluster implementation
>>>>>>>
>>>>>>
>>>>>> I still don't know tribes, but then my non-educated guess at this point
>>>>>> would be that there is something wrong in your configuration.
>>>>>> Can you copy/paste it here ? (remove sensible things like passwords,
>>>>>> public
>>>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>>>
>>>>>> Then maybe some tribes-specialist can take over ?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>>>> cluster
>>>>>>>
>>>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>>>> nodes ping and telnet are OK
>>>>>>>
>>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Plz can i get some help here?
>>>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>>>> Production server
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Fady
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi.
>>>>>>>> If you want quick answers, you should provide more information.
>>>>>>>> What is the host OS ?
>>>>>>>> What is the Tomcat version ?
>>>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>>>
>>>>>>>> I do not know tribes at all, but according to the logfile below, it
>>>>>>>> seems
>>>>>>>> that something is trying to "ping" the address 10.114.43.103, port
>>>>>>>> 4000,
>>>>>>>> and
>>>>>>>> never getting a response (or at least not within 3000ms).
>>>>>>>> Is there actually something listening on that address/port ?
>>>>>>>> The "netstat" command (available both on Linux and Windows) can tell
>>>>>>>> you.
>>>>>>>> If there is something listening there, can it respond to whatever is
>>>>>>>> ping-ing it ?
>>>>>>>> (routing, firewall, ..)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please
>>>>>>>>>> advise
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has
>>>>>>>>>> timed
>>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>>>> at
>>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>>>>>>>>>> to
>>>>>>>>>> process request in NioReceiver
>>>>>>>>>> java.io.IOException: A non-blocking socket operation could not
>>>>>>>>>> be
>>>>>>>>>> completed immediately
>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Fady
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Hi,
telnet IP 4000 is working fine, i installed a tool for network
monitoring at the level of IP and Port and i didnt see any
disconnection, and yes i'm sure that no firewall is enabled.
I saw some strange think on the server that I have tried to ping the
multicast IP (228.0.0.4) and i get reply from different IPs in the
network, i don't know why and how i get those IPs, after checking with
the network team they told me that those IPs are related to the SAN
storage taking into consideration that the Tomcat servers are not
connected in anyway to that SUN storage.
On Tue, Feb 7, 2017 at 4:51 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
> Hi.
>
> This is for the Tomcat/Tribes experts on the list.
>
> I know nothing of Tribes, but the on-line documentation seems to say that
> the communication happens over TCP and that the protocol used is not
> encrypted.
> Fady previously tried a standard "ping" and a "telnet" between the two
> nodes, and that is the base for him mentioning that "there is no network
> disconnection" between the nodes.
> Nevertheless, the calling pinging node seems to say that it times out
> without getting a response fom the target node. There is evidently a
> contradiction there.
> So this could still be some kind of network issue.
>
> Considering that the protocol command for this "ping" should be known by
> someone here, would it not be possible to imagine a little program in some
> scripting language (or even java, God forbid), which would open a TCP
> channel with the target node IP/port, send such a "ping" message, wait for a
> response and report the result ?
> That would at least confirm/deny that the problem is with the network.
>
> The log below does not for example say if the error happens when opening the
> TCP communication channel, or after sending the ping message on it,
> (Of course, testing the TCP open could be done with "telnet IP 4000", but I
> don't know if Fady tried this).
> Maybe tribes also already contains some löw-level debugging options ?
> wireshark maybe another option, but it has quite a learning curve.
> And this is on Windows.
>
> By the way Fady, are you sure that your "Windows Firewall with Enhanced
> Security" is not just dropping TCP packets to/from port 40xx (or from
> "java.exe") ? There are some "network policies" there which can have
> wide-ranging side-effects.
>
>
>
>
> On 07.02.2017 14:42, Fady Haikal wrote:
>>
>> Hi, issue still not fixed. Tomcat session replication is not able to
>> replicate the key from node to node, please find below the error,
>> taking into consideration that there is no network disconnection
>> between 2 nodes
>>
>>
>> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
>> Unable to replicate backup
>> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
>> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
>> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
>> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
>> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
>> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
>> 114, 43, 102}:4000;
>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
>> at
>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>> at
>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> at
>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>> at
>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>> at
>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>> at
>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>> at
>> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
>> at
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
>> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
>> at
>> org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
>> at
>> org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
>> at
>> org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
>> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
>> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
>> at
>> org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
>> at
>> javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
>> at
>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
>> at
>> org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
>> at
>> org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
>> at
>> org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
>> at
>> org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
>> at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
>> at
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
>> at
>> org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
>> at
>> org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
>> at
>> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
>> at
>> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
>> at
>> org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
>> at
>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
>> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
>> at
>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> at
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.lang.Thread.run(Unknown Source)
>>
>> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com>
>> wrote:
>>>
>>> Fady,
>>>
>>> Sorry for top posting.
>>>
>>> If I remember correctly, the Cluster Element goes into the Container and
>>> not the Host.
>>> Plus I see in our (working) case, a DeltaManager and a
>>> JvmRouteSessionIDBinderListener
>>>
>>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>>> expireSessionsOnShutdown="false"
>>> notifyListenersOnReplication="true"/>
>>> ...
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>>
>>> Besides this, only ports, limits and values are different.
>>>
>>> You may want to filter out the replication for static resources as gifs
>>> jpg or css.
>>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>>>
>>> Best regards
>>>
>>> Peter
>>>
>>>> below is the server.xml configuration, as mentioened earlier the issue
>>>> is related to the cluster configuration, and as per my research i can
>>>> see that some users are facing the same issue but i didnt found the
>>>> solution of it
>>>>
>>>>
>>>> <?xml version='1.0' encoding='utf-8'?>
>>>> <!--
>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>> contributor license agreements. See the NOTICE file distributed with
>>>> this work for additional information regarding copyright ownership.
>>>> The ASF licenses this file to You under the Apache License, Version
>>>> 2.0
>>>> (the "License"); you may not use this file except in compliance with
>>>> the License. You may obtain a copy of the License at
>>>>
>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>
>>>> Unless required by applicable law or agreed to in writing, software
>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>> See the License for the specific language governing permissions and
>>>> limitations under the License.
>>>> -->
>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>> define subcomponents such as "Valves" at this level.
>>>> Documentation at /docs/config/server.html
>>>> -->
>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>> <Listener
>>>> className="org.apache.catalina.startup.VersionLoggerListener" />
>>>> <!-- Security listener. Documentation at /docs/config/listeners.html
>>>> <Listener className="org.apache.catalina.security.SecurityListener"
>>>> />
>>>> -->
>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>>>> SSLEngine="on" />
>>>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>>>> <Listener
>>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>>> />
>>>> <Listener
>>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>>> />
>>>> <Listener
>>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>>> />
>>>>
>>>> <!-- Global JNDI resources
>>>> Documentation at /docs/jndi-resources-howto.html
>>>> -->
>>>> <GlobalNamingResources>
>>>> <!-- Editable user database that can also be used by
>>>> UserDatabaseRealm to authenticate users
>>>> -->
>>>> <Resource name="UserDatabase" auth="Container"
>>>> type="org.apache.catalina.UserDatabase"
>>>> description="User database that can be updated and saved"
>>>>
>>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>>> pathname="conf/tomcat-users.xml" />
>>>> </GlobalNamingResources>
>>>>
>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>> share
>>>> a single "Container" Note: A "Service" is not itself a
>>>> "Container",
>>>> so you may not define subcomponents such as "Valves" at this
>>>> level.
>>>> Documentation at /docs/config/service.html
>>>> -->
>>>> <Service name="Catalina">
>>>>
>>>> <!--The connectors can use a shared executor, you can define one
>>>> or more named thread pools-->
>>>> <!--
>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>> maxThreads="150" minSpareThreads="4"/>
>>>> -->
>>>>
>>>>
>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>> received
>>>> and responses are returned. Documentation at :
>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>> non-blocking)
>>>> Java AJP Connector: /docs/config/ajp.html
>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>> -->
>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>> connectionTimeout="60000" maxThreads="500"
>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>> <!-- A "Connector" using the shared thread pool-->
>>>> <!--
>>>> <Connector executor="tomcatThreadPool"
>>>> port="8080" protocol="HTTP/1.1"
>>>> connectionTimeout="60000"
>>>> redirectPort="8443" />
>>>> -->
>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>> This connector uses the NIO implementation that requires the
>>>> JSSE
>>>> style configuration. When using the APR/native implementation,
>>>> the
>>>> OpenSSL style configuration is required as described in the
>>>> APR/native
>>>> documentation -->
>>>> <!--
>>>> <Connector port="8443"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>> secure="true"
>>>> clientAuth="false" sslProtocol="TLS" />
>>>> -->
>>>>
>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>
>>>>
>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>> processes
>>>> every request. The Engine implementation for Tomcat stand
>>>> alone
>>>> analyzes the HTTP headers included with the request, and
>>>> passes them
>>>> on to the appropriate Host (virtual host).
>>>> Documentation at /docs/config/engine.html -->
>>>>
>>>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>>>
>>>> -->
>>>> <Engine name="Catalina" defaultHost="localhost"
>>>> jvmRoute="TomcatNode1">
>>>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>>>> via a brute-force attack -->
>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>> <!-- This Realm uses the UserDatabase configured in the global
>>>> JNDI
>>>> resources under the key "UserDatabase". Any edits
>>>> that are performed against this UserDatabase are
>>>> immediately
>>>> available for use by the Realm. -->
>>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>> resourceName="UserDatabase"/>
>>>> </Realm>
>>>>
>>>> <Host name="localhost" appBase="webapps"
>>>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>>> <!--For clustering, please take a look at documentation at:
>>>> /docs/cluster-howto.html (simple how to)
>>>> /docs/config/cluster.html (reference documentation) -->
>>>> <!--
>>>> <Cluster
>>>> className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>>> -->
>>>>
>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>> channelSendOptions="4">
>>>> <Channel
>>>> className="org.apache.catalina.tribes.group.GroupChannel">
>>>> <Membership
>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>> address="228.0.0.4"
>>>> port="45564"
>>>> frequency="500"
>>>> dropTime="9000"/>
>>>> <Receiver
>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>> address="auto"
>>>> port="4000"
>>>> autoBind="100"
>>>> selectorTimeout="5000"
>>>> maxThreads="6"/>
>>>>
>>>> <Sender
>>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>>> <Transport
>>>>
>>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>>> </Sender>
>>>> <Interceptor
>>>>
>>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>>> <Interceptor
>>>>
>>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>>> </Channel>
>>>>
>>>> <Valve
>>>> className="org.apache.catalina.ha.tcp.ReplicationValve"
>>>> filter=""/>
>>>> <Valve
>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>
>>>> <Deployer
>>>> className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>> watchEnabled="false"/>
>>>>
>>>> <ClusterListener
>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>> </Cluster>
>>>> <!-- SingleSignOn valve, share authentication between web
>>>> applications
>>>> Documentation at: /docs/config/valve.html -->
>>>> <!--
>>>> <Valve
>>>> className="org.apache.catalina.authenticator.SingleSignOn" />
>>>> -->
>>>>
>>>> <!-- Access log processes all example.
>>>> Documentation at: /docs/config/valve.html
>>>> Note: The pattern used is equivalent to using
>>>> pattern="common" -->
>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>> directory="logs"
>>>> prefix="localhost_access_log" suffix=".txt"
>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>> threshold="900" />
>>>>
>>>> </Host>
>>>> </Engine>
>>>> </Service>
>>>> </Server>
>>>>
>>>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>>>> wrote:
>>>>>
>>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>> What is the host OS ? Windows Server 2012
>>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>>
>>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>>> cluster implementation
>>>>>>
>>>>>
>>>>> I still don't know tribes, but then my non-educated guess at this point
>>>>> would be that there is something wrong in your configuration.
>>>>> Can you copy/paste it here ? (remove sensible things like passwords,
>>>>> public
>>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>>
>>>>> Then maybe some tribes-specialist can take over ?
>>>>>
>>>>>
>>>>>>
>>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>>> cluster
>>>>>>
>>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>>> nodes ping and telnet are OK
>>>>>>
>>>>>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Plz can i get some help here?
>>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>>> Production server
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Fady
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi.
>>>>>>> If you want quick answers, you should provide more information.
>>>>>>> What is the host OS ?
>>>>>>> What is the Tomcat version ?
>>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>>
>>>>>>> I do not know tribes at all, but according to the logfile below, it
>>>>>>> seems
>>>>>>> that something is trying to "ping" the address 10.114.43.103, port
>>>>>>> 4000,
>>>>>>> and
>>>>>>> never getting a response (or at least not within 3000ms).
>>>>>>> Is there actually something listening on that address/port ?
>>>>>>> The "netstat" command (available both on Linux and Windows) can tell
>>>>>>> you.
>>>>>>> If there is something listening there, can it respond to whatever is
>>>>>>> ping-ing it ?
>>>>>>> (routing, firewall, ..)
>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please
>>>>>>>>> advise
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has
>>>>>>>>> timed
>>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>>> at
>>>>>>>>> org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>>>>>>>>> to
>>>>>>>>> process request in NioReceiver
>>>>>>>>> java.io.IOException: A non-blocking socket operation could not
>>>>>>>>> be
>>>>>>>>> completed immediately
>>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Fady
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
Hi.
This is for the Tomcat/Tribes experts on the list.
I know nothing of Tribes, but the on-line documentation seems to say that the
communication happens over TCP and that the protocol used is not encrypted.
Fady previously tried a standard "ping" and a "telnet" between the two nodes, and that is
the base for him mentioning that "there is no network disconnection" between the nodes.
Nevertheless, the calling pinging node seems to say that it times out without getting a
response fom the target node. There is evidently a contradiction there.
So this could still be some kind of network issue.
Considering that the protocol command for this "ping" should be known by someone here,
would it not be possible to imagine a little program in some scripting language (or even
java, God forbid), which would open a TCP channel with the target node IP/port, send such
a "ping" message, wait for a response and report the result ?
That would at least confirm/deny that the problem is with the network.
The log below does not for example say if the error happens when opening the TCP
communication channel, or after sending the ping message on it,
(Of course, testing the TCP open could be done with "telnet IP 4000", but I don't know if
Fady tried this).
Maybe tribes also already contains some l�w-level debugging options ?
wireshark maybe another option, but it has quite a learning curve.
And this is on Windows.
By the way Fady, are you sure that your "Windows Firewall with Enhanced Security" is not
just dropping TCP packets to/from port 40xx (or from "java.exe") ? There are some "network
policies" there which can have wide-ranging side-effects.
On 07.02.2017 14:42, Fady Haikal wrote:
> Hi, issue still not fixed. Tomcat session replication is not able to
> replicate the key from node to node, please find below the error,
> taking into consideration that there is no network disconnection
> between 2 nodes
>
>
> 07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
> org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
> Unable to replicate backup
> key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
> backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
> 114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
> securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
> -85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
> Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
> 114, 43, 102}:4000;
> org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
> at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
> at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
> at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
> at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
> at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
> at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
> at org.apache.catalina.connector.Request.getSession(Request.java:2260)
> at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
> at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
> at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
> at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
> at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
> at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
> at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
> at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
> at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
> at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
> at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
> at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
> at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
> at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
> at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
> at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
> at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
> at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Unknown Source)
>
> On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com> wrote:
>> Fady,
>>
>> Sorry for top posting.
>>
>> If I remember correctly, the Cluster Element goes into the Container and not the Host.
>> Plus I see in our (working) case, a DeltaManager and a JvmRouteSessionIDBinderListener
>>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> notifyListenersOnReplication="true"/>
>> ...
>> <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>>
>> Besides this, only ports, limits and values are different.
>>
>> You may want to filter out the replication for static resources as gifs jpg or css.
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>>
>> Best regards
>>
>> Peter
>>
>>> below is the server.xml configuration, as mentioened earlier the issue
>>> is related to the cluster configuration, and as per my research i can
>>> see that some users are facing the same issue but i didnt found the
>>> solution of it
>>>
>>>
>>> <?xml version='1.0' encoding='utf-8'?>
>>> <!--
>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>> contributor license agreements. See the NOTICE file distributed with
>>> this work for additional information regarding copyright ownership.
>>> The ASF licenses this file to You under the Apache License, Version 2.0
>>> (the "License"); you may not use this file except in compliance with
>>> the License. You may obtain a copy of the License at
>>>
>>> http://www.apache.org/licenses/LICENSE-2.0
>>>
>>> Unless required by applicable law or agreed to in writing, software
>>> distributed under the License is distributed on an "AS IS" BASIS,
>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>> See the License for the specific language governing permissions and
>>> limitations under the License.
>>> -->
>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>> define subcomponents such as "Valves" at this level.
>>> Documentation at /docs/config/server.html
>>> -->
>>> <Server port="8005" shutdown="SHUTDOWN">
>>> <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
>>> <!-- Security listener. Documentation at /docs/config/listeners.html
>>> <Listener className="org.apache.catalina.security.SecurityListener" />
>>> -->
>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>>> SSLEngine="on" />
>>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>>> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>>> />
>>> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>>> />
>>> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>>> />
>>>
>>> <!-- Global JNDI resources
>>> Documentation at /docs/jndi-resources-howto.html
>>> -->
>>> <GlobalNamingResources>
>>> <!-- Editable user database that can also be used by
>>> UserDatabaseRealm to authenticate users
>>> -->
>>> <Resource name="UserDatabase" auth="Container"
>>> type="org.apache.catalina.UserDatabase"
>>> description="User database that can be updated and saved"
>>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>>> pathname="conf/tomcat-users.xml" />
>>> </GlobalNamingResources>
>>>
>>> <!-- A "Service" is a collection of one or more "Connectors" that share
>>> a single "Container" Note: A "Service" is not itself a "Container",
>>> so you may not define subcomponents such as "Valves" at this level.
>>> Documentation at /docs/config/service.html
>>> -->
>>> <Service name="Catalina">
>>>
>>> <!--The connectors can use a shared executor, you can define one
>>> or more named thread pools-->
>>> <!--
>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>> maxThreads="150" minSpareThreads="4"/>
>>> -->
>>>
>>>
>>> <!-- A "Connector" represents an endpoint by which requests are received
>>> and responses are returned. Documentation at :
>>> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
>>> Java AJP Connector: /docs/config/ajp.html
>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>> -->
>>> <Connector port="8080" protocol="HTTP/1.1"
>>> connectionTimeout="60000" maxThreads="500"
>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>> <!-- A "Connector" using the shared thread pool-->
>>> <!--
>>> <Connector executor="tomcatThreadPool"
>>> port="8080" protocol="HTTP/1.1"
>>> connectionTimeout="60000"
>>> redirectPort="8443" />
>>> -->
>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>> This connector uses the NIO implementation that requires the JSSE
>>> style configuration. When using the APR/native implementation, the
>>> OpenSSL style configuration is required as described in the APR/native
>>> documentation -->
>>> <!--
>>> <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
>>> clientAuth="false" sslProtocol="TLS" />
>>> -->
>>>
>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>
>>>
>>> <!-- An Engine represents the entry point (within Catalina) that processes
>>> every request. The Engine implementation for Tomcat stand alone
>>> analyzes the HTTP headers included with the request, and passes them
>>> on to the appropriate Host (virtual host).
>>> Documentation at /docs/config/engine.html -->
>>>
>>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>>
>>> -->
>>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatNode1">
>>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>>> via a brute-force attack -->
>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>>> resources under the key "UserDatabase". Any edits
>>> that are performed against this UserDatabase are immediately
>>> available for use by the Realm. -->
>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>> resourceName="UserDatabase"/>
>>> </Realm>
>>>
>>> <Host name="localhost" appBase="webapps"
>>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>> <!--For clustering, please take a look at documentation at:
>>> /docs/cluster-howto.html (simple how to)
>>> /docs/config/cluster.html (reference documentation) -->
>>> <!--
>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>>> -->
>>>
>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>> channelSendOptions="4">
>>> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>>> <Membership
>>> className="org.apache.catalina.tribes.membership.McastService"
>>> address="228.0.0.4"
>>> port="45564"
>>> frequency="500"
>>> dropTime="9000"/>
>>> <Receiver
>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>> address="auto"
>>> port="4000"
>>> autoBind="100"
>>> selectorTimeout="5000"
>>> maxThreads="6"/>
>>>
>>> <Sender
>>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>>> <Transport
>>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>>> </Sender>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>>> <Interceptor
>>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>>> </Channel>
>>>
>>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>> filter=""/>
>>> <Valve
>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>
>>> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>> watchEnabled="false"/>
>>>
>>> <ClusterListener
>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>> </Cluster>
>>> <!-- SingleSignOn valve, share authentication between web applications
>>> Documentation at: /docs/config/valve.html -->
>>> <!--
>>> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>>> -->
>>>
>>> <!-- Access log processes all example.
>>> Documentation at: /docs/config/valve.html
>>> Note: The pattern used is equivalent to using pattern="common" -->
>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>> directory="logs"
>>> prefix="localhost_access_log" suffix=".txt"
>>> pattern="%h %l %u %t "%r" %s %b" />
>>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>> threshold="900" />
>>>
>>> </Host>
>>> </Engine>
>>> </Service>
>>> </Server>
>>>
>>> On Mon, Feb 6, 2017 at 6:51 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com> wrote:
>>>> On 06.02.2017 17:45, Fady Haikal wrote:
>>>>>
>>>>> Hi,
>>>>> What is the host OS ? Windows Server 2012
>>>>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>>>>
>>>>> Is this problem new ? was this working before ? how long ? Since
>>>>> cluster implementation
>>>>>
>>>>
>>>> I still don't know tribes, but then my non-educated guess at this point
>>>> would be that there is something wrong in your configuration.
>>>> Can you copy/paste it here ? (remove sensible things like passwords, public
>>>> IP addresses etc..)(but not to the point of making it uncheckable).
>>>>
>>>> Then maybe some tribes-specialist can take over ?
>>>>
>>>>
>>>>>
>>>>> Is there actually something listening on that address/port ? Tomcat
>>>>> cluster
>>>>>
>>>>> the Port 4000 is listening and there is no disconnection between 2
>>>>> nodes ping and telnet are OK
>>>>>
>>>>> On Mon, Feb 6, 2017 at 6:42 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com>
>>>>> wrote:
>>>>>>
>>>>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>>>>
>>>>>>>
>>>>>>> Plz can i get some help here?
>>>>>>> This issue is still occurring and it's filling the log file in the
>>>>>>> Production server
>>>>>>>
>>>>>>> Regards,
>>>>>>> Fady
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi.
>>>>>> If you want quick answers, you should provide more information.
>>>>>> What is the host OS ?
>>>>>> What is the Tomcat version ?
>>>>>> Is this problem new ? was this working before ? how long ?
>>>>>>
>>>>>> I do not know tribes at all, but according to the logfile below, it seems
>>>>>> that something is trying to "ping" the address 10.114.43.103, port 4000,
>>>>>> and
>>>>>> never getting a response (or at least not within 3000ms).
>>>>>> Is there actually something listening on that address/port ?
>>>>>> The "netstat" command (available both on Linux and Windows) can tell you.
>>>>>> If there is something listening there, can it respond to whatever is
>>>>>> ping-ing it ?
>>>>>> (routing, firewall, ..)
>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>>>>>>
>>>>>>>>
>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>>>>
>>>>>>>>
>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>>>>>>>> process request in NioReceiver
>>>>>>>> java.io.IOException: A non-blocking socket operation could not be
>>>>>>>> completed immediately
>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Fady
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Hi, issue still not fixed. Tomcat session replication is not able to
replicate the key from node to node, please find below the error,
taking into consideration that there is no network disconnection
between 2 nodes
07-Feb-2017 16:36:06.186 SEVERE [http-nio-8080-exec-8]
org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo
Unable to replicate backup
key:58291D242C742A8A4B1657BA42C831A4.TomcatNode2 to
backup:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
114, 43, 102}:4000,{10, 114, 43, 102},4000, alive=68841350,
securePort=-1, UDP Port=-1, id={85 5 -62 -66 106 -12 64 12 -102 -14
-85 -87 15 9 -51 -112 }, payload={}, command={}, domain={}, ].
Reason:Operation has timed out(3000 ms.).; Faulty members:tcp://{10,
114, 43, 102}:4000;
org.apache.catalina.tribes.ChannelException: Operation has timed
out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 102}:4000;
at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
at org.apache.catalina.tribes.tipis.LazyReplicatedMap.publishEntryInfo(LazyReplicatedMap.java:170)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1040)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.put(AbstractReplicatedMap.java:1024)
at org.apache.catalina.session.ManagerBase.add(ManagerBase.java:647)
at org.apache.catalina.session.StandardSession.setId(StandardSession.java:374)
at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:279)
at org.apache.catalina.session.ManagerBase.createSession(ManagerBase.java:708)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2936)
at org.apache.catalina.connector.Request.getSession(Request.java:2260)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:895)
at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:231)
at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:568)
at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:513)
at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:137)
at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:109)
at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:60)
at org.apache.jsp.WEB_002dINF.jsp._401_jsp._jspService(_401_jsp.java:100)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:719)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:390)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:317)
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:445)
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:304)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:181)
at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:194)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:318)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:208)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
On Tue, Feb 7, 2017 at 1:02 PM, Kreuser, Peter <pk...@airplus.com> wrote:
> Fady,
>
> Sorry for top posting.
>
> If I remember correctly, the Cluster Element goes into the Container and not the Host.
> Plus I see in our (working) case, a DeltaManager and a JvmRouteSessionIDBinderListener
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"/>
> ...
> <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
> Besides this, only ports, limits and values are different.
>
> You may want to filter out the replication for static resources as gifs jpg or css.
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
>
> Best regards
>
> Peter
>
>> below is the server.xml configuration, as mentioened earlier the issue
>> is related to the cluster configuration, and as per my research i can
>> see that some users are facing the same issue but i didnt found the
>> solution of it
>>
>>
>> <?xml version='1.0' encoding='utf-8'?>
>> <!--
>> Licensed to the Apache Software Foundation (ASF) under one or more
>> contributor license agreements. See the NOTICE file distributed with
>> this work for additional information regarding copyright ownership.
>> The ASF licenses this file to You under the Apache License, Version 2.0
>> (the "License"); you may not use this file except in compliance with
>> the License. You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in writing, software
>> distributed under the License is distributed on an "AS IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> See the License for the specific language governing permissions and
>> limitations under the License.
>> -->
>> <!-- Note: A "Server" is not itself a "Container", so you may not
>> define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/server.html
>> -->
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
>> <!-- Security listener. Documentation at /docs/config/listeners.html
>> <Listener className="org.apache.catalina.security.SecurityListener" />
>> -->
>> <!--APR library loader. Documentation at /docs/apr.html -->
>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
>> />
>> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
>> />
>> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
>> />
>>
>> <!-- Global JNDI resources
>> Documentation at /docs/jndi-resources-howto.html
>> -->
>> <GlobalNamingResources>
>> <!-- Editable user database that can also be used by
>> UserDatabaseRealm to authenticate users
>> -->
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <!-- A "Service" is a collection of one or more "Connectors" that share
>> a single "Container" Note: A "Service" is not itself a "Container",
>> so you may not define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/service.html
>> -->
>> <Service name="Catalina">
>>
>> <!--The connectors can use a shared executor, you can define one
>> or more named thread pools-->
>> <!--
>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> maxThreads="150" minSpareThreads="4"/>
>> -->
>>
>>
>> <!-- A "Connector" represents an endpoint by which requests are received
>> and responses are returned. Documentation at :
>> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
>> Java AJP Connector: /docs/config/ajp.html
>> APR (HTTP/AJP) Connector: /docs/apr.html
>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> -->
>> <Connector port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000" maxThreads="500"
>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> <!-- A "Connector" using the shared thread pool-->
>> <!--
>> <Connector executor="tomcatThreadPool"
>> port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000"
>> redirectPort="8443" />
>> -->
>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> This connector uses the NIO implementation that requires the JSSE
>> style configuration. When using the APR/native implementation, the
>> OpenSSL style configuration is required as described in the APR/native
>> documentation -->
>> <!--
>> <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
>> maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
>> clientAuth="false" sslProtocol="TLS" />
>> -->
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>
>>
>> <!-- An Engine represents the entry point (within Catalina) that processes
>> every request. The Engine implementation for Tomcat stand alone
>> analyzes the HTTP headers included with the request, and passes them
>> on to the appropriate Host (virtual host).
>> Documentation at /docs/config/engine.html -->
>>
>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>
>> -->
>> <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatNode1">
>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>> via a brute-force attack -->
>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>> resources under the key "UserDatabase". Any edits
>> that are performed against this UserDatabase are immediately
>> available for use by the Realm. -->
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase"/>
>> </Realm>
>>
>> <Host name="localhost" appBase="webapps"
>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>> <!--For clustering, please take a look at documentation at:
>> /docs/cluster-howto.html (simple how to)
>> /docs/config/cluster.html (reference documentation) -->
>> <!--
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>> -->
>>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="4">
>> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45564"
>> frequency="500"
>> dropTime="9000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="auto"
>> port="4000"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>> filter=""/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> <!-- SingleSignOn valve, share authentication between web applications
>> Documentation at: /docs/config/valve.html -->
>> <!--
>> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
>> -->
>>
>> <!-- Access log processes all example.
>> Documentation at: /docs/config/valve.html
>> Note: The pattern used is equivalent to using pattern="common" -->
>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> directory="logs"
>> prefix="localhost_access_log" suffix=".txt"
>> pattern="%h %l %u %t "%r" %s %b" />
>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>> threshold="900" />
>>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
>> > On 06.02.2017 17:45, Fady Haikal wrote:
>> >>
>> >> Hi,
>> >> What is the host OS ? Windows Server 2012
>> >> What is the Tomcat version ? Apache Tomcat/8.0.30
>> >>
>> >> Is this problem new ? was this working before ? how long ? Since
>> >> cluster implementation
>> >>
>> >
>> > I still don't know tribes, but then my non-educated guess at this point
>> > would be that there is something wrong in your configuration.
>> > Can you copy/paste it here ? (remove sensible things like passwords, public
>> > IP addresses etc..)(but not to the point of making it uncheckable).
>> >
>> > Then maybe some tribes-specialist can take over ?
>> >
>> >
>> >>
>> >> Is there actually something listening on that address/port ? Tomcat
>> >> cluster
>> >>
>> >> the Port 4000 is listening and there is no disconnection between 2
>> >> nodes ping and telnet are OK
>> >>
>> >> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>> >> wrote:
>> >>>
>> >>> On 06.02.2017 17:24, Fady Haikal wrote:
>> >>>>
>> >>>>
>> >>>> Plz can i get some help here?
>> >>>> This issue is still occurring and it's filling the log file in the
>> >>>> Production server
>> >>>>
>> >>>> Regards,
>> >>>> Fady
>> >>>
>> >>>
>> >>>
>> >>> Hi.
>> >>> If you want quick answers, you should provide more information.
>> >>> What is the host OS ?
>> >>> What is the Tomcat version ?
>> >>> Is this problem new ? was this working before ? how long ?
>> >>>
>> >>> I do not know tribes at all, but according to the logfile below, it seems
>> >>> that something is trying to "ping" the address 10.114.43.103, port 4000,
>> >>> and
>> >>> never getting a response (or at least not within 3000ms).
>> >>> Is there actually something listening on that address/port ?
>> >>> The "netstat" command (available both on Linux and Windows) can tell you.
>> >>> If there is something listening there, can it respond to whatever is
>> >>> ping-ing it ?
>> >>> (routing, firewall, ..)
>> >>>
>> >>>>
>> >>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>> >>>>>
>> >>>>>
>> >>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >>>>> Unable to send AbstractReplicatedMap.ping message
>> >>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> >>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>> >>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>> >>>>>
>> >>>>>
>> >>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> >>>>> process request in NioReceiver
>> >>>>> java.io.IOException: A non-blocking socket operation could not be
>> >>>>> completed immediately
>> >>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> >>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>> >>>>> at
>> >>>>>
>> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>> >>>>> at java.lang.Thread.run(Unknown Source)
>> >>>>>
>> >>>>> Regards,
>> >>>>> Fady
>> >>>>
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>
>> >>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Operation has timed out
Posted by "Kreuser, Peter" <pk...@airplus.com>.
Fady,
Sorry for top posting.
If I remember correctly, the Cluster Element goes into the Container and not the Host.
Plus I see in our (working) case, a DeltaManager and a JvmRouteSessionIDBinderListener
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
...
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
Besides this, only ports, limits and values are different.
You may want to filter out the replication for static resources as gifs jpg or css.
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;" />
Best regards
Peter
> below is the server.xml configuration, as mentioened earlier the issue
> is related to the cluster configuration, and as per my research i can
> see that some users are facing the same issue but i didnt found the
> solution of it
>
>
> <?xml version='1.0' encoding='utf-8'?>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
> <!-- Note: A "Server" is not itself a "Container", so you may not
> define subcomponents such as "Valves" at this level.
> Documentation at /docs/config/server.html
> -->
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
> <!-- Security listener. Documentation at /docs/config/listeners.html
> <Listener className="org.apache.catalina.security.SecurityListener" />
> -->
> <!--APR library loader. Documentation at /docs/apr.html -->
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> />
> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> />
>
> <!-- Global JNDI resources
> Documentation at /docs/jndi-resources-howto.html
> -->
> <GlobalNamingResources>
> <!-- Editable user database that can also be used by
> UserDatabaseRealm to authenticate users
> -->
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <!-- A "Service" is a collection of one or more "Connectors" that share
> a single "Container" Note: A "Service" is not itself a "Container",
> so you may not define subcomponents such as "Valves" at this level.
> Documentation at /docs/config/service.html
> -->
> <Service name="Catalina">
>
> <!--The connectors can use a shared executor, you can define one
> or more named thread pools-->
> <!--
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="150" minSpareThreads="4"/>
> -->
>
>
> <!-- A "Connector" represents an endpoint by which requests are received
> and responses are returned. Documentation at :
> Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
> Java AJP Connector: /docs/config/ajp.html
> APR (HTTP/AJP) Connector: /docs/apr.html
> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> -->
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="60000" maxThreads="500"
> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> <!-- A "Connector" using the shared thread pool-->
> <!--
> <Connector executor="tomcatThreadPool"
> port="8080" protocol="HTTP/1.1"
> connectionTimeout="60000"
> redirectPort="8443" />
> -->
> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> This connector uses the NIO implementation that requires the JSSE
> style configuration. When using the APR/native implementation, the
> OpenSSL style configuration is required as described in the APR/native
> documentation -->
> <!--
> <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
> maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
> clientAuth="false" sslProtocol="TLS" />
> -->
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
>
> <!-- An Engine represents the entry point (within Catalina) that processes
> every request. The Engine implementation for Tomcat stand alone
> analyzes the HTTP headers included with the request, and passes them
> on to the appropriate Host (virtual host).
> Documentation at /docs/config/engine.html -->
>
> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>
> -->
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatNode1">
> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
> via a brute-force attack -->
> <Realm className="org.apache.catalina.realm.LockOutRealm">
> <!-- This Realm uses the UserDatabase configured in the global JNDI
> resources under the key "UserDatabase". Any edits
> that are performed against this UserDatabase are immediately
> available for use by the Realm. -->
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase"/>
> </Realm>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true" startStopThreads="0">
> <!--For clustering, please take a look at documentation at:
> /docs/cluster-howto.html (simple how to)
> /docs/config/cluster.html (reference documentation) -->
> <!--
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
> -->
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="4">
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45564"
> frequency="500"
> dropTime="9000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> <!-- SingleSignOn valve, share authentication between web applications
> Documentation at: /docs/config/valve.html -->
> <!--
> <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
> -->
>
> <!-- Access log processes all example.
> Documentation at: /docs/config/valve.html
> Note: The pattern used is equivalent to using pattern="common" -->
> <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
> prefix="localhost_access_log" suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b" />
> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> threshold="900" />
>
> </Host>
> </Engine>
> </Service>
> </Server>
>
> On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
> > On 06.02.2017 17:45, Fady Haikal wrote:
> >>
> >> Hi,
> >> What is the host OS ? Windows Server 2012
> >> What is the Tomcat version ? Apache Tomcat/8.0.30
> >>
> >> Is this problem new ? was this working before ? how long ? Since
> >> cluster implementation
> >>
> >
> > I still don't know tribes, but then my non-educated guess at this point
> > would be that there is something wrong in your configuration.
> > Can you copy/paste it here ? (remove sensible things like passwords, public
> > IP addresses etc..)(but not to the point of making it uncheckable).
> >
> > Then maybe some tribes-specialist can take over ?
> >
> >
> >>
> >> Is there actually something listening on that address/port ? Tomcat
> >> cluster
> >>
> >> the Port 4000 is listening and there is no disconnection between 2
> >> nodes ping and telnet are OK
> >>
> >> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <aw...@ice-sa.com>
> >> wrote:
> >>>
> >>> On 06.02.2017 17:24, Fady Haikal wrote:
> >>>>
> >>>>
> >>>> Plz can i get some help here?
> >>>> This issue is still occurring and it's filling the log file in the
> >>>> Production server
> >>>>
> >>>> Regards,
> >>>> Fady
> >>>
> >>>
> >>>
> >>> Hi.
> >>> If you want quick answers, you should provide more information.
> >>> What is the host OS ?
> >>> What is the Tomcat version ?
> >>> Is this problem new ? was this working before ? how long ?
> >>>
> >>> I do not know tribes at all, but according to the logfile below, it seems
> >>> that something is trying to "ping" the address 10.114.43.103, port 4000,
> >>> and
> >>> never getting a response (or at least not within 3000ms).
> >>> Is there actually something listening on that address/port ?
> >>> The "netstat" command (available both on Linux and Windows) can tell you.
> >>> If there is something listening there, can it respond to whatever is
> >>> ping-ing it ?
> >>> (routing, firewall, ..)
> >>>
> >>>>
> >>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>
> >>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
> >>>>>
> >>>>>
> >>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >>>>> Unable to send AbstractReplicatedMap.ping message
> >>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
> >>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
> >>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
> >>>>>
> >>>>>
> >>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> >>>>> process request in NioReceiver
> >>>>> java.io.IOException: A non-blocking socket operation could not be
> >>>>> completed immediately
> >>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> >>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
> >>>>> at
> >>>>>
> >>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
> >>>>> at java.lang.Thread.run(Unknown Source)
> >>>>>
> >>>>> Regards,
> >>>>> Fady
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
below is the server.xml configuration, as mentioened earlier the issue
is related to the cluster configuration, and as per my research i can
see that some users are facing the same issue but i didnt found the
solution of it
<?xml version='1.0' encoding='utf-8'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Note: A "Server" is not itself a "Container", so you may not
define subcomponents such as "Valves" at this level.
Documentation at /docs/config/server.html
-->
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<!-- Security listener. Documentation at /docs/config/listeners.html
<Listener className="org.apache.catalina.security.SecurityListener" />
-->
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
/>
<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<!-- Editable user database that can also be used by
UserDatabaseRealm to authenticate users
-->
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<!-- A "Service" is a collection of one or more "Connectors" that share
a single "Container" Note: A "Service" is not itself a "Container",
so you may not define subcomponents such as "Valves" at this level.
Documentation at /docs/config/service.html
-->
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one
or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="60000" maxThreads="500"
minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="60000"
redirectPort="8443" />
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
This connector uses the NIO implementation that requires the JSSE
style configuration. When using the APR/native implementation, the
OpenSSL style configuration is required as described in the APR/native
documentation -->
<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS" />
-->
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<!-- An Engine represents the entry point (within Catalina) that processes
every request. The Engine implementation for Tomcat stand alone
analyzes the HTTP headers included with the request, and passes them
on to the appropriate Host (virtual host).
Documentation at /docs/config/engine.html -->
<!-- You should set jvmRoute to support load-balancing via AJP ie :
-->
<Engine name="Catalina" defaultHost="localhost" jvmRoute="TomcatNode1">
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
<!-- This Realm uses the UserDatabase configured in the global JNDI
resources under the key "UserDatabase". Any edits
that are performed against this UserDatabase are immediately
available for use by the Realm. -->
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true" startStopThreads="0">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="4">
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="9000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- SingleSignOn valve, share authentication between web applications
Documentation at: /docs/config/valve.html -->
<!--
<Valve className="org.apache.catalina.authenticator.SingleSignOn" />
-->
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
threshold="900" />
</Host>
</Engine>
</Service>
</Server>
On Mon, Feb 6, 2017 at 6:51 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
> On 06.02.2017 17:45, Fady Haikal wrote:
>>
>> Hi,
>> What is the host OS ? Windows Server 2012
>> What is the Tomcat version ? Apache Tomcat/8.0.30
>>
>> Is this problem new ? was this working before ? how long ? Since
>> cluster implementation
>>
>
> I still don't know tribes, but then my non-educated guess at this point
> would be that there is something wrong in your configuration.
> Can you copy/paste it here ? (remove sensible things like passwords, public
> IP addresses etc..)(but not to the point of making it uncheckable).
>
> Then maybe some tribes-specialist can take over ?
>
>
>>
>> Is there actually something listening on that address/port ? Tomcat
>> cluster
>>
>> the Port 4000 is listening and there is no disconnection between 2
>> nodes ping and telnet are OK
>>
>> On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <aw...@ice-sa.com>
>> wrote:
>>>
>>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>>
>>>>
>>>> Plz can i get some help here?
>>>> This issue is still occurring and it's filling the log file in the
>>>> Production server
>>>>
>>>> Regards,
>>>> Fady
>>>
>>>
>>>
>>> Hi.
>>> If you want quick answers, you should provide more information.
>>> What is the host OS ?
>>> What is the Tomcat version ?
>>> Is this problem new ? was this working before ? how long ?
>>>
>>> I do not know tribes at all, but according to the logfile below, it seems
>>> that something is trying to "ping" the address 10.114.43.103, port 4000,
>>> and
>>> never getting a response (or at least not within 3000ms).
>>> Is there actually something listening on that address/port ?
>>> The "netstat" command (available both on Linux and Windows) can tell you.
>>> If there is something listening there, can it respond to whatever is
>>> ping-ing it ?
>>> (routing, firewall, ..)
>>>
>>>>
>>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>>>
>>>>>
>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>>
>>>>>
>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>>>>> process request in NioReceiver
>>>>> java.io.IOException: A non-blocking socket operation could not be
>>>>> completed immediately
>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>>> at
>>>>>
>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>
>>>>> Regards,
>>>>> Fady
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 06.02.2017 17:45, Fady Haikal wrote:
> Hi,
> What is the host OS ? Windows Server 2012
> What is the Tomcat version ? Apache Tomcat/8.0.30
>
> Is this problem new ? was this working before ? how long ? Since
> cluster implementation
>
I still don't know tribes, but then my non-educated guess at this point would be that
there is something wrong in your configuration.
Can you copy/paste it here ? (remove sensible things like passwords, public IP addresses
etc..)(but not to the point of making it uncheckable).
Then maybe some tribes-specialist can take over ?
>
> Is there actually something listening on that address/port ? Tomcat cluster
>
> the Port 4000 is listening and there is no disconnection between 2
> nodes ping and telnet are OK
>
> On Mon, Feb 6, 2017 at 6:42 PM, Andr� Warnier (tomcat) <aw...@ice-sa.com> wrote:
>> On 06.02.2017 17:24, Fady Haikal wrote:
>>>
>>> Plz can i get some help here?
>>> This issue is still occurring and it's filling the log file in the
>>> Production server
>>>
>>> Regards,
>>> Fady
>>
>>
>> Hi.
>> If you want quick answers, you should provide more information.
>> What is the host OS ?
>> What is the Tomcat version ?
>> Is this problem new ? was this working before ? how long ?
>>
>> I do not know tribes at all, but according to the logfile below, it seems
>> that something is trying to "ping" the address 10.114.43.103, port 4000, and
>> never getting a response (or at least not within 3000ms).
>> Is there actually something listening on that address/port ?
>> The "netstat" command (available both on Linux and Windows) can tell you.
>> If there is something listening there, can it respond to whatever is
>> ping-ing it ?
>> (routing, firewall, ..)
>>
>>>
>>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com> wrote:
>>>>
>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>>
>>>>
>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>> Unable to send AbstractReplicatedMap.ping message
>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>>> at
>>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>>> at
>>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>>> at
>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>>> at
>>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>>
>>>>
>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>>>> process request in NioReceiver
>>>> java.io.IOException: A non-blocking socket operation could not be
>>>> completed immediately
>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>>> at
>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>>> at java.lang.Thread.run(Unknown Source)
>>>>
>>>> Regards,
>>>> Fady
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Hi,
What is the host OS ? Windows Server 2012
What is the Tomcat version ? Apache Tomcat/8.0.30
Is this problem new ? was this working before ? how long ? Since
cluster implementation
Is there actually something listening on that address/port ? Tomcat cluster
the Port 4000 is listening and there is no disconnection between 2
nodes ping and telnet are OK
On Mon, Feb 6, 2017 at 6:42 PM, André Warnier (tomcat) <aw...@ice-sa.com> wrote:
> On 06.02.2017 17:24, Fady Haikal wrote:
>>
>> Plz can i get some help here?
>> This issue is still occurring and it's filling the log file in the
>> Production server
>>
>> Regards,
>> Fady
>
>
> Hi.
> If you want quick answers, you should provide more information.
> What is the host OS ?
> What is the Tomcat version ?
> Is this problem new ? was this working before ? how long ?
>
> I do not know tribes at all, but according to the logfile below, it seems
> that something is trying to "ping" the address 10.114.43.103, port 4000, and
> never getting a response (or at least not within 3000ms).
> Is there actually something listening on that address/port ?
> The "netstat" command (available both on Linux and Windows) can tell you.
> If there is something listening there, can it respond to whatever is
> ping-ing it ?
> (routing, firewall, ..)
>
>>
>> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com> wrote:
>>>
>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>
>>>
>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>> Unable to send AbstractReplicatedMap.ping message
>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>> at
>>> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>>> at
>>> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>>> at
>>> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>>> at
>>> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>>> at
>>> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>>> at
>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>>> at
>>> org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>>
>>>
>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>>> process request in NioReceiver
>>> java.io.IOException: A non-blocking socket operation could not be
>>> completed immediately
>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>>> at
>>> org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>>> at java.lang.Thread.run(Unknown Source)
>>>
>>> Regards,
>>> Fady
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 06.02.2017 17:24, Fady Haikal wrote:
> Plz can i get some help here?
> This issue is still occurring and it's filling the log file in the
> Production server
>
> Regards,
> Fady
Hi.
If you want quick answers, you should provide more information.
What is the host OS ?
What is the Tomcat version ?
Is this problem new ? was this working before ? how long ?
I do not know tribes at all, but according to the logfile below, it seems that something
is trying to "ping" the address 10.114.43.103, port 4000, and never getting a response (or
at least not within 3000ms).
Is there actually something listening on that address/port ?
The "netstat" command (available both on Linux and Windows) can tell you.
If there is something listening there, can it respond to whatever is ping-ing it ?
(routing, firewall, ..)
>
> On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com> wrote:
>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>
>>
>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
>> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
>> at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
>> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
>> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>>
>>
>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> process request in NioReceiver
>> java.io.IOException: A non-blocking socket operation could not be
>> completed immediately
>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
>> at java.lang.Thread.run(Unknown Source)
>>
>> Regards,
>> Fady
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Plz can i get some help here?
This issue is still occurring and it's filling the log file in the
Production server
Regards,
Fady
On Mon, Feb 6, 2017 at 8:52 AM, Fady Haikal <fa...@gmail.com> wrote:
> Guys, we are facing the below errors in Tomcat cluster, please advise
>
>
> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:108)
> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:48)
> at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:54)
> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:82)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:233)
> at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:186)
> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:267)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:161)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>
>
> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> process request in NioReceiver
> java.io.IOException: A non-blocking socket operation could not be
> completed immediately
> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:272)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:425)
> at java.lang.Thread.run(Unknown Source)
>
> Regards,
> Fady
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
2017-02-09 18:45 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> i will try that but this required tomcat restart so i will be
> scheduling tonight a restart, anyway from where i can enable the
> org.apache.catalina.tribes.MESSAGES i didn't found it in the
> logging.properties file
>
The org.apache.catalina.tribes.MESSAGES is the logger name.
You have to add settings to the logging.properties file.
The following is a sample. Please refer to.
Note for using the org.apache.catalina.tribes.MESSAGES:
The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs.
So It should get logs in a short period of time.
=
handlers = 1catalina.org.apache.juli.AsyncFileHandler,
2localhost.org.apache.juli.AsyncFileHandler,
3manager.org.apache.juli.AsyncFileHandler,
4host-manager.org.apache.juli.AsyncFileHandler,
java.util.logging.ConsoleHandler,
10catalina.org.apache.juli.AsyncFileHandler
//omit
# FOR DEBUG
10catalina.org.apache.juli.AsyncFileHandler.formatter =
org.apache.juli.VerbatimFormatter
10catalina.org.apache.juli.AsyncFileHandler.level = FINEST
10catalina.org.apache.juli.AsyncFileHandler.directory =
${catalina.base}/logs
10catalina.org.apache.juli.AsyncFileHandler.prefix = MESSAGES.
10catalina.org.apache.juli.AsyncFileHandler.bufferSize = -1
org.apache.catalina.tribes.MESSAGES.level = FINEST
org.apache.catalina.tribes.MESSAGES.handlers =
10catalina.org.apache.juli.AsyncFileHandler
> On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org>
> wrote:
> > Hi
> >
> > Please try mapSendOptions = 8.
> > This means that it does not receive ACK of the messages.
> > Processing is returned immediately after message transmission is
> completed.
> >
> > As another way,
> > Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
> > response mail.)
> >
> >
> >
> > 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >
> >> The <Cluster><Manager> is set under the context.xml
> >>
> >> <Manager className="org.apache.catalina.ha.session.BackupManager"
> >> notifyListenersOnReplication="true"
> >> mapSendOptions="6"/>
> >>
> >> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
> >> wrote:
> >> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >
> >> >> Keiichi,
> >> >> mapSendoption is not configured anyway please check my configuration
> >> below
> >> >>
> >> >
> >> >
> >> > You have not set <Cluster><Manager>.
> >> > If you do not use Cluster Manager template, DeltaManager is used by
> >> default.
> >> > However, in the logs, it seems to be using BackupManager.
> >> > Have you set <Context><Manager> in webapps/{your app}
> >> /META-INF/context.xml
> >> > or conf/{Engine}/{Host}/{your app}.xml?
> >> >
> >> >
> >> >
> >> >> <?xml version='1.0' encoding='utf-8'?>
> >> >> <!--
> >> >> Licensed to the Apache Software Foundation (ASF) under one or more
> >> >> contributor license agreements. See the NOTICE file distributed
> with
> >> >> this work for additional information regarding copyright ownership.
> >> >> The ASF licenses this file to You under the Apache License, Version
> >> 2.0
> >> >> (the "License"); you may not use this file except in compliance
> with
> >> >> the License. You may obtain a copy of the License at
> >> >>
> >> >> http://www.apache.org/licenses/LICENSE-2.0
> >> >>
> >> >> Unless required by applicable law or agreed to in writing, software
> >> >> distributed under the License is distributed on an "AS IS" BASIS,
> >> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >> implied.
> >> >> See the License for the specific language governing permissions and
> >> >> limitations under the License.
> >> >> -->
> >> >> <!-- Note: A "Server" is not itself a "Container", so you may not
> >> >> define subcomponents such as "Valves" at this level.
> >> >> Documentation at /docs/config/server.html
> >> >> -->
> >> >> <Server port="8005" shutdown="SHUTDOWN">
> >> >> <Listener className="org.apache.catalina.startup.
> >> VersionLoggerListener"
> >> >> />
> >> >> <!-- Security listener. Documentation at
> /docs/config/listeners.html
> >> >> <Listener className="org.apache.catalina.security.
> SecurityListener"
> >> />
> >> >> -->
> >> >> <!--APR library loader. Documentation at /docs/apr.html -->
> >> >> <Listener className="org.apache.catalina.core.
> AprLifecycleListener"
> >> >> SSLEngine="on" />
> >> >> <!-- Prevent memory leaks due to use of particular java/javax
> APIs-->
> >> >> <Listener className="org.apache.catalina.core.
> >> >> JreMemoryLeakPreventionListener"
> >> >> />
> >> >> <Listener className="org.apache.catalina.mbeans.
> >> >> GlobalResourcesLifecycleListener"
> >> >> />
> >> >> <Listener className="org.apache.catalina.core.
> >> >> ThreadLocalLeakPreventionListener"
> >> >> />
> >> >>
> >> >> <!-- Global JNDI resources
> >> >> Documentation at /docs/jndi-resources-howto.html
> >> >> -->
> >> >> <GlobalNamingResources>
> >> >> <!-- Editable user database that can also be used by
> >> >> UserDatabaseRealm to authenticate users
> >> >> -->
> >> >> <Resource name="UserDatabase" auth="Container"
> >> >> type="org.apache.catalina.UserDatabase"
> >> >> description="User database that can be updated and
> saved"
> >> >> factory="org.apache.catalina.users.
> >> >> MemoryUserDatabaseFactory"
> >> >> pathname="conf/tomcat-users.xml" />
> >> >> </GlobalNamingResources>
> >> >>
> >> >> <!-- A "Service" is a collection of one or more "Connectors" that
> >> share
> >> >> a single "Container" Note: A "Service" is not itself a
> >> "Container",
> >> >> so you may not define subcomponents such as "Valves" at this
> >> level.
> >> >> Documentation at /docs/config/service.html
> >> >> -->
> >> >> <Service name="Catalina">
> >> >>
> >> >> <!--The connectors can use a shared executor, you can define one
> >> >> or more named thread pools-->
> >> >> <!--
> >> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> >> >> maxThreads="150" minSpareThreads="4"/>
> >> >> -->
> >> >>
> >> >>
> >> >> <!-- A "Connector" represents an endpoint by which requests are
> >> >> received
> >> >> and responses are returned. Documentation at :
> >> >> Java HTTP Connector: /docs/config/http.html (blocking &
> >> >> non-blocking)
> >> >> Java AJP Connector: /docs/config/ajp.html
> >> >> APR (HTTP/AJP) Connector: /docs/apr.html
> >> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> >> >> -->
> >> >> <Connector port="8080" protocol="HTTP/1.1"
> >> >> connectionTimeout="60000" maxThreads="500"
> >> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> >> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> >> >> <!-- A "Connector" using the shared thread pool-->
> >> >> <!--
> >> >> <Connector executor="tomcatThreadPool"
> >> >> port="8080" protocol="HTTP/1.1"
> >> >> connectionTimeout="60000"
> >> >> redirectPort="8443" />
> >> >> -->
> >> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> >> >> This connector uses the NIO implementation that requires the
> >> JSSE
> >> >> style configuration. When using the APR/native
> implementation,
> >> the
> >> >> OpenSSL style configuration is required as described in the
> >> >> APR/native
> >> >> documentation -->
> >> >> <!--
> >> >> <Connector port="8443" protocol="org.apache.coyote.
> >> >> http11.Http11NioProtocol"
> >> >> maxThreads="150" SSLEnabled="true" scheme="https"
> >> >> secure="true"
> >> >> clientAuth="false" sslProtocol="TLS" />
> >> >> -->
> >> >>
> >> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> >> >>
> >> >>
> >> >> <!-- An Engine represents the entry point (within Catalina) that
> >> >> processes
> >> >> every request. The Engine implementation for Tomcat stand
> >> alone
> >> >> analyzes the HTTP headers included with the request, and
> passes
> >> >> them
> >> >> on to the appropriate Host (virtual host).
> >> >> Documentation at /docs/config/engine.html -->
> >> >>
> >> >> <!-- You should set jvmRoute to support load-balancing via AJP
> ie :
> >> >>
> >> >> -->
> >> >> <Engine name="Catalina" defaultHost="10.114.43.102"
> >> >> jvmRoute="TomcatNode1">
> >> >> <!-- Use the LockOutRealm to prevent attempts to guess user
> passwords
> >> >> via a brute-force attack -->
> >> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
> >> >> <!-- This Realm uses the UserDatabase configured in the
> global
> >> JNDI
> >> >> resources under the key "UserDatabase". Any edits
> >> >> that are performed against this UserDatabase are
> >> immediately
> >> >> available for use by the Realm. -->
> >> >> <Realm className="org.apache.catalina.realm.
> UserDatabaseRealm"
> >> >> resourceName="UserDatabase"/>
> >> >> </Realm>
> >> >>
> >> >> <Host name="10.114.43.102" appBase="webapps"
> >> >> unpackWARs="true" autoDeploy="true" startStopThreads="0">
> >> >> <!--For clustering, please take a look at documentation at:
> >> >> /docs/cluster-howto.html (simple how to)
> >> >> /docs/config/cluster.html (reference documentation) -->
> >> >> <!--
> >> >> <Cluster className="org.apache.catalina.ha.tcp.
> >> SimpleTcpCluster"/>
> >> >> -->
> >> >>
> >> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> >> >> channelSendOptions="4">
> >> >> <Channel className="org.apache.catalina.tribes.group.
> >> >> GroupChannel">
> >> >> <Membership
> >> >> className="org.apache.catalina.tribes.membership.McastService"
> >> >> address="228.0.0.4"
> >> >> port="45560"
> >> >> frequency="500"
> >> >> dropTime="9000"/>
> >> >> <Receiver
> >> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> >> >> address="auto"
> >> >> port="4000"
> >> >> autoBind="100"
> >> >> selectorTimeout="5000"
> >> >> maxThreads="6"/>
> >> >>
> >> >> <Sender
> >> >> className="org.apache.catalina.tribes.transport.
> >> ReplicationTransmitter">
> >> >> <Transport
> >> >> className="org.apache.catalina.tribes.transport.nio.
> >> >> PooledParallelSender"/>
> >> >> </Sender>
> >> >> <Interceptor
> >> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> TcpFailureDetector"/>
> >> >> <Interceptor
> >> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> MessageDispatch15Interceptor"/>
> >> >> </Channel>
> >> >>
> >> >> <Valve className="org.apache.catalina.ha.tcp.
> >> ReplicationValve"
> >> >>
> >> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >> >> *\.html|.*\.css|.*\.txt"/>
> >> >> <Valve
> >> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
> >> >>
> >> >> <Deployer className="org.apache.catalina.ha.deploy.
> >> >> FarmWarDeployer"
> >> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> >> >> deployDir="D:/imaljava/
> TomcatNode1/tmp/war-deploy/"
> >> >> watchDir="D:/imaljava/
> TomcatNode1/tmp/war-listen/"
> >> >> watchEnabled="false"/>
> >> >>
> >> >> <ClusterListener
> >> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> >> >> </Cluster>
> >> >> <!-- SingleSignOn valve, share authentication between web
> >> >> applications
> >> >> Documentation at: /docs/config/valve.html -->
> >> >> <!--
> >> >> <Valve className="org.apache.catalina.authenticator.
> >> SingleSignOn"
> >> >> />
> >> >> -->
> >> >>
> >> >> <!-- Access log processes all example.
> >> >> Documentation at: /docs/config/valve.html
> >> >> Note: The pattern used is equivalent to using
> >> >> pattern="common" -->
> >> >> <Valve className="org.apache.catalina.valves.AccessLogValve"
> >> >> directory="logs"
> >> >> prefix="localhost_access_log" suffix=".txt"
> >> >> pattern="%h %l %u %t "%r" %s %b" />
> >> >> <Valve className="org.apache.catalina.valves.
> StuckThreadDetectionValve"
> >> >> threshold="900" />
> >> >>
> >> >> </Host>
> >> >> </Engine>
> >> >> </Service>
> >> >> </Server>
> >> >>
> >> >>
> >> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
> >> >> wrote:
> >> >> > Hi
> >> >> >
> >> >> > You seems to be using BackupManager.
> >> >> > What is set for mapSendOptions?
> >> >> >
> >> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> >> > so you can enable logging on the org.apache.catalina.tribes.
> MESSAGES,
> >> >> you
> >> >> > can track messages.
> >> >> >
> >> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
> level
> >> >> logs.
> >> >> > So It should get logs in a short period of time.
> >> >> >
> >> >> >
> >> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >> >
> >> >> >> Guys, we are facing the below errors in Tomcat cluster, please
> advise
> >> >> >>
> >> >> >>
> >> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >> >> org.apache.catalina.tribes.ChannelException: Operation has timed
> >> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >> >> >> at org.apache.catalina.tribes.transport.nio.
> >> >> ParallelNioSender.sendMessage(
> >> >> >> ParallelNioSender.java:108)
> >> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> >> >> >> sendMessage(PooledParallelSender.java:48)
> >> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> >> >> >> sendMessage(ReplicationTransmitter.java:54)
> >> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
> sendMessage(
> >> >> >> ChannelCoordinator.java:82)
> >> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> sendMessage(
> >> >> >> ChannelInterceptorBase.java:76)
> >> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >> >> MessageDispatchInterceptor.sendMessage(
> MessageDispatchInterceptor.
> >> >> java:81)
> >> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> sendMessage(
> >> >> >> ChannelInterceptorBase.java:76)
> >> >> >> at org.apache.catalina.tribes.group.interceptors.
> TcpFailureDetector.
> >> >> >> sendMessage(TcpFailureDetector.java:93)
> >> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> sendMessage(
> >> >> >> ChannelInterceptorBase.java:76)
> >> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> >> GroupChannel.java:233)
> >> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> >> GroupChannel.java:186)
> >> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> >> RpcChannel.java:99)
> >> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> >> ping(AbstractReplicatedMap.java:267)
> >> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> heartbeat(
> >> >> >> AbstractReplicatedMap.java:885)
> >> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> >> >> GroupChannel.java:161)
> >> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> >> HeartbeatThread.run(
> >> >> >> GroupChannel.java:697)
> >> >> >>
> >> >> >>
> >> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
> Unable
> >> to
> >> >> >> process request in NioReceiver
> >> >> >> java.io.IOException: A non-blocking socket operation could not be
> >> >> >> completed immediately
> >> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> >> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> >> >> listen(NioReceiver.java:272)
> >> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >> >> >> NioReceiver.java:425)
> >> >> >> at java.lang.Thread.run(Unknown Source)
> >> >> >>
> >> >> >> Regards,
> >> >> >> Fady
> >> >> >>
> >> >> >> ------------------------------------------------------------
> >> ---------
> >> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Keiichi.Fujino
> >> >>
> >> >> ------------------------------------------------------------
> ---------
> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>
> >> >> --
> >> >> Keiichi.Fujino
> >> >> <us...@tomcat.apache.org>
> >> >> <us...@tomcat.apache.org>
> >> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >> --
> >> Keiichi.Fujino
> >> <us...@tomcat.apache.org>
> >> <us...@tomcat.apache.org>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> --
> Keiichi.Fujino
> <us...@tomcat.apache.org>
> <us...@tomcat.apache.org>
>
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
2017-02-09 19:21 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> i already restarted it after changing the parameter to 8
>
However, I can not see the log that context has been restarted in the log
you showed.
You may need to restart Tomcat.
>
> On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujino <kf...@apache.org>
> wrote:
> > It is realy strange.
> > Have you failed to restart Context?
> > When starting up the AbstractReplicatedMap, the followings should be
> logged.
> >
> > "Initializing AbstractReplicatedMap with context name:xxx"
> > "AbstractReplicatedMap[xxx] initialization was completed in xxx ms."
> >
> >
> > 2017-02-09 19:09 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >
> >> i can only see this error related to AbstractReplicatedMap
> >>
> >>
> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> Unable to send AbstractReplicatedMap.ping message
> >> java.lang.NullPointerException
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> ping(AbstractReplicatedMap.java:256)
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> AbstractReplicatedMap.java:885)
> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> GroupChannel.java:166)
> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> >> GroupChannel.java:697)
> >>
> >> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino <kf...@apache.org>
> >> wrote:
> >> > These logs seem to be the tomcat memory leak detections logs during
> >> context
> >> > restart.
> >> > It is a different issue from this thread's topic.
> >> >
> >> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> >> > initialization was completed in xxx ms." in your log file?
> >> >
> >> > 2017-02-09 18:49 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >
> >> >> Keiichi,
> >> >> once i changed the mapSendOptions and without restarting the below
> >> >> shown in the tomcat
> >> >>
> >> >>
> >> >> 09-Feb-2017 12:44:12.299 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
> >> >> Process}]) but failed to remove it when the web application was
> >> >> stopped. Threads are going to be renewed over time to try and avoid a
> >> >> probable memory leak.
> >> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.302 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.303 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_services] created a ThreadLocal with key of
> >> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >> java.lang.NullPointerException
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> ping(AbstractReplicatedMap.java:256)
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> >> AbstractReplicatedMap.java:885)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> >> GroupChannel.java:166)
> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> HeartbeatThread.run(
> >> >> GroupChannel.java:697)
> >> >>
> >> >> 09-Feb-2017 12:44:14.834 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.835 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.838 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.838 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:14.840 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_services] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.146 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.147 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.147 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.150 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.150 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@6433b5ab]) and a value of type
> >> >> [com.opensymphony.xwork2.ActionContext] (value
> >> >> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to
> remove
> >> >> it when the web application was stopped. Threads are going to be
> >> >> renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@25f6ef48]) and a value of type
> >> >> [org.apache.struts2.dispatcher.Dispatcher] (value
> >> >> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
> >> >> remove it when the web application was stopped. Threads are going to
> >> >> be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.159 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@3adbfb04]) and a value of type
> >> >> [org.springframework.security.core.context.SecurityContextImpl]
> (value
> >> >> [org.springframework.security.core.context.
> SecurityContextImpl@1e3583d5
> >> :
> >> >> Authentication:
> >> >> org.springframework.security.authentication.
> >> UsernamePasswordAuthentication
> >> >> Token@1e3583d5:
> >> >> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
> >> >> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
> >> >> remove it when the web application was stopped. Threads are going to
> >> >> be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.160 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.160 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.161 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.161 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.164 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.164 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.165 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.165 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.166 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.529 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [com.path.lib.common.util.ThreadAttributes$1]
> (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
> >> >> syncClusterOperation}]) but failed to remove it when the web
> >> >> application was stopped. Threads are going to be renewed over time to
> >> >> try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.531 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
> >> >> failed to remove it when the web application was stopped. Threads are
> >> >> going to be renewed over time to try and avoid a probable memory
> leak.
> >> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret5_1486633428990_956105]) but
> failed
> >> >> to remove it when the web application was stopped. Threads are going
> >> >> to be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret7_1486633316229_329875]) but
> failed
> >> >> to remove it when the web application was stopped. Threads are going
> >> >> to be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret13_1486633418550_38295]) but
> failed
> >> >> to remove it when the web application was stopped. Threads are going
> >> >> to be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
> >> >> failed to remove it when the web application was stopped. Threads are
> >> >> going to be renewed over time to try and avoid a probable memory
> leak.
> >> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> >> [java.lang.Class] (value [class ret7_1486633389730_569340]) but
> failed
> >> >> to remove it when the web application was stopped. Threads are going
> >> >> to be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.545 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1]
> (value
> >> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> >> the web application was stopped. Threads are going to be renewed over
> >> >> time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.545 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_reporting_services] created a ThreadLocal
> >> >> with key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> >> [org.apache.commons.collections.ReferenceMap] (value
> >> >> [{WebappClassLoader
> >> >> context: imal_reporting_services
> >> >> delegate: false
> >> >> ----------> Parent Classloader:
> >> >> java.net.URLClassLoader@2e510e35
> >> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8
> }])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [org.springframework.core.NamedThreadLocal] (value
> [Actual
> >> >> transaction active]) and a value of type [java.lang.Boolean] (value
> >> >> [true]) but failed to remove it when the web application was stopped.
> >> >> Threads are going to be renewed over time to try and avoid a probable
> >> >> memory leak.
> >> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> >> [Transactional resources]) and a value of type [java.util.HashMap]
> >> >> (value [{org.apache.ibatis.session.defaults.
> DefaultSqlSessionFactory@
> >> >> 6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
> >> >> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.
> >> >> springframework.jdbc.datasource.ConnectionHolder@73222923}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> >> [Current transaction name]) and a value of type [java.lang.String]
> >> >> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.
> >> >> trxMgntListCountsAndRecords])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.744 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> >> [Current aspect-driven transaction]) and a value of type
> >> >> [org.springframework.transaction.interceptor.
> TransactionAspectSupport.
> >> >> TransactionInfo]
> >> >> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but
> failed
> >> >> to remove it when the web application was stopped. Threads are going
> >> >> to be renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.744 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> >> [Transaction synchronizations]) and a value of type
> >> >> [java.util.LinkedList] (value
> >> >> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$
> >> >> SqlSessionSynchronization@6abd3931]])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.745 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.745 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [java.lang.ThreadLocal] (value
> >> >> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
> >> >> [org.apache.ibatis.executor.ErrorContext] (value [
> >> >> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-
> Inline
> >> >> ### The error occurred while setting parameters]) but failed to
> remove
> >> >> it when the web application was stopped. Threads are going to be
> >> >> renewed over time to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.749 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN
> RCVALERT}])
> >> >> but failed to remove it when the web application was stopped. Threads
> >> >> are going to be renewed over time to try and avoid a probable memory
> >> >> leak.
> >> >> 09-Feb-2017 12:44:17.751 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:17.751 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_core_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.197 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.198 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.198 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.200 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.200 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.201 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.201 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.202 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.364 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.371 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_admin_portal] created a ThreadLocal with
> key
> >> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value
> of
> >> >> type [java.util.HashMap] (value [{}]) but failed to remove it when
> the
> >> >> web application was stopped. Threads are going to be renewed over
> time
> >> >> to try and avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_swift_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> >> >> failed to remove it when the web application was stopped. Threads are
> >> >> going to be renewed over time to try and avoid a probable memory
> leak.
> >> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_swift_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
> >> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> >> was stopped. Threads are going to be renewed over time to try and
> >> >> avoid a probable memory leak.
> >> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> >> checkThreadLocalMapForLeaks
> >> >> The web application [imal_swift_services] created a ThreadLocal with
> >> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value
> of
> >> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> >> >> failed to remove it when the web application was stopped. Threads are
> >> >> going to be renewed over time to try and avoid a probable memory
> leak.
> >> >>
> >> >> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com>
> >> wrote:
> >> >> > i will try that but this required tomcat restart so i will be
> >> >> > scheduling tonight a restart, anyway from where i can enable the
> >> >> > org.apache.catalina.tribes.MESSAGES i didn't found it in the
> >> >> > logging.properties file
> >> >> >
> >> >> > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <
> kfujino@apache.org>
> >> >> wrote:
> >> >> >> Hi
> >> >> >>
> >> >> >> Please try mapSendOptions = 8.
> >> >> >> This means that it does not receive ACK of the messages.
> >> >> >> Processing is returned immediately after message transmission is
> >> >> completed.
> >> >> >>
> >> >> >> As another way,
> >> >> >> Please try the org.apache.catalina.tribes.MESSAGES. (see
> previous my
> >> >> >> response mail.)
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >> >>
> >> >> >>> The <Cluster><Manager> is set under the context.xml
> >> >> >>>
> >> >> >>> <Manager className="org.apache.catalina.ha.session.
> BackupManager"
> >> >> >>> notifyListenersOnReplication="true"
> >> >> >>> mapSendOptions="6"/>
> >> >> >>>
> >> >> >>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <
> kfujino@apache.org
> >> >
> >> >> >>> wrote:
> >> >> >>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >> >>> >
> >> >> >>> >> Keiichi,
> >> >> >>> >> mapSendoption is not configured anyway please check my
> >> configuration
> >> >> >>> below
> >> >> >>> >>
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > You have not set <Cluster><Manager>.
> >> >> >>> > If you do not use Cluster Manager template, DeltaManager is
> used
> >> by
> >> >> >>> default.
> >> >> >>> > However, in the logs, it seems to be using BackupManager.
> >> >> >>> > Have you set <Context><Manager> in webapps/{your app}
> >> >> >>> /META-INF/context.xml
> >> >> >>> > or conf/{Engine}/{Host}/{your app}.xml?
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >> <?xml version='1.0' encoding='utf-8'?>
> >> >> >>> >> <!--
> >> >> >>> >> Licensed to the Apache Software Foundation (ASF) under one
> or
> >> more
> >> >> >>> >> contributor license agreements. See the NOTICE file
> >> distributed
> >> >> with
> >> >> >>> >> this work for additional information regarding copyright
> >> >> ownership.
> >> >> >>> >> The ASF licenses this file to You under the Apache License,
> >> >> Version
> >> >> >>> 2.0
> >> >> >>> >> (the "License"); you may not use this file except in
> compliance
> >> >> with
> >> >> >>> >> the License. You may obtain a copy of the License at
> >> >> >>> >>
> >> >> >>> >> http://www.apache.org/licenses/LICENSE-2.0
> >> >> >>> >>
> >> >> >>> >> Unless required by applicable law or agreed to in writing,
> >> >> software
> >> >> >>> >> distributed under the License is distributed on an "AS IS"
> >> BASIS,
> >> >> >>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or
> >> >> >>> implied.
> >> >> >>> >> See the License for the specific language governing
> permissions
> >> >> and
> >> >> >>> >> limitations under the License.
> >> >> >>> >> -->
> >> >> >>> >> <!-- Note: A "Server" is not itself a "Container", so you may
> >> not
> >> >> >>> >> define subcomponents such as "Valves" at this level.
> >> >> >>> >> Documentation at /docs/config/server.html
> >> >> >>> >> -->
> >> >> >>> >> <Server port="8005" shutdown="SHUTDOWN">
> >> >> >>> >> <Listener className="org.apache.catalina.startup.
> >> >> >>> VersionLoggerListener"
> >> >> >>> >> />
> >> >> >>> >> <!-- Security listener. Documentation at
> >> >> /docs/config/listeners.html
> >> >> >>> >> <Listener className="org.apache.catalina.security.
> >> >> SecurityListener"
> >> >> >>> />
> >> >> >>> >> -->
> >> >> >>> >> <!--APR library loader. Documentation at /docs/apr.html -->
> >> >> >>> >> <Listener className="org.apache.catalina.core.
> >> >> AprLifecycleListener"
> >> >> >>> >> SSLEngine="on" />
> >> >> >>> >> <!-- Prevent memory leaks due to use of particular
> java/javax
> >> >> APIs-->
> >> >> >>> >> <Listener className="org.apache.catalina.core.
> >> >> >>> >> JreMemoryLeakPreventionListener"
> >> >> >>> >> />
> >> >> >>> >> <Listener className="org.apache.catalina.mbeans.
> >> >> >>> >> GlobalResourcesLifecycleListener"
> >> >> >>> >> />
> >> >> >>> >> <Listener className="org.apache.catalina.core.
> >> >> >>> >> ThreadLocalLeakPreventionListener"
> >> >> >>> >> />
> >> >> >>> >>
> >> >> >>> >> <!-- Global JNDI resources
> >> >> >>> >> Documentation at /docs/jndi-resources-howto.html
> >> >> >>> >> -->
> >> >> >>> >> <GlobalNamingResources>
> >> >> >>> >> <!-- Editable user database that can also be used by
> >> >> >>> >> UserDatabaseRealm to authenticate users
> >> >> >>> >> -->
> >> >> >>> >> <Resource name="UserDatabase" auth="Container"
> >> >> >>> >> type="org.apache.catalina.UserDatabase"
> >> >> >>> >> description="User database that can be updated
> and
> >> >> saved"
> >> >> >>> >> factory="org.apache.catalina.users.
> >> >> >>> >> MemoryUserDatabaseFactory"
> >> >> >>> >> pathname="conf/tomcat-users.xml" />
> >> >> >>> >> </GlobalNamingResources>
> >> >> >>> >>
> >> >> >>> >> <!-- A "Service" is a collection of one or more "Connectors"
> >> that
> >> >> >>> share
> >> >> >>> >> a single "Container" Note: A "Service" is not itself a
> >> >> >>> "Container",
> >> >> >>> >> so you may not define subcomponents such as "Valves" at
> >> this
> >> >> >>> level.
> >> >> >>> >> Documentation at /docs/config/service.html
> >> >> >>> >> -->
> >> >> >>> >> <Service name="Catalina">
> >> >> >>> >>
> >> >> >>> >> <!--The connectors can use a shared executor, you can
> define
> >> one
> >> >> >>> >> or more named thread pools-->
> >> >> >>> >> <!--
> >> >> >>> >> <Executor name="tomcatThreadPool"
> namePrefix="catalina-exec-"
> >> >> >>> >> maxThreads="150" minSpareThreads="4"/>
> >> >> >>> >> -->
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> <!-- A "Connector" represents an endpoint by which
> requests
> >> are
> >> >> >>> >> received
> >> >> >>> >> and responses are returned. Documentation at :
> >> >> >>> >> Java HTTP Connector: /docs/config/http.html
> (blocking &
> >> >> >>> >> non-blocking)
> >> >> >>> >> Java AJP Connector: /docs/config/ajp.html
> >> >> >>> >> APR (HTTP/AJP) Connector: /docs/apr.html
> >> >> >>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> >> >> >>> >> -->
> >> >> >>> >> <Connector port="8080" protocol="HTTP/1.1"
> >> >> >>> >> connectionTimeout="60000" maxThreads="500"
> >> >> >>> >> minSpareThreads="25" maxSpareThreads="75"
> enableLookups="false"
> >> >> >>> >> disableUploadTimeout="true" acceptCount="100"
> >> redirectPort="8443" />
> >> >> >>> >> <!-- A "Connector" using the shared thread pool-->
> >> >> >>> >> <!--
> >> >> >>> >> <Connector executor="tomcatThreadPool"
> >> >> >>> >> port="8080" protocol="HTTP/1.1"
> >> >> >>> >> connectionTimeout="60000"
> >> >> >>> >> redirectPort="8443" />
> >> >> >>> >> -->
> >> >> >>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> >> >> >>> >> This connector uses the NIO implementation that
> requires
> >> >> the
> >> >> >>> JSSE
> >> >> >>> >> style configuration. When using the APR/native
> >> >> implementation,
> >> >> >>> the
> >> >> >>> >> OpenSSL style configuration is required as described
> in
> >> the
> >> >> >>> >> APR/native
> >> >> >>> >> documentation -->
> >> >> >>> >> <!--
> >> >> >>> >> <Connector port="8443" protocol="org.apache.coyote.
> >> >> >>> >> http11.Http11NioProtocol"
> >> >> >>> >> maxThreads="150" SSLEnabled="true"
> scheme="https"
> >> >> >>> >> secure="true"
> >> >> >>> >> clientAuth="false" sslProtocol="TLS" />
> >> >> >>> >> -->
> >> >> >>> >>
> >> >> >>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >> >> >>> >> <Connector port="8009" protocol="AJP/1.3"
> >> redirectPort="8443" />
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> <!-- An Engine represents the entry point (within
> Catalina)
> >> that
> >> >> >>> >> processes
> >> >> >>> >> every request. The Engine implementation for Tomcat
> >> stand
> >> >> >>> alone
> >> >> >>> >> analyzes the HTTP headers included with the request,
> and
> >> >> passes
> >> >> >>> >> them
> >> >> >>> >> on to the appropriate Host (virtual host).
> >> >> >>> >> Documentation at /docs/config/engine.html -->
> >> >> >>> >>
> >> >> >>> >> <!-- You should set jvmRoute to support load-balancing via
> >> AJP
> >> >> ie :
> >> >> >>> >>
> >> >> >>> >> -->
> >> >> >>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
> >> >> >>> >> jvmRoute="TomcatNode1">
> >> >> >>> >> <!-- Use the LockOutRealm to prevent attempts to guess user
> >> >> passwords
> >> >> >>> >> via a brute-force attack -->
> >> >> >>> >> <Realm className="org.apache.
> catalina.realm.LockOutRealm">
> >> >> >>> >> <!-- This Realm uses the UserDatabase configured in
> the
> >> >> global
> >> >> >>> JNDI
> >> >> >>> >> resources under the key "UserDatabase". Any
> edits
> >> >> >>> >> that are performed against this UserDatabase are
> >> >> >>> immediately
> >> >> >>> >> available for use by the Realm. -->
> >> >> >>> >> <Realm className="org.apache.catalina.realm.
> >> >> UserDatabaseRealm"
> >> >> >>> >> resourceName="UserDatabase"/>
> >> >> >>> >> </Realm>
> >> >> >>> >>
> >> >> >>> >> <Host name="10.114.43.102" appBase="webapps"
> >> >> >>> >> unpackWARs="true" autoDeploy="true"
> >> >> startStopThreads="0">
> >> >> >>> >> <!--For clustering, please take a look at documentation
> at:
> >> >> >>> >> /docs/cluster-howto.html (simple how to)
> >> >> >>> >> /docs/config/cluster.html (reference documentation)
> -->
> >> >> >>> >> <!--
> >> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
> >> >> >>> SimpleTcpCluster"/>
> >> >> >>> >> -->
> >> >> >>> >>
> >> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
> >> SimpleTcpCluster"
> >> >> >>> >> channelSendOptions="4">
> >> >> >>> >> <Channel className="org.apache.
> catalina.tribes.group.
> >> >> >>> >> GroupChannel">
> >> >> >>> >> <Membership
> >> >> >>> >> className="org.apache.catalina.tribes.membership.
> McastService"
> >> >> >>> >> address="228.0.0.4"
> >> >> >>> >> port="45560"
> >> >> >>> >> frequency="500"
> >> >> >>> >> dropTime="9000"/>
> >> >> >>> >> <Receiver
> >> >> >>> >> className="org.apache.catalina.tribes.transport.nio.
> NioReceiver"
> >> >> >>> >> address="auto"
> >> >> >>> >> port="4000"
> >> >> >>> >> autoBind="100"
> >> >> >>> >> selectorTimeout="5000"
> >> >> >>> >> maxThreads="6"/>
> >> >> >>> >>
> >> >> >>> >> <Sender
> >> >> >>> >> className="org.apache.catalina.tribes.transport.
> >> >> >>> ReplicationTransmitter">
> >> >> >>> >> <Transport
> >> >> >>> >> className="org.apache.catalina.tribes.transport.nio.
> >> >> >>> >> PooledParallelSender"/>
> >> >> >>> >> </Sender>
> >> >> >>> >> <Interceptor
> >> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> >>> >> TcpFailureDetector"/>
> >> >> >>> >> <Interceptor
> >> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >> >>> >> MessageDispatch15Interceptor"/>
> >> >> >>> >> </Channel>
> >> >> >>> >>
> >> >> >>> >> <Valve className="org.apache.catalina.ha.tcp.
> >> >> >>> ReplicationValve"
> >> >> >>> >>
> >> >> >>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >> >> >>> >> *\.html|.*\.css|.*\.txt"/>
> >> >> >>> >> <Valve
> >> >> >>> >> className="org.apache.catalina.ha.session.
> JvmRouteBinderValve"/>
> >> >> >>> >>
> >> >> >>> >> <Deployer className="org.apache.catalina.ha.deploy.
> >> >> >>> >> FarmWarDeployer"
> >> >> >>> >> tempDir="D:/imaljava/
> >> TomcatNode1/tmp/war-temp/"
> >> >> >>> >> deployDir="D:/imaljava/
> >> >> TomcatNode1/tmp/war-deploy/"
> >> >> >>> >> watchDir="D:/imaljava/
> >> >> TomcatNode1/tmp/war-listen/"
> >> >> >>> >> watchEnabled="false"/>
> >> >> >>> >>
> >> >> >>> >> <ClusterListener
> >> >> >>> >> className="org.apache.catalina.ha.session.
> >> ClusterSessionListener"/>
> >> >> >>> >> </Cluster>
> >> >> >>> >> <!-- SingleSignOn valve, share authentication between
> web
> >> >> >>> >> applications
> >> >> >>> >> Documentation at: /docs/config/valve.html -->
> >> >> >>> >> <!--
> >> >> >>> >> <Valve className="org.apache.catalina.authenticator.
> >> >> >>> SingleSignOn"
> >> >> >>> >> />
> >> >> >>> >> -->
> >> >> >>> >>
> >> >> >>> >> <!-- Access log processes all example.
> >> >> >>> >> Documentation at: /docs/config/valve.html
> >> >> >>> >> Note: The pattern used is equivalent to using
> >> >> >>> >> pattern="common" -->
> >> >> >>> >> <Valve className="org.apache.catalina.valves.
> >> >> AccessLogValve"
> >> >> >>> >> directory="logs"
> >> >> >>> >> prefix="localhost_access_log" suffix=".txt"
> >> >> >>> >> pattern="%h %l %u %t "%r" %s %b" />
> >> >> >>> >> <Valve className="org.apache.catalina.valves.
> >> >> StuckThreadDetectionValve"
> >> >> >>> >> threshold="900" />
> >> >> >>> >>
> >> >> >>> >> </Host>
> >> >> >>> >> </Engine>
> >> >> >>> >> </Service>
> >> >> >>> >> </Server>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <
> >> kfujino@apache.org
> >> >> >
> >> >> >>> >> wrote:
> >> >> >>> >> > Hi
> >> >> >>> >> >
> >> >> >>> >> > You seems to be using BackupManager.
> >> >> >>> >> > What is set for mapSendOptions?
> >> >> >>> >> >
> >> >> >>> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> >> >>> >> > so you can enable logging on the org.apache.catalina.tribes.
> >> >> MESSAGES,
> >> >> >>> >> you
> >> >> >>> >> > can track messages.
> >> >> >>> >> >
> >> >> >>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> >> >>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of
> trace
> >> >> level
> >> >> >>> >> logs.
> >> >> >>> >> > So It should get logs in a short period of time.
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <
> fadyhaikal@gmail.com>:
> >> >> >>> >> >
> >> >> >>> >> >> Guys, we are facing the below errors in Tomcat cluster,
> please
> >> >> advise
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> >> >>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> heartbeat
> >> >> >>> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >> >>> >> >> org.apache.catalina.tribes.ChannelException: Operation
> has
> >> >> timed
> >> >> >>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43,
> 103}:4000;
> >> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> >> >> >>> >> ParallelNioSender.sendMessage(
> >> >> >>> >> >> ParallelNioSender.java:108)
> >> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> >> >> PooledParallelSender.
> >> >> >>> >> >> sendMessage(PooledParallelSender.java:48)
> >> >> >>> >> >> at org.apache.catalina.tribes.transport.
> >> ReplicationTransmitter.
> >> >> >>> >> >> sendMessage(ReplicationTransmitter.java:54)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
> >> >> sendMessage(
> >> >> >>> >> >> ChannelCoordinator.java:82)
> >> >> >>> >> >> at org.apache.catalina.tribes.
> group.ChannelInterceptorBase.
> >> >> >>> sendMessage(
> >> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >> >>> >> >> MessageDispatchInterceptor.sendMessage(
> >> >> MessageDispatchInterceptor.
> >> >> >>> >> java:81)
> >> >> >>> >> >> at org.apache.catalina.tribes.
> group.ChannelInterceptorBase.
> >> >> >>> sendMessage(
> >> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >> TcpFailureDetector.
> >> >> >>> >> >> sendMessage(TcpFailureDetector.java:93)
> >> >> >>> >> >> at org.apache.catalina.tribes.
> group.ChannelInterceptorBase.
> >> >> >>> sendMessage(
> >> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> >>> >> >> GroupChannel.java:233)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> >>> >> >> GroupChannel.java:186)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> >> >> >>> RpcChannel.java:99)
> >> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> >>> >> >> ping(AbstractReplicatedMap.java:267)
> >> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> heartbeat(
> >> >> >>> >> >> AbstractReplicatedMap.java:885)
> >> >> >>> >> >> at org.apache.catalina.tribes.
> group.GroupChannel.heartbeat(
> >> >> >>> >> >> GroupChannel.java:161)
> >> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> >> >> >>> HeartbeatThread.run(
> >> >> >>> >> >> GroupChannel.java:697)
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> >> >>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.
> listen
> >> >> Unable
> >> >> >>> to
> >> >> >>> >> >> process request in NioReceiver
> >> >> >>> >> >> java.io.IOException: A non-blocking socket operation could
> >> not
> >> >> be
> >> >> >>> >> >> completed immediately
> >> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> >> >>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown
> Source)
> >> >> >>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
> >> >> Source)
> >> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> >> >>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> >> >>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> >> >>> >> >> listen(NioReceiver.java:272)
> >> >> >>> >> >> at org.apache.catalina.tribes.
> transport.nio.NioReceiver.run(
> >> >> >>> >> >> NioReceiver.java:425)
> >> >> >>> >> >> at java.lang.Thread.run(Unknown Source)
> >> >> >>> >> >>
> >> >> >>> >> >> Regards,
> >> >> >>> >> >> Fady
> >> >> >>> >> >>
> >> >> >>> >> >> ------------------------------
> ------------------------------
> >> >> >>> ---------
> >> >> >>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.
> apache.org
> >> >> >>> >> >> For additional commands, e-mail:
> users-help@tomcat.apache.org
> >> >> >>> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > --
> >> >> >>> >> > Keiichi.Fujino
> >> >> >>> >>
> >> >> >>> >> ------------------------------------------------------------
> >> >> ---------
> >> >> >>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> >>> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >> >>> >>
> >> >> >>> >> --
> >> >> >>> >> Keiichi.Fujino
> >> >> >>> >> <us...@tomcat.apache.org>
> >> >> >>> >> <us...@tomcat.apache.org>
> >> >> >>> >>
> >> >> >>>
> >> >> >>> ------------------------------------------------------------
> >> ---------
> >> >> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >> >>>
> >> >> >>> --
> >> >> >>> Keiichi.Fujino
> >> >> >>> <us...@tomcat.apache.org>
> >> >> >>> <us...@tomcat.apache.org>
> >> >> >>>
> >> >>
> >> >> ------------------------------------------------------------
> ---------
> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Keiichi.Fujino
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >> --
> >> Keiichi.Fujino
> >> <us...@tomcat.apache.org>
> >
> > <us...@tomcat.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> --
> Keiichi.Fujino
> <us...@tomcat.apache.org>
> <us...@tomcat.apache.org>
>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
i already restarted it after changing the parameter to 8
On Thu, Feb 9, 2017 at 12:16 PM, Keiichi Fujino <kf...@apache.org> wrote:
> It is realy strange.
> Have you failed to restart Context?
> When starting up the AbstractReplicatedMap, the followings should be logged.
>
> "Initializing AbstractReplicatedMap with context name:xxx"
> "AbstractReplicatedMap[xxx] initialization was completed in xxx ms."
>
>
> 2017-02-09 19:09 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>
>> i can only see this error related to AbstractReplicatedMap
>>
>>
>> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>> java.lang.NullPointerException
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> ping(AbstractReplicatedMap.java:256)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> GroupChannel.java:166)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> GroupChannel.java:697)
>>
>> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino <kf...@apache.org>
>> wrote:
>> > These logs seem to be the tomcat memory leak detections logs during
>> context
>> > restart.
>> > It is a different issue from this thread's topic.
>> >
>> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
>> > initialization was completed in xxx ms." in your log file?
>> >
>> > 2017-02-09 18:49 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >
>> >> Keiichi,
>> >> once i changed the mapSendOptions and without restarting the below
>> >> shown in the tomcat
>> >>
>> >>
>> >> 09-Feb-2017 12:44:12.299 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.300 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.300 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.301 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
>> >> Process}]) but failed to remove it when the web application was
>> >> stopped. Threads are going to be renewed over time to try and avoid a
>> >> probable memory leak.
>> >> 09-Feb-2017 12:44:12.301 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.302 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.303 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_services] created a ThreadLocal with key of
>> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
>> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >> Unable to send AbstractReplicatedMap.ping message
>> >> java.lang.NullPointerException
>> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> ping(AbstractReplicatedMap.java:256)
>> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> >> AbstractReplicatedMap.java:885)
>> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >> GroupChannel.java:166)
>> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> >> GroupChannel.java:697)
>> >>
>> >> 09-Feb-2017 12:44:14.834 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.835 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.838 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.838 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.839 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.839 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.839 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:14.840 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_services] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.146 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.147 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.147 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.148 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.148 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.148 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.149 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.149 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.149 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.150 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.150 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.151 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@6433b5ab]) and a value of type
>> >> [com.opensymphony.xwork2.ActionContext] (value
>> >> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
>> >> it when the web application was stopped. Threads are going to be
>> >> renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.151 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@25f6ef48]) and a value of type
>> >> [org.apache.struts2.dispatcher.Dispatcher] (value
>> >> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
>> >> remove it when the web application was stopped. Threads are going to
>> >> be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.151 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.159 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@3adbfb04]) and a value of type
>> >> [org.springframework.security.core.context.SecurityContextImpl] (value
>> >> [org.springframework.security.core.context.SecurityContextImpl@1e3583d5
>> :
>> >> Authentication:
>> >> org.springframework.security.authentication.
>> UsernamePasswordAuthentication
>> >> Token@1e3583d5:
>> >> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
>> >> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
>> >> remove it when the web application was stopped. Threads are going to
>> >> be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.160 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.160 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.161 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.161 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.163 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.163 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.163 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.164 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.164 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.165 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.165 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.166 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.529 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
>> >> syncClusterOperation}]) but failed to remove it when the web
>> >> application was stopped. Threads are going to be renewed over time to
>> >> try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.531 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
>> >> failed to remove it when the web application was stopped. Threads are
>> >> going to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.541 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.541 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.541 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
>> >> to remove it when the web application was stopped. Threads are going
>> >> to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.541 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.542 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.542 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
>> >> to remove it when the web application was stopped. Threads are going
>> >> to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.542 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.542 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.543 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
>> >> to remove it when the web application was stopped. Threads are going
>> >> to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.543 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.543 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.543 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
>> >> failed to remove it when the web application was stopped. Threads are
>> >> going to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.544 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.544 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.544 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> >> [java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
>> >> to remove it when the web application was stopped. Threads are going
>> >> to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.544 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.545 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> >> the web application was stopped. Threads are going to be renewed over
>> >> time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.545 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_reporting_services] created a ThreadLocal
>> >> with key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> >> [org.apache.commons.collections.ReferenceMap] (value
>> >> [{WebappClassLoader
>> >> context: imal_reporting_services
>> >> delegate: false
>> >> ----------> Parent Classloader:
>> >> java.net.URLClassLoader@2e510e35
>> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.739 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.739 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [org.springframework.core.NamedThreadLocal] (value [Actual
>> >> transaction active]) and a value of type [java.lang.Boolean] (value
>> >> [true]) but failed to remove it when the web application was stopped.
>> >> Threads are going to be renewed over time to try and avoid a probable
>> >> memory leak.
>> >> 09-Feb-2017 12:44:17.739 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [org.springframework.core.NamedThreadLocal] (value
>> >> [Transactional resources]) and a value of type [java.util.HashMap]
>> >> (value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@
>> >> 6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
>> >> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.
>> >> springframework.jdbc.datasource.ConnectionHolder@73222923}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.739 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [org.springframework.core.NamedThreadLocal] (value
>> >> [Current transaction name]) and a value of type [java.lang.String]
>> >> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.
>> >> trxMgntListCountsAndRecords])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.744 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [org.springframework.core.NamedThreadLocal] (value
>> >> [Current aspect-driven transaction]) and a value of type
>> >> [org.springframework.transaction.interceptor.TransactionAspectSupport.
>> >> TransactionInfo]
>> >> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
>> >> to remove it when the web application was stopped. Threads are going
>> >> to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.744 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [org.springframework.core.NamedThreadLocal] (value
>> >> [Transaction synchronizations]) and a value of type
>> >> [java.util.LinkedList] (value
>> >> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$
>> >> SqlSessionSynchronization@6abd3931]])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.745 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.745 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [java.lang.ThreadLocal] (value
>> >> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
>> >> [org.apache.ibatis.executor.ErrorContext] (value [
>> >> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
>> >> ### The error occurred while setting parameters]) but failed to remove
>> >> it when the web application was stopped. Threads are going to be
>> >> renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.749 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.750 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.750 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.750 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
>> >> but failed to remove it when the web application was stopped. Threads
>> >> are going to be renewed over time to try and avoid a probable memory
>> >> leak.
>> >> 09-Feb-2017 12:44:17.751 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:17.751 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_core_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.197 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.198 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.198 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.199 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.199 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.199 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.200 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.200 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.201 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.201 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.202 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_tfa_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.364 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.365 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.365 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.365 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.365 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.366 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.366 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.366 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.366 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.367 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.367 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.367 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.368 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.368 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.368 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.368 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.369 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.369 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.369 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.370 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.370 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.370 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.370 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.371 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_admin_portal] created a ThreadLocal with key
>> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> >> web application was stopped. Threads are going to be renewed over time
>> >> to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.746 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_swift_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
>> >> failed to remove it when the web application was stopped. Threads are
>> >> going to be renewed over time to try and avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.746 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_swift_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
>> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> >> was stopped. Threads are going to be renewed over time to try and
>> >> avoid a probable memory leak.
>> >> 09-Feb-2017 12:44:18.746 SEVERE
>> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> >> org.apache.catalina.loader.WebappClassLoaderBase.
>> >> checkThreadLocalMapForLeaks
>> >> The web application [imal_swift_services] created a ThreadLocal with
>> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
>> >> failed to remove it when the web application was stopped. Threads are
>> >> going to be renewed over time to try and avoid a probable memory leak.
>> >>
>> >> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com>
>> wrote:
>> >> > i will try that but this required tomcat restart so i will be
>> >> > scheduling tonight a restart, anyway from where i can enable the
>> >> > org.apache.catalina.tribes.MESSAGES i didn't found it in the
>> >> > logging.properties file
>> >> >
>> >> > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org>
>> >> wrote:
>> >> >> Hi
>> >> >>
>> >> >> Please try mapSendOptions = 8.
>> >> >> This means that it does not receive ACK of the messages.
>> >> >> Processing is returned immediately after message transmission is
>> >> completed.
>> >> >>
>> >> >> As another way,
>> >> >> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
>> >> >> response mail.)
>> >> >>
>> >> >>
>> >> >>
>> >> >> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >> >>
>> >> >>> The <Cluster><Manager> is set under the context.xml
>> >> >>>
>> >> >>> <Manager className="org.apache.catalina.ha.session.BackupManager"
>> >> >>> notifyListenersOnReplication="true"
>> >> >>> mapSendOptions="6"/>
>> >> >>>
>> >> >>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kfujino@apache.org
>> >
>> >> >>> wrote:
>> >> >>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >> >>> >
>> >> >>> >> Keiichi,
>> >> >>> >> mapSendoption is not configured anyway please check my
>> configuration
>> >> >>> below
>> >> >>> >>
>> >> >>> >
>> >> >>> >
>> >> >>> > You have not set <Cluster><Manager>.
>> >> >>> > If you do not use Cluster Manager template, DeltaManager is used
>> by
>> >> >>> default.
>> >> >>> > However, in the logs, it seems to be using BackupManager.
>> >> >>> > Have you set <Context><Manager> in webapps/{your app}
>> >> >>> /META-INF/context.xml
>> >> >>> > or conf/{Engine}/{Host}/{your app}.xml?
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> >> <?xml version='1.0' encoding='utf-8'?>
>> >> >>> >> <!--
>> >> >>> >> Licensed to the Apache Software Foundation (ASF) under one or
>> more
>> >> >>> >> contributor license agreements. See the NOTICE file
>> distributed
>> >> with
>> >> >>> >> this work for additional information regarding copyright
>> >> ownership.
>> >> >>> >> The ASF licenses this file to You under the Apache License,
>> >> Version
>> >> >>> 2.0
>> >> >>> >> (the "License"); you may not use this file except in compliance
>> >> with
>> >> >>> >> the License. You may obtain a copy of the License at
>> >> >>> >>
>> >> >>> >> http://www.apache.org/licenses/LICENSE-2.0
>> >> >>> >>
>> >> >>> >> Unless required by applicable law or agreed to in writing,
>> >> software
>> >> >>> >> distributed under the License is distributed on an "AS IS"
>> BASIS,
>> >> >>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> >> >>> implied.
>> >> >>> >> See the License for the specific language governing permissions
>> >> and
>> >> >>> >> limitations under the License.
>> >> >>> >> -->
>> >> >>> >> <!-- Note: A "Server" is not itself a "Container", so you may
>> not
>> >> >>> >> define subcomponents such as "Valves" at this level.
>> >> >>> >> Documentation at /docs/config/server.html
>> >> >>> >> -->
>> >> >>> >> <Server port="8005" shutdown="SHUTDOWN">
>> >> >>> >> <Listener className="org.apache.catalina.startup.
>> >> >>> VersionLoggerListener"
>> >> >>> >> />
>> >> >>> >> <!-- Security listener. Documentation at
>> >> /docs/config/listeners.html
>> >> >>> >> <Listener className="org.apache.catalina.security.
>> >> SecurityListener"
>> >> >>> />
>> >> >>> >> -->
>> >> >>> >> <!--APR library loader. Documentation at /docs/apr.html -->
>> >> >>> >> <Listener className="org.apache.catalina.core.
>> >> AprLifecycleListener"
>> >> >>> >> SSLEngine="on" />
>> >> >>> >> <!-- Prevent memory leaks due to use of particular java/javax
>> >> APIs-->
>> >> >>> >> <Listener className="org.apache.catalina.core.
>> >> >>> >> JreMemoryLeakPreventionListener"
>> >> >>> >> />
>> >> >>> >> <Listener className="org.apache.catalina.mbeans.
>> >> >>> >> GlobalResourcesLifecycleListener"
>> >> >>> >> />
>> >> >>> >> <Listener className="org.apache.catalina.core.
>> >> >>> >> ThreadLocalLeakPreventionListener"
>> >> >>> >> />
>> >> >>> >>
>> >> >>> >> <!-- Global JNDI resources
>> >> >>> >> Documentation at /docs/jndi-resources-howto.html
>> >> >>> >> -->
>> >> >>> >> <GlobalNamingResources>
>> >> >>> >> <!-- Editable user database that can also be used by
>> >> >>> >> UserDatabaseRealm to authenticate users
>> >> >>> >> -->
>> >> >>> >> <Resource name="UserDatabase" auth="Container"
>> >> >>> >> type="org.apache.catalina.UserDatabase"
>> >> >>> >> description="User database that can be updated and
>> >> saved"
>> >> >>> >> factory="org.apache.catalina.users.
>> >> >>> >> MemoryUserDatabaseFactory"
>> >> >>> >> pathname="conf/tomcat-users.xml" />
>> >> >>> >> </GlobalNamingResources>
>> >> >>> >>
>> >> >>> >> <!-- A "Service" is a collection of one or more "Connectors"
>> that
>> >> >>> share
>> >> >>> >> a single "Container" Note: A "Service" is not itself a
>> >> >>> "Container",
>> >> >>> >> so you may not define subcomponents such as "Valves" at
>> this
>> >> >>> level.
>> >> >>> >> Documentation at /docs/config/service.html
>> >> >>> >> -->
>> >> >>> >> <Service name="Catalina">
>> >> >>> >>
>> >> >>> >> <!--The connectors can use a shared executor, you can define
>> one
>> >> >>> >> or more named thread pools-->
>> >> >>> >> <!--
>> >> >>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> >> >>> >> maxThreads="150" minSpareThreads="4"/>
>> >> >>> >> -->
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> <!-- A "Connector" represents an endpoint by which requests
>> are
>> >> >>> >> received
>> >> >>> >> and responses are returned. Documentation at :
>> >> >>> >> Java HTTP Connector: /docs/config/http.html (blocking &
>> >> >>> >> non-blocking)
>> >> >>> >> Java AJP Connector: /docs/config/ajp.html
>> >> >>> >> APR (HTTP/AJP) Connector: /docs/apr.html
>> >> >>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> >> >>> >> -->
>> >> >>> >> <Connector port="8080" protocol="HTTP/1.1"
>> >> >>> >> connectionTimeout="60000" maxThreads="500"
>> >> >>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> >> >>> >> disableUploadTimeout="true" acceptCount="100"
>> redirectPort="8443" />
>> >> >>> >> <!-- A "Connector" using the shared thread pool-->
>> >> >>> >> <!--
>> >> >>> >> <Connector executor="tomcatThreadPool"
>> >> >>> >> port="8080" protocol="HTTP/1.1"
>> >> >>> >> connectionTimeout="60000"
>> >> >>> >> redirectPort="8443" />
>> >> >>> >> -->
>> >> >>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> >> >>> >> This connector uses the NIO implementation that requires
>> >> the
>> >> >>> JSSE
>> >> >>> >> style configuration. When using the APR/native
>> >> implementation,
>> >> >>> the
>> >> >>> >> OpenSSL style configuration is required as described in
>> the
>> >> >>> >> APR/native
>> >> >>> >> documentation -->
>> >> >>> >> <!--
>> >> >>> >> <Connector port="8443" protocol="org.apache.coyote.
>> >> >>> >> http11.Http11NioProtocol"
>> >> >>> >> maxThreads="150" SSLEnabled="true" scheme="https"
>> >> >>> >> secure="true"
>> >> >>> >> clientAuth="false" sslProtocol="TLS" />
>> >> >>> >> -->
>> >> >>> >>
>> >> >>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> >> >>> >> <Connector port="8009" protocol="AJP/1.3"
>> redirectPort="8443" />
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> <!-- An Engine represents the entry point (within Catalina)
>> that
>> >> >>> >> processes
>> >> >>> >> every request. The Engine implementation for Tomcat
>> stand
>> >> >>> alone
>> >> >>> >> analyzes the HTTP headers included with the request, and
>> >> passes
>> >> >>> >> them
>> >> >>> >> on to the appropriate Host (virtual host).
>> >> >>> >> Documentation at /docs/config/engine.html -->
>> >> >>> >>
>> >> >>> >> <!-- You should set jvmRoute to support load-balancing via
>> AJP
>> >> ie :
>> >> >>> >>
>> >> >>> >> -->
>> >> >>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
>> >> >>> >> jvmRoute="TomcatNode1">
>> >> >>> >> <!-- Use the LockOutRealm to prevent attempts to guess user
>> >> passwords
>> >> >>> >> via a brute-force attack -->
>> >> >>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> >> >>> >> <!-- This Realm uses the UserDatabase configured in the
>> >> global
>> >> >>> JNDI
>> >> >>> >> resources under the key "UserDatabase". Any edits
>> >> >>> >> that are performed against this UserDatabase are
>> >> >>> immediately
>> >> >>> >> available for use by the Realm. -->
>> >> >>> >> <Realm className="org.apache.catalina.realm.
>> >> UserDatabaseRealm"
>> >> >>> >> resourceName="UserDatabase"/>
>> >> >>> >> </Realm>
>> >> >>> >>
>> >> >>> >> <Host name="10.114.43.102" appBase="webapps"
>> >> >>> >> unpackWARs="true" autoDeploy="true"
>> >> startStopThreads="0">
>> >> >>> >> <!--For clustering, please take a look at documentation at:
>> >> >>> >> /docs/cluster-howto.html (simple how to)
>> >> >>> >> /docs/config/cluster.html (reference documentation) -->
>> >> >>> >> <!--
>> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
>> >> >>> SimpleTcpCluster"/>
>> >> >>> >> -->
>> >> >>> >>
>> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
>> SimpleTcpCluster"
>> >> >>> >> channelSendOptions="4">
>> >> >>> >> <Channel className="org.apache.catalina.tribes.group.
>> >> >>> >> GroupChannel">
>> >> >>> >> <Membership
>> >> >>> >> className="org.apache.catalina.tribes.membership.McastService"
>> >> >>> >> address="228.0.0.4"
>> >> >>> >> port="45560"
>> >> >>> >> frequency="500"
>> >> >>> >> dropTime="9000"/>
>> >> >>> >> <Receiver
>> >> >>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> >> >>> >> address="auto"
>> >> >>> >> port="4000"
>> >> >>> >> autoBind="100"
>> >> >>> >> selectorTimeout="5000"
>> >> >>> >> maxThreads="6"/>
>> >> >>> >>
>> >> >>> >> <Sender
>> >> >>> >> className="org.apache.catalina.tribes.transport.
>> >> >>> ReplicationTransmitter">
>> >> >>> >> <Transport
>> >> >>> >> className="org.apache.catalina.tribes.transport.nio.
>> >> >>> >> PooledParallelSender"/>
>> >> >>> >> </Sender>
>> >> >>> >> <Interceptor
>> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >> >>> >> TcpFailureDetector"/>
>> >> >>> >> <Interceptor
>> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >> >>> >> MessageDispatch15Interceptor"/>
>> >> >>> >> </Channel>
>> >> >>> >>
>> >> >>> >> <Valve className="org.apache.catalina.ha.tcp.
>> >> >>> ReplicationValve"
>> >> >>> >>
>> >> >>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> >> >>> >> *\.html|.*\.css|.*\.txt"/>
>> >> >>> >> <Valve
>> >> >>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>> >> >>> >>
>> >> >>> >> <Deployer className="org.apache.catalina.ha.deploy.
>> >> >>> >> FarmWarDeployer"
>> >> >>> >> tempDir="D:/imaljava/
>> TomcatNode1/tmp/war-temp/"
>> >> >>> >> deployDir="D:/imaljava/
>> >> TomcatNode1/tmp/war-deploy/"
>> >> >>> >> watchDir="D:/imaljava/
>> >> TomcatNode1/tmp/war-listen/"
>> >> >>> >> watchEnabled="false"/>
>> >> >>> >>
>> >> >>> >> <ClusterListener
>> >> >>> >> className="org.apache.catalina.ha.session.
>> ClusterSessionListener"/>
>> >> >>> >> </Cluster>
>> >> >>> >> <!-- SingleSignOn valve, share authentication between web
>> >> >>> >> applications
>> >> >>> >> Documentation at: /docs/config/valve.html -->
>> >> >>> >> <!--
>> >> >>> >> <Valve className="org.apache.catalina.authenticator.
>> >> >>> SingleSignOn"
>> >> >>> >> />
>> >> >>> >> -->
>> >> >>> >>
>> >> >>> >> <!-- Access log processes all example.
>> >> >>> >> Documentation at: /docs/config/valve.html
>> >> >>> >> Note: The pattern used is equivalent to using
>> >> >>> >> pattern="common" -->
>> >> >>> >> <Valve className="org.apache.catalina.valves.
>> >> AccessLogValve"
>> >> >>> >> directory="logs"
>> >> >>> >> prefix="localhost_access_log" suffix=".txt"
>> >> >>> >> pattern="%h %l %u %t "%r" %s %b" />
>> >> >>> >> <Valve className="org.apache.catalina.valves.
>> >> StuckThreadDetectionValve"
>> >> >>> >> threshold="900" />
>> >> >>> >>
>> >> >>> >> </Host>
>> >> >>> >> </Engine>
>> >> >>> >> </Service>
>> >> >>> >> </Server>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <
>> kfujino@apache.org
>> >> >
>> >> >>> >> wrote:
>> >> >>> >> > Hi
>> >> >>> >> >
>> >> >>> >> > You seems to be using BackupManager.
>> >> >>> >> > What is set for mapSendOptions?
>> >> >>> >> >
>> >> >>> >> > The Cluster module uses the Tomcat JULI logging FW,
>> >> >>> >> > so you can enable logging on the org.apache.catalina.tribes.
>> >> MESSAGES,
>> >> >>> >> you
>> >> >>> >> > can track messages.
>> >> >>> >> >
>> >> >>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
>> >> >>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
>> >> level
>> >> >>> >> logs.
>> >> >>> >> > So It should get logs in a short period of time.
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >> >>> >> >
>> >> >>> >> >> Guys, we are facing the below errors in Tomcat cluster, please
>> >> advise
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >> >>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> heartbeat
>> >> >>> >> >> Unable to send AbstractReplicatedMap.ping message
>> >> >>> >> >> org.apache.catalina.tribes.ChannelException: Operation has
>> >> timed
>> >> >>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
>> >> >>> >> ParallelNioSender.sendMessage(
>> >> >>> >> >> ParallelNioSender.java:108)
>> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
>> >> PooledParallelSender.
>> >> >>> >> >> sendMessage(PooledParallelSender.java:48)
>> >> >>> >> >> at org.apache.catalina.tribes.transport.
>> ReplicationTransmitter.
>> >> >>> >> >> sendMessage(ReplicationTransmitter.java:54)
>> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
>> >> sendMessage(
>> >> >>> >> >> ChannelCoordinator.java:82)
>> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >> >>> sendMessage(
>> >> >>> >> >> ChannelInterceptorBase.java:76)
>> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> >> >>> >> >> MessageDispatchInterceptor.sendMessage(
>> >> MessageDispatchInterceptor.
>> >> >>> >> java:81)
>> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >> >>> sendMessage(
>> >> >>> >> >> ChannelInterceptorBase.java:76)
>> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> >> TcpFailureDetector.
>> >> >>> >> >> sendMessage(TcpFailureDetector.java:93)
>> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >> >>> sendMessage(
>> >> >>> >> >> ChannelInterceptorBase.java:76)
>> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >>> >> >> GroupChannel.java:233)
>> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >>> >> >> GroupChannel.java:186)
>> >> >>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
>> >> >>> RpcChannel.java:99)
>> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> >>> >> >> ping(AbstractReplicatedMap.java:267)
>> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> heartbeat(
>> >> >>> >> >> AbstractReplicatedMap.java:885)
>> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >> >>> >> >> GroupChannel.java:161)
>> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
>> >> >>> HeartbeatThread.run(
>> >> >>> >> >> GroupChannel.java:697)
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >> >>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>> >> Unable
>> >> >>> to
>> >> >>> >> >> process request in NioReceiver
>> >> >>> >> >> java.io.IOException: A non-blocking socket operation could
>> not
>> >> be
>> >> >>> >> >> completed immediately
>> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >> >>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >> >>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
>> >> Source)
>> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >> >>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >> >>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> >> >>> >> >> listen(NioReceiver.java:272)
>> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> >> >>> >> >> NioReceiver.java:425)
>> >> >>> >> >> at java.lang.Thread.run(Unknown Source)
>> >> >>> >> >>
>> >> >>> >> >> Regards,
>> >> >>> >> >> Fady
>> >> >>> >> >>
>> >> >>> >> >> ------------------------------------------------------------
>> >> >>> ---------
>> >> >>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >>> >> >>
>> >> >>> >> >>
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > --
>> >> >>> >> > Keiichi.Fujino
>> >> >>> >>
>> >> >>> >> ------------------------------------------------------------
>> >> ---------
>> >> >>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> Keiichi.Fujino
>> >> >>> >> <us...@tomcat.apache.org>
>> >> >>> >> <us...@tomcat.apache.org>
>> >> >>> >>
>> >> >>>
>> >> >>> ------------------------------------------------------------
>> ---------
>> >> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >>>
>> >> >>> --
>> >> >>> Keiichi.Fujino
>> >> >>> <us...@tomcat.apache.org>
>> >> >>> <us...@tomcat.apache.org>
>> >> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > Keiichi.Fujino
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> --
>> Keiichi.Fujino
>> <us...@tomcat.apache.org>
>
> <us...@tomcat.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
It is realy strange.
Have you failed to restart Context?
When starting up the AbstractReplicatedMap, the followings should be logged.
"Initializing AbstractReplicatedMap with context name:xxx"
"AbstractReplicatedMap[xxx] initialization was completed in xxx ms."
2017-02-09 19:09 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> i can only see this error related to AbstractReplicatedMap
>
>
> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
> java.lang.NullPointerException
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:256)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> GroupChannel.java:166)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> GroupChannel.java:697)
>
> On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino <kf...@apache.org>
> wrote:
> > These logs seem to be the tomcat memory leak detections logs during
> context
> > restart.
> > It is a different issue from this thread's topic.
> >
> > Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> > initialization was completed in xxx ms." in your log file?
> >
> > 2017-02-09 18:49 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >
> >> Keiichi,
> >> once i changed the mapSendOptions and without restarting the below
> >> shown in the tomcat
> >>
> >>
> >> 09-Feb-2017 12:44:12.299 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.300 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
> >> Process}]) but failed to remove it when the web application was
> >> stopped. Threads are going to be renewed over time to try and avoid a
> >> probable memory leak.
> >> 09-Feb-2017 12:44:12.301 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.302 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.303 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_services] created a ThreadLocal with key of
> >> type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> Unable to send AbstractReplicatedMap.ping message
> >> java.lang.NullPointerException
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> ping(AbstractReplicatedMap.java:256)
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> AbstractReplicatedMap.java:885)
> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> GroupChannel.java:166)
> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> >> GroupChannel.java:697)
> >>
> >> 09-Feb-2017 12:44:14.834 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.835 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.838 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.838 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.839 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:14.840 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_services] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.146 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.147 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.147 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.148 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.149 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.150 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.150 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@6433b5ab]) and a value of type
> >> [com.opensymphony.xwork2.ActionContext] (value
> >> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
> >> it when the web application was stopped. Threads are going to be
> >> renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@25f6ef48]) and a value of type
> >> [org.apache.struts2.dispatcher.Dispatcher] (value
> >> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
> >> remove it when the web application was stopped. Threads are going to
> >> be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.151 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.159 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@3adbfb04]) and a value of type
> >> [org.springframework.security.core.context.SecurityContextImpl] (value
> >> [org.springframework.security.core.context.SecurityContextImpl@1e3583d5
> :
> >> Authentication:
> >> org.springframework.security.authentication.
> UsernamePasswordAuthentication
> >> Token@1e3583d5:
> >> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
> >> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
> >> remove it when the web application was stopped. Threads are going to
> >> be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.160 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.160 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.161 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.161 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.163 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.164 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.164 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.165 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.165 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.166 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.529 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
> >> syncClusterOperation}]) but failed to remove it when the web
> >> application was stopped. Threads are going to be renewed over time to
> >> try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.531 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
> >> failed to remove it when the web application was stopped. Threads are
> >> going to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
> >> to remove it when the web application was stopped. Threads are going
> >> to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.541 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
> >> to remove it when the web application was stopped. Threads are going
> >> to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.542 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
> >> to remove it when the web application was stopped. Threads are going
> >> to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.543 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
> >> failed to remove it when the web application was stopped. Threads are
> >> going to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@68650f9d]) and a value of type
> >> [java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
> >> to remove it when the web application was stopped. Threads are going
> >> to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.544 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.545 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> >> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> >> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> >> the web application was stopped. Threads are going to be renewed over
> >> time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.545 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_reporting_services] created a ThreadLocal
> >> with key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@42e441f6]) and a value of type
> >> [org.apache.commons.collections.ReferenceMap] (value
> >> [{WebappClassLoader
> >> context: imal_reporting_services
> >> delegate: false
> >> ----------> Parent Classloader:
> >> java.net.URLClassLoader@2e510e35
> >> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [org.springframework.core.NamedThreadLocal] (value [Actual
> >> transaction active]) and a value of type [java.lang.Boolean] (value
> >> [true]) but failed to remove it when the web application was stopped.
> >> Threads are going to be renewed over time to try and avoid a probable
> >> memory leak.
> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> [Transactional resources]) and a value of type [java.util.HashMap]
> >> (value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@
> >> 6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
> >> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.
> >> springframework.jdbc.datasource.ConnectionHolder@73222923}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.739 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> [Current transaction name]) and a value of type [java.lang.String]
> >> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.
> >> trxMgntListCountsAndRecords])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.744 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> [Current aspect-driven transaction]) and a value of type
> >> [org.springframework.transaction.interceptor.TransactionAspectSupport.
> >> TransactionInfo]
> >> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
> >> to remove it when the web application was stopped. Threads are going
> >> to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.744 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [org.springframework.core.NamedThreadLocal] (value
> >> [Transaction synchronizations]) and a value of type
> >> [java.util.LinkedList] (value
> >> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$
> >> SqlSessionSynchronization@6abd3931]])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.745 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.745 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [java.lang.ThreadLocal] (value
> >> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
> >> [org.apache.ibatis.executor.ErrorContext] (value [
> >> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
> >> ### The error occurred while setting parameters]) but failed to remove
> >> it when the web application was stopped. Threads are going to be
> >> renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.749 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.750 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
> >> but failed to remove it when the web application was stopped. Threads
> >> are going to be renewed over time to try and avoid a probable memory
> >> leak.
> >> 09-Feb-2017 12:44:17.751 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:17.751 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_core_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.197 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.198 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.198 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.199 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.200 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.200 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.201 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.201 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.202 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_tfa_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.364 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.365 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.366 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.367 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.368 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.369 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.370 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.371 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_admin_portal] created a ThreadLocal with key
> >> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> >> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> >> web application was stopped. Threads are going to be renewed over time
> >> to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_swift_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> >> failed to remove it when the web application was stopped. Threads are
> >> going to be renewed over time to try and avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_swift_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
> >> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> >> was stopped. Threads are going to be renewed over time to try and
> >> avoid a probable memory leak.
> >> 09-Feb-2017 12:44:18.746 SEVERE
> >> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> >> org.apache.catalina.loader.WebappClassLoaderBase.
> >> checkThreadLocalMapForLeaks
> >> The web application [imal_swift_services] created a ThreadLocal with
> >> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> >> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> >> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> >> failed to remove it when the web application was stopped. Threads are
> >> going to be renewed over time to try and avoid a probable memory leak.
> >>
> >> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com>
> wrote:
> >> > i will try that but this required tomcat restart so i will be
> >> > scheduling tonight a restart, anyway from where i can enable the
> >> > org.apache.catalina.tribes.MESSAGES i didn't found it in the
> >> > logging.properties file
> >> >
> >> > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org>
> >> wrote:
> >> >> Hi
> >> >>
> >> >> Please try mapSendOptions = 8.
> >> >> This means that it does not receive ACK of the messages.
> >> >> Processing is returned immediately after message transmission is
> >> completed.
> >> >>
> >> >> As another way,
> >> >> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
> >> >> response mail.)
> >> >>
> >> >>
> >> >>
> >> >> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >>
> >> >>> The <Cluster><Manager> is set under the context.xml
> >> >>>
> >> >>> <Manager className="org.apache.catalina.ha.session.BackupManager"
> >> >>> notifyListenersOnReplication="true"
> >> >>> mapSendOptions="6"/>
> >> >>>
> >> >>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kfujino@apache.org
> >
> >> >>> wrote:
> >> >>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >>> >
> >> >>> >> Keiichi,
> >> >>> >> mapSendoption is not configured anyway please check my
> configuration
> >> >>> below
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> > You have not set <Cluster><Manager>.
> >> >>> > If you do not use Cluster Manager template, DeltaManager is used
> by
> >> >>> default.
> >> >>> > However, in the logs, it seems to be using BackupManager.
> >> >>> > Have you set <Context><Manager> in webapps/{your app}
> >> >>> /META-INF/context.xml
> >> >>> > or conf/{Engine}/{Host}/{your app}.xml?
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >> <?xml version='1.0' encoding='utf-8'?>
> >> >>> >> <!--
> >> >>> >> Licensed to the Apache Software Foundation (ASF) under one or
> more
> >> >>> >> contributor license agreements. See the NOTICE file
> distributed
> >> with
> >> >>> >> this work for additional information regarding copyright
> >> ownership.
> >> >>> >> The ASF licenses this file to You under the Apache License,
> >> Version
> >> >>> 2.0
> >> >>> >> (the "License"); you may not use this file except in compliance
> >> with
> >> >>> >> the License. You may obtain a copy of the License at
> >> >>> >>
> >> >>> >> http://www.apache.org/licenses/LICENSE-2.0
> >> >>> >>
> >> >>> >> Unless required by applicable law or agreed to in writing,
> >> software
> >> >>> >> distributed under the License is distributed on an "AS IS"
> BASIS,
> >> >>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >> >>> implied.
> >> >>> >> See the License for the specific language governing permissions
> >> and
> >> >>> >> limitations under the License.
> >> >>> >> -->
> >> >>> >> <!-- Note: A "Server" is not itself a "Container", so you may
> not
> >> >>> >> define subcomponents such as "Valves" at this level.
> >> >>> >> Documentation at /docs/config/server.html
> >> >>> >> -->
> >> >>> >> <Server port="8005" shutdown="SHUTDOWN">
> >> >>> >> <Listener className="org.apache.catalina.startup.
> >> >>> VersionLoggerListener"
> >> >>> >> />
> >> >>> >> <!-- Security listener. Documentation at
> >> /docs/config/listeners.html
> >> >>> >> <Listener className="org.apache.catalina.security.
> >> SecurityListener"
> >> >>> />
> >> >>> >> -->
> >> >>> >> <!--APR library loader. Documentation at /docs/apr.html -->
> >> >>> >> <Listener className="org.apache.catalina.core.
> >> AprLifecycleListener"
> >> >>> >> SSLEngine="on" />
> >> >>> >> <!-- Prevent memory leaks due to use of particular java/javax
> >> APIs-->
> >> >>> >> <Listener className="org.apache.catalina.core.
> >> >>> >> JreMemoryLeakPreventionListener"
> >> >>> >> />
> >> >>> >> <Listener className="org.apache.catalina.mbeans.
> >> >>> >> GlobalResourcesLifecycleListener"
> >> >>> >> />
> >> >>> >> <Listener className="org.apache.catalina.core.
> >> >>> >> ThreadLocalLeakPreventionListener"
> >> >>> >> />
> >> >>> >>
> >> >>> >> <!-- Global JNDI resources
> >> >>> >> Documentation at /docs/jndi-resources-howto.html
> >> >>> >> -->
> >> >>> >> <GlobalNamingResources>
> >> >>> >> <!-- Editable user database that can also be used by
> >> >>> >> UserDatabaseRealm to authenticate users
> >> >>> >> -->
> >> >>> >> <Resource name="UserDatabase" auth="Container"
> >> >>> >> type="org.apache.catalina.UserDatabase"
> >> >>> >> description="User database that can be updated and
> >> saved"
> >> >>> >> factory="org.apache.catalina.users.
> >> >>> >> MemoryUserDatabaseFactory"
> >> >>> >> pathname="conf/tomcat-users.xml" />
> >> >>> >> </GlobalNamingResources>
> >> >>> >>
> >> >>> >> <!-- A "Service" is a collection of one or more "Connectors"
> that
> >> >>> share
> >> >>> >> a single "Container" Note: A "Service" is not itself a
> >> >>> "Container",
> >> >>> >> so you may not define subcomponents such as "Valves" at
> this
> >> >>> level.
> >> >>> >> Documentation at /docs/config/service.html
> >> >>> >> -->
> >> >>> >> <Service name="Catalina">
> >> >>> >>
> >> >>> >> <!--The connectors can use a shared executor, you can define
> one
> >> >>> >> or more named thread pools-->
> >> >>> >> <!--
> >> >>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> >> >>> >> maxThreads="150" minSpareThreads="4"/>
> >> >>> >> -->
> >> >>> >>
> >> >>> >>
> >> >>> >> <!-- A "Connector" represents an endpoint by which requests
> are
> >> >>> >> received
> >> >>> >> and responses are returned. Documentation at :
> >> >>> >> Java HTTP Connector: /docs/config/http.html (blocking &
> >> >>> >> non-blocking)
> >> >>> >> Java AJP Connector: /docs/config/ajp.html
> >> >>> >> APR (HTTP/AJP) Connector: /docs/apr.html
> >> >>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> >> >>> >> -->
> >> >>> >> <Connector port="8080" protocol="HTTP/1.1"
> >> >>> >> connectionTimeout="60000" maxThreads="500"
> >> >>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> >> >>> >> disableUploadTimeout="true" acceptCount="100"
> redirectPort="8443" />
> >> >>> >> <!-- A "Connector" using the shared thread pool-->
> >> >>> >> <!--
> >> >>> >> <Connector executor="tomcatThreadPool"
> >> >>> >> port="8080" protocol="HTTP/1.1"
> >> >>> >> connectionTimeout="60000"
> >> >>> >> redirectPort="8443" />
> >> >>> >> -->
> >> >>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> >> >>> >> This connector uses the NIO implementation that requires
> >> the
> >> >>> JSSE
> >> >>> >> style configuration. When using the APR/native
> >> implementation,
> >> >>> the
> >> >>> >> OpenSSL style configuration is required as described in
> the
> >> >>> >> APR/native
> >> >>> >> documentation -->
> >> >>> >> <!--
> >> >>> >> <Connector port="8443" protocol="org.apache.coyote.
> >> >>> >> http11.Http11NioProtocol"
> >> >>> >> maxThreads="150" SSLEnabled="true" scheme="https"
> >> >>> >> secure="true"
> >> >>> >> clientAuth="false" sslProtocol="TLS" />
> >> >>> >> -->
> >> >>> >>
> >> >>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >> >>> >> <Connector port="8009" protocol="AJP/1.3"
> redirectPort="8443" />
> >> >>> >>
> >> >>> >>
> >> >>> >> <!-- An Engine represents the entry point (within Catalina)
> that
> >> >>> >> processes
> >> >>> >> every request. The Engine implementation for Tomcat
> stand
> >> >>> alone
> >> >>> >> analyzes the HTTP headers included with the request, and
> >> passes
> >> >>> >> them
> >> >>> >> on to the appropriate Host (virtual host).
> >> >>> >> Documentation at /docs/config/engine.html -->
> >> >>> >>
> >> >>> >> <!-- You should set jvmRoute to support load-balancing via
> AJP
> >> ie :
> >> >>> >>
> >> >>> >> -->
> >> >>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
> >> >>> >> jvmRoute="TomcatNode1">
> >> >>> >> <!-- Use the LockOutRealm to prevent attempts to guess user
> >> passwords
> >> >>> >> via a brute-force attack -->
> >> >>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
> >> >>> >> <!-- This Realm uses the UserDatabase configured in the
> >> global
> >> >>> JNDI
> >> >>> >> resources under the key "UserDatabase". Any edits
> >> >>> >> that are performed against this UserDatabase are
> >> >>> immediately
> >> >>> >> available for use by the Realm. -->
> >> >>> >> <Realm className="org.apache.catalina.realm.
> >> UserDatabaseRealm"
> >> >>> >> resourceName="UserDatabase"/>
> >> >>> >> </Realm>
> >> >>> >>
> >> >>> >> <Host name="10.114.43.102" appBase="webapps"
> >> >>> >> unpackWARs="true" autoDeploy="true"
> >> startStopThreads="0">
> >> >>> >> <!--For clustering, please take a look at documentation at:
> >> >>> >> /docs/cluster-howto.html (simple how to)
> >> >>> >> /docs/config/cluster.html (reference documentation) -->
> >> >>> >> <!--
> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
> >> >>> SimpleTcpCluster"/>
> >> >>> >> -->
> >> >>> >>
> >> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
> SimpleTcpCluster"
> >> >>> >> channelSendOptions="4">
> >> >>> >> <Channel className="org.apache.catalina.tribes.group.
> >> >>> >> GroupChannel">
> >> >>> >> <Membership
> >> >>> >> className="org.apache.catalina.tribes.membership.McastService"
> >> >>> >> address="228.0.0.4"
> >> >>> >> port="45560"
> >> >>> >> frequency="500"
> >> >>> >> dropTime="9000"/>
> >> >>> >> <Receiver
> >> >>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> >> >>> >> address="auto"
> >> >>> >> port="4000"
> >> >>> >> autoBind="100"
> >> >>> >> selectorTimeout="5000"
> >> >>> >> maxThreads="6"/>
> >> >>> >>
> >> >>> >> <Sender
> >> >>> >> className="org.apache.catalina.tribes.transport.
> >> >>> ReplicationTransmitter">
> >> >>> >> <Transport
> >> >>> >> className="org.apache.catalina.tribes.transport.nio.
> >> >>> >> PooledParallelSender"/>
> >> >>> >> </Sender>
> >> >>> >> <Interceptor
> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >>> >> TcpFailureDetector"/>
> >> >>> >> <Interceptor
> >> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >> >>> >> MessageDispatch15Interceptor"/>
> >> >>> >> </Channel>
> >> >>> >>
> >> >>> >> <Valve className="org.apache.catalina.ha.tcp.
> >> >>> ReplicationValve"
> >> >>> >>
> >> >>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >> >>> >> *\.html|.*\.css|.*\.txt"/>
> >> >>> >> <Valve
> >> >>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
> >> >>> >>
> >> >>> >> <Deployer className="org.apache.catalina.ha.deploy.
> >> >>> >> FarmWarDeployer"
> >> >>> >> tempDir="D:/imaljava/
> TomcatNode1/tmp/war-temp/"
> >> >>> >> deployDir="D:/imaljava/
> >> TomcatNode1/tmp/war-deploy/"
> >> >>> >> watchDir="D:/imaljava/
> >> TomcatNode1/tmp/war-listen/"
> >> >>> >> watchEnabled="false"/>
> >> >>> >>
> >> >>> >> <ClusterListener
> >> >>> >> className="org.apache.catalina.ha.session.
> ClusterSessionListener"/>
> >> >>> >> </Cluster>
> >> >>> >> <!-- SingleSignOn valve, share authentication between web
> >> >>> >> applications
> >> >>> >> Documentation at: /docs/config/valve.html -->
> >> >>> >> <!--
> >> >>> >> <Valve className="org.apache.catalina.authenticator.
> >> >>> SingleSignOn"
> >> >>> >> />
> >> >>> >> -->
> >> >>> >>
> >> >>> >> <!-- Access log processes all example.
> >> >>> >> Documentation at: /docs/config/valve.html
> >> >>> >> Note: The pattern used is equivalent to using
> >> >>> >> pattern="common" -->
> >> >>> >> <Valve className="org.apache.catalina.valves.
> >> AccessLogValve"
> >> >>> >> directory="logs"
> >> >>> >> prefix="localhost_access_log" suffix=".txt"
> >> >>> >> pattern="%h %l %u %t "%r" %s %b" />
> >> >>> >> <Valve className="org.apache.catalina.valves.
> >> StuckThreadDetectionValve"
> >> >>> >> threshold="900" />
> >> >>> >>
> >> >>> >> </Host>
> >> >>> >> </Engine>
> >> >>> >> </Service>
> >> >>> >> </Server>
> >> >>> >>
> >> >>> >>
> >> >>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <
> kfujino@apache.org
> >> >
> >> >>> >> wrote:
> >> >>> >> > Hi
> >> >>> >> >
> >> >>> >> > You seems to be using BackupManager.
> >> >>> >> > What is set for mapSendOptions?
> >> >>> >> >
> >> >>> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> >>> >> > so you can enable logging on the org.apache.catalina.tribes.
> >> MESSAGES,
> >> >>> >> you
> >> >>> >> > can track messages.
> >> >>> >> >
> >> >>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> >>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
> >> level
> >> >>> >> logs.
> >> >>> >> > So It should get logs in a short period of time.
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >>> >> >
> >> >>> >> >> Guys, we are facing the below errors in Tomcat cluster, please
> >> advise
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> >>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> heartbeat
> >> >>> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >>> >> >> org.apache.catalina.tribes.ChannelException: Operation has
> >> timed
> >> >>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> >> >>> >> ParallelNioSender.sendMessage(
> >> >>> >> >> ParallelNioSender.java:108)
> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> >> PooledParallelSender.
> >> >>> >> >> sendMessage(PooledParallelSender.java:48)
> >> >>> >> >> at org.apache.catalina.tribes.transport.
> ReplicationTransmitter.
> >> >>> >> >> sendMessage(ReplicationTransmitter.java:54)
> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
> >> sendMessage(
> >> >>> >> >> ChannelCoordinator.java:82)
> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> >>> sendMessage(
> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >>> >> >> MessageDispatchInterceptor.sendMessage(
> >> MessageDispatchInterceptor.
> >> >>> >> java:81)
> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> >>> sendMessage(
> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> TcpFailureDetector.
> >> >>> >> >> sendMessage(TcpFailureDetector.java:93)
> >> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >> >>> sendMessage(
> >> >>> >> >> ChannelInterceptorBase.java:76)
> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >>> >> >> GroupChannel.java:233)
> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >>> >> >> GroupChannel.java:186)
> >> >>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> >> >>> RpcChannel.java:99)
> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >>> >> >> ping(AbstractReplicatedMap.java:267)
> >> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> heartbeat(
> >> >>> >> >> AbstractReplicatedMap.java:885)
> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> >>> >> >> GroupChannel.java:161)
> >> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> >> >>> HeartbeatThread.run(
> >> >>> >> >> GroupChannel.java:697)
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> >>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
> >> Unable
> >> >>> to
> >> >>> >> >> process request in NioReceiver
> >> >>> >> >> java.io.IOException: A non-blocking socket operation could
> not
> >> be
> >> >>> >> >> completed immediately
> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> >>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> >>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >> >>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
> >> Source)
> >> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> >>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> >>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> >>> >> >> listen(NioReceiver.java:272)
> >> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >> >>> >> >> NioReceiver.java:425)
> >> >>> >> >> at java.lang.Thread.run(Unknown Source)
> >> >>> >> >>
> >> >>> >> >> Regards,
> >> >>> >> >> Fady
> >> >>> >> >>
> >> >>> >> >> ------------------------------------------------------------
> >> >>> ---------
> >> >>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > --
> >> >>> >> > Keiichi.Fujino
> >> >>> >>
> >> >>> >> ------------------------------------------------------------
> >> ---------
> >> >>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >>> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>> >>
> >> >>> >> --
> >> >>> >> Keiichi.Fujino
> >> >>> >> <us...@tomcat.apache.org>
> >> >>> >> <us...@tomcat.apache.org>
> >> >>> >>
> >> >>>
> >> >>> ------------------------------------------------------------
> ---------
> >> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>>
> >> >>> --
> >> >>> Keiichi.Fujino
> >> >>> <us...@tomcat.apache.org>
> >> >>> <us...@tomcat.apache.org>
> >> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> >
> > --
> > Keiichi.Fujino
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> --
> Keiichi.Fujino
> <us...@tomcat.apache.org>
<us...@tomcat.apache.org>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
i can only see this error related to AbstractReplicatedMap
09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
java.lang.NullPointerException
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
On Thu, Feb 9, 2017 at 12:04 PM, Keiichi Fujino <kf...@apache.org> wrote:
> These logs seem to be the tomcat memory leak detections logs during context
> restart.
> It is a different issue from this thread's topic.
>
> Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
> initialization was completed in xxx ms." in your log file?
>
> 2017-02-09 18:49 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>
>> Keiichi,
>> once i changed the mapSendOptions and without restarting the below
>> shown in the tomcat
>>
>>
>> 09-Feb-2017 12:44:12.299 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.300 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.300 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.301 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
>> Process}]) but failed to remove it when the web application was
>> stopped. Threads are going to be renewed over time to try and avoid a
>> probable memory leak.
>> 09-Feb-2017 12:44:12.301 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.302 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.303 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_services] created a ThreadLocal with key of
>> type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>> java.lang.NullPointerException
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> ping(AbstractReplicatedMap.java:256)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> GroupChannel.java:166)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> GroupChannel.java:697)
>>
>> 09-Feb-2017 12:44:14.834 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.835 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.838 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.838 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.839 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.839 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.839 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:14.840 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_services] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.146 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.147 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.147 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.148 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.148 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.148 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.149 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.149 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.149 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.150 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.150 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.151 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@6433b5ab]) and a value of type
>> [com.opensymphony.xwork2.ActionContext] (value
>> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
>> it when the web application was stopped. Threads are going to be
>> renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.151 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@25f6ef48]) and a value of type
>> [org.apache.struts2.dispatcher.Dispatcher] (value
>> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
>> remove it when the web application was stopped. Threads are going to
>> be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.151 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.159 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@3adbfb04]) and a value of type
>> [org.springframework.security.core.context.SecurityContextImpl] (value
>> [org.springframework.security.core.context.SecurityContextImpl@1e3583d5:
>> Authentication:
>> org.springframework.security.authentication.UsernamePasswordAuthentication
>> Token@1e3583d5:
>> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
>> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
>> remove it when the web application was stopped. Threads are going to
>> be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.160 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.160 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.161 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.161 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.163 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.163 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.163 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.164 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.164 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.165 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.165 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.166 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.529 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
>> syncClusterOperation}]) but failed to remove it when the web
>> application was stopped. Threads are going to be renewed over time to
>> try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.531 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
>> failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.541 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.541 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.541 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
>> to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.541 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.542 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.542 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
>> to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.542 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.542 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.543 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
>> to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.543 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.543 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.543 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
>> failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.544 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.544 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.544 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@68650f9d]) and a value of type
>> [java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
>> to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.544 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.545 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
>> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
>> of type [java.util.HashSet] (value [[]]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.545 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_reporting_services] created a ThreadLocal
>> with key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@42e441f6]) and a value of type
>> [org.apache.commons.collections.ReferenceMap] (value
>> [{WebappClassLoader
>> context: imal_reporting_services
>> delegate: false
>> ----------> Parent Classloader:
>> java.net.URLClassLoader@2e510e35
>> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.739 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.739 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [org.springframework.core.NamedThreadLocal] (value [Actual
>> transaction active]) and a value of type [java.lang.Boolean] (value
>> [true]) but failed to remove it when the web application was stopped.
>> Threads are going to be renewed over time to try and avoid a probable
>> memory leak.
>> 09-Feb-2017 12:44:17.739 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [org.springframework.core.NamedThreadLocal] (value
>> [Transactional resources]) and a value of type [java.util.HashMap]
>> (value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@
>> 6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.
>> springframework.jdbc.datasource.ConnectionHolder@73222923}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.739 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [org.springframework.core.NamedThreadLocal] (value
>> [Current transaction name]) and a value of type [java.lang.String]
>> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.
>> trxMgntListCountsAndRecords])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.744 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [org.springframework.core.NamedThreadLocal] (value
>> [Current aspect-driven transaction]) and a value of type
>> [org.springframework.transaction.interceptor.TransactionAspectSupport.
>> TransactionInfo]
>> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
>> to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.744 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [org.springframework.core.NamedThreadLocal] (value
>> [Transaction synchronizations]) and a value of type
>> [java.util.LinkedList] (value
>> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$
>> SqlSessionSynchronization@6abd3931]])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.745 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.745 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [java.lang.ThreadLocal] (value
>> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
>> [org.apache.ibatis.executor.ErrorContext] (value [
>> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
>> ### The error occurred while setting parameters]) but failed to remove
>> it when the web application was stopped. Threads are going to be
>> renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.749 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.750 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.750 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.750 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
>> but failed to remove it when the web application was stopped. Threads
>> are going to be renewed over time to try and avoid a probable memory
>> leak.
>> 09-Feb-2017 12:44:17.751 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:17.751 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_core_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.197 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.198 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.198 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.199 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.199 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.199 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.200 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.200 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.201 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.201 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.202 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_tfa_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.364 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.365 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.365 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.365 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.365 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.366 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.366 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.366 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.366 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.367 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.367 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.367 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.368 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.368 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.368 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.368 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.369 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.369 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.369 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.370 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.370 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.370 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.370 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.371 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_admin_portal] created a ThreadLocal with key
>> of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
>> type [java.util.HashMap] (value [{}]) but failed to remove it when the
>> web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.746 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_swift_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
>> failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.746 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_swift_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
>> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
>> was stopped. Threads are going to be renewed over time to try and
>> avoid a probable memory leak.
>> 09-Feb-2017 12:44:18.746 SEVERE
>> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
>> org.apache.catalina.loader.WebappClassLoaderBase.
>> checkThreadLocalMapForLeaks
>> The web application [imal_swift_services] created a ThreadLocal with
>> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
>> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
>> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
>> failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>>
>> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com> wrote:
>> > i will try that but this required tomcat restart so i will be
>> > scheduling tonight a restart, anyway from where i can enable the
>> > org.apache.catalina.tribes.MESSAGES i didn't found it in the
>> > logging.properties file
>> >
>> > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org>
>> wrote:
>> >> Hi
>> >>
>> >> Please try mapSendOptions = 8.
>> >> This means that it does not receive ACK of the messages.
>> >> Processing is returned immediately after message transmission is
>> completed.
>> >>
>> >> As another way,
>> >> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
>> >> response mail.)
>> >>
>> >>
>> >>
>> >> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >>
>> >>> The <Cluster><Manager> is set under the context.xml
>> >>>
>> >>> <Manager className="org.apache.catalina.ha.session.BackupManager"
>> >>> notifyListenersOnReplication="true"
>> >>> mapSendOptions="6"/>
>> >>>
>> >>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
>> >>> wrote:
>> >>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >>> >
>> >>> >> Keiichi,
>> >>> >> mapSendoption is not configured anyway please check my configuration
>> >>> below
>> >>> >>
>> >>> >
>> >>> >
>> >>> > You have not set <Cluster><Manager>.
>> >>> > If you do not use Cluster Manager template, DeltaManager is used by
>> >>> default.
>> >>> > However, in the logs, it seems to be using BackupManager.
>> >>> > Have you set <Context><Manager> in webapps/{your app}
>> >>> /META-INF/context.xml
>> >>> > or conf/{Engine}/{Host}/{your app}.xml?
>> >>> >
>> >>> >
>> >>> >
>> >>> >> <?xml version='1.0' encoding='utf-8'?>
>> >>> >> <!--
>> >>> >> Licensed to the Apache Software Foundation (ASF) under one or more
>> >>> >> contributor license agreements. See the NOTICE file distributed
>> with
>> >>> >> this work for additional information regarding copyright
>> ownership.
>> >>> >> The ASF licenses this file to You under the Apache License,
>> Version
>> >>> 2.0
>> >>> >> (the "License"); you may not use this file except in compliance
>> with
>> >>> >> the License. You may obtain a copy of the License at
>> >>> >>
>> >>> >> http://www.apache.org/licenses/LICENSE-2.0
>> >>> >>
>> >>> >> Unless required by applicable law or agreed to in writing,
>> software
>> >>> >> distributed under the License is distributed on an "AS IS" BASIS,
>> >>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> >>> implied.
>> >>> >> See the License for the specific language governing permissions
>> and
>> >>> >> limitations under the License.
>> >>> >> -->
>> >>> >> <!-- Note: A "Server" is not itself a "Container", so you may not
>> >>> >> define subcomponents such as "Valves" at this level.
>> >>> >> Documentation at /docs/config/server.html
>> >>> >> -->
>> >>> >> <Server port="8005" shutdown="SHUTDOWN">
>> >>> >> <Listener className="org.apache.catalina.startup.
>> >>> VersionLoggerListener"
>> >>> >> />
>> >>> >> <!-- Security listener. Documentation at
>> /docs/config/listeners.html
>> >>> >> <Listener className="org.apache.catalina.security.
>> SecurityListener"
>> >>> />
>> >>> >> -->
>> >>> >> <!--APR library loader. Documentation at /docs/apr.html -->
>> >>> >> <Listener className="org.apache.catalina.core.
>> AprLifecycleListener"
>> >>> >> SSLEngine="on" />
>> >>> >> <!-- Prevent memory leaks due to use of particular java/javax
>> APIs-->
>> >>> >> <Listener className="org.apache.catalina.core.
>> >>> >> JreMemoryLeakPreventionListener"
>> >>> >> />
>> >>> >> <Listener className="org.apache.catalina.mbeans.
>> >>> >> GlobalResourcesLifecycleListener"
>> >>> >> />
>> >>> >> <Listener className="org.apache.catalina.core.
>> >>> >> ThreadLocalLeakPreventionListener"
>> >>> >> />
>> >>> >>
>> >>> >> <!-- Global JNDI resources
>> >>> >> Documentation at /docs/jndi-resources-howto.html
>> >>> >> -->
>> >>> >> <GlobalNamingResources>
>> >>> >> <!-- Editable user database that can also be used by
>> >>> >> UserDatabaseRealm to authenticate users
>> >>> >> -->
>> >>> >> <Resource name="UserDatabase" auth="Container"
>> >>> >> type="org.apache.catalina.UserDatabase"
>> >>> >> description="User database that can be updated and
>> saved"
>> >>> >> factory="org.apache.catalina.users.
>> >>> >> MemoryUserDatabaseFactory"
>> >>> >> pathname="conf/tomcat-users.xml" />
>> >>> >> </GlobalNamingResources>
>> >>> >>
>> >>> >> <!-- A "Service" is a collection of one or more "Connectors" that
>> >>> share
>> >>> >> a single "Container" Note: A "Service" is not itself a
>> >>> "Container",
>> >>> >> so you may not define subcomponents such as "Valves" at this
>> >>> level.
>> >>> >> Documentation at /docs/config/service.html
>> >>> >> -->
>> >>> >> <Service name="Catalina">
>> >>> >>
>> >>> >> <!--The connectors can use a shared executor, you can define one
>> >>> >> or more named thread pools-->
>> >>> >> <!--
>> >>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> >>> >> maxThreads="150" minSpareThreads="4"/>
>> >>> >> -->
>> >>> >>
>> >>> >>
>> >>> >> <!-- A "Connector" represents an endpoint by which requests are
>> >>> >> received
>> >>> >> and responses are returned. Documentation at :
>> >>> >> Java HTTP Connector: /docs/config/http.html (blocking &
>> >>> >> non-blocking)
>> >>> >> Java AJP Connector: /docs/config/ajp.html
>> >>> >> APR (HTTP/AJP) Connector: /docs/apr.html
>> >>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> >>> >> -->
>> >>> >> <Connector port="8080" protocol="HTTP/1.1"
>> >>> >> connectionTimeout="60000" maxThreads="500"
>> >>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> >>> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> >>> >> <!-- A "Connector" using the shared thread pool-->
>> >>> >> <!--
>> >>> >> <Connector executor="tomcatThreadPool"
>> >>> >> port="8080" protocol="HTTP/1.1"
>> >>> >> connectionTimeout="60000"
>> >>> >> redirectPort="8443" />
>> >>> >> -->
>> >>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> >>> >> This connector uses the NIO implementation that requires
>> the
>> >>> JSSE
>> >>> >> style configuration. When using the APR/native
>> implementation,
>> >>> the
>> >>> >> OpenSSL style configuration is required as described in the
>> >>> >> APR/native
>> >>> >> documentation -->
>> >>> >> <!--
>> >>> >> <Connector port="8443" protocol="org.apache.coyote.
>> >>> >> http11.Http11NioProtocol"
>> >>> >> maxThreads="150" SSLEnabled="true" scheme="https"
>> >>> >> secure="true"
>> >>> >> clientAuth="false" sslProtocol="TLS" />
>> >>> >> -->
>> >>> >>
>> >>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> >>> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>> >>> >>
>> >>> >>
>> >>> >> <!-- An Engine represents the entry point (within Catalina) that
>> >>> >> processes
>> >>> >> every request. The Engine implementation for Tomcat stand
>> >>> alone
>> >>> >> analyzes the HTTP headers included with the request, and
>> passes
>> >>> >> them
>> >>> >> on to the appropriate Host (virtual host).
>> >>> >> Documentation at /docs/config/engine.html -->
>> >>> >>
>> >>> >> <!-- You should set jvmRoute to support load-balancing via AJP
>> ie :
>> >>> >>
>> >>> >> -->
>> >>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
>> >>> >> jvmRoute="TomcatNode1">
>> >>> >> <!-- Use the LockOutRealm to prevent attempts to guess user
>> passwords
>> >>> >> via a brute-force attack -->
>> >>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> >>> >> <!-- This Realm uses the UserDatabase configured in the
>> global
>> >>> JNDI
>> >>> >> resources under the key "UserDatabase". Any edits
>> >>> >> that are performed against this UserDatabase are
>> >>> immediately
>> >>> >> available for use by the Realm. -->
>> >>> >> <Realm className="org.apache.catalina.realm.
>> UserDatabaseRealm"
>> >>> >> resourceName="UserDatabase"/>
>> >>> >> </Realm>
>> >>> >>
>> >>> >> <Host name="10.114.43.102" appBase="webapps"
>> >>> >> unpackWARs="true" autoDeploy="true"
>> startStopThreads="0">
>> >>> >> <!--For clustering, please take a look at documentation at:
>> >>> >> /docs/cluster-howto.html (simple how to)
>> >>> >> /docs/config/cluster.html (reference documentation) -->
>> >>> >> <!--
>> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
>> >>> SimpleTcpCluster"/>
>> >>> >> -->
>> >>> >>
>> >>> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> >>> >> channelSendOptions="4">
>> >>> >> <Channel className="org.apache.catalina.tribes.group.
>> >>> >> GroupChannel">
>> >>> >> <Membership
>> >>> >> className="org.apache.catalina.tribes.membership.McastService"
>> >>> >> address="228.0.0.4"
>> >>> >> port="45560"
>> >>> >> frequency="500"
>> >>> >> dropTime="9000"/>
>> >>> >> <Receiver
>> >>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> >>> >> address="auto"
>> >>> >> port="4000"
>> >>> >> autoBind="100"
>> >>> >> selectorTimeout="5000"
>> >>> >> maxThreads="6"/>
>> >>> >>
>> >>> >> <Sender
>> >>> >> className="org.apache.catalina.tribes.transport.
>> >>> ReplicationTransmitter">
>> >>> >> <Transport
>> >>> >> className="org.apache.catalina.tribes.transport.nio.
>> >>> >> PooledParallelSender"/>
>> >>> >> </Sender>
>> >>> >> <Interceptor
>> >>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >>> >> TcpFailureDetector"/>
>> >>> >> <Interceptor
>> >>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >>> >> MessageDispatch15Interceptor"/>
>> >>> >> </Channel>
>> >>> >>
>> >>> >> <Valve className="org.apache.catalina.ha.tcp.
>> >>> ReplicationValve"
>> >>> >>
>> >>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> >>> >> *\.html|.*\.css|.*\.txt"/>
>> >>> >> <Valve
>> >>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>> >>> >>
>> >>> >> <Deployer className="org.apache.catalina.ha.deploy.
>> >>> >> FarmWarDeployer"
>> >>> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> >>> >> deployDir="D:/imaljava/
>> TomcatNode1/tmp/war-deploy/"
>> >>> >> watchDir="D:/imaljava/
>> TomcatNode1/tmp/war-listen/"
>> >>> >> watchEnabled="false"/>
>> >>> >>
>> >>> >> <ClusterListener
>> >>> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> >>> >> </Cluster>
>> >>> >> <!-- SingleSignOn valve, share authentication between web
>> >>> >> applications
>> >>> >> Documentation at: /docs/config/valve.html -->
>> >>> >> <!--
>> >>> >> <Valve className="org.apache.catalina.authenticator.
>> >>> SingleSignOn"
>> >>> >> />
>> >>> >> -->
>> >>> >>
>> >>> >> <!-- Access log processes all example.
>> >>> >> Documentation at: /docs/config/valve.html
>> >>> >> Note: The pattern used is equivalent to using
>> >>> >> pattern="common" -->
>> >>> >> <Valve className="org.apache.catalina.valves.
>> AccessLogValve"
>> >>> >> directory="logs"
>> >>> >> prefix="localhost_access_log" suffix=".txt"
>> >>> >> pattern="%h %l %u %t "%r" %s %b" />
>> >>> >> <Valve className="org.apache.catalina.valves.
>> StuckThreadDetectionValve"
>> >>> >> threshold="900" />
>> >>> >>
>> >>> >> </Host>
>> >>> >> </Engine>
>> >>> >> </Service>
>> >>> >> </Server>
>> >>> >>
>> >>> >>
>> >>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kfujino@apache.org
>> >
>> >>> >> wrote:
>> >>> >> > Hi
>> >>> >> >
>> >>> >> > You seems to be using BackupManager.
>> >>> >> > What is set for mapSendOptions?
>> >>> >> >
>> >>> >> > The Cluster module uses the Tomcat JULI logging FW,
>> >>> >> > so you can enable logging on the org.apache.catalina.tribes.
>> MESSAGES,
>> >>> >> you
>> >>> >> > can track messages.
>> >>> >> >
>> >>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
>> >>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
>> level
>> >>> >> logs.
>> >>> >> > So It should get logs in a short period of time.
>> >>> >> >
>> >>> >> >
>> >>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >>> >> >
>> >>> >> >> Guys, we are facing the below errors in Tomcat cluster, please
>> advise
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >>> >> >> Unable to send AbstractReplicatedMap.ping message
>> >>> >> >> org.apache.catalina.tribes.ChannelException: Operation has
>> timed
>> >>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >>> >> >> at org.apache.catalina.tribes.transport.nio.
>> >>> >> ParallelNioSender.sendMessage(
>> >>> >> >> ParallelNioSender.java:108)
>> >>> >> >> at org.apache.catalina.tribes.transport.nio.
>> PooledParallelSender.
>> >>> >> >> sendMessage(PooledParallelSender.java:48)
>> >>> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> >>> >> >> sendMessage(ReplicationTransmitter.java:54)
>> >>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
>> sendMessage(
>> >>> >> >> ChannelCoordinator.java:82)
>> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >>> sendMessage(
>> >>> >> >> ChannelInterceptorBase.java:76)
>> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> >>> >> >> MessageDispatchInterceptor.sendMessage(
>> MessageDispatchInterceptor.
>> >>> >> java:81)
>> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >>> sendMessage(
>> >>> >> >> ChannelInterceptorBase.java:76)
>> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> TcpFailureDetector.
>> >>> >> >> sendMessage(TcpFailureDetector.java:93)
>> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> >>> sendMessage(
>> >>> >> >> ChannelInterceptorBase.java:76)
>> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >>> >> >> GroupChannel.java:233)
>> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >>> >> >> GroupChannel.java:186)
>> >>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
>> >>> RpcChannel.java:99)
>> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >>> >> >> ping(AbstractReplicatedMap.java:267)
>> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> heartbeat(
>> >>> >> >> AbstractReplicatedMap.java:885)
>> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >>> >> >> GroupChannel.java:161)
>> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
>> >>> HeartbeatThread.run(
>> >>> >> >> GroupChannel.java:697)
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
>> Unable
>> >>> to
>> >>> >> >> process request in NioReceiver
>> >>> >> >> java.io.IOException: A non-blocking socket operation could not
>> be
>> >>> >> >> completed immediately
>> >>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
>> Source)
>> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> >>> >> >> listen(NioReceiver.java:272)
>> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> >>> >> >> NioReceiver.java:425)
>> >>> >> >> at java.lang.Thread.run(Unknown Source)
>> >>> >> >>
>> >>> >> >> Regards,
>> >>> >> >> Fady
>> >>> >> >>
>> >>> >> >> ------------------------------------------------------------
>> >>> ---------
>> >>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>> >> >>
>> >>> >> >>
>> >>> >> >
>> >>> >> >
>> >>> >> > --
>> >>> >> > Keiichi.Fujino
>> >>> >>
>> >>> >> ------------------------------------------------------------
>> ---------
>> >>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>> >>
>> >>> >> --
>> >>> >> Keiichi.Fujino
>> >>> >> <us...@tomcat.apache.org>
>> >>> >> <us...@tomcat.apache.org>
>> >>> >>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>
>> >>> --
>> >>> Keiichi.Fujino
>> >>> <us...@tomcat.apache.org>
>> >>> <us...@tomcat.apache.org>
>> >>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> Keiichi.Fujino
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
These logs seem to be the tomcat memory leak detections logs during context
restart.
It is a different issue from this thread's topic.
Anyway, Can you see the log likea "AbstractReplicatedMap[xxx]
initialization was completed in xxx ms." in your log file?
2017-02-09 18:49 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> Keiichi,
> once i changed the mapSendOptions and without restarting the below
> shown in the tomcat
>
>
> 09-Feb-2017 12:44:12.299 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
> Process}]) but failed to remove it when the web application was
> stopped. Threads are going to be renewed over time to try and avoid a
> probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.302 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.303 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
> java.lang.NullPointerException
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:256)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> GroupChannel.java:166)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> GroupChannel.java:697)
>
> 09-Feb-2017 12:44:14.834 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.835 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.838 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.838 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.840 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.146 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.147 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.147 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.150 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.150 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@6433b5ab]) and a value of type
> [com.opensymphony.xwork2.ActionContext] (value
> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
> it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@25f6ef48]) and a value of type
> [org.apache.struts2.dispatcher.Dispatcher] (value
> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
> remove it when the web application was stopped. Threads are going to
> be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.159 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@3adbfb04]) and a value of type
> [org.springframework.security.core.context.SecurityContextImpl] (value
> [org.springframework.security.core.context.SecurityContextImpl@1e3583d5:
> Authentication:
> org.springframework.security.authentication.UsernamePasswordAuthentication
> Token@1e3583d5:
> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
> remove it when the web application was stopped. Threads are going to
> be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.160 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.160 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.161 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.161 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.164 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.164 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.165 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.165 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.166 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.529 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
> syncClusterOperation}]) but failed to remove it when the web
> application was stopped. Threads are going to be renewed over time to
> try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.531 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.545 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.545 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value [Actual
> transaction active]) and a value of type [java.lang.Boolean] (value
> [true]) but failed to remove it when the web application was stopped.
> Threads are going to be renewed over time to try and avoid a probable
> memory leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Transactional resources]) and a value of type [java.util.HashMap]
> (value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@
> 6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.
> springframework.jdbc.datasource.ConnectionHolder@73222923}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Current transaction name]) and a value of type [java.lang.String]
> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.
> trxMgntListCountsAndRecords])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.744 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Current aspect-driven transaction]) and a value of type
> [org.springframework.transaction.interceptor.TransactionAspectSupport.
> TransactionInfo]
> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.744 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Transaction synchronizations]) and a value of type
> [java.util.LinkedList] (value
> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$
> SqlSessionSynchronization@6abd3931]])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.745 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.745 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
> [org.apache.ibatis.executor.ErrorContext] (value [
> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
> ### The error occurred while setting parameters]) but failed to remove
> it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.749 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.751 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:17.751 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:18.197 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.198 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.198 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.200 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.200 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.201 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.201 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.202 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.364 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.371 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.
> checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
>
> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com> wrote:
> > i will try that but this required tomcat restart so i will be
> > scheduling tonight a restart, anyway from where i can enable the
> > org.apache.catalina.tribes.MESSAGES i didn't found it in the
> > logging.properties file
> >
> > On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org>
> wrote:
> >> Hi
> >>
> >> Please try mapSendOptions = 8.
> >> This means that it does not receive ACK of the messages.
> >> Processing is returned immediately after message transmission is
> completed.
> >>
> >> As another way,
> >> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
> >> response mail.)
> >>
> >>
> >>
> >> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >>
> >>> The <Cluster><Manager> is set under the context.xml
> >>>
> >>> <Manager className="org.apache.catalina.ha.session.BackupManager"
> >>> notifyListenersOnReplication="true"
> >>> mapSendOptions="6"/>
> >>>
> >>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
> >>> wrote:
> >>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >>> >
> >>> >> Keiichi,
> >>> >> mapSendoption is not configured anyway please check my configuration
> >>> below
> >>> >>
> >>> >
> >>> >
> >>> > You have not set <Cluster><Manager>.
> >>> > If you do not use Cluster Manager template, DeltaManager is used by
> >>> default.
> >>> > However, in the logs, it seems to be using BackupManager.
> >>> > Have you set <Context><Manager> in webapps/{your app}
> >>> /META-INF/context.xml
> >>> > or conf/{Engine}/{Host}/{your app}.xml?
> >>> >
> >>> >
> >>> >
> >>> >> <?xml version='1.0' encoding='utf-8'?>
> >>> >> <!--
> >>> >> Licensed to the Apache Software Foundation (ASF) under one or more
> >>> >> contributor license agreements. See the NOTICE file distributed
> with
> >>> >> this work for additional information regarding copyright
> ownership.
> >>> >> The ASF licenses this file to You under the Apache License,
> Version
> >>> 2.0
> >>> >> (the "License"); you may not use this file except in compliance
> with
> >>> >> the License. You may obtain a copy of the License at
> >>> >>
> >>> >> http://www.apache.org/licenses/LICENSE-2.0
> >>> >>
> >>> >> Unless required by applicable law or agreed to in writing,
> software
> >>> >> distributed under the License is distributed on an "AS IS" BASIS,
> >>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >>> implied.
> >>> >> See the License for the specific language governing permissions
> and
> >>> >> limitations under the License.
> >>> >> -->
> >>> >> <!-- Note: A "Server" is not itself a "Container", so you may not
> >>> >> define subcomponents such as "Valves" at this level.
> >>> >> Documentation at /docs/config/server.html
> >>> >> -->
> >>> >> <Server port="8005" shutdown="SHUTDOWN">
> >>> >> <Listener className="org.apache.catalina.startup.
> >>> VersionLoggerListener"
> >>> >> />
> >>> >> <!-- Security listener. Documentation at
> /docs/config/listeners.html
> >>> >> <Listener className="org.apache.catalina.security.
> SecurityListener"
> >>> />
> >>> >> -->
> >>> >> <!--APR library loader. Documentation at /docs/apr.html -->
> >>> >> <Listener className="org.apache.catalina.core.
> AprLifecycleListener"
> >>> >> SSLEngine="on" />
> >>> >> <!-- Prevent memory leaks due to use of particular java/javax
> APIs-->
> >>> >> <Listener className="org.apache.catalina.core.
> >>> >> JreMemoryLeakPreventionListener"
> >>> >> />
> >>> >> <Listener className="org.apache.catalina.mbeans.
> >>> >> GlobalResourcesLifecycleListener"
> >>> >> />
> >>> >> <Listener className="org.apache.catalina.core.
> >>> >> ThreadLocalLeakPreventionListener"
> >>> >> />
> >>> >>
> >>> >> <!-- Global JNDI resources
> >>> >> Documentation at /docs/jndi-resources-howto.html
> >>> >> -->
> >>> >> <GlobalNamingResources>
> >>> >> <!-- Editable user database that can also be used by
> >>> >> UserDatabaseRealm to authenticate users
> >>> >> -->
> >>> >> <Resource name="UserDatabase" auth="Container"
> >>> >> type="org.apache.catalina.UserDatabase"
> >>> >> description="User database that can be updated and
> saved"
> >>> >> factory="org.apache.catalina.users.
> >>> >> MemoryUserDatabaseFactory"
> >>> >> pathname="conf/tomcat-users.xml" />
> >>> >> </GlobalNamingResources>
> >>> >>
> >>> >> <!-- A "Service" is a collection of one or more "Connectors" that
> >>> share
> >>> >> a single "Container" Note: A "Service" is not itself a
> >>> "Container",
> >>> >> so you may not define subcomponents such as "Valves" at this
> >>> level.
> >>> >> Documentation at /docs/config/service.html
> >>> >> -->
> >>> >> <Service name="Catalina">
> >>> >>
> >>> >> <!--The connectors can use a shared executor, you can define one
> >>> >> or more named thread pools-->
> >>> >> <!--
> >>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> >>> >> maxThreads="150" minSpareThreads="4"/>
> >>> >> -->
> >>> >>
> >>> >>
> >>> >> <!-- A "Connector" represents an endpoint by which requests are
> >>> >> received
> >>> >> and responses are returned. Documentation at :
> >>> >> Java HTTP Connector: /docs/config/http.html (blocking &
> >>> >> non-blocking)
> >>> >> Java AJP Connector: /docs/config/ajp.html
> >>> >> APR (HTTP/AJP) Connector: /docs/apr.html
> >>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> >>> >> -->
> >>> >> <Connector port="8080" protocol="HTTP/1.1"
> >>> >> connectionTimeout="60000" maxThreads="500"
> >>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> >>> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> >>> >> <!-- A "Connector" using the shared thread pool-->
> >>> >> <!--
> >>> >> <Connector executor="tomcatThreadPool"
> >>> >> port="8080" protocol="HTTP/1.1"
> >>> >> connectionTimeout="60000"
> >>> >> redirectPort="8443" />
> >>> >> -->
> >>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> >>> >> This connector uses the NIO implementation that requires
> the
> >>> JSSE
> >>> >> style configuration. When using the APR/native
> implementation,
> >>> the
> >>> >> OpenSSL style configuration is required as described in the
> >>> >> APR/native
> >>> >> documentation -->
> >>> >> <!--
> >>> >> <Connector port="8443" protocol="org.apache.coyote.
> >>> >> http11.Http11NioProtocol"
> >>> >> maxThreads="150" SSLEnabled="true" scheme="https"
> >>> >> secure="true"
> >>> >> clientAuth="false" sslProtocol="TLS" />
> >>> >> -->
> >>> >>
> >>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >>> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> >>> >>
> >>> >>
> >>> >> <!-- An Engine represents the entry point (within Catalina) that
> >>> >> processes
> >>> >> every request. The Engine implementation for Tomcat stand
> >>> alone
> >>> >> analyzes the HTTP headers included with the request, and
> passes
> >>> >> them
> >>> >> on to the appropriate Host (virtual host).
> >>> >> Documentation at /docs/config/engine.html -->
> >>> >>
> >>> >> <!-- You should set jvmRoute to support load-balancing via AJP
> ie :
> >>> >>
> >>> >> -->
> >>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
> >>> >> jvmRoute="TomcatNode1">
> >>> >> <!-- Use the LockOutRealm to prevent attempts to guess user
> passwords
> >>> >> via a brute-force attack -->
> >>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
> >>> >> <!-- This Realm uses the UserDatabase configured in the
> global
> >>> JNDI
> >>> >> resources under the key "UserDatabase". Any edits
> >>> >> that are performed against this UserDatabase are
> >>> immediately
> >>> >> available for use by the Realm. -->
> >>> >> <Realm className="org.apache.catalina.realm.
> UserDatabaseRealm"
> >>> >> resourceName="UserDatabase"/>
> >>> >> </Realm>
> >>> >>
> >>> >> <Host name="10.114.43.102" appBase="webapps"
> >>> >> unpackWARs="true" autoDeploy="true"
> startStopThreads="0">
> >>> >> <!--For clustering, please take a look at documentation at:
> >>> >> /docs/cluster-howto.html (simple how to)
> >>> >> /docs/config/cluster.html (reference documentation) -->
> >>> >> <!--
> >>> >> <Cluster className="org.apache.catalina.ha.tcp.
> >>> SimpleTcpCluster"/>
> >>> >> -->
> >>> >>
> >>> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> >>> >> channelSendOptions="4">
> >>> >> <Channel className="org.apache.catalina.tribes.group.
> >>> >> GroupChannel">
> >>> >> <Membership
> >>> >> className="org.apache.catalina.tribes.membership.McastService"
> >>> >> address="228.0.0.4"
> >>> >> port="45560"
> >>> >> frequency="500"
> >>> >> dropTime="9000"/>
> >>> >> <Receiver
> >>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> >>> >> address="auto"
> >>> >> port="4000"
> >>> >> autoBind="100"
> >>> >> selectorTimeout="5000"
> >>> >> maxThreads="6"/>
> >>> >>
> >>> >> <Sender
> >>> >> className="org.apache.catalina.tribes.transport.
> >>> ReplicationTransmitter">
> >>> >> <Transport
> >>> >> className="org.apache.catalina.tribes.transport.nio.
> >>> >> PooledParallelSender"/>
> >>> >> </Sender>
> >>> >> <Interceptor
> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >>> >> TcpFailureDetector"/>
> >>> >> <Interceptor
> >>> >> className="org.apache.catalina.tribes.group.interceptors.
> >>> >> MessageDispatch15Interceptor"/>
> >>> >> </Channel>
> >>> >>
> >>> >> <Valve className="org.apache.catalina.ha.tcp.
> >>> ReplicationValve"
> >>> >>
> >>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >>> >> *\.html|.*\.css|.*\.txt"/>
> >>> >> <Valve
> >>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
> >>> >>
> >>> >> <Deployer className="org.apache.catalina.ha.deploy.
> >>> >> FarmWarDeployer"
> >>> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> >>> >> deployDir="D:/imaljava/
> TomcatNode1/tmp/war-deploy/"
> >>> >> watchDir="D:/imaljava/
> TomcatNode1/tmp/war-listen/"
> >>> >> watchEnabled="false"/>
> >>> >>
> >>> >> <ClusterListener
> >>> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> >>> >> </Cluster>
> >>> >> <!-- SingleSignOn valve, share authentication between web
> >>> >> applications
> >>> >> Documentation at: /docs/config/valve.html -->
> >>> >> <!--
> >>> >> <Valve className="org.apache.catalina.authenticator.
> >>> SingleSignOn"
> >>> >> />
> >>> >> -->
> >>> >>
> >>> >> <!-- Access log processes all example.
> >>> >> Documentation at: /docs/config/valve.html
> >>> >> Note: The pattern used is equivalent to using
> >>> >> pattern="common" -->
> >>> >> <Valve className="org.apache.catalina.valves.
> AccessLogValve"
> >>> >> directory="logs"
> >>> >> prefix="localhost_access_log" suffix=".txt"
> >>> >> pattern="%h %l %u %t "%r" %s %b" />
> >>> >> <Valve className="org.apache.catalina.valves.
> StuckThreadDetectionValve"
> >>> >> threshold="900" />
> >>> >>
> >>> >> </Host>
> >>> >> </Engine>
> >>> >> </Service>
> >>> >> </Server>
> >>> >>
> >>> >>
> >>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kfujino@apache.org
> >
> >>> >> wrote:
> >>> >> > Hi
> >>> >> >
> >>> >> > You seems to be using BackupManager.
> >>> >> > What is set for mapSendOptions?
> >>> >> >
> >>> >> > The Cluster module uses the Tomcat JULI logging FW,
> >>> >> > so you can enable logging on the org.apache.catalina.tribes.
> MESSAGES,
> >>> >> you
> >>> >> > can track messages.
> >>> >> >
> >>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace
> level
> >>> >> logs.
> >>> >> > So It should get logs in a short period of time.
> >>> >> >
> >>> >> >
> >>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >>> >> >
> >>> >> >> Guys, we are facing the below errors in Tomcat cluster, please
> advise
> >>> >> >>
> >>> >> >>
> >>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >>> >> >> Unable to send AbstractReplicatedMap.ping message
> >>> >> >> org.apache.catalina.tribes.ChannelException: Operation has
> timed
> >>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> >>> >> ParallelNioSender.sendMessage(
> >>> >> >> ParallelNioSender.java:108)
> >>> >> >> at org.apache.catalina.tribes.transport.nio.
> PooledParallelSender.
> >>> >> >> sendMessage(PooledParallelSender.java:48)
> >>> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> >>> >> >> sendMessage(ReplicationTransmitter.java:54)
> >>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.
> sendMessage(
> >>> >> >> ChannelCoordinator.java:82)
> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >>> sendMessage(
> >>> >> >> ChannelInterceptorBase.java:76)
> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> >>> >> >> MessageDispatchInterceptor.sendMessage(
> MessageDispatchInterceptor.
> >>> >> java:81)
> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >>> sendMessage(
> >>> >> >> ChannelInterceptorBase.java:76)
> >>> >> >> at org.apache.catalina.tribes.group.interceptors.
> TcpFailureDetector.
> >>> >> >> sendMessage(TcpFailureDetector.java:93)
> >>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> >>> sendMessage(
> >>> >> >> ChannelInterceptorBase.java:76)
> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >>> >> >> GroupChannel.java:233)
> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >>> >> >> GroupChannel.java:186)
> >>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> >>> RpcChannel.java:99)
> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >>> >> >> ping(AbstractReplicatedMap.java:267)
> >>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> heartbeat(
> >>> >> >> AbstractReplicatedMap.java:885)
> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >>> >> >> GroupChannel.java:161)
> >>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> >>> HeartbeatThread.run(
> >>> >> >> GroupChannel.java:697)
> >>> >> >>
> >>> >> >>
> >>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen
> Unable
> >>> to
> >>> >> >> process request in NioReceiver
> >>> >> >> java.io.IOException: A non-blocking socket operation could not
> be
> >>> >> >> completed immediately
> >>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
> Source)
> >>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >>> >> >> listen(NioReceiver.java:272)
> >>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >>> >> >> NioReceiver.java:425)
> >>> >> >> at java.lang.Thread.run(Unknown Source)
> >>> >> >>
> >>> >> >> Regards,
> >>> >> >> Fady
> >>> >> >>
> >>> >> >> ------------------------------------------------------------
> >>> ---------
> >>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Keiichi.Fujino
> >>> >>
> >>> >> ------------------------------------------------------------
> ---------
> >>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>> >>
> >>> >> --
> >>> >> Keiichi.Fujino
> >>> >> <us...@tomcat.apache.org>
> >>> >> <us...@tomcat.apache.org>
> >>> >>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>
> >>> --
> >>> Keiichi.Fujino
> >>> <us...@tomcat.apache.org>
> >>> <us...@tomcat.apache.org>
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Keiichi.Fujino
Re: Operation has timed out
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
Let's not mix issues here.
1) your Host entry is
<Host name="10.114.43.102" appBase="webapps"
unpackWARs="true" autoDeploy="true" startStopThreads="0">
that means that Tomcat will automatically reload any application when it notices that it
has been changed.
2) the parameter you changed is in the application's context, so the application has been
changed
3) to reload the application, Tomcat first stops the running application.
That's when it noticed all the potential leak issues below.
But they have always been there, you just never saw them before.
On 09.02.2017 10:49, Fady Haikal wrote:
> Keiichi,
> once i changed the mapSendOptions and without restarting the below
> shown in the tomcat
>
>
> 09-Feb-2017 12:44:12.299 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.300 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
> Process}]) but failed to remove it when the web application was
> stopped. Threads are going to be renewed over time to try and avoid a
> probable memory leak.
> 09-Feb-2017 12:44:12.301 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.302 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.303 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_services] created a ThreadLocal with key of
> type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
> java.lang.NullPointerException
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
>
> 09-Feb-2017 12:44:14.834 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.835 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.838 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.838 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.839 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:14.840 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_services] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.146 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.147 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.147 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.148 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.149 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.150 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.150 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@6433b5ab]) and a value of type
> [com.opensymphony.xwork2.ActionContext] (value
> [com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
> it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@25f6ef48]) and a value of type
> [org.apache.struts2.dispatcher.Dispatcher] (value
> [org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
> remove it when the web application was stopped. Threads are going to
> be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.151 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.159 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@3adbfb04]) and a value of type
> [org.springframework.security.core.context.SecurityContextImpl] (value
> [org.springframework.security.core.context.SecurityContextImpl@1e3583d5:
> Authentication:
> org.springframework.security.authentication.UsernamePasswordAuthenticationToken@1e3583d5:
> Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
> Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
> remove it when the web application was stopped. Threads are going to
> be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.160 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.160 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.161 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.161 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.163 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.164 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.164 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.165 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.165 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.166 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.529 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
> syncClusterOperation}]) but failed to remove it when the web
> application was stopped. Threads are going to be renewed over time to
> try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.531 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret11_1486633106472_296031]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.541 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.542 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.543 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret11_1486633409440_961930]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@68650f9d]) and a value of type
> [java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.544 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.545 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
> [net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
> of type [java.util.HashSet] (value [[]]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.545 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_reporting_services] created a ThreadLocal
> with key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@42e441f6]) and a value of type
> [org.apache.commons.collections.ReferenceMap] (value
> [{WebappClassLoader
> context: imal_reporting_services
> delegate: false
> ----------> Parent Classloader:
> java.net.URLClassLoader@2e510e35
> =com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value [Actual
> transaction active]) and a value of type [java.lang.Boolean] (value
> [true]) but failed to remove it when the web application was stopped.
> Threads are going to be renewed over time to try and avoid a probable
> memory leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Transactional resources]) and a value of type [java.util.HashMap]
> (value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.springframework.jdbc.datasource.ConnectionHolder@73222923}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.739 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Current transaction name]) and a value of type [java.lang.String]
> (value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.trxMgntListCountsAndRecords])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.744 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Current aspect-driven transaction]) and a value of type
> [org.springframework.transaction.interceptor.TransactionAspectSupport.TransactionInfo]
> (value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
> to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.744 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [org.springframework.core.NamedThreadLocal] (value
> [Transaction synchronizations]) and a value of type
> [java.util.LinkedList] (value
> [[com.path.lib.common.base.ibatis3.SqlSessionUtils$SqlSessionSynchronization@6abd3931]])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.745 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.745 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [java.lang.ThreadLocal] (value
> [java.lang.ThreadLocal@2cf5f6b6]) and a value of type
> [org.apache.ibatis.executor.ErrorContext] (value [
> ### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
> ### The error occurred while setting parameters]) but failed to remove
> it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:17.749 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.750 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
> but failed to remove it when the web application was stopped. Threads
> are going to be renewed over time to try and avoid a probable memory
> leak.
> 09-Feb-2017 12:44:17.751 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:17.751 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_core_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:18.197 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.198 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.198 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.199 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.200 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.200 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.201 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.201 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.202 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_tfa_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.364 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.365 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.366 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.367 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.368 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.369 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.370 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.371 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_admin_portal] created a ThreadLocal with key
> of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
> type [java.util.HashMap] (value [{}]) but failed to remove it when the
> web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
> UNKNOWN_PROG_REF}]) but failed to remove it when the web application
> was stopped. Threads are going to be renewed over time to try and
> avoid a probable memory leak.
> 09-Feb-2017 12:44:18.746 SEVERE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [imal_swift_services] created a ThreadLocal with
> key of type [com.path.lib.common.util.ThreadAttributes$1] (value
> [com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
> type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
> failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
>
> On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com> wrote:
>> i will try that but this required tomcat restart so i will be
>> scheduling tonight a restart, anyway from where i can enable the
>> org.apache.catalina.tribes.MESSAGES i didn't found it in the
>> logging.properties file
>>
>> On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org> wrote:
>>> Hi
>>>
>>> Please try mapSendOptions = 8.
>>> This means that it does not receive ACK of the messages.
>>> Processing is returned immediately after message transmission is completed.
>>>
>>> As another way,
>>> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
>>> response mail.)
>>>
>>>
>>>
>>> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>>
>>>> The <Cluster><Manager> is set under the context.xml
>>>>
>>>> <Manager className="org.apache.catalina.ha.session.BackupManager"
>>>> notifyListenersOnReplication="true"
>>>> mapSendOptions="6"/>
>>>>
>>>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
>>>> wrote:
>>>>> 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>>>>
>>>>>> Keiichi,
>>>>>> mapSendoption is not configured anyway please check my configuration
>>>> below
>>>>>>
>>>>>
>>>>>
>>>>> You have not set <Cluster><Manager>.
>>>>> If you do not use Cluster Manager template, DeltaManager is used by
>>>> default.
>>>>> However, in the logs, it seems to be using BackupManager.
>>>>> Have you set <Context><Manager> in webapps/{your app}
>>>> /META-INF/context.xml
>>>>> or conf/{Engine}/{Host}/{your app}.xml?
>>>>>
>>>>>
>>>>>
>>>>>> <?xml version='1.0' encoding='utf-8'?>
>>>>>> <!--
>>>>>> Licensed to the Apache Software Foundation (ASF) under one or more
>>>>>> contributor license agreements. See the NOTICE file distributed with
>>>>>> this work for additional information regarding copyright ownership.
>>>>>> The ASF licenses this file to You under the Apache License, Version
>>>> 2.0
>>>>>> (the "License"); you may not use this file except in compliance with
>>>>>> the License. You may obtain a copy of the License at
>>>>>>
>>>>>> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>
>>>>>> Unless required by applicable law or agreed to in writing, software
>>>>>> distributed under the License is distributed on an "AS IS" BASIS,
>>>>>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>>>> See the License for the specific language governing permissions and
>>>>>> limitations under the License.
>>>>>> -->
>>>>>> <!-- Note: A "Server" is not itself a "Container", so you may not
>>>>>> define subcomponents such as "Valves" at this level.
>>>>>> Documentation at /docs/config/server.html
>>>>>> -->
>>>>>> <Server port="8005" shutdown="SHUTDOWN">
>>>>>> <Listener className="org.apache.catalina.startup.
>>>> VersionLoggerListener"
>>>>>> />
>>>>>> <!-- Security listener. Documentation at /docs/config/listeners.html
>>>>>> <Listener className="org.apache.catalina.security.SecurityListener"
>>>> />
>>>>>> -->
>>>>>> <!--APR library loader. Documentation at /docs/apr.html -->
>>>>>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>>>>>> SSLEngine="on" />
>>>>>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>>>>>> <Listener className="org.apache.catalina.core.
>>>>>> JreMemoryLeakPreventionListener"
>>>>>> />
>>>>>> <Listener className="org.apache.catalina.mbeans.
>>>>>> GlobalResourcesLifecycleListener"
>>>>>> />
>>>>>> <Listener className="org.apache.catalina.core.
>>>>>> ThreadLocalLeakPreventionListener"
>>>>>> />
>>>>>>
>>>>>> <!-- Global JNDI resources
>>>>>> Documentation at /docs/jndi-resources-howto.html
>>>>>> -->
>>>>>> <GlobalNamingResources>
>>>>>> <!-- Editable user database that can also be used by
>>>>>> UserDatabaseRealm to authenticate users
>>>>>> -->
>>>>>> <Resource name="UserDatabase" auth="Container"
>>>>>> type="org.apache.catalina.UserDatabase"
>>>>>> description="User database that can be updated and saved"
>>>>>> factory="org.apache.catalina.users.
>>>>>> MemoryUserDatabaseFactory"
>>>>>> pathname="conf/tomcat-users.xml" />
>>>>>> </GlobalNamingResources>
>>>>>>
>>>>>> <!-- A "Service" is a collection of one or more "Connectors" that
>>>> share
>>>>>> a single "Container" Note: A "Service" is not itself a
>>>> "Container",
>>>>>> so you may not define subcomponents such as "Valves" at this
>>>> level.
>>>>>> Documentation at /docs/config/service.html
>>>>>> -->
>>>>>> <Service name="Catalina">
>>>>>>
>>>>>> <!--The connectors can use a shared executor, you can define one
>>>>>> or more named thread pools-->
>>>>>> <!--
>>>>>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>>>>> maxThreads="150" minSpareThreads="4"/>
>>>>>> -->
>>>>>>
>>>>>>
>>>>>> <!-- A "Connector" represents an endpoint by which requests are
>>>>>> received
>>>>>> and responses are returned. Documentation at :
>>>>>> Java HTTP Connector: /docs/config/http.html (blocking &
>>>>>> non-blocking)
>>>>>> Java AJP Connector: /docs/config/ajp.html
>>>>>> APR (HTTP/AJP) Connector: /docs/apr.html
>>>>>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>>>>> -->
>>>>>> <Connector port="8080" protocol="HTTP/1.1"
>>>>>> connectionTimeout="60000" maxThreads="500"
>>>>>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>>>>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>>>>> <!-- A "Connector" using the shared thread pool-->
>>>>>> <!--
>>>>>> <Connector executor="tomcatThreadPool"
>>>>>> port="8080" protocol="HTTP/1.1"
>>>>>> connectionTimeout="60000"
>>>>>> redirectPort="8443" />
>>>>>> -->
>>>>>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>>>>> This connector uses the NIO implementation that requires the
>>>> JSSE
>>>>>> style configuration. When using the APR/native implementation,
>>>> the
>>>>>> OpenSSL style configuration is required as described in the
>>>>>> APR/native
>>>>>> documentation -->
>>>>>> <!--
>>>>>> <Connector port="8443" protocol="org.apache.coyote.
>>>>>> http11.Http11NioProtocol"
>>>>>> maxThreads="150" SSLEnabled="true" scheme="https"
>>>>>> secure="true"
>>>>>> clientAuth="false" sslProtocol="TLS" />
>>>>>> -->
>>>>>>
>>>>>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>>>>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>>>>>
>>>>>>
>>>>>> <!-- An Engine represents the entry point (within Catalina) that
>>>>>> processes
>>>>>> every request. The Engine implementation for Tomcat stand
>>>> alone
>>>>>> analyzes the HTTP headers included with the request, and passes
>>>>>> them
>>>>>> on to the appropriate Host (virtual host).
>>>>>> Documentation at /docs/config/engine.html -->
>>>>>>
>>>>>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>>>>>
>>>>>> -->
>>>>>> <Engine name="Catalina" defaultHost="10.114.43.102"
>>>>>> jvmRoute="TomcatNode1">
>>>>>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>>>>>> via a brute-force attack -->
>>>>>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>>>>> <!-- This Realm uses the UserDatabase configured in the global
>>>> JNDI
>>>>>> resources under the key "UserDatabase". Any edits
>>>>>> that are performed against this UserDatabase are
>>>> immediately
>>>>>> available for use by the Realm. -->
>>>>>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>>>>> resourceName="UserDatabase"/>
>>>>>> </Realm>
>>>>>>
>>>>>> <Host name="10.114.43.102" appBase="webapps"
>>>>>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>>>>> <!--For clustering, please take a look at documentation at:
>>>>>> /docs/cluster-howto.html (simple how to)
>>>>>> /docs/config/cluster.html (reference documentation) -->
>>>>>> <!--
>>>>>> <Cluster className="org.apache.catalina.ha.tcp.
>>>> SimpleTcpCluster"/>
>>>>>> -->
>>>>>>
>>>>>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>>>>> channelSendOptions="4">
>>>>>> <Channel className="org.apache.catalina.tribes.group.
>>>>>> GroupChannel">
>>>>>> <Membership
>>>>>> className="org.apache.catalina.tribes.membership.McastService"
>>>>>> address="228.0.0.4"
>>>>>> port="45560"
>>>>>> frequency="500"
>>>>>> dropTime="9000"/>
>>>>>> <Receiver
>>>>>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>>>>> address="auto"
>>>>>> port="4000"
>>>>>> autoBind="100"
>>>>>> selectorTimeout="5000"
>>>>>> maxThreads="6"/>
>>>>>>
>>>>>> <Sender
>>>>>> className="org.apache.catalina.tribes.transport.
>>>> ReplicationTransmitter">
>>>>>> <Transport
>>>>>> className="org.apache.catalina.tribes.transport.nio.
>>>>>> PooledParallelSender"/>
>>>>>> </Sender>
>>>>>> <Interceptor
>>>>>> className="org.apache.catalina.tribes.group.interceptors.
>>>>>> TcpFailureDetector"/>
>>>>>> <Interceptor
>>>>>> className="org.apache.catalina.tribes.group.interceptors.
>>>>>> MessageDispatch15Interceptor"/>
>>>>>> </Channel>
>>>>>>
>>>>>> <Valve className="org.apache.catalina.ha.tcp.
>>>> ReplicationValve"
>>>>>>
>>>>>> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>>>>>> *\.html|.*\.css|.*\.txt"/>
>>>>>> <Valve
>>>>>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>>>>>
>>>>>> <Deployer className="org.apache.catalina.ha.deploy.
>>>>>> FarmWarDeployer"
>>>>>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>>>>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>>>>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>>>>> watchEnabled="false"/>
>>>>>>
>>>>>> <ClusterListener
>>>>>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>>>>> </Cluster>
>>>>>> <!-- SingleSignOn valve, share authentication between web
>>>>>> applications
>>>>>> Documentation at: /docs/config/valve.html -->
>>>>>> <!--
>>>>>> <Valve className="org.apache.catalina.authenticator.
>>>> SingleSignOn"
>>>>>> />
>>>>>> -->
>>>>>>
>>>>>> <!-- Access log processes all example.
>>>>>> Documentation at: /docs/config/valve.html
>>>>>> Note: The pattern used is equivalent to using
>>>>>> pattern="common" -->
>>>>>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>>>>> directory="logs"
>>>>>> prefix="localhost_access_log" suffix=".txt"
>>>>>> pattern="%h %l %u %t "%r" %s %b" />
>>>>>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>>>>> threshold="900" />
>>>>>>
>>>>>> </Host>
>>>>>> </Engine>
>>>>>> </Service>
>>>>>> </Server>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
>>>>>> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> You seems to be using BackupManager.
>>>>>>> What is set for mapSendOptions?
>>>>>>>
>>>>>>> The Cluster module uses the Tomcat JULI logging FW,
>>>>>>> so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
>>>>>> you
>>>>>>> can track messages.
>>>>>>>
>>>>>>> Note for using the org.apache.catalina.tribes.MESSAGES:
>>>>>>> The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
>>>>>> logs.
>>>>>>> So It should get logs in a short period of time.
>>>>>>>
>>>>>>>
>>>>>>> 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>>>>>>
>>>>>>>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>>>>>>>
>>>>>>>>
>>>>>>>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>>>>>>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>>>>>>> Unable to send AbstractReplicatedMap.ping message
>>>>>>>> org.apache.catalina.tribes.ChannelException: Operation has timed
>>>>>>>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>>>>>>> at org.apache.catalina.tribes.transport.nio.
>>>>>> ParallelNioSender.sendMessage(
>>>>>>>> ParallelNioSender.java:108)
>>>>>>>> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>>>>>>>> sendMessage(PooledParallelSender.java:48)
>>>>>>>> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>>>>>>>> sendMessage(ReplicationTransmitter.java:54)
>>>>>>>> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>>>>>>>> ChannelCoordinator.java:82)
>>>>>>>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>>> sendMessage(
>>>>>>>> ChannelInterceptorBase.java:76)
>>>>>>>> at org.apache.catalina.tribes.group.interceptors.
>>>>>>>> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
>>>>>> java:81)
>>>>>>>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>>> sendMessage(
>>>>>>>> ChannelInterceptorBase.java:76)
>>>>>>>> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>>>>>>>> sendMessage(TcpFailureDetector.java:93)
>>>>>>>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>>> sendMessage(
>>>>>>>> ChannelInterceptorBase.java:76)
>>>>>>>> at org.apache.catalina.tribes.group.GroupChannel.send(
>>>>>>>> GroupChannel.java:233)
>>>>>>>> at org.apache.catalina.tribes.group.GroupChannel.send(
>>>>>>>> GroupChannel.java:186)
>>>>>>>> at org.apache.catalina.tribes.group.RpcChannel.send(
>>>> RpcChannel.java:99)
>>>>>>>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>>>>>>>> ping(AbstractReplicatedMap.java:267)
>>>>>>>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>>>>>>>> AbstractReplicatedMap.java:885)
>>>>>>>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>>>>>>>> GroupChannel.java:161)
>>>>>>>> at org.apache.catalina.tribes.group.GroupChannel$
>>>> HeartbeatThread.run(
>>>>>>>> GroupChannel.java:697)
>>>>>>>>
>>>>>>>>
>>>>>>>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>>>>>>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>>>> to
>>>>>>>> process request in NioReceiver
>>>>>>>> java.io.IOException: A non-blocking socket operation could not be
>>>>>>>> completed immediately
>>>>>>>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>>>>>>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>>>>>>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>>>>>>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>>>>>>> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>>>>>>>> listen(NioReceiver.java:272)
>>>>>>>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>>>>>>>> NioReceiver.java:425)
>>>>>>>> at java.lang.Thread.run(Unknown Source)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Fady
>>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>> ---------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Keiichi.Fujino
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>> --
>>>>>> Keiichi.Fujino
>>>>>> <us...@tomcat.apache.org>
>>>>>> <us...@tomcat.apache.org>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>> --
>>>> Keiichi.Fujino
>>>> <us...@tomcat.apache.org>
>>>> <us...@tomcat.apache.org>
>>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Keiichi,
once i changed the mapSendOptions and without restarting the below
shown in the tomcat
09-Feb-2017 12:44:12.299 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS A.ALMANA
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.300 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.SHAIKH LOGOUT
Process}]) but failed to remove it when the web application was
stopped. Threads are going to be renewed over time to try and avoid a
probable memory leak.
09-Feb-2017 12:44:12.301 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.302 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.303 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_services] created a ThreadLocal with key of
type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6ca9b927]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=IBIS M.HANAFI
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:12.676 SEVERE [GroupChannel-Heartbeat-1]
org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
Unable to send AbstractReplicatedMap.ping message
java.lang.NullPointerException
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.ping(AbstractReplicatedMap.java:256)
at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(AbstractReplicatedMap.java:885)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.java:166)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(GroupChannel.java:697)
09-Feb-2017 12:44:14.834 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.835 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.838 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.838 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA SA.FADUL RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.839 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.839 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA YAMEN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.839 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN IB01MT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:14.840 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_services] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@2f1fffbd]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=TFA S.HUSSIN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.146 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.147 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.147 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.148 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.148 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.148 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.149 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.149 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.149 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.150 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.150 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.151 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@6433b5ab]) and a value of type
[com.opensymphony.xwork2.ActionContext] (value
[com.opensymphony.xwork2.ActionContext@7edc20f1]) but failed to remove
it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.151 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@25f6ef48]) and a value of type
[org.apache.struts2.dispatcher.Dispatcher] (value
[org.apache.struts2.dispatcher.Dispatcher@76d77370]) but failed to
remove it when the web application was stopped. Threads are going to
be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.151 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.159 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@3adbfb04]) and a value of type
[org.springframework.security.core.context.SecurityContextImpl] (value
[org.springframework.security.core.context.SecurityContextImpl@1e3583d5:
Authentication:
org.springframework.security.authentication.UsernamePasswordAuthenticationToken@1e3583d5:
Principal: H.YOUSIF; Credentials: [PROTECTED]; Authenticated: true;
Details: null; Granted Authorities: ROLE_FULL_ACCESS]) but failed to
remove it when the web application was stopped. Threads are going to
be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.160 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.160 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.161 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.161 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.163 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.163 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.163 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.164 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.164 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.165 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.165 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.166 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@556cf4ad]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.529 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@1f7999df]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=REP RepSCHED null
syncClusterOperation}]) but failed to remove it when the web
application was stopped. Threads are going to be renewed over time to
try and avoid a probable memory leak.
09-Feb-2017 12:44:17.531 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret11_1486633106472_296031]) but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.541 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@197c6a4a}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.541 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.541 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret5_1486633428990_956105]) but failed
to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.541 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@2b6b08b1}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.542 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.542 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret7_1486633316229_329875]) but failed
to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.542 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7b2bc}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.542 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.543 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret13_1486633418550_38295]) but failed
to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.543 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@796754a7}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.543 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.543 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret11_1486633409440_961930]) but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.544 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@6ee6de35}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.544 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.544 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@68650f9d]) and a value of type
[java.lang.Class] (value [class ret7_1486633389730_569340]) but failed
to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.544 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@7706a274}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.545 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [net.sf.jasperreports.engine.fonts.FontUtil$1] (value
[net.sf.jasperreports.engine.fonts.FontUtil$1@37262d58]) and a value
of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. Threads are going to be renewed over
time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.545 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_reporting_services] created a ThreadLocal
with key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@42e441f6]) and a value of type
[org.apache.commons.collections.ReferenceMap] (value
[{WebappClassLoader
context: imal_reporting_services
delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@2e510e35
=com.sun.org.apache.xerces.internal.util.XMLGrammarPoolImpl@5493aab8}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.739 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET AF.MOHMD RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.739 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [org.springframework.core.NamedThreadLocal] (value [Actual
transaction active]) and a value of type [java.lang.Boolean] (value
[true]) but failed to remove it when the web application was stopped.
Threads are going to be renewed over time to try and avoid a probable
memory leak.
09-Feb-2017 12:44:17.739 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [org.springframework.core.NamedThreadLocal] (value
[Transactional resources]) and a value of type [java.util.HashMap]
(value [{org.apache.ibatis.session.defaults.DefaultSqlSessionFactory@6e6c2157=com.path.lib.common.base.ibatis3.SqlSessionHolder@5afa7cf6,
org.apache.tomcat.dbcp.dbcp2.BasicDataSource@4824e5d1=org.springframework.jdbc.datasource.ConnectionHolder@73222923}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.739 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [org.springframework.core.NamedThreadLocal] (value
[Current transaction name]) and a value of type [java.lang.String]
(value [com.path.core.bo.trxmgnt.impl.TrxMgntBOImpl.trxMgntListCountsAndRecords])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.744 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [org.springframework.core.NamedThreadLocal] (value
[Current aspect-driven transaction]) and a value of type
[org.springframework.transaction.interceptor.TransactionAspectSupport.TransactionInfo]
(value [PROPAGATION_REQUIRED,ISOLATION_DEFAULT,-Exception]) but failed
to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.744 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [org.springframework.core.NamedThreadLocal] (value
[Transaction synchronizations]) and a value of type
[java.util.LinkedList] (value
[[com.path.lib.common.base.ibatis3.SqlSessionUtils$SqlSessionSynchronization@6abd3931]])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.745 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF D001MT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.745 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@2cf5f6b6]) and a value of type
[org.apache.ibatis.executor.ErrorContext] (value [
### The error may involve trxMgntMapper.updateCtsTrsTempFiltered-Inline
### The error occurred while setting parameters]) but failed to remove
it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:17.749 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET ES.OSMAN D001MT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.750 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET H.HAMID
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:17.750 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET N.KBSHI RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.750 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET N.RAHMAN RCVALERT}])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
09-Feb-2017 12:44:17.751 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET E.AHMED
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:17.751 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_core_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@61b59f7f]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=RET H.YOUSIF
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:18.197 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.198 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.198 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.199 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.199 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.199 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.200 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.200 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.201 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.201 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.202 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_tfa_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6fd140af]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.364 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.365 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.365 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.365 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.365 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.366 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.366 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.366 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.366 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.367 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.367 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.367 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.368 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.368 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.368 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.368 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.369 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.369 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.369 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.370 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.370 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.370 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.370 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.371 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_admin_portal] created a ThreadLocal with key
of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@6c804389]) and a value of
type [java.util.HashMap] (value [{}]) but failed to remove it when the
web application was stopped. Threads are going to be renewed over time
to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.746 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_swift_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
09-Feb-2017 12:44:18.746 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_swift_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN
UNKNOWN_PROG_REF}]) but failed to remove it when the web application
was stopped. Threads are going to be renewed over time to try and
avoid a probable memory leak.
09-Feb-2017 12:44:18.746 SEVERE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [imal_swift_services] created a ThreadLocal with
key of type [com.path.lib.common.util.ThreadAttributes$1] (value
[com.path.lib.common.util.ThreadAttributes$1@761fbca0]) and a value of
type [java.util.HashMap] (value [{PATH_INFO=SWFT YAMEN Y002}]) but
failed to remove it when the web application was stopped. Threads are
going to be renewed over time to try and avoid a probable memory leak.
On Thu, Feb 9, 2017 at 11:45 AM, Fady Haikal <fa...@gmail.com> wrote:
> i will try that but this required tomcat restart so i will be
> scheduling tonight a restart, anyway from where i can enable the
> org.apache.catalina.tribes.MESSAGES i didn't found it in the
> logging.properties file
>
> On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org> wrote:
>> Hi
>>
>> Please try mapSendOptions = 8.
>> This means that it does not receive ACK of the messages.
>> Processing is returned immediately after message transmission is completed.
>>
>> As another way,
>> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
>> response mail.)
>>
>>
>>
>> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>
>>> The <Cluster><Manager> is set under the context.xml
>>>
>>> <Manager className="org.apache.catalina.ha.session.BackupManager"
>>> notifyListenersOnReplication="true"
>>> mapSendOptions="6"/>
>>>
>>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
>>> wrote:
>>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>> >
>>> >> Keiichi,
>>> >> mapSendoption is not configured anyway please check my configuration
>>> below
>>> >>
>>> >
>>> >
>>> > You have not set <Cluster><Manager>.
>>> > If you do not use Cluster Manager template, DeltaManager is used by
>>> default.
>>> > However, in the logs, it seems to be using BackupManager.
>>> > Have you set <Context><Manager> in webapps/{your app}
>>> /META-INF/context.xml
>>> > or conf/{Engine}/{Host}/{your app}.xml?
>>> >
>>> >
>>> >
>>> >> <?xml version='1.0' encoding='utf-8'?>
>>> >> <!--
>>> >> Licensed to the Apache Software Foundation (ASF) under one or more
>>> >> contributor license agreements. See the NOTICE file distributed with
>>> >> this work for additional information regarding copyright ownership.
>>> >> The ASF licenses this file to You under the Apache License, Version
>>> 2.0
>>> >> (the "License"); you may not use this file except in compliance with
>>> >> the License. You may obtain a copy of the License at
>>> >>
>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>> >>
>>> >> Unless required by applicable law or agreed to in writing, software
>>> >> distributed under the License is distributed on an "AS IS" BASIS,
>>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>> implied.
>>> >> See the License for the specific language governing permissions and
>>> >> limitations under the License.
>>> >> -->
>>> >> <!-- Note: A "Server" is not itself a "Container", so you may not
>>> >> define subcomponents such as "Valves" at this level.
>>> >> Documentation at /docs/config/server.html
>>> >> -->
>>> >> <Server port="8005" shutdown="SHUTDOWN">
>>> >> <Listener className="org.apache.catalina.startup.
>>> VersionLoggerListener"
>>> >> />
>>> >> <!-- Security listener. Documentation at /docs/config/listeners.html
>>> >> <Listener className="org.apache.catalina.security.SecurityListener"
>>> />
>>> >> -->
>>> >> <!--APR library loader. Documentation at /docs/apr.html -->
>>> >> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>>> >> SSLEngine="on" />
>>> >> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>>> >> <Listener className="org.apache.catalina.core.
>>> >> JreMemoryLeakPreventionListener"
>>> >> />
>>> >> <Listener className="org.apache.catalina.mbeans.
>>> >> GlobalResourcesLifecycleListener"
>>> >> />
>>> >> <Listener className="org.apache.catalina.core.
>>> >> ThreadLocalLeakPreventionListener"
>>> >> />
>>> >>
>>> >> <!-- Global JNDI resources
>>> >> Documentation at /docs/jndi-resources-howto.html
>>> >> -->
>>> >> <GlobalNamingResources>
>>> >> <!-- Editable user database that can also be used by
>>> >> UserDatabaseRealm to authenticate users
>>> >> -->
>>> >> <Resource name="UserDatabase" auth="Container"
>>> >> type="org.apache.catalina.UserDatabase"
>>> >> description="User database that can be updated and saved"
>>> >> factory="org.apache.catalina.users.
>>> >> MemoryUserDatabaseFactory"
>>> >> pathname="conf/tomcat-users.xml" />
>>> >> </GlobalNamingResources>
>>> >>
>>> >> <!-- A "Service" is a collection of one or more "Connectors" that
>>> share
>>> >> a single "Container" Note: A "Service" is not itself a
>>> "Container",
>>> >> so you may not define subcomponents such as "Valves" at this
>>> level.
>>> >> Documentation at /docs/config/service.html
>>> >> -->
>>> >> <Service name="Catalina">
>>> >>
>>> >> <!--The connectors can use a shared executor, you can define one
>>> >> or more named thread pools-->
>>> >> <!--
>>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>>> >> maxThreads="150" minSpareThreads="4"/>
>>> >> -->
>>> >>
>>> >>
>>> >> <!-- A "Connector" represents an endpoint by which requests are
>>> >> received
>>> >> and responses are returned. Documentation at :
>>> >> Java HTTP Connector: /docs/config/http.html (blocking &
>>> >> non-blocking)
>>> >> Java AJP Connector: /docs/config/ajp.html
>>> >> APR (HTTP/AJP) Connector: /docs/apr.html
>>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>>> >> -->
>>> >> <Connector port="8080" protocol="HTTP/1.1"
>>> >> connectionTimeout="60000" maxThreads="500"
>>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>>> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>>> >> <!-- A "Connector" using the shared thread pool-->
>>> >> <!--
>>> >> <Connector executor="tomcatThreadPool"
>>> >> port="8080" protocol="HTTP/1.1"
>>> >> connectionTimeout="60000"
>>> >> redirectPort="8443" />
>>> >> -->
>>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>>> >> This connector uses the NIO implementation that requires the
>>> JSSE
>>> >> style configuration. When using the APR/native implementation,
>>> the
>>> >> OpenSSL style configuration is required as described in the
>>> >> APR/native
>>> >> documentation -->
>>> >> <!--
>>> >> <Connector port="8443" protocol="org.apache.coyote.
>>> >> http11.Http11NioProtocol"
>>> >> maxThreads="150" SSLEnabled="true" scheme="https"
>>> >> secure="true"
>>> >> clientAuth="false" sslProtocol="TLS" />
>>> >> -->
>>> >>
>>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
>>> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>> >>
>>> >>
>>> >> <!-- An Engine represents the entry point (within Catalina) that
>>> >> processes
>>> >> every request. The Engine implementation for Tomcat stand
>>> alone
>>> >> analyzes the HTTP headers included with the request, and passes
>>> >> them
>>> >> on to the appropriate Host (virtual host).
>>> >> Documentation at /docs/config/engine.html -->
>>> >>
>>> >> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>> >>
>>> >> -->
>>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
>>> >> jvmRoute="TomcatNode1">
>>> >> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>>> >> via a brute-force attack -->
>>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
>>> >> <!-- This Realm uses the UserDatabase configured in the global
>>> JNDI
>>> >> resources under the key "UserDatabase". Any edits
>>> >> that are performed against this UserDatabase are
>>> immediately
>>> >> available for use by the Realm. -->
>>> >> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>>> >> resourceName="UserDatabase"/>
>>> >> </Realm>
>>> >>
>>> >> <Host name="10.114.43.102" appBase="webapps"
>>> >> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>>> >> <!--For clustering, please take a look at documentation at:
>>> >> /docs/cluster-howto.html (simple how to)
>>> >> /docs/config/cluster.html (reference documentation) -->
>>> >> <!--
>>> >> <Cluster className="org.apache.catalina.ha.tcp.
>>> SimpleTcpCluster"/>
>>> >> -->
>>> >>
>>> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>>> >> channelSendOptions="4">
>>> >> <Channel className="org.apache.catalina.tribes.group.
>>> >> GroupChannel">
>>> >> <Membership
>>> >> className="org.apache.catalina.tribes.membership.McastService"
>>> >> address="228.0.0.4"
>>> >> port="45560"
>>> >> frequency="500"
>>> >> dropTime="9000"/>
>>> >> <Receiver
>>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>>> >> address="auto"
>>> >> port="4000"
>>> >> autoBind="100"
>>> >> selectorTimeout="5000"
>>> >> maxThreads="6"/>
>>> >>
>>> >> <Sender
>>> >> className="org.apache.catalina.tribes.transport.
>>> ReplicationTransmitter">
>>> >> <Transport
>>> >> className="org.apache.catalina.tribes.transport.nio.
>>> >> PooledParallelSender"/>
>>> >> </Sender>
>>> >> <Interceptor
>>> >> className="org.apache.catalina.tribes.group.interceptors.
>>> >> TcpFailureDetector"/>
>>> >> <Interceptor
>>> >> className="org.apache.catalina.tribes.group.interceptors.
>>> >> MessageDispatch15Interceptor"/>
>>> >> </Channel>
>>> >>
>>> >> <Valve className="org.apache.catalina.ha.tcp.
>>> ReplicationValve"
>>> >>
>>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>>> >> *\.html|.*\.css|.*\.txt"/>
>>> >> <Valve
>>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>> >>
>>> >> <Deployer className="org.apache.catalina.ha.deploy.
>>> >> FarmWarDeployer"
>>> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>>> >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>>> >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>>> >> watchEnabled="false"/>
>>> >>
>>> >> <ClusterListener
>>> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>>> >> </Cluster>
>>> >> <!-- SingleSignOn valve, share authentication between web
>>> >> applications
>>> >> Documentation at: /docs/config/valve.html -->
>>> >> <!--
>>> >> <Valve className="org.apache.catalina.authenticator.
>>> SingleSignOn"
>>> >> />
>>> >> -->
>>> >>
>>> >> <!-- Access log processes all example.
>>> >> Documentation at: /docs/config/valve.html
>>> >> Note: The pattern used is equivalent to using
>>> >> pattern="common" -->
>>> >> <Valve className="org.apache.catalina.valves.AccessLogValve"
>>> >> directory="logs"
>>> >> prefix="localhost_access_log" suffix=".txt"
>>> >> pattern="%h %l %u %t "%r" %s %b" />
>>> >> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>>> >> threshold="900" />
>>> >>
>>> >> </Host>
>>> >> </Engine>
>>> >> </Service>
>>> >> </Server>
>>> >>
>>> >>
>>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
>>> >> wrote:
>>> >> > Hi
>>> >> >
>>> >> > You seems to be using BackupManager.
>>> >> > What is set for mapSendOptions?
>>> >> >
>>> >> > The Cluster module uses the Tomcat JULI logging FW,
>>> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
>>> >> you
>>> >> > can track messages.
>>> >> >
>>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
>>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
>>> >> logs.
>>> >> > So It should get logs in a short period of time.
>>> >> >
>>> >> >
>>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>>> >> >
>>> >> >> Guys, we are facing the below errors in Tomcat cluster, please advise
>>> >> >>
>>> >> >>
>>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>>> >> >> Unable to send AbstractReplicatedMap.ping message
>>> >> >> org.apache.catalina.tribes.ChannelException: Operation has timed
>>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>>> >> >> at org.apache.catalina.tribes.transport.nio.
>>> >> ParallelNioSender.sendMessage(
>>> >> >> ParallelNioSender.java:108)
>>> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>>> >> >> sendMessage(PooledParallelSender.java:48)
>>> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>>> >> >> sendMessage(ReplicationTransmitter.java:54)
>>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>>> >> >> ChannelCoordinator.java:82)
>>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>> sendMessage(
>>> >> >> ChannelInterceptorBase.java:76)
>>> >> >> at org.apache.catalina.tribes.group.interceptors.
>>> >> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
>>> >> java:81)
>>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>> sendMessage(
>>> >> >> ChannelInterceptorBase.java:76)
>>> >> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>>> >> >> sendMessage(TcpFailureDetector.java:93)
>>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>>> sendMessage(
>>> >> >> ChannelInterceptorBase.java:76)
>>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>>> >> >> GroupChannel.java:233)
>>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>>> >> >> GroupChannel.java:186)
>>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
>>> RpcChannel.java:99)
>>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>>> >> >> ping(AbstractReplicatedMap.java:267)
>>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>>> >> >> AbstractReplicatedMap.java:885)
>>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>>> >> >> GroupChannel.java:161)
>>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
>>> HeartbeatThread.run(
>>> >> >> GroupChannel.java:697)
>>> >> >>
>>> >> >>
>>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>>> to
>>> >> >> process request in NioReceiver
>>> >> >> java.io.IOException: A non-blocking socket operation could not be
>>> >> >> completed immediately
>>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>>> >> >> listen(NioReceiver.java:272)
>>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>>> >> >> NioReceiver.java:425)
>>> >> >> at java.lang.Thread.run(Unknown Source)
>>> >> >>
>>> >> >> Regards,
>>> >> >> Fady
>>> >> >>
>>> >> >> ------------------------------------------------------------
>>> ---------
>>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Keiichi.Fujino
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>>> >>
>>> >> --
>>> >> Keiichi.Fujino
>>> >> <us...@tomcat.apache.org>
>>> >> <us...@tomcat.apache.org>
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>> --
>>> Keiichi.Fujino
>>> <us...@tomcat.apache.org>
>>> <us...@tomcat.apache.org>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
i will try that but this required tomcat restart so i will be
scheduling tonight a restart, anyway from where i can enable the
org.apache.catalina.tribes.MESSAGES i didn't found it in the
logging.properties file
On Thu, Feb 9, 2017 at 11:40 AM, Keiichi Fujino <kf...@apache.org> wrote:
> Hi
>
> Please try mapSendOptions = 8.
> This means that it does not receive ACK of the messages.
> Processing is returned immediately after message transmission is completed.
>
> As another way,
> Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
> response mail.)
>
>
>
> 2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>
>> The <Cluster><Manager> is set under the context.xml
>>
>> <Manager className="org.apache.catalina.ha.session.BackupManager"
>> notifyListenersOnReplication="true"
>> mapSendOptions="6"/>
>>
>> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
>> wrote:
>> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >
>> >> Keiichi,
>> >> mapSendoption is not configured anyway please check my configuration
>> below
>> >>
>> >
>> >
>> > You have not set <Cluster><Manager>.
>> > If you do not use Cluster Manager template, DeltaManager is used by
>> default.
>> > However, in the logs, it seems to be using BackupManager.
>> > Have you set <Context><Manager> in webapps/{your app}
>> /META-INF/context.xml
>> > or conf/{Engine}/{Host}/{your app}.xml?
>> >
>> >
>> >
>> >> <?xml version='1.0' encoding='utf-8'?>
>> >> <!--
>> >> Licensed to the Apache Software Foundation (ASF) under one or more
>> >> contributor license agreements. See the NOTICE file distributed with
>> >> this work for additional information regarding copyright ownership.
>> >> The ASF licenses this file to You under the Apache License, Version
>> 2.0
>> >> (the "License"); you may not use this file except in compliance with
>> >> the License. You may obtain a copy of the License at
>> >>
>> >> http://www.apache.org/licenses/LICENSE-2.0
>> >>
>> >> Unless required by applicable law or agreed to in writing, software
>> >> distributed under the License is distributed on an "AS IS" BASIS,
>> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>> implied.
>> >> See the License for the specific language governing permissions and
>> >> limitations under the License.
>> >> -->
>> >> <!-- Note: A "Server" is not itself a "Container", so you may not
>> >> define subcomponents such as "Valves" at this level.
>> >> Documentation at /docs/config/server.html
>> >> -->
>> >> <Server port="8005" shutdown="SHUTDOWN">
>> >> <Listener className="org.apache.catalina.startup.
>> VersionLoggerListener"
>> >> />
>> >> <!-- Security listener. Documentation at /docs/config/listeners.html
>> >> <Listener className="org.apache.catalina.security.SecurityListener"
>> />
>> >> -->
>> >> <!--APR library loader. Documentation at /docs/apr.html -->
>> >> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>> >> SSLEngine="on" />
>> >> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>> >> <Listener className="org.apache.catalina.core.
>> >> JreMemoryLeakPreventionListener"
>> >> />
>> >> <Listener className="org.apache.catalina.mbeans.
>> >> GlobalResourcesLifecycleListener"
>> >> />
>> >> <Listener className="org.apache.catalina.core.
>> >> ThreadLocalLeakPreventionListener"
>> >> />
>> >>
>> >> <!-- Global JNDI resources
>> >> Documentation at /docs/jndi-resources-howto.html
>> >> -->
>> >> <GlobalNamingResources>
>> >> <!-- Editable user database that can also be used by
>> >> UserDatabaseRealm to authenticate users
>> >> -->
>> >> <Resource name="UserDatabase" auth="Container"
>> >> type="org.apache.catalina.UserDatabase"
>> >> description="User database that can be updated and saved"
>> >> factory="org.apache.catalina.users.
>> >> MemoryUserDatabaseFactory"
>> >> pathname="conf/tomcat-users.xml" />
>> >> </GlobalNamingResources>
>> >>
>> >> <!-- A "Service" is a collection of one or more "Connectors" that
>> share
>> >> a single "Container" Note: A "Service" is not itself a
>> "Container",
>> >> so you may not define subcomponents such as "Valves" at this
>> level.
>> >> Documentation at /docs/config/service.html
>> >> -->
>> >> <Service name="Catalina">
>> >>
>> >> <!--The connectors can use a shared executor, you can define one
>> >> or more named thread pools-->
>> >> <!--
>> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> >> maxThreads="150" minSpareThreads="4"/>
>> >> -->
>> >>
>> >>
>> >> <!-- A "Connector" represents an endpoint by which requests are
>> >> received
>> >> and responses are returned. Documentation at :
>> >> Java HTTP Connector: /docs/config/http.html (blocking &
>> >> non-blocking)
>> >> Java AJP Connector: /docs/config/ajp.html
>> >> APR (HTTP/AJP) Connector: /docs/apr.html
>> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> >> -->
>> >> <Connector port="8080" protocol="HTTP/1.1"
>> >> connectionTimeout="60000" maxThreads="500"
>> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> >> <!-- A "Connector" using the shared thread pool-->
>> >> <!--
>> >> <Connector executor="tomcatThreadPool"
>> >> port="8080" protocol="HTTP/1.1"
>> >> connectionTimeout="60000"
>> >> redirectPort="8443" />
>> >> -->
>> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> >> This connector uses the NIO implementation that requires the
>> JSSE
>> >> style configuration. When using the APR/native implementation,
>> the
>> >> OpenSSL style configuration is required as described in the
>> >> APR/native
>> >> documentation -->
>> >> <!--
>> >> <Connector port="8443" protocol="org.apache.coyote.
>> >> http11.Http11NioProtocol"
>> >> maxThreads="150" SSLEnabled="true" scheme="https"
>> >> secure="true"
>> >> clientAuth="false" sslProtocol="TLS" />
>> >> -->
>> >>
>> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>> >>
>> >>
>> >> <!-- An Engine represents the entry point (within Catalina) that
>> >> processes
>> >> every request. The Engine implementation for Tomcat stand
>> alone
>> >> analyzes the HTTP headers included with the request, and passes
>> >> them
>> >> on to the appropriate Host (virtual host).
>> >> Documentation at /docs/config/engine.html -->
>> >>
>> >> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>> >>
>> >> -->
>> >> <Engine name="Catalina" defaultHost="10.114.43.102"
>> >> jvmRoute="TomcatNode1">
>> >> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>> >> via a brute-force attack -->
>> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> >> <!-- This Realm uses the UserDatabase configured in the global
>> JNDI
>> >> resources under the key "UserDatabase". Any edits
>> >> that are performed against this UserDatabase are
>> immediately
>> >> available for use by the Realm. -->
>> >> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> >> resourceName="UserDatabase"/>
>> >> </Realm>
>> >>
>> >> <Host name="10.114.43.102" appBase="webapps"
>> >> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>> >> <!--For clustering, please take a look at documentation at:
>> >> /docs/cluster-howto.html (simple how to)
>> >> /docs/config/cluster.html (reference documentation) -->
>> >> <!--
>> >> <Cluster className="org.apache.catalina.ha.tcp.
>> SimpleTcpCluster"/>
>> >> -->
>> >>
>> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> >> channelSendOptions="4">
>> >> <Channel className="org.apache.catalina.tribes.group.
>> >> GroupChannel">
>> >> <Membership
>> >> className="org.apache.catalina.tribes.membership.McastService"
>> >> address="228.0.0.4"
>> >> port="45560"
>> >> frequency="500"
>> >> dropTime="9000"/>
>> >> <Receiver
>> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> >> address="auto"
>> >> port="4000"
>> >> autoBind="100"
>> >> selectorTimeout="5000"
>> >> maxThreads="6"/>
>> >>
>> >> <Sender
>> >> className="org.apache.catalina.tribes.transport.
>> ReplicationTransmitter">
>> >> <Transport
>> >> className="org.apache.catalina.tribes.transport.nio.
>> >> PooledParallelSender"/>
>> >> </Sender>
>> >> <Interceptor
>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >> TcpFailureDetector"/>
>> >> <Interceptor
>> >> className="org.apache.catalina.tribes.group.interceptors.
>> >> MessageDispatch15Interceptor"/>
>> >> </Channel>
>> >>
>> >> <Valve className="org.apache.catalina.ha.tcp.
>> ReplicationValve"
>> >>
>> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> >> *\.html|.*\.css|.*\.txt"/>
>> >> <Valve
>> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>> >>
>> >> <Deployer className="org.apache.catalina.ha.deploy.
>> >> FarmWarDeployer"
>> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> >> watchEnabled="false"/>
>> >>
>> >> <ClusterListener
>> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> >> </Cluster>
>> >> <!-- SingleSignOn valve, share authentication between web
>> >> applications
>> >> Documentation at: /docs/config/valve.html -->
>> >> <!--
>> >> <Valve className="org.apache.catalina.authenticator.
>> SingleSignOn"
>> >> />
>> >> -->
>> >>
>> >> <!-- Access log processes all example.
>> >> Documentation at: /docs/config/valve.html
>> >> Note: The pattern used is equivalent to using
>> >> pattern="common" -->
>> >> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> >> directory="logs"
>> >> prefix="localhost_access_log" suffix=".txt"
>> >> pattern="%h %l %u %t "%r" %s %b" />
>> >> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>> >> threshold="900" />
>> >>
>> >> </Host>
>> >> </Engine>
>> >> </Service>
>> >> </Server>
>> >>
>> >>
>> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
>> >> wrote:
>> >> > Hi
>> >> >
>> >> > You seems to be using BackupManager.
>> >> > What is set for mapSendOptions?
>> >> >
>> >> > The Cluster module uses the Tomcat JULI logging FW,
>> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
>> >> you
>> >> > can track messages.
>> >> >
>> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
>> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
>> >> logs.
>> >> > So It should get logs in a short period of time.
>> >> >
>> >> >
>> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >> >
>> >> >> Guys, we are facing the below errors in Tomcat cluster, please advise
>> >> >>
>> >> >>
>> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >> >> Unable to send AbstractReplicatedMap.ping message
>> >> >> org.apache.catalina.tribes.ChannelException: Operation has timed
>> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >> >> at org.apache.catalina.tribes.transport.nio.
>> >> ParallelNioSender.sendMessage(
>> >> >> ParallelNioSender.java:108)
>> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>> >> >> sendMessage(PooledParallelSender.java:48)
>> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> >> >> sendMessage(ReplicationTransmitter.java:54)
>> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>> >> >> ChannelCoordinator.java:82)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.interceptors.
>> >> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
>> >> java:81)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> >> >> sendMessage(TcpFailureDetector.java:93)
>> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
>> sendMessage(
>> >> >> ChannelInterceptorBase.java:76)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >> GroupChannel.java:233)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> >> GroupChannel.java:186)
>> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
>> RpcChannel.java:99)
>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> >> ping(AbstractReplicatedMap.java:267)
>> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> >> >> AbstractReplicatedMap.java:885)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >> >> GroupChannel.java:161)
>> >> >> at org.apache.catalina.tribes.group.GroupChannel$
>> HeartbeatThread.run(
>> >> >> GroupChannel.java:697)
>> >> >>
>> >> >>
>> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
>> to
>> >> >> process request in NioReceiver
>> >> >> java.io.IOException: A non-blocking socket operation could not be
>> >> >> completed immediately
>> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> >> >> listen(NioReceiver.java:272)
>> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> >> >> NioReceiver.java:425)
>> >> >> at java.lang.Thread.run(Unknown Source)
>> >> >>
>> >> >> Regards,
>> >> >> Fady
>> >> >>
>> >> >> ------------------------------------------------------------
>> ---------
>> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> > --
>> >> > Keiichi.Fujino
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >> --
>> >> Keiichi.Fujino
>> >> <us...@tomcat.apache.org>
>> >> <us...@tomcat.apache.org>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> --
>> Keiichi.Fujino
>> <us...@tomcat.apache.org>
>> <us...@tomcat.apache.org>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
Hi
Please try mapSendOptions = 8.
This means that it does not receive ACK of the messages.
Processing is returned immediately after message transmission is completed.
As another way,
Please try the org.apache.catalina.tribes.MESSAGES. (see previous my
response mail.)
2017-02-09 18:23 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> The <Cluster><Manager> is set under the context.xml
>
> <Manager className="org.apache.catalina.ha.session.BackupManager"
> notifyListenersOnReplication="true"
> mapSendOptions="6"/>
>
> On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org>
> wrote:
> > 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >
> >> Keiichi,
> >> mapSendoption is not configured anyway please check my configuration
> below
> >>
> >
> >
> > You have not set <Cluster><Manager>.
> > If you do not use Cluster Manager template, DeltaManager is used by
> default.
> > However, in the logs, it seems to be using BackupManager.
> > Have you set <Context><Manager> in webapps/{your app}
> /META-INF/context.xml
> > or conf/{Engine}/{Host}/{your app}.xml?
> >
> >
> >
> >> <?xml version='1.0' encoding='utf-8'?>
> >> <!--
> >> Licensed to the Apache Software Foundation (ASF) under one or more
> >> contributor license agreements. See the NOTICE file distributed with
> >> this work for additional information regarding copyright ownership.
> >> The ASF licenses this file to You under the Apache License, Version
> 2.0
> >> (the "License"); you may not use this file except in compliance with
> >> the License. You may obtain a copy of the License at
> >>
> >> http://www.apache.org/licenses/LICENSE-2.0
> >>
> >> Unless required by applicable law or agreed to in writing, software
> >> distributed under the License is distributed on an "AS IS" BASIS,
> >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> >> See the License for the specific language governing permissions and
> >> limitations under the License.
> >> -->
> >> <!-- Note: A "Server" is not itself a "Container", so you may not
> >> define subcomponents such as "Valves" at this level.
> >> Documentation at /docs/config/server.html
> >> -->
> >> <Server port="8005" shutdown="SHUTDOWN">
> >> <Listener className="org.apache.catalina.startup.
> VersionLoggerListener"
> >> />
> >> <!-- Security listener. Documentation at /docs/config/listeners.html
> >> <Listener className="org.apache.catalina.security.SecurityListener"
> />
> >> -->
> >> <!--APR library loader. Documentation at /docs/apr.html -->
> >> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> >> SSLEngine="on" />
> >> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
> >> <Listener className="org.apache.catalina.core.
> >> JreMemoryLeakPreventionListener"
> >> />
> >> <Listener className="org.apache.catalina.mbeans.
> >> GlobalResourcesLifecycleListener"
> >> />
> >> <Listener className="org.apache.catalina.core.
> >> ThreadLocalLeakPreventionListener"
> >> />
> >>
> >> <!-- Global JNDI resources
> >> Documentation at /docs/jndi-resources-howto.html
> >> -->
> >> <GlobalNamingResources>
> >> <!-- Editable user database that can also be used by
> >> UserDatabaseRealm to authenticate users
> >> -->
> >> <Resource name="UserDatabase" auth="Container"
> >> type="org.apache.catalina.UserDatabase"
> >> description="User database that can be updated and saved"
> >> factory="org.apache.catalina.users.
> >> MemoryUserDatabaseFactory"
> >> pathname="conf/tomcat-users.xml" />
> >> </GlobalNamingResources>
> >>
> >> <!-- A "Service" is a collection of one or more "Connectors" that
> share
> >> a single "Container" Note: A "Service" is not itself a
> "Container",
> >> so you may not define subcomponents such as "Valves" at this
> level.
> >> Documentation at /docs/config/service.html
> >> -->
> >> <Service name="Catalina">
> >>
> >> <!--The connectors can use a shared executor, you can define one
> >> or more named thread pools-->
> >> <!--
> >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> >> maxThreads="150" minSpareThreads="4"/>
> >> -->
> >>
> >>
> >> <!-- A "Connector" represents an endpoint by which requests are
> >> received
> >> and responses are returned. Documentation at :
> >> Java HTTP Connector: /docs/config/http.html (blocking &
> >> non-blocking)
> >> Java AJP Connector: /docs/config/ajp.html
> >> APR (HTTP/AJP) Connector: /docs/apr.html
> >> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> >> -->
> >> <Connector port="8080" protocol="HTTP/1.1"
> >> connectionTimeout="60000" maxThreads="500"
> >> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> >> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> >> <!-- A "Connector" using the shared thread pool-->
> >> <!--
> >> <Connector executor="tomcatThreadPool"
> >> port="8080" protocol="HTTP/1.1"
> >> connectionTimeout="60000"
> >> redirectPort="8443" />
> >> -->
> >> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> >> This connector uses the NIO implementation that requires the
> JSSE
> >> style configuration. When using the APR/native implementation,
> the
> >> OpenSSL style configuration is required as described in the
> >> APR/native
> >> documentation -->
> >> <!--
> >> <Connector port="8443" protocol="org.apache.coyote.
> >> http11.Http11NioProtocol"
> >> maxThreads="150" SSLEnabled="true" scheme="https"
> >> secure="true"
> >> clientAuth="false" sslProtocol="TLS" />
> >> -->
> >>
> >> <!-- Define an AJP 1.3 Connector on port 8009 -->
> >> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> >>
> >>
> >> <!-- An Engine represents the entry point (within Catalina) that
> >> processes
> >> every request. The Engine implementation for Tomcat stand
> alone
> >> analyzes the HTTP headers included with the request, and passes
> >> them
> >> on to the appropriate Host (virtual host).
> >> Documentation at /docs/config/engine.html -->
> >>
> >> <!-- You should set jvmRoute to support load-balancing via AJP ie :
> >>
> >> -->
> >> <Engine name="Catalina" defaultHost="10.114.43.102"
> >> jvmRoute="TomcatNode1">
> >> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
> >> via a brute-force attack -->
> >> <Realm className="org.apache.catalina.realm.LockOutRealm">
> >> <!-- This Realm uses the UserDatabase configured in the global
> JNDI
> >> resources under the key "UserDatabase". Any edits
> >> that are performed against this UserDatabase are
> immediately
> >> available for use by the Realm. -->
> >> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> >> resourceName="UserDatabase"/>
> >> </Realm>
> >>
> >> <Host name="10.114.43.102" appBase="webapps"
> >> unpackWARs="true" autoDeploy="true" startStopThreads="0">
> >> <!--For clustering, please take a look at documentation at:
> >> /docs/cluster-howto.html (simple how to)
> >> /docs/config/cluster.html (reference documentation) -->
> >> <!--
> >> <Cluster className="org.apache.catalina.ha.tcp.
> SimpleTcpCluster"/>
> >> -->
> >>
> >> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> >> channelSendOptions="4">
> >> <Channel className="org.apache.catalina.tribes.group.
> >> GroupChannel">
> >> <Membership
> >> className="org.apache.catalina.tribes.membership.McastService"
> >> address="228.0.0.4"
> >> port="45560"
> >> frequency="500"
> >> dropTime="9000"/>
> >> <Receiver
> >> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> >> address="auto"
> >> port="4000"
> >> autoBind="100"
> >> selectorTimeout="5000"
> >> maxThreads="6"/>
> >>
> >> <Sender
> >> className="org.apache.catalina.tribes.transport.
> ReplicationTransmitter">
> >> <Transport
> >> className="org.apache.catalina.tribes.transport.nio.
> >> PooledParallelSender"/>
> >> </Sender>
> >> <Interceptor
> >> className="org.apache.catalina.tribes.group.interceptors.
> >> TcpFailureDetector"/>
> >> <Interceptor
> >> className="org.apache.catalina.tribes.group.interceptors.
> >> MessageDispatch15Interceptor"/>
> >> </Channel>
> >>
> >> <Valve className="org.apache.catalina.ha.tcp.
> ReplicationValve"
> >>
> >> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> >> *\.html|.*\.css|.*\.txt"/>
> >> <Valve
> >> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
> >>
> >> <Deployer className="org.apache.catalina.ha.deploy.
> >> FarmWarDeployer"
> >> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> >> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
> >> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
> >> watchEnabled="false"/>
> >>
> >> <ClusterListener
> >> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> >> </Cluster>
> >> <!-- SingleSignOn valve, share authentication between web
> >> applications
> >> Documentation at: /docs/config/valve.html -->
> >> <!--
> >> <Valve className="org.apache.catalina.authenticator.
> SingleSignOn"
> >> />
> >> -->
> >>
> >> <!-- Access log processes all example.
> >> Documentation at: /docs/config/valve.html
> >> Note: The pattern used is equivalent to using
> >> pattern="common" -->
> >> <Valve className="org.apache.catalina.valves.AccessLogValve"
> >> directory="logs"
> >> prefix="localhost_access_log" suffix=".txt"
> >> pattern="%h %l %u %t "%r" %s %b" />
> >> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >> threshold="900" />
> >>
> >> </Host>
> >> </Engine>
> >> </Service>
> >> </Server>
> >>
> >>
> >> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
> >> wrote:
> >> > Hi
> >> >
> >> > You seems to be using BackupManager.
> >> > What is set for mapSendOptions?
> >> >
> >> > The Cluster module uses the Tomcat JULI logging FW,
> >> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
> >> you
> >> > can track messages.
> >> >
> >> > Note for using the org.apache.catalina.tribes.MESSAGES:
> >> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
> >> logs.
> >> > So It should get logs in a short period of time.
> >> >
> >> >
> >> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >> >
> >> >> Guys, we are facing the below errors in Tomcat cluster, please advise
> >> >>
> >> >>
> >> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> >> Unable to send AbstractReplicatedMap.ping message
> >> >> org.apache.catalina.tribes.ChannelException: Operation has timed
> >> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >> >> at org.apache.catalina.tribes.transport.nio.
> >> ParallelNioSender.sendMessage(
> >> >> ParallelNioSender.java:108)
> >> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> >> >> sendMessage(PooledParallelSender.java:48)
> >> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> >> >> sendMessage(ReplicationTransmitter.java:54)
> >> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
> >> >> ChannelCoordinator.java:82)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.interceptors.
> >> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
> >> java:81)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> >> >> sendMessage(TcpFailureDetector.java:93)
> >> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.
> sendMessage(
> >> >> ChannelInterceptorBase.java:76)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> GroupChannel.java:233)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> >> GroupChannel.java:186)
> >> >> at org.apache.catalina.tribes.group.RpcChannel.send(
> RpcChannel.java:99)
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> >> ping(AbstractReplicatedMap.java:267)
> >> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> >> AbstractReplicatedMap.java:885)
> >> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> >> GroupChannel.java:161)
> >> >> at org.apache.catalina.tribes.group.GroupChannel$
> HeartbeatThread.run(
> >> >> GroupChannel.java:697)
> >> >>
> >> >>
> >> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable
> to
> >> >> process request in NioReceiver
> >> >> java.io.IOException: A non-blocking socket operation could not be
> >> >> completed immediately
> >> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> >> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> >> listen(NioReceiver.java:272)
> >> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >> >> NioReceiver.java:425)
> >> >> at java.lang.Thread.run(Unknown Source)
> >> >>
> >> >> Regards,
> >> >> Fady
> >> >>
> >> >> ------------------------------------------------------------
> ---------
> >> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Keiichi.Fujino
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >> --
> >> Keiichi.Fujino
> >> <us...@tomcat.apache.org>
> >> <us...@tomcat.apache.org>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> --
> Keiichi.Fujino
> <us...@tomcat.apache.org>
> <us...@tomcat.apache.org>
>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
The <Cluster><Manager> is set under the context.xml
<Manager className="org.apache.catalina.ha.session.BackupManager"
notifyListenersOnReplication="true"
mapSendOptions="6"/>
On Thu, Feb 9, 2017 at 11:21 AM, Keiichi Fujino <kf...@apache.org> wrote:
> 2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>
>> Keiichi,
>> mapSendoption is not configured anyway please check my configuration below
>>
>
>
> You have not set <Cluster><Manager>.
> If you do not use Cluster Manager template, DeltaManager is used by default.
> However, in the logs, it seems to be using BackupManager.
> Have you set <Context><Manager> in webapps/{your app} /META-INF/context.xml
> or conf/{Engine}/{Host}/{your app}.xml?
>
>
>
>> <?xml version='1.0' encoding='utf-8'?>
>> <!--
>> Licensed to the Apache Software Foundation (ASF) under one or more
>> contributor license agreements. See the NOTICE file distributed with
>> this work for additional information regarding copyright ownership.
>> The ASF licenses this file to You under the Apache License, Version 2.0
>> (the "License"); you may not use this file except in compliance with
>> the License. You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in writing, software
>> distributed under the License is distributed on an "AS IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> See the License for the specific language governing permissions and
>> limitations under the License.
>> -->
>> <!-- Note: A "Server" is not itself a "Container", so you may not
>> define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/server.html
>> -->
>> <Server port="8005" shutdown="SHUTDOWN">
>> <Listener className="org.apache.catalina.startup.VersionLoggerListener"
>> />
>> <!-- Security listener. Documentation at /docs/config/listeners.html
>> <Listener className="org.apache.catalina.security.SecurityListener" />
>> -->
>> <!--APR library loader. Documentation at /docs/apr.html -->
>> <Listener className="org.apache.catalina.core.AprLifecycleListener"
>> SSLEngine="on" />
>> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
>> <Listener className="org.apache.catalina.core.
>> JreMemoryLeakPreventionListener"
>> />
>> <Listener className="org.apache.catalina.mbeans.
>> GlobalResourcesLifecycleListener"
>> />
>> <Listener className="org.apache.catalina.core.
>> ThreadLocalLeakPreventionListener"
>> />
>>
>> <!-- Global JNDI resources
>> Documentation at /docs/jndi-resources-howto.html
>> -->
>> <GlobalNamingResources>
>> <!-- Editable user database that can also be used by
>> UserDatabaseRealm to authenticate users
>> -->
>> <Resource name="UserDatabase" auth="Container"
>> type="org.apache.catalina.UserDatabase"
>> description="User database that can be updated and saved"
>> factory="org.apache.catalina.users.
>> MemoryUserDatabaseFactory"
>> pathname="conf/tomcat-users.xml" />
>> </GlobalNamingResources>
>>
>> <!-- A "Service" is a collection of one or more "Connectors" that share
>> a single "Container" Note: A "Service" is not itself a "Container",
>> so you may not define subcomponents such as "Valves" at this level.
>> Documentation at /docs/config/service.html
>> -->
>> <Service name="Catalina">
>>
>> <!--The connectors can use a shared executor, you can define one
>> or more named thread pools-->
>> <!--
>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
>> maxThreads="150" minSpareThreads="4"/>
>> -->
>>
>>
>> <!-- A "Connector" represents an endpoint by which requests are
>> received
>> and responses are returned. Documentation at :
>> Java HTTP Connector: /docs/config/http.html (blocking &
>> non-blocking)
>> Java AJP Connector: /docs/config/ajp.html
>> APR (HTTP/AJP) Connector: /docs/apr.html
>> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
>> -->
>> <Connector port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000" maxThreads="500"
>> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
>> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
>> <!-- A "Connector" using the shared thread pool-->
>> <!--
>> <Connector executor="tomcatThreadPool"
>> port="8080" protocol="HTTP/1.1"
>> connectionTimeout="60000"
>> redirectPort="8443" />
>> -->
>> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
>> This connector uses the NIO implementation that requires the JSSE
>> style configuration. When using the APR/native implementation, the
>> OpenSSL style configuration is required as described in the
>> APR/native
>> documentation -->
>> <!--
>> <Connector port="8443" protocol="org.apache.coyote.
>> http11.Http11NioProtocol"
>> maxThreads="150" SSLEnabled="true" scheme="https"
>> secure="true"
>> clientAuth="false" sslProtocol="TLS" />
>> -->
>>
>> <!-- Define an AJP 1.3 Connector on port 8009 -->
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>>
>>
>> <!-- An Engine represents the entry point (within Catalina) that
>> processes
>> every request. The Engine implementation for Tomcat stand alone
>> analyzes the HTTP headers included with the request, and passes
>> them
>> on to the appropriate Host (virtual host).
>> Documentation at /docs/config/engine.html -->
>>
>> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>>
>> -->
>> <Engine name="Catalina" defaultHost="10.114.43.102"
>> jvmRoute="TomcatNode1">
>> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
>> via a brute-force attack -->
>> <Realm className="org.apache.catalina.realm.LockOutRealm">
>> <!-- This Realm uses the UserDatabase configured in the global JNDI
>> resources under the key "UserDatabase". Any edits
>> that are performed against this UserDatabase are immediately
>> available for use by the Realm. -->
>> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
>> resourceName="UserDatabase"/>
>> </Realm>
>>
>> <Host name="10.114.43.102" appBase="webapps"
>> unpackWARs="true" autoDeploy="true" startStopThreads="0">
>> <!--For clustering, please take a look at documentation at:
>> /docs/cluster-howto.html (simple how to)
>> /docs/config/cluster.html (reference documentation) -->
>> <!--
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
>> -->
>>
>> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
>> channelSendOptions="4">
>> <Channel className="org.apache.catalina.tribes.group.
>> GroupChannel">
>> <Membership
>> className="org.apache.catalina.tribes.membership.McastService"
>> address="228.0.0.4"
>> port="45560"
>> frequency="500"
>> dropTime="9000"/>
>> <Receiver
>> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
>> address="auto"
>> port="4000"
>> autoBind="100"
>> selectorTimeout="5000"
>> maxThreads="6"/>
>>
>> <Sender
>> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
>> <Transport
>> className="org.apache.catalina.tribes.transport.nio.
>> PooledParallelSender"/>
>> </Sender>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.
>> TcpFailureDetector"/>
>> <Interceptor
>> className="org.apache.catalina.tribes.group.interceptors.
>> MessageDispatch15Interceptor"/>
>> </Channel>
>>
>> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>>
>> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
>> *\.html|.*\.css|.*\.txt"/>
>> <Valve
>> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>>
>> <Deployer className="org.apache.catalina.ha.deploy.
>> FarmWarDeployer"
>> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
>> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
>> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
>> watchEnabled="false"/>
>>
>> <ClusterListener
>> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
>> </Cluster>
>> <!-- SingleSignOn valve, share authentication between web
>> applications
>> Documentation at: /docs/config/valve.html -->
>> <!--
>> <Valve className="org.apache.catalina.authenticator.SingleSignOn"
>> />
>> -->
>>
>> <!-- Access log processes all example.
>> Documentation at: /docs/config/valve.html
>> Note: The pattern used is equivalent to using
>> pattern="common" -->
>> <Valve className="org.apache.catalina.valves.AccessLogValve"
>> directory="logs"
>> prefix="localhost_access_log" suffix=".txt"
>> pattern="%h %l %u %t "%r" %s %b" />
>> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
>> threshold="900" />
>>
>> </Host>
>> </Engine>
>> </Service>
>> </Server>
>>
>>
>> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
>> wrote:
>> > Hi
>> >
>> > You seems to be using BackupManager.
>> > What is set for mapSendOptions?
>> >
>> > The Cluster module uses the Tomcat JULI logging FW,
>> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
>> you
>> > can track messages.
>> >
>> > Note for using the org.apache.catalina.tribes.MESSAGES:
>> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
>> logs.
>> > So It should get logs in a short period of time.
>> >
>> >
>> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>> >
>> >> Guys, we are facing the below errors in Tomcat cluster, please advise
>> >>
>> >>
>> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> >> Unable to send AbstractReplicatedMap.ping message
>> >> org.apache.catalina.tribes.ChannelException: Operation has timed
>> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> >> at org.apache.catalina.tribes.transport.nio.
>> ParallelNioSender.sendMessage(
>> >> ParallelNioSender.java:108)
>> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>> >> sendMessage(PooledParallelSender.java:48)
>> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> >> sendMessage(ReplicationTransmitter.java:54)
>> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>> >> ChannelCoordinator.java:82)
>> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> >> ChannelInterceptorBase.java:76)
>> >> at org.apache.catalina.tribes.group.interceptors.
>> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
>> java:81)
>> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> >> ChannelInterceptorBase.java:76)
>> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> >> sendMessage(TcpFailureDetector.java:93)
>> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> >> ChannelInterceptorBase.java:76)
>> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> GroupChannel.java:233)
>> >> at org.apache.catalina.tribes.group.GroupChannel.send(
>> >> GroupChannel.java:186)
>> >> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> >> ping(AbstractReplicatedMap.java:267)
>> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> >> AbstractReplicatedMap.java:885)
>> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> >> GroupChannel.java:161)
>> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> >> GroupChannel.java:697)
>> >>
>> >>
>> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> >> process request in NioReceiver
>> >> java.io.IOException: A non-blocking socket operation could not be
>> >> completed immediately
>> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> >> listen(NioReceiver.java:272)
>> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> >> NioReceiver.java:425)
>> >> at java.lang.Thread.run(Unknown Source)
>> >>
>> >> Regards,
>> >> Fady
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > Keiichi.Fujino
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> --
>> Keiichi.Fujino
>> <us...@tomcat.apache.org>
>> <us...@tomcat.apache.org>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
2017-02-09 18:10 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> Keiichi,
> mapSendoption is not configured anyway please check my configuration below
>
You have not set <Cluster><Manager>.
If you do not use Cluster Manager template, DeltaManager is used by default.
However, in the logs, it seems to be using BackupManager.
Have you set <Context><Manager> in webapps/{your app} /META-INF/context.xml
or conf/{Engine}/{Host}/{your app}.xml?
> <?xml version='1.0' encoding='utf-8'?>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
> <!-- Note: A "Server" is not itself a "Container", so you may not
> define subcomponents such as "Valves" at this level.
> Documentation at /docs/config/server.html
> -->
> <Server port="8005" shutdown="SHUTDOWN">
> <Listener className="org.apache.catalina.startup.VersionLoggerListener"
> />
> <!-- Security listener. Documentation at /docs/config/listeners.html
> <Listener className="org.apache.catalina.security.SecurityListener" />
> -->
> <!--APR library loader. Documentation at /docs/apr.html -->
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
> <!-- Prevent memory leaks due to use of particular java/javax APIs-->
> <Listener className="org.apache.catalina.core.
> JreMemoryLeakPreventionListener"
> />
> <Listener className="org.apache.catalina.mbeans.
> GlobalResourcesLifecycleListener"
> />
> <Listener className="org.apache.catalina.core.
> ThreadLocalLeakPreventionListener"
> />
>
> <!-- Global JNDI resources
> Documentation at /docs/jndi-resources-howto.html
> -->
> <GlobalNamingResources>
> <!-- Editable user database that can also be used by
> UserDatabaseRealm to authenticate users
> -->
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.
> MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <!-- A "Service" is a collection of one or more "Connectors" that share
> a single "Container" Note: A "Service" is not itself a "Container",
> so you may not define subcomponents such as "Valves" at this level.
> Documentation at /docs/config/service.html
> -->
> <Service name="Catalina">
>
> <!--The connectors can use a shared executor, you can define one
> or more named thread pools-->
> <!--
> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
> maxThreads="150" minSpareThreads="4"/>
> -->
>
>
> <!-- A "Connector" represents an endpoint by which requests are
> received
> and responses are returned. Documentation at :
> Java HTTP Connector: /docs/config/http.html (blocking &
> non-blocking)
> Java AJP Connector: /docs/config/ajp.html
> APR (HTTP/AJP) Connector: /docs/apr.html
> Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
> -->
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="60000" maxThreads="500"
> minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
> disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
> <!-- A "Connector" using the shared thread pool-->
> <!--
> <Connector executor="tomcatThreadPool"
> port="8080" protocol="HTTP/1.1"
> connectionTimeout="60000"
> redirectPort="8443" />
> -->
> <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
> This connector uses the NIO implementation that requires the JSSE
> style configuration. When using the APR/native implementation, the
> OpenSSL style configuration is required as described in the
> APR/native
> documentation -->
> <!--
> <Connector port="8443" protocol="org.apache.coyote.
> http11.Http11NioProtocol"
> maxThreads="150" SSLEnabled="true" scheme="https"
> secure="true"
> clientAuth="false" sslProtocol="TLS" />
> -->
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
>
> <!-- An Engine represents the entry point (within Catalina) that
> processes
> every request. The Engine implementation for Tomcat stand alone
> analyzes the HTTP headers included with the request, and passes
> them
> on to the appropriate Host (virtual host).
> Documentation at /docs/config/engine.html -->
>
> <!-- You should set jvmRoute to support load-balancing via AJP ie :
>
> -->
> <Engine name="Catalina" defaultHost="10.114.43.102"
> jvmRoute="TomcatNode1">
> <!-- Use the LockOutRealm to prevent attempts to guess user passwords
> via a brute-force attack -->
> <Realm className="org.apache.catalina.realm.LockOutRealm">
> <!-- This Realm uses the UserDatabase configured in the global JNDI
> resources under the key "UserDatabase". Any edits
> that are performed against this UserDatabase are immediately
> available for use by the Realm. -->
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase"/>
> </Realm>
>
> <Host name="10.114.43.102" appBase="webapps"
> unpackWARs="true" autoDeploy="true" startStopThreads="0">
> <!--For clustering, please take a look at documentation at:
> /docs/cluster-howto.html (simple how to)
> /docs/config/cluster.html (reference documentation) -->
> <!--
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
> -->
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="4">
> <Channel className="org.apache.catalina.tribes.group.
> GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="228.0.0.4"
> port="45560"
> frequency="500"
> dropTime="9000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.
> PooledParallelSender"/>
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.
> TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.
> MessageDispatch15Interceptor"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
>
> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.
> *\.html|.*\.css|.*\.txt"/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <Deployer className="org.apache.catalina.ha.deploy.
> FarmWarDeployer"
> tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
> deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
> watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
> watchEnabled="false"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
> <!-- SingleSignOn valve, share authentication between web
> applications
> Documentation at: /docs/config/valve.html -->
> <!--
> <Valve className="org.apache.catalina.authenticator.SingleSignOn"
> />
> -->
>
> <!-- Access log processes all example.
> Documentation at: /docs/config/valve.html
> Note: The pattern used is equivalent to using
> pattern="common" -->
> <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
> prefix="localhost_access_log" suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b" />
> <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
> threshold="900" />
>
> </Host>
> </Engine>
> </Service>
> </Server>
>
>
> On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org>
> wrote:
> > Hi
> >
> > You seems to be using BackupManager.
> > What is set for mapSendOptions?
> >
> > The Cluster module uses the Tomcat JULI logging FW,
> > so you can enable logging on the org.apache.catalina.tribes.MESSAGES,
> you
> > can track messages.
> >
> > Note for using the org.apache.catalina.tribes.MESSAGES:
> > The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level
> logs.
> > So It should get logs in a short period of time.
> >
> >
> > 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> >
> >> Guys, we are facing the below errors in Tomcat cluster, please advise
> >>
> >>
> >> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> >> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> >> Unable to send AbstractReplicatedMap.ping message
> >> org.apache.catalina.tribes.ChannelException: Operation has timed
> >> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> >> at org.apache.catalina.tribes.transport.nio.
> ParallelNioSender.sendMessage(
> >> ParallelNioSender.java:108)
> >> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> >> sendMessage(PooledParallelSender.java:48)
> >> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> >> sendMessage(ReplicationTransmitter.java:54)
> >> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
> >> ChannelCoordinator.java:82)
> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> >> ChannelInterceptorBase.java:76)
> >> at org.apache.catalina.tribes.group.interceptors.
> >> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.
> java:81)
> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> >> ChannelInterceptorBase.java:76)
> >> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> >> sendMessage(TcpFailureDetector.java:93)
> >> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> >> ChannelInterceptorBase.java:76)
> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> GroupChannel.java:233)
> >> at org.apache.catalina.tribes.group.GroupChannel.send(
> >> GroupChannel.java:186)
> >> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> >> ping(AbstractReplicatedMap.java:267)
> >> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> >> AbstractReplicatedMap.java:885)
> >> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> >> GroupChannel.java:161)
> >> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> >> GroupChannel.java:697)
> >>
> >>
> >> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> >> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> >> process request in NioReceiver
> >> java.io.IOException: A non-blocking socket operation could not be
> >> completed immediately
> >> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> >> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> >> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> >> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> >> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> >> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> >> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> >> listen(NioReceiver.java:272)
> >> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> >> NioReceiver.java:425)
> >> at java.lang.Thread.run(Unknown Source)
> >>
> >> Regards,
> >> Fady
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> >
> > --
> > Keiichi.Fujino
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> --
> Keiichi.Fujino
> <us...@tomcat.apache.org>
> <us...@tomcat.apache.org>
>
Re: Operation has timed out
Posted by Fady Haikal <fa...@gmail.com>.
Keiichi,
mapSendoption is not configured anyway please check my configuration below
<?xml version='1.0' encoding='utf-8'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Note: A "Server" is not itself a "Container", so you may not
define subcomponents such as "Valves" at this level.
Documentation at /docs/config/server.html
-->
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<!-- Security listener. Documentation at /docs/config/listeners.html
<Listener className="org.apache.catalina.security.SecurityListener" />
-->
<!--APR library loader. Documentation at /docs/apr.html -->
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<!-- Prevent memory leaks due to use of particular java/javax APIs-->
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
/>
<!-- Global JNDI resources
Documentation at /docs/jndi-resources-howto.html
-->
<GlobalNamingResources>
<!-- Editable user database that can also be used by
UserDatabaseRealm to authenticate users
-->
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<!-- A "Service" is a collection of one or more "Connectors" that share
a single "Container" Note: A "Service" is not itself a "Container",
so you may not define subcomponents such as "Valves" at this level.
Documentation at /docs/config/service.html
-->
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one
or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="60000" maxThreads="500"
minSpareThreads="25" maxSpareThreads="75" enableLookups="false"
disableUploadTimeout="true" acceptCount="100" redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="60000"
redirectPort="8443" />
-->
<!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
This connector uses the NIO implementation that requires the JSSE
style configuration. When using the APR/native implementation, the
OpenSSL style configuration is required as described in the APR/native
documentation -->
<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS" />
-->
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<!-- An Engine represents the entry point (within Catalina) that processes
every request. The Engine implementation for Tomcat stand alone
analyzes the HTTP headers included with the request, and passes them
on to the appropriate Host (virtual host).
Documentation at /docs/config/engine.html -->
<!-- You should set jvmRoute to support load-balancing via AJP ie :
-->
<Engine name="Catalina" defaultHost="10.114.43.102" jvmRoute="TomcatNode1">
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
<!-- This Realm uses the UserDatabase configured in the global JNDI
resources under the key "UserDatabase". Any edits
that are performed against this UserDatabase are immediately
available for use by the Realm. -->
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<Host name="10.114.43.102" appBase="webapps"
unpackWARs="true" autoDeploy="true" startStopThreads="0">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="4">
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45560"
frequency="500"
dropTime="9000"/>
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="D:/imaljava/TomcatNode1/tmp/war-temp/"
deployDir="D:/imaljava/TomcatNode1/tmp/war-deploy/"
watchDir="D:/imaljava/TomcatNode1/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- SingleSignOn valve, share authentication between web applications
Documentation at: /docs/config/valve.html -->
<!--
<Valve className="org.apache.catalina.authenticator.SingleSignOn" />
-->
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common" -->
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve"
threshold="900" />
</Host>
</Engine>
</Service>
</Server>
On Thu, Feb 9, 2017 at 10:32 AM, Keiichi Fujino <kf...@apache.org> wrote:
> Hi
>
> You seems to be using BackupManager.
> What is set for mapSendOptions?
>
> The Cluster module uses the Tomcat JULI logging FW,
> so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you
> can track messages.
>
> Note for using the org.apache.catalina.tribes.MESSAGES:
> The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs.
> So It should get logs in a short period of time.
>
>
> 2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
>
>> Guys, we are facing the below errors in Tomcat cluster, please advise
>>
>>
>> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
>> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
>> Unable to send AbstractReplicatedMap.ping message
>> org.apache.catalina.tribes.ChannelException: Operation has timed
>> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
>> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(
>> ParallelNioSender.java:108)
>> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
>> sendMessage(PooledParallelSender.java:48)
>> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
>> sendMessage(ReplicationTransmitter.java:54)
>> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
>> ChannelCoordinator.java:82)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.
>> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
>> sendMessage(TcpFailureDetector.java:93)
>> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
>> ChannelInterceptorBase.java:76)
>> at org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.java:233)
>> at org.apache.catalina.tribes.group.GroupChannel.send(
>> GroupChannel.java:186)
>> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
>> ping(AbstractReplicatedMap.java:267)
>> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
>> AbstractReplicatedMap.java:885)
>> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
>> GroupChannel.java:161)
>> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
>> GroupChannel.java:697)
>>
>>
>> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
>> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
>> process request in NioReceiver
>> java.io.IOException: A non-blocking socket operation could not be
>> completed immediately
>> at sun.nio.ch.SocketDispatcher.close0(Native Method)
>> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
>> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>> at sun.nio.ch.SelectorImpl.select(Unknown Source)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.
>> listen(NioReceiver.java:272)
>> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
>> NioReceiver.java:425)
>> at java.lang.Thread.run(Unknown Source)
>>
>> Regards,
>> Fady
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> --
> Keiichi.Fujino
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Operation has timed out
Posted by Keiichi Fujino <kf...@apache.org>.
Hi
You seems to be using BackupManager.
What is set for mapSendOptions?
The Cluster module uses the Tomcat JULI logging FW,
so you can enable logging on the org.apache.catalina.tribes.MESSAGES, you
can track messages.
Note for using the org.apache.catalina.tribes.MESSAGES:
The org.apache.catalina.tribes.MESSAGES outputs a lot of trace level logs.
So It should get logs in a short period of time.
2017-02-06 15:52 GMT+09:00 Fady Haikal <fa...@gmail.com>:
> Guys, we are facing the below errors in Tomcat cluster, please advise
>
>
> 06-Feb-2017 01:14:20.718 SEVERE [GroupChannel-Heartbeat-1]
> org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat
> Unable to send AbstractReplicatedMap.ping message
> org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{10, 114, 43, 103}:4000;
> at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(
> ParallelNioSender.java:108)
> at org.apache.catalina.tribes.transport.nio.PooledParallelSender.
> sendMessage(PooledParallelSender.java:48)
> at org.apache.catalina.tribes.transport.ReplicationTransmitter.
> sendMessage(ReplicationTransmitter.java:54)
> at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(
> ChannelCoordinator.java:82)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.
> MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:81)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.
> sendMessage(TcpFailureDetector.java:93)
> at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(
> ChannelInterceptorBase.java:76)
> at org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.java:233)
> at org.apache.catalina.tribes.group.GroupChannel.send(
> GroupChannel.java:186)
> at org.apache.catalina.tribes.group.RpcChannel.send(RpcChannel.java:99)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.
> ping(AbstractReplicatedMap.java:267)
> at org.apache.catalina.tribes.tipis.AbstractReplicatedMap.heartbeat(
> AbstractReplicatedMap.java:885)
> at org.apache.catalina.tribes.group.GroupChannel.heartbeat(
> GroupChannel.java:161)
> at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(
> GroupChannel.java:697)
>
>
> 06-Feb-2017 01:20:51.437 SEVERE [NioReceiver]
> org.apache.catalina.tribes.transport.nio.NioReceiver.listen Unable to
> process request in NioReceiver
> java.io.IOException: A non-blocking socket operation could not be
> completed immediately
> at sun.nio.ch.SocketDispatcher.close0(Native Method)
> at sun.nio.ch.SocketDispatcher.close(Unknown Source)
> at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.implDereg(Unknown Source)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
> at sun.nio.ch.SelectorImpl.select(Unknown Source)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.
> listen(NioReceiver.java:272)
> at org.apache.catalina.tribes.transport.nio.NioReceiver.run(
> NioReceiver.java:425)
> at java.lang.Thread.run(Unknown Source)
>
> Regards,
> Fady
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Keiichi.Fujino