You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by "Simon, Leonard" <le...@hsn.net> on 2012/07/02 22:32:20 UTC

General Architecture Question for multiple websites on a single RedHat server

Our goals are as follows:

 The operating system will be RHEL 6.X.

We want to run Apache and Tomcat on a single server running five websites
using different ports for each.

Our goal is to manage each web server independently, i.e., separate
shutdown and startup scripts and hopefully  different log files for each
web server.

We noticed the bundled Apache/Tomcat with RedHat might only give us a
standard install. Is there a way to do an alternate path install?

Thanks and look forward to some ideas on how to accomplish this.

RE: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Jeff Sturm <je...@eprize.com>.

> -----Original Message-----
> From: Simon, Leonard [mailto:leonard.simon@hsn.net]
> Sent: Monday, July 09, 2012 4:24 PM
> 
> Well our Tomcat went out to lunch again and we had to recycle the webserver
> to get things stablized. By this I mean we get reports from the users that
> screens become unresponsive and looking at a top we see tomcat process
> taking 100% CPU.

<cut>

> PSPermGen       total 54336K, used 54288K [0x00007f73d3200000,
> 0x00007f73d6710000, 0x00007f73dda00000)
>   object space 54336K, 99% used

Try growing your perm gen space--this is at 99% and may be thrashing a bit.  Start Tomcat with -XX:MaxPermSize= and a suitable value.

You may find perm gen usage stabilizes at some point, or keeps growing indefinitely.  If the latter, look for excessive use of class loading in your application.   Excessive application redeploying in Tomcat may cause the same thing.

-Jeff



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] RE: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Simon, Leonard" <le...@hsn.net>.

We hit the leap second when it happened and rebooted the servers which
solved that problem.

Thanks anyway

On Mon, Jul 9, 2012 at 5:24 PM, Martin Gainty <mg...@hotmail.com> wrote:

>
> apparently for RH 6.1 stopping and starting the ntp daemon works
> /etc/init.d/ntpd stop; date; date `date +"%m%d%H%M%C%y.%S"`; date ;
> /etc/init.d/ntpd starthttps://bugzilla.redhat.com/show_bug.cgi?id=836748
>
> Martin Gainty
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene
> Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte
> Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht
> dient lediglich dem Austausch von Informationen und entfaltet keine
> rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von
> E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le
> destinataire prévu, nous te demandons avec bonté que pour satisfaire
> informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie
> de ceci est interdite. Ce message sert à l'information seulement et n'aura
> pas n'importe quel effet légalement obligatoire. Étant donné que les email
> peuvent facilement être sujets à la manipulation, nous ne pouvons accepter
> aucune responsabilité pour le contenu fourni.
>
>
> > Date: Mon, 9 Jul 2012 22:42:21 +0200
> > From: aw@ice-sa.com
> > To: users@tomcat.apache.org
> > Subject: Re: [EXTERNAL] Re: General Architecture Question for multiple
> websites on a single RedHat server
> >
> > Simon, Leonard wrote:
> > > Well our Tomcat went out to lunch again and we had to recycle the
> webserver
> > > to get things stablized. By this I mean we get reports from the users
> that
> > > screens become unresponsive and looking at a top we see tomcat process
> > > taking 100% CPU.
> > >
> > > Was able to do a thread dump captured with a kill -3 PID and here it
> is if
> > > anyone is so inclined to comment on it.
> > >
> >
> > Have not followed this thread and not looked at the thread dump (which
> would not tell me
> > anything anyway), but by any chance have you googled for "leap second
> bug" ?
> > Mentioning that because of your "100% CPU" above.
> > See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=479765
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>

RE: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Martin Gainty <mg...@hotmail.com>.

apparently for RH 6.1 stopping and starting the ntp daemon works
/etc/init.d/ntpd stop; date; date `date +"%m%d%H%M%C%y.%S"`; date ; 
/etc/init.d/ntpd starthttps://bugzilla.redhat.com/show_bug.cgi?id=836748

Martin Gainty 
______________________________________________ 
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.

> Date: Mon, 9 Jul 2012 22:42:21 +0200
> From: aw@ice-sa.com
> To: users@tomcat.apache.org
> Subject: Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server
> 
> Simon, Leonard wrote:
> > Well our Tomcat went out to lunch again and we had to recycle the webserver
> > to get things stablized. By this I mean we get reports from the users that
> > screens become unresponsive and looking at a top we see tomcat process
> > taking 100% CPU.
> > 
> > Was able to do a thread dump captured with a kill -3 PID and here it is if
> > anyone is so inclined to comment on it.
> > 
> 
> Have not followed this thread and not looked at the thread dump (which would not tell me 
> anything anyway), but by any chance have you googled for "leap second bug" ?
> Mentioning that because of your "100% CPU" above.
> See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=479765
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by André Warnier <aw...@ice-sa.com>.

Simon, Leonard wrote:
> Well our Tomcat went out to lunch again and we had to recycle the webserver
> to get things stablized. By this I mean we get reports from the users that
> screens become unresponsive and looking at a top we see tomcat process
> taking 100% CPU.
> 
> Was able to do a thread dump captured with a kill -3 PID and here it is if
> anyone is so inclined to comment on it.
> 

Have not followed this thread and not looked at the thread dump (which would not tell me 
anything anyway), but by any chance have you googled for "leap second bug" ?
Mentioning that because of your "100% CPU" above.
See e.g. https://bugzilla.redhat.com/show_bug.cgi?id=479765

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.

> From: Christopher Schultz [mailto:chris@christopherschultz.net] 
> Subject: Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

> This thread dump shows a mostly-idle server with the exception of
> those threads in socketAccept() (not sure why these count as RUNNABLE
> when they are really blocking)

Because they blocked in the underlying native code part of the platform-specific JRE, not in Java code.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Pid <pi...@pidster.com>.

On 11/07/2012 13:21, Simon, Leonard wrote:
> Mark,
> I really do appreciate your time in assisting,
> It's these up front sessions that could lead to consulting gigs later on.

There's something to look forward to.


> Free consultations pay off in the end.

Not very often they don't.


p

Re: [EXTERNAL] Re: Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Simon, Leonard" <le...@hsn.net>.

Mark,
I really do appreciate your time in assisting,
It's these up front sessions that could lead to consulting gigs later on.
Free consultations pay off in the end.



On Tue, Jul 10, 2012 at 3:33 PM, Mark Eggers <it...@yahoo.com> wrote:

> ----- Original Message -----
>
> > From: André Warnier <aw...@ice-sa.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Cc:
> > Sent: Tuesday, July 10, 2012 12:12 PM
> > Subject: Re: [EXTERNAL] Re: Re: General Architecture Question for
> multiple websites on a single RedHat server
> >
> > Mark Eggers wrote:
> > ...
> >
> >>
> >>  . . . . just my two cents.
> >
> > As a consultant in my professional capacity, I'd charge just about
> 100,000
> > times more for that.
>
>
> Writing quick how-to mail messages on a mailing list - cheap.
>
> As a consultant in my professional capacity, writing the scripts, running
> the analysis, replaying the access log on a test system, helping / dealing
> with developers, system admins, system architects, management . . . agreed.
>
> Oh, and document, document, document.
> (but please to always call it research - apologies to Tom Lehrer)
>
> /mde/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Mark Eggers <it...@yahoo.com>.

----- Original Message -----

> From: André Warnier <aw...@ice-sa.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Cc: 
> Sent: Tuesday, July 10, 2012 12:12 PM
> Subject: Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
> 
> Mark Eggers wrote:
> ...
> 
>> 
>>  . . . . just my two cents.
> 
> As a consultant in my professional capacity, I'd charge just about 100,000 
> times more for that.

Writing quick how-to mail messages on a mailing list - cheap.

As a consultant in my professional capacity, writing the scripts, running the analysis, replaying the access log on a test system, helping / dealing with developers, system admins, system architects, management . . . agreed.

Oh, and document, document, document.
(but please to always call it research - apologies to Tom Lehrer)

/mde/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by André Warnier <aw...@ice-sa.com>.

Mark Eggers wrote:
...

> 
> . . . . just my two cents.

As a consultant in my professional capacity, I'd charge just about 100,000 times more for 
that.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Mark Eggers <it...@yahoo.com>.

----- Original Message -----

> From: "Simon, Leonard" <le...@hsn.net>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Cc: 
> Sent: Tuesday, July 10, 2012 9:54 AM
> Subject: Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server
> 
> Chris,
> 
> Thanks for looking at this.
> 
> Tomcat version is 6.0.32.
> mod_jk is at 1.2.31
> 
> 
> Someone else did the thread dump so I'm assuming they did it on the right
> process.
> 
> On Tue, Jul 10, 2012 at 12:19 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
> 
>>  -----BEGIN PGP SIGNED MESSAGE-----
>>  Hash: SHA1
>> 
>>  Simon,
>> 
>>  On 7/9/12 4:24 PM, Simon, Leonard wrote:
>>  > Well our Tomcat went out to lunch again and we had to recycle the
>>  > webserver to get things stablized. By this I mean we get reports
>>  > from the users that screens become unresponsive and looking at a
>>  > top we see tomcat process taking 100% CPU.
>> 
>>  Are you sure this is the right process?
>> 
>>  > Was able to do a thread dump captured with a kill -3 PID and here
>>  > it is if anyone is so inclined to comment on it.
>> 
>>  This thread dump shows a mostly-idle server with the exception of
>>  those threads in socketAccept() (not sure why these count as RUNNABLE
>>  when they are really blocking) and those executing reads from the
>>  client connection(s).
>> 
>>  What exact version of Tomcat are you using, and what version of mod_jk
>>  (or, if you are using mox_proxy_ajp, what httpd version)? IIRC, there
>>  have been some stability improvements in recent Tomcat versions around
>>  the worker threads being returned to their associated connectors.
>> 
>>  - -chris

I didn't see much in the way that rang immediate alarm bells. It looks like you're processing about 18 client connections, and everything else is pretty quiet. These client connections are going through the AJP connector (as you've noted in your reply above).

A few things though:

As someone in this thread has already mentioned, permgen is pretty full. You might try increasing that with -XX:MaxPermSize=128m.

There are a lot of garbage collection threads. You can see this on a multi-core system. From digging around, it appears that the number of parallel garbage collection threads follows this formula:

8 + (5/8)X = GCT

You get one GCT (garbage collection thread) per core for the first 8 cores, and then 5/8 of a thread for every core after that. So in your case:

8 + (5/8)X = 18
X = 16

This means that your system has 24 cores. Are you running on a 24 core system, or have you tuned garbage collection with JVM arguments.

In general, if you're not running into GC issues, tuning GC parameters is counter-productive. If you do have to tune GC parameters, lots of testing is in order.

I noticed that you also have an MQ Trace monitor running. Are you using MQ? Directly accessing an MQ service without going through a pool configured for graceful restarts / retries can cause a system to become unresponsive. However, I don't see any evidence of that in this thread dump.

As I've said off line, it's really difficult to see what's consuming CPU from one thread dump. Here's how to start figuring out what is going on with your system.

1. Keep access logs

If you don't, then start. You'll want the access logs to replay on a test environment to see if you can recreate the problem. JMeter is a good tool for replaying information from access logs.

2. When the problem occurs

a. Multiple thread dumps, about 5 seconds apart. Use a tool like jstack so it's scriptable

   jstack -l [process-id]
   where [process-id] is the process id of the distressed Tomcat

The -l generates a long listing, and may not be necessary. You'll need to have the right permissions (either root or the user running the JVM being targeted with the process id).

b. At the same time use something like the following to see which thread is consuming CPU:

   ps -L -o pcpu,lwp -p [process-id]
   where [process-id] is the process id of the distressed Tomcat

This will show all the threads of the process, the percentage of CPU used for each thread, and the thread process ID. You can then correlate the thread process ID with the thread dump to see exactly what is consuming the CPU.

This will generate tons of output, so it's best to put both in a script and direct the output to files.

Now you'll end up with the following:

1. What requests were being made of your server when the problem occurred
2. Multiple thread dumps while the problem is occurring
3. The identity of the thread (or threads) that is consuming the CPU

Once you get this information, you'll be in a much better position to determine what is causing your problems.

. . . . just my two cents.
/mde/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Simon, Leonard" <le...@hsn.net>.

Chris,

Thanks for looking at this.

Tomcat version is 6.0.32.
mod_jk is at 1.2.31


Someone else did the thread dump so I'm assuming they did it on the right
process.

On Tue, Jul 10, 2012 at 12:19 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Simon,
>
> On 7/9/12 4:24 PM, Simon, Leonard wrote:
> > Well our Tomcat went out to lunch again and we had to recycle the
> > webserver to get things stablized. By this I mean we get reports
> > from the users that screens become unresponsive and looking at a
> > top we see tomcat process taking 100% CPU.
>
> Are you sure this is the right process?
>
> > Was able to do a thread dump captured with a kill -3 PID and here
> > it is if anyone is so inclined to comment on it.
>
> This thread dump shows a mostly-idle server with the exception of
> those threads in socketAccept() (not sure why these count as RUNNABLE
> when they are really blocking) and those executing reads from the
> client connection(s).
>
> What exact version of Tomcat are you using, and what version of mod_jk
> (or, if you are using mox_proxy_ajp, what httpd version)? IIRC, there
> have been some stability improvements in recent Tomcat versions around
> the worker threads being returned to their associated connectors.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/8VfwACgkQ9CaO5/Lv0PCgWwCfbN/E29q4DKee4q1A+IEMmED6
> 8+0AnivucFDMS/7lhPiOb+0tv5I/6vim
> =ABXf
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Simon,

On 7/9/12 4:24 PM, Simon, Leonard wrote:
> Well our Tomcat went out to lunch again and we had to recycle the
> webserver to get things stablized. By this I mean we get reports
> from the users that screens become unresponsive and looking at a
> top we see tomcat process taking 100% CPU.

Are you sure this is the right process?

> Was able to do a thread dump captured with a kill -3 PID and here
> it is if anyone is so inclined to comment on it.

This thread dump shows a mostly-idle server with the exception of
those threads in socketAccept() (not sure why these count as RUNNABLE
when they are really blocking) and those executing reads from the
client connection(s).

What exact version of Tomcat are you using, and what version of mod_jk
(or, if you are using mox_proxy_ajp, what httpd version)? IIRC, there
have been some stability improvements in recent Tomcat versions around
the worker threads being returned to their associated connectors.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/8VfwACgkQ9CaO5/Lv0PCgWwCfbN/E29q4DKee4q1A+IEMmED6
8+0AnivucFDMS/7lhPiOb+0tv5I/6vim
=ABXf
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Simon, Leonard" <le...@hsn.net>.

Well our Tomcat went out to lunch again and we had to recycle the webserver
to get things stablized. By this I mean we get reports from the users that
screens become unresponsive and looking at a top we see tomcat process
taking 100% CPU.

Was able to do a thread dump captured with a kill -3 PID and here it is if
anyone is so inclined to comment on it.

Thanks in advance:

2012-07-09 11:35:11
Full thread dump OpenJDK 64-Bit Server VM (14.0-b16 mixed mode):

"Java2D Disposer" daemon prio=10 tid=0x00007f72c0024800 nid=0x70b in
Object.wait() [0x00007f73b0ecd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ef3d3bd0> (a
java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
        - locked <0x00007f73ef3d3bd0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
        at sun.java2d.Disposer.run(Disposer.java:143)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor68" daemon prio=10 tid=0x00007f732006e800 nid=0x71bf in
Object.wait() [0x00007f73b0fce000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee9847e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee9847e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor67" daemon prio=10 tid=0x00007f732006c800 nid=0x71be in
Object.wait() [0x00007f73b10cf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee984428> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee984428> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor66" daemon prio=10 tid=0x00007f732006a800 nid=0x71bd in
Object.wait() [0x00007f73b11d0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee984068> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee984068> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor65" daemon prio=10 tid=0x00007f7320068800 nid=0x71bc in
Object.wait() [0x00007f73b12d1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee983ca8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee983ca8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor64" daemon prio=10 tid=0x00007f7320066800 nid=0x71b8 in
Object.wait() [0x00007f73b13d2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee939970> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee939970> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor63" daemon prio=10 tid=0x00007f7320064800 nid=0x71b7 in
Object.wait() [0x00007f73b14d3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee9395b0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee9395b0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor62" daemon prio=10 tid=0x00007f7320062800 nid=0x71b6 in
Object.wait() [0x00007f73b15d4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee9391f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee9391f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor61" daemon prio=10 tid=0x00007f7320060800 nid=0x71b5 in
Object.wait() [0x00007f73b16d5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee9389f8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee9389f8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor60" daemon prio=10 tid=0x00007f732005e800 nid=0x71b4 in
Object.wait() [0x00007f73b17d6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee938170> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee938170> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor58" daemon prio=10 tid=0x00007f732005c000 nid=0x71b2 in
Object.wait() [0x00007f73b1ad9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ee937db0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ee937db0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor57" daemon prio=10 tid=0x00007f732005b000 nid=0x71b1 runnable
[0x00007f73b18d7000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f7419229ba8> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor56" daemon prio=10 tid=0x00007f7320059000 nid=0x5c40 in
Object.wait() [0x00007f73b1bda000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ed0cb450> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ed0cb450> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor55" daemon prio=10 tid=0x00007f7320057800 nid=0x5c3f in
Object.wait() [0x00007f73b1cdb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ed0cb340> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ed0cb340> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor54" daemon prio=10 tid=0x00007f7320056000 nid=0x5c3e in
Object.wait() [0x00007f73b1ddc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ed0cb230> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ed0cb230> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor53" daemon prio=10 tid=0x00007f7320055000 nid=0x5c3d in
Object.wait() [0x00007f73b1edd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ed0cb560> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ed0cb560> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor52" daemon prio=10 tid=0x00007f7320053000 nid=0x4d1b in
Object.wait() [0x00007f73b1fde000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73eccb4888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73eccb4888> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor51" daemon prio=10 tid=0x00007f7320051000 nid=0x4d1a runnable
[0x00007f73b20df000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f741920cce8> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor50" daemon prio=10 tid=0x00007f732004f000 nid=0x4d19 in
Object.wait() [0x00007f73b21e0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73eccb4bc8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73eccb4bc8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor49" daemon prio=10 tid=0x00007f732004e800 nid=0x4d18 runnable
[0x00007f73b22e1000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f741955bb18> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor48" daemon prio=10 tid=0x00007f732004c800 nid=0x33d7 in
Object.wait() [0x00007f73b23e2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e9173380> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e9173380> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor47" daemon prio=10 tid=0x00007f732004a800 nid=0x33d6 in
Object.wait() [0x00007f73b24e3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e9174578> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e9174578> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor46" daemon prio=10 tid=0x00007f7320048800 nid=0x33d5 in
Object.wait() [0x00007f73b25e4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e9174908> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e9174908> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor45" daemon prio=10 tid=0x00007f7320046800 nid=0x33d4 in
Object.wait() [0x00007f73b26e5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e9174c98> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e9174c98> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor44" daemon prio=10 tid=0x00007f7320044800 nid=0x33d3 runnable
[0x00007f73b27e6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73fa9119c0> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor42" daemon prio=10 tid=0x00007f7320041000 nid=0x33d1 in
Object.wait() [0x00007f73b29e8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e91b45a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e91b45a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor41" daemon prio=10 tid=0x00007f732003f800 nid=0x33d0 in
Object.wait() [0x00007f73b2ae9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e91c1798> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e91c1798> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor40" daemon prio=10 tid=0x00007f732003d800 nid=0x3326 in
Object.wait() [0x00007f73b2bea000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e91fb738> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e91fb738> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor39" daemon prio=10 tid=0x00007f732003c000 nid=0x3325 in
Object.wait() [0x00007f73b2ceb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e923baf8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e923baf8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor38" daemon prio=10 tid=0x00007f732003a800 nid=0x3324 in
Object.wait() [0x00007f73b2dec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e9243b00> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e9243b00> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor37" daemon prio=10 tid=0x00007f7320039800 nid=0x3323 runnable
[0x00007f73b2eed000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73f9c698f0> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor36" daemon prio=10 tid=0x00007f7320037800 nid=0x3254 runnable
[0x00007f73b2fee000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73f9dd91a8> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor35" daemon prio=10 tid=0x00007f7320035800 nid=0x3253 in
Object.wait() [0x00007f73b30ef000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e6e8bce8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e6e8bce8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor34" daemon prio=10 tid=0x00007f7320034000 nid=0x3252 in
Object.wait() [0x00007f73b31f0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e895fb98> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e895fb98> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor33" daemon prio=10 tid=0x00007f7320032800 nid=0x3251 in
Object.wait() [0x00007f73b32f1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e6e901e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e6e901e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor32" daemon prio=10 tid=0x00007f7320030800 nid=0x310f in
Object.wait() [0x00007f73b33f2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e59be4a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e59be4a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor31" daemon prio=10 tid=0x00007f732002e800 nid=0x310e in
Object.wait() [0x00007f73b34f3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e59be5b0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e59be5b0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor30" daemon prio=10 tid=0x00007f732002d000 nid=0x310d in
Object.wait() [0x00007f73b35f4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e59be7d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e59be7d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor29" daemon prio=10 tid=0x00007f732002c000 nid=0x310c runnable
[0x00007f73b36f5000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73fb0f5468> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor28" daemon prio=10 tid=0x00007f732002a000 nid=0x2f0f in
Object.wait() [0x00007f73b37f6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e35477e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e35477e0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor27" daemon prio=10 tid=0x00007f7320028000 nid=0x2f0e in
Object.wait() [0x00007f73b38f7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e35478f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e35478f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor26" daemon prio=10 tid=0x00007f7320026800 nid=0x2f0d runnable
[0x00007f73b39f8000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f74194ed160> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor24" daemon prio=10 tid=0x00007f7320023800 nid=0x2e61 runnable
[0x00007f73b3bfa000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73f86cf6d0> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor23" daemon prio=10 tid=0x00007f7320022000 nid=0x2e60 in
Object.wait() [0x00007f73b3cfb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d96538> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d96538> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor22" daemon prio=10 tid=0x00007f7320020800 nid=0x2e5f in
Object.wait() [0x00007f73b3dfc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d96648> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d96648> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor21" daemon prio=10 tid=0x00007f7320020000 nid=0x2e5e runnable
[0x00007f73d03be000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f741922fd08> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor20" daemon prio=10 tid=0x00007f732001e000 nid=0x2e32 in
Object.wait() [0x00007f73b3efd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d4c468> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d4c468> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor19" daemon prio=10 tid=0x00007f732001c000 nid=0x2e31 in
Object.wait() [0x00007f73b3ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d472f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d472f0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor18" daemon prio=10 tid=0x00007f732001a800 nid=0x2e30 in
Object.wait() [0x00007f73d01bc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d47400> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d47400> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor17" daemon prio=10 tid=0x00007f7320019000 nid=0x2e2f in
Object.wait() [0x00007f73d02bd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73e2d4c578> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73e2d4c578> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor15" daemon prio=10 tid=0x00007f7320016000 nid=0x2368 in
Object.wait() [0x00007f73d05c0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73dfcbd590> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73dfcbd590> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor13" daemon prio=10 tid=0x00007f7320014000 nid=0x2366 runnable
[0x00007f73d07c2000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73fa315aa0> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"Thread-17" daemon prio=10 tid=0x00007f7318003800 nid=0xb0f waiting on
condition [0x00007f73d08f8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
sun.print.UnixPrintServiceLookup$PrinterChangeListener.run(UnixPrintServiceLookup.java:782)

"WebSphere MQ Trace Monitor" daemon prio=10 tid=0x00007f732c0de800
nid=0x86f waiting on condition [0x00007f73d09f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
com.ibm.mq.commonservices.internal.monitor.TraceMonitor.run(TraceMonitor.java:134)

"TP-Processor12" daemon prio=10 tid=0x00007f7320011800 nid=0x86e in
Object.wait() [0x00007f73d0afa000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de4db8d8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de4db8d8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor11" daemon prio=10 tid=0x00007f7320010000 nid=0x86d in
Object.wait() [0x00007f73d0bfb000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de4db9e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de4db9e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor10" daemon prio=10 tid=0x00007f732000e800 nid=0x86c in
Object.wait() [0x00007f73d0cfc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de4dbd78> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de4dbd78> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Processor9" daemon prio=10 tid=0x00007f732000d000 nid=0x86b runnable
[0x00007f73d0dfd000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:146)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        - locked <0x00007f73fa8824a0> (a java.io.BufferedInputStream)
        at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
        at
org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
        at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor8" daemon prio=10 tid=0x00007f732000b800 nid=0x831 in
Object.wait() [0x00007f73d0efe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de3f8cc0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de3f8cc0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor6" daemon prio=10 tid=0x00007f7320008800 nid=0x82e in
Object.wait() [0x00007f73d1100000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de3f8ee0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de3f8ee0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor5" daemon prio=10 tid=0x00007f7320007800 nid=0x82d in
Object.wait() [0x00007f73d1201000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73de3f8dd0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73de3f8dd0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
         at java.lang.Thread.run(Thread.java:636)

"TP-Monitor" daemon prio=10 tid=0x00007f75583ca800 nid=0x35e in
Object.wait() [0x00007f73d13bb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ddd65e20> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
        at
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:565)
        - locked <0x00007f73ddd65e20> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor4" daemon prio=10 tid=0x00007f75583ca000 nid=0x35d runnable
[0x00007f73d14bc000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
        at java.net.ServerSocket.implAccept(ServerSocket.java:470)
        at java.net.ServerSocket.accept(ServerSocket.java:438)
        at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:311)
        at
org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:668)
        at
org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor2" daemon prio=10 tid=0x00007f755846a800 nid=0x35b in
Object.wait() [0x00007f73d16be000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73dddd6ed8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73dddd6ed8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"TP-Processor1" daemon prio=10 tid=0x00007f7558438800 nid=0x35a in
Object.wait() [0x00007f73d17bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73ddd664f8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Object.wait(Object.java:502)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:662)
        - locked <0x00007f73ddd664f8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
        at java.lang.Thread.run(Thread.java:636)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
tid=0x00007f7558384800 nid=0x359 waiting on condition [0x00007f73d1ad5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1627)
        at java.lang.Thread.run(Thread.java:636)

"GC Daemon" daemon prio=10 tid=0x00007f75582a7800 nid=0x32a in
Object.wait() [0x00007f73d1bdf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73dda0dda0> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:117)
        - locked <0x00007f73dda0dda0> (a sun.misc.GC$LatencyLock)

"Low Memory Detector" daemon prio=10 tid=0x00007f75580b9000 nid=0x31a
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007f75580b5800 nid=0x319 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00007f75580b3800 nid=0x318 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f75580b1800 nid=0x317 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f7558093000 nid=0x2ef in
Object.wait() [0x00007f73d2e4f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73dda21410> (a
java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
        - locked <0x00007f73dda21410> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x00007f7558091000 nid=0x2ec in
Object.wait() [0x00007f73d2f50000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f73dda21400> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        - locked <0x00007f73dda21400> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f7558006800 nid=0x29c runnable
[0x00007f755f34b000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
        at java.net.ServerSocket.implAccept(ServerSocket.java:470)
        at java.net.ServerSocket.accept(ServerSocket.java:438)
        at
org.apache.catalina.core.StandardServer.await(StandardServer.java:431)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:628)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

"VM Thread" prio=10 tid=0x00007f7558089800 nid=0x2df runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f7558010800 nid=0x29f
runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f7558012800 nid=0x2a0
runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f7558014000 nid=0x2a1
runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f7558016000 nid=0x2a2
runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f7558018000 nid=0x2bc
runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f755801a000 nid=0x2bd
runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f755801b800 nid=0x2be
runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f755801d800 nid=0x2bf
runnable

"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f755801f800 nid=0x2c0
runnable

"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f7558021000 nid=0x2c1
runnable

"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f7558023000 nid=0x2c2
runnable

"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f7558025000 nid=0x2c3
runnable

"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f7558026800 nid=0x2c5
runnable

"GC task thread#13 (ParallelGC)" prio=10 tid=0x00007f7558028800 nid=0x2c6
runnable

"GC task thread#14 (ParallelGC)" prio=10 tid=0x00007f755802a800 nid=0x2c7
runnable

"GC task thread#15 (ParallelGC)" prio=10 tid=0x00007f755802c000 nid=0x2c8
runnable

"GC task thread#16 (ParallelGC)" prio=10 tid=0x00007f755802e000 nid=0x2c9
runnable

"GC task thread#17 (ParallelGC)" prio=10 tid=0x00007f7558030000 nid=0x2ca
runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f75580bb800 nid=0x31b waiting
on condition

JNI global references: 1206

Heap
 PSYoungGen      total 446656K, used 470K [0x00007f74d7e00000,
0x00007f74f3430000, 0x00007f7555000000)
  eden space 444992K, 0% used
[0x00007f74d7e00000,0x00007f74d7e75880,0x00007f74f3090000)
  from space 1664K, 0% used
[0x00007f74f3250000,0x00007f74f3250000,0x00007f74f33f0000)
  to   space 1792K, 0% used
[0x00007f74f3090000,0x00007f74f3090000,0x00007f74f3250000)
 PSOldGen        total 4100096K, used 982518K [0x00007f73dda00000,
0x00007f74d7e00000, 0x00007f74d7e00000)
  object space 4100096K, 23% used
[0x00007f73dda00000,0x00007f741997da40,0x00007f74d7e00000)
 PSPermGen       total 54336K, used 54288K [0x00007f73d3200000,
0x00007f73d6710000, 0x00007f73dda00000)
  object space 54336K, 99% used
[0x00007f73d3200000,0x00007f73d6704318,0x00007f73d6710000)

Jul 9, 2012 11:37:47 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina


On Wed, Jul 4, 2012 at 9:02 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leonard,
>
> On 7/2/12 4:32 PM, Simon, Leonard wrote:
> > Our goals are as follows:
> >
> > The operating system will be RHEL 6.X.
> >
> > We want to run Apache and Tomcat on a single server running five
> > websites using different ports for each.
> >
> > Our goal is to manage each web server independently, i.e.,
> > separate shutdown and startup scripts and hopefully  different log
> > files for each web server.
> >
> > We noticed the bundled Apache/Tomcat with RedHat might only give us
> > a standard install. Is there a way to do an alternate path
> > install?
> >
> > Thanks and look forward to some ideas on how to accomplish this.
>
> I periodically threaten to post my multi-instance Apache Ant scripts
> to the list. I have recently split my build process into multiple
> build scripts including a build-tomcat.xml which will manage that kind
> of thing for you. It's still not a perfect separation, but it allows
> you to do things like define 'projectname.property=value' in
> ~/.ant.properties, set <project name="name"> in build.xml, then
> duplicate the whole thing over and over again.
>
> So, I have, for instance, "myproject1.tomcat-port" and
> "myproject1.tomcat-shutdown-port", etc. and then (near) duplicates for
> each project. Each developer on a shared dev server gets a batch of
> port numbers assigned for their own uses and then sets those up
> accordingly. Each webapp can also have a separate projectX.tomcat-home
> and projectX.java-home property, so that simply typing 'ant
> tomcat-start' in any project's directory will build an appropriate
> CATALINA_BASE (ports and all) for that webapp and then launch the
> webapp in that instance.
>
> All of this requires a fair amount of infrastructure to set up (build
> scripts, environment-specific server.xml, etc. files, etc.) but once
> it's done it makes your life sooooo much easier.
>
> I can share what I've got, but it will likely require you to heavily
> modify it to work in your environment since it's so deeply integrated
> into our complete build process.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/055sACgkQ9CaO5/Lv0PCr/QCbBAC4LQmZxoMzdtarFxBXHbb8
> qjEAn3ldY08iFkVxlWrEkTLDgfxA7qql
> =ev/P
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leonard,

On 7/5/12 1:05 PM, Simon, Leonard wrote:
> I don't envision us using that but thanks anyway. We are looking
> more at a vendor supported solution because of the many things the
> system administration group supports and do not have the expertise
> nor time for customized solutions.

So, you want a vendor-supported solution and you're asking a
volunteer-based community? I think you should be talking to RedHat.

Honestly, this kind of thing really needs to be heavily customized to
meet your needs. Just because my port-assignment conventions make
sense to me doesn't mean they will work for you.

For example, most of our webapps (which run in their own JVM/Tomcat
combinations) have a localhost-only HTTP connector which other webapps
can use as a loopback connection. That means that each webapp actually
requires *three* port numbers: one for AJP (for proxying from actual
clients), one for shutdown, and one for localhost-HTTP. (I suppose
technically, we could disable socket-based shutdown to free-up a port,
but the convenience of a clean shutdown is a nice thing to have).

If I were to submit my build scripts and get them included (even as an
unsupported contribution) in the distribution, they would need
tweaking to be useful to you, and then you have the same maintenance
issue that you would have with my (original) scripts: anytime an
update was released, you'd have to merge-in any changes. I suspect the
same will be true with any vendor-specific solution unless you pay
them directly to support something custom-built.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/22YUACgkQ9CaO5/Lv0PAVDACeIf1BbYgMFzsMoUJGEs7RWEgp
5ewAoL4Moo+LGkQlTlbs5J4SyndaJVp1
=nkWQ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: [EXTERNAL] Re: General Architecture Question for multiple websites on a single RedHat server

Posted by "Simon, Leonard" <le...@hsn.net>.

Chris,

I don't envision us using that but thanks anyway.
We are looking more at a vendor supported solution because of the many
things the system administration group supports and do not have the
expertise nor time for customized solutions.

But again thanks for offering.

We need to figure out what is causing Tomcat to use 100% of the CPU for
what we currently have in place. We were able to capture a thread dump at
the time of slowdown using a kill -3 but when the logs rotated we lost the
data. Will have to wait for the next slowdown.

On Wed, Jul 4, 2012 at 9:02 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Leonard,
>
> On 7/2/12 4:32 PM, Simon, Leonard wrote:
> > Our goals are as follows:
> >
> > The operating system will be RHEL 6.X.
> >
> > We want to run Apache and Tomcat on a single server running five
> > websites using different ports for each.
> >
> > Our goal is to manage each web server independently, i.e.,
> > separate shutdown and startup scripts and hopefully  different log
> > files for each web server.
> >
> > We noticed the bundled Apache/Tomcat with RedHat might only give us
> > a standard install. Is there a way to do an alternate path
> > install?
> >
> > Thanks and look forward to some ideas on how to accomplish this.
>
> I periodically threaten to post my multi-instance Apache Ant scripts
> to the list. I have recently split my build process into multiple
> build scripts including a build-tomcat.xml which will manage that kind
> of thing for you. It's still not a perfect separation, but it allows
> you to do things like define 'projectname.property=value' in
> ~/.ant.properties, set <project name="name"> in build.xml, then
> duplicate the whole thing over and over again.
>
> So, I have, for instance, "myproject1.tomcat-port" and
> "myproject1.tomcat-shutdown-port", etc. and then (near) duplicates for
> each project. Each developer on a shared dev server gets a batch of
> port numbers assigned for their own uses and then sets those up
> accordingly. Each webapp can also have a separate projectX.tomcat-home
> and projectX.java-home property, so that simply typing 'ant
> tomcat-start' in any project's directory will build an appropriate
> CATALINA_BASE (ports and all) for that webapp and then launch the
> webapp in that instance.
>
> All of this requires a fair amount of infrastructure to set up (build
> scripts, environment-specific server.xml, etc. files, etc.) but once
> it's done it makes your life sooooo much easier.
>
> I can share what I've got, but it will likely require you to heavily
> modify it to work in your environment since it's so deeply integrated
> into our complete build process.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/055sACgkQ9CaO5/Lv0PCr/QCbBAC4LQmZxoMzdtarFxBXHbb8
> qjEAn3ldY08iFkVxlWrEkTLDgfxA7qql
> =ev/P
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leonard,

On 7/2/12 4:32 PM, Simon, Leonard wrote:
> Our goals are as follows:
> 
> The operating system will be RHEL 6.X.
> 
> We want to run Apache and Tomcat on a single server running five
> websites using different ports for each.
> 
> Our goal is to manage each web server independently, i.e.,
> separate shutdown and startup scripts and hopefully  different log
> files for each web server.
> 
> We noticed the bundled Apache/Tomcat with RedHat might only give us
> a standard install. Is there a way to do an alternate path
> install?
> 
> Thanks and look forward to some ideas on how to accomplish this.

I periodically threaten to post my multi-instance Apache Ant scripts
to the list. I have recently split my build process into multiple
build scripts including a build-tomcat.xml which will manage that kind
of thing for you. It's still not a perfect separation, but it allows
you to do things like define 'projectname.property=value' in
~/.ant.properties, set <project name="name"> in build.xml, then
duplicate the whole thing over and over again.

So, I have, for instance, "myproject1.tomcat-port" and
"myproject1.tomcat-shutdown-port", etc. and then (near) duplicates for
each project. Each developer on a shared dev server gets a batch of
port numbers assigned for their own uses and then sets those up
accordingly. Each webapp can also have a separate projectX.tomcat-home
and projectX.java-home property, so that simply typing 'ant
tomcat-start' in any project's directory will build an appropriate
CATALINA_BASE (ports and all) for that webapp and then launch the
webapp in that instance.

All of this requires a fair amount of infrastructure to set up (build
scripts, environment-specific server.xml, etc. files, etc.) but once
it's done it makes your life sooooo much easier.

I can share what I've got, but it will likely require you to heavily
modify it to work in your environment since it's so deeply integrated
into our complete build process.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/055sACgkQ9CaO5/Lv0PCr/QCbBAC4LQmZxoMzdtarFxBXHbb8
qjEAn3ldY08iFkVxlWrEkTLDgfxA7qql
=ev/P
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Pid <pi...@pidster.com>.

On 02/07/2012 22:34, Mark Eggers wrote:
> ----- Original Message -----
> 
>> From: "Simon, Leonard" <le...@hsn.net>
>> To: users@tomcat.apache.org
>> Cc: 
>> Sent: Monday, July 2, 2012 1:32 PM
>> Subject: General Architecture Question for multiple websites on a single RedHat server
>>
>> Our goals are as follows:
>>
>> The operating system will be RHEL 6.X.
>>
>> We want to run Apache and Tomcat on a single server running five websites
>> using different ports for each.
>>
>> Our goal is to manage each web server independently, i.e., separate
>> shutdown and startup scripts and hopefully  different log files for each
>> web server.
>>
>> We noticed the bundled Apache/Tomcat with RedHat might only give us a
>> standard install. Is there a way to do an alternate path install?
>>
>> Thanks and look forward to some ideas on how to accomplish this.
>>
> 
> 1. Don't use the distribution package - download from tomcat.apache.org
> 2. Use CATALINA_HOME / CATALINA_BASE - see RUNNING.txt for pointers
> 3. Create an unprivileged user for running the Tomcats
> 4. Create a template server init script for starting and stopping (version, status, etc) Tomcat
> 5. Copy the template init script to each named service (tomcat1, tomcat2, tomcat3, etc.)
> 6. Optionally place service-specific parameters in an /etc/sysconfig subdirectory
> 7. Add the services with chkconfig
> 8. Start and stop each service separately
> 
> Don't forget to change the relevant ports in each server.xml file. An ant script with the xmltask add-on is quite handy for doing this. You can then write a properties file with configuration specifics for each CATALINA_BASE, run the ant tasks to create the CATALINA_BASE, service support files, etc., and then install. The properties files then become a nice base document for all of the Tomcat servers that you've installed.
> 
> Alternatively, if all of your web sites have similar uptime / control / access requirements, you can run all 5 sites using a combination of named virtual hosts on Apache HTTPD and virtual hosts on Tomcat.
> 
> Or a combination of the above . . . It all depends on your requirements.

+1


p

> . . . . just my two cents.
> /mde/
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


-- 

[key:62590808]

Re: General Architecture Question for multiple websites on a single RedHat server

Posted by Mark Eggers <it...@yahoo.com>.

----- Original Message -----

> From: "Simon, Leonard" <le...@hsn.net>
> To: users@tomcat.apache.org
> Cc: 
> Sent: Monday, July 2, 2012 1:32 PM
> Subject: General Architecture Question for multiple websites on a single RedHat server
> 
> Our goals are as follows:
> 
> The operating system will be RHEL 6.X.
> 
> We want to run Apache and Tomcat on a single server running five websites
> using different ports for each.
> 
> Our goal is to manage each web server independently, i.e., separate
> shutdown and startup scripts and hopefully  different log files for each
> web server.
> 
> We noticed the bundled Apache/Tomcat with RedHat might only give us a
> standard install. Is there a way to do an alternate path install?
> 
> Thanks and look forward to some ideas on how to accomplish this.
> 

1. Don't use the distribution package - download from tomcat.apache.org
2. Use CATALINA_HOME / CATALINA_BASE - see RUNNING.txt for pointers
3. Create an unprivileged user for running the Tomcats
4. Create a template server init script for starting and stopping (version, status, etc) Tomcat
5. Copy the template init script to each named service (tomcat1, tomcat2, tomcat3, etc.)
6. Optionally place service-specific parameters in an /etc/sysconfig subdirectory
7. Add the services with chkconfig
8. Start and stop each service separately

Don't forget to change the relevant ports in each server.xml file. An ant script with the xmltask add-on is quite handy for doing this. You can then write a properties file with configuration specifics for each CATALINA_BASE, run the ant tasks to create the CATALINA_BASE, service support files, etc., and then install. The properties files then become a nice base document for all of the Tomcat servers that you've installed.

Alternatively, if all of your web sites have similar uptime / control / access requirements, you can run all 5 sites using a combination of named virtual hosts on Apache HTTPD and virtual hosts on Tomcat.

Or a combination of the above . . . It all depends on your requirements.

. . . . just my two cents.
/mde/

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org