You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by Yuval Schwartz <yu...@gmail.com> on 2016/01/27 17:15:54 UTC

Installing APR based Apache Tomcat Native Library

Hello,

tomcat version: 8.0.22
java: jdk1.8.0_05
server: Amazon Linux AMI

When deploying my web application to my production environment (detailed
above), I get a message:



*The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path*
So I wanted to install the Apache Tomcat Native library (does this improve
performance even for a web app that doesn't use SSL?)
According to the documentation: http://tomcat.apache.org/native-doc/
I installed the apr-devel and openssl-devel packages with the command:

yum install apr-devel openssl-devel

However, I don't understand the next part of the instructions which
discusses the "make && make install" command.
>From where do I run this command? I searched and I could not find a
"jni/native" directory.
>From where do I run the "./configure --help" command and the other
"./configure" commands?

Thank you.

Re: Installing APR based Apache Tomcat Native Library

Posted by Yuval Schwartz <yu...@gmail.com>.

Thank you for all of your help, very informative.
Regarding your last answer, I will look at it in greater detail and let you
know if I have questions (I am less familiar with some of the term you
mention there).

On Fri, Feb 5, 2016 at 11:48 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Yuval,
>
> On 2/4/16 3:14 AM, Yuval Schwartz wrote:
> >> On Wed, Feb 3, 2016 at 11:15 PM, Christopher Schultz <
> >> chris@christopherschultz.net> wrote:
> >
> > ELB usually uses HTTP keepalive, so that might be part of the
> > issue.
> >
> >> ELB has a feature called "connection draining" which keeps
> >> connections open while the instance is de-registering (my
> >> instance begins de-registering when I shutdown tomcat) for a set
> >> period of time (default: 300 s). It is this connection that stays
> >> open when tomcat shuts down that causes the WARNING message to
> >> appear. If I disable connection draining then I stop seeing the
> >> WARNING when I shutdown tomcat.
>
> That'll do it.
>
> So, if you had waited 5 minutes between removing your node from ELB
> and shutting-down Tomcat, you'd find that there also no errors.
>
> >> However, "connection draining" seems like a nice feature. One
> >> option to overcome this WARNING while still keeping connection
> >> draining would be to first undeploy my application, then, after
> >> the connection draining time period expires, shutdown tomcat
> >> (tested this, doesn't throw the warning).
> >
> >> Do you have any other recommendations for how I might keep this
> >> feature and adjust something else in my configuration? Any light
> >> to shed on the matter?
>
> Wait 5 minutes?
>
> 5 minutes is quite a long time to drain the node. How long is the
> average request? In production, we use ELB with 30-second draining.
>
> (Then again, we have a layer of stateless httpd instances in
> between... when we drain our Tomcat nodes, we typically wait over an
> hour.)
>
> I think this might come down to the lack of flexibility of ELB.
>
> The way we use ELB, we are balancing between a set of httpd instances
> which are *all* proxying to the same set of back-end Tomcat nodes. So
> the currently-enabled set of httpd instances is irrelevant to the
> client: any httpd can get to any Tomcat.
>
> When we want to service an httpd instance, we remove it from the ELB
> with a short draining period and reboot, upgrade, whatever. No
> interruption to the Tomcat nodes.
>
> If we want to service a Tomcat node, we bring-down the node from
> *httpd*, not from ELB. We are using mod_jk which allows you to put a
> Tomcat node into a "disabled" state, which means existing users will
> still be routed to the node that is "going down" but new users will go
> to other nodes instead. Once the number of sessions on that node goes
> to zero, we STOP the node in mod_jk (NO TRAFFIC AT ALL to the node)
> and then service it.
>
> ELB lacks this two-phase removal feature that mod_jk has. That means,
> when you remove a node from ELB, the countdown starts for your
> users... if they don't log off from that node, they will lose their
> sessions in T minus 5 minutes. If you are using sticky sessions with
> no clustering (like we are), then those users are just out of luck
> when the 5-minute timer runs out.
>
> If you want better behavior, you are going to need more complexity.
>
> /On the other hand/, if you *are* clustered and don't use (or need)
> sticky sessions, then you can significantly reduce that user-draining
> period to the maximum reasonable response time you expect to be able
> to serve. If you have a super-fast entirely-clustered (or stateless)
> service, set the drain time to 1 second. If you have requests that
> routinely last 45 seconds, set the drain-time to more like 90 seconds.
> 5 minutes doesn't seem like it makes any sense to be in any scenario,
> though.
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAla1GMcACgkQ9CaO5/Lv0PBO+ACeJLP4z3DUfcUqVmG30w7sYSsN
> 5ggAoLDU8DSnA1vjVppiN9QU11GFsfng
> =RhN2
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Installing APR based Apache Tomcat Native Library

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yuval,

On 2/4/16 3:14 AM, Yuval Schwartz wrote:
>> On Wed, Feb 3, 2016 at 11:15 PM, Christopher Schultz < 
>> chris@christopherschultz.net> wrote:
> 
> ELB usually uses HTTP keepalive, so that might be part of the
> issue.
> 
>> ELB has a feature called "connection draining" which keeps
>> connections open while the instance is de-registering (my
>> instance begins de-registering when I shutdown tomcat) for a set
>> period of time (default: 300 s). It is this connection that stays
>> open when tomcat shuts down that causes the WARNING message to
>> appear. If I disable connection draining then I stop seeing the
>> WARNING when I shutdown tomcat.

That'll do it.

So, if you had waited 5 minutes between removing your node from ELB
and shutting-down Tomcat, you'd find that there also no errors.

>> However, "connection draining" seems like a nice feature. One
>> option to overcome this WARNING while still keeping connection
>> draining would be to first undeploy my application, then, after
>> the connection draining time period expires, shutdown tomcat
>> (tested this, doesn't throw the warning).
> 
>> Do you have any other recommendations for how I might keep this
>> feature and adjust something else in my configuration? Any light
>> to shed on the matter?

Wait 5 minutes?

5 minutes is quite a long time to drain the node. How long is the
average request? In production, we use ELB with 30-second draining.

(Then again, we have a layer of stateless httpd instances in
between... when we drain our Tomcat nodes, we typically wait over an
hour.)

I think this might come down to the lack of flexibility of ELB.

The way we use ELB, we are balancing between a set of httpd instances
which are *all* proxying to the same set of back-end Tomcat nodes. So
the currently-enabled set of httpd instances is irrelevant to the
client: any httpd can get to any Tomcat.

When we want to service an httpd instance, we remove it from the ELB
with a short draining period and reboot, upgrade, whatever. No
interruption to the Tomcat nodes.

If we want to service a Tomcat node, we bring-down the node from
*httpd*, not from ELB. We are using mod_jk which allows you to put a
Tomcat node into a "disabled" state, which means existing users will
still be routed to the node that is "going down" but new users will go
to other nodes instead. Once the number of sessions on that node goes
to zero, we STOP the node in mod_jk (NO TRAFFIC AT ALL to the node)
and then service it.

ELB lacks this two-phase removal feature that mod_jk has. That means,
when you remove a node from ELB, the countdown starts for your
users... if they don't log off from that node, they will lose their
sessions in T minus 5 minutes. If you are using sticky sessions with
no clustering (like we are), then those users are just out of luck
when the 5-minute timer runs out.

If you want better behavior, you are going to need more complexity.

/On the other hand/, if you *are* clustered and don't use (or need)
sticky sessions, then you can significantly reduce that user-draining
period to the maximum reasonable response time you expect to be able
to serve. If you have a super-fast entirely-clustered (or stateless)
service, set the drain time to 1 second. If you have requests that
routinely last 45 seconds, set the drain-time to more like 90 seconds.
5 minutes doesn't seem like it makes any sense to be in any scenario,
though.

Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla1GMcACgkQ9CaO5/Lv0PBO+ACeJLP4z3DUfcUqVmG30w7sYSsN
5ggAoLDU8DSnA1vjVppiN9QU11GFsfng
=RhN2
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Installing APR based Apache Tomcat Native Library

Posted by Yuval Schwartz <yu...@gmail.com>.

Update below.

On Thu, Feb 4, 2016 at 8:15 AM, Yuval Schwartz <yu...@gmail.com>
wrote:

> Thanks, answers below.
>
> On Wed, Feb 3, 2016 at 11:15 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Yuval,
>>
>> On 2/3/16 12:29 PM, Yuval Schwartz wrote:
>> > Thanks a lot. Comments below.
>> >
>> > On Wed, Feb 3, 2016 at 5:02 PM, Christopher Schultz <
>> > chris@christopherschultz.net> wrote:
>> >
>> > Try this. Before you are ready to shut down, take note of the pid
>> > of the JVM process. Then run this command:
>> >
>> > $ CATALINA_HOME/bin/shutdown.sh && sleep 2 && jstack [pid] >
>> > stack.txt
>> >
>> >
>> >> (Nice idea!) I performed this and it worked. What stood out was a
>> >> line in 2 http-apr-8080 threads: - locked <[some number]> (a
>> >> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
>> >
>> >> Does this tell us something?
>>
>> Can you post the rest of the stack trace?
>>
>
> Yes, sure:
>
> "http-apr-8080-exec-7" #32 daemon prio=5 os_prio=0 tid=0x00007f118c015800
> nid=0x2d10 runnable [0x00007f1192ef1000]
>    java.lang.Thread.State: RUNNABLE
>         at org.apache.tomcat.jni.Socket.recvbb(Native Method)
>         at
> org.apache.coyote.http11.InternalAprInputBuffer.doReadSocket(InternalAprInputBuffer.java:631)
>         at
> org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:583)
>         at
> org.apache.coyote.http11.InternalAprInputBuffer.parseRequestLine(InternalAprInputBuffer.java:140)
>         at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1005)
>         at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
>         at
> org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2400)
>         - locked <0x00000000f12d0080> (a
> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:745)
>
> The thing is, I found an identical thread from a thread dump that I called
> before the shutdown (ie: before the warning was thrown). Only it was named
> "http-apr-8080-exec-6".
>
>
>>
>> > This should request a Tomcat shutdown and take a thread dump 2
>> > seconds later. If that doesn't work, adjust the sleep time until
>> > you get a good snapshot. You should see a thread with a name like
>> > http-apr-8080-[#] that is doing something inside your application.
>> > then its up to you to figure out why it's still running.
>> >
>> >
>> >> As far as I can tell, I only get this WARNING when my elastic
>> >> load balancer is pinging pages on my web app.
>>
>> ELB usually uses HTTP keepalive, so that might be part of the issue.
>>
>
ELB has a feature called "connection draining" which keeps connections open
while the instance is de-registering (my instance begins de-registering
when I shutdown tomcat) for a set period of time (default: 300 s). It is
this connection that stays open when tomcat shuts down that causes the
WARNING message to appear.
If I disable connection draining then I stop seeing the WARNING when I
shutdown tomcat.

However, "connection draining" seems like a nice feature.
One option to overcome this WARNING while still keeping connection draining
would be to first undeploy my application, then, after the connection
draining time period expires, shutdown tomcat (tested this, doesn't throw
the warning).

Do you have any other recommendations for how I might keep this feature and
adjust something else in my configuration? Any light to shed on the matter?

Thanks a lot.


>
>> > What is your connector configuration?
>> >
>> >> The connector element in $CATALINA_HOME/conf/server.xml looks
>> >> like this: <Connector port="8080" protocol="HTTP/1.1"
>> >> connectionTimeout="20000" redirectPort="8443" /> (there is a
>> >> commented connector after it with an attribute of
>> >> "executor=tomcatThreadPool"...but this is commented out).
>>
>> So that's pretty much the default configuration. Good to know.
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iEYEARECAAYFAlaybdUACgkQ9CaO5/Lv0PCS7wCeMKCjgMLSrGAAj5lQUjGG5oza
>> w34An2e7kYagW6zIN6tPTS7kXu/XcAf7
>> =eZ7k
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>

Re: Installing APR based Apache Tomcat Native Library

Posted by Yuval Schwartz <yu...@gmail.com>.

Thanks, answers below.

On Wed, Feb 3, 2016 at 11:15 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Yuval,
>
> On 2/3/16 12:29 PM, Yuval Schwartz wrote:
> > Thanks a lot. Comments below.
> >
> > On Wed, Feb 3, 2016 at 5:02 PM, Christopher Schultz <
> > chris@christopherschultz.net> wrote:
> >
> > Try this. Before you are ready to shut down, take note of the pid
> > of the JVM process. Then run this command:
> >
> > $ CATALINA_HOME/bin/shutdown.sh && sleep 2 && jstack [pid] >
> > stack.txt
> >
> >
> >> (Nice idea!) I performed this and it worked. What stood out was a
> >> line in 2 http-apr-8080 threads: - locked <[some number]> (a
> >> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
> >
> >> Does this tell us something?
>
> Can you post the rest of the stack trace?
>

Yes, sure:

"http-apr-8080-exec-7" #32 daemon prio=5 os_prio=0 tid=0x00007f118c015800
nid=0x2d10 runnable [0x00007f1192ef1000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.tomcat.jni.Socket.recvbb(Native Method)
        at
org.apache.coyote.http11.InternalAprInputBuffer.doReadSocket(InternalAprInputBuffer.java:631)
        at
org.apache.coyote.http11.InternalAprInputBuffer.fill(InternalAprInputBuffer.java:583)
        at
org.apache.coyote.http11.InternalAprInputBuffer.parseRequestLine(InternalAprInputBuffer.java:140)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1005)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
        at
org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2400)
        - locked <0x00000000f12d0080> (a
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

The thing is, I found an identical thread from a thread dump that I called
before the shutdown (ie: before the warning was thrown). Only it was named
"http-apr-8080-exec-6".


>
> > This should request a Tomcat shutdown and take a thread dump 2
> > seconds later. If that doesn't work, adjust the sleep time until
> > you get a good snapshot. You should see a thread with a name like
> > http-apr-8080-[#] that is doing something inside your application.
> > then its up to you to figure out why it's still running.
> >
> >
> >> As far as I can tell, I only get this WARNING when my elastic
> >> load balancer is pinging pages on my web app.
>
> ELB usually uses HTTP keepalive, so that might be part of the issue.
>
> > What is your connector configuration?
> >
> >> The connector element in $CATALINA_HOME/conf/server.xml looks
> >> like this: <Connector port="8080" protocol="HTTP/1.1"
> >> connectionTimeout="20000" redirectPort="8443" /> (there is a
> >> commented connector after it with an attribute of
> >> "executor=tomcatThreadPool"...but this is commented out).
>
> So that's pretty much the default configuration. Good to know.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlaybdUACgkQ9CaO5/Lv0PCS7wCeMKCjgMLSrGAAj5lQUjGG5oza
> w34An2e7kYagW6zIN6tPTS7kXu/XcAf7
> =eZ7k
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Installing APR based Apache Tomcat Native Library

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yuval,

On 2/3/16 12:29 PM, Yuval Schwartz wrote:
> Thanks a lot. Comments below.
> 
> On Wed, Feb 3, 2016 at 5:02 PM, Christopher Schultz < 
> chris@christopherschultz.net> wrote:
> 
> Try this. Before you are ready to shut down, take note of the pid
> of the JVM process. Then run this command:
> 
> $ CATALINA_HOME/bin/shutdown.sh && sleep 2 && jstack [pid] >
> stack.txt
> 
> 
>> (Nice idea!) I performed this and it worked. What stood out was a
>> line in 2 http-apr-8080 threads: - locked <[some number]> (a 
>> org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
> 
>> Does this tell us something?

Can you post the rest of the stack trace?

> This should request a Tomcat shutdown and take a thread dump 2
> seconds later. If that doesn't work, adjust the sleep time until
> you get a good snapshot. You should see a thread with a name like 
> http-apr-8080-[#] that is doing something inside your application. 
> then its up to you to figure out why it's still running.
> 
> 
>> As far as I can tell, I only get this WARNING when my elastic
>> load balancer is pinging pages on my web app.

ELB usually uses HTTP keepalive, so that might be part of the issue.

> What is your connector configuration?
> 
>> The connector element in $CATALINA_HOME/conf/server.xml looks
>> like this: <Connector port="8080" protocol="HTTP/1.1" 
>> connectionTimeout="20000" redirectPort="8443" /> (there is a
>> commented connector after it with an attribute of 
>> "executor=tomcatThreadPool"...but this is commented out).

So that's pretty much the default configuration. Good to know.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlaybdUACgkQ9CaO5/Lv0PCS7wCeMKCjgMLSrGAAj5lQUjGG5oza
w34An2e7kYagW6zIN6tPTS7kXu/XcAf7
=eZ7k
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Installing APR based Apache Tomcat Native Library

Posted by Yuval Schwartz <yu...@gmail.com>.

Thanks a lot. Comments below.

On Wed, Feb 3, 2016 at 5:02 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Yuval,
>
> On 2/2/16 6:11 PM, Yuval Schwartz wrote:
> > Thanks. Answers below
> >
> > On Tuesday, 2 February 2016, Christopher Schultz <
> > chris@christopherschultz.net
> > <javascript:_e(%7B%7D,'cvml','chris@christopherschultz.net');>>
> > wrote:
> >
> > Yuval,
> >
> > On 2/2/16 9:28 AM, Yuval Schwartz wrote:
> >>>> On Tue, Feb 2, 2016 at 4:15 PM, Christopher Schultz <
> >>>> chris@christopherschultz.net> wrote:
> >>>>
> >>>> Yuval,
> >>>>
> >>>> On 2/2/16 6:21 AM, Yuval Schwartz wrote:
> >>>>>>> Hello again Christoph,
> >>>>>>>
> >>>>>>> I think there is a problem with the way that I
> >>>>>>> installed the APR library. If you recall, I installed a
> >>>>>>> compiled version from the distros with the commands yum
> >>>>>>> install apr-devel openssl-devel tomcat-native
> >>>>>>>
> >>>>>>> I get the desired "Loaded APR based library" in my
> >>>>>>> logs. However, every time I shut down tomcat I get a
> >>>>>>> warning:
> >>>>>>>
> >>>>>>> WARNING [main]
> >>>>>>> org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
> >>>>>>>
> >>>>>>>
> The executor associated with thread pool [http-apr-8080] has
> >>>>>>> not fully shutdown. Some application threads may still
> >>>>>>> be running
> >>>>>>>
> >>>>>>> If I remove the packages that I installed (yum remove
> >>>>>>> apr-devel openssl-devel tomcat-native) then this
> >>>>>>> warning goes away.
> >>>>>>>
> >>>>>>> Is there something that I forgot to configure when
> >>>>>>> installing the APR library?
> >>>>
> >>>> What makes you think that something is wrong with the
> >>>> library? Perhaps you are getting that WARNING because you
> >>>> have still-running request-processing threads?
> >>>>
> >>>>
> >>>>> Thanks, any ideas on how I can look into this then? I've
> >>>>> tried looking at thread dumps and at the threads that are
> >>>>> created and destroyed in my ServletContextListener's
> >>>>> contextInitialize and contextDestroyed methods.
> >
> > So, just to confirm, when you attempt to shut-down Tomcat, you get
> > that WARNING and if you take a thread dump after that warning, you
> > have threads still running contextInitialized and
> > contextDestroyed?
> >
> >
> >> When I shut down I get that message (I am able to shut down
> >> successfully, I just get that message right before). I can't take
> >> a thread dump after I get the message because tomcat is no longer
> >> running (i.e. There are no Java processes taking place for me to
> >> perform a thread dump on).
>
> Try this. Before you are ready to shut down, take note of the pid of
> the JVM process. Then run this command:
>
> $ CATALINA_HOME/bin/shutdown.sh && sleep 2 && jstack [pid] > stack.txt
>

(Nice idea!)
I performed this and it worked.
What stood out was a line in 2 http-apr-8080 threads:
- locked <[some number]> (a
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)

Does this tell us something?


>
> This should request a Tomcat shutdown and take a thread dump 2 seconds
> later. If that doesn't work, adjust the sleep time until you get a
> good snapshot. You should see a thread with a name like
> http-apr-8080-[#] that is doing something inside your application.
> then its up to you to figure out why it's still running.
>

As far as I can tell, I only get this WARNING when my elastic load balancer
is pinging pages on my web app.



>
> >> I mentioned contextInitialized and contextDestroyed simply
> >> because I used those methods to print information about threads
> >> that were running in order to try to get a better idea of what's
> >> going on (i.e. Using Thread.getAllStackTraces()).
>
> Well, if you are using contextInitialized and contextDestroyed to dump
> the stacks of all threads, then of course those methods will be listed
> - -- they are currently running (to generate the thread dumps).
>
> > The thread the WARNING is talking about will be one of the HTTP
> > request processor threads (it even tells you which thread pool is
> > still in use: http-apr-8080). If you have some long-running
> > requests, you may get that WARNING during shutdown, but shutdown
> > can still succeed after those threads complete.
> >
> >
> >> When I take a thread dump while the application is running I see
> >> threads called http-apr-8080-exec-1, http-apr-8080-exec-2, etc.
> >> do these threads belong to the pool that the warning is referring
> >> to?
>
> Yes, but while the server is running, you should be expecting to
> handle requests from clients, right? The fact that those threads are
> running is not out of the ordinary. Remember that even threads not
> actively servicing requests will have a non-null stack trace: they'll
> be waiting for new requests.
>
> >> Are you saying I don't need to pay attention to this warning? As
> >> a side question: why should it concern me that a thread might
> >> stay open when tomcat is shutting down? If tomcat is shutting
> >> down then won't all threads be destroyed anyway?
>
> If a request is in-progress when you ask Tomcat to shut down, Tomcat
> will wait for a certain amount of time for those requests to complete.
> If they don't complete within a certain amount of time (configurable:
> see the Configuration section of the user's guide), Tomcat will stop
> them anyway and shut down.
>
> If you get this error every time you shut down, you should find out
> which thread is running long and figure out why.
>
> >>>> If the library loads, I think you installed it properly ;)
> >>>>
> >>>>
> >>>>> Yes, I just reinstalled by compiling the source code and I
> >>>>> still get the same error...so it's not the library.
> >
> > If you are okay with the library version available from your Linux
> > distro, you should stick with that.
> >
> >> I mentioned this point because, interestingly, when I don't use
> >> the apr library (i.e. I use the standard nio) then I don't see
> >> the warning that we're discussing.
>
> So if you use the BIO connector, you never get this warning?
>
> What is your connector configuration?
>

The connector element in $CATALINA_HOME/conf/server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
(there is a commented connector after it with an attribute of
"executor=tomcatThreadPool"...but this is commented out).


>
> - -chris
>

Thank you for your help


> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlayFpgACgkQ9CaO5/Lv0PAJ0ACghxKS1I+POEIlrCvIGOrp0nzM
> 0tcAn1Fxi1neSvL5z8bTS5cE+xEEw5VN
> =/4Yk
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Installing APR based Apache Tomcat Native Library

Posted by Christopher Schultz <ch...@christopherschultz.net>.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yuval,

On 2/2/16 6:11 PM, Yuval Schwartz wrote:
> Thanks. Answers below
> 
> On Tuesday, 2 February 2016, Christopher Schultz < 
> chris@christopherschultz.net 
> <javascript:_e(%7B%7D,'cvml','chris@christopherschultz.net');>>
> wrote:
> 
> Yuval,
> 
> On 2/2/16 9:28 AM, Yuval Schwartz wrote:
>>>> On Tue, Feb 2, 2016 at 4:15 PM, Christopher Schultz < 
>>>> chris@christopherschultz.net> wrote:
>>>> 
>>>> Yuval,
>>>> 
>>>> On 2/2/16 6:21 AM, Yuval Schwartz wrote:
>>>>>>> Hello again Christoph,
>>>>>>> 
>>>>>>> I think there is a problem with the way that I
>>>>>>> installed the APR library. If you recall, I installed a
>>>>>>> compiled version from the distros with the commands yum
>>>>>>> install apr-devel openssl-devel tomcat-native
>>>>>>> 
>>>>>>> I get the desired "Loaded APR based library" in my
>>>>>>> logs. However, every time I shut down tomcat I get a
>>>>>>> warning:
>>>>>>> 
>>>>>>> WARNING [main] 
>>>>>>> org.apache.tomcat.util.net.AbstractEndpoint.shutdownExecutor
>>>>>>>
>>>>>>> 
The executor associated with thread pool [http-apr-8080] has
>>>>>>> not fully shutdown. Some application threads may still
>>>>>>> be running
>>>>>>> 
>>>>>>> If I remove the packages that I installed (yum remove 
>>>>>>> apr-devel openssl-devel tomcat-native) then this
>>>>>>> warning goes away.
>>>>>>> 
>>>>>>> Is there something that I forgot to configure when
>>>>>>> installing the APR library?
>>>> 
>>>> What makes you think that something is wrong with the
>>>> library? Perhaps you are getting that WARNING because you
>>>> have still-running request-processing threads?
>>>> 
>>>> 
>>>>> Thanks, any ideas on how I can look into this then? I've
>>>>> tried looking at thread dumps and at the threads that are
>>>>> created and destroyed in my ServletContextListener's
>>>>> contextInitialize and contextDestroyed methods.
> 
> So, just to confirm, when you attempt to shut-down Tomcat, you get 
> that WARNING and if you take a thread dump after that warning, you 
> have threads still running contextInitialized and
> contextDestroyed?
> 
> 
>> When I shut down I get that message (I am able to shut down
>> successfully, I just get that message right before). I can't take
>> a thread dump after I get the message because tomcat is no longer
>> running (i.e. There are no Java processes taking place for me to 
>> perform a thread dump on).

Try this. Before you are ready to shut down, take note of the pid of
the JVM process. Then run this command:

$ CATALINA_HOME/bin/shutdown.sh && sleep 2 && jstack [pid] > stack.txt

This should request a Tomcat shutdown and take a thread dump 2 seconds
later. If that doesn't work, adjust the sleep time until you get a
good snapshot. You should see a thread with a name like
http-apr-8080-[#] that is doing something inside your application.
then its up to you to figure out why it's still running.

>> I mentioned contextInitialized and contextDestroyed simply
>> because I used those methods to print information about threads
>> that were running in order to try to get a better idea of what's
>> going on (i.e. Using Thread.getAllStackTraces()).

Well, if you are using contextInitialized and contextDestroyed to dump
the stacks of all threads, then of course those methods will be listed
- -- they are currently running (to generate the thread dumps).

> The thread the WARNING is talking about will be one of the HTTP 
> request processor threads (it even tells you which thread pool is 
> still in use: http-apr-8080). If you have some long-running
> requests, you may get that WARNING during shutdown, but shutdown
> can still succeed after those threads complete.
> 
> 
>> When I take a thread dump while the application is running I see
>> threads called http-apr-8080-exec-1, http-apr-8080-exec-2, etc.
>> do these threads belong to the pool that the warning is referring
>> to?

Yes, but while the server is running, you should be expecting to
handle requests from clients, right? The fact that those threads are
running is not out of the ordinary. Remember that even threads not
actively servicing requests will have a non-null stack trace: they'll
be waiting for new requests.

>> Are you saying I don't need to pay attention to this warning? As
>> a side question: why should it concern me that a thread might
>> stay open when tomcat is shutting down? If tomcat is shutting
>> down then won't all threads be destroyed anyway?

If a request is in-progress when you ask Tomcat to shut down, Tomcat
will wait for a certain amount of time for those requests to complete.
If they don't complete within a certain amount of time (configurable:
see the Configuration section of the user's guide), Tomcat will stop
them anyway and shut down.

If you get this error every time you shut down, you should find out
which thread is running long and figure out why.

>>>> If the library loads, I think you installed it properly ;)
>>>> 
>>>> 
>>>>> Yes, I just reinstalled by compiling the source code and I
>>>>> still get the same error...so it's not the library.
> 
> If you are okay with the library version available from your Linux 
> distro, you should stick with that.
> 
>> I mentioned this point because, interestingly, when I don't use
>> the apr library (i.e. I use the standard nio) then I don't see
>> the warning that we're discussing.

So if you use the BIO connector, you never get this warning?

What is your connector configuration?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlayFpgACgkQ9CaO5/Lv0PAJ0ACghxKS1I+POEIlrCvIGOrp0nzM
0tcAn1Fxi1neSvL5z8bTS5cE+xEEw5VN
=/4Yk
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org