You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rohit Kelapure <ke...@gmail.com> on 2011/10/26 17:55:13 UTC
ThreadLocal Memory Leak Protection - Executor threadRenewalDelay questions
* Reposting from the dev list as advised *
Dear All,
After going through the thread renewal code in
/tomcat-8.0.x/java/org/apache/tomcat/util/threads/TaskQueue.java ,
/tomcat-8.0.x/java/org/apache/tomcat/util/threads/ThreadPoolExecutor.java
and the bug (Improve ThreadLocal memory leak clean-up)
https://issues.apache.org/bugzilla/show_bug.cgi?id=49159 I had the
following questions-
1. Once the thread pool has been renewed after No. of active threads
in pool * max (threadKeepAliveTimeout, longestRequest+
threadRenewalDelay) seconds, why does the ThreadPoolExecutor still
keep paying the price of thread renewal. ?
After org.apache.tomcat.util.threads.ThreadPoolExecutor.threadRenewalDelay
is set to 1000L it never goes back to -1. Ideally once the all the
threads in the threadpool is renewed we should revert the
threadRenewalDelay back to -1 and NOT call
ThreadPoolExecutor.currentThreadShouldBeStopped()/ThreadPoolExecutor.stopCurrentThreadIfNeeded()
in TaskQueue.poll(long, TimeUnit) or TaskQueue.take().
Is this because we are never quite sure as to when *all* of the
threads in the pool have been renewed ?
2. Does the thread renewal approach scale under load (i.e. all threads
in the pool busy servicing requests and CPU close to 90%) ? Is it
meant for production deployments ?
3. How about threads that are servicing long running HTTP Keep-alive
connections that never let the thread return to the pool. Are these
threads renewed before the server is stopped.
--Thanks,
Rohit Kelapure
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ThreadLocal Memory Leak Protection - Executor threadRenewalDelay
questions
Posted by Mark Thomas <ma...@apache.org>.
On 26/10/2011 16:55, Rohit Kelapure wrote:
> Is this because we are never quite sure as to when *all* of the
> threads in the pool have been renewed ?
I assume so, yes.
> 2. Does the thread renewal approach scale under load (i.e. all threads
> in the pool busy servicing requests and CPU close to 90%) ?
Don't know. It hasn't been tested. Looking at when it tests for renewal
it should only ever do it when there is some slack.
> Is it meant for production deployments ?
Will it work? Yes. Would you be better off fixing whatever memory leak
means you need to use this in the first place. Absolutely yes.
> 3. How about threads that are servicing long running HTTP Keep-alive
> connections that never let the thread return to the pool. Are these
> threads renewed before the server is stopped.
If the threads are never returned then they will never be renewed.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ThreadLocal Memory Leak Protection - Executor threadRenewalDelay questions
Posted by Rohit Kelapure <ke...@gmail.com>.
Sylvain and Mark,
Thank you for answering my questions!
On Thu, Oct 27, 2011 at 3:16 PM, Sylvain Laurent <sl...@apache.org> wrote:
> Hello,
>
> On 26 oct. 2011, at 17:55, Rohit Kelapure wrote:
>
>> * Reposting from the dev list as advised *
>>
>> Dear All,
>>
>> After going through the thread renewal code in
>> /tomcat-8.0.x/java/org/apache/tomcat/util/threads/TaskQueue.java ,
>> /tomcat-8.0.x/java/org/apache/tomcat/util/threads/ThreadPoolExecutor.java
>> and the bug (Improve ThreadLocal memory leak clean-up)
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=49159 I had the
>> following questions-
>>
>> 1. Once the thread pool has been renewed after No. of active threads
>> in pool * max (threadKeepAliveTimeout, longestRequest+
>> threadRenewalDelay) seconds, why does the ThreadPoolExecutor still
>> keep paying the price of thread renewal. ?
>> After org.apache.tomcat.util.threads.ThreadPoolExecutor.threadRenewalDelay
>> is set to 1000L it never goes back to -1. Ideally once the all the
>> threads in the threadpool is renewed we should revert the
>> threadRenewalDelay back to -1 and NOT call
>> ThreadPoolExecutor.currentThreadShouldBeStopped()/ThreadPoolExecutor.stopCurrentThreadIfNeeded()
>> in TaskQueue.poll(long, TimeUnit) or TaskQueue.take().
>> Is this because we are never quite sure as to when *all* of the
>> threads in the pool have been renewed ?
>
> Yes. threadRenewalDelay is a configuration parameter of the ThreadPoolExecutor and its value never changes at runtime (unless you play with JMX).
> The thing is that it may take a long time to renew all the threads of the pool because there might be very long requests. Besides, a second renewal can be triggered (i.e. another webapp being stopped) while some threads have not been renewed yet after the first renewal was asked. By comparing the timestamps of when the thread was created and when the last renewal was triggered, we are sure that all threads are eventually renewed (provided they are returned to the pool).
>
>>
>> 2. Does the thread renewal approach scale under load (i.e. all threads
>> in the pool busy servicing requests and CPU close to 90%) ? Is it
>> meant for production deployments ?
> Yes, it was meant for both development and production and is enabled by default.
> When I was developing it I performed some load tests with JMeter to check that there is no functional impact while threads are being renewed, but I did not measure the performance impact of recreating new threads. Anyway, threadRenewalDelay is here to throttle the rate of renewal, so that not all threads are re-created at the same time, thus the performance impact should be negligible (and configurable).
>
>>
>> 3. How about threads that are servicing long running HTTP Keep-alive
>> connections that never let the thread return to the pool. Are these
>> threads renewed before the server is stopped.
> With tomcat 6 (and the BIO connector, I don't know for others), threads were tied to the TCP connection as you describe, so that they were not returned to the pool if the connection was kept alive. But kee-palive is not eternal, the number of requests is limited and there's a timeout if the client does not send a new request for some time.
>
> From tomcat 7 (with the BIO connector at least), threads are decoupled from TCP connections, so that even with keep-alive, threads are returned to the pool after servicing each request. So, in this case threads are newed if needed.
>
> Sylvain
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ThreadLocal Memory Leak Protection - Executor threadRenewalDelay questions
Posted by Sylvain Laurent <sl...@apache.org>.
Hello,
On 26 oct. 2011, at 17:55, Rohit Kelapure wrote:
> * Reposting from the dev list as advised *
>
> Dear All,
>
> After going through the thread renewal code in
> /tomcat-8.0.x/java/org/apache/tomcat/util/threads/TaskQueue.java ,
> /tomcat-8.0.x/java/org/apache/tomcat/util/threads/ThreadPoolExecutor.java
> and the bug (Improve ThreadLocal memory leak clean-up)
> https://issues.apache.org/bugzilla/show_bug.cgi?id=49159 I had the
> following questions-
>
> 1. Once the thread pool has been renewed after No. of active threads
> in pool * max (threadKeepAliveTimeout, longestRequest+
> threadRenewalDelay) seconds, why does the ThreadPoolExecutor still
> keep paying the price of thread renewal. ?
> After org.apache.tomcat.util.threads.ThreadPoolExecutor.threadRenewalDelay
> is set to 1000L it never goes back to -1. Ideally once the all the
> threads in the threadpool is renewed we should revert the
> threadRenewalDelay back to -1 and NOT call
> ThreadPoolExecutor.currentThreadShouldBeStopped()/ThreadPoolExecutor.stopCurrentThreadIfNeeded()
> in TaskQueue.poll(long, TimeUnit) or TaskQueue.take().
> Is this because we are never quite sure as to when *all* of the
> threads in the pool have been renewed ?
Yes. threadRenewalDelay is a configuration parameter of the ThreadPoolExecutor and its value never changes at runtime (unless you play with JMX).
The thing is that it may take a long time to renew all the threads of the pool because there might be very long requests. Besides, a second renewal can be triggered (i.e. another webapp being stopped) while some threads have not been renewed yet after the first renewal was asked. By comparing the timestamps of when the thread was created and when the last renewal was triggered, we are sure that all threads are eventually renewed (provided they are returned to the pool).
>
> 2. Does the thread renewal approach scale under load (i.e. all threads
> in the pool busy servicing requests and CPU close to 90%) ? Is it
> meant for production deployments ?
Yes, it was meant for both development and production and is enabled by default.
When I was developing it I performed some load tests with JMeter to check that there is no functional impact while threads are being renewed, but I did not measure the performance impact of recreating new threads. Anyway, threadRenewalDelay is here to throttle the rate of renewal, so that not all threads are re-created at the same time, thus the performance impact should be negligible (and configurable).
>
> 3. How about threads that are servicing long running HTTP Keep-alive
> connections that never let the thread return to the pool. Are these
> threads renewed before the server is stopped.
With tomcat 6 (and the BIO connector, I don't know for others), threads were tied to the TCP connection as you describe, so that they were not returned to the pool if the connection was kept alive. But kee-palive is not eternal, the number of requests is limited and there's a timeout if the client does not send a new request for some time.
From tomcat 7 (with the BIO connector at least), threads are decoupled from TCP connections, so that even with keep-alive, threads are returned to the pool after servicing each request. So, in this case threads are newed if needed.
Sylvain
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org