You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Dan Checkoway <dc...@gmail.com> on 2011/09/20 20:29:27 UTC
Executor thread lifecycle
I'm curious how long Executor threads live prior to being
recycled/replaced/what have you. Is there a way to control this lifecycle?
I'm using ThreadLocal to allocate some resources per thread...and the
behavior I'm seeing is that after some period of time, my ThreadLocal<Thing>
goes away and needs to be reinitialized. It's implying to me that the
thread itself has been recycled.
What's making this more confusing is that the thread IDs appear to be
getting reused, so logging the thread ID isn't helping (it's probably
confusing me more than anything).
Anyway, can somebody confirm if Tomcat has explicit deliberate control over
the Executor thread lifecycle, and if so, how I might be able to control
it? And if not, is there a way to control it?
Thanks,
Dan
Re: Executor thread lifecycle
Posted by Dan Checkoway <dc...@gmail.com>.
If commons-pool supported bulk borrow/return in a single blocking operation,
I'd be all over it. But the fact that if I want to borrow 3000 objects from
the pool, it's 3000 blocking ops, that's a non-starter.
Unless I'm missing something? If there's a "minimally blocking" pool impl
that supports bulk borrow/return, shoot it my way...thx.
On Tue, Sep 20, 2011 at 3:11 PM, Mark Thomas <ma...@apache.org> wrote:
> On 20/09/2011 20:06, Dan Checkoway wrote:
> > To be specific, we're having a major issue with garbage generation. To
> > avoid this, we're pre-allocating pools of instances (up until now
> > ThreadLocal, but that's obviously changing). When a request is processed
> it
> > may need anywhere between 1 and N objects from the pool, which it grabs
> and
> > uses, and then returns them to the pool when finished. If N > pool size,
> > new instances get allocated on the fly, and they get added to the pool as
> > well.
> >
> > Sounds like a big memory leak in the making, right? Yeah, except there's
> a
> > rational cap on the total # of instances each thread will ever need. So
> > this ended up being dramatically more efficient because it generated
> > essentially zero garbage over time. GC could keep up again...
> >
> > Until this proved to be its own garbage generator. :-) As threads got
> > recycled, the ThreadLocal pools got turned to garbage. Foiled again!
> :-)
> >
> > So yeah, I'm going to centralize the pool and just synchronize management
> of
> > it. Oy.
>
> You might want to take a look at Commons Pool trunk. It has been
> completely re-written for 2.0 for performance. There is still some work
> to do before a release but the GenericObjectPool should be OK and
> performs well in a multi-threaded environment (the
> GenericKeyedObjectPool has a known bug I need to fix).
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Executor thread lifecycle
Posted by Mark Thomas <ma...@apache.org>.
On 20/09/2011 20:06, Dan Checkoway wrote:
> To be specific, we're having a major issue with garbage generation. To
> avoid this, we're pre-allocating pools of instances (up until now
> ThreadLocal, but that's obviously changing). When a request is processed it
> may need anywhere between 1 and N objects from the pool, which it grabs and
> uses, and then returns them to the pool when finished. If N > pool size,
> new instances get allocated on the fly, and they get added to the pool as
> well.
>
> Sounds like a big memory leak in the making, right? Yeah, except there's a
> rational cap on the total # of instances each thread will ever need. So
> this ended up being dramatically more efficient because it generated
> essentially zero garbage over time. GC could keep up again...
>
> Until this proved to be its own garbage generator. :-) As threads got
> recycled, the ThreadLocal pools got turned to garbage. Foiled again! :-)
>
> So yeah, I'm going to centralize the pool and just synchronize management of
> it. Oy.
You might want to take a look at Commons Pool trunk. It has been
completely re-written for 2.0 for performance. There is still some work
to do before a release but the GenericObjectPool should be OK and
performs well in a multi-threaded environment (the
GenericKeyedObjectPool has a known bug I need to fix).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Executor thread lifecycle
Posted by Dan Checkoway <dc...@gmail.com>.
To be specific, we're having a major issue with garbage generation. To
avoid this, we're pre-allocating pools of instances (up until now
ThreadLocal, but that's obviously changing). When a request is processed it
may need anywhere between 1 and N objects from the pool, which it grabs and
uses, and then returns them to the pool when finished. If N > pool size,
new instances get allocated on the fly, and they get added to the pool as
well.
Sounds like a big memory leak in the making, right? Yeah, except there's a
rational cap on the total # of instances each thread will ever need. So
this ended up being dramatically more efficient because it generated
essentially zero garbage over time. GC could keep up again...
Until this proved to be its own garbage generator. :-) As threads got
recycled, the ThreadLocal pools got turned to garbage. Foiled again! :-)
So yeah, I'm going to centralize the pool and just synchronize management of
it. Oy.
On Tue, Sep 20, 2011 at 3:02 PM, Mark Thomas <ma...@apache.org> wrote:
> On 20/09/2011 19:59, Dan Checkoway wrote:
> > Thanks for the quick reply, Mark. Much appreciated. I was afraid that
> was
> > the case.
> >
> > What I was trying to accomplish was...data that's not request/response
> > specific in any way, that survives requests. I was using ThreadLocal so
> as
> > to make it non-blocking, i.e. I didn't want to centralize management of
> this
> > stuff and have to synchronize access to it across my hundreds of executor
> > threads.
> >
> > Sounds like I'll need to bite the bullet on that. Oh well... :-)
>
> I assume it changes rarely (and you would have had to deal with that
> even if using ThreadLocals and had the same concurrency issues). A
> read/write lock should do the trick if reads >> writes.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Executor thread lifecycle
Posted by Mark Thomas <ma...@apache.org>.
On 20/09/2011 19:59, Dan Checkoway wrote:
> Thanks for the quick reply, Mark. Much appreciated. I was afraid that was
> the case.
>
> What I was trying to accomplish was...data that's not request/response
> specific in any way, that survives requests. I was using ThreadLocal so as
> to make it non-blocking, i.e. I didn't want to centralize management of this
> stuff and have to synchronize access to it across my hundreds of executor
> threads.
>
> Sounds like I'll need to bite the bullet on that. Oh well... :-)
I assume it changes rarely (and you would have had to deal with that
even if using ThreadLocals and had the same concurrency issues). A
read/write lock should do the trick if reads >> writes.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Executor thread lifecycle
Posted by Dan Checkoway <dc...@gmail.com>.
Thanks for the quick reply, Mark. Much appreciated. I was afraid that was
the case.
What I was trying to accomplish was...data that's not request/response
specific in any way, that survives requests. I was using ThreadLocal so as
to make it non-blocking, i.e. I didn't want to centralize management of this
stuff and have to synchronize access to it across my hundreds of executor
threads.
Sounds like I'll need to bite the bullet on that. Oh well... :-)
On Tue, Sep 20, 2011 at 2:55 PM, Mark Thomas <ma...@apache.org> wrote:
> On 20/09/2011 19:29, Dan Checkoway wrote:
> > I'm curious how long Executor threads live prior to being
> > recycled/replaced/what have you. Is there a way to control this
> lifecycle?
>
> No.
>
> > I'm using ThreadLocal to allocate some resources per thread...
>
> That is asking for a memory leak. There is a simple rule for correctly
> using a ThreadLocal in a web application.
>
> The ThreadLocal must be removed from the thread before the
> request/response in which it was created completes processing.
>
> > and the
> > behavior I'm seeing is that after some period of time, my
> ThreadLocal<Thing>
> > goes away and needs to be reinitialized. It's implying to me that the
> > thread itself has been recycled.
>
> More recent versions of Tomcat will re-cycle threads and/or remove
> ThreadLocals detected when an application is reloaded as part of the
> memory leak prevention code.
>
> > What's making this more confusing is that the thread IDs appear to be
> > getting reused, so logging the thread ID isn't helping (it's probably
> > confusing me more than anything).
> >
> > Anyway, can somebody confirm if Tomcat has explicit deliberate control
> over
> > the Executor thread lifecycle, and if so, how I might be able to control
> > it? And if not, is there a way to control it?
>
> ThreadLocals are likely to be the wrong solution to the problem. If you
> need a resource across multiple, unrelated requests, the ServletContext
> is probably the way to go.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Executor thread lifecycle
Posted by Mark Thomas <ma...@apache.org>.
On 20/09/2011 19:29, Dan Checkoway wrote:
> I'm curious how long Executor threads live prior to being
> recycled/replaced/what have you. Is there a way to control this lifecycle?
No.
> I'm using ThreadLocal to allocate some resources per thread...
That is asking for a memory leak. There is a simple rule for correctly
using a ThreadLocal in a web application.
The ThreadLocal must be removed from the thread before the
request/response in which it was created completes processing.
> and the
> behavior I'm seeing is that after some period of time, my ThreadLocal<Thing>
> goes away and needs to be reinitialized. It's implying to me that the
> thread itself has been recycled.
More recent versions of Tomcat will re-cycle threads and/or remove
ThreadLocals detected when an application is reloaded as part of the
memory leak prevention code.
> What's making this more confusing is that the thread IDs appear to be
> getting reused, so logging the thread ID isn't helping (it's probably
> confusing me more than anything).
>
> Anyway, can somebody confirm if Tomcat has explicit deliberate control over
> the Executor thread lifecycle, and if so, how I might be able to control
> it? And if not, is there a way to control it?
ThreadLocals are likely to be the wrong solution to the problem. If you
need a resource across multiple, unrelated requests, the ServletContext
is probably the way to go.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Executor thread lifecycle
Posted by Dan Checkoway <dc...@gmail.com>.
Chris,
I do have one other webapp deployed alongside, but neither of them ever gets
reloaded. We always do a full stop/start of tomcat to roll out new builds
(which is about the only time we ever stop these apps).
There's absolutely no performance penalty that I'm aware of. It was just
the yet-unexplained recycling of threads.
FWIW, we've got the pool based solution going now, so I'm off the whole
ThreadLocal bandwagon (really really small wagon). This is uber low prio
from my standpoint, in case you guys want to forget I ever brought it up.
:-)
Thanks,
Dan
On Wed, Sep 21, 2011 at 9:46 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dan,
>
> On 9/20/2011 4:47 PM, Dan Checkoway wrote:
> > Thanks Chris. Those threads are *never* idle in this app. :-)
> > They're still getting recycled periodically, it looks like, despite
> > lack of idle time.
>
> Hmm, that does sound weird. Do you have other webapps also deployed
> alongside this one? If you reload those webapps, the threads in the
> thread pool will be recycled after a webapp undeployment to flush-out
> any nasty stuff that might be tied to them. You can try setting the
> "threadRenewalDelay" to a negative number to disable this particular
> kind of recycling.
>
> > Does that make sense or am I on crack?
>
> You might still be on crack.
>
> Are you observing a performance penalty for this, or are you just
> trying to explain an as-yet-unexplained recycling of threads?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk556q0ACgkQ9CaO5/Lv0PADgwCfYqXFN9SBljy1LbEMUw2wEHpZ
> 7NsAoK/kACgQm3tx+k9Uy0Xa9XyWIjEn
> =g84G
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Executor thread lifecycle
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dan,
On 9/20/2011 4:47 PM, Dan Checkoway wrote:
> Thanks Chris. Those threads are *never* idle in this app. :-)
> They're still getting recycled periodically, it looks like, despite
> lack of idle time.
Hmm, that does sound weird. Do you have other webapps also deployed
alongside this one? If you reload those webapps, the threads in the
thread pool will be recycled after a webapp undeployment to flush-out
any nasty stuff that might be tied to them. You can try setting the
"threadRenewalDelay" to a negative number to disable this particular
kind of recycling.
> Does that make sense or am I on crack?
You might still be on crack.
Are you observing a performance penalty for this, or are you just
trying to explain an as-yet-unexplained recycling of threads?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk556q0ACgkQ9CaO5/Lv0PADgwCfYqXFN9SBljy1LbEMUw2wEHpZ
7NsAoK/kACgQm3tx+k9Uy0Xa9XyWIjEn
=g84G
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Executor thread lifecycle
Posted by Dan Checkoway <dc...@gmail.com>.
Thanks Chris. Those threads are *never* idle in this app. :-) They're
still getting recycled periodically, it looks like, despite lack of idle
time. Does that make sense or am I on crack?
On Tue, Sep 20, 2011 at 4:38 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dan,
>
> On 9/20/2011 2:29 PM, Dan Checkoway wrote:
> > I'm curious how long Executor threads live prior to being
> > recycled/replaced/what have you. Is there a way to control this
> > lifecycle?
>
> You can set maxIdleTime to something other than the default (which is
> 1 minute). I think that would do it. But you have to weigh the
> memory/performance benefit of idle threads versus your thread-based
> caching system.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk54+bYACgkQ9CaO5/Lv0PC+SACbB96YLw4douQmXPGw4Js0+2qN
> OVkAn21tNDayxCm/1335VZvpxlwjbrBS
> =I50V
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Executor thread lifecycle
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dan,
On 9/20/2011 2:29 PM, Dan Checkoway wrote:
> I'm curious how long Executor threads live prior to being
> recycled/replaced/what have you. Is there a way to control this
> lifecycle?
You can set maxIdleTime to something other than the default (which is
1 minute). I think that would do it. But you have to weigh the
memory/performance benefit of idle threads versus your thread-based
caching system.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk54+bYACgkQ9CaO5/Lv0PC+SACbB96YLw4douQmXPGw4Js0+2qN
OVkAn21tNDayxCm/1335VZvpxlwjbrBS
=I50V
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org