You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geronimo.apache.org by Jeremy Boynes <jb...@apache.org> on 2005/06/02 15:45:47 UTC
CORBA eating threads
The CORBA implementation consumes 9 threads from the pool which with
default settings can cause the EJB transport to hang.
What is it using these threads for? Is there some bug which prevents
them being returned to the pool, or are these permanent threads it needs
and if so shouldn't it be allocating them directly?
--
Jeremy
Re: CORBA eating threads
Posted by Aaron Mulder <am...@alumni.princeton.edu>.
I'm not sure about allocating them directly. From my point of
view, I'd rather see all the thread pools through Geronimo, even if at the
present time we don't allow you to tweak them meaningfully. It's much
nicer to see a fixed pool of 9 or 10 for CORBA than to see nothing and
know it's just doing something under the covers.
Aaron
On Thu, 2 Jun 2005, Jeremy Boynes wrote:
> The CORBA implementation consumes 9 threads from the pool which with
> default settings can cause the EJB transport to hang.
>
> What is it using these threads for? Is there some bug which prevents
> them being returned to the pool, or are these permanent threads it needs
> and if so shouldn't it be allocating them directly?
>
> --
> Jeremy
>
Re: CORBA eating threads
Posted by Dain Sundstrom <da...@iq80.com>.
On Jun 2, 2005, at 9:43 AM, Dain Sundstrom wrote:
>> How do we set it up so that if the pool is undersized then we get
>> an error rather than just have the transport hang?
>>
>
> I'm looking into this one. The default for the current thread pool
> is to have the requesting thread run the task, which bad for our
> case since the requesting thread is the socket acceptor. I think
> we should instead set abortWhenBlocked on the thread pool so we get
> an exception. It would be nice if there were a wait 10sec option
> to deal with floods, but I don't see any options for that.
The problem was in the ThreadPool class we were using a work queue in
addition to the thread pooling, so when we ran out of threads the
work was just queued forever. Normally a small fixed size buffer is
desireable, since it can even out connection bursts, but since we had
a total of one threads available any code that needs two threads to
work (like a test using a timer) would lock up. For now I have
turned off the work queue and set the pool to throw an exception if a
thread can not be immediately acquired. This will help us drive out
other bugs in the system. Of course later on when tuning the server,
the thread pool will be an obvious place to start.
-dain
Re: CORBA eating threads
Posted by Dain Sundstrom <da...@iq80.com>.
On Jun 2, 2005, at 8:59 AM, Jeremy Boynes wrote:
> Dain Sundstrom wrote:
>
>> On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:
>>
>>> The CORBA implementation consumes 9 threads from the pool which
>>> with default settings can cause the EJB transport to hang.
>>>
>>> What is it using these threads for? Is there some bug which
>>> prevents them being returned to the pool, or are these permanent
>>> threads it needs and if so shouldn't it be allocating them directly?
>>>
>> It is pretty simple. The each CSS Bean deployed acquires 2
>> threads, one for the name service orb and one to start the css
>> orb, and the CORBABean consumes one thread. We have 4 CSS beans
>> deployed and one CORBABean giving us a total thread usage of 9.
>> You can see the thread dump below.
>>
>
> Should all of these come from the same thread pool as the non-CORBA
> transport?
I believe that is a configuration/service organization discussion we
should have once certification is complete. The good thing is we
have the thread creation externalized so we can choose how we want to
handle this later.
> How does this scale, both with the number of EJBs deployed and the
> number of inbound requests?
You have two threads per CSS bean. I believe that a css bean is a
corba protocol configuration and we have 4 in the configuration you
are looking at. This is not a per EJB allocation. As for scaling
with inbound requests, I have no idea. I don't think this thread is
related to the inbound requests at all. I believe it is simply an
internal orb management thread used to control shutdown of the orb.
This is just a guess, Alan it the expert.
> With the current setup, what criteria should a user use to size the
> pool?
Since we have no reasonable way for a user to configure our server,
we just need to pick a good (large) default. We can deal with
helping our users tune the server post certification, and after we
get a way for users to configure the system.
> How do we set it up so that if the pool is undersized then we get
> an error rather than just have the transport hang?
I'm looking into this one. The default for the current thread pool
is to have the requesting thread run the task, which bad for our case
since the requesting thread is the socket acceptor. I think we
should instead set abortWhenBlocked on the thread pool so we get an
exception. It would be nice if there were a wait 10sec option to
deal with floods, but I don't see any options for that.
-dain
Re: CORBA eating threads
Posted by Jeremy Boynes <jb...@apache.org>.
Dain Sundstrom wrote:
> On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:
>
>> The CORBA implementation consumes 9 threads from the pool which with
>> default settings can cause the EJB transport to hang.
>>
>> What is it using these threads for? Is there some bug which prevents
>> them being returned to the pool, or are these permanent threads it
>> needs and if so shouldn't it be allocating them directly?
>
>
> It is pretty simple. The each CSS Bean deployed acquires 2 threads,
> one for the name service orb and one to start the css orb, and the
> CORBABean consumes one thread. We have 4 CSS beans deployed and one
> CORBABean giving us a total thread usage of 9. You can see the thread
> dump below.
>
Should all of these come from the same thread pool as the non-CORBA
transport?
How does this scale, both with the number of EJBs deployed and the
number of inbound requests?
With the current setup, what criteria should a user use to size the pool?
How do we set it up so that if the pool is undersized then we get an
error rather than just have the transport hang?
--
Jeremy
Re: CORBA eating threads
Posted by Dain Sundstrom <da...@iq80.com>.
On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:
> The CORBA implementation consumes 9 threads from the pool which
> with default settings can cause the EJB transport to hang.
>
> What is it using these threads for? Is there some bug which
> prevents them being returned to the pool, or are these permanent
> threads it needs and if so shouldn't it be allocating them directly?
It is pretty simple. The each CSS Bean deployed acquires 2 threads,
one for the name service orb and one to start the css orb, and the
CORBABean consumes one thread. We have 4 CSS beans deployed and one
CORBABean giving us a total thread usage of 9. You can see the
thread dump below.
-dain
DefaultThreadPool 8@1b prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():233, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 7@1c prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():222, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 6@1d prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():233, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 5@1e prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():222, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 4@1f prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():233, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 3@20 prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():222, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 2@21 prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():233, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 1@22 prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():222, CSSBean.java
run():-1
run():552, Thread.java
DefaultThreadPool 0@31 prio=5, in group "main", status: WAIT
wait():-1, Object.java
wait():429, Object.java
run():2521, ORB.java
run():158, CORBABean.java
run():-1
run():552, Thread.java