Cassandra - Thread leak when high concurrent load

[JKnight JKnight <be...@gmail.com>]

Dear all,

My Cassandra server had thread leak when high concurrent load. I used
jconsole and saw many, many thread occur.

I knew Cassandra use TThreadPoolServer for handling request. And Cassandra
use DebuggableThreadPoolExecutor to handling command (read/write).

I want to know the reason of thread leak problem.

Anybody can help me?

Thank a lot for support.
-- 
Best regards,
JKnight

Re: Cassandra - Thread leak when high concurrent load

[Brandon Williams <dr...@gmail.com>]

On Sun, Apr 25, 2010 at 12:09 PM, JKnight JKnight <be...@gmail.com>wrote:

> Thanks Robson,
>
> The number of thread gradually increase to 7000. And the server hang up.
> I know threadpool is used to prevent creating large number of thread.
>
> So why Cassandra create large number of thread when high concurrent load.


The only thing I can surmise from this is that you are creating nearly 7000
client connections to Cassandra.  It is thread-per-connection, and a normal
amount of threads is in the low hundreds.

-Brandon

Re: Cassandra - Thread leak when high concurrent load

[JKnight JKnight <be...@gmail.com>]

Thanks Robson,

The number of thread gradually increase to 7000. And the server hang up.
I know threadpool is used to prevent creating large number of thread.

So why Cassandra create large number of thread when high concurrent load.

On Sun, Apr 25, 2010 at 5:38 PM, Mark Robson <ma...@gmail.com> wrote:

>
>
> On 25 April 2010 10:48, JKnight JKnight <be...@gmail.com> wrote:
>
>> Dear all,
>>
>> My Cassandra server had thread leak when high concurrent load. I used
>> jconsole and saw many, many thread occur.
>>
>
> Just because there are a lot of threads, need not imply a thread leak.
> Cassandra uses a lot of threads.
>
> Do you see the number of threads gradually increase during a soak test on
> your test cluster? Can you dump the JVM info (I believe sending a signal
> makes it dump this)?
>
> The JMX information viewable via JConsole also gives you a list of threads.
> Assuming you see this gradually increasing during the soak test, can you
> tellwhich thread pool is increasing?
>
> Mark
>



-- 
Best regards,
JKnight

Re: Cassandra - Thread leak when high concurrent load

[Mark Robson <ma...@gmail.com>]

On 25 April 2010 10:48, JKnight JKnight <be...@gmail.com> wrote:

> Dear all,
>
> My Cassandra server had thread leak when high concurrent load. I used
> jconsole and saw many, many thread occur.
>

Just because there are a lot of threads, need not imply a thread leak.
Cassandra uses a lot of threads.

Do you see the number of threads gradually increase during a soak test on
your test cluster? Can you dump the JVM info (I believe sending a signal
makes it dump this)?

The JMX information viewable via JConsole also gives you a list of threads.
Assuming you see this gradually increasing during the soak test, can you
tellwhich thread pool is increasing?

Mark