You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Alessandro Torrisi <al...@eurone.it> on 2006/01/11 00:04:08 UTC

[MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Hi ! Yesterday I put in stress my direct connect hub software on Mac  
Os X 10.3.4 Tiger with latest Java implementation.
Max fake users reached number is 2500. When I reached this number the  
application has thrown this exception:

Exception in thread "IoThreadPool-1" java.lang.OutOfMemoryError: Java  
heap space
Exception in thread "IoThreadPool-4" java.lang.OutOfMemoryError: Java  
heap space
com.mchange.v2.async.ThreadPoolAsynchronousRunner 
$DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency  
threads for unassigned pending tasks!
Exception in thread "IoThreadPool-3" java.lang.OutOfMemoryError: Java  
heap space
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap  
space
4212917 [SocketAcceptor-0] WARN  
org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected  
exception.
java.lang.OutOfMemoryError: Java heap space
Exception in thread "IoThreadPool-2" java.lang.OutOfMemoryError: Java  
heap space

What's this ? Is it directly caused by the Java heap space ? How can  
I avoid it ?
Thanks and best regards,

Alessandro Torrisi

http://www.alessandrotorrisi.it

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Alessandro Torrisi <al...@eurone.it>.
I probably do the simplest thing that can be done...I cast the input 
object into a ByteBuffer then decode it with my CP-1252 decoder
and then I call my dispatcher void dispatchAction on a "JxHubPipeline" 
which simply process an input string which represents the
protocol command itself.

====== SNIPPET 1 ======

   @Override
    public void messageReceived(IoSession session, Object obj) throws 
Exception {
        try {
            ByteBuffer buf = (ByteBuffer) obj;
            String msg = buf.getString(decoderCp1252);
            log.debug("Received message was " + msg);

            pipeline.dispatchAction(session, msg);

        } catch (Exception _e) {
            // msg is null
            _e.printStackTrace();
        }
    }


====== SNIPPET 2 ======
   if (message.startsWith(JxHubProtocol.DC_KEY.command())) {
           processKey(session, message);
           log.debug("*** DC_KEY " + message);
          ....
 
At the end message is read and processKey makes the dirty work.

Best regards,
Alex


fedechicco@gmail.com wrote:

> When you receve the chat messages do you make a copy of your 
> ByteBuffer for each user?
>
> ----- Original Message ----- From: "Alessandro Torrisi" 
> <al...@eurone.it>
> To: "Apache Directory Developers List" <de...@directory.apache.org>
> Sent: Wednesday, January 11, 2006 11:43 AM
> Subject: Re: [MINA] Apparent DEADLOCK in 
> com.mchange.v2.async.ThreadPoolAsynchronousRunner
>
>
>> Yes I'm using c3p0 with hibernate, so I think it's directly managed 
>> by Hibernate...
>> How can I do to free ByteBuffers much more as possible ?
>> With 2000 opened IoSessions I have about 312 Mb of ram busy.
>> When I receive chat messages I see memory occupation to grow...Is 
>> there some natural mina
>> activity I'm losing during reading/writing operations ?
>>
>> I switched from 10 threads to 5 threads yesterday, it seems to change 
>> very little.
>> But I don't have an optimal view of the performance cause Java 
>> implementation on
>> Mac Os X is very slow...
>>
>> The applications main behaviour is to communicate in "blocks", slow, 
>> then fast, then again slow,
>> it is not constant. How can be avoided this ?
>>
>> I know that probably performance it's not the main target of this 
>> mailing list, but if a network
>> application is not performant as you well know, it is not used if 
>> there are many alternatives.
>> I have to understand if it's a physical architectural limit of 
>> Java/Mina before seriously going on.
>>
>> However I can say my application has improved from the beginning ! 
>> Thanks too this mailing list !
>>
>> Bye, Alex
>>
>> Justin Rudd wrote:
>>
>>>>> Hi ! Yesterday I put in stress my direct connect hub software on Mac
>>>>> Os X 10.3.4 Tiger with latest Java implementation.
>>>>> Max fake users reached number is 2500. When I reached this number the
>>>>> application has thrown this exception:
>>>>>
>>>
>>> [snip]
>>>
>>>
>>>>> com.mchange.v2.async.ThreadPoolAsynchronousRunner
>>>>> $DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
>>>>> threads for unassigned pending tasks!
>>>>>
>>>
>>> Are you using c3p0?  I haven't done an exhaustive look through the
>>> source of 0.8.x, but that package prefix - com.mchange - looks like
>>> c3p0 to me.  If so, are you releasing all your connections?  Because
>>> c3p0 by default will use some background threads to clean things up. 
>>> It will spin them off to try and find connections it can use, create
>>> new connection for clients that are waiting for them, etc.  So if you
>>> aren't cleaning all your resources up, it will keep spinning off
>>> threads which will cause it to run out of memory.
>>>
>>> If you aren't using c3p0, what other client libraries are you using?
>>>
>>> -- 
>>> Justin Rudd
>>> "Get the facts first.  You can distort them later."
>>>   - Mark Twain
>>> http://seagecko.org/thoughts/
>>>
>>
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by fe...@gmail.com.
When you receve the chat messages do you make a copy of your ByteBuffer for 
each user?

----- Original Message ----- 
From: "Alessandro Torrisi" <al...@eurone.it>
To: "Apache Directory Developers List" <de...@directory.apache.org>
Sent: Wednesday, January 11, 2006 11:43 AM
Subject: Re: [MINA] Apparent DEADLOCK in 
com.mchange.v2.async.ThreadPoolAsynchronousRunner


> Yes I'm using c3p0 with hibernate, so I think it's directly managed by 
> Hibernate...
> How can I do to free ByteBuffers much more as possible ?
> With 2000 opened IoSessions I have about 312 Mb of ram busy.
> When I receive chat messages I see memory occupation to grow...Is there 
> some natural mina
> activity I'm losing during reading/writing operations ?
>
> I switched from 10 threads to 5 threads yesterday, it seems to change very 
> little.
> But I don't have an optimal view of the performance cause Java 
> implementation on
> Mac Os X is very slow...
>
> The applications main behaviour is to communicate in "blocks", slow, then 
> fast, then again slow,
> it is not constant. How can be avoided this ?
>
> I know that probably performance it's not the main target of this mailing 
> list, but if a network
> application is not performant as you well know, it is not used if there 
> are many alternatives.
> I have to understand if it's a physical architectural limit of Java/Mina 
> before seriously going on.
>
> However I can say my application has improved from the beginning ! Thanks 
> too this mailing list !
>
> Bye, Alex
>
> Justin Rudd wrote:
>
>>>>Hi ! Yesterday I put in stress my direct connect hub software on Mac
>>>>Os X 10.3.4 Tiger with latest Java implementation.
>>>>Max fake users reached number is 2500. When I reached this number the
>>>>application has thrown this exception:
>>>>
>>
>>[snip]
>>
>>
>>>>com.mchange.v2.async.ThreadPoolAsynchronousRunner
>>>>$DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
>>>>threads for unassigned pending tasks!
>>>>
>>
>>Are you using c3p0?  I haven't done an exhaustive look through the
>>source of 0.8.x, but that package prefix - com.mchange - looks like
>>c3p0 to me.  If so, are you releasing all your connections?  Because
>>c3p0 by default will use some background threads to clean things up. It 
>>will spin them off to try and find connections it can use, create
>>new connection for clients that are waiting for them, etc.  So if you
>>aren't cleaning all your resources up, it will keep spinning off
>>threads which will cause it to run out of memory.
>>
>>If you aren't using c3p0, what other client libraries are you using?
>>
>>--
>>Justin Rudd
>>"Get the facts first.  You can distort them later."
>>   - Mark Twain
>>http://seagecko.org/thoughts/
>>
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by fe...@gmail.com.
Ok, I'll create a new Issue in jira, it's the first time I do it, so I hope to do all right.
  ----- Original Message ----- 
  From: Trustin Lee 
  To: Apache Directory Developers List 
  Sent: Friday, January 13, 2006 1:10 PM
  Subject: Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner


  2006/1/13, fedechicco@gmail.com <fe...@gmail.com>:
    I have had a similar problems implementing broadcast, and I have modified the ByteBuffer of MINA 8.0 making a method of
    asReadOnly() that create a java.nio.ByteBuffer from the asReadOnly of the wrapped nio.buffer.

    The problem is that the readOnly will share the original byte[] or direct array, so I had hacked the mina bytebuffer just a little in this way:

    In the MINA byteBuffer.asReadOnly() I create a new mina DefaultByteBuffer container and get into that the nio.buffer copy and the reference to the original mina bytebuffer, and a boolean flag that says if the DefaultByteBuffer wrap a readOnly copy or a real writable nio.bytebuffer. I do an aquire() of the original MINA byteBuffer.
    So when the MINA ByteBuffer copy is released, the release method just look at the flag and release the original mina bytebuffer.
    The MINA bytebuffer copy _must_ be not pooled obviusly.

    In this way there is no need to make a copy of the data, but only a new nio.bytebuffer that use the same byte[] or direct array, and the original ByteBuffer will be released only when all the copies will be already written to the channel.

    If you'd like it I can modify the MINA ByteBuffer of the 9.1 in the same way, it shouldn't be hard.

    by Fed

  This is a great idea.  Why don't you post this to the JIRA?

  http://issues.apache.org/jira/browse/DIRMINA 

  We'll really appreciate it because it makes us really easy to track the changes and known issues.

  Thanks in advance,
  Trustin

  -- 
  what we call human nature is actually human habit
  --
  http://gleamynode.net/
  PGP Key ID: 0x854B996C

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Trustin Lee <tr...@gmail.com>.
2006/1/13, fedechicco@gmail.com <fe...@gmail.com>:
>
> I have had a similar problems implementing broadcast, and I have modified
> the ByteBuffer of MINA 8.0 making a method of
> asReadOnly() that create a java.nio.ByteBuffer from the asReadOnly of the
> wrapped nio.buffer.
>
> The problem is that the readOnly will share the original byte[] or direct
> array, so I had hacked the mina bytebuffer just a little in this way:
>
> In the MINA byteBuffer.asReadOnly() I create a new mina DefaultByteBuffer
> container and get into that the nio.buffer copy and the reference to the
> original mina bytebuffer, and a boolean flag that says if the
> DefaultByteBuffer wrap a readOnly copy or a real writable nio.bytebuffer.
> I do an aquire() of the original MINA byteBuffer.
> So when the MINA ByteBuffer copy is released, the release method just look
> at the flag and release the original mina bytebuffer.
> The MINA bytebuffer copy _must_ be not pooled obviusly.
>
> In this way there is no need to make a copy of the data, but only a new
> nio.bytebuffer that use the same byte[] or direct array, and the original
> ByteBuffer will be released only when all the copies will be already written
> to the channel.
>
> If you'd like it I can modify the MINA ByteBuffer of the 9.1 in the same
> way, it shouldn't be hard.
>
> by Fed
>

This is a great idea.  Why don't you post this to the JIRA?

http://issues.apache.org/jira/browse/DIRMINA

We'll really appreciate it because it makes us really easy to track the
changes and known issues.

Thanks in advance,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Emmanuel Lecharny <el...@gmail.com>.
How can it be connected to Directory ? The initial problem, back in
2006, was related to MINA, which was then a subprojet to Directory.

If you have a problem with MINA, just post to the MINA mailing list.

>From the stack trace you are posting, I don't think this is even a
MINA poblem, so please post to the mailing list which is associated
with the tools you ar eusong.

Thanks !


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by rohita <ro...@411web.com>.
Hi All,

I am having the same problem of APPARENT DEADLOCK
I am very new to Java.
I am Hibernate3 and c3p0
Data base i am using is MySQL.

Please help me out as i am not able to find out which settings i have to
change to remove this deadlock,
Whenever this deadlock comes site goes down and because of this i am loosing
traffic from site

Here is the stack trace for my error

09:29:04,924  WARN ThreadPoolAsynchronousRunner:608 -
com.mchange.v2.async.ThreadPool
AsynchronousRunner$DeadlockDetector@2f922e72 -- APPARENT DEADLOCK!!!
Creating emergen
cy threads for unassigned pending tasks!
09:29:04,926  WARN ThreadPoolAsynchronousRunner:624 -
com.mchange.v2.async.ThreadPool
AsynchronousRunner$DeadlockDetector@2f922e72 -- APPARENT DEADLOCK!!!
Complete Status:
 
        Managed Threads: 3
        Active Threads: 3
        Active Tasks: 
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTa
sk@b3249a2 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTa
sk@1280306c
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTa
sk@1deb0bf1
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
        Pending Tasks: 
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTa
sk@325d0a8f
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTa
sk@7a604c1d
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@482542af
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@63d142a
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@13f6d499
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@31ff930c
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@20f1279
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2b85c6fd
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5bcb225d
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@753d36bf
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5d571bb
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1daefb
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1ccae0cc
               
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4b6c7fd8

Thanks
Rohit



Alessandro Torrisi wrote:
> 
> I want to refer to a standard library, so if the patch becomes official 
> it's better ! I understood this patch improves performance...
> I wait for your moves before modify it myself.
> 
> Bye,
> Alex
> 
> fedechicco@gmail.com wrote:
> 
>> I have had a similar problems implementing broadcast, and I have 
>> modified the ByteBuffer of MINA 8.0 making a method of
>> asReadOnly() that create a java.nio.ByteBuffer from the asReadOnly of 
>> the wrapped nio.buffer.
>>  
>> The problem is that the readOnly will share the original byte[] or 
>> direct array, so I had hacked the mina bytebuffer just a little in 
>> this way:
>>  
>> In the MINA byteBuffer.asReadOnly() I create a new mina 
>> DefaultByteBuffer container and get into that the nio.buffer copy and 
>> the reference to the original mina bytebuffer, and a boolean flag that 
>> says if the DefaultByteBuffer wrap a readOnly copy or a real writable 
>> nio.bytebuffer. I do an aquire() of the original MINA byteBuffer.
>> So when the MINA ByteBuffer copy is released, the release method just 
>> look at the flag and release the original mina bytebuffer.
>> The MINA bytebuffer copy _must_ be not pooled obviusly.
>>  
>> In this way there is no need to make a copy of the data, but only a 
>> new nio.bytebuffer that use the same byte[] or direct array, and the 
>> original ByteBuffer will be released only when all the copies will be 
>> already written to the channel.
>>  
>> If you'd like it I can modify the MINA ByteBuffer of the 9.1 in the 
>> same way, it shouldn't be hard.
>>  
>> by Fed
>>
>>     ----- Original Message -----
>>     *From:* Trustin Lee <ma...@gmail.com>
>>     *To:* Apache Directory Developers List
>>     <ma...@directory.apache.org>
>>     *Sent:* Friday, January 13, 2006 1:21 AM
>>     *Subject:* Re: [MINA] Apparent DEADLOCK in
>>     com.mchange.v2.async.ThreadPoolAsynchronousRunner
>>
>>     Hi Alex,
>>
>>     2006/1/12, Alessandro Torrisi <alessandro.torrisi@eurone.it
>>     <ma...@eurone.it>>:
>>
>>         I use a buffer size of 2048 bytes...not so much
>>
>>         Can you explain me this better ? Can I do something to improve
>>         or it's simply a fact ?
>>
>>
>>     Could you try to turn off the 'pooled' property of the buffers
>>     when you broadcast:
>>
>>     buf.setPooled(false);
>>
>>     Please let me know if this helps.  And please make sure you increased
>>     the direct buffer size as we suggested in the FAQ section.
>>
>>     HTH,
>>     Trustin
>>     -- 
>>     what we call human nature is actually human habit
>>     --
>>     http://gleamynode.net/
>>     PGP Key ID: 0x854B996C 
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/-MINA--Apparent-DEADLOCK-in-com.mchange.v2.async.ThreadPoolAsynchronousRunner-tp2310251p20627848.html
Sent from the Apache Directory Project mailing list archive at Nabble.com.

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Alessandro Torrisi <al...@eurone.it>.
I want to refer to a standard library, so if the patch becomes official 
it's better ! I understood this patch improves performance...
I wait for your moves before modify it myself.

Bye,
Alex

fedechicco@gmail.com wrote:

> I have had a similar problems implementing broadcast, and I have 
> modified the ByteBuffer of MINA 8.0 making a method of
> asReadOnly() that create a java.nio.ByteBuffer from the asReadOnly of 
> the wrapped nio.buffer.
>  
> The problem is that the readOnly will share the original byte[] or 
> direct array, so I had hacked the mina bytebuffer just a little in 
> this way:
>  
> In the MINA byteBuffer.asReadOnly() I create a new mina 
> DefaultByteBuffer container and get into that the nio.buffer copy and 
> the reference to the original mina bytebuffer, and a boolean flag that 
> says if the DefaultByteBuffer wrap a readOnly copy or a real writable 
> nio.bytebuffer. I do an aquire() of the original MINA byteBuffer.
> So when the MINA ByteBuffer copy is released, the release method just 
> look at the flag and release the original mina bytebuffer.
> The MINA bytebuffer copy _must_ be not pooled obviusly.
>  
> In this way there is no need to make a copy of the data, but only a 
> new nio.bytebuffer that use the same byte[] or direct array, and the 
> original ByteBuffer will be released only when all the copies will be 
> already written to the channel.
>  
> If you'd like it I can modify the MINA ByteBuffer of the 9.1 in the 
> same way, it shouldn't be hard.
>  
> by Fed
>
>     ----- Original Message -----
>     *From:* Trustin Lee <ma...@gmail.com>
>     *To:* Apache Directory Developers List
>     <ma...@directory.apache.org>
>     *Sent:* Friday, January 13, 2006 1:21 AM
>     *Subject:* Re: [MINA] Apparent DEADLOCK in
>     com.mchange.v2.async.ThreadPoolAsynchronousRunner
>
>     Hi Alex,
>
>     2006/1/12, Alessandro Torrisi <alessandro.torrisi@eurone.it
>     <ma...@eurone.it>>:
>
>         I use a buffer size of 2048 bytes...not so much
>
>         Can you explain me this better ? Can I do something to improve
>         or it's simply a fact ?
>
>
>     Could you try to turn off the 'pooled' property of the buffers
>     when you broadcast:
>
>     buf.setPooled(false);
>
>     Please let me know if this helps.  And please make sure you increased
>     the direct buffer size as we suggested in the FAQ section.
>
>     HTH,
>     Trustin
>     -- 
>     what we call human nature is actually human habit
>     --
>     http://gleamynode.net/
>     PGP Key ID: 0x854B996C 
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by fe...@gmail.com.
I have had a similar problems implementing broadcast, and I have modified the ByteBuffer of MINA 8.0 making a method of
asReadOnly() that create a java.nio.ByteBuffer from the asReadOnly of the wrapped nio.buffer.

The problem is that the readOnly will share the original byte[] or direct array, so I had hacked the mina bytebuffer just a little in this way:

In the MINA byteBuffer.asReadOnly() I create a new mina DefaultByteBuffer container and get into that the nio.buffer copy and the reference to the original mina bytebuffer, and a boolean flag that says if the DefaultByteBuffer wrap a readOnly copy or a real writable nio.bytebuffer. I do an aquire() of the original MINA byteBuffer.
So when the MINA ByteBuffer copy is released, the release method just look at the flag and release the original mina bytebuffer.
The MINA bytebuffer copy _must_ be not pooled obviusly.

In this way there is no need to make a copy of the data, but only a new nio.bytebuffer that use the same byte[] or direct array, and the original ByteBuffer will be released only when all the copies will be already written to the channel.

If you'd like it I can modify the MINA ByteBuffer of the 9.1 in the same way, it shouldn't be hard.

by Fed
  ----- Original Message ----- 
  From: Trustin Lee 
  To: Apache Directory Developers List 
  Sent: Friday, January 13, 2006 1:21 AM
  Subject: Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner


  Hi Alex,


  2006/1/12, Alessandro Torrisi <al...@eurone.it>:
    I use a buffer size of 2048 bytes...not so much


    Can you explain me this better ? Can I do something to improve or it's simply a fact ?

  Could you try to turn off the 'pooled' property of the buffers when you broadcast: 

  buf.setPooled(false);

  Please let me know if this helps.  And please make sure you increased
  the direct buffer size as we suggested in the FAQ section.

  HTH,
  Trustin
  -- 
  what we call human nature is actually human habit 
  --
  http://gleamynode.net/
  PGP Key ID: 0x854B996C

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Trustin Lee <tr...@gmail.com>.
Hi Alex,

2006/1/12, Alessandro Torrisi <al...@eurone.it>:
>
> I use a buffer size of 2048 bytes...not so much
>
> Can you explain me this better ? Can I do something to improve or it's
> simply a fact ?
>

Could you try to turn off the 'pooled' property of the buffers when you
broadcast:

buf.setPooled(false);

Please let me know if this helps.  And please make sure you increased
the direct buffer size as we suggested in the FAQ section.

HTH,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Alessandro Torrisi <al...@eurone.it>.
I use a buffer size of 2048 bytes...not so much

Can you explain me this better ? Can I do something to improve or  
it's simply a fact ?

On 11/gen/06, at 18:18, Kaspar Luethi wrote:

> i think memory occupation grows when *sending* the chat messages
> back to the users. even if you send a single message, if you send
> them to 2000 users, the single message will get copied into 2000
> mina byte buffers. that's why i still miss ByteBuffer.duplicate() in
> mina. it was probably made just for that kind of situation.

Best regards,
Alex

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Kaspar Luethi <ka...@humantools.ch>.
>How can I do to free ByteBuffers much more as possible ?
>With 2000 opened IoSessions I have about 312 Mb of ram busy.

312/2000 = 156 KB/user, not too much. java is very memory hungry.

i only have small bytebuffers (<4k) and with 500 users, the server
consumes about 80MB (linux).

i also tested for memory leaks with a profiler, and i must confess
there were many, mostly improperly released JDBC resources.

there is the danger in java-garbage-collection, that the programmer
relies too much on the runtime system to clean up the memory.

developing long-running application needs you to take an extra care
for that kind of problems. another example on my suse linux testserver,
MySQL closes all connections after 8 hours.


>When I receive chat messages I see memory occupation to grow...Is
>there some natural mina
>activity I'm losing during reading/writing operations ?

i think memory occupation grows when *sending* the chat messages
back to the users. even if you send a single message, if you send
them to 2000 users, the single message will get copied into 2000
mina byte buffers. that's why i still miss ByteBuffer.duplicate() in
mina. it was probably made just for that kind of situation.

mina-experts, please correct me if i am wrong about the copying.

kaspar
-- 
---------------------------------------------
  http://www.humantools.com

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Alessandro Torrisi <al...@eurone.it>.
Yes I'm using c3p0 with hibernate, so I think it's directly managed by 
Hibernate...
How can I do to free ByteBuffers much more as possible ?
With 2000 opened IoSessions I have about 312 Mb of ram busy.
When I receive chat messages I see memory occupation to grow...Is there 
some natural mina
activity I'm losing during reading/writing operations ?

I switched from 10 threads to 5 threads yesterday, it seems to change 
very little.
But I don't have an optimal view of the performance cause Java 
implementation on
Mac Os X is very slow...

The applications main behaviour is to communicate in "blocks", slow, 
then fast, then again slow,
it is not constant. How can be avoided this ?

I know that probably performance it's not the main target of this 
mailing list, but if a network
application is not performant as you well know, it is not used if there 
are many alternatives.
I have to understand if it's a physical architectural limit of Java/Mina 
before seriously going on.

However I can say my application has improved from the beginning ! 
Thanks too this mailing list !

Bye, Alex

Justin Rudd wrote:

>>>Hi ! Yesterday I put in stress my direct connect hub software on Mac
>>>Os X 10.3.4 Tiger with latest Java implementation.
>>>Max fake users reached number is 2500. When I reached this number the
>>>application has thrown this exception:
>>>      
>>>
>
>[snip]
>
>  
>
>>>com.mchange.v2.async.ThreadPoolAsynchronousRunner
>>>$DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
>>>threads for unassigned pending tasks!
>>>      
>>>
>
>Are you using c3p0?  I haven't done an exhaustive look through the
>source of 0.8.x, but that package prefix - com.mchange - looks like
>c3p0 to me.  If so, are you releasing all your connections?  Because
>c3p0 by default will use some background threads to clean things up. 
>It will spin them off to try and find connections it can use, create
>new connection for clients that are waiting for them, etc.  So if you
>aren't cleaning all your resources up, it will keep spinning off
>threads which will cause it to run out of memory.
>
>If you aren't using c3p0, what other client libraries are you using?
>
>--
>Justin Rudd
>"Get the facts first.  You can distort them later."
>   - Mark Twain
>http://seagecko.org/thoughts/
>  
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Justin Rudd <ju...@gmail.com>.
> > Hi ! Yesterday I put in stress my direct connect hub software on Mac
> > Os X 10.3.4 Tiger with latest Java implementation.
> > Max fake users reached number is 2500. When I reached this number the
> > application has thrown this exception:

[snip]

> > com.mchange.v2.async.ThreadPoolAsynchronousRunner
> > $DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
> > threads for unassigned pending tasks!

Are you using c3p0?  I haven't done an exhaustive look through the
source of 0.8.x, but that package prefix - com.mchange - looks like
c3p0 to me.  If so, are you releasing all your connections?  Because
c3p0 by default will use some background threads to clean things up. 
It will spin them off to try and find connections it can use, create
new connection for clients that are waiting for them, etc.  So if you
aren't cleaning all your resources up, it will keep spinning off
threads which will cause it to run out of memory.

If you aren't using c3p0, what other client libraries are you using?

--
Justin Rudd
"Get the facts first.  You can distort them later."
   - Mark Twain
http://seagecko.org/thoughts/

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Emmanuel Lecharny <el...@gmail.com>.
hey,

have you tried to tune the JVM?

like :
- increasing the maximum memory with -Xmx=512m
- decreasing the stack size with -Xss=64k or lower ?

Is it a Sun jvm? 1.5.0_06 ? (may be 1.5.0_05 on Mac OSX)

-- Emmanuel Lécharny

Alessandro Torrisi a écrit :

> Hi ! Yesterday I put in stress my direct connect hub software on Mac  
> Os X 10.3.4 Tiger with latest Java implementation.
> Max fake users reached number is 2500. When I reached this number the  
> application has thrown this exception:
>
> Exception in thread "IoThreadPool-1" java.lang.OutOfMemoryError: Java  
> heap space
> Exception in thread "IoThreadPool-4" java.lang.OutOfMemoryError: Java  
> heap space
> com.mchange.v2.async.ThreadPoolAsynchronousRunner 
> $DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency  
> threads for unassigned pending tasks!
> Exception in thread "IoThreadPool-3" java.lang.OutOfMemoryError: Java  
> heap space
> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap  
> space
> 4212917 [SocketAcceptor-0] WARN  
> org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected  
> exception.
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "IoThreadPool-2" java.lang.OutOfMemoryError: Java  
> heap space
>
> What's this ? Is it directly caused by the Java heap space ? How can  
> I avoid it ?
> Thanks and best regards,
>
> Alessandro Torrisi
>
> http://www.alessandrotorrisi.it
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Emmanuel Lecharny <el...@gmail.com>.
Alessandro Torrisi a écrit :

> I used top to monitor application, but it's too expensive:


Just try with vmstat. It's less expensive, I think

>
> with 1581 users I have 312 Mb in memory and a cpu usage between the  
> 6% and the 20%
> as average.
>
that's not too much. Do you still have the pb with 2500 users ?

--Emmanuel Lécharny

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Alessandro Torrisi <al...@eurone.it>.
Now I'm using -Xms32M and -Xmx1024M and it looks better, but code  
could be optimized.
If someone wants too look the code or access cvs tell me please.

I used top to monitor application, but it's too expensive:

with 1581 users I have 312 Mb in memory and a cpu usage between the  
6% and the 20%
as average.

My buffers are of 2048 bytes, are enough ? Is it too much ?

Bye, Alex :D


On 11/gen/06, at 00:29, Kaspar Luethi wrote:

> hi alessandro
>
> 2500 users sounds good. were they also sending actual data and
> it being broacasted to all other users?
>
> i think this is a memory problem and not a deadlock case.
> deadlocks can occurr easily, when objects cannot be created
> due to memory shortage and somecode is waiting for it...
>
> with 2500 users you likely also have more than 2500 mina ByteBuffers
> in memory, eventually a multiple of that. now it depends how big those
> buffers are...2KB? 20MB?! maybe your mac just has not enough RAM :)
>
> so you should watch the memory consumption while stress testing.
> do you have something like the "top" command on OS X? it helps too
> for watching the cpu load.
>
> try to give your JVM more memory. at first you could try the -server
> switch, which does allow the JVM to allocate more memory by default
> along with some other differences in the JRE and garbage collection.
> i don't know if apple's java implementation has this option.
>
> for the sun JVM there is a list of options (many for solaris only) at
> http://java.sun.com/docs/hotspot/VMOptions.html
>
> kaspar
>
>
> Alessandro Torrisi wrote:
>> Hi ! Yesterday I put in stress my direct connect hub software on Mac
>> Os X 10.3.4 Tiger with latest Java implementation.
>> Max fake users reached number is 2500. When I reached this number the
>> application has thrown this exception:
>>
>> Exception in thread "IoThreadPool-1" java.lang.OutOfMemoryError: Java
>> heap space
>> Exception in thread "IoThreadPool-4" java.lang.OutOfMemoryError: Java
>> heap space
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner
>> $DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
>> threads for unassigned pending tasks!
>> Exception in thread "IoThreadPool-3" java.lang.OutOfMemoryError: Java
>> heap space
>> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap
>> space
>> 4212917 [SocketAcceptor-0] WARN
>> org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected
>> exception.
>> java.lang.OutOfMemoryError: Java heap space
>> Exception in thread "IoThreadPool-2" java.lang.OutOfMemoryError: Java
>> heap space
>>
>> What's this ? Is it directly caused by the Java heap space ? How can
>> I avoid it ?
>> Thanks and best regards,
>>
>> Alessandro Torrisi
>>
>> http://www.alessandrotorrisi.it
>>
>
>

Re: [MINA] Apparent DEADLOCK in com.mchange.v2.async.ThreadPoolAsynchronousRunner

Posted by Kaspar Luethi <ka...@humantools.com>.
hi alessandro

2500 users sounds good. were they also sending actual data and
it being broacasted to all other users?

i think this is a memory problem and not a deadlock case.
deadlocks can occurr easily, when objects cannot be created
due to memory shortage and somecode is waiting for it...

with 2500 users you likely also have more than 2500 mina ByteBuffers
in memory, eventually a multiple of that. now it depends how big those
buffers are...2KB? 20MB?! maybe your mac just has not enough RAM :)

so you should watch the memory consumption while stress testing.
do you have something like the "top" command on OS X? it helps too
for watching the cpu load.

try to give your JVM more memory. at first you could try the -server
switch, which does allow the JVM to allocate more memory by default
along with some other differences in the JRE and garbage collection.
i don't know if apple's java implementation has this option.

for the sun JVM there is a list of options (many for solaris only) at
http://java.sun.com/docs/hotspot/VMOptions.html

kaspar


Alessandro Torrisi wrote:
> Hi ! Yesterday I put in stress my direct connect hub software on Mac
> Os X 10.3.4 Tiger with latest Java implementation.
> Max fake users reached number is 2500. When I reached this number the
> application has thrown this exception:
>
> Exception in thread "IoThreadPool-1" java.lang.OutOfMemoryError: Java
> heap space
> Exception in thread "IoThreadPool-4" java.lang.OutOfMemoryError: Java
> heap space
> com.mchange.v2.async.ThreadPoolAsynchronousRunner
> $DeadlockDetector@27d98b -- APPARENT DEADLOCK!!! Creating emergency
> threads for unassigned pending tasks!
> Exception in thread "IoThreadPool-3" java.lang.OutOfMemoryError: Java
> heap space
> Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap
> space
> 4212917 [SocketAcceptor-0] WARN
> org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected
> exception.
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "IoThreadPool-2" java.lang.OutOfMemoryError: Java
> heap space
>
> What's this ? Is it directly caused by the Java heap space ? How can
> I avoid it ?
> Thanks and best regards,
>
> Alessandro Torrisi
>
> http://www.alessandrotorrisi.it
>