You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jordan Zimmerman <jo...@jordanzimmerman.com> on 2012/07/09 04:29:29 UTC
OOM/no more threads?
Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=59462] throws
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.run(Thread.java:636)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.start(Thread.java:614)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.start0(Native Method)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: java.lang.OutOfMemoryError: unable to create new native thread
RE: OOM/no more threads?
Posted by Rakesh R <ra...@huawei.com>.
1) Probably good to relook the Linux thread stack size. Use “ulimit -s <size in kb>” for setting slightly bigger value and observe the test case.
2) Check also your ulimit on virtual size, i.e. ulimit -v. Increasing this limit should let you create more threads with the same stack size.
3) Also, lowering -Xmx can help by leaving more memory for thread stacks.
All these are like tuning options. Still would be good to observe the jstack once again in long running instance and will see the possibilities of leaks as Camille suggested?
-Rakesh
________________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com]
Sent: Monday, July 09, 2012 12:57 PM
To: user@zookeeper.apache.org
Subject: Re: OOM/no more threads?
-Xms35004M -Xmx35004M -Xss128k
64 bit Centos
On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:
> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss parameters? OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [jordan@jordanzimmerman.com]
> Sent: Monday, July 09, 2012 8:43 AM
> To: user@zookeeper.apache.org
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> jordan@jordanzimmerman.com> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> jordan@jordanzimmerman.com> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr=
>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
Re: OOM/no more threads?
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
-Xms35004M -Xmx35004M -Xss128k
64 bit Centos
On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:
> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss parameters? OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [jordan@jordanzimmerman.com]
> Sent: Monday, July 09, 2012 8:43 AM
> To: user@zookeeper.apache.org
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> jordan@jordanzimmerman.com> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> jordan@jordanzimmerman.com> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr=
>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
RE: OOM/no more threads?
Posted by Rakesh R <ra...@huawei.com>.
Hi Jordan,
BTW, could you give details of -Xmx, -Xss parameters? OS ? 32bit/62bit system?
As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
Also, there could be chances of thread leak in the system.
-Rakesh
________________________________________
From: Jordan Zimmerman [jordan@jordanzimmerman.com]
Sent: Monday, July 09, 2012 8:43 AM
To: user@zookeeper.apache.org
Subject: Re: OOM/no more threads?
Good idea - I'll try that.
-JZ
On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
> What does a jstack on another long-running instance show? Anything out of
> the norm?
>
> C
>
> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> I'm not sure which of the instances went down, but it could be a few weeks.
>>
>> -JZ
>>
>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>
>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>> was
>>> the last time you restarted these instances?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>> It's not really out of memory. It's "unable to create new native
>> thread".
>>>> It appears to have to do with JMX (notice the RMI).
>>>>
>>>> -JZ
>>>>
>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>
>>>>> Not known specifically that I know of. Can you give us more details?
>> Did
>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>> tell you?
>>>>>
>>>>> C
>>>>>
>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>> jordan@jordanzimmerman.com> wrote:
>>>>>
>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>> Accept-0: accept loop for ServerSocket[addr=
>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>> 11:01:28
>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start0(Native Method)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: OOM/no more threads?
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Good idea - I'll try that.
-JZ
On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
> What does a jstack on another long-running instance show? Anything out of
> the norm?
>
> C
>
> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> I'm not sure which of the instances went down, but it could be a few weeks.
>>
>> -JZ
>>
>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>
>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>> was
>>> the last time you restarted these instances?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>> It's not really out of memory. It's "unable to create new native
>> thread".
>>>> It appears to have to do with JMX (notice the RMI).
>>>>
>>>> -JZ
>>>>
>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>
>>>>> Not known specifically that I know of. Can you give us more details?
>> Did
>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>> tell you?
>>>>>
>>>>> C
>>>>>
>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>> jordan@jordanzimmerman.com> wrote:
>>>>>
>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>> Accept-0: accept loop for ServerSocket[addr=
>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>> 11:01:28
>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start0(Native Method)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: OOM/no more threads?
Posted by Camille Fournier <ca...@apache.org>.
What does a jstack on another long-running instance show? Anything out of
the norm?
C
On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> I'm not sure which of the instances went down, but it could be a few weeks.
>
> -JZ
>
> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>
> > Yeah a tickle in the back of my mind says that this is a JVM bug. When
> was
> > the last time you restarted these instances?
> >
> > C
> >
> > On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> >
> >> It's not really out of memory. It's "unable to create new native
> thread".
> >> It appears to have to do with JMX (notice the RMI).
> >>
> >> -JZ
> >>
> >> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
> >>
> >>> Not known specifically that I know of. Can you give us more details?
> Did
> >>> you actually run out of memory? File descriptors? What does a heap dump
> >>> tell you?
> >>>
> >>> C
> >>>
> >>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> >>> jordan@jordanzimmerman.com> wrote:
> >>>
> >>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
> >>>>
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> >>>> Accept-0: accept loop for ServerSocket[addr=
> >>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
> >> 11:01:28
> >>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.run(Thread.java:636)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.start(Thread.java:614)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.start0(Native Method)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> >>>> java.lang.OutOfMemoryError: unable to create new native thread
> >>>>
> >>>>
> >>
> >>
>
>
Re: OOM/no more threads?
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I'm not sure which of the instances went down, but it could be a few weeks.
-JZ
On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
> Yeah a tickle in the back of my mind says that this is a JVM bug. When was
> the last time you restarted these instances?
>
> C
>
> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> It's not really out of memory. It's "unable to create new native thread".
>> It appears to have to do with JMX (notice the RMI).
>>
>> -JZ
>>
>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>
>>> Not known specifically that I know of. Can you give us more details? Did
>>> you actually run out of memory? File descriptors? What does a heap dump
>>> tell you?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>> Accept-0: accept loop for ServerSocket[addr=
>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>> 11:01:28
>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.run(Thread.java:636)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.start(Thread.java:614)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.start0(Native Method)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>
>>>>
>>
>>
Re: OOM/no more threads?
Posted by Camille Fournier <ca...@apache.org>.
Yeah a tickle in the back of my mind says that this is a JVM bug. When was
the last time you restarted these instances?
C
On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> It's not really out of memory. It's "unable to create new native thread".
> It appears to have to do with JMX (notice the RMI).
>
> -JZ
>
> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>
> > Not known specifically that I know of. Can you give us more details? Did
> > you actually run out of memory? File descriptors? What does a heap dump
> > tell you?
> >
> > C
> >
> > On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> >
> >> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
> >>
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> >> Accept-0: accept loop for ServerSocket[addr=
> >> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
> 11:01:28
> >> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.run(Thread.java:636)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.start(Thread.java:614)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.start0(Native Method)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> >> java.lang.OutOfMemoryError: unable to create new native thread
> >>
> >>
>
>
Re: OOM/no more threads?
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
It's not really out of memory. It's "unable to create new native thread". It appears to have to do with JMX (notice the RMI).
-JZ
On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
> Not known specifically that I know of. Can you give us more details? Did
> you actually run out of memory? File descriptors? What does a heap dump
> tell you?
>
> C
>
> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>> Accept-0: accept loop for ServerSocket[addr=
>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28
>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.run(Thread.java:636)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.start(Thread.java:614)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.start0(Native Method)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>>
Re: OOM/no more threads?
Posted by Camille Fournier <ca...@apache.org>.
Not known specifically that I know of. Can you give us more details? Did
you actually run out of memory? File descriptors? What does a heap dump
tell you?
C
On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> Accept-0: accept loop for ServerSocket[addr=
> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28
> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.run(Thread.java:636)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.start(Thread.java:614)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.start0(Native Method)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> java.lang.OutOfMemoryError: unable to create new native thread
>
>