You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2016/11/01 05:32:48 UTC
Re: java.lang.OutOfMemoryError: unable to create new native thread
Hi Vadim,
Thank you so much this was a very useful command. This conversation is
going on here
https://www.mail-archive.com/user@spark.apache.org/msg58656.html
or you can just google "
why spark driver program is creating so many threads? How can I limit this
number?
<https://www.mail-archive.com/search?l=user@spark.apache.org&q=subject:%22Re%5C%3A+why+spark+driver+program+is+creating+so+many+threads%5C%3F+How+can+I+limit+this+number%5C%3F%22&o=newest>
"
please take a look if you are interested.
Thanks a lot!
On Mon, Oct 31, 2016 at 8:14 AM, Vadim Semenov <va...@datadoghq.com>
wrote:
> Have you tried to get number of threads in a running process using `cat
> /proc/<pid>/status` ?
>
> On Sun, Oct 30, 2016 at 11:04 PM, kant kodali <ka...@gmail.com> wrote:
>
>> yes I did run ps -ef | grep "app_name" and it is root.
>>
>>
>>
>> On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <ch...@indetail.co.jp>
>> wrote:
>>
>>> sorry, the UID
>>>
>>> On 10/31/16 11:59 AM, Chan Chor Pang wrote:
>>>
>>> actually if the max user processes is not the problem, i have no idea
>>>
>>> but i still suspecting the user,
>>> as the user who run spark-submit is not necessary the pid for the JVM
>>> process
>>>
>>> can u make sure when you "ps -ef | grep {your app id} " the PID is root?
>>> On 10/31/16 11:21 AM, kant kodali wrote:
>>>
>>> The java process is run by the root and it has the same config
>>>
>>> sudo -i
>>>
>>> ulimit -a
>>>
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size (blocks, -f) unlimited
>>> pending signals (-i) 120242
>>> max locked memory (kbytes, -l) 64
>>> max memory size (kbytes, -m) unlimited
>>> open files (-n) 1024
>>> pipe size (512 bytes, -p) 8
>>> POSIX message queues (bytes, -q) 819200
>>> real-time priority (-r) 0
>>> stack size (kbytes, -s) 8192
>>> cpu time (seconds, -t) unlimited
>>> max user processes (-u) 120242
>>> virtual memory (kbytes, -v) unlimited
>>> file locks (-x) unlimited
>>>
>>>
>>>
>>> On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <ch...@indetail.co.jp>
>>> wrote:
>>>
>>>> I have the same Exception before and the problem fix after i change the
>>>> nproc conf.
>>>>
>>>> > max user processes (-u) 120242
>>>> ↑this config does looks good.
>>>> are u sure the user who run ulimit -a is the same user who run the Java
>>>> process?
>>>> depend on how u submit the job and your setting, spark job may execute
>>>> by other user.
>>>>
>>>>
>>>> On 10/31/16 10:38 AM, kant kodali wrote:
>>>>
>>>> when I did this
>>>>
>>>> cat /proc/sys/kernel/pid_max
>>>>
>>>> I got 32768
>>>>
>>>> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <ka...@gmail.com>
>>>> wrote:
>>>>
>>>>> I believe for ubuntu it is unlimited but I am not 100% sure (I just
>>>>> read somewhere online). I ran ulimit -a and this is what I get
>>>>>
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 120242
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 1024
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 8192
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 120242
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang <
>>>>> chin-sh@indetail.co.jp> wrote:
>>>>>
>>>>>> not sure for ubuntu, but i think you can just create the file by
>>>>>> yourself
>>>>>> the syntax will be the same as /etc/security/limits.conf
>>>>>>
>>>>>> nproc.conf not only limit java process but all process by the same
>>>>>> user
>>>>>> so even the jvm process does nothing, if the corresponding user is
>>>>>> busy in other way
>>>>>> the jvm process will still not able to create new thread.
>>>>>>
>>>>>> btw the default limit for centos is 1024
>>>>>>
>>>>>>
>>>>>> On 10/31/16 9:51 AM, kant kodali wrote:
>>>>>>
>>>>>>
>>>>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang <
>>>>>> chin-sh@indetail.co.jp> wrote:
>>>>>>
>>>>>>> /etc/security/limits.d/90-nproc.conf
>>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using Ubuntu 16.04 LTS. I have this directory
>>>>>> /etc/security/limits.d/ but I don't have any files underneath it. This
>>>>>> error happens after running for 4 to 5 hours. I wonder if this is a GC
>>>>>> issue? And I am thinking if I should use CMS. I have also posted this on SO
>>>>>> since I havent got much response for this question
>>>>>> http://stackoverflow.com/questions/40315589/dag-sch
>>>>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat
>>>>>> e-new-native
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> kant
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---*------------------------------------------------*---*---*---*---
>>>>>> 株式会社INDETAIL
>>>>>> ニアショア総合サービス事業本部
>>>>>> ゲームサービス事業部
>>>>>> 陳 楚鵬
>>>>>> E-mail :chin-sh@indetail.co.jp
>>>>>> URL : http://www.indetail.co.jp
>>>>>>
>>>>>> 【札幌本社／LABO／LABO2】
>>>>>> 〒060-0042
>>>>>> 札幌市中央区大通西9丁目3番地33
>>>>>> キタコーセンタービルディング
>>>>>> （札幌本社／LABO2：2階、LABO：9階）
>>>>>> TEL：011-206-9235 FAX：011-206-9236
>>>>>>
>>>>>> 【東京支店】
>>>>>> 〒108-0014
>>>>>> 東京都港区芝5丁目29番20号 クロスオフィス三田
>>>>>> TEL：03-6809-6502 FAX：03-6809-6504
>>>>>>
>>>>>> 【名古屋サテライト】
>>>>>> 〒460-0002
>>>>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD
>>>>>> TEL：052-971-0086
>>>>>>
>>>>>>
>>
>
Re: java.lang.OutOfMemoryError: unable to create new native thread
Posted by kant kodali <ka...@gmail.com>.
Here is a UI of my thread dump.
http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTYvMTEvMS8tLWpzdGFja19kdW1wX3dpbmRvd19pbnRlcnZhbF8xbWluX2JhdGNoX2ludGVydmFsXzFzLnR4dC0tNi0xNy00Ng==
On Mon, Oct 31, 2016 at 10:32 PM, kant kodali <ka...@gmail.com> wrote:
> Hi Vadim,
>
> Thank you so much this was a very useful command. This conversation is
> going on here
>
> https://www.mail-archive.com/user@spark.apache.org/msg58656.html
>
> or you can just google "
>
> why spark driver program is creating so many threads? How can I limit this
> number?
> <https://www.mail-archive.com/search?l=user@spark.apache.org&q=subject:%22Re%5C%3A+why+spark+driver+program+is+creating+so+many+threads%5C%3F+How+can+I+limit+this+number%5C%3F%22&o=newest>
> "
>
> please take a look if you are interested.
>
> Thanks a lot!
>
> On Mon, Oct 31, 2016 at 8:14 AM, Vadim Semenov <
> vadim.semenov@datadoghq.com> wrote:
>
>> Have you tried to get number of threads in a running process using `cat
>> /proc/<pid>/status` ?
>>
>> On Sun, Oct 30, 2016 at 11:04 PM, kant kodali <ka...@gmail.com> wrote:
>>
>>> yes I did run ps -ef | grep "app_name" and it is root.
>>>
>>>
>>>
>>> On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <ch...@indetail.co.jp>
>>> wrote:
>>>
>>>> sorry, the UID
>>>>
>>>> On 10/31/16 11:59 AM, Chan Chor Pang wrote:
>>>>
>>>> actually if the max user processes is not the problem, i have no idea
>>>>
>>>> but i still suspecting the user,
>>>> as the user who run spark-submit is not necessary the pid for the JVM
>>>> process
>>>>
>>>> can u make sure when you "ps -ef | grep {your app id} " the PID is root?
>>>> On 10/31/16 11:21 AM, kant kodali wrote:
>>>>
>>>> The java process is run by the root and it has the same config
>>>>
>>>> sudo -i
>>>>
>>>> ulimit -a
>>>>
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 120242
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 1024
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 120242
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>>
>>>>
>>>> On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <chin-sh@indetail.co.jp
>>>> > wrote:
>>>>
>>>>> I have the same Exception before and the problem fix after i change
>>>>> the nproc conf.
>>>>>
>>>>> > max user processes (-u) 120242
>>>>> ↑this config does looks good.
>>>>> are u sure the user who run ulimit -a is the same user who run the
>>>>> Java process?
>>>>> depend on how u submit the job and your setting, spark job may execute
>>>>> by other user.
>>>>>
>>>>>
>>>>> On 10/31/16 10:38 AM, kant kodali wrote:
>>>>>
>>>>> when I did this
>>>>>
>>>>> cat /proc/sys/kernel/pid_max
>>>>>
>>>>> I got 32768
>>>>>
>>>>> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <ka...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I believe for ubuntu it is unlimited but I am not 100% sure (I just
>>>>>> read somewhere online). I ran ulimit -a and this is what I get
>>>>>>
>>>>>> core file size (blocks, -c) 0
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 120242
>>>>>> max locked memory (kbytes, -l) 64
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 1024
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 8192
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 120242
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang <
>>>>>> chin-sh@indetail.co.jp> wrote:
>>>>>>
>>>>>>> not sure for ubuntu, but i think you can just create the file by
>>>>>>> yourself
>>>>>>> the syntax will be the same as /etc/security/limits.conf
>>>>>>>
>>>>>>> nproc.conf not only limit java process but all process by the same
>>>>>>> user
>>>>>>> so even the jvm process does nothing, if the corresponding user is
>>>>>>> busy in other way
>>>>>>> the jvm process will still not able to create new thread.
>>>>>>>
>>>>>>> btw the default limit for centos is 1024
>>>>>>>
>>>>>>>
>>>>>>> On 10/31/16 9:51 AM, kant kodali wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang <
>>>>>>> chin-sh@indetail.co.jp> wrote:
>>>>>>>
>>>>>>>> /etc/security/limits.d/90-nproc.conf
>>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am using Ubuntu 16.04 LTS. I have this directory
>>>>>>> /etc/security/limits.d/ but I don't have any files underneath it. This
>>>>>>> error happens after running for 4 to 5 hours. I wonder if this is a GC
>>>>>>> issue? And I am thinking if I should use CMS. I have also posted this on SO
>>>>>>> since I havent got much response for this question
>>>>>>> http://stackoverflow.com/questions/40315589/dag-sch
>>>>>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat
>>>>>>> e-new-native
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> kant
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ---*------------------------------------------------*---*---*---*---
>>>>>>> 株式会社INDETAIL
>>>>>>> ニアショア総合サービス事業本部
>>>>>>> ゲームサービス事業部
>>>>>>> 陳 楚鵬
>>>>>>> E-mail :chin-sh@indetail.co.jp
>>>>>>> URL : http://www.indetail.co.jp
>>>>>>>
>>>>>>> 【札幌本社／LABO／LABO2】
>>>>>>> 〒060-0042
>>>>>>> 札幌市中央区大通西9丁目3番地33
>>>>>>> キタコーセンタービルディング
>>>>>>> （札幌本社／LABO2：2階、LABO：9階）
>>>>>>> TEL：011-206-9235 FAX：011-206-9236
>>>>>>>
>>>>>>> 【東京支店】
>>>>>>> 〒108-0014
>>>>>>> 東京都港区芝5丁目29番20号 クロスオフィス三田
>>>>>>> TEL：03-6809-6502 FAX：03-6809-6504
>>>>>>>
>>>>>>> 【名古屋サテライト】
>>>>>>> 〒460-0002
>>>>>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD
>>>>>>> TEL：052-971-0086
>>>>>>>
>>>>>>>
>>>
>>
>