You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yatong Zhang <bl...@gmail.com> on 2014/09/16 02:55:35 UTC
hs_err_pid3013.log, out of memory?
Hi there,
I just encountered an error which left a log '/hs_err_pid3013.log'. So is
there a way to solve this?
#
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (malloc) failed to allocate 12288 bytes for
> committing reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # In 32 bit mode, the process size limit was hit
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Use 64 bit Java on a 64 bit OS
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2718), pid=3013, tid=140294995289856
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build
> 1.7.0_45-b18)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode
> linux-amd64 compressed oops)
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
>
> --------------- T H R E A D ---------------
>
> Current thread (0x00007f994ace8000): JavaThread "HANDSHAKE-/10.0.10.9"
> daemon [_thread_new, id=3274, stack(0x00007f98f957a000,0x00007f98f95bb000)]
>
> Stack: [0x00007f98f957a000,0x00007f98f95bb000], sp=0x00007f98f95b9800,
> free space=254k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0x992c8a] VMError::report_and_die()+0x2ea
> V [libjvm.so+0x49319b] report_vm_out_of_memory(char const*, int,
> unsigned long, char const*)+0x9b
> V [libjvm.so+0x81310e] os::Linux::commit_memory_impl(char*, unsigned
> long, bool)+0xfe
> V [libjvm.so+0x8131cc] os::pd_commit_memory(char*, unsigned long,
> bool)+0xc
> V [libjvm.so+0x80b3ba] os::commit_memory(char*, unsigned long, bool)+0x2a
> V [libjvm.so+0x8116ed] os::pd_create_stack_guard_pages(char*, unsigned
> long)+0x6d
> V [libjvm.so+0x94693e] JavaThread::create_stack_guard_pages()+0x5e
> V [libjvm.so+0x94d3d4] JavaThread::run()+0x34
> V [libjvm.so+0x815288] java_start(Thread*)+0x108
>
>
> --------------- P R O C E S S ---------------
>
Re: hs_err_pid3013.log, out of memory?
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang <bl...@gmail.com> wrote:
> @Chris Lohfink I have 16G memory per node, all the other settings are
> default
>
> @J. Ryan Earl I am not sure. I am using the default settings.
>
> But I've found out it might be because some settings in
> '/etc/sysctl.conf'. I am still testing it
>
If JNA is installed, it will try to memlock all of the JVM process. For
this to happen, you have to adjust the settings for the user you run
Cassandra as under /etc/security/limits.conf or limits.d/ and you have to
modifying kernel.shmmax in sysctl.conf accordingly. If you do not, and JNA
is installed, the memlock will fail with the error you gave.
Re: hs_err_pid3013.log, out of memory?
Posted by Chris Lohfink <cl...@blackbirdit.com>.
Can you post the output of "free" before starting cassandra? Is there anything in logs before it runs out of memory (ie in /var/log/cassandra/output.log)? If an exception is thrown with an OOM it might narrow down if theres a kernel resource (ie mmap file limit, file descriptor limit etc) that you have maxed out of instead of memory.
---
Chris Lohfink
On Sep 17, 2014, at 8:35 PM, Yatong Zhang <bl...@gmail.com> wrote:
> @Chris Lohfink I have 16G memory per node, all the other settings are default
>
> @J. Ryan Earl I am not sure. I am using the default settings.
>
> But I've found out it might be because some settings in '/etc/sysctl.conf'. I am still testing it
>
> On Wed, Sep 17, 2014 at 11:46 AM, J. Ryan Earl <os...@jryanearl.us> wrote:
> Are you using JNA? Did you adjust your memlock limit?
>
> On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <cl...@blackbirdit.com> wrote:
> How much memory does your system have? How much memory is system utilizing before starting Cassandra (use command free)? What are the heap setting it tries to use?
>
> Chris
>
> On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> It's during the startup. I tried to upgrade cassandra from 2.0.7 to 2.0.10, but looks like cassandra could not start again. Also I found the following log at '/var/log/messages':
>>
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H 2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0 4971 1 0x00000080
>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082 ffffffff81ed4ff0 ffff8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370 ffff88042b591ca0 ffff8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8 000000000000fbc8 ffff8803b4f01af8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ? wake_up_new_task+0xd9/0x130
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? do_fork+0x13d/0x480
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H 2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0 4972 1 0x00000080
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082 ffffffff81ed6d78 ffff8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370 ffff8803b4d7fca0 ffff8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8 000000000000fbc8 ffff8803b4cf1ab8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ? wake_up_state+0x10/0x20
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ? thread_return+0x4e/0x76e
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more than 120 seconds.
>>
>>
>> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote:
>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com> wrote:
>> I just encountered an error which left a log '/hs_err_pid3013.log'. So is there a way to solve this?
>>
>> # There is insufficient memory for the Java Runtime Environment to continue.
>> # Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.
>>
>> Use less heap memory?
>>
>> You haven't specified under which circumstances this occurred, so I can only conjecture that it is likely being caused by "writing too fast".
>>
>> Write more slowly.
>>
>> =Rob
>>
>>
>
>
>
Re: hs_err_pid3013.log, out of memory?
Posted by Yatong Zhang <bl...@gmail.com>.
@Chris Lohfink I have 16G memory per node, all the other settings are
default
@J. Ryan Earl I am not sure. I am using the default settings.
But I've found out it might be because some settings in '/etc/sysctl.conf'.
I am still testing it
On Wed, Sep 17, 2014 at 11:46 AM, J. Ryan Earl <os...@jryanearl.us> wrote:
> Are you using JNA? Did you adjust your memlock limit?
>
> On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <cl...@blackbirdit.com>
> wrote:
>
>> How much memory does your system have? How much memory is system
>> utilizing before starting Cassandra (use command free)? What are the heap
>> setting it tries to use?
>>
>> Chris
>>
>> On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bl...@gmail.com> wrote:
>>
>> It's during the startup. I tried to upgrade cassandra from 2.0.7 to
>> 2.0.10, but looks like cassandra could not start again. Also I found the
>> following log at '/var/log/messages':
>>
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more
>>> than 120 seconds.
>>> Sep 16 09:06:59 storage6 kernel: Tainted: G
>>> --------------- H 2.6.32-431.el6.x86_64 #1
>>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0
>>> 4971 1 0x00000080
>>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082
>>> ffffffff81ed4ff0 ffff8803b4f01540
>>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370
>>> ffff88042b591ca0 ffff8803b4f01540
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8
>>> 000000000000fbc8 ffff8803b4f01af8
>>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>>> exit_robust_list+0x90/0x160
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>>> do_group_exit+0x58/0xd0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>>> get_signal_to_deliver+0x1f6/0x460
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>]
>>> do_signal+0x75/0x800
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ?
>>> wake_up_new_task+0xd9/0x130
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ?
>>> do_fork+0x13d/0x480
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>>> sys_futex+0x7b/0x170
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>>> do_notify_resume+0x90/0xc0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>]
>>> int_signal+0x12/0x17
>>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more
>>> than 120 seconds.
>>> Sep 16 09:06:59 storage6 kernel: Tainted: G
>>> --------------- H 2.6.32-431.el6.x86_64 #1
>>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0
>>> 4972 1 0x00000080
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082
>>> ffffffff81ed6d78 ffff8803b4cf1500
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370
>>> ffff8803b4d7fca0 ffff8803b4cf1500
>>> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8
>>> 000000000000fbc8 ffff8803b4cf1ab8
>>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>>> exit_robust_list+0x90/0x160
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ?
>>> wake_up_state+0x10/0x20
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>>> do_group_exit+0x58/0xd0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>>> get_signal_to_deliver+0x1f6/0x460
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>]
>>> do_signal+0x75/0x800
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ?
>>> __switch_to+0x1ac/0x320
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ?
>>> thread_return+0x4e/0x76e
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>>> sys_futex+0x7b/0x170
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>>> do_notify_resume+0x90/0xc0
>>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>]
>>> int_signal+0x12/0x17
>>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more
>>> than 120 seconds.
>>>
>>
>>
>> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com>
>> wrote:
>>
>>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com>
>>> wrote:
>>>
>>>> I just encountered an error which left a log '/hs_err_pid3013.log'. So
>>>> is there a way to solve this?
>>>>
>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>> continue.
>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>>>> committing reserved memory.
>>>>>
>>>>
>>> Use less heap memory?
>>>
>>> You haven't specified under which circumstances this occurred, so I can
>>> only conjecture that it is likely being caused by "writing too fast".
>>>
>>> Write more slowly.
>>>
>>> =Rob
>>>
>>>
>>
>>
>
Re: hs_err_pid3013.log, out of memory?
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
Are you using JNA? Did you adjust your memlock limit?
On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink <cl...@blackbirdit.com>
wrote:
> How much memory does your system have? How much memory is system utilizing
> before starting Cassandra (use command free)? What are the heap setting it
> tries to use?
>
> Chris
>
> On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
> It's during the startup. I tried to upgrade cassandra from 2.0.7 to
> 2.0.10, but looks like cassandra could not start again. Also I found the
> following log at '/var/log/messages':
>
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more
>> than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel: Tainted: G
>> --------------- H 2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0
>> 4971 1 0x00000080
>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082
>> ffffffff81ed4ff0 ffff8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370
>> ffff88042b591ca0 ffff8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8
>> 000000000000fbc8 ffff8803b4f01af8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>> exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>> do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>> get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ?
>> wake_up_new_task+0xd9/0x130
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ?
>> do_fork+0x13d/0x480
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>> sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>> do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more
>> than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel: Tainted: G
>> --------------- H 2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0
>> 4972 1 0x00000080
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082
>> ffffffff81ed6d78 ffff8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370
>> ffff8803b4d7fca0 ffff8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8
>> 000000000000fbc8 ffff8803b4cf1ab8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
>> exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ?
>> wake_up_state+0x10/0x20
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
>> do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
>> get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ?
>> __switch_to+0x1ac/0x320
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ?
>> thread_return+0x4e/0x76e
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
>> sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
>> do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more
>> than 120 seconds.
>>
>
>
> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com>
>> wrote:
>>
>>> I just encountered an error which left a log '/hs_err_pid3013.log'. So
>>> is there a way to solve this?
>>>
>>> # There is insufficient memory for the Java Runtime Environment to
>>>> continue.
>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>>> committing reserved memory.
>>>>
>>>
>> Use less heap memory?
>>
>> You haven't specified under which circumstances this occurred, so I can
>> only conjecture that it is likely being caused by "writing too fast".
>>
>> Write more slowly.
>>
>> =Rob
>>
>>
>
>
Re: hs_err_pid3013.log, out of memory?
Posted by Chris Lohfink <cl...@blackbirdit.com>.
How much memory does your system have? How much memory is system utilizing before starting Cassandra (use command free)? What are the heap setting it tries to use?
Chris
On Sep 15, 2014, at 8:16 PM, Yatong Zhang <bl...@gmail.com> wrote:
> It's during the startup. I tried to upgrade cassandra from 2.0.7 to 2.0.10, but looks like cassandra could not start again. Also I found the following log at '/var/log/messages':
>
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more than 120 seconds.
> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H 2.6.32-431.el6.x86_64 #1
> Sep 16 09:06:59 storage6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0 4971 1 0x00000080
> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082 ffffffff81ed4ff0 ffff8803b4f01540
> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370 ffff88042b591ca0 ffff8803b4f01540
> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8 000000000000fbc8 ffff8803b4f01af8
> Sep 16 09:06:59 storage6 kernel: Call Trace:
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? exit_robust_list+0x90/0x160
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] get_signal_to_deliver+0x1f6/0x460
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ? wake_up_new_task+0xd9/0x130
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? do_fork+0x13d/0x480
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] do_notify_resume+0x90/0xc0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more than 120 seconds.
> Sep 16 09:06:59 storage6 kernel: Tainted: G --------------- H 2.6.32-431.el6.x86_64 #1
> Sep 16 09:06:59 storage6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0 4972 1 0x00000080
> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082 ffffffff81ed6d78 ffff8803b4cf1500
> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370 ffff8803b4d7fca0 ffff8803b4cf1500
> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8 000000000000fbc8 ffff8803b4cf1ab8
> Sep 16 09:06:59 storage6 kernel: Call Trace:
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ? exit_robust_list+0x90/0x160
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ? wake_up_state+0x10/0x20
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>] get_signal_to_deliver+0x1f6/0x460
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ? thread_return+0x4e/0x76e
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ? sys_futex+0x7b/0x170
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>] do_notify_resume+0x90/0xc0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more than 120 seconds.
>
>
> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com> wrote:
> I just encountered an error which left a log '/hs_err_pid3013.log'. So is there a way to solve this?
>
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.
>
> Use less heap memory?
>
> You haven't specified under which circumstances this occurred, so I can only conjecture that it is likely being caused by "writing too fast".
>
> Write more slowly.
>
> =Rob
>
>
Re: hs_err_pid3013.log, out of memory?
Posted by Yatong Zhang <bl...@gmail.com>.
It's during the startup. I tried to upgrade cassandra from 2.0.7 to 2.0.10,
but looks like cassandra could not start again. Also I found the following
log at '/var/log/messages':
Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more than
> 120 seconds.
> Sep 16 09:06:59 storage6 kernel: Tainted: G ---------------
> H 2.6.32-431.el6.x86_64 #1
> Sep 16 09:06:59 storage6 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 16 09:06:59 storage6 kernel: java D 0000000000000003 0
> 4971 1 0x00000080
> Sep 16 09:06:59 storage6 kernel: ffff88042b591c98 0000000000000082
> ffffffff81ed4ff0 ffff8803b4f01540
> Sep 16 09:06:59 storage6 kernel: ffff88042b591c68 ffffffff810af370
> ffff88042b591ca0 ffff8803b4f01540
> Sep 16 09:06:59 storage6 kernel: ffff8803b4f01af8 ffff88042b591fd8
> 000000000000fbc8 ffff8803b4f01af8
> Sep 16 09:06:59 storage6 kernel: Call Trace:
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
> exit_robust_list+0x90/0x160
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
> do_group_exit+0x58/0xd0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
> get_signal_to_deliver+0x1f6/0x460
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81066629>] ?
> wake_up_new_task+0xd9/0x130
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81070ead>] ? do_fork+0x13d/0x480
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
> sys_futex+0x7b/0x170
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
> do_notify_resume+0x90/0xc0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more
> than 120 seconds.
> Sep 16 09:06:59 storage6 kernel: Tainted: G ---------------
> H 2.6.32-431.el6.x86_64 #1
> Sep 16 09:06:59 storage6 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 16 09:06:59 storage6 kernel: java D 0000000000000000 0
> 4972 1 0x00000080
> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc98 0000000000000082
> ffffffff81ed6d78 ffff8803b4cf1500
> Sep 16 09:06:59 storage6 kernel: ffff8803b4d7fc68 ffffffff810af370
> ffff8803b4d7fca0 ffff8803b4cf1500
> Sep 16 09:06:59 storage6 kernel: ffff8803b4cf1ab8 ffff8803b4d7ffd8
> 000000000000fbc8 ffff8803b4cf1ab8
> Sep 16 09:06:59 storage6 kernel: Call Trace:
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810af370>] ?
> exit_robust_list+0x90/0x160
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076ad5>] exit_mm+0x95/0x180
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81076f1f>] do_exit+0x15f/0x870
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81065e20>] ?
> wake_up_state+0x10/0x20
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81077688>]
> do_group_exit+0x58/0xd0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8108d046>]
> get_signal_to_deliver+0x1f6/0x460
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100a265>] do_signal+0x75/0x800
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810097cc>] ?
> __switch_to+0x1ac/0x320
> Sep 16 09:06:59 storage6 kernel: [<ffffffff81527910>] ?
> thread_return+0x4e/0x76e
> Sep 16 09:06:59 storage6 kernel: [<ffffffff810b1c0b>] ?
> sys_futex+0x7b/0x170
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100aa80>]
> do_notify_resume+0x90/0xc0
> Sep 16 09:06:59 storage6 kernel: [<ffffffff8100b341>] int_signal+0x12/0x17
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more
> than 120 seconds.
>
On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com> wrote:
>
>> I just encountered an error which left a log '/hs_err_pid3013.log'. So is
>> there a way to solve this?
>>
>> # There is insufficient memory for the Java Runtime Environment to
>>> continue.
>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>> committing reserved memory.
>>>
>>
> Use less heap memory?
>
> You haven't specified under which circumstances this occurred, so I can
> only conjecture that it is likely being caused by "writing too fast".
>
> Write more slowly.
>
> =Rob
>
>
Re: hs_err_pid3013.log, out of memory?
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang <bl...@gmail.com> wrote:
> I just encountered an error which left a log '/hs_err_pid3013.log'. So is
> there a way to solve this?
>
> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>> committing reserved memory.
>>
>
Use less heap memory?
You haven't specified under which circumstances this occurred, so I can
only conjecture that it is likely being caused by "writing too fast".
Write more slowly.
=Rob