You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Abhishek Kumar Maheshwari <Ab...@timesinternet.in> on 2017/03/22 05:45:06 UTC
[Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
Thanks Jayesh,
I found the fix for the same.
I make below changes :
In /etc/sysctl.conf I make below change:
vm.max_map_count = 1048575
in the /etc/security/limits.d file:
root - memlock unlimited
root - nofile 100000
root - nproc 32768
root - as unlimited
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: Thursday, March 23, 2017 8:36 PM
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>; user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Dmesg will often print a message saying that it had to kill a process if the server was short of memory, so you will have to dump the output to a file and check.
If a process is killed to reclaim memory for the system, then it will dump a list of all processes and the actual process that was killed.
So maybe, you can check for a kill like this - "dmesg | grep -i kill"
If you do find a line (or two), then you need to examine the output carefully.
In production, I tend to dump a lot of GC output also which helps troubleshooting.
E.g. Below is what I have.
If you look, I also have a flag that says that if the heap runs out of memory (which is rare), then dump files.
If dmesg does not show your processes being killed, then you may have to dump gc logging info to get some insight.
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms16G
-Xmx16G
-Xmn4800M
-XX:+HeapDumpOnOutOfMemoryError
-Xss256k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseParNewGC
-XX:MaxTenuringThreshold=2
-XX:SurvivorRatio=8
-XX:+UnlockDiagnosticVMOptions
-XX:ParGCCardsPerStrideChunk=32768
-XX:NewSize=750m
-XX:MaxNewSize=750m
-XX:+UseCondCardMark
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-Xloggc:<some-path-for-gc-logs>
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1M
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=<true|false>
-Dcom.sun.management.jmxremote.authenticate=<true|false>
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 5:18 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
JVM config is as below:
-Xms16G
-Xmx16G
-Xmn3000M
What I need to check in dmesg?
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 23 March 2017 03:39
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.
Jayesh
________________________________
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
No only Cassandra is running on these servers.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
Re: [Cassandra 3.0.9] Cannot allocate memory
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Dmesg will often print a message saying that it had to kill a process if the server was short of memory, so you will have to dump the output to a file and check.
If a process is killed to reclaim memory for the system, then it will dump a list of all processes and the actual process that was killed.
So maybe, you can check for a kill like this - "dmesg | grep -i kill"
If you do find a line (or two), then you need to examine the output carefully.
In production, I tend to dump a lot of GC output also which helps troubleshooting.
E.g. Below is what I have.
If you look, I also have a flag that says that if the heap runs out of memory (which is rare), then dump files.
If dmesg does not show your processes being killed, then you may have to dump gc logging info to get some insight.
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms16G
-Xmx16G
-Xmn4800M
-XX:+HeapDumpOnOutOfMemoryError
-Xss256k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=80
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseParNewGC
-XX:MaxTenuringThreshold=2
-XX:SurvivorRatio=8
-XX:+UnlockDiagnosticVMOptions
-XX:ParGCCardsPerStrideChunk=32768
-XX:NewSize=750m
-XX:MaxNewSize=750m
-XX:+UseCondCardMark
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure
-Xloggc:<some-path-for-gc-logs>
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1M
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=<true|false>
-Dcom.sun.management.jmxremote.authenticate=<true|false>
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
Date: Wednesday, March 22, 2017 at 5:18 PM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
JVM config is as below:
-Xms16G
-Xmx16G
-Xmn3000M
What I need to check in dmesg?
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 23 March 2017 03:39
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>; user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.
Jayesh
________________________________
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
No only Cassandra is running on these servers.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
Re: [Cassandra 3.0.9] Cannot allocate memory
Posted by Nate McCall <na...@thelastpickle.com>.
On Thu, Mar 23, 2017 at 11:18 AM, Abhishek Kumar Maheshwari <
Abhishek.Maheshwari@timesinternet.in> wrote:
> JVM config is as below:
>
>
>
> -Xms16G
>
> -Xmx16G
>
> -Xmn3000M
>
>
>
I don't think it is the cause, but you need to remove Xmn when using G1GC.
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
JVM config is as below:
-Xms16G
-Xmx16G
-Xmn3000M
What I need to check in dmesg?
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 23 March 2017 03:39
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>; user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.
Jayesh
________________________________
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
No only Cassandra is running on these servers.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>; user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
And what is the configured max heap?
Sometimes you may also be able to see some useful messages in "dmesg" output.
Jayesh
________________________________
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
Sent: Wednesday, March 22, 2017 5:05:14 PM
To: Thakrar, Jayesh; user@cassandra.apache.org
Subject: RE: [Cassandra 3.0.9] Cannot allocate memory
No only Cassandra is running on these servers.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>; user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
No only Cassandra is running on these servers.
From: Thakrar, Jayesh [mailto:jthakrar@conversantmedia.com]
Sent: 22 March 2017 22:27
To: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>; user@cassandra.apache.org
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
Hi Abhishek,
In sysctl.conf file we have below setting:
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2
vm.max_map_count = 1048575
so I need to apply patch for same?
From: Abhishek Verma [mailto:verma@uber.com]
Sent: 22 March 2017 23:04
To: user@cassandra.apache.org
Cc: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
Subject: Re: [Cassandra 3.0.9] Cannot allocate memory
Just a shot in the dark, but what is your setting of vm.max_map_count in /etc/sysctl.conf ?
It is recommended to set it to:
vm.max_map_count = 1048575
Source: https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html
We saw a similar problem in the past where mmap failed and we added a check to emit warning as a part of https://issues.apache.org/jira/browse/CASSANDRA-13008.
On Wed, Mar 22, 2017 at 9:57 AM, Thakrar, Jayesh <jt...@conversantmedia.com>> wrote:
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591<tel:+91%2099998%2005591> (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com<http://www.et-gbs.com>
Re: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Verma <ve...@uber.com>.
Just a shot in the dark, but what is your setting of
vm.max_map_count in /etc/sysctl.conf ?
It is recommended to set it to:
vm.max_map_count = 1048575
Source:
https://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettingsLinux.html
We saw a similar problem in the past where mmap failed and we added a check
to emit warning as a part of
https://issues.apache.org/jira/browse/CASSANDRA-13008.
On Wed, Mar 22, 2017 at 9:57 AM, Thakrar, Jayesh <
jthakrar@conversantmedia.com> wrote:
> Is/are the Cassandra server(s) shared?
>
> E.g. do they run mesos + spark?
>
>
>
> *From: *Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
> *Date: *Wednesday, March 22, 2017 at 12:45 AM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *[Cassandra 3.0.9] Cannot allocate memory
>
>
>
> Hi all,
>
>
>
> I am using Cassandra 3.0.9. while I am adding new server after some time I
> am getting below exception. JVM option is attaches.
>
> Hardware info:
>
> Ram 64 GB.
>
> Core: 40
>
>
>
>
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> #
>
> # There is insufficient memory for the Java Runtime Environment to
> continue.
>
> # Native memory allocation (mmap) failed to map 12288 bytes for committing
> reserved memory.
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> [thread 140033204860672 also had an error]
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000,
> 12288, 0) failed; error='Cannot allocate memory' (errno=12)
>
> [thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server
> VM warning:
>
> INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed;
> error='Cannot allocate memory' (errno=12)
>
> Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also
> had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0)
> failed; error='Cannot allocate memory' (errno=12)
>
>
>
> Please let me know what I miss?
>
>
>
> *Thanks & Regards,*
> *Abhishek Kumar Maheshwari*
> *+91- 9999805591 <+91%2099998%2005591> (Mobile)*
>
> Times Internet Ltd. | A Times of India Group Company
>
> FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
>
> *P** Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.*
>
>
>
> YES Bank & The Economic Times Global Business Summit (GBS) is back on
> 27-28 March. The 3rd edition of GBS will see participation of 2000+
> delegates from over 20 countries as they bear witness to leaders sharing
> insights into how to best navigate a dynamic future. Visit www.et-gbs.com
>
Re: [Cassandra 3.0.9] Cannot allocate memory
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Is/are the Cassandra server(s) shared?
E.g. do they run mesos + spark?
From: Abhishek Kumar Maheshwari <Ab...@timesinternet.in>
Date: Wednesday, March 22, 2017 at 12:45 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.
YES Bank & The Economic Times Global Business Summit (GBS) is back on 27-28 March. The 3rd edition of GBS will see participation of 2000+ delegates from over 20 countries as they bear witness to leaders sharing insights into how to best navigate a dynamic future. Visit www.et-gbs.com
RE: [Cassandra 3.0.9] Cannot allocate memory
Posted by Abhishek Kumar Maheshwari <Ab...@timesinternet.in>.
The exception is as below:
INFO 17:42:37 Index build of til_lineitem_productsku_item_id,til_lineitem_productsku_status complete
WARN 17:43:28 G1 Old Generation GC in 2906ms. G1 Eden Space: 1560281088 -> 0; G1 Old Gen: 3033393144 -> 1127339984; G1 Survivor Space
: 150994944 -> 0;
INFO 17:43:28 Pool Name Active Pending Completed Blocked All Time Blocked
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9f31a55000, 12288, 0) failed; error='Cannot allocate memory'
(errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/apache-cassandra-3.0.9/bin/hs_err_pid8264.log
INFO 17:43:28 MutationStage 1 0 55409726 0 0
INFO 17:43:28 ViewMutationStage 0 0 0 0 0
INFO 17:43:28 ReadStage 0 0 0 0 0
INFO 17:43:28 RequestResponseStage 0 0 17 0 0
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9f830d8000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)INFO 17:43:28 ReadRepairStage 0 0 0 0 0
[thread 140321932023552 also had an error]
[thread 140321811961600 also had an error]
ERROR 17:43:28 Exception in thread Thread[CompactionExecutor:2482,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) ~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) ~[apache-cassandra-3.0.9.jar:3.0.9]
From: Abhishek Kumar Maheshwari
Sent: 22 March 2017 11:15
To: user@cassandra.apache.org
Subject: [Cassandra 3.0.9] Cannot allocate memory
Hi all,
I am using Cassandra 3.0.9. while I am adding new server after some time I am getting below exception. JVM option is attaches.
Hardware info:
Ram 64 GB.
Core: 40
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fe9c44ee000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c056ab000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204860672 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5c0566a000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
[thread 140033204594432 also had an error]Java HotSpot(TM) 64-Bit Server VM warning:
INFO: os::commit_memory(0x00007fe9c420c000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: [thread 140641994852096 also had an error]INFO: os::commit_memory(0x00007f5c055a7000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
Please let me know what I miss?
Thanks & Regards,
Abhishek Kumar Maheshwari
+91- 9999805591 (Mobile)
Times Internet Ltd. | A Times of India Group Company
FC - 6, Sector 16A, Film City, Noida, U.P. 201301 | INDIA
P Please do not print this email unless it is absolutely necessary. Spread environmental awareness.