You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rahul Bhardwaj <ra...@indiamart.com> on 2015/05/11 11:58:38 UTC
error='Cannot allocate memory' (errno=12)
Hi All,
We have cluster of 3 nodes with 64GB RAM each. My cluster was running in
healthy state. Suddenly one machine's cassandra daemon stops working and
shut down.
On restarting it after 2 minutes it again stops and is getting stop after
returning below error in cassandra.log
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 12288 bytes for
committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid23215.log
INFO 09:50:41 Loading settings from
file:/etc/cassandra/default.conf/cassandra.yaml
INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
authorizer=AllowAllAuthorizer; auto_snapshot=true;
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
cluster_name=Test Cluster; column_index_size_in_kb=64;
commit_failure_policy=stop;
commitlog_directory=/var/lib/cassandra/commitlog;
commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
concurrent_writes=32; counter_cache_save_period=7200;
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
dynamic_snitch_reset_interval_in_ms=600000;
dynamic_snitch_update_interval_in_ms=100;
endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
index_summary_capacity_in_mb=null;
index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
internode_compression=all; key_cache_save_period=14400;
key_cache_size_in_mb=null; listen_address=null;
max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
memtable_allocation_type=heap_buffers; native_transport_port=9042;
num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
read_request_timeout_in_ms=999999999;
request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
request_timeout_in_ms=999999999; row_cache_save_period=0;
row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
rpc_port=9160; rpc_server_type=sync;
saved_caches_directory=/var/lib/cassandra/saved_caches;
seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
parameters=[{seeds=206.191.151.199}]}];
server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
start_native_transport=true; start_rpc=true; storage_port=7000;
thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
tombstone_warn_threshold=1000; trickle_fsync=false;
trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
write_request_timeout_in_ms=999999999]
ERROR 09:50:41 Exception encountered during startup
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
~[na:1.7.0_60]
at
java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
~[na:1.7.0_60]
at
java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
~[na:1.7.0_60]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at
java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
at
java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
Exception encountered during startup: unable to create new native thread
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
allocate memory' (errno=12
This is our production cluster and rest 2 nodes are still in healthy state.
We have also changed heap size from 10G to 20 G but problem still persists.
Please help in finding the reason behind this.
Regards:
Rahul Bhardwaj
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
Re: error='Cannot allocate memory' (errno=12)
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
I see your ulimit -a above, missed that. You should increase nofile
ulimit. If you used JNA, you'd need to increase memlock too, but you
probably aren't using JNA. 1024 nofile is default and far too small, try
making that like 64K. Thread handles can count against file descriptors
limits, similar to how pipes, sockets, etc can count against file
descriptor limits.
On Tue, May 12, 2015 at 7:55 PM, J. Ryan Earl <os...@jryanearl.us> wrote:
> What's your ulimit -a output? Did you adjust nproc and nofile ulimits
> up? Do you have JNA installed? What about memlock ulimit and in
> sysctl.conf => kernel.shmmax?
>
> What's in cassandra.log?
>
> On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Hi Robert,
>>
>> I saw somewhere you answering the same prob but no solution found. plz
>> check again.
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>>
>>
>>
>> On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> Bu it is giving same error with java 7 and open jdk
>>>
>>> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> Well i havent used 2.1.x cassandra neither java 8 but any reason for
>>>> not using oracle JDK as i thought thats what is recommended. i saw a thread
>>>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>>>> 2.1.x versions.
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>>> PFA of error log
>>>>> hs_err_pid9656.log
>>>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>>>>
>>>>>
>>>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>>> free RAM:
>>>>>>
>>>>>>
>>>>>> free -m
>>>>>> total used free shared buffers
>>>>>> cached
>>>>>> Mem: 64398 23753 40644 0 108
>>>>>> 8324
>>>>>> -/+ buffers/cache: 15319 49078
>>>>>> Swap: 2925 15 2909
>>>>>>
>>>>>>
>>>>>> ulimit -a
>>>>>> core file size (blocks, -c) 0
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 515041
>>>>>> max locked memory (kbytes, -l) 64
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 1024
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 10240
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 515041
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>>
>>>>>> Also attaching complete error file
>>>>>>
>>>>>>
>>>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>>>>> there is no hardware failure on the machine. what is the free ram on the
>>>>>>> box ?
>>>>>>>
>>>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was
>>>>>>>> running in healthy state. Suddenly one machine's cassandra daemon stops
>>>>>>>> working and shut down.
>>>>>>>>
>>>>>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>>>>>> after returning below error in cassandra.log
>>>>>>>>
>>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>>>>> allocate memory' (errno=12)
>>>>>>>> #
>>>>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>>>>> continue.
>>>>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes
>>>>>>>> for committing reserved memory.
>>>>>>>> # An error report file with more information is saved as:
>>>>>>>> # /tmp/hs_err_pid23215.log
>>>>>>>> INFO 09:50:41 Loading settings from
>>>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>>>>> INFO 09:50:41 Node
>>>>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>>>>> commit_failure_policy=stop;
>>>>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>>>>> index_summary_capacity_in_mb=null;
>>>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>>>>> read_request_timeout_in_ms=999999999;
>>>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>>>>> write_request_timeout_in_ms=999999999]
>>>>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>>> ~[na:1.7.0_60]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>> at java.lang.Thread.start0(Native Method)
>>>>>>>> at java.lang.Thread.start(Thread.java:714)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>>> at
>>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>>> Exception encountered during startup: unable to create new native
>>>>>>>> thread
>>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>>>>> allocate memory' (errno=12
>>>>>>>>
>>>>>>>> This is our production cluster and rest 2 nodes are still in
>>>>>>>> healthy state.
>>>>>>>> We have also changed heap size from 10G to 20 G but problem still
>>>>>>>> persists. Please help in finding the reason behind this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards:
>>>>>>>> Rahul Bhardwaj
>>>>>>>>
>>>>>>>>
>>>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates
>>>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>>>> <https://www.facebook.com/IndiaMART>
>>>>>>>> <https://twitter.com/IndiaMART> Mobile Channel:
>>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>>> <http://m.indiamart.com/>
>>>>>>>>
>>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>>> this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Check out some smart tips to generate more business - *IndiaMART
>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>
>
>
Re: error='Cannot allocate memory' (errno=12)
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
What's your ulimit -a output? Did you adjust nproc and nofile ulimits up?
Do you have JNA installed? What about memlock ulimit and in sysctl.conf =>
kernel.shmmax?
What's in cassandra.log?
On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> Hi Robert,
>
> I saw somewhere you answering the same prob but no solution found. plz
> check again.
>
>
> regards:
> Rahul Bhardwaj
>
>
>
>
> On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Bu it is giving same error with java 7 and open jdk
>>
>> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <an...@gmail.com>
>> wrote:
>>
>>> Well i havent used 2.1.x cassandra neither java 8 but any reason for not
>>> using oracle JDK as i thought thats what is recommended. i saw a thread
>>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>>> 2.1.x versions.
>>>
>>>
>>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> PFA of error log
>>>> hs_err_pid9656.log
>>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>>> free RAM:
>>>>>
>>>>>
>>>>> free -m
>>>>> total used free shared buffers
>>>>> cached
>>>>> Mem: 64398 23753 40644 0 108
>>>>> 8324
>>>>> -/+ buffers/cache: 15319 49078
>>>>> Swap: 2925 15 2909
>>>>>
>>>>>
>>>>> ulimit -a
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 515041
>>>>> max locked memory (kbytes, -l) 64
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 1024
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 10240
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 515041
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>>
>>>>> Also attaching complete error file
>>>>>
>>>>>
>>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>>>> there is no hardware failure on the machine. what is the free ram on the
>>>>>> box ?
>>>>>>
>>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was
>>>>>>> running in healthy state. Suddenly one machine's cassandra daemon stops
>>>>>>> working and shut down.
>>>>>>>
>>>>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>>>>> after returning below error in cassandra.log
>>>>>>>
>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>>>> allocate memory' (errno=12)
>>>>>>> #
>>>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>>>> continue.
>>>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes
>>>>>>> for committing reserved memory.
>>>>>>> # An error report file with more information is saved as:
>>>>>>> # /tmp/hs_err_pid23215.log
>>>>>>> INFO 09:50:41 Loading settings from
>>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>>>> INFO 09:50:41 Node
>>>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>>>> commit_failure_policy=stop;
>>>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>>>> index_summary_capacity_in_mb=null;
>>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>>>> read_request_timeout_in_ms=999999999;
>>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>>>> write_request_timeout_in_ms=999999999]
>>>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>> ~[na:1.7.0_60]
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>> ~[na:1.7.0_60]
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>> ~[na:1.7.0_60]
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>> ~[na:1.7.0_60]
>>>>>>> at
>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>> at java.lang.Thread.start0(Native Method)
>>>>>>> at java.lang.Thread.start(Thread.java:714)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>>> at
>>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>>> at
>>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>>> at
>>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>>> Exception encountered during startup: unable to create new native
>>>>>>> thread
>>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>>>> allocate memory' (errno=12
>>>>>>>
>>>>>>> This is our production cluster and rest 2 nodes are still in healthy
>>>>>>> state.
>>>>>>> We have also changed heap size from 10G to 20 G but problem still
>>>>>>> persists. Please help in finding the reason behind this.
>>>>>>>
>>>>>>>
>>>>>>> Regards:
>>>>>>> Rahul Bhardwaj
>>>>>>>
>>>>>>>
>>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates
>>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>>> Mobile Channel:
>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>> <http://m.indiamart.com/>
>>>>>>>
>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Check out some smart tips to generate more business - *IndiaMART
>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>
>>>
>>>
>>
>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Check out some smart tips to generate more business - *IndiaMART Learning
> Centre* <http://corporate.indiamart.com/quick-learn/>
>
Re: error='Cannot allocate memory' (errno=12)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Hi Robert,
I saw somewhere you answering the same prob but no solution found. plz
check again.
regards:
Rahul Bhardwaj
On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> Bu it is giving same error with java 7 and open jdk
>
> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> Well i havent used 2.1.x cassandra neither java 8 but any reason for not
>> using oracle JDK as i thought thats what is recommended. i saw a thread
>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>> 2.1.x versions.
>>
>>
>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> PFA of error log
>>> hs_err_pid9656.log
>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>>
>>>
>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> free RAM:
>>>>
>>>>
>>>> free -m
>>>> total used free shared buffers
>>>> cached
>>>> Mem: 64398 23753 40644 0 108
>>>> 8324
>>>> -/+ buffers/cache: 15319 49078
>>>> Swap: 2925 15 2909
>>>>
>>>>
>>>> ulimit -a
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 515041
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 1024
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 10240
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 515041
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>>
>>>> Also attaching complete error file
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>>> there is no hardware failure on the machine. what is the free ram on the
>>>>> box ?
>>>>>
>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was
>>>>>> running in healthy state. Suddenly one machine's cassandra daemon stops
>>>>>> working and shut down.
>>>>>>
>>>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>>>> after returning below error in cassandra.log
>>>>>>
>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>>> allocate memory' (errno=12)
>>>>>> #
>>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>>> continue.
>>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes
>>>>>> for committing reserved memory.
>>>>>> # An error report file with more information is saved as:
>>>>>> # /tmp/hs_err_pid23215.log
>>>>>> INFO 09:50:41 Loading settings from
>>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>>> INFO 09:50:41 Node
>>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>>> commit_failure_policy=stop;
>>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>>> index_summary_capacity_in_mb=null;
>>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>>> read_request_timeout_in_ms=999999999;
>>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>>> write_request_timeout_in_ms=999999999]
>>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>> ~[na:1.7.0_60]
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>> ~[na:1.7.0_60]
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>> ~[na:1.7.0_60]
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>> ~[na:1.7.0_60]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>> at java.lang.Thread.start0(Native Method)
>>>>>> at java.lang.Thread.start(Thread.java:714)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>>> at
>>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>>> Exception encountered during startup: unable to create new native
>>>>>> thread
>>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>>> allocate memory' (errno=12
>>>>>>
>>>>>> This is our production cluster and rest 2 nodes are still in healthy
>>>>>> state.
>>>>>> We have also changed heap size from 10G to 20 G but problem still
>>>>>> persists. Please help in finding the reason behind this.
>>>>>>
>>>>>>
>>>>>> Regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>>
>>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates
>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>> Mobile Channel:
>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>> <http://m.indiamart.com/>
>>>>>>
>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Check out some smart tips to generate more business - *IndiaMART
>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>
>>
>>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
Re: error='Cannot allocate memory' (errno=12)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Bu it is giving same error with java 7 and open jdk
On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal <an...@gmail.com> wrote:
> Well i havent used 2.1.x cassandra neither java 8 but any reason for not
> using oracle JDK as i thought thats what is recommended. i saw a thread
> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
> 2.1.x versions.
>
>
> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> PFA of error log
>> hs_err_pid9656.log
>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>
>>
>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> free RAM:
>>>
>>>
>>> free -m
>>> total used free shared buffers cached
>>> Mem: 64398 23753 40644 0 108 8324
>>> -/+ buffers/cache: 15319 49078
>>> Swap: 2925 15 2909
>>>
>>>
>>> ulimit -a
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size (blocks, -f) unlimited
>>> pending signals (-i) 515041
>>> max locked memory (kbytes, -l) 64
>>> max memory size (kbytes, -m) unlimited
>>> open files (-n) 1024
>>> pipe size (512 bytes, -p) 8
>>> POSIX message queues (bytes, -q) 819200
>>> real-time priority (-r) 0
>>> stack size (kbytes, -s) 10240
>>> cpu time (seconds, -t) unlimited
>>> max user processes (-u) 515041
>>> virtual memory (kbytes, -v) unlimited
>>> file locks (-x) unlimited
>>>
>>>
>>> Also attaching complete error file
>>>
>>>
>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>> there is no hardware failure on the machine. what is the free ram on the
>>>> box ?
>>>>
>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running
>>>>> in healthy state. Suddenly one machine's cassandra daemon stops working and
>>>>> shut down.
>>>>>
>>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>>> after returning below error in cassandra.log
>>>>>
>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>>> allocate memory' (errno=12)
>>>>> #
>>>>> # There is insufficient memory for the Java Runtime Environment to
>>>>> continue.
>>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>>>> committing reserved memory.
>>>>> # An error report file with more information is saved as:
>>>>> # /tmp/hs_err_pid23215.log
>>>>> INFO 09:50:41 Loading settings from
>>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>>> INFO 09:50:41 Node
>>>>> configuration:[authenticator=AllowAllAuthenticator;
>>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>>> commit_failure_policy=stop;
>>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>>> dynamic_snitch_update_interval_in_ms=100;
>>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>>> index_summary_capacity_in_mb=null;
>>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>>> internode_compression=all; key_cache_save_period=14400;
>>>>> key_cache_size_in_mb=null; listen_address=null;
>>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>>> read_request_timeout_in_ms=999999999;
>>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>>> rpc_port=9160; rpc_server_type=sync;
>>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>>> parameters=[{seeds=206.191.151.199}]}];
>>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>>> write_request_timeout_in_ms=999999999]
>>>>> ERROR 09:50:41 Exception encountered during startup
>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>> ~[na:1.7.0_60]
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>> ~[na:1.7.0_60]
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>> ~[na:1.7.0_60]
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>> ~[na:1.7.0_60]
>>>>> at
>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>> at java.lang.Thread.start0(Native Method)
>>>>> at java.lang.Thread.start(Thread.java:714)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>>> at
>>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>>> at
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>>> at
>>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>>> Exception encountered during startup: unable to create new native
>>>>> thread
>>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>>> allocate memory' (errno=12
>>>>>
>>>>> This is our production cluster and rest 2 nodes are still in healthy
>>>>> state.
>>>>> We have also changed heap size from 10G to 20 G but problem still
>>>>> persists. Please help in finding the reason behind this.
>>>>>
>>>>>
>>>>> Regards:
>>>>> Rahul Bhardwaj
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>>> this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Check out some smart tips to generate more business - *IndiaMART
>>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>>
>>>>
>>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Check out some smart tips to generate more business - *IndiaMART
>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
Re: error='Cannot allocate memory' (errno=12)
Posted by Anishek Agarwal <an...@gmail.com>.
Well i havent used 2.1.x cassandra neither java 8 but any reason for not
using oracle JDK as i thought thats what is recommended. i saw a thread
earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
2.1.x versions.
On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> PFA of error log
> hs_err_pid9656.log
> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>
>
> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> free RAM:
>>
>>
>> free -m
>> total used free shared buffers cached
>> Mem: 64398 23753 40644 0 108 8324
>> -/+ buffers/cache: 15319 49078
>> Swap: 2925 15 2909
>>
>>
>> ulimit -a
>> core file size (blocks, -c) 0
>> data seg size (kbytes, -d) unlimited
>> scheduling priority (-e) 0
>> file size (blocks, -f) unlimited
>> pending signals (-i) 515041
>> max locked memory (kbytes, -l) 64
>> max memory size (kbytes, -m) unlimited
>> open files (-n) 1024
>> pipe size (512 bytes, -p) 8
>> POSIX message queues (bytes, -q) 819200
>> real-time priority (-r) 0
>> stack size (kbytes, -s) 10240
>> cpu time (seconds, -t) unlimited
>> max user processes (-u) 515041
>> virtual memory (kbytes, -v) unlimited
>> file locks (-x) unlimited
>>
>>
>> Also attaching complete error file
>>
>>
>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
>> wrote:
>>
>>> the memory cassandra is trying to allocate is pretty small. you sure
>>> there is no hardware failure on the machine. what is the free ram on the
>>> box ?
>>>
>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running
>>>> in healthy state. Suddenly one machine's cassandra daemon stops working and
>>>> shut down.
>>>>
>>>> On restarting it after 2 minutes it again stops and is getting stop
>>>> after returning below error in cassandra.log
>>>>
>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>>> allocate memory' (errno=12)
>>>> #
>>>> # There is insufficient memory for the Java Runtime Environment to
>>>> continue.
>>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>>> committing reserved memory.
>>>> # An error report file with more information is saved as:
>>>> # /tmp/hs_err_pid23215.log
>>>> INFO 09:50:41 Loading settings from
>>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>>> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
>>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>>> commit_failure_policy=stop;
>>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>>> concurrent_writes=32; counter_cache_save_period=7200;
>>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>>> dynamic_snitch_reset_interval_in_ms=600000;
>>>> dynamic_snitch_update_interval_in_ms=100;
>>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>>> index_summary_capacity_in_mb=null;
>>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>>> internode_compression=all; key_cache_save_period=14400;
>>>> key_cache_size_in_mb=null; listen_address=null;
>>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>>> read_request_timeout_in_ms=999999999;
>>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>>> rpc_port=9160; rpc_server_type=sync;
>>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>>> parameters=[{seeds=206.191.151.199}]}];
>>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>>> write_request_timeout_in_ms=999999999]
>>>> ERROR 09:50:41 Exception encountered during startup
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>> ~[na:1.7.0_60]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>> ~[na:1.7.0_60]
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>> ~[na:1.7.0_60]
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>> ~[na:1.7.0_60]
>>>> at
>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>> at java.lang.Thread.start0(Native Method)
>>>> at java.lang.Thread.start(Thread.java:714)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>>> at
>>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>>> at
>>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>>> at
>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>>> at
>>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>>> Exception encountered during startup: unable to create new native thread
>>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>>> allocate memory' (errno=12
>>>>
>>>> This is our production cluster and rest 2 nodes are still in healthy
>>>> state.
>>>> We have also changed heap size from 10G to 20 G but problem still
>>>> persists. Please help in finding the reason behind this.
>>>>
>>>>
>>>> Regards:
>>>> Rahul Bhardwaj
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Check out some smart tips to generate more business - *IndiaMART
>>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>>
>>>
>>>
>>
>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Check out some smart tips to generate more business - *IndiaMART Learning
> Centre* <http://corporate.indiamart.com/quick-learn/>
>
Re: error='Cannot allocate memory' (errno=12)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
PFA of error log
hs_err_pid9656.log
<https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> free RAM:
>
>
> free -m
> total used free shared buffers cached
> Mem: 64398 23753 40644 0 108 8324
> -/+ buffers/cache: 15319 49078
> Swap: 2925 15 2909
>
>
> ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 515041
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 515041
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> Also attaching complete error file
>
>
> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> the memory cassandra is trying to allocate is pretty small. you sure
>> there is no hardware failure on the machine. what is the free ram on the
>> box ?
>>
>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> Hi All,
>>>
>>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running
>>> in healthy state. Suddenly one machine's cassandra daemon stops working and
>>> shut down.
>>>
>>> On restarting it after 2 minutes it again stops and is getting stop
>>> after returning below error in cassandra.log
>>>
>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>>> allocate memory' (errno=12)
>>> #
>>> # There is insufficient memory for the Java Runtime Environment to
>>> continue.
>>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>>> committing reserved memory.
>>> # An error report file with more information is saved as:
>>> # /tmp/hs_err_pid23215.log
>>> INFO 09:50:41 Loading settings from
>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>> commit_failure_policy=stop;
>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>>> concurrent_writes=32; counter_cache_save_period=7200;
>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>> dynamic_snitch_reset_interval_in_ms=600000;
>>> dynamic_snitch_update_interval_in_ms=100;
>>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>>> index_summary_capacity_in_mb=null;
>>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>>> internode_compression=all; key_cache_save_period=14400;
>>> key_cache_size_in_mb=null; listen_address=null;
>>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>>> read_request_timeout_in_ms=999999999;
>>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>>> rpc_port=9160; rpc_server_type=sync;
>>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>>> parameters=[{seeds=206.191.151.199}]}];
>>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>>> start_native_transport=true; start_rpc=true; storage_port=7000;
>>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>>> tombstone_warn_threshold=1000; trickle_fsync=false;
>>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>>> write_request_timeout_in_ms=999999999]
>>> ERROR 09:50:41 Exception encountered during startup
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>> ~[na:1.7.0_60]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>> ~[na:1.7.0_60]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>> ~[na:1.7.0_60]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>> ~[na:1.7.0_60]
>>> at
>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>> at java.lang.Thread.start0(Native Method)
>>> at java.lang.Thread.start(Thread.java:714)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>>> at
>>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>>> at
>>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>>> Exception encountered during startup: unable to create new native thread
>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>>> allocate memory' (errno=12
>>>
>>> This is our production cluster and rest 2 nodes are still in healthy
>>> state.
>>> We have also changed heap size from 10G to 20 G but problem still
>>> persists. Please help in finding the reason behind this.
>>>
>>>
>>> Regards:
>>> Rahul Bhardwaj
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Check out some smart tips to generate more business - *IndiaMART
>>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>>
>>
>>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
Re: error='Cannot allocate memory' (errno=12)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
free RAM:
free -m
total used free shared buffers cached
Mem: 64398 23753 40644 0 108 8324
-/+ buffers/cache: 15319 49078
Swap: 2925 15 2909
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515041
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 515041
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Also attaching complete error file
On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal <an...@gmail.com> wrote:
> the memory cassandra is trying to allocate is pretty small. you sure there
> is no hardware failure on the machine. what is the free ram on the box ?
>
> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Hi All,
>>
>> We have cluster of 3 nodes with 64GB RAM each. My cluster was running in
>> healthy state. Suddenly one machine's cassandra daemon stops working and
>> shut down.
>>
>> On restarting it after 2 minutes it again stops and is getting stop after
>> returning below error in cassandra.log
>>
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
>> allocate memory' (errno=12)
>> #
>> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>> # Native memory allocation (malloc) failed to allocate 12288 bytes for
>> committing reserved memory.
>> # An error report file with more information is saved as:
>> # /tmp/hs_err_pid23215.log
>> INFO 09:50:41 Loading settings from
>> file:/etc/cassandra/default.conf/cassandra.yaml
>> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>> commit_failure_policy=stop;
>> commitlog_directory=/var/lib/cassandra/commitlog;
>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
>> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
>> concurrent_writes=32; counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>> dynamic_snitch_reset_interval_in_ms=600000;
>> dynamic_snitch_update_interval_in_ms=100;
>> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
>> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
>> index_summary_capacity_in_mb=null;
>> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
>> internode_compression=all; key_cache_save_period=14400;
>> key_cache_size_in_mb=null; listen_address=null;
>> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
>> memtable_allocation_type=heap_buffers; native_transport_port=9042;
>> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
>> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
>> read_request_timeout_in_ms=999999999;
>> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
>> request_timeout_in_ms=999999999; row_cache_save_period=0;
>> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
>> rpc_port=9160; rpc_server_type=sync;
>> saved_caches_directory=/var/lib/cassandra/saved_caches;
>> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
>> parameters=[{seeds=206.191.151.199}]}];
>> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
>> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
>> start_native_transport=true; start_rpc=true; storage_port=7000;
>> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
>> tombstone_warn_threshold=1000; trickle_fsync=false;
>> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
>> write_request_timeout_in_ms=999999999]
>> ERROR 09:50:41 Exception encountered during startup
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
>> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>> ~[na:1.7.0_60]
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>> ~[na:1.7.0_60]
>> at
>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:714)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at
>> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
>> at
>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
>> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
>> at
>> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
>> Exception encountered during startup: unable to create new native thread
>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
>> allocate memory' (errno=12
>>
>> This is our production cluster and rest 2 nodes are still in healthy
>> state.
>> We have also changed heap size from 10G to 20 G but problem still
>> persists. Please help in finding the reason behind this.
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Check out some smart tips to generate more business - *IndiaMART
>> Learning Centre* <http://corporate.indiamart.com/quick-learn/>
>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Check out some smart tips to generate more business - *IndiaMART Learning
Centre* <http://corporate.indiamart.com/quick-learn/>
Re: error='Cannot allocate memory' (errno=12)
Posted by Anishek Agarwal <an...@gmail.com>.
the memory cassandra is trying to allocate is pretty small. you sure there
is no hardware failure on the machine. what is the free ram on the box ?
On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> Hi All,
>
> We have cluster of 3 nodes with 64GB RAM each. My cluster was running in
> healthy state. Suddenly one machine's cassandra daemon stops working and
> shut down.
>
> On restarting it after 2 minutes it again stops and is getting stop after
> returning below error in cassandra.log
>
> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> os::commit_memory(0x00007fd064dc6000, 12288, 0) failed; error='Cannot
> allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (malloc) failed to allocate 12288 bytes for
> committing reserved memory.
> # An error report file with more information is saved as:
> # /tmp/hs_err_pid23215.log
> INFO 09:50:41 Loading settings from
> file:/etc/cassandra/default.conf/cassandra.yaml
> INFO 09:50:41 Node configuration:[authenticator=AllowAllAuthenticator;
> authorizer=AllowAllAuthorizer; auto_snapshot=true;
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>;
> cluster_name=Test Cluster; column_index_size_in_kb=64;
> commit_failure_policy=stop;
> commitlog_directory=/var/lib/cassandra/commitlog;
> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
> commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16;
> concurrent_compactors=4; concurrent_counter_writes=32; concurrent_reads=32;
> concurrent_writes=32; counter_cache_save_period=7200;
> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
> cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data];
> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
> dynamic_snitch_reset_interval_in_ms=600000;
> dynamic_snitch_update_interval_in_ms=100;
> endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true;
> hinted_handoff_throttle_in_kb=1024; incremental_backups=false;
> index_summary_capacity_in_mb=null;
> index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
> internode_compression=all; key_cache_save_period=14400;
> key_cache_size_in_mb=null; listen_address=null;
> max_hint_window_in_ms=10800000; max_hints_delivery_threads=2;
> memtable_allocation_type=heap_buffers; native_transport_port=9042;
> num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
> permissions_validity_in_ms=2000; range_request_timeout_in_ms=1000000;
> read_request_timeout_in_ms=999999999;
> request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
> request_timeout_in_ms=999999999; row_cache_save_period=0;
> row_cache_size_in_mb=0; rpc_address=null; rpc_keepalive=true;
> rpc_port=9160; rpc_server_type=sync;
> saved_caches_directory=/var/lib/cassandra/saved_caches;
> seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider,
> parameters=[{seeds=206.191.151.199}]}];
> server_encryption_options=<REDACTED>; snapshot_before_compaction=false;
> ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50;
> start_native_transport=true; start_rpc=true; storage_port=7000;
> thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000;
> tombstone_warn_threshold=1000; trickle_fsync=false;
> trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000;
> write_request_timeout_in_ms=999999999]
> ERROR 09:50:41 Exception encountered during startup
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method) ~[na:1.7.0_60]
> at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> ~[na:1.7.0_60]
> at
> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
> ~[na:1.7.0_60]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
> ~[na:1.7.0_60]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
> ~[na:1.7.0_60]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at
> java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.scheduleWithFixedDelay(DebuggableScheduledThreadPoolExecutor.java:61)
> at org.apache.cassandra.gms.Gossiper.start(Gossiper.java:1188)
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:721)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:643)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:535)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:448)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:537)
> Exception encountered during startup: unable to create new native thread
> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> os::commit_memory(0x00007fd459520000, 65536, 1) failed; error='Cannot
> allocate memory' (errno=12
>
> This is our production cluster and rest 2 nodes are still in healthy state.
> We have also changed heap size from 10G to 20 G but problem still
> persists. Please help in finding the reason behind this.
>
>
> Regards:
> Rahul Bhardwaj
>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Check out some smart tips to generate more business - *IndiaMART Learning
> Centre* <http://corporate.indiamart.com/quick-learn/>