You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roy Burstein <bu...@gmail.com> on 2019/01/14 16:33:06 UTC
Cassandra 3.11.3 map errors
Hi ,
We are testing C* 3.11.3 and we have mapping issue and possibly leaked
memory.
It might be related to our configuration,any ideas would be helpful .
Cassandra version: 3.11.3
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.1.3.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
*LEAK DETECTED:*
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Index.db
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@5b90823b) to class
org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Data.db
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763) to class
org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1710583516:[Memory@[0..3e24),
Memory@[0..45e88)] was not released before the reference was garbage
collected
*Limits of Cassandra process:*
[root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v
grep | awk '\{print $2}'`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 32768 32768 processes
Max open files 100000 100000 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 766985 766985 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
*max_map_count parameter on OS:*
[root@cass063 ~]# sysctl vm.max_map_count
vm.max_map_count = 1073741824
*cassandra.yaml:*
cluster_name: 'Cass Cluster'
num_tokens: 256
hinted_handoff_enabled: false
max_hint_window_in_ms: 10800000
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
hints_directory: /var/lib/cassandra/hints
hints_flush_period_in_ms: 10000
max_hints_file_size_in_mb: 128
batchlog_replay_throttle_in_kb: 1024
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
credentials_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data/disk1
commitlog_directory: /var/lib/cassandra/data/disk1/commitlog
cdc_enabled: false
disk_failure_policy: stop
commit_failure_policy: stop
prepared_statements_cache_size_mb:
thrift_prepared_statements_cache_size_mb:
key_cache_size_in_mb: 0
key_cache_save_period: 3600
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "10.110.30.1,10.110.30.2,10.110.30.3"
concurrent_reads: 48
concurrent_writes: 96
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32
file_cache_size_in_mb: 10240
memtable_offheap_space_in_mb: 10240
memtable_cleanup_threshold: 0.1
memtable_allocation_type: offheap_buffers
commitlog_total_space_in_mb: 8192
memtable_flush_writers: 8
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: true
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.106.62.34
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: 0.0.0.0
rpc_port: 9160
broadcast_rpc_address: 10.106.62.34
rpc_keepalive: true
rpc_server_type: hsha
rpc_max_threads: 128
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
column_index_size_in_kb: 64
column_index_cache_size_in_kb: 2
concurrent_compactors: 32
compaction_throughput_mb_per_sec: 500
sstable_preemptive_open_interval_in_mb: 50
stream_throughput_outbound_megabits_per_sec: 0
read_request_timeout_in_ms: 10000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 60000
counter_write_request_timeout_in_ms: 10000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
slow_query_log_timeout_in_ms: 500
cross_node_timeout: false
phi_convict_threshold: 12
endpoint_snitch: GossipingPropertyFileSnitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.5
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra
client_encryption_options:
enabled: false
optional: false
keystore: conf/.keystore
keystore_password: cassandra
internode_compression: dc
inter_dc_tcp_nodelay: false
tracetype_query_ttl: 86400
tracetype_repair_ttl: 604800
enable_user_defined_functions: false
enable_scripted_user_defined_functions: false
enable_materialized_views: true
windows_timer_interval: 1
transparent_data_encryption_options:
enabled: false
chunk_length_kb: 64
cipher: AES/CBC/PKCS5Padding
key_alias: testing:1
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: conf/.keystore
keystore_password: cassandra
store_type: JCEKS
key_password: cassandra
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
unlogged_batch_across_partitions_warn_threshold: 10
compaction_large_partition_warning_threshold_mb: 10
gc_warn_threshold_in_ms: 1000
back_pressure_enabled: false
back_pressure_strategy:
- class_name: org.apache.cassandra.net.RateBasedBackPressure
parameters:
- high_ratio: 0.90
factor: 5
flow: FAST
*A lot of maps, 200K maps of cassandra process,*:
[root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v
grep | awk '{print $2}'`/maps
239587 /proc/202664/maps
Thanks,
Roy
Re: Cassandra 3.11.3 map errors
Posted by Laxmikant Upadhyay <la...@gmail.com>.
I have observed a memory leak in 3.11.2 version similar situation which
Bobbie has mentioned, We faced corrupt sstable issue after disk space was
100% full (due to snapshots) in multiple nodes (we have default
disk_failure_policy)
After clearing the snapshots we ran repair and observed following issues
when running 'nodetool repair -full -pr'
*1. Failed giving multiple warnings followed
by ArrayIndexOutOfBoundsException *
WARN [CompactionExecutor:302] 2019-01-21 06:04:01,502
LeveledCompactionStrategy.java:144 - Could not acquire references for
compacting SSTables ......which is not a problem per se,unless it happens
frequently, in which case it must be reported. Will retry later.
ERROR [ValidationExecutor:49] 2019-01-21 05:46:39,810 Validator.java:268 -
Failed creating a merkle tree for [repair
#441ad710-1d11-11e9-acec-43289f88a823 on ks1/table1, [
ERROR [ValidationExecutor:49] 2019-01-21 05:46:39,811
CassandraDaemon.java:228 - Exception in thread
Thread[ValidationExecutor:49,1,main]
java.lang.ArrayIndexOutOfBoundsException: null
2. * After this failure, we restarted the node and compaction got triggered
after startup and failed giving CorruptSSTableException error: *
Caused by: org.apache.cassandra.io.compress.CorruptBlockException:
(/var/lib/cassandra/data/ks1/table2-f7a6e260155a11e996298b24fac245dc/mc-2154-big-Data.db):
corruption detected, chunk at 51859444 of length 10387.
ERROR [CompactionExecutor:3] 2019-01-21 07:27:03,282
CassandraDaemon.java:244 - Exception in thread
Thread[CompactionExecutor:3,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/lib/cassandra/data/ks1/table2-f7a6e260155a11e9962
98b24fac245dc/mc-2154-big-Data.db
Not sure why it was not giving CorruptSSTableException at first place.
*3. One node went down without writing any logs in system.log ..further
checking the messages log of linux it was found that the cassandra process
went due to OOM.* After starting the node, system.log or compactionHistory
also *not showing record of any activity at the time when the node went
down*.
@Bobbie have you raised any jira for this issue ?
On Wed, Jan 16, 2019 at 1:15 PM Roy Burstein <bu...@gmail.com> wrote:
> Hi Bob ,
> We did not use the reaper(yet) just trying to run tests against the new C*
> v3 cluster .
> The nodes keeps crashing all the time and these are the error we are
> getting .
> Any other ideas ?
> Thanks!
> Roy
>
> On Wed, Jan 16, 2019 at 8:51 AM Bobbie Haynes <ha...@gmail.com>
> wrote:
>
>> Hi Roy,
>> I don't think the Memory Leak issue is related to MAP errors
>> .I was also using Reaper in our cluster.I have seen Memory Leak issue (ERROR
>> [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED)
>> when some of SStables got corrupted because of disk space issue we had when
>> compactions were running.
>> I guess you have to report this Memory Leak issue to Reaper tool JIRA.
>>
>> Thanks,
>> Bob
>>
>> On Mon, Jan 14, 2019 at 8:44 AM Roy Burstein <bu...@gmail.com>
>> wrote:
>>
>>> Hi ,
>>>
>>> We are testing C* 3.11.3 and we have mapping issue and possibly leaked
>>> memory.
>>> It might be related to our configuration,any ideas would be helpful .
>>>
>>>
>>>
>>> Cassandra version: 3.11.3
>>> OS: CentOS Linux release 7.4.1708 (Core)
>>> Kernel: 3.10.0-957.1.3.el7.x86_64
>>> JDK: jdk1.8.0_131
>>> Heap: same errors with 16GB / 32GB / 64GB.
>>>
>>> *We are seeing this errors in production:*
>>>
>>> *java.io.IOException: Map failed:*
>>>
>>> ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:5017,1,main]
>>> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>>> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
>>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
>>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
>>> Caused by: java.io.IOException: Map failed
>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_131]
>>> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.3.jar:3.11.3]
>>> ... 23 common frames omitted
>>> Caused by: java.lang.OutOfMemoryError: Map failed
>>> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_131]
>>> ... 24 common frames omitted
>>>
>>> *LEAK DETECTED:*
>>>
>>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290) was not released before the reference was garbage collected
>>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Index.db was not released before the reference was garbage collected
>>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b90823b) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Data.db was not released before the reference was garbage collected
>>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1710583516:[Memory@[0..3e24), Memory@[0..45e88)] was not released before the reference was garbage collected
>>>
>>>
>>> *Limits of Cassandra process:*
>>>
>>> [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '\{print $2}'`/limits
>>> Limit Soft Limit Hard Limit Units
>>> Max cpu time unlimited unlimited seconds
>>> Max file size unlimited unlimited bytes
>>> Max data size unlimited unlimited bytes
>>> Max stack size 8388608 unlimited bytes
>>> Max core file size 0 unlimited bytes
>>> Max resident set unlimited unlimited bytes
>>> Max processes 32768 32768 processes
>>> Max open files 100000 100000 files
>>> Max locked memory unlimited unlimited bytes
>>> Max address space unlimited unlimited bytes
>>> Max file locks unlimited unlimited locks
>>> Max pending signals 766985 766985 signals
>>> Max msgqueue size 819200 819200 bytes
>>> Max nice priority 0 0
>>> Max realtime priority 0 0
>>> Max realtime timeout unlimited unlimited us
>>>
>>>
>>>
>>> *max_map_count parameter on OS:*
>>>
>>> [root@cass063 ~]# sysctl vm.max_map_count
>>> vm.max_map_count = 1073741824
>>>
>>>
>>>
>>>
>>> *cassandra.yaml:*
>>>
>>> cluster_name: 'Cass Cluster'
>>> num_tokens: 256
>>> hinted_handoff_enabled: false
>>> max_hint_window_in_ms: 10800000
>>> hinted_handoff_throttle_in_kb: 1024
>>> max_hints_delivery_threads: 2
>>> hints_directory: /var/lib/cassandra/hints
>>> hints_flush_period_in_ms: 10000
>>> max_hints_file_size_in_mb: 128
>>> batchlog_replay_throttle_in_kb: 1024
>>> authenticator: AllowAllAuthenticator
>>> authorizer: AllowAllAuthorizer
>>> role_manager: CassandraRoleManager
>>> roles_validity_in_ms: 2000
>>> permissions_validity_in_ms: 2000
>>> credentials_validity_in_ms: 2000
>>> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> data_file_directories:
>>> - /var/lib/cassandra/data/disk1
>>> commitlog_directory: /var/lib/cassandra/data/disk1/commitlog
>>> cdc_enabled: false
>>> disk_failure_policy: stop
>>> commit_failure_policy: stop
>>> prepared_statements_cache_size_mb:
>>> thrift_prepared_statements_cache_size_mb:
>>> key_cache_size_in_mb: 0
>>> key_cache_save_period: 3600
>>> row_cache_size_in_mb: 0
>>> row_cache_save_period: 0
>>> counter_cache_size_in_mb:
>>> counter_cache_save_period: 7200
>>> saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches
>>> commitlog_sync: periodic
>>> commitlog_sync_period_in_ms: 10000
>>> commitlog_segment_size_in_mb: 32
>>> seed_provider:
>>> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>> parameters:
>>> - seeds: "10.110.30.1,10.110.30.2,10.110.30.3"
>>> concurrent_reads: 48
>>> concurrent_writes: 96
>>> concurrent_counter_writes: 32
>>> concurrent_materialized_view_writes: 32
>>> file_cache_size_in_mb: 10240
>>> memtable_offheap_space_in_mb: 10240
>>> memtable_cleanup_threshold: 0.1
>>> memtable_allocation_type: offheap_buffers
>>> commitlog_total_space_in_mb: 8192
>>> memtable_flush_writers: 8
>>> index_summary_capacity_in_mb:
>>> index_summary_resize_interval_in_minutes: 60
>>> trickle_fsync: true
>>> trickle_fsync_interval_in_kb: 10240
>>> storage_port: 7000
>>> ssl_storage_port: 7001
>>> listen_address: 10.106.62.34
>>> start_native_transport: true
>>> native_transport_port: 9042
>>> start_rpc: false
>>> rpc_address: 0.0.0.0
>>> rpc_port: 9160
>>> broadcast_rpc_address: 10.106.62.34
>>> rpc_keepalive: true
>>> rpc_server_type: hsha
>>> rpc_max_threads: 128
>>> thrift_framed_transport_size_in_mb: 15
>>> incremental_backups: false
>>> snapshot_before_compaction: false
>>> auto_snapshot: true
>>> column_index_size_in_kb: 64
>>> column_index_cache_size_in_kb: 2
>>> concurrent_compactors: 32
>>> compaction_throughput_mb_per_sec: 500
>>> sstable_preemptive_open_interval_in_mb: 50
>>> stream_throughput_outbound_megabits_per_sec: 0
>>> read_request_timeout_in_ms: 10000
>>> range_request_timeout_in_ms: 10000
>>> write_request_timeout_in_ms: 60000
>>> counter_write_request_timeout_in_ms: 10000
>>> cas_contention_timeout_in_ms: 1000
>>> truncate_request_timeout_in_ms: 60000
>>> request_timeout_in_ms: 10000
>>> slow_query_log_timeout_in_ms: 500
>>> cross_node_timeout: false
>>> phi_convict_threshold: 12
>>> endpoint_snitch: GossipingPropertyFileSnitch
>>> dynamic_snitch_update_interval_in_ms: 100
>>> dynamic_snitch_reset_interval_in_ms: 600000
>>> dynamic_snitch_badness_threshold: 0.5
>>> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
>>> server_encryption_options:
>>> internode_encryption: none
>>> keystore: conf/.keystore
>>> keystore_password: cassandra
>>> truststore: conf/.truststore
>>> truststore_password: cassandra
>>> client_encryption_options:
>>> enabled: false
>>> optional: false
>>> keystore: conf/.keystore
>>> keystore_password: cassandra
>>> internode_compression: dc
>>> inter_dc_tcp_nodelay: false
>>> tracetype_query_ttl: 86400
>>> tracetype_repair_ttl: 604800
>>> enable_user_defined_functions: false
>>> enable_scripted_user_defined_functions: false
>>> enable_materialized_views: true
>>> windows_timer_interval: 1
>>> transparent_data_encryption_options:
>>> enabled: false
>>> chunk_length_kb: 64
>>> cipher: AES/CBC/PKCS5Padding
>>> key_alias: testing:1
>>> key_provider:
>>> - class_name: org.apache.cassandra.security.JKSKeyProvider
>>> parameters:
>>> - keystore: conf/.keystore
>>> keystore_password: cassandra
>>> store_type: JCEKS
>>> key_password: cassandra
>>> tombstone_warn_threshold: 1000
>>> tombstone_failure_threshold: 100000
>>> batch_size_warn_threshold_in_kb: 5
>>> batch_size_fail_threshold_in_kb: 50
>>> unlogged_batch_across_partitions_warn_threshold: 10
>>> compaction_large_partition_warning_threshold_mb: 10
>>> gc_warn_threshold_in_ms: 1000
>>> back_pressure_enabled: false
>>> back_pressure_strategy:
>>> - class_name: org.apache.cassandra.net.RateBasedBackPressure
>>> parameters:
>>> - high_ratio: 0.90
>>> factor: 5
>>> flow: FAST
>>>
>>>
>>>
>>> *A lot of maps, 200K maps of cassandra process,*:
>>>
>>> [root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '{print $2}'`/maps
>>> 239587 /proc/202664/maps
>>>
>>> Thanks,
>>> Roy
>>>
>>
--
regards,
Laxmikant Upadhyay
Re: Cassandra 3.11.3 map errors
Posted by Roy Burstein <bu...@gmail.com>.
Hi Bob ,
We did not use the reaper(yet) just trying to run tests against the new C*
v3 cluster .
The nodes keeps crashing all the time and these are the error we are
getting .
Any other ideas ?
Thanks!
Roy
On Wed, Jan 16, 2019 at 8:51 AM Bobbie Haynes <ha...@gmail.com> wrote:
> Hi Roy,
> I don't think the Memory Leak issue is related to MAP errors
> .I was also using Reaper in our cluster.I have seen Memory Leak issue (ERROR
> [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED)
> when some of SStables got corrupted because of disk space issue we had when
> compactions were running.
> I guess you have to report this Memory Leak issue to Reaper tool JIRA.
>
> Thanks,
> Bob
>
> On Mon, Jan 14, 2019 at 8:44 AM Roy Burstein <bu...@gmail.com>
> wrote:
>
>> Hi ,
>>
>> We are testing C* 3.11.3 and we have mapping issue and possibly leaked
>> memory.
>> It might be related to our configuration,any ideas would be helpful .
>>
>>
>>
>> Cassandra version: 3.11.3
>> OS: CentOS Linux release 7.4.1708 (Core)
>> Kernel: 3.10.0-957.1.3.el7.x86_64
>> JDK: jdk1.8.0_131
>> Heap: same errors with 16GB / 32GB / 64GB.
>>
>> *We are seeing this errors in production:*
>>
>> *java.io.IOException: Map failed:*
>>
>> ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:5017,1,main]
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
>> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
>> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
>> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
>> Caused by: java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_131]
>> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.3.jar:3.11.3]
>> ... 23 common frames omitted
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_131]
>> ... 24 common frames omitted
>>
>> *LEAK DETECTED:*
>>
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290) was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Index.db was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b90823b) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Data.db was not released before the reference was garbage collected
>> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1710583516:[Memory@[0..3e24), Memory@[0..45e88)] was not released before the reference was garbage collected
>>
>>
>> *Limits of Cassandra process:*
>>
>> [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '\{print $2}'`/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 8388608 unlimited bytes
>> Max core file size 0 unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 32768 32768 processes
>> Max open files 100000 100000 files
>> Max locked memory unlimited unlimited bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 766985 766985 signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>>
>>
>>
>> *max_map_count parameter on OS:*
>>
>> [root@cass063 ~]# sysctl vm.max_map_count
>> vm.max_map_count = 1073741824
>>
>>
>>
>>
>> *cassandra.yaml:*
>>
>> cluster_name: 'Cass Cluster'
>> num_tokens: 256
>> hinted_handoff_enabled: false
>> max_hint_window_in_ms: 10800000
>> hinted_handoff_throttle_in_kb: 1024
>> max_hints_delivery_threads: 2
>> hints_directory: /var/lib/cassandra/hints
>> hints_flush_period_in_ms: 10000
>> max_hints_file_size_in_mb: 128
>> batchlog_replay_throttle_in_kb: 1024
>> authenticator: AllowAllAuthenticator
>> authorizer: AllowAllAuthorizer
>> role_manager: CassandraRoleManager
>> roles_validity_in_ms: 2000
>> permissions_validity_in_ms: 2000
>> credentials_validity_in_ms: 2000
>> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> data_file_directories:
>> - /var/lib/cassandra/data/disk1
>> commitlog_directory: /var/lib/cassandra/data/disk1/commitlog
>> cdc_enabled: false
>> disk_failure_policy: stop
>> commit_failure_policy: stop
>> prepared_statements_cache_size_mb:
>> thrift_prepared_statements_cache_size_mb:
>> key_cache_size_in_mb: 0
>> key_cache_save_period: 3600
>> row_cache_size_in_mb: 0
>> row_cache_save_period: 0
>> counter_cache_size_in_mb:
>> counter_cache_save_period: 7200
>> saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches
>> commitlog_sync: periodic
>> commitlog_sync_period_in_ms: 10000
>> commitlog_segment_size_in_mb: 32
>> seed_provider:
>> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>> parameters:
>> - seeds: "10.110.30.1,10.110.30.2,10.110.30.3"
>> concurrent_reads: 48
>> concurrent_writes: 96
>> concurrent_counter_writes: 32
>> concurrent_materialized_view_writes: 32
>> file_cache_size_in_mb: 10240
>> memtable_offheap_space_in_mb: 10240
>> memtable_cleanup_threshold: 0.1
>> memtable_allocation_type: offheap_buffers
>> commitlog_total_space_in_mb: 8192
>> memtable_flush_writers: 8
>> index_summary_capacity_in_mb:
>> index_summary_resize_interval_in_minutes: 60
>> trickle_fsync: true
>> trickle_fsync_interval_in_kb: 10240
>> storage_port: 7000
>> ssl_storage_port: 7001
>> listen_address: 10.106.62.34
>> start_native_transport: true
>> native_transport_port: 9042
>> start_rpc: false
>> rpc_address: 0.0.0.0
>> rpc_port: 9160
>> broadcast_rpc_address: 10.106.62.34
>> rpc_keepalive: true
>> rpc_server_type: hsha
>> rpc_max_threads: 128
>> thrift_framed_transport_size_in_mb: 15
>> incremental_backups: false
>> snapshot_before_compaction: false
>> auto_snapshot: true
>> column_index_size_in_kb: 64
>> column_index_cache_size_in_kb: 2
>> concurrent_compactors: 32
>> compaction_throughput_mb_per_sec: 500
>> sstable_preemptive_open_interval_in_mb: 50
>> stream_throughput_outbound_megabits_per_sec: 0
>> read_request_timeout_in_ms: 10000
>> range_request_timeout_in_ms: 10000
>> write_request_timeout_in_ms: 60000
>> counter_write_request_timeout_in_ms: 10000
>> cas_contention_timeout_in_ms: 1000
>> truncate_request_timeout_in_ms: 60000
>> request_timeout_in_ms: 10000
>> slow_query_log_timeout_in_ms: 500
>> cross_node_timeout: false
>> phi_convict_threshold: 12
>> endpoint_snitch: GossipingPropertyFileSnitch
>> dynamic_snitch_update_interval_in_ms: 100
>> dynamic_snitch_reset_interval_in_ms: 600000
>> dynamic_snitch_badness_threshold: 0.5
>> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
>> server_encryption_options:
>> internode_encryption: none
>> keystore: conf/.keystore
>> keystore_password: cassandra
>> truststore: conf/.truststore
>> truststore_password: cassandra
>> client_encryption_options:
>> enabled: false
>> optional: false
>> keystore: conf/.keystore
>> keystore_password: cassandra
>> internode_compression: dc
>> inter_dc_tcp_nodelay: false
>> tracetype_query_ttl: 86400
>> tracetype_repair_ttl: 604800
>> enable_user_defined_functions: false
>> enable_scripted_user_defined_functions: false
>> enable_materialized_views: true
>> windows_timer_interval: 1
>> transparent_data_encryption_options:
>> enabled: false
>> chunk_length_kb: 64
>> cipher: AES/CBC/PKCS5Padding
>> key_alias: testing:1
>> key_provider:
>> - class_name: org.apache.cassandra.security.JKSKeyProvider
>> parameters:
>> - keystore: conf/.keystore
>> keystore_password: cassandra
>> store_type: JCEKS
>> key_password: cassandra
>> tombstone_warn_threshold: 1000
>> tombstone_failure_threshold: 100000
>> batch_size_warn_threshold_in_kb: 5
>> batch_size_fail_threshold_in_kb: 50
>> unlogged_batch_across_partitions_warn_threshold: 10
>> compaction_large_partition_warning_threshold_mb: 10
>> gc_warn_threshold_in_ms: 1000
>> back_pressure_enabled: false
>> back_pressure_strategy:
>> - class_name: org.apache.cassandra.net.RateBasedBackPressure
>> parameters:
>> - high_ratio: 0.90
>> factor: 5
>> flow: FAST
>>
>>
>>
>> *A lot of maps, 200K maps of cassandra process,*:
>>
>> [root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '{print $2}'`/maps
>> 239587 /proc/202664/maps
>>
>> Thanks,
>> Roy
>>
>
Re: Cassandra 3.11.3 map errors
Posted by Bobbie Haynes <ha...@gmail.com>.
Hi Roy,
I don't think the Memory Leak issue is related to MAP errors .I
was also using Reaper in our cluster.I have seen Memory Leak issue (ERROR
[Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED)
when some of SStables got corrupted because of disk space issue we had when
compactions were running.
I guess you have to report this Memory Leak issue to Reaper tool JIRA.
Thanks,
Bob
On Mon, Jan 14, 2019 at 8:44 AM Roy Burstein <bu...@gmail.com> wrote:
> Hi ,
>
> We are testing C* 3.11.3 and we have mapping issue and possibly leaked
> memory.
> It might be related to our configuration,any ideas would be helpful .
>
>
>
> Cassandra version: 3.11.3
> OS: CentOS Linux release 7.4.1708 (Core)
> Kernel: 3.10.0-957.1.3.el7.x86_64
> JDK: jdk1.8.0_131
> Heap: same errors with 16GB / 32GB / 64GB.
>
> *We are seeing this errors in production:*
>
> *java.io.IOException: Map failed:*
>
> ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:5017,1,main]
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274) ~[apache-cassandra-3.11.3.jar:3.11.3]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.3.jar:3.11.3]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) ~[na:1.8.0_131]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.3.jar:3.11.3]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_131]
> ... 24 common frames omitted
>
> *LEAK DETECTED:*
>
> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290) was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1179238225:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Index.db was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5b90823b) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@783549664:/var/lib/cassandra/data/disk1/sessions_rawdata/sessions_v2_2019_01_13-19be8e90037011e9a45847402874bbd7/mc-1209-big-Data.db was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6ecdf763) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@1710583516:[Memory@[0..3e24), Memory@[0..45e88)] was not released before the reference was garbage collected
>
>
> *Limits of Cassandra process:*
>
> [root@cass063 ~ ]# cat /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '\{print $2}'`/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size 0 unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 32768 32768 processes
> Max open files 100000 100000 files
> Max locked memory unlimited unlimited bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 766985 766985 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
>
>
> *max_map_count parameter on OS:*
>
> [root@cass063 ~]# sysctl vm.max_map_count
> vm.max_map_count = 1073741824
>
>
>
>
> *cassandra.yaml:*
>
> cluster_name: 'Cass Cluster'
> num_tokens: 256
> hinted_handoff_enabled: false
> max_hint_window_in_ms: 10800000
> hinted_handoff_throttle_in_kb: 1024
> max_hints_delivery_threads: 2
> hints_directory: /var/lib/cassandra/hints
> hints_flush_period_in_ms: 10000
> max_hints_file_size_in_mb: 128
> batchlog_replay_throttle_in_kb: 1024
> authenticator: AllowAllAuthenticator
> authorizer: AllowAllAuthorizer
> role_manager: CassandraRoleManager
> roles_validity_in_ms: 2000
> permissions_validity_in_ms: 2000
> credentials_validity_in_ms: 2000
> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> data_file_directories:
> - /var/lib/cassandra/data/disk1
> commitlog_directory: /var/lib/cassandra/data/disk1/commitlog
> cdc_enabled: false
> disk_failure_policy: stop
> commit_failure_policy: stop
> prepared_statements_cache_size_mb:
> thrift_prepared_statements_cache_size_mb:
> key_cache_size_in_mb: 0
> key_cache_save_period: 3600
> row_cache_size_in_mb: 0
> row_cache_save_period: 0
> counter_cache_size_in_mb:
> counter_cache_save_period: 7200
> saved_caches_directory: /var/lib/cassandra/data/disk1/saved_caches
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 10000
> commitlog_segment_size_in_mb: 32
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "10.110.30.1,10.110.30.2,10.110.30.3"
> concurrent_reads: 48
> concurrent_writes: 96
> concurrent_counter_writes: 32
> concurrent_materialized_view_writes: 32
> file_cache_size_in_mb: 10240
> memtable_offheap_space_in_mb: 10240
> memtable_cleanup_threshold: 0.1
> memtable_allocation_type: offheap_buffers
> commitlog_total_space_in_mb: 8192
> memtable_flush_writers: 8
> index_summary_capacity_in_mb:
> index_summary_resize_interval_in_minutes: 60
> trickle_fsync: true
> trickle_fsync_interval_in_kb: 10240
> storage_port: 7000
> ssl_storage_port: 7001
> listen_address: 10.106.62.34
> start_native_transport: true
> native_transport_port: 9042
> start_rpc: false
> rpc_address: 0.0.0.0
> rpc_port: 9160
> broadcast_rpc_address: 10.106.62.34
> rpc_keepalive: true
> rpc_server_type: hsha
> rpc_max_threads: 128
> thrift_framed_transport_size_in_mb: 15
> incremental_backups: false
> snapshot_before_compaction: false
> auto_snapshot: true
> column_index_size_in_kb: 64
> column_index_cache_size_in_kb: 2
> concurrent_compactors: 32
> compaction_throughput_mb_per_sec: 500
> sstable_preemptive_open_interval_in_mb: 50
> stream_throughput_outbound_megabits_per_sec: 0
> read_request_timeout_in_ms: 10000
> range_request_timeout_in_ms: 10000
> write_request_timeout_in_ms: 60000
> counter_write_request_timeout_in_ms: 10000
> cas_contention_timeout_in_ms: 1000
> truncate_request_timeout_in_ms: 60000
> request_timeout_in_ms: 10000
> slow_query_log_timeout_in_ms: 500
> cross_node_timeout: false
> phi_convict_threshold: 12
> endpoint_snitch: GossipingPropertyFileSnitch
> dynamic_snitch_update_interval_in_ms: 100
> dynamic_snitch_reset_interval_in_ms: 600000
> dynamic_snitch_badness_threshold: 0.5
> request_scheduler: org.apache.cassandra.scheduler.NoScheduler
> server_encryption_options:
> internode_encryption: none
> keystore: conf/.keystore
> keystore_password: cassandra
> truststore: conf/.truststore
> truststore_password: cassandra
> client_encryption_options:
> enabled: false
> optional: false
> keystore: conf/.keystore
> keystore_password: cassandra
> internode_compression: dc
> inter_dc_tcp_nodelay: false
> tracetype_query_ttl: 86400
> tracetype_repair_ttl: 604800
> enable_user_defined_functions: false
> enable_scripted_user_defined_functions: false
> enable_materialized_views: true
> windows_timer_interval: 1
> transparent_data_encryption_options:
> enabled: false
> chunk_length_kb: 64
> cipher: AES/CBC/PKCS5Padding
> key_alias: testing:1
> key_provider:
> - class_name: org.apache.cassandra.security.JKSKeyProvider
> parameters:
> - keystore: conf/.keystore
> keystore_password: cassandra
> store_type: JCEKS
> key_password: cassandra
> tombstone_warn_threshold: 1000
> tombstone_failure_threshold: 100000
> batch_size_warn_threshold_in_kb: 5
> batch_size_fail_threshold_in_kb: 50
> unlogged_batch_across_partitions_warn_threshold: 10
> compaction_large_partition_warning_threshold_mb: 10
> gc_warn_threshold_in_ms: 1000
> back_pressure_enabled: false
> back_pressure_strategy:
> - class_name: org.apache.cassandra.net.RateBasedBackPressure
> parameters:
> - high_ratio: 0.90
> factor: 5
> flow: FAST
>
>
>
> *A lot of maps, 200K maps of cassandra process,*:
>
> [root@cass063 ~]# wc -l /proc/`ps -ef | grep CassandraDaemon | grep -v grep | awk '{print $2}'`/maps
> 239587 /proc/202664/maps
>
> Thanks,
> Roy
>