[jira] [Created] (CASSANDRA-11827) Node is not getting back to UN

["Gaurav Sharma (JIRA)" <ji...@apache.org>]

Gaurav Sharma created CASSANDRA-11827:
-----------------------------------------

             Summary: Node is not getting back to UN
                 Key: CASSANDRA-11827
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11827
             Project: Cassandra
          Issue Type: Task
          Components: Configuration, Observability
         Environment: Ubuntu 12.04.5 LTS
            Reporter: Gaurav Sharma
             Fix For: 2.1.x


Hi , 
One of node went down due to high CPU/Memory utilization , now when we are trying to start it again , it is not joining cluster . On other nodes in system.log we can see below error ::

INFO  [HANDSHAKE-/172.16.1.71] 2016-05-18 15:21:27,011 OutboundTcpConnection.java:485 - Handshaking version with /172.16.1.71
DEBUG [MessagingService-Outgoing-/172.16.1.71] 2016-05-18 15:21:32,011 OutboundTcpConnection.java:412 - Target max version is -2147483648; no version information yet, will retry

INFO  [MessagingService-Outgoing-/172.16.1.71] 2016-05-18 15:21:32,011 YamlConfigurationLoader.java:92 - Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  [MessagingService-Outgoing-/172.16.1.71] 2016-05-18 15:21:32,013 YamlConfigurationLoader.java:135 - Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAl
lAuthorizer; auto_snapshot=false; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=172.16.1.73; cas_contention_timeout_in_ms=5000; clien
t_encryption_options=<REDACTED>; cluster_name=wynkframework; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/mnt/cassandra/commitlog; commitlog_segment_
size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=32; concurrent_compa
ctors=4; concurrent_counter_writes=48; concurrent_reads=48; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=
10000; cross_node_timeout=true; data_file_directories=[/var/lib/cassandra/data, /data/cassandra, /data1/cassandra]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; d
ynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=false; hinted_handoff_throttle_in_kb=1024;
incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_peri
od=14400; key_cache_size_in_mb=1024; listen_address=172.16.1.73; max_hint_window_in_ms=10800000; max_hints_delivery_threads=4; memory_allocator=JEMallocAllocator; memtable_allocatio
n_type=offheap_objects; memtable_cleanup_threshold=0.11; memtable_flush_writers=4; native_transport_max_concurrent_connections=100; native_transport_port=9042; num_tokens=256; parti
tioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=30000; read_request_timeout_in_ms=20000; request_scheduler=org.apach
e.cassandra.scheduler.NoScheduler; request_timeout_in_ms=30000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=172.16.1.73; rpc_keepalive=true; rpc_max_threads=2048; r
pc_min_threads=16; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/mnt/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvide
r, parameters=[{seeds=172.16.1.103,172.16.1.73}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_i
n_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tr
ickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=30000]
TRACE [HANDSHAKE-/172.16.1.71] 2016-05-18 15:21:32,015 OutboundTcpConnection.java:492 - Cannot handshake version with /172.16.1.71
java.nio.channels.AsynchronousCloseException: null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)