[jira] [Created] (CASSANDRA-10291) Bootstrap hangs on adding new node

["Ara Sadoyan (JIRA)" <ji...@apache.org>]

Ara Sadoyan created CASSANDRA-10291:
---------------------------------------

             Summary: Bootstrap hangs on adding new node 
                 Key: CASSANDRA-10291
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10291
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Debian 7 64 bit
HotSpot JDK 1.7.0_79
Cassandra-2.2.1 via apt-get 
1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive 
            Reporter: Ara Sadoyan
            Priority: Blocker
         Attachments: cassandra.log

Adding new node in heavy loaded environment freeze bootstrap. No errors are reported in log files.  Some of other other nodes throws "String didn't validate" error, but I;m not sure that this is related. 
After restarting node it start bootstrap again and hangs after some time . 

nodetool netstats shows : 
/data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701   bytes(27%)  received  from idx:0/192.168.220.16
/data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671   bytes(38%)  received  from idx:0/192.168.220.22
/data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398   bytes(48%)  received  from idx:0/192.168.220.12
/data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610  bytes(85%)  received  from idx:0/192.168.220.18
/data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085   bytes(48%)  received  from idx:0/192.168.220.20

but listing files on local FS shows "No such file or directory"

This happens only if there is significant amount of data. I have 1.5 TB per node on 13 node cluster, we use STCS compaction strategy and flat network topology . 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)