You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maxim Kramarenko <ma...@trackstudio.com> on 2010/05/23 09:02:21 UTC

Very slow bootstrap

Hello!

I have to 2 node cluster:
[root@cas2 bin]# sh nodetool -h localhost ring
Address       Status     Load          Range 
           Ring
 
47311629213338587668692978196312911227
172.19.0.32   Up         80.06 GB 
15179341530891532433339729554474051162     |<--|
172.19.0.30   Up         169.42 GB 
47311629213338587668692978196312911227     |-->|

Also, I am trying to add another node to a cluser, node 3.

The problems is: node 3 stay in bootstrapping stage 30+ hrs for now, 
load only 5 GB data and this process makes node 1 (and whole cluster) 
unusable for reading data.

[root@cas3 bin]# sh nodetool -h localhost info
31243541907542829729148671132285303944
Load             : 5.54 GB
Generation No    : 1274480008
Uptime (seconds) : 117681
Heap Memory (MB) : 674.74 / 4094.75
[root@cas3 bin]# sh nodetool -h localhost streams
Mode: Bootstrapping
Not sending any streams.
Not receiving any streams.


Each server have 8-24 GB RAM (run it with -Xmx4g or -Xmx20g), 16 cores. 
Why bootstrapping so slow ?

Probably, something wrong with my setup ? Will adding more nodes helps? 
Larger replication factor ?

Re: Very slow bootstrap

Posted by Jonathan Ellis <jb...@gmail.com>.
If you check the other nodes you will probably see that one of them
thinks it is still trying to send to node 3.  You will probably need
to restart that node, and then retry the bootstrap from 3.

Alternatively you could force 3 into the ring by restarting w/
autobootstrap off (be sure to set InitialToken to what it was
bootstrapping to, it's in the log) and then run repair on it to make
sure it got everything.

2010/5/23 Maxim Kramarenko <ma...@trackstudio.com>:
> Hello!
>
> I have to 2 node cluster:
> [root@cas2 bin]# sh nodetool -h localhost ring
> Address       Status     Load          Range           Ring
>
> 47311629213338587668692978196312911227
> 172.19.0.32   Up         80.06 GB 15179341530891532433339729554474051162
> |<--|
> 172.19.0.30   Up         169.42 GB 47311629213338587668692978196312911227
>   |-->|
>
> Also, I am trying to add another node to a cluser, node 3.
>
> The problems is: node 3 stay in bootstrapping stage 30+ hrs for now, load
> only 5 GB data and this process makes node 1 (and whole cluster) unusable
> for reading data.
>
> [root@cas3 bin]# sh nodetool -h localhost info
> 31243541907542829729148671132285303944
> Load             : 5.54 GB
> Generation No    : 1274480008
> Uptime (seconds) : 117681
> Heap Memory (MB) : 674.74 / 4094.75
> [root@cas3 bin]# sh nodetool -h localhost streams
> Mode: Bootstrapping
> Not sending any streams.
> Not receiving any streams.
>
>
> Each server have 8-24 GB RAM (run it with -Xmx4g or -Xmx20g), 16 cores. Why
> bootstrapping so slow ?
>
> Probably, something wrong with my setup ? Will adding more nodes helps?
> Larger replication factor ?
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com