the problem of elasticity

[魏金仙 <se...@126.com>]

I've no idea why it doesn't work well.
We are testing Elasticity of Cassandra 0.6.6.We choose orderPreservingPartitioner and set replicationFactor as 2.
 We start from 6-server cluster(node A, B,C,D,E,F), which is load balanced. roughly every node has 12GB. we then add node G between A and B. the InitialToken of G is set to divide the data firstly owned by B, which means if we start with 7-server cluster, node G and node B will gain 6GB(other nodes: 12GB).
The problem is that
1.we go  on with running the workload(read/write=1), then data of node G  increases very slowly and data of node B doesn't decrease apparently. exactly, after three and a half hours, node G gets only 2.85GB.
2.we then stop running the workload and wait for 2 hours. the result is  that  no data is populated from node B  to node G .
that is, 5.5hours later after adding a new node, we haven't found any data moving from Node B to the new Node named G.   
we tried the monitor tool——streams on the new node, result in "nothing streaming to ..."and "nothing streaming from.."
Did I miss something?

Re: the problem of elasticity

[Peter Schuller <pe...@infidyne.com>]

> In general if you think the data is not distributed correctly run nodetool repair on the node. http://wiki.apache.org/cassandra/Operations#Repairing_missing_or_inconsistent_data

And before expecting the old node to throw away it's data, 'nodetool
cleanup' is required (but don't do this while you still have concerns
that the data has not propagated correctly).

-- 
/ Peter Schuller

Re: the problem of elasticity

[Aaron Morton <aa...@thelastpickle.com>]

If you can supply the output from nodetool repair it will help. As well as checking the streams you can check the nodetool tpstats.

Also, if you can provide the the steps you used to add the new node.

In general if you think the data is not distributed correctly run nodetool repair on the node. http://wiki.apache.org/cassandra/Operations#Repairing_missing_or_inconsistent_data

Hope that helps
Aaron

On 22/01/2011, at 2:31 AM, 魏金仙 <se...@126.com> wrote:

> I've no idea why it doesn't work well. 
> We are testing Elasticity of Cassandra 0.6.6.We choose orderPreservingPartitioner and set replicationFactor as 2.
>  We start from 6-server cluster(node A, B,C,D,E,F), which is load balanced. roughly every node has 12GB. we then add node G between A and B. the InitialToken of G is set to divide the data firstly owned by B, which means if we start with 7-server cluster, node G and node B will gain 6GB(other nodes: 12GB).
> The problem is that
> 1.we go  on with running the workload(read/write=1), then data of node G  increases very slowly and data of node B doesn't decrease apparently. exactly, after three and a half hours, node G gets only 2.85GB.
> 2.we then stop running the workload and wait for 2 hours. the result is  that  no data is populated from node B  to node G .
> that is, 5.5hours later after adding a new node, we haven't found any data moving from Node B to the new Node named G.    
> we tried the monitor tool——streams on the new node, result in "nothing streaming to ..."and "nothing streaming from.."
> Did I miss something?
> 
> 
> 
> 
> 
> 
> 

Re: the problem of elasticity

[Robert Coli <rc...@digg.com>]

On 1/21/11, 魏金仙 <se...@126.com> wrote:
> We are testing Elasticity of Cassandra 0.6.6.We choose
> orderPreservingPartitioner and set replicationFactor as 2.
> [ ... we tried to change our topology ... ]
> we tried the monitor tool——streams on the new node, result in "nothing
> streaming to ..."and "nothing streaming from.."

https://issues.apache.org/jira/browse/CASSANDRA-1676

If you do not have this patch and intra-node streaming operations ever
work, you are lucky.

For what it's worth, it's relatively easy to patch just 1676 against 0.6.6.

See the following threads for more details :

http://www.mail-archive.com/user@cassandra.apache.org/msg07106.html
http://www.mail-archive.com/user@cassandra.apache.org/msg08379.html
http://www.mail-archive.com/user@cassandra.apache.org/msg08441.html

=Rob