Cluster processing not balanced

[Rene Kochen <re...@schange.com>]

Hi All,

I have the following situation:

- Cassandra 1.0.11
- A 6 node cluster
- Random partitioner
- Tokens are balanced (according to node-tool)
- Data-load is balanced (according to node-tool)

I have a customers column-family with 1000000 customers. I also have a test
client which requests random customers from the cluster in a round-robin
fashion, i.e. each request is targeted to a different machine and each
request contains a random customer-id (a number in the range of 0 -
1000000).

What I see is that one node (node 4) is only proxying. Node 4 does receive
requests. I can see that in the storage-proxy counters. However, I see in
the Customers column-family a zero in the ReadCount counter (local reads).
Node 4 should also serve reads because I have 1000000 customers in the
cluster, the cluster is balanced and I do requests with random
customer-ids. In fact, two of the four nodes have this behavior. The result
is that the processing load within the cluster is not balanced.

I have never witnessed this behavior before.

Am I missing something?

Thanks,

Rene

Re: Cluster processing not balanced

[Rene Kochen <re...@schange.com>]

I found the issue.

Stupid mistake.

Although I have 1000000 customers in the cluster, I do a get_range_slices
to get the first 10000 customers. I use these customers as a seed for my
test program. I random select customers from that set and fire requests at
the cluster. Those 10000 customers all reside on the first node (actually
the first three nodes because of the replication factor of three).

Thanks,

Rene


2013/6/28 Rene Kochen <re...@schange.com>

> Hi All,
>
> I have the following situation:
>
> - Cassandra 1.0.11
> - A 6 node cluster
> - Random partitioner
> - Tokens are balanced (according to node-tool)
> - Data-load is balanced (according to node-tool)
>
> I have a customers column-family with 1000000 customers. I also have a
> test client which requests random customers from the cluster in a
> round-robin fashion, i.e. each request is targeted to a different machine
> and each request contains a random customer-id (a number in the range of 0
> - 1000000).
>
> What I see is that one node (node 4) is only proxying. Node 4 does receive
> requests. I can see that in the storage-proxy counters. However, I see in
> the Customers column-family a zero in the ReadCount counter (local reads).
> Node 4 should also serve reads because I have 1000000 customers in the
> cluster, the cluster is balanced and I do requests with random
> customer-ids. In fact, two of the four nodes have this behavior. The result
> is that the processing load within the cluster is not balanced.
>
> I have never witnessed this behavior before.
>
> Am I missing something?
>
> Thanks,
>
> Rene
>