You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ian douglas <ia...@armorgames.com> on 2011/01/03 22:30:39 UTC
How can I correct this Cassandra load imbalance?
Hi everyone,
I've been lurking in the #cassandra IRC channel lately looking for help
on this, but wanted to try the mailing list as well.
We have 3 nodes, and last week it was suggested that I run 'nodetool
move' to reset our token values on the 3 nodes because they were
randomly assigned when starting the nodes the first time.
Our ReplicationFactor is set to 2, and we're using the Random partitioner.
Our row min/max/mean values are mostly the same, but have some small
discrepancies, but for one of our column families, we have a pretty
varied disk space live/total count. We'd expect that after issuing a
"nodetool move" and "nodetool cleanup" on each node (and waiting for
each to complete before repeating on the next node) that the data load
would be more balanced, and give us approximately the same disk load on
all 3 nodes.
nodetool ring:
Address Status Load
Range Ring
113427455640312821154458202477256070485
10.242.142.16 Up 2.66 GB
0 |<--|
10.242.142.111Up 1.24 GB
56713727820156410577229101238628035243 | |
10.242.6.242 Up 5.86 GB
113427455640312821154458202477256070485 |-->|
Disk space used (live/total) and row size min/max/mean for each of the
nodes for the largest ColumnFamily of about a dozen CF's (all are
defined on each node. I added commas for readability:
Node 1:
Column Family: UserGameshareData
Space used (live): 1,992,054,724
Space used (total): 1,992,054,724
Compacted row minimum size: 307
Compacted row maximum size: 123,498
Compacted row mean size: 1,409
Node 2:
Column Family: UserGameshareData
Space used (live): 782,974,992
Space used (total): 806,168,719
Compacted row minimum size: 306
Compacted row maximum size: 88,379
Compacted row mean size: 1,405
Node 3:
Column Family: UserGameshareData
Space used (live): 4,435,011,932
Space used (total): 4,435,011,932
Compacted row minimum size: 306
Compacted row maximum size: 68,636
Compacted row mean size: 1,387
We also have a cron job running every night that runs a 'nodetool
cleanup' on one of the nodes (day of year % number of nodes).
I'm happy to share any additional information that can help us balance
things a little bit better. I had debated setting our ReplicationFactor
to 3 to see if our nodes eventually copied data to every node.
Cheers,
Ian Douglas, Armor Games
Re: How can I correct this Cassandra load imbalance?
Posted by Peter Schuller <pe...@infidyne.com>.
> Node 1:
> strings/grep/wc: 979,123
> space used: 2,061,497,786
>
> Node 2:
> strings/grep/wc: 443,558
> space used: 854,213,778
>
> Node 3:
> strings/grep/wc: 2,103,294
> space used: 4,505,048,405
Was this figured out? Could it be so simple as a compaction
discrepancy (did you try running compact on them all)? Given
sufficient amounts of overwrites/deletions, variations in compacting
timing could account for differences.
--
/ Peter Schuller
Re: How can I correct this Cassandra load imbalance?
Posted by ian douglas <ia...@armorgames.com>.
Thanks Richard!
strings UserGameshareData-*-Index.db | grep ':' | wc -l
Node 1:
strings/grep/wc: 979,123
space used: 2,061,497,786
Node 2:
strings/grep/wc: 443,558
space used: 854,213,778
Node 3:
strings/grep/wc: 2,103,294
space used: 4,505,048,405
On 01/06/2011 11:43 AM, Robert Coli wrote:
> On Thu, Jan 6, 2011 at 10:50 AM, ian douglas<ia...@armorgames.com> wrote:
>> Is there any way to determine via a "nodetool cfstats" (or similar) how many
>> rows we have per column family to help answer your second question a little
>> better?
> In 0.6, you can get an (inexact, but probably sufficient for this
> purpose) estimate of this by counting the number of lines in :
>
> "strings ColumnFamily-*-Index.db |grep :"
>
> =Rob
Re: How can I correct this Cassandra load imbalance?
Posted by Robert Coli <rc...@digg.com>.
On Thu, Jan 6, 2011 at 10:50 AM, ian douglas <ia...@armorgames.com> wrote:
> Is there any way to determine via a "nodetool cfstats" (or similar) how many
> rows we have per column family to help answer your second question a little
> better?
In 0.6, you can get an (inexact, but probably sufficient for this
purpose) estimate of this by counting the number of lines in :
"strings ColumnFamily-*-Index.db |grep :"
=Rob
Re: How can I correct this Cassandra load imbalance?
Posted by ian douglas <ia...@armorgames.com>.
We're currently on 0.6.0 waiting for the full release of 0.7 before we
upgrade. We have other Thrift/PHP code to update whenever we upgrade
Cassandra, so we don't want to upgrade to a release candidate on our
production system.
We *did* have a problem with a column family setup where we had few rows
(probably hundreds?), and those few rows exceeding 100MB in size, so we
migrated that column family's data to a new column family and stored
that old data into what would now be hundreds of thousands of rows.
Our largest row size right now is in the ballpark of a few hundred
kilobytes.
Is there any way to determine via a "nodetool cfstats" (or similar) how
many rows we have per column family to help answer your second question
a little better? I do know from our migration that we created something
like 30,000 smaller rows for each of the bigger rows, and then we
deleted that old column family from our nodes (first by removing it from
the XML configuration, then by deleting files at the OS level). When our
migration finished, we *still* saw this large imbalance, which is what
prompted my questions, and led to "nodetool move" to reset our token
values, etc., but even running cleanups, flushes and repairs on each
node individually, we're still left with this imbalanced load.
Thanks for your help. Let me know if there's any additional information
I can give.
On 01/06/2011 10:39 AM, Peter Schuller wrote:
>> I posted row sizes (min/max/mean) of our largest data set in my original
>> message, but had zero responses on the mailing list. The folks in IRC told
>> me to wait it out, see if to rebalanced on its own (it didn't), or to run a
>> repair on each node one at a time (didn't help), and that it wasn't a big
>> concern until we had "dozens of GBs" worth of data.
> Ok. It may not be a concern practically right now, but an unexplained
> imbalance is not good. First off, is this the very latest 0.6 release
> or else one of the 0.7 rc:s, or is this an old 0.6? Not that I
> remember off hand whether there were any bugs fixed in the 0.6 series
> that would explain this particular behavior, but it's probably a good
> start to ask if you have the latest version.
>
> Also, you mentioned originally that "Our row min/max/mean values are
> mostly the same". I'm not entirely positive to what you are referring;
> the important points I wanted to ask about are:
>
> (1) Do you have "many" keys (say, thousands or more) so that there
> should be no statistically significant imbalance between the nodes in
> terms of the *number* of rows?
>
> (2) How sure are you about the distribution of row sizes; is it
> possible you have a small number of very large rows that are screwing
> up the statistics?
>
Re: How can I correct this Cassandra load imbalance?
Posted by Peter Schuller <pe...@infidyne.com>.
> I posted row sizes (min/max/mean) of our largest data set in my original
> message, but had zero responses on the mailing list. The folks in IRC told
> me to wait it out, see if to rebalanced on its own (it didn't), or to run a
> repair on each node one at a time (didn't help), and that it wasn't a big
> concern until we had "dozens of GBs" worth of data.
Ok. It may not be a concern practically right now, but an unexplained
imbalance is not good. First off, is this the very latest 0.6 release
or else one of the 0.7 rc:s, or is this an old 0.6? Not that I
remember off hand whether there were any bugs fixed in the 0.6 series
that would explain this particular behavior, but it's probably a good
start to ask if you have the latest version.
Also, you mentioned originally that "Our row min/max/mean values are
mostly the same". I'm not entirely positive to what you are referring;
the important points I wanted to ask about are:
(1) Do you have "many" keys (say, thousands or more) so that there
should be no statistically significant imbalance between the nodes in
terms of the *number* of rows?
(2) How sure are you about the distribution of row sizes; is it
possible you have a small number of very large rows that are screwing
up the statistics?
--
/ Peter Schuller
Re: How can I correct this Cassandra load imbalance?
Posted by ian douglas <ia...@armorgames.com>.
Hi Peter,
I posted row sizes (min/max/mean) of our largest data set in my original
message, but had zero responses on the mailing list. The folks in IRC
told me to wait it out, see if to rebalanced on its own (it didn't), or
to run a repair on each node one at a time (didn't help), and that it
wasn't a big concern until we had "dozens of GBs" worth of data.
On 01/06/2011 10:08 AM, Peter Schuller wrote:
>> I've been lurking in the #cassandra IRC channel lately looking for help on
>> this, but wanted to try the mailing list as well.
> Was this resolved off-list, and if so what was the problem?
>
> I don't see a problem in your description to explain the imbalance,
> assuming you don't have extreme variation in the size of rows (or very
> few rows). I was hoping someone else would spot something but the
> thread seems dead still :)
>
Re: How can I correct this Cassandra load imbalance?
Posted by Peter Schuller <pe...@infidyne.com>.
> I've been lurking in the #cassandra IRC channel lately looking for help on
> this, but wanted to try the mailing list as well.
Was this resolved off-list, and if so what was the problem?
I don't see a problem in your description to explain the imbalance,
assuming you don't have extreme variation in the size of rows (or very
few rows). I was hoping someone else would spot something but the
thread seems dead still :)
--
/ Peter Schuller