one node with very high loads

[Senthilvel Rangaswamy <se...@gmail.com>]

We are running 1.1.2 on m1.xlarge with ephemeral store for data. We are
seeing very high loads on one
of the nodes in the ring, 30+.

The node is not doing any compaction, or streaming. Yet, there seems to be
lot of disk i/o activity. What
could be the reason ? If it is high reads, I expect it to be across the
cluster.


 cassandra]$ nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All
time blocked
ReadStage                        32        72       44327505
0                 0
RequestResponseStage              0         0      168321146
0                 0
MutationStage                    32       588      109240939
0                 0
ReadRepairStage                   0         0        3264320
0                 0
ReplicateOnWriteStage             8         8       20080122
0                 0
GossipStage                       0         0        1617877
0                 0
AntiEntropyStage                  0         0             14
0                 0
MigrationStage                    0         0              7
0                 0
MemtablePostFlusher               0         0           1432
0                 0
StreamStage                       0         0              8
0                 0
FlushWriter                       0         0           1432
0                 0
MiscStage                         0         0              0
0                 0
commitlog_archiver                0         0              0
0                 0
AntiEntropySessions               0         0              0
0                 0
InternalResponseStage             0         0              3
0                 0
HintedHandoff                     0         0           1169
0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR             227246
BINARY                       0
READ                   6743268
MUTATION              10662391
REQUEST_RESPONSE            35




-- 
..Senthil

"If there's anything more important than my ego around, I want it
 caught and shot now."
                                                    - Douglas Adams.

Re: one node with very high loads

[Rob Coli <rc...@palominodb.com>]

On Mon, Aug 27, 2012 at 9:25 AM, Senthilvel Rangaswamy
<se...@gmail.com> wrote:
> We are running 1.1.2 on m1.xlarge with ephemeral store for data. We are
> seeing very high loads on one of the nodes in the ring, 30+.

My first hunch would be that you are sending all client requests to
this one node, so it is coordinating 30x as many requests as it
should.

If that's not the case, if I were you I would attempt to determine if
the high i/o is high read or write on the node, via a tool like iotop.
You can also compare the tpstats of two nodes with similar uptimes to
see if your node is performing more of any stage than other members of
its cohort.

Once you determine whether it's read or write, determine which files
are being read or written.. :)

=Rob

-- 
=Robert Coli
AIM&GTALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb