You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Daning Wang <da...@netseer.com> on 2012/01/06 19:11:24 UTC
Pending on ReadStage
Hi all,
We have 5 nodes cluster(0.8.6), but the performance from one node is way
behind others, I checked tpstats, It always show non-zero pending
ReadStage, I don't see this problem on other nodes.
What caused the problem? I/O? Memory? Cpu usage is still low. How to fix
this problem?
~/bin/nodetool -h localhost tpstats
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 11 15 56960
0 0
RequestResponseStage 0 0 606695
0 0
MutationStage 0 0 538634
0 0
ReadRepairStage 0 0 199997
0 0
ReplicateOnWriteStage 0 0 0
0 0
GossipStage 0 0 5734
0 0
AntiEntropyStage 0 0 0
0 0
MigrationStage 0 0 0
0 0
MemtablePostFlusher 0 0 7
0 0
StreamStage 0 0 0
0 0
FlushWriter 0 0 8
0 0
MiscStage 0 0 0
0 0
FlushSorter 0 0 0
0 0
InternalResponseStage 0 0 0
0 0
HintedHandoff 1 4 0
0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 9082
MUTATION 0
REQUEST_RESPONSE 0
Thanks you in advance.
Daning
Re: Pending on ReadStage
Posted by aaron morton <aa...@thelastpickle.com>.
It may be IO, it's a good place to look
http://spyced.blogspot.com/2010/01/linux-performance-basics.html
Look at the logs, when messages are dropped you will see logs from StatusLogger that include TP stats and some other things.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 8:35 AM, Daning Wang wrote:
> Thanks for your reply.
>
> Nodes are equally balanced. and it is RandomPartitioner. I think that machine is slower, Are you saying it is IO issue?
>
> Daning
>
> On Fri, Jan 6, 2012 at 10:25 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> Are all your nodes equally balanced in terms of read requests? Are you
> using RandomPartitioner? Are you reading using indexes?
>
> First thing you can do is compare iostat -x output between the 2 nodes
> to rule out any io issues assuming your read requests are equally
> balanced.
>
> On Fri, Jan 6, 2012 at 10:11 AM, Daning Wang <da...@netseer.com> wrote:
> > Hi all,
> >
> > We have 5 nodes cluster(0.8.6), but the performance from one node is way
> > behind others, I checked tpstats, It always show non-zero pending ReadStage,
> > I don't see this problem on other nodes.
> >
> > What caused the problem? I/O? Memory? Cpu usage is still low. How to fix
> > this problem?
> >
> > ~/bin/nodetool -h localhost tpstats
> > Pool Name Active Pending Completed Blocked All
> > time blocked
> > ReadStage 11 15 56960
> > 0 0
> > RequestResponseStage 0 0 606695
> > 0 0
> > MutationStage 0 0 538634
> > 0 0
> > ReadRepairStage 0 0 199997
> > 0 0
> > ReplicateOnWriteStage 0 0 0
> > 0 0
> > GossipStage 0 0 5734
> > 0 0
> > AntiEntropyStage 0 0 0
> > 0 0
> > MigrationStage 0 0 0
> > 0 0
> > MemtablePostFlusher 0 0 7
> > 0 0
> > StreamStage 0 0 0
> > 0 0
> > FlushWriter 0 0 8
> > 0 0
> > MiscStage 0 0 0
> > 0 0
> > FlushSorter 0 0 0
> > 0 0
> > InternalResponseStage 0 0 0
> > 0 0
> > HintedHandoff 1 4 0
> > 0 0
> >
> > Message type Dropped
> > RANGE_SLICE 0
> > READ_REPAIR 0
> > BINARY 0
> > READ 9082
> > MUTATION 0
> > REQUEST_RESPONSE 0
> >
> > Thanks you in advance.
> >
> > Daning
> >
>
Re: Pending on ReadStage
Posted by Daning Wang <da...@netseer.com>.
Thanks for your reply.
Nodes are equally balanced. and it is RandomPartitioner. I think that
machine is slower, Are you saying it is IO issue?
Daning
On Fri, Jan 6, 2012 at 10:25 AM, Mohit Anchlia <mo...@gmail.com>wrote:
> Are all your nodes equally balanced in terms of read requests? Are you
> using RandomPartitioner? Are you reading using indexes?
>
> First thing you can do is compare iostat -x output between the 2 nodes
> to rule out any io issues assuming your read requests are equally
> balanced.
>
> On Fri, Jan 6, 2012 at 10:11 AM, Daning Wang <da...@netseer.com> wrote:
> > Hi all,
> >
> > We have 5 nodes cluster(0.8.6), but the performance from one node is way
> > behind others, I checked tpstats, It always show non-zero pending
> ReadStage,
> > I don't see this problem on other nodes.
> >
> > What caused the problem? I/O? Memory? Cpu usage is still low. How to fix
> > this problem?
> >
> > ~/bin/nodetool -h localhost tpstats
> > Pool Name Active Pending Completed Blocked
> All
> > time blocked
> > ReadStage 11 15 56960
> > 0 0
> > RequestResponseStage 0 0 606695
> > 0 0
> > MutationStage 0 0 538634
> > 0 0
> > ReadRepairStage 0 0 199997
> > 0 0
> > ReplicateOnWriteStage 0 0 0
> > 0 0
> > GossipStage 0 0 5734
> > 0 0
> > AntiEntropyStage 0 0 0
> > 0 0
> > MigrationStage 0 0 0
> > 0 0
> > MemtablePostFlusher 0 0 7
> > 0 0
> > StreamStage 0 0 0
> > 0 0
> > FlushWriter 0 0 8
> > 0 0
> > MiscStage 0 0 0
> > 0 0
> > FlushSorter 0 0 0
> > 0 0
> > InternalResponseStage 0 0 0
> > 0 0
> > HintedHandoff 1 4 0
> > 0 0
> >
> > Message type Dropped
> > RANGE_SLICE 0
> > READ_REPAIR 0
> > BINARY 0
> > READ 9082
> > MUTATION 0
> > REQUEST_RESPONSE 0
> >
> > Thanks you in advance.
> >
> > Daning
> >
>
Re: Pending on ReadStage
Posted by Mohit Anchlia <mo...@gmail.com>.
Are all your nodes equally balanced in terms of read requests? Are you
using RandomPartitioner? Are you reading using indexes?
First thing you can do is compare iostat -x output between the 2 nodes
to rule out any io issues assuming your read requests are equally
balanced.
On Fri, Jan 6, 2012 at 10:11 AM, Daning Wang <da...@netseer.com> wrote:
> Hi all,
>
> We have 5 nodes cluster(0.8.6), but the performance from one node is way
> behind others, I checked tpstats, It always show non-zero pending ReadStage,
> I don't see this problem on other nodes.
>
> What caused the problem? I/O? Memory? Cpu usage is still low. How to fix
> this problem?
>
> ~/bin/nodetool -h localhost tpstats
> Pool Name Active Pending Completed Blocked All
> time blocked
> ReadStage 11 15 56960
> 0 0
> RequestResponseStage 0 0 606695
> 0 0
> MutationStage 0 0 538634
> 0 0
> ReadRepairStage 0 0 199997
> 0 0
> ReplicateOnWriteStage 0 0 0
> 0 0
> GossipStage 0 0 5734
> 0 0
> AntiEntropyStage 0 0 0
> 0 0
> MigrationStage 0 0 0
> 0 0
> MemtablePostFlusher 0 0 7
> 0 0
> StreamStage 0 0 0
> 0 0
> FlushWriter 0 0 8
> 0 0
> MiscStage 0 0 0
> 0 0
> FlushSorter 0 0 0
> 0 0
> InternalResponseStage 0 0 0
> 0 0
> HintedHandoff 1 4 0
> 0 0
>
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> BINARY 0
> READ 9082
> MUTATION 0
> REQUEST_RESPONSE 0
>
> Thanks you in advance.
>
> Daning
>