You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ian Danforth <id...@numenta.com> on 2011/08/11 02:17:07 UTC
tpstats confusion
Hello all!
Thanks for taking the time to read this, I'm new to Cassandra and really
want to get it working :)
Below you'll find the output from tpstats on the three nodes in my .8.1
cluster. As you'll note from ring (also below), my cluster appears balanced.
However, please note that NODE1 is the only node that has completed any
ReplicateOnWriteStage actions. I've been having a lot of trouble with RoW
and seen a single node in my various test clusters get hugely backed up on
this stage to the point where it slows all writes to a crawl.
I'm using counters in a column family defined like this:
==== KEYSPACE/CF ====
create keyspace Keyspace1
with strategy_options=[{replication_factor:3}]
and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy';
use Keyspace1;
create column family TwitterTest
with default_validation_class = CounterColumnType
and replicate_on_write = true;
====
With any given batch_insert() I'm incrementing ~17,000 counters. I've backed
off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
elsewhere that ANY or ONE could overwhelm the system.
I hope I'm just doing something stupid and one you experienced folks can
point out my error.
Thanks in advance!
Ian
==== MORE DATA ====
Address DC Rack Status State Load Owns
Token
113427455640312821154458202477256070484
10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
33.33% 0
10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
33.33% 56713727820156410577229101238628035242
10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
33.33% 113427455640312821154458202477256070484
SEED
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 1479839
MutationStage 0 0 1479908
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 0
GossipStage 0 0 5176
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 28
StreamStage 0 0 0
FlushWriter 0 0 28
MiscStage 0 0 2
FlushSorter 0 0 0
InternalResponseStage 0 0 363
HintedHandoff 0 0 0
NODE1
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 2960208
MutationStage 0 0 1480104
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 1480104
GossipStage 0 0 4151
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 31
StreamStage 0 0 8
FlushWriter 0 0 31
MiscStage 0 0 8
FlushSorter 0 0 0
InternalResponseStage 0 0 1
HintedHandoff 0 0 0
NODE2
Pool Name Active Pending Completed
ReadStage 0 0 7
RequestResponseStage 0 0 0
MutationStage 0 0 1481530
ReadRepairStage 0 0 0
ReplicateOnWriteStage 0 0 0
GossipStage 0 0 4201
AntiEntropyStage 0 0 0
MigrationStage 0 0 128
MemtablePostFlusher 0 0 31
StreamStage 0 0 6
FlushWriter 0 0 31
MiscStage 0 0 6
FlushSorter 0 0 0
InternalResponseStage 0 0 1
HintedHandoff 0 0 0
Re: tpstats confusion
Posted by Jonathan Ellis <jb...@gmail.com>.
are you i/o bound? I believe counter ROW needs to perform a read of
the old value.
On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com> wrote:
> Hello all!
> Thanks for taking the time to read this, I'm new to Cassandra and really
> want to get it working :)
> Below you'll find the output from tpstats on the three nodes in my .8.1
> cluster. As you'll note from ring (also below), my cluster appears balanced.
> However, please note that NODE1 is the only node that has completed any
> ReplicateOnWriteStage actions. I've been having a lot of trouble with RoW
> and seen a single node in my various test clusters get hugely backed up on
> this stage to the point where it slows all writes to a crawl.
> I'm using counters in a column family defined like this:
> ==== KEYSPACE/CF ====
> create keyspace Keyspace1
> with strategy_options=[{replication_factor:3}]
> and placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy';
> use Keyspace1;
> create column family TwitterTest
> with default_validation_class = CounterColumnType
> and replicate_on_write = true;
> ====
>
> With any given batch_insert() I'm incrementing ~17,000 counters. I've backed
> off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
> elsewhere that ANY or ONE could overwhelm the system.
>
> I hope I'm just doing something stupid and one you experienced folks can
> point out my error.
> Thanks in advance!
> Ian
> ==== MORE DATA ====
> Address DC Rack Status State Load Owns
> Token
>
> 113427455640312821154458202477256070484
> 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
> 33.33% 0
> 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
> 33.33% 56713727820156410577229101238628035242
> 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
> 33.33% 113427455640312821154458202477256070484
>
>
>
> SEED
> Pool Name Active Pending Completed
> ReadStage 0 0 7
> RequestResponseStage 0 0 1479839
> MutationStage 0 0 1479908
> ReadRepairStage 0 0 0
> ReplicateOnWriteStage 0 0 0
> GossipStage 0 0 5176
> AntiEntropyStage 0 0 0
> MigrationStage 0 0 128
> MemtablePostFlusher 0 0 28
> StreamStage 0 0 0
> FlushWriter 0 0 28
> MiscStage 0 0 2
> FlushSorter 0 0 0
> InternalResponseStage 0 0 363
> HintedHandoff 0 0 0
> NODE1
> Pool Name Active Pending Completed
> ReadStage 0 0 7
> RequestResponseStage 0 0 2960208
> MutationStage 0 0 1480104
> ReadRepairStage 0 0 0
> ReplicateOnWriteStage 0 0 1480104
> GossipStage 0 0 4151
> AntiEntropyStage 0 0 0
> MigrationStage 0 0 128
> MemtablePostFlusher 0 0 31
> StreamStage 0 0 8
> FlushWriter 0 0 31
> MiscStage 0 0 8
> FlushSorter 0 0 0
> InternalResponseStage 0 0 1
> HintedHandoff 0 0 0
> NODE2
> Pool Name Active Pending Completed
> ReadStage 0 0 7
> RequestResponseStage 0 0 0
> MutationStage 0 0 1481530
> ReadRepairStage 0 0 0
> ReplicateOnWriteStage 0 0 0
> GossipStage 0 0 4201
> AntiEntropyStage 0 0 0
> MigrationStage 0 0 128
> MemtablePostFlusher 0 0 31
> StreamStage 0 0 6
> FlushWriter 0 0 31
> MiscStage 0 0 6
> FlushSorter 0 0 0
> InternalResponseStage 0 0 1
> HintedHandoff 0 0 0
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: tpstats confusion
Posted by Jonathan Ellis <jb...@gmail.com>.
https://issues.apache.org/jira/browse/CASSANDRA-2889
On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <id...@numenta.com> wrote:
> I don't think so, perhaps more succinctly, why would ROW actions only be
> performed on a single node?
> Ian
>
> On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> are you i/o bound? I believe counter ROW needs to perform a read of
>> the old value.
>>
>> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
>> wrote:
>> > Hello all!
>> > Thanks for taking the time to read this, I'm new to Cassandra and really
>> > want to get it working :)
>> > Below you'll find the output from tpstats on the three nodes in my .8.1
>> > cluster. As you'll note from ring (also below), my cluster appears
>> > balanced.
>> > However, please note that NODE1 is the only node that has completed any
>> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> > RoW
>> > and seen a single node in my various test clusters get hugely backed up
>> > on
>> > this stage to the point where it slows all writes to a crawl.
>> > I'm using counters in a column family defined like this:
>> > ==== KEYSPACE/CF ====
>> > create keyspace Keyspace1
>> > with strategy_options=[{replication_factor:3}]
>> > and placement_strategy =
>> > 'org.apache.cassandra.locator.SimpleStrategy';
>> > use Keyspace1;
>> > create column family TwitterTest
>> > with default_validation_class = CounterColumnType
>> > and replicate_on_write = true;
>> > ====
>> >
>> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> > backed
>> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
>> > elsewhere that ANY or ONE could overwhelm the system.
>> >
>> > I hope I'm just doing something stupid and one you experienced folks can
>> > point out my error.
>> > Thanks in advance!
>> > Ian
>> > ==== MORE DATA ====
>> > Address DC Rack Status State Load
>> > Owns
>> > Token
>> >
>> > 113427455640312821154458202477256070484
>> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
>> > 33.33% 0
>> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
>> > 33.33% 56713727820156410577229101238628035242
>> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
>> > 33.33% 113427455640312821154458202477256070484
>> >
>> >
>> >
>> > SEED
>> > Pool Name Active Pending Completed
>> > ReadStage 0 0 7
>> > RequestResponseStage 0 0 1479839
>> > MutationStage 0 0 1479908
>> > ReadRepairStage 0 0 0
>> > ReplicateOnWriteStage 0 0 0
>> > GossipStage 0 0 5176
>> > AntiEntropyStage 0 0 0
>> > MigrationStage 0 0 128
>> > MemtablePostFlusher 0 0 28
>> > StreamStage 0 0 0
>> > FlushWriter 0 0 28
>> > MiscStage 0 0 2
>> > FlushSorter 0 0 0
>> > InternalResponseStage 0 0 363
>> > HintedHandoff 0 0 0
>> > NODE1
>> > Pool Name Active Pending Completed
>> > ReadStage 0 0 7
>> > RequestResponseStage 0 0 2960208
>> > MutationStage 0 0 1480104
>> > ReadRepairStage 0 0 0
>> > ReplicateOnWriteStage 0 0 1480104
>> > GossipStage 0 0 4151
>> > AntiEntropyStage 0 0 0
>> > MigrationStage 0 0 128
>> > MemtablePostFlusher 0 0 31
>> > StreamStage 0 0 8
>> > FlushWriter 0 0 31
>> > MiscStage 0 0 8
>> > FlushSorter 0 0 0
>> > InternalResponseStage 0 0 1
>> > HintedHandoff 0 0 0
>> > NODE2
>> > Pool Name Active Pending Completed
>> > ReadStage 0 0 7
>> > RequestResponseStage 0 0 0
>> > MutationStage 0 0 1481530
>> > ReadRepairStage 0 0 0
>> > ReplicateOnWriteStage 0 0 0
>> > GossipStage 0 0 4201
>> > AntiEntropyStage 0 0 0
>> > MigrationStage 0 0 128
>> > MemtablePostFlusher 0 0 31
>> > StreamStage 0 0 6
>> > FlushWriter 0 0 31
>> > MiscStage 0 0 6
>> > FlushSorter 0 0 0
>> > InternalResponseStage 0 0 1
>> > HintedHandoff 0 0 0
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: tpstats confusion
Posted by aaron morton <aa...@thelastpickle.com>.
I've not checked the code but from memory when the nodes are ordered in proximity to the coordinator the local node is always first if it's in the replica set. So with RF=3 and N=3 the closest node is always the local one.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 12 Aug 2011, at 10:10, Ian Danforth wrote:
> I'm writing at QUORUM though and (pardon me for being dense) it would seem that the first replica shouldn't always be on the same server if I'm using RP.
>
> I very much appreciate your time, I'm sure there is something basic that just hasn't clicked.
>
> Ian
>
> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-2889
>
> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <id...@numenta.com> wrote:
> > I don't think so, perhaps more succinctly, why would ROW actions only be
> > performed on a single node?
> > Ian
> >
> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> >>
> >> are you i/o bound? I believe counter ROW needs to perform a read of
> >> the old value.
> >>
> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
> >> wrote:
> >> > Hello all!
> >> > Thanks for taking the time to read this, I'm new to Cassandra and really
> >> > want to get it working :)
> >> > Below you'll find the output from tpstats on the three nodes in my .8.1
> >> > cluster. As you'll note from ring (also below), my cluster appears
> >> > balanced.
> >> > However, please note that NODE1 is the only node that has completed any
> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
> >> > RoW
> >> > and seen a single node in my various test clusters get hugely backed up
> >> > on
> >> > this stage to the point where it slows all writes to a crawl.
> >> > I'm using counters in a column family defined like this:
> >> > ==== KEYSPACE/CF ====
> >> > create keyspace Keyspace1
> >> > with strategy_options=[{replication_factor:3}]
> >> > and placement_strategy =
> >> > 'org.apache.cassandra.locator.SimpleStrategy';
> >> > use Keyspace1;
> >> > create column family TwitterTest
> >> > with default_validation_class = CounterColumnType
> >> > and replicate_on_write = true;
> >> > ====
> >> >
> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> >> > backed
> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
> >> > elsewhere that ANY or ONE could overwhelm the system.
> >> >
> >> > I hope I'm just doing something stupid and one you experienced folks can
> >> > point out my error.
> >> > Thanks in advance!
> >> > Ian
> >> > ==== MORE DATA ====
> >> > Address DC Rack Status State Load
> >> > Owns
> >> > Token
> >> >
> >> > 113427455640312821154458202477256070484
> >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
> >> > 33.33% 0
> >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
> >> > 33.33% 56713727820156410577229101238628035242
> >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
> >> > 33.33% 113427455640312821154458202477256070484
> >> >
> >> >
> >> >
> >> > SEED
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 1479839
> >> > MutationStage 0 0 1479908
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 0
> >> > GossipStage 0 0 5176
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 28
> >> > StreamStage 0 0 0
> >> > FlushWriter 0 0 28
> >> > MiscStage 0 0 2
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 363
> >> > HintedHandoff 0 0 0
> >> > NODE1
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 2960208
> >> > MutationStage 0 0 1480104
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 1480104
> >> > GossipStage 0 0 4151
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 31
> >> > StreamStage 0 0 8
> >> > FlushWriter 0 0 31
> >> > MiscStage 0 0 8
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 1
> >> > HintedHandoff 0 0 0
> >> > NODE2
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 0
> >> > MutationStage 0 0 1481530
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 0
> >> > GossipStage 0 0 4201
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 31
> >> > StreamStage 0 0 6
> >> > FlushWriter 0 0 31
> >> > MiscStage 0 0 6
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 1
> >> > HintedHandoff 0 0 0
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: tpstats confusion
Posted by aaron morton <aa...@thelastpickle.com>.
Turn the logging up to DEBUG, if you want to you can do it just for org.apache.cassandra.service.StorageProxy , and look for…
* forwarding counter update of key… when the coordinator is forwarding the counter update
* insert writing local & replicate… when a counter mutation is applied locally
* insert writing key… when the ROW is been sent
btw I was wrong about the sort by proximity. The default SimpleSnitch does no sorting, so the endpoint order is the token order. The DynamicSnitch will re-order according to the observed latency.
Hope that helps.
Aaron
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 13 Aug 2011, at 12:52, Ian Danforth wrote:
> Thanks! This would make sense but I'm not sure it maps to what I'm seeing. Specifically if I start processes that write to each of the three nodes, so I know there is a constant stream of writes to each node, instead of seeing replicate on write on all nodes, I see it only on a single node.
>
> It appears that all increments are being routed to a single node which then has to do all replication. However I don't have a good way to see the write flow
>
> Connection made to node: XXX
> Write sent to node: XXX
> Replicas sent to nodes: XXX, XXX
>
> If I could see that I'd have a lot more confidence about what's going on.
>
> Ian
>
> On Thu, Aug 11, 2011 at 6:38 PM, aaron morton <aa...@thelastpickle.com> wrote:
> I've not checked the code but from memory when the nodes are ordered in proximity to the coordinator the local node is always first if it's in the replica set. So with RF=3 and N=3 the closest node is always the local one.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12 Aug 2011, at 10:10, Ian Danforth wrote:
>
>> I'm writing at QUORUM though and (pardon me for being dense) it would seem that the first replica shouldn't always be on the same server if I'm using RP.
>>
>> I very much appreciate your time, I'm sure there is something basic that just hasn't clicked.
>>
>> Ian
>>
>> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> https://issues.apache.org/jira/browse/CASSANDRA-2889
>>
>> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <id...@numenta.com> wrote:
>> > I don't think so, perhaps more succinctly, why would ROW actions only be
>> > performed on a single node?
>> > Ian
>> >
>> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> >>
>> >> are you i/o bound? I believe counter ROW needs to perform a read of
>> >> the old value.
>> >>
>> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
>> >> wrote:
>> >> > Hello all!
>> >> > Thanks for taking the time to read this, I'm new to Cassandra and really
>> >> > want to get it working :)
>> >> > Below you'll find the output from tpstats on the three nodes in my .8.1
>> >> > cluster. As you'll note from ring (also below), my cluster appears
>> >> > balanced.
>> >> > However, please note that NODE1 is the only node that has completed any
>> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> >> > RoW
>> >> > and seen a single node in my various test clusters get hugely backed up
>> >> > on
>> >> > this stage to the point where it slows all writes to a crawl.
>> >> > I'm using counters in a column family defined like this:
>> >> > ==== KEYSPACE/CF ====
>> >> > create keyspace Keyspace1
>> >> > with strategy_options=[{replication_factor:3}]
>> >> > and placement_strategy =
>> >> > 'org.apache.cassandra.locator.SimpleStrategy';
>> >> > use Keyspace1;
>> >> > create column family TwitterTest
>> >> > with default_validation_class = CounterColumnType
>> >> > and replicate_on_write = true;
>> >> > ====
>> >> >
>> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> >> > backed
>> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
>> >> > elsewhere that ANY or ONE could overwhelm the system.
>> >> >
>> >> > I hope I'm just doing something stupid and one you experienced folks can
>> >> > point out my error.
>> >> > Thanks in advance!
>> >> > Ian
>> >> > ==== MORE DATA ====
>> >> > Address DC Rack Status State Load
>> >> > Owns
>> >> > Token
>> >> >
>> >> > 113427455640312821154458202477256070484
>> >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
>> >> > 33.33% 0
>> >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
>> >> > 33.33% 56713727820156410577229101238628035242
>> >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
>> >> > 33.33% 113427455640312821154458202477256070484
>> >> >
>> >> >
>> >> >
>> >> > SEED
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 1479839
>> >> > MutationStage 0 0 1479908
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 5176
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 28
>> >> > StreamStage 0 0 0
>> >> > FlushWriter 0 0 28
>> >> > MiscStage 0 0 2
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 363
>> >> > HintedHandoff 0 0 0
>> >> > NODE1
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 2960208
>> >> > MutationStage 0 0 1480104
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 1480104
>> >> > GossipStage 0 0 4151
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 8
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 8
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >> > NODE2
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 0
>> >> > MutationStage 0 0 1481530
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 4201
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 6
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 6
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
Re: tpstats confusion
Posted by Ian Danforth <id...@numenta.com>.
Thanks! This would make sense but I'm not sure it maps to what I'm seeing.
Specifically if I start processes that write to each of the three nodes, so
I know there is a constant stream of writes to each node, instead of seeing
replicate on write on all nodes, I see it only on a single node.
It appears that all increments are being routed to a single node which then
has to do all replication. However I don't have a good way to see the write
flow
Connection made to node: XXX
Write sent to node: XXX
Replicas sent to nodes: XXX, XXX
If I could see that I'd have a lot more confidence about what's going on.
Ian
On Thu, Aug 11, 2011 at 6:38 PM, aaron morton <aa...@thelastpickle.com>wrote:
> I've not checked the code but from memory when the nodes are ordered in
> proximity to the coordinator the local node is always first if it's in the
> replica set. So with RF=3 and N=3 the closest node is always the local one.
>
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12 Aug 2011, at 10:10, Ian Danforth wrote:
>
> I'm writing at QUORUM though and (pardon me for being dense) it would seem
> that the first replica shouldn't always be on the same server if I'm using
> RP.
>
> I very much appreciate your time, I'm sure there is something basic that
> just hasn't clicked.
>
> Ian
>
> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-2889
>>
>> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <id...@numenta.com>
>> wrote:
>> > I don't think so, perhaps more succinctly, why would ROW actions only be
>> > performed on a single node?
>> > Ian
>> >
>> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com>
>> wrote:
>> >>
>> >> are you i/o bound? I believe counter ROW needs to perform a read of
>> >> the old value.
>> >>
>> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
>> >> wrote:
>> >> > Hello all!
>> >> > Thanks for taking the time to read this, I'm new to Cassandra and
>> really
>> >> > want to get it working :)
>> >> > Below you'll find the output from tpstats on the three nodes in my
>> .8.1
>> >> > cluster. As you'll note from ring (also below), my cluster appears
>> >> > balanced.
>> >> > However, please note that NODE1 is the only node that has completed
>> any
>> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> >> > RoW
>> >> > and seen a single node in my various test clusters get hugely backed
>> up
>> >> > on
>> >> > this stage to the point where it slows all writes to a crawl.
>> >> > I'm using counters in a column family defined like this:
>> >> > ==== KEYSPACE/CF ====
>> >> > create keyspace Keyspace1
>> >> > with strategy_options=[{replication_factor:3}]
>> >> > and placement_strategy =
>> >> > 'org.apache.cassandra.locator.SimpleStrategy';
>> >> > use Keyspace1;
>> >> > create column family TwitterTest
>> >> > with default_validation_class = CounterColumnType
>> >> > and replicate_on_write = true;
>> >> > ====
>> >> >
>> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> >> > backed
>> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was
>> suggested
>> >> > elsewhere that ANY or ONE could overwhelm the system.
>> >> >
>> >> > I hope I'm just doing something stupid and one you experienced folks
>> can
>> >> > point out my error.
>> >> > Thanks in advance!
>> >> > Ian
>> >> > ==== MORE DATA ====
>> >> > Address DC Rack Status State Load
>> >> > Owns
>> >> > Token
>> >> >
>> >> > 113427455640312821154458202477256070484
>> >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
>> >> > 33.33% 0
>> >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
>> >> > 33.33% 56713727820156410577229101238628035242
>> >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
>> >> > 33.33% 113427455640312821154458202477256070484
>> >> >
>> >> >
>> >> >
>> >> > SEED
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 1479839
>> >> > MutationStage 0 0 1479908
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 5176
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 28
>> >> > StreamStage 0 0 0
>> >> > FlushWriter 0 0 28
>> >> > MiscStage 0 0 2
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 363
>> >> > HintedHandoff 0 0 0
>> >> > NODE1
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 2960208
>> >> > MutationStage 0 0 1480104
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 1480104
>> >> > GossipStage 0 0 4151
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 8
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 8
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >> > NODE2
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 0
>> >> > MutationStage 0 0 1481530
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 4201
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 6
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 6
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
Re: tpstats confusion
Posted by Ian Danforth <id...@numenta.com>.
I'm writing at QUORUM though and (pardon me for being dense) it would seem
that the first replica shouldn't always be on the same server if I'm using
RP.
I very much appreciate your time, I'm sure there is something basic that
just hasn't clicked.
Ian
On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/CASSANDRA-2889
>
> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <id...@numenta.com>
> wrote:
> > I don't think so, perhaps more succinctly, why would ROW actions only be
> > performed on a single node?
> > Ian
> >
> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> are you i/o bound? I believe counter ROW needs to perform a read of
> >> the old value.
> >>
> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
> >> wrote:
> >> > Hello all!
> >> > Thanks for taking the time to read this, I'm new to Cassandra and
> really
> >> > want to get it working :)
> >> > Below you'll find the output from tpstats on the three nodes in my
> .8.1
> >> > cluster. As you'll note from ring (also below), my cluster appears
> >> > balanced.
> >> > However, please note that NODE1 is the only node that has completed
> any
> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
> >> > RoW
> >> > and seen a single node in my various test clusters get hugely backed
> up
> >> > on
> >> > this stage to the point where it slows all writes to a crawl.
> >> > I'm using counters in a column family defined like this:
> >> > ==== KEYSPACE/CF ====
> >> > create keyspace Keyspace1
> >> > with strategy_options=[{replication_factor:3}]
> >> > and placement_strategy =
> >> > 'org.apache.cassandra.locator.SimpleStrategy';
> >> > use Keyspace1;
> >> > create column family TwitterTest
> >> > with default_validation_class = CounterColumnType
> >> > and replicate_on_write = true;
> >> > ====
> >> >
> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> >> > backed
> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was
> suggested
> >> > elsewhere that ANY or ONE could overwhelm the system.
> >> >
> >> > I hope I'm just doing something stupid and one you experienced folks
> can
> >> > point out my error.
> >> > Thanks in advance!
> >> > Ian
> >> > ==== MORE DATA ====
> >> > Address DC Rack Status State Load
> >> > Owns
> >> > Token
> >> >
> >> > 113427455640312821154458202477256070484
> >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
> >> > 33.33% 0
> >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
> >> > 33.33% 56713727820156410577229101238628035242
> >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
> >> > 33.33% 113427455640312821154458202477256070484
> >> >
> >> >
> >> >
> >> > SEED
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 1479839
> >> > MutationStage 0 0 1479908
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 0
> >> > GossipStage 0 0 5176
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 28
> >> > StreamStage 0 0 0
> >> > FlushWriter 0 0 28
> >> > MiscStage 0 0 2
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 363
> >> > HintedHandoff 0 0 0
> >> > NODE1
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 2960208
> >> > MutationStage 0 0 1480104
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 1480104
> >> > GossipStage 0 0 4151
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 31
> >> > StreamStage 0 0 8
> >> > FlushWriter 0 0 31
> >> > MiscStage 0 0 8
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 1
> >> > HintedHandoff 0 0 0
> >> > NODE2
> >> > Pool Name Active Pending Completed
> >> > ReadStage 0 0 7
> >> > RequestResponseStage 0 0 0
> >> > MutationStage 0 0 1481530
> >> > ReadRepairStage 0 0 0
> >> > ReplicateOnWriteStage 0 0 0
> >> > GossipStage 0 0 4201
> >> > AntiEntropyStage 0 0 0
> >> > MigrationStage 0 0 128
> >> > MemtablePostFlusher 0 0 31
> >> > StreamStage 0 0 6
> >> > FlushWriter 0 0 31
> >> > MiscStage 0 0 6
> >> > FlushSorter 0 0 0
> >> > InternalResponseStage 0 0 1
> >> > HintedHandoff 0 0 0
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: tpstats confusion
Posted by Ian Danforth <id...@numenta.com>.
I don't think so, perhaps more succinctly, why would ROW actions only be
performed on a single node?
Ian
On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> are you i/o bound? I believe counter ROW needs to perform a read of
> the old value.
>
> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <id...@numenta.com>
> wrote:
> > Hello all!
> > Thanks for taking the time to read this, I'm new to Cassandra and really
> > want to get it working :)
> > Below you'll find the output from tpstats on the three nodes in my .8.1
> > cluster. As you'll note from ring (also below), my cluster appears
> balanced.
> > However, please note that NODE1 is the only node that has completed any
> > ReplicateOnWriteStage actions. I've been having a lot of trouble with RoW
> > and seen a single node in my various test clusters get hugely backed up
> on
> > this stage to the point where it slows all writes to a crawl.
> > I'm using counters in a column family defined like this:
> > ==== KEYSPACE/CF ====
> > create keyspace Keyspace1
> > with strategy_options=[{replication_factor:3}]
> > and placement_strategy =
> 'org.apache.cassandra.locator.SimpleStrategy';
> > use Keyspace1;
> > create column family TwitterTest
> > with default_validation_class = CounterColumnType
> > and replicate_on_write = true;
> > ====
> >
> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
> backed
> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
> > elsewhere that ANY or ONE could overwhelm the system.
> >
> > I hope I'm just doing something stupid and one you experienced folks can
> > point out my error.
> > Thanks in advance!
> > Ian
> > ==== MORE DATA ====
> > Address DC Rack Status State Load
> Owns
> > Token
> >
> > 113427455640312821154458202477256070484
> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
> > 33.33% 0
> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
> > 33.33% 56713727820156410577229101238628035242
> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
> > 33.33% 113427455640312821154458202477256070484
> >
> >
> >
> > SEED
> > Pool Name Active Pending Completed
> > ReadStage 0 0 7
> > RequestResponseStage 0 0 1479839
> > MutationStage 0 0 1479908
> > ReadRepairStage 0 0 0
> > ReplicateOnWriteStage 0 0 0
> > GossipStage 0 0 5176
> > AntiEntropyStage 0 0 0
> > MigrationStage 0 0 128
> > MemtablePostFlusher 0 0 28
> > StreamStage 0 0 0
> > FlushWriter 0 0 28
> > MiscStage 0 0 2
> > FlushSorter 0 0 0
> > InternalResponseStage 0 0 363
> > HintedHandoff 0 0 0
> > NODE1
> > Pool Name Active Pending Completed
> > ReadStage 0 0 7
> > RequestResponseStage 0 0 2960208
> > MutationStage 0 0 1480104
> > ReadRepairStage 0 0 0
> > ReplicateOnWriteStage 0 0 1480104
> > GossipStage 0 0 4151
> > AntiEntropyStage 0 0 0
> > MigrationStage 0 0 128
> > MemtablePostFlusher 0 0 31
> > StreamStage 0 0 8
> > FlushWriter 0 0 31
> > MiscStage 0 0 8
> > FlushSorter 0 0 0
> > InternalResponseStage 0 0 1
> > HintedHandoff 0 0 0
> > NODE2
> > Pool Name Active Pending Completed
> > ReadStage 0 0 7
> > RequestResponseStage 0 0 0
> > MutationStage 0 0 1481530
> > ReadRepairStage 0 0 0
> > ReplicateOnWriteStage 0 0 0
> > GossipStage 0 0 4201
> > AntiEntropyStage 0 0 0
> > MigrationStage 0 0 128
> > MemtablePostFlusher 0 0 31
> > StreamStage 0 0 6
> > FlushWriter 0 0 31
> > MiscStage 0 0 6
> > FlushSorter 0 0 0
> > InternalResponseStage 0 0 1
> > HintedHandoff 0 0 0
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>