You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anishek Agarwal <an...@gmail.com> on 2016/04/12 08:59:22 UTC
Traffic inconsistent across nodes
hello,
we have 8 nodes in one cluster and attached is the traffic patterns across
the nodes.
its very surprising that only 4 nodes show transmitting (purple) packets.
our driver configuration on clients has the following load balancing
configuration :
new TokenAwarePolicy(
new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
"WDC")),
true)
any idea what is that we are missing which is leading to this skewed data
read patterns
cassandra drivers as below:
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.6</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-mapping</artifactId>
<version>2.1.6</version>
</dependency>
cassandra version is 2.0.17
Thanks in advance for the help.
Anishek
Re: Traffic inconsistent across nodes
Posted by Eric Stevens <mi...@gmail.com>.
Glad that worked out, that was going to be my next suspicion, since
everything thought it was up and happy, I can't think of a way that
Cassandra and the driver could both consider the cluster happy if some
nodes were not transmitting at least some data (they have to at least for
gossip).
On Mon, Apr 18, 2016 at 10:02 PM Anishek Agarwal <an...@gmail.com> wrote:
> Looks like some problem with our monitoring framework. Thanks for you help
> !
>
> On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs
>> Cent OS 7)
>> All of them are running Cassandra 2.0.17
>>
>> output of the test :
>>
>> host ip: 10.124.114.113
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.108
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.110
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.118
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.125.138.59
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.97
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.105
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>> host ip: 10.124.114.98
>>
>> host DC : WDC
>>
>> distance of host: LOCAL
>>
>> host is up: true
>>
>> cassandra version : 2.0.17
>>
>>
>> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <mi...@gmail.com> wrote:
>>
>>> Thanks for that, that helps a lot. The next thing to check might be
>>> whether or not your application actually has access to the other nodes.
>>> With that topology, and assuming all the nodes you included in your
>>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>>> network issue of some kind.
>>>
>>> Also, it probably doesn't matter, but what OS / Distribution are you
>>> running the servers and clients on?
>>>
>>> Check with netcat or something that you can reach all the configured
>>> ports from your application server, but also the driver itself offers some
>>> introspection into its view of individual connection health. This is a
>>> little bit ugly, but this is how we include information about connection
>>> status in an API for health monitoring from a Scala application using the
>>> Java driver; hopefully you can use it to see how to access information
>>> about the driver's view of host health from the application's perspective.
>>> Most importantly I'd suggest looking for host.isUp status and
>>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>>> in your target datacenter to be LOCAL.
>>>
>>> "hosts" -> {
>>> val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>>> connection.getMetadata
>>> .getAllHosts.asScala
>>> .groupBy(_.getDatacenter)
>>> .mapValues(_.groupBy(_.getRack))
>>> val lbp: LoadBalancingPolicy = connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>>> JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>>> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: mutable.Set[Host]) =>
>>> rack -> JsArray(hosts.map { host =>
>>> Json.obj(
>>> "address" -> host.getAddress.toString,
>>> "socketAddress" -> host.getSocketAddress.toString,
>>> "cassandraVersion" -> host.getCassandraVersion.toString,
>>> "isUp" -> host.isUp,
>>> "hostDistance" -> lbp.distance(host).toString
>>> )
>>> }.toSeq)
>>> }.toSeq)
>>> }.toSeq)
>>> },
>>>
>>>
>>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> here is the output: every node in a single DC is in the same rack.
>>>>
>>>> Datacenter: WDC5
>>>>
>>>> ================
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> -- Address Load Tokens Owns (effective) Host ID
>>>> Rack
>>>>
>>>> UN 10.125.138.33 299.22 GB 256 64.2%
>>>> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>>>>
>>>> UN 10.125.138.125 329.38 GB 256 70.3%
>>>> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>>>>
>>>> UN 10.125.138.129 305.11 GB 256 65.5%
>>>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>>>>
>>>> Datacenter: WDC
>>>>
>>>> ===============
>>>>
>>>> Status=Up/Down
>>>>
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>
>>>> -- Address Load Tokens Owns (effective) Host ID
>>>> Rack
>>>>
>>>> UN 10.124.114.105 151.09 GB 256 38.0%
>>>> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>>>>
>>>> UN 10.124.114.110 150.15 GB 256 36.9%
>>>> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>>>>
>>>> UN 10.124.114.108 170.1 GB 256 41.3%
>>>> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>>>>
>>>> UN 10.124.114.98 165.34 GB 256 37.6%
>>>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>>>>
>>>> UN 10.124.114.113 145.22 GB 256 35.7%
>>>> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>>>>
>>>> UN 10.125.138.59 162.65 GB 256 38.6%
>>>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>>>>
>>>> UN 10.124.114.97 164.03 GB 256 36.9%
>>>> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>>>>
>>>> UN 10.124.114.118 139.62 GB 256 35.1%
>>>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>>>>
>>>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> The output of nodetool status would really help answer some
>>>>> questions. I take it the 8 hosts in your graph are in the same DC. Are
>>>>> the four serving writes in the same logical or physical rack (as Cassandra
>>>>> sees it), while the others are not?
>>>>>
>>>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Maybe include nodetool status here? Are the four nodes serving
>>>>>>> reads in one DC (local to your driver's config) while the others are in
>>>>>>> another?
>>>>>>>
>>>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> hello,
>>>>>>>>
>>>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>>>> across the nodes.
>>>>>>>>
>>>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>>>> packets.
>>>>>>>>
>>>>>>>> our driver configuration on clients has the following load
>>>>>>>> balancing configuration :
>>>>>>>>
>>>>>>>> new TokenAwarePolicy(
>>>>>>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>>>>>>> true)
>>>>>>>>
>>>>>>>>
>>>>>>>> any idea what is that we are missing which is leading to this
>>>>>>>> skewed data read patterns
>>>>>>>>
>>>>>>>> cassandra drivers as below:
>>>>>>>>
>>>>>>>> <dependency>
>>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>>>>> <version>2.1.6</version>
>>>>>>>> </dependency>
>>>>>>>> <dependency>
>>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>>>>> <version>2.1.6</version>
>>>>>>>> </dependency>
>>>>>>>>
>>>>>>>> cassandra version is 2.0.17
>>>>>>>>
>>>>>>>> Thanks in advance for the help.
>>>>>>>>
>>>>>>>> Anishek
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
>
Re: Traffic inconsistent across nodes
Posted by Anishek Agarwal <an...@gmail.com>.
Looks like some problem with our monitoring framework. Thanks for you help !
On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal <an...@gmail.com> wrote:
> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
> OS 7)
> All of them are running Cassandra 2.0.17
>
> output of the test :
>
> host ip: 10.124.114.113
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.108
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.110
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.118
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.125.138.59
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.97
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.105
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.98
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
>
> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <mi...@gmail.com> wrote:
>
>> Thanks for that, that helps a lot. The next thing to check might be
>> whether or not your application actually has access to the other nodes.
>> With that topology, and assuming all the nodes you included in your
>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>> network issue of some kind.
>>
>> Also, it probably doesn't matter, but what OS / Distribution are you
>> running the servers and clients on?
>>
>> Check with netcat or something that you can reach all the configured
>> ports from your application server, but also the driver itself offers some
>> introspection into its view of individual connection health. This is a
>> little bit ugly, but this is how we include information about connection
>> status in an API for health monitoring from a Scala application using the
>> Java driver; hopefully you can use it to see how to access information
>> about the driver's view of host health from the application's perspective.
>> Most importantly I'd suggest looking for host.isUp status and
>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>> in your target datacenter to be LOCAL.
>>
>> "hosts" -> {
>> val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>> connection.getMetadata
>> .getAllHosts.asScala
>> .groupBy(_.getDatacenter)
>> .mapValues(_.groupBy(_.getRack))
>> val lbp: LoadBalancingPolicy = connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>> JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: mutable.Set[Host]) =>
>> rack -> JsArray(hosts.map { host =>
>> Json.obj(
>> "address" -> host.getAddress.toString,
>> "socketAddress" -> host.getSocketAddress.toString,
>> "cassandraVersion" -> host.getCassandraVersion.toString,
>> "isUp" -> host.isUp,
>> "hostDistance" -> lbp.distance(host).toString
>> )
>> }.toSeq)
>> }.toSeq)
>> }.toSeq)
>> },
>>
>>
>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <an...@gmail.com>
>> wrote:
>>
>>> here is the output: every node in a single DC is in the same rack.
>>>
>>> Datacenter: WDC5
>>>
>>> ================
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>>
>>> UN 10.125.138.33 299.22 GB 256 64.2%
>>> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>>>
>>> UN 10.125.138.125 329.38 GB 256 70.3%
>>> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>>>
>>> UN 10.125.138.129 305.11 GB 256 65.5%
>>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>>>
>>> Datacenter: WDC
>>>
>>> ===============
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>>
>>> UN 10.124.114.105 151.09 GB 256 38.0%
>>> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>>>
>>> UN 10.124.114.110 150.15 GB 256 36.9%
>>> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>>>
>>> UN 10.124.114.108 170.1 GB 256 41.3%
>>> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>>>
>>> UN 10.124.114.98 165.34 GB 256 37.6%
>>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>>>
>>> UN 10.124.114.113 145.22 GB 256 35.7%
>>> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>>>
>>> UN 10.125.138.59 162.65 GB 256 38.6%
>>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>>>
>>> UN 10.124.114.97 164.03 GB 256 36.9%
>>> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>>>
>>> UN 10.124.114.118 139.62 GB 256 35.1%
>>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>>>
>>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mi...@gmail.com> wrote:
>>>
>>>> The output of nodetool status would really help answer some questions.
>>>> I take it the 8 hosts in your graph are in the same DC. Are the four
>>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>>> while the others are not?
>>>>
>>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Maybe include nodetool status here? Are the four nodes serving reads
>>>>>> in one DC (local to your driver's config) while the others are in another?
>>>>>>
>>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> hello,
>>>>>>>
>>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>>> across the nodes.
>>>>>>>
>>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>>> packets.
>>>>>>>
>>>>>>> our driver configuration on clients has the following load balancing
>>>>>>> configuration :
>>>>>>>
>>>>>>> new TokenAwarePolicy(
>>>>>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>>>>>> true)
>>>>>>>
>>>>>>>
>>>>>>> any idea what is that we are missing which is leading to this skewed
>>>>>>> data read patterns
>>>>>>>
>>>>>>> cassandra drivers as below:
>>>>>>>
>>>>>>> <dependency>
>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>>>> <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>> <dependency>
>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>>>> <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> cassandra version is 2.0.17
>>>>>>>
>>>>>>> Thanks in advance for the help.
>>>>>>>
>>>>>>> Anishek
>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: Traffic inconsistent across nodes
Posted by Anishek Agarwal <an...@gmail.com>.
OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
OS 7)
All of them are running Cassandra 2.0.17
output of the test :
host ip: 10.124.114.113
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.108
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.110
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.118
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.125.138.59
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.97
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.105
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
host ip: 10.124.114.98
host DC : WDC
distance of host: LOCAL
host is up: true
cassandra version : 2.0.17
On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <mi...@gmail.com> wrote:
> Thanks for that, that helps a lot. The next thing to check might be
> whether or not your application actually has access to the other nodes.
> With that topology, and assuming all the nodes you included in your
> original graph are in the 'WDC' data center, I'd be inclined to look for a
> network issue of some kind.
>
> Also, it probably doesn't matter, but what OS / Distribution are you
> running the servers and clients on?
>
> Check with netcat or something that you can reach all the configured ports
> from your application server, but also the driver itself offers some
> introspection into its view of individual connection health. This is a
> little bit ugly, but this is how we include information about connection
> status in an API for health monitoring from a Scala application using the
> Java driver; hopefully you can use it to see how to access information
> about the driver's view of host health from the application's perspective.
> Most importantly I'd suggest looking for host.isUp status and
> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
> in your target datacenter to be LOCAL.
>
> "hosts" -> {
> val hosts: Map[String, Map[String, mutable.Set[Host]]] =
> connection.getMetadata
> .getAllHosts.asScala
> .groupBy(_.getDatacenter)
> .mapValues(_.groupBy(_.getRack))
> val lbp: LoadBalancingPolicy = connection.getConfiguration.getPolicies.getLoadBalancingPolicy
> JsObject(hosts.map { case (dc: String, rackAndHosts) =>
> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts: mutable.Set[Host]) =>
> rack -> JsArray(hosts.map { host =>
> Json.obj(
> "address" -> host.getAddress.toString,
> "socketAddress" -> host.getSocketAddress.toString,
> "cassandraVersion" -> host.getCassandraVersion.toString,
> "isUp" -> host.isUp,
> "hostDistance" -> lbp.distance(host).toString
> )
> }.toSeq)
> }.toSeq)
> }.toSeq)
> },
>
>
> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> here is the output: every node in a single DC is in the same rack.
>>
>> Datacenter: WDC5
>>
>> ================
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 10.125.138.33 299.22 GB 256 64.2%
>> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>>
>> UN 10.125.138.125 329.38 GB 256 70.3%
>> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>>
>> UN 10.125.138.129 305.11 GB 256 65.5%
>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>>
>> Datacenter: WDC
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN 10.124.114.105 151.09 GB 256 38.0%
>> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>>
>> UN 10.124.114.110 150.15 GB 256 36.9%
>> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>>
>> UN 10.124.114.108 170.1 GB 256 41.3%
>> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>>
>> UN 10.124.114.98 165.34 GB 256 37.6%
>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>>
>> UN 10.124.114.113 145.22 GB 256 35.7%
>> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>>
>> UN 10.125.138.59 162.65 GB 256 38.6%
>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>>
>> UN 10.124.114.97 164.03 GB 256 36.9%
>> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>>
>> UN 10.124.114.118 139.62 GB 256 35.1%
>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>>
>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mi...@gmail.com> wrote:
>>
>>> The output of nodetool status would really help answer some questions.
>>> I take it the 8 hosts in your graph are in the same DC. Are the four
>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>> while the others are not?
>>>
>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com>
>>> wrote:
>>>
>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>
>>>>
>>>>
>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Maybe include nodetool status here? Are the four nodes serving reads
>>>>> in one DC (local to your driver's config) while the others are in another?
>>>>>
>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hello,
>>>>>>
>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>> across the nodes.
>>>>>>
>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>> packets.
>>>>>>
>>>>>> our driver configuration on clients has the following load balancing
>>>>>> configuration :
>>>>>>
>>>>>> new TokenAwarePolicy(
>>>>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>>>>> true)
>>>>>>
>>>>>>
>>>>>> any idea what is that we are missing which is leading to this skewed
>>>>>> data read patterns
>>>>>>
>>>>>> cassandra drivers as below:
>>>>>>
>>>>>> <dependency>
>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>>> <version>2.1.6</version>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>>> <version>2.1.6</version>
>>>>>> </dependency>
>>>>>>
>>>>>> cassandra version is 2.0.17
>>>>>>
>>>>>> Thanks in advance for the help.
>>>>>>
>>>>>> Anishek
>>>>>>
>>>>>>
>>>>
>>
Re: Traffic inconsistent across nodes
Posted by Eric Stevens <mi...@gmail.com>.
Thanks for that, that helps a lot. The next thing to check might be
whether or not your application actually has access to the other nodes.
With that topology, and assuming all the nodes you included in your
original graph are in the 'WDC' data center, I'd be inclined to look for a
network issue of some kind.
Also, it probably doesn't matter, but what OS / Distribution are you
running the servers and clients on?
Check with netcat or something that you can reach all the configured ports
from your application server, but also the driver itself offers some
introspection into its view of individual connection health. This is a
little bit ugly, but this is how we include information about connection
status in an API for health monitoring from a Scala application using the
Java driver; hopefully you can use it to see how to access information
about the driver's view of host health from the application's perspective.
Most importantly I'd suggest looking for host.isUp status and
LoadBalancingPolicy.distance(host) to see that it considers all the hosts
in your target datacenter to be LOCAL.
"hosts" -> {
val hosts: Map[String, Map[String, mutable.Set[Host]]] =
connection.getMetadata
.getAllHosts.asScala
.groupBy(_.getDatacenter)
.mapValues(_.groupBy(_.getRack))
val lbp: LoadBalancingPolicy =
connection.getConfiguration.getPolicies.getLoadBalancingPolicy
JsObject(hosts.map { case (dc: String, rackAndHosts) =>
dc -> JsObject(rackAndHosts.map { case (rack: String, hosts:
mutable.Set[Host]) =>
rack -> JsArray(hosts.map { host =>
Json.obj(
"address" -> host.getAddress.toString,
"socketAddress" -> host.getSocketAddress.toString,
"cassandraVersion" -> host.getCassandraVersion.toString,
"isUp" -> host.isUp,
"hostDistance" -> lbp.distance(host).toString
)
}.toSeq)
}.toSeq)
}.toSeq)
},
On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <an...@gmail.com> wrote:
> here is the output: every node in a single DC is in the same rack.
>
> Datacenter: WDC5
>
> ================
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns (effective) Host ID
> Rack
>
> UN 10.125.138.33 299.22 GB 256 64.2%
> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>
> UN 10.125.138.125 329.38 GB 256 70.3%
> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>
> UN 10.125.138.129 305.11 GB 256 65.5%
> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>
> Datacenter: WDC
>
> ===============
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns (effective) Host ID
> Rack
>
> UN 10.124.114.105 151.09 GB 256 38.0%
> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>
> UN 10.124.114.110 150.15 GB 256 36.9%
> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>
> UN 10.124.114.108 170.1 GB 256 41.3%
> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>
> UN 10.124.114.98 165.34 GB 256 37.6%
> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>
> UN 10.124.114.113 145.22 GB 256 35.7%
> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>
> UN 10.125.138.59 162.65 GB 256 38.6%
> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>
> UN 10.124.114.97 164.03 GB 256 36.9%
> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>
> UN 10.124.114.118 139.62 GB 256 35.1%
> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>
> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mi...@gmail.com> wrote:
>
>> The output of nodetool status would really help answer some questions. I
>> take it the 8 hosts in your graph are in the same DC. Are the four serving
>> writes in the same logical or physical rack (as Cassandra sees it), while
>> the others are not?
>>
>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com>
>> wrote:
>>
>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>
>>>
>>>
>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com> wrote:
>>>
>>>> Maybe include nodetool status here? Are the four nodes serving reads
>>>> in one DC (local to your driver's config) while the others are in another?
>>>>
>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> hello,
>>>>>
>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>> across the nodes.
>>>>>
>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>> packets.
>>>>>
>>>>> our driver configuration on clients has the following load balancing
>>>>> configuration :
>>>>>
>>>>> new TokenAwarePolicy(
>>>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>>>> true)
>>>>>
>>>>>
>>>>> any idea what is that we are missing which is leading to this skewed
>>>>> data read patterns
>>>>>
>>>>> cassandra drivers as below:
>>>>>
>>>>> <dependency>
>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>> <version>2.1.6</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>> <version>2.1.6</version>
>>>>> </dependency>
>>>>>
>>>>> cassandra version is 2.0.17
>>>>>
>>>>> Thanks in advance for the help.
>>>>>
>>>>> Anishek
>>>>>
>>>>>
>>>
>
Re: Traffic inconsistent across nodes
Posted by Anishek Agarwal <an...@gmail.com>.
here is the output: every node in a single DC is in the same rack.
Datacenter: WDC5
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.125.138.33 299.22 GB 256 64.2%
8aaa6015-d444-4551-a3c5-3257536df476 RAC1
UN 10.125.138.125 329.38 GB 256 70.3%
70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
UN 10.125.138.129 305.11 GB 256 65.5%
0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
Datacenter: WDC
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.124.114.105 151.09 GB 256 38.0%
c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
UN 10.124.114.110 150.15 GB 256 36.9%
6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
UN 10.124.114.108 170.1 GB 256 41.3%
040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
UN 10.124.114.98 165.34 GB 256 37.6%
cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
UN 10.124.114.113 145.22 GB 256 35.7%
1557af04-e658-4751-b984-8e0cdc41376e RAC1
UN 10.125.138.59 162.65 GB 256 38.6%
9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
UN 10.124.114.97 164.03 GB 256 36.9%
c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
UN 10.124.114.118 139.62 GB 256 35.1%
2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <mi...@gmail.com> wrote:
> The output of nodetool status would really help answer some questions. I
> take it the 8 hosts in your graph are in the same DC. Are the four serving
> writes in the same logical or physical rack (as Cassandra sees it), while
> the others are not?
>
> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com>
> wrote:
>
>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>
>>
>>
>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com> wrote:
>>
>>> Maybe include nodetool status here? Are the four nodes serving reads in
>>> one DC (local to your driver's config) while the others are in another?
>>>
>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com> wrote:
>>>
>>>> hello,
>>>>
>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>> across the nodes.
>>>>
>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>> packets.
>>>>
>>>> our driver configuration on clients has the following load balancing
>>>> configuration :
>>>>
>>>> new TokenAwarePolicy(
>>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>>> true)
>>>>
>>>>
>>>> any idea what is that we are missing which is leading to this skewed
>>>> data read patterns
>>>>
>>>> cassandra drivers as below:
>>>>
>>>> <dependency>
>>>> <groupId>com.datastax.cassandra</groupId>
>>>> <artifactId>cassandra-driver-core</artifactId>
>>>> <version>2.1.6</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>com.datastax.cassandra</groupId>
>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>> <version>2.1.6</version>
>>>> </dependency>
>>>>
>>>> cassandra version is 2.0.17
>>>>
>>>> Thanks in advance for the help.
>>>>
>>>> Anishek
>>>>
>>>>
>>
Re: Traffic inconsistent across nodes
Posted by Eric Stevens <mi...@gmail.com>.
The output of nodetool status would really help answer some questions. I
take it the 8 hosts in your graph are in the same DC. Are the four serving
writes in the same logical or physical rack (as Cassandra sees it), while
the others are not?
On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <an...@gmail.com> wrote:
> We have two DC one with the above 8 nodes and other with 3 nodes.
>
>
>
> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com> wrote:
>
>> Maybe include nodetool status here? Are the four nodes serving reads in
>> one DC (local to your driver's config) while the others are in another?
>>
>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com> wrote:
>>
>>> hello,
>>>
>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>> across the nodes.
>>>
>>> its very surprising that only 4 nodes show transmitting (purple) packets.
>>>
>>> our driver configuration on clients has the following load balancing
>>> configuration :
>>>
>>> new TokenAwarePolicy(
>>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>>> true)
>>>
>>>
>>> any idea what is that we are missing which is leading to this skewed
>>> data read patterns
>>>
>>> cassandra drivers as below:
>>>
>>> <dependency>
>>> <groupId>com.datastax.cassandra</groupId>
>>> <artifactId>cassandra-driver-core</artifactId>
>>> <version>2.1.6</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>com.datastax.cassandra</groupId>
>>> <artifactId>cassandra-driver-mapping</artifactId>
>>> <version>2.1.6</version>
>>> </dependency>
>>>
>>> cassandra version is 2.0.17
>>>
>>> Thanks in advance for the help.
>>>
>>> Anishek
>>>
>>>
>
Re: Traffic inconsistent across nodes
Posted by Anishek Agarwal <an...@gmail.com>.
We have two DC one with the above 8 nodes and other with 3 nodes.
On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <mi...@gmail.com> wrote:
> Maybe include nodetool status here? Are the four nodes serving reads in
> one DC (local to your driver's config) while the others are in another?
>
> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com> wrote:
>
>> hello,
>>
>> we have 8 nodes in one cluster and attached is the traffic patterns
>> across the nodes.
>>
>> its very surprising that only 4 nodes show transmitting (purple) packets.
>>
>> our driver configuration on clients has the following load balancing
>> configuration :
>>
>> new TokenAwarePolicy(
>> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
>> true)
>>
>>
>> any idea what is that we are missing which is leading to this skewed data
>> read patterns
>>
>> cassandra drivers as below:
>>
>> <dependency>
>> <groupId>com.datastax.cassandra</groupId>
>> <artifactId>cassandra-driver-core</artifactId>
>> <version>2.1.6</version>
>> </dependency>
>> <dependency>
>> <groupId>com.datastax.cassandra</groupId>
>> <artifactId>cassandra-driver-mapping</artifactId>
>> <version>2.1.6</version>
>> </dependency>
>>
>> cassandra version is 2.0.17
>>
>> Thanks in advance for the help.
>>
>> Anishek
>>
>>
Re: Traffic inconsistent across nodes
Posted by Eric Stevens <mi...@gmail.com>.
Maybe include nodetool status here? Are the four nodes serving reads in
one DC (local to your driver's config) while the others are in another?
On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <an...@gmail.com> wrote:
> hello,
>
> we have 8 nodes in one cluster and attached is the traffic patterns across
> the nodes.
>
> its very surprising that only 4 nodes show transmitting (purple) packets.
>
> our driver configuration on clients has the following load balancing
> configuration :
>
> new TokenAwarePolicy(
> new DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME, "WDC")),
> true)
>
>
> any idea what is that we are missing which is leading to this skewed data
> read patterns
>
> cassandra drivers as below:
>
> <dependency>
> <groupId>com.datastax.cassandra</groupId>
> <artifactId>cassandra-driver-core</artifactId>
> <version>2.1.6</version>
> </dependency>
> <dependency>
> <groupId>com.datastax.cassandra</groupId>
> <artifactId>cassandra-driver-mapping</artifactId>
> <version>2.1.6</version>
> </dependency>
>
> cassandra version is 2.0.17
>
> Thanks in advance for the help.
>
> Anishek
>
>