You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andrew Bialecki <an...@klaviyo.com> on 2016/11/14 05:21:53 UTC
cassandra python driver routing requests to one node?
We have an odd situation where all of a sudden of our cluster started
seeing a disproportionate number of writes go to one node. We're using the
Python driver version 3.7.1. I'm not sure if this is a driver issue or
possibly a network issue causing requests to get routed in an odd way. It's
not absolute, there are requests going to all nodes.
Tried restarting the problematic node, no luck (those are the quiet
periods). Tried restarting the clients, also no luck. Checked nodetool
status and ownership is even across the cluster.
Curious if anyone's seen this behavior before. Seems like the next step
will be to debug the client and see why it's choosing that node.
[image: Inline image 1]
--
AB
Re: cassandra python driver routing requests to one node?
Posted by Andrew Bialecki <an...@klaviyo.com>.
Is the node selection based on key deterministic across multiple clients?
If it is, that sounds plausible. For this particular workload it's
definitely possible to have a hot key / spot, but it was surprising it
wasn't three nodes that got hot, it was just one.
On Mon, Nov 14, 2016 at 6:26 PM, Alex Popescu <al...@datastax.com> wrote:
> I'm wondering if what you are seeing is https://datastax-oss.
> atlassian.net/browse/PYTHON-643 (that could still be a sign of a
> potential data hotspot)
>
> On Sun, Nov 13, 2016 at 10:57 PM, Andrew Bialecki <
> andrew.bialecki@klaviyo.com> wrote:
>
>> We're using the "default" TokenAwarePolicy. Our nodes are spread across
>> different racks within one datacenter. I've turned on debug logging for the
>> Python driver, but it doesn't look like it logs which Casandra node each
>> request goes to, but maybe I haven't got the right logging set to debug.
>>
>> On Mon, Nov 14, 2016 at 12:39 AM, Ben Slater <be...@instaclustr.com>
>> wrote:
>>
>>> What load balancing policies are you using in your client code (
>>> https://datastax.github.io/python-driver/api/cassandra/policies.html)?
>>>
>>> Cheers
>>> Ben
>>>
>>> On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki <
>>> andrew.bialecki@klaviyo.com> wrote:
>>>
>>>> We have an odd situation where all of a sudden of our cluster started
>>>> seeing a disproportionate number of writes go to one node. We're using the
>>>> Python driver version 3.7.1. I'm not sure if this is a driver issue or
>>>> possibly a network issue causing requests to get routed in an odd way. It's
>>>> not absolute, there are requests going to all nodes.
>>>>
>>>> Tried restarting the problematic node, no luck (those are the quiet
>>>> periods). Tried restarting the clients, also no luck. Checked nodetool
>>>> status and ownership is even across the cluster.
>>>>
>>>> Curious if anyone's seen this behavior before. Seems like the next step
>>>> will be to debug the client and see why it's choosing that node.
>>>>
>>>> [image: Inline image 1]
>>>>
>>>>
>>>> --
>>>> AB
>>>>
>>>
>>
>>
>> --
>> AB
>>
>
>
>
> --
> Bests,
>
> Alex Popescu | @al3xandru
> Sen. Product Manager @ DataStax
>
>
>
>
--
AB
Re: cassandra python driver routing requests to one node?
Posted by Alex Popescu <al...@datastax.com>.
I'm wondering if what you are seeing is
https://datastax-oss.atlassian.net/browse/PYTHON-643 (that could still be a
sign of a potential data hotspot)
On Sun, Nov 13, 2016 at 10:57 PM, Andrew Bialecki <
andrew.bialecki@klaviyo.com> wrote:
> We're using the "default" TokenAwarePolicy. Our nodes are spread across
> different racks within one datacenter. I've turned on debug logging for the
> Python driver, but it doesn't look like it logs which Casandra node each
> request goes to, but maybe I haven't got the right logging set to debug.
>
> On Mon, Nov 14, 2016 at 12:39 AM, Ben Slater <be...@instaclustr.com>
> wrote:
>
>> What load balancing policies are you using in your client code (
>> https://datastax.github.io/python-driver/api/cassandra/policies.html)?
>>
>> Cheers
>> Ben
>>
>> On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki <an...@klaviyo.com>
>> wrote:
>>
>>> We have an odd situation where all of a sudden of our cluster started
>>> seeing a disproportionate number of writes go to one node. We're using the
>>> Python driver version 3.7.1. I'm not sure if this is a driver issue or
>>> possibly a network issue causing requests to get routed in an odd way. It's
>>> not absolute, there are requests going to all nodes.
>>>
>>> Tried restarting the problematic node, no luck (those are the quiet
>>> periods). Tried restarting the clients, also no luck. Checked nodetool
>>> status and ownership is even across the cluster.
>>>
>>> Curious if anyone's seen this behavior before. Seems like the next step
>>> will be to debug the client and see why it's choosing that node.
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> --
>>> AB
>>>
>>
>
>
> --
> AB
>
--
Bests,
Alex Popescu | @al3xandru
Sen. Product Manager @ DataStax
Re: cassandra python driver routing requests to one node?
Posted by Andrew Bialecki <an...@klaviyo.com>.
We're using the "default" TokenAwarePolicy. Our nodes are spread across
different racks within one datacenter. I've turned on debug logging for the
Python driver, but it doesn't look like it logs which Casandra node each
request goes to, but maybe I haven't got the right logging set to debug.
On Mon, Nov 14, 2016 at 12:39 AM, Ben Slater <be...@instaclustr.com>
wrote:
> What load balancing policies are you using in your client code (
> https://datastax.github.io/python-driver/api/cassandra/policies.html)?
>
> Cheers
> Ben
>
> On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki <an...@klaviyo.com>
> wrote:
>
>> We have an odd situation where all of a sudden of our cluster started
>> seeing a disproportionate number of writes go to one node. We're using the
>> Python driver version 3.7.1. I'm not sure if this is a driver issue or
>> possibly a network issue causing requests to get routed in an odd way. It's
>> not absolute, there are requests going to all nodes.
>>
>> Tried restarting the problematic node, no luck (those are the quiet
>> periods). Tried restarting the clients, also no luck. Checked nodetool
>> status and ownership is even across the cluster.
>>
>> Curious if anyone's seen this behavior before. Seems like the next step
>> will be to debug the client and see why it's choosing that node.
>>
>> [image: Inline image 1]
>>
>>
>> --
>> AB
>>
>
--
AB
Re: cassandra python driver routing requests to one node?
Posted by Ben Slater <be...@instaclustr.com>.
What load balancing policies are you using in your client code (
https://datastax.github.io/python-driver/api/cassandra/policies.html)?
Cheers
Ben
On Mon, 14 Nov 2016 at 16:22 Andrew Bialecki <an...@klaviyo.com>
wrote:
> We have an odd situation where all of a sudden of our cluster started
> seeing a disproportionate number of writes go to one node. We're using the
> Python driver version 3.7.1. I'm not sure if this is a driver issue or
> possibly a network issue causing requests to get routed in an odd way. It's
> not absolute, there are requests going to all nodes.
>
> Tried restarting the problematic node, no luck (those are the quiet
> periods). Tried restarting the clients, also no luck. Checked nodetool
> status and ownership is even across the cluster.
>
> Curious if anyone's seen this behavior before. Seems like the next step
> will be to debug the client and see why it's choosing that node.
>
> [image: Inline image 1]
>
>
> --
> AB
>