You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Anubhav Kale <An...@microsoft.com.INVALID> on 2017/05/09 21:01:58 UTC
Cassandra 2.1.13: Using JOIN_RING=False
Hello,
With some inspiration from the Cassandra Summit talk from last year, we are trying to setup a cluster with coordinator-only nodes. We setup join_ring=false in env.sh, disabled auth in YAML and the nodes are able to start just fine. However, we're running into a few problems
1] The nodes marked with join_ring=false continue to store data. Why ?
2] We tried Python driver's whitelistedpolicy. But we notice message like below, so we are not able to send queries to all nodes marked as coordinators. We also changed the Scala driver to support whitelisting, but see the same thing. What are we missing ?
3] Is there any way to concretely tell that only coordinator nodes are getting requests from clients ? We don't have OpsCenter.
Thanks !
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] Removing host not found in peers metadata: <Host: 10.80.10.128>
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.128 removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.128
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] Removing host not found in peers metadata: <Host: 10.80.10.127>
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.127 removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.127
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] Removing host not found in peers metadata: <Host: 10.80.10.129>
2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host 10.80.10.129 removed
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host 10.80.10.129
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] Finished fetching ring info
2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection] Rebuilding token map due to topology changes
2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table not found
2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table not found
2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection created
2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection for host 10.80.10.125
2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection for host 10.80.10.126
Re: Cassandra 2.1.13: Using JOIN_RING=False
Posted by Ben Bromhead <be...@instaclustr.com>.
A rough guess on the best way to do this with cql, without looking too
deeply into it:
You would likely have to implement a custom load balancing policy within
the driver that only uses coordinator nodes and/or modify
refreshNodeListAndTokenMap() behaviour in ControlConnection.java (there
should also be an equivalent class in the python driver).
On Thu, 11 May 2017 at 08:19 Dikang Gu <di...@gmail.com> wrote:
> 1. The coordinator still store system data and hints, but they should not
> store any user data since they are not part of ring.
> 2. We are using coordinator for thrift client. For cql based drivers, they
> needs to talk to nodes in the ring, so I think coordinator mode won't work
> for them.
>
> -Dikang
>
> On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale <
> Anubhav.Kale@microsoft.com.invalid> wrote:
>
>> Hello,
>>
>>
>>
>> With some inspiration from the Cassandra Summit talk from last year, we
>> are trying to setup a cluster with coordinator-only nodes. We setup
>> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to
>> start just fine. However, we’re running into a few problems
>>
>>
>>
>> 1] The nodes marked with join_ring=false continue to store data. Why ?
>>
>> 2] We tried Python driver’s whitelistedpolicy. But we notice message like
>> below, so we are not able to send queries to all nodes marked as
>> coordinators. We also changed the Scala driver to support whitelisting, but
>> see the same thing. What are we missing ?
>>
>> 3] Is there any way to concretely tell that only coordinator nodes are
>> getting requests from clients ? We don’t have OpsCenter.
>>
>>
>>
>> Thanks !
>>
>>
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: <Host: 10.80.10.128>
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.128 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.128
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: <Host: 10.80.10.127>
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.127 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.127
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Removing host not found in peers metadata: <Host: 10.80.10.129>
>>
>> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
>> 10.80.10.129 removed
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
>> 10.80.10.129
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Finished fetching ring info
>>
>> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
>> Rebuilding token map due to topology changes
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table
>> not found
>>
>> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table
>> not found
>>
>> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection
>> created
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.125
>>
>> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
>> for host 10.80.10.126
>>
>>
>>
>
>
>
> --
> Dikang
>
> --
Ben Bromhead
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer
Re: Cassandra 2.1.13: Using JOIN_RING=False
Posted by Dikang Gu <di...@gmail.com>.
1. The coordinator still store system data and hints, but they should not
store any user data since they are not part of ring.
2. We are using coordinator for thrift client. For cql based drivers, they
needs to talk to nodes in the ring, so I think coordinator mode won't work
for them.
-Dikang
On Tue, May 9, 2017 at 2:01 PM, Anubhav Kale <
Anubhav.Kale@microsoft.com.invalid> wrote:
> Hello,
>
>
>
> With some inspiration from the Cassandra Summit talk from last year, we
> are trying to setup a cluster with coordinator-only nodes. We setup
> join_ring=false in env.sh, disabled auth in YAML and the nodes are able to
> start just fine. However, we’re running into a few problems
>
>
>
> 1] The nodes marked with join_ring=false continue to store data. Why ?
>
> 2] We tried Python driver’s whitelistedpolicy. But we notice message like
> below, so we are not able to send queries to all nodes marked as
> coordinators. We also changed the Scala driver to support whitelisting, but
> see the same thing. What are we missing ?
>
> 3] Is there any way to concretely tell that only coordinator nodes are
> getting requests from clients ? We don’t have OpsCenter.
>
>
>
> Thanks !
>
>
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: <Host: 10.80.10.128>
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.128 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.128
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: <Host: 10.80.10.127>
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.127 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.127
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Removing host not found in peers metadata: <Host: 10.80.10.129>
>
> 2017-05-09 20:45:25,060 [INFO] cassandra.cluster: Cassandra host
> 10.80.10.129 removed
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: Removing host
> 10.80.10.129
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Finished fetching ring info
>
> 2017-05-09 20:45:25,060 [DEBUG] cassandra.cluster: [control connection]
> Rebuilding token map due to topology changes
>
> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user functions table
> not found
>
> 2017-05-09 20:45:25,081 [DEBUG] cassandra.metadata: user aggregates table
> not found
>
> 2017-05-09 20:45:25,098 [DEBUG] cassandra.cluster: Control connection
> created
>
> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
> for host 10.80.10.125
>
> 2017-05-09 20:45:25,099 [DEBUG] cassandra.pool: Initializing connection
> for host 10.80.10.126
>
>
>
--
Dikang