You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Apoorva Gaurav <ap...@myntra.com> on 2014/04/01 12:50:30 UTC
Dead node appearing in datastax driver
Hello All,
We had a 4 node cassandra 2.0.4 cluster ( lets call them host1, host2,
host3 and host4), out of which we've removed one node (host4) using
nodetool removenode command. Now using nodetool status or nodetool ring we
no longer see host4. It's also not appearing in Datastax opscenter. But its
intermittently appearing in Metadata.getAllHosts() while connecting using
datastax driver 1.0.4.
Couple of questions :-
-How is it appearing.
-Can this have impact on read / write performance of client.
Code which we are using to connect is
public void connect() {
PoolingOptions poolingOptions = new PoolingOptions();
cluster = Cluster.builder()
.addContactPoints(inetAddresses.toArray(new String[]{}))
.withLoadBalancingPolicy(new RoundRobinPolicy())
.withPoolingOptions(poolingOptions)
.withPort(port)
.withCredentials(username, password)
.build();
Metadata metadata = cluster.getMetadata();
System.out.printf("Connected to cluster: %s\n",
metadata.getClusterName());
for (Host host : metadata.getAllHosts()) {
System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
host.getDatacenter(), host.getAddress(), host.getRack());
}
}
--
Thanks & Regards,
Apoorva
Re: Dead node appearing in datastax driver
Posted by Apoorva Gaurav <ap...@myntra.com>.
I manually removed entries from System.peers.
The improvements can well be coincidental as various other apps were also
running on the same test bed.
On Tue, Apr 1, 2014 at 8:43 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> What does "Did that" mean? Does that means "I upgraded to 2.0.6", or does
> that mean "I manually removed entries from System.peers". If the latter,
> I'd need more info on what you did exactly, what your peers table looked
> like before and how they look like now: there is no reason deleting the
> peers entries for hosts that at not part of the cluster anymore would have
> anything to do with write latency (but if say you've removed wrong entries,
> that might have make the driver think some live host had been removed and
> if the drivers has less nodes to use to dispatch queries, that might impact
> latency I suppose -- at least that's the only related thing I can think of).
>
> --
> Sylvain
>
>
> On Tue, Apr 1, 2014 at 2:44 PM, Apoorva Gaurav <ap...@myntra.com>wrote:
>
>> Did that and I actually see a significant reduction in write latency.
>>
>>
>> On Tue, Apr 1, 2014 at 5:35 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Tue, Apr 1, 2014 at 1:49 PM, Apoorva Gaurav <
>>> apoorva.gaurav@myntra.com> wrote:
>>>
>>>> Hello Sylvian,
>>>>
>>>> Queried system.peers on three live nodes and host4 is appearing on two
>>>> of these.
>>>>
>>>
>>> That's why the driver thinks they are still there. You're most probably
>>> running into https://issues.apache.org/jira/browse/CASSANDRA-6053 since
>>> you are on C* 2.0.4. As said, this is relatively harmless, but you should
>>> think about upgrading to 2.0.6 to fix it in the future (you could manually
>>> remove the bad entries in System.peers in the meantime if you want, they
>>> are really just leftover that shouldn't be here).
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>>
>>>> On Tue, Apr 1, 2014 at 5:06 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>>>
>>>>> On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav <
>>>>> apoorva.gaurav@myntra.com> wrote:
>>>>>
>>>>>> Hello All,
>>>>>>
>>>>>> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1,
>>>>>> host2, host3 and host4), out of which we've removed one node (host4) using
>>>>>> nodetool removenode command. Now using nodetool status or nodetool ring we
>>>>>> no longer see host4. It's also not appearing in Datastax opscenter. But its
>>>>>> intermittently appearing in Metadata.getAllHosts() while connecting using
>>>>>> datastax driver 1.0.4.
>>>>>>
>>>>>> Couple of questions :-
>>>>>> -How is it appearing.
>>>>>>
>>>>>
>>>>> Not sure. Can you try querying the peers system table on each of your
>>>>> nodes (with cqlsh: SELECT * FROM system.peers) and see if the host4 is
>>>>> still mentioned somewhere?
>>>>>
>>>>>
>>>>>> -Can this have impact on read / write performance of client.
>>>>>>
>>>>>
>>>>> No. If the host doesn't exists, the driver might try to reconnect to
>>>>> it at times, but since it won't be able to, it won't try to use it for
>>>>> reads and writes. That does mean you might have a reconnection task running
>>>>> with some regularity, but 1) it's not on the write/read path of queries and
>>>>> 2) provided you've left the default reconnection policy, this will happen
>>>>> once every 10 minutes and will be pretty cheap so that it will consume an
>>>>> completely negligible amount of ressources. That doesn't mean I'm not
>>>>> interested tracking down why that happens in the first place though.
>>>>>
>>>>> --
>>>>> Sylvain
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Code which we are using to connect is
>>>>>>
>>>>>> public void connect() {
>>>>>>
>>>>>> PoolingOptions poolingOptions = new PoolingOptions();
>>>>>>
>>>>>> cluster = Cluster.builder()
>>>>>>
>>>>>> .addContactPoints(inetAddresses.toArray(newString[]{}))
>>>>>>
>>>>>> .withLoadBalancingPolicy(new RoundRobinPolicy())
>>>>>>
>>>>>> .withPoolingOptions(poolingOptions)
>>>>>>
>>>>>> .withPort(port)
>>>>>>
>>>>>> .withCredentials(username, password)
>>>>>>
>>>>>> .build();
>>>>>>
>>>>>> Metadata metadata = cluster.getMetadata();
>>>>>>
>>>>>> System.out.printf("Connected to cluster: %s\n",
>>>>>> metadata.getClusterName());
>>>>>>
>>>>>> for (Host host : metadata.getAllHosts()) {
>>>>>>
>>>>>> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
>>>>>> host.getDatacenter(), host.getAddress(), host.getRack());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Apoorva
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Apoorva
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Apoorva
>>
>
>
--
Thanks & Regards,
Apoorva
Re: Dead node appearing in datastax driver
Posted by Sylvain Lebresne <sy...@datastax.com>.
What does "Did that" mean? Does that means "I upgraded to 2.0.6", or does
that mean "I manually removed entries from System.peers". If the latter,
I'd need more info on what you did exactly, what your peers table looked
like before and how they look like now: there is no reason deleting the
peers entries for hosts that at not part of the cluster anymore would have
anything to do with write latency (but if say you've removed wrong entries,
that might have make the driver think some live host had been removed and
if the drivers has less nodes to use to dispatch queries, that might impact
latency I suppose -- at least that's the only related thing I can think of).
--
Sylvain
On Tue, Apr 1, 2014 at 2:44 PM, Apoorva Gaurav <ap...@myntra.com>wrote:
> Did that and I actually see a significant reduction in write latency.
>
>
> On Tue, Apr 1, 2014 at 5:35 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Tue, Apr 1, 2014 at 1:49 PM, Apoorva Gaurav <apoorva.gaurav@myntra.com
>> > wrote:
>>
>>> Hello Sylvian,
>>>
>>> Queried system.peers on three live nodes and host4 is appearing on two
>>> of these.
>>>
>>
>> That's why the driver thinks they are still there. You're most probably
>> running into https://issues.apache.org/jira/browse/CASSANDRA-6053 since
>> you are on C* 2.0.4. As said, this is relatively harmless, but you should
>> think about upgrading to 2.0.6 to fix it in the future (you could manually
>> remove the bad entries in System.peers in the meantime if you want, they
>> are really just leftover that shouldn't be here).
>>
>> --
>> Sylvain
>>
>>
>>>
>>> On Tue, Apr 1, 2014 at 5:06 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>>
>>>> On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav <
>>>> apoorva.gaurav@myntra.com> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1,
>>>>> host2, host3 and host4), out of which we've removed one node (host4) using
>>>>> nodetool removenode command. Now using nodetool status or nodetool ring we
>>>>> no longer see host4. It's also not appearing in Datastax opscenter. But its
>>>>> intermittently appearing in Metadata.getAllHosts() while connecting using
>>>>> datastax driver 1.0.4.
>>>>>
>>>>> Couple of questions :-
>>>>> -How is it appearing.
>>>>>
>>>>
>>>> Not sure. Can you try querying the peers system table on each of your
>>>> nodes (with cqlsh: SELECT * FROM system.peers) and see if the host4 is
>>>> still mentioned somewhere?
>>>>
>>>>
>>>>> -Can this have impact on read / write performance of client.
>>>>>
>>>>
>>>> No. If the host doesn't exists, the driver might try to reconnect to it
>>>> at times, but since it won't be able to, it won't try to use it for reads
>>>> and writes. That does mean you might have a reconnection task running with
>>>> some regularity, but 1) it's not on the write/read path of queries and 2)
>>>> provided you've left the default reconnection policy, this will happen once
>>>> every 10 minutes and will be pretty cheap so that it will consume an
>>>> completely negligible amount of ressources. That doesn't mean I'm not
>>>> interested tracking down why that happens in the first place though.
>>>>
>>>> --
>>>> Sylvain
>>>>
>>>>
>>>>
>>>>>
>>>>> Code which we are using to connect is
>>>>>
>>>>> public void connect() {
>>>>>
>>>>> PoolingOptions poolingOptions = new PoolingOptions();
>>>>>
>>>>> cluster = Cluster.builder()
>>>>>
>>>>> .addContactPoints(inetAddresses.toArray(newString[]{}))
>>>>>
>>>>> .withLoadBalancingPolicy(new RoundRobinPolicy())
>>>>>
>>>>> .withPoolingOptions(poolingOptions)
>>>>>
>>>>> .withPort(port)
>>>>>
>>>>> .withCredentials(username, password)
>>>>>
>>>>> .build();
>>>>>
>>>>> Metadata metadata = cluster.getMetadata();
>>>>>
>>>>> System.out.printf("Connected to cluster: %s\n",
>>>>> metadata.getClusterName());
>>>>>
>>>>> for (Host host : metadata.getAllHosts()) {
>>>>>
>>>>> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
>>>>> host.getDatacenter(), host.getAddress(), host.getRack());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Apoorva
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Apoorva
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Apoorva
>
Re: Dead node appearing in datastax driver
Posted by Apoorva Gaurav <ap...@myntra.com>.
Did that and I actually see a significant reduction in write latency.
On Tue, Apr 1, 2014 at 5:35 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Tue, Apr 1, 2014 at 1:49 PM, Apoorva Gaurav <ap...@myntra.com>wrote:
>
>> Hello Sylvian,
>>
>> Queried system.peers on three live nodes and host4 is appearing on two of
>> these.
>>
>
> That's why the driver thinks they are still there. You're most probably
> running into https://issues.apache.org/jira/browse/CASSANDRA-6053 since
> you are on C* 2.0.4. As said, this is relatively harmless, but you should
> think about upgrading to 2.0.6 to fix it in the future (you could manually
> remove the bad entries in System.peers in the meantime if you want, they
> are really just leftover that shouldn't be here).
>
> --
> Sylvain
>
>
>>
>> On Tue, Apr 1, 2014 at 5:06 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>>
>>> On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav <
>>> apoorva.gaurav@myntra.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1, host2,
>>>> host3 and host4), out of which we've removed one node (host4) using
>>>> nodetool removenode command. Now using nodetool status or nodetool ring we
>>>> no longer see host4. It's also not appearing in Datastax opscenter. But its
>>>> intermittently appearing in Metadata.getAllHosts() while connecting using
>>>> datastax driver 1.0.4.
>>>>
>>>> Couple of questions :-
>>>> -How is it appearing.
>>>>
>>>
>>> Not sure. Can you try querying the peers system table on each of your
>>> nodes (with cqlsh: SELECT * FROM system.peers) and see if the host4 is
>>> still mentioned somewhere?
>>>
>>>
>>>> -Can this have impact on read / write performance of client.
>>>>
>>>
>>> No. If the host doesn't exists, the driver might try to reconnect to it
>>> at times, but since it won't be able to, it won't try to use it for reads
>>> and writes. That does mean you might have a reconnection task running with
>>> some regularity, but 1) it's not on the write/read path of queries and 2)
>>> provided you've left the default reconnection policy, this will happen once
>>> every 10 minutes and will be pretty cheap so that it will consume an
>>> completely negligible amount of ressources. That doesn't mean I'm not
>>> interested tracking down why that happens in the first place though.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>>>
>>>> Code which we are using to connect is
>>>>
>>>> public void connect() {
>>>>
>>>> PoolingOptions poolingOptions = new PoolingOptions();
>>>>
>>>> cluster = Cluster.builder()
>>>>
>>>> .addContactPoints(inetAddresses.toArray(newString[]{}))
>>>>
>>>> .withLoadBalancingPolicy(new RoundRobinPolicy())
>>>>
>>>> .withPoolingOptions(poolingOptions)
>>>>
>>>> .withPort(port)
>>>>
>>>> .withCredentials(username, password)
>>>>
>>>> .build();
>>>>
>>>> Metadata metadata = cluster.getMetadata();
>>>>
>>>> System.out.printf("Connected to cluster: %s\n",
>>>> metadata.getClusterName());
>>>>
>>>> for (Host host : metadata.getAllHosts()) {
>>>>
>>>> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
>>>> host.getDatacenter(), host.getAddress(), host.getRack());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Apoorva
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Apoorva
>>
>
>
--
Thanks & Regards,
Apoorva
Re: Dead node appearing in datastax driver
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, Apr 1, 2014 at 1:49 PM, Apoorva Gaurav <ap...@myntra.com>wrote:
> Hello Sylvian,
>
> Queried system.peers on three live nodes and host4 is appearing on two of
> these.
>
That's why the driver thinks they are still there. You're most probably
running into https://issues.apache.org/jira/browse/CASSANDRA-6053 since you
are on C* 2.0.4. As said, this is relatively harmless, but you should think
about upgrading to 2.0.6 to fix it in the future (you could manually remove
the bad entries in System.peers in the meantime if you want, they are
really just leftover that shouldn't be here).
--
Sylvain
>
> On Tue, Apr 1, 2014 at 5:06 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav <
>> apoorva.gaurav@myntra.com> wrote:
>>
>>> Hello All,
>>>
>>> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1, host2,
>>> host3 and host4), out of which we've removed one node (host4) using
>>> nodetool removenode command. Now using nodetool status or nodetool ring we
>>> no longer see host4. It's also not appearing in Datastax opscenter. But its
>>> intermittently appearing in Metadata.getAllHosts() while connecting using
>>> datastax driver 1.0.4.
>>>
>>> Couple of questions :-
>>> -How is it appearing.
>>>
>>
>> Not sure. Can you try querying the peers system table on each of your
>> nodes (with cqlsh: SELECT * FROM system.peers) and see if the host4 is
>> still mentioned somewhere?
>>
>>
>>> -Can this have impact on read / write performance of client.
>>>
>>
>> No. If the host doesn't exists, the driver might try to reconnect to it
>> at times, but since it won't be able to, it won't try to use it for reads
>> and writes. That does mean you might have a reconnection task running with
>> some regularity, but 1) it's not on the write/read path of queries and 2)
>> provided you've left the default reconnection policy, this will happen once
>> every 10 minutes and will be pretty cheap so that it will consume an
>> completely negligible amount of ressources. That doesn't mean I'm not
>> interested tracking down why that happens in the first place though.
>>
>> --
>> Sylvain
>>
>>
>>
>>>
>>> Code which we are using to connect is
>>>
>>> public void connect() {
>>>
>>> PoolingOptions poolingOptions = new PoolingOptions();
>>>
>>> cluster = Cluster.builder()
>>>
>>> .addContactPoints(inetAddresses.toArray(new String[]{}))
>>>
>>> .withLoadBalancingPolicy(new RoundRobinPolicy())
>>>
>>> .withPoolingOptions(poolingOptions)
>>>
>>> .withPort(port)
>>>
>>> .withCredentials(username, password)
>>>
>>> .build();
>>>
>>> Metadata metadata = cluster.getMetadata();
>>>
>>> System.out.printf("Connected to cluster: %s\n",
>>> metadata.getClusterName());
>>>
>>> for (Host host : metadata.getAllHosts()) {
>>>
>>> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
>>> host.getDatacenter(), host.getAddress(), host.getRack());
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Apoorva
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Apoorva
>
Re: Dead node appearing in datastax driver
Posted by Apoorva Gaurav <ap...@myntra.com>.
Hello Sylvian,
Queried system.peers on three live nodes and host4 is appearing on two of
these.
On Tue, Apr 1, 2014 at 5:06 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav <apoorva.gaurav@myntra.com
> > wrote:
>
>> Hello All,
>>
>> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1, host2,
>> host3 and host4), out of which we've removed one node (host4) using
>> nodetool removenode command. Now using nodetool status or nodetool ring we
>> no longer see host4. It's also not appearing in Datastax opscenter. But its
>> intermittently appearing in Metadata.getAllHosts() while connecting using
>> datastax driver 1.0.4.
>>
>> Couple of questions :-
>> -How is it appearing.
>>
>
> Not sure. Can you try querying the peers system table on each of your
> nodes (with cqlsh: SELECT * FROM system.peers) and see if the host4 is
> still mentioned somewhere?
>
>
>> -Can this have impact on read / write performance of client.
>>
>
> No. If the host doesn't exists, the driver might try to reconnect to it at
> times, but since it won't be able to, it won't try to use it for reads and
> writes. That does mean you might have a reconnection task running with some
> regularity, but 1) it's not on the write/read path of queries and 2)
> provided you've left the default reconnection policy, this will happen once
> every 10 minutes and will be pretty cheap so that it will consume an
> completely negligible amount of ressources. That doesn't mean I'm not
> interested tracking down why that happens in the first place though.
>
> --
> Sylvain
>
>
>
>>
>> Code which we are using to connect is
>>
>> public void connect() {
>>
>> PoolingOptions poolingOptions = new PoolingOptions();
>>
>> cluster = Cluster.builder()
>>
>> .addContactPoints(inetAddresses.toArray(new String[]{}))
>>
>> .withLoadBalancingPolicy(new RoundRobinPolicy())
>>
>> .withPoolingOptions(poolingOptions)
>>
>> .withPort(port)
>>
>> .withCredentials(username, password)
>>
>> .build();
>>
>> Metadata metadata = cluster.getMetadata();
>>
>> System.out.printf("Connected to cluster: %s\n",
>> metadata.getClusterName());
>>
>> for (Host host : metadata.getAllHosts()) {
>>
>> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
>> host.getDatacenter(), host.getAddress(), host.getRack());
>>
>> }
>>
>> }
>>
>>
>>
>> --
>> Thanks & Regards,
>> Apoorva
>>
>
>
--
Thanks & Regards,
Apoorva
Re: Dead node appearing in datastax driver
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, Apr 1, 2014 at 12:50 PM, Apoorva Gaurav
<ap...@myntra.com>wrote:
> Hello All,
>
> We had a 4 node cassandra 2.0.4 cluster ( lets call them host1, host2,
> host3 and host4), out of which we've removed one node (host4) using
> nodetool removenode command. Now using nodetool status or nodetool ring we
> no longer see host4. It's also not appearing in Datastax opscenter. But its
> intermittently appearing in Metadata.getAllHosts() while connecting using
> datastax driver 1.0.4.
>
> Couple of questions :-
> -How is it appearing.
>
Not sure. Can you try querying the peers system table on each of your nodes
(with cqlsh: SELECT * FROM system.peers) and see if the host4 is still
mentioned somewhere?
> -Can this have impact on read / write performance of client.
>
No. If the host doesn't exists, the driver might try to reconnect to it at
times, but since it won't be able to, it won't try to use it for reads and
writes. That does mean you might have a reconnection task running with some
regularity, but 1) it's not on the write/read path of queries and 2)
provided you've left the default reconnection policy, this will happen once
every 10 minutes and will be pretty cheap so that it will consume an
completely negligible amount of ressources. That doesn't mean I'm not
interested tracking down why that happens in the first place though.
--
Sylvain
>
> Code which we are using to connect is
>
> public void connect() {
>
> PoolingOptions poolingOptions = new PoolingOptions();
>
> cluster = Cluster.builder()
>
> .addContactPoints(inetAddresses.toArray(new String[]{}))
>
> .withLoadBalancingPolicy(new RoundRobinPolicy())
>
> .withPoolingOptions(poolingOptions)
>
> .withPort(port)
>
> .withCredentials(username, password)
>
> .build();
>
> Metadata metadata = cluster.getMetadata();
>
> System.out.printf("Connected to cluster: %s\n",
> metadata.getClusterName());
>
> for (Host host : metadata.getAllHosts()) {
>
> System.out.printf("Datacenter: %s; Host: %s; Rack: %s\n",
> host.getDatacenter(), host.getAddress(), host.getRack());
>
> }
>
> }
>
>
>
> --
> Thanks & Regards,
> Apoorva
>