You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bertrand Brelier <be...@gmail.com> on 2016/11/20 18:35:59 UTC
data not replicated on new node
Hello everybody,
I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
I recently added a new node (to make it a 3-node cluster).
I am using a replication factor of 3 , so I expected to have a copy of
the same data on each node :
CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
But the new node has less data that the other 2 :
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
On the new node :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 1.2G
on one of the old nodes :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 52G
I am monitoring the amount of data on each node, and they grow at the
same rate. So I suspect that my new data are replicated on the 3 nodes
but the old data stored on the first 2 nodes are not replicated on the
new node.
I ran nodetool repair (on each node, one at a time), but the new node
still does not have a copy of the old data.
Could you please help me understand why the old data is not replicated
to the new node ? Please let me know if you need further information.
Thank you,
Cheers,
Bertrand
Re: data not replicated on new node
Posted by Malte Pickhan <ma...@zalando.de>.
Not sure if it's really related, but we experienced something similar last
friday. I summarized it in the following Issue:
https://issues.apache.org/jira/browse/CASSANDRA-12947
Best,
Malte
2016-11-23 10:21 GMT+01:00 Oleksandr Shulgin <ol...@zalando.de>:
> On Tue, Nov 22, 2016 at 5:23 PM, Bertrand Brelier <
> bertrand.brelier@gmail.com> wrote:
>
>> Hello Shalom.
>>
>> No I really went from 3.1.1 to 3.0.9 .
>>
> So you've just installed the 3.0.9 version and re-started with it? I
> wonder if it's really supported?
>
> Regards,
> --
> Alex
>
>
Re: data not replicated on new node
Posted by Malte Pickhan <ma...@zalando.de>.
Not sure if it's really related, but we experienced something similar last
friday. I summarized it in the following Issue:
https://issues.apache.org/jira/browse/CASSANDRA-12947
Best,
Malte
2016-11-23 10:21 GMT+01:00 Oleksandr Shulgin <ol...@zalando.de>:
> On Tue, Nov 22, 2016 at 5:23 PM, Bertrand Brelier <
> bertrand.brelier@gmail.com> wrote:
>
>> Hello Shalom.
>>
>> No I really went from 3.1.1 to 3.0.9 .
>>
> So you've just installed the 3.0.9 version and re-started with it? I
> wonder if it's really supported?
>
> Regards,
> --
> Alex
>
>
Re: data not replicated on new node
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Tue, Nov 22, 2016 at 5:23 PM, Bertrand Brelier <
bertrand.brelier@gmail.com> wrote:
> Hello Shalom.
>
> No I really went from 3.1.1 to 3.0.9 .
>
So you've just installed the 3.0.9 version and re-started with it? I
wonder if it's really supported?
Regards,
--
Alex
Re: data not replicated on new node
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Tue, Nov 22, 2016 at 5:23 PM, Bertrand Brelier <
bertrand.brelier@gmail.com> wrote:
> Hello Shalom.
>
> No I really went from 3.1.1 to 3.0.9 .
>
So you've just installed the 3.0.9 version and re-started with it? I
wonder if it's really supported?
Regards,
--
Alex
Re: data not replicated on new node
Posted by Bertrand Brelier <be...@gmail.com>.
Hello Shalom.
No I really went from 3.1.1 to 3.0.9 .
Cheers.
Bertrand
On Nov 22, 2016 1:57 AM, "Shalom Sagges" <sh...@liveperson.com> wrote:
>
> *I took that opportunity to upgrade from 3.1.1 to 3.0.9*
>
> If my guess is right and you meant that you upgraded from 2.1.1 to 3.0.9
> directly, then this might cause some issues (not necessarily the issue at
> hand though). The proper upgrade process should be to 2.1.9 and from there
> upgrade to 3.0.x.
>
> https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/
> upgrdCassandra.html
>
> Hope this helps.
>
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
> <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
>
>
> On Tue, Nov 22, 2016 at 2:44 AM, Bertrand Brelier <
> bertrand.brelier@gmail.com> wrote:
>
>> Hello Shalom, Vladimir,
>>
>> Thanks for your help.
>>
>> I had initially 3 nodes, had a hardware failure and reinstalled Cassandra
>> on the node (I took that opportunity to upgrade from 3.1.1 to 3.0.9). I ran
>> nodetool upgradesstables and nodetool repair on each node once I updated
>> Cassandra.
>>
>> The 3 nodes are in the same private network, I am using the private IPs
>> for the seeds and the listen_address and the public IPs for rpc_address
>>
>> I am using ssl to encrypt the communication between the nodes, so I am
>> using the port 7001 :
>>
>> telnet PRIVATEIP 7001
>> Trying PRIVATEIP...
>> Connected to PRIVATEIP.
>>
>> Each node can connect with any other node.
>>
>> I selected some old data from the new node :
>>
>> CONSISTENCY;
>> Current consistency level is ONE.
>> select count(*) from ....;
>>
>> count
>> -------
>> 0
>>
>> CONSISTENCY ALL;
>> Consistency level set to ALL.
>> count(*) from ....;
>>
>> count
>> -------
>> 64
>>
>> When I switched to ALL I could get the data while the initial level ONE
>> did not have any data. I did not expect to get any data with ALL, am I
>> missing something ?
>>
>> I do not know if this is related, but while I was inquiring the database,
>> I had the following messages in the debug.log :
>>
>> DEBUG [ReadRepairStage:15292] 2016-11-21 18:15:59,719
>> ReadCallback.java:234 - Digest mismatch:
>> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
>> DecoratedKey(2288259866140251828, 00040000002a0000040000421500)
>> (d41d8cd98f00b204e9800998ecf8427e vs ce211ac5533e1a146d9fee734fd8de26)
>> at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
>> ~[apache-cassandra-3.0.10.jar:3.0.10]
>> at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:225)
>> ~[apache-cassandra-3.0.10.jar:3.0.10]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_111]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_111]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
>>
>>
>> Thanks for your help,
>>
>> Cheers,
>>
>> Bertrand
>>
>>
>> On 16-11-21 01:28 AM, Shalom Sagges wrote:
>>
>> I believe the logs should show you what the issue is.
>> Also, can the node "talk" with the others? (i.e. telnet to the other
>> nodes on port 7000).
>>
>>
>> Shalom Sagges
>> DBA
>> T: +972-74-700-4035
>> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
>> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>>
>> <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
>>
>>
>> On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier <
>> bertrand.brelier@gmail.com> wrote:
>>
>>> Hello Jonathan,
>>>
>>> No, the new node is not a seed in my cluster.
>>>
>>> When I ran nodetool bootstrap resume
>>> Node is already bootstrapped.
>>>
>>> Cheers,
>>>
>>> Bertrand
>>>
>>> On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jo...@jonhaddad.com>
>>> wrote:
>>>
>>>> Did you add the new node as a seed? If you did, it wouldn't bootstrap,
>>>> and you should run repair.
>>>> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <
>>>> bertrand.brelier@gmail.com> wrote:
>>>>
>>>>> Hello everybody,
>>>>>
>>>>> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>>>>>
>>>>> I recently added a new node (to make it a 3-node cluster).
>>>>>
>>>>> I am using a replication factor of 3 , so I expected to have a copy of
>>>>> the same data on each node :
>>>>>
>>>>> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
>>>>> 'replication_factor': '3'} AND durable_writes = true;
>>>>>
>>>>> But the new node has less data that the other 2 :
>>>>>
>>>>> Datacenter: datacenter1
>>>>> =======================
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> -- Address Load Tokens Owns (effective) Host
>>>>> ID Rack
>>>>> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
>>>>> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
>>>>> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>>>>>
>>>>>
>>>>> On the new node :
>>>>>
>>>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>>>> total 1.2G
>>>>>
>>>>> on one of the old nodes :
>>>>>
>>>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>>>> total 52G
>>>>>
>>>>>
>>>>> I am monitoring the amount of data on each node, and they grow at the
>>>>> same rate. So I suspect that my new data are replicated on the 3 nodes
>>>>> but the old data stored on the first 2 nodes are not replicated on the
>>>>> new node.
>>>>>
>>>>> I ran nodetool repair (on each node, one at a time), but the new node
>>>>> still does not have a copy of the old data.
>>>>>
>>>>> Could you please help me understand why the old data is not replicated
>>>>> to the new node ? Please let me know if you need further information.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Bertrand
>>>>>
>>>>>
>>>
>>
>> This message may contain confidential and/or privileged information.
>> If you are not the addressee or authorized to receive this on behalf of
>> the addressee you must not use, copy, disclose or take action based on this
>> message or any information herein.
>> If you have received this message in error, please advise the sender
>> immediately by reply email and delete this message. Thank you.
>>
>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
Re: data not replicated on new node
Posted by Shalom Sagges <sh...@liveperson.com>.
*I took that opportunity to upgrade from 3.1.1 to 3.0.9*
If my guess is right and you meant that you upgraded from 2.1.1 to 3.0.9
directly, then this might cause some issues (not necessarily the issue at
hand though). The proper upgrade process should be to 2.1.9 and from there
upgrade to 3.0.x.
https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgrdCassandra.html
Hope this helps.
Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
On Tue, Nov 22, 2016 at 2:44 AM, Bertrand Brelier <
bertrand.brelier@gmail.com> wrote:
> Hello Shalom, Vladimir,
>
> Thanks for your help.
>
> I had initially 3 nodes, had a hardware failure and reinstalled Cassandra
> on the node (I took that opportunity to upgrade from 3.1.1 to 3.0.9). I ran
> nodetool upgradesstables and nodetool repair on each node once I updated
> Cassandra.
>
> The 3 nodes are in the same private network, I am using the private IPs
> for the seeds and the listen_address and the public IPs for rpc_address
>
> I am using ssl to encrypt the communication between the nodes, so I am
> using the port 7001 :
>
> telnet PRIVATEIP 7001
> Trying PRIVATEIP...
> Connected to PRIVATEIP.
>
> Each node can connect with any other node.
>
> I selected some old data from the new node :
>
> CONSISTENCY;
> Current consistency level is ONE.
> select count(*) from ....;
>
> count
> -------
> 0
>
> CONSISTENCY ALL;
> Consistency level set to ALL.
> count(*) from ....;
>
> count
> -------
> 64
>
> When I switched to ALL I could get the data while the initial level ONE
> did not have any data. I did not expect to get any data with ALL, am I
> missing something ?
>
> I do not know if this is related, but while I was inquiring the database,
> I had the following messages in the debug.log :
>
> DEBUG [ReadRepairStage:15292] 2016-11-21 18:15:59,719
> ReadCallback.java:234 - Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key
> DecoratedKey(2288259866140251828, 00040000002a0000040000421500) (
> d41d8cd98f00b204e9800998ecf8427e vs ce211ac5533e1a146d9fee734fd8de26)
> at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.service.ReadCallback$
> AsyncRepairRunner.run(ReadCallback.java:225)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
>
>
> Thanks for your help,
>
> Cheers,
>
> Bertrand
>
>
> On 16-11-21 01:28 AM, Shalom Sagges wrote:
>
> I believe the logs should show you what the issue is.
> Also, can the node "talk" with the others? (i.e. telnet to the other nodes
> on port 7000).
>
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035
> <http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
> <http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
>
> <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
>
>
> On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier <
> bertrand.brelier@gmail.com> wrote:
>
>> Hello Jonathan,
>>
>> No, the new node is not a seed in my cluster.
>>
>> When I ran nodetool bootstrap resume
>> Node is already bootstrapped.
>>
>> Cheers,
>>
>> Bertrand
>>
>> On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>>> Did you add the new node as a seed? If you did, it wouldn't bootstrap,
>>> and you should run repair.
>>> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <
>>> bertrand.brelier@gmail.com> wrote:
>>>
>>>> Hello everybody,
>>>>
>>>> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>>>>
>>>> I recently added a new node (to make it a 3-node cluster).
>>>>
>>>> I am using a replication factor of 3 , so I expected to have a copy of
>>>> the same data on each node :
>>>>
>>>> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
>>>> 'replication_factor': '3'} AND durable_writes = true;
>>>>
>>>> But the new node has less data that the other 2 :
>>>>
>>>> Datacenter: datacenter1
>>>> =======================
>>>> Status=Up/Down
>>>> |/ State=Normal/Leaving/Joining/Moving
>>>> -- Address Load Tokens Owns (effective) Host
>>>> ID Rack
>>>> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
>>>> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
>>>> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>>>>
>>>>
>>>> On the new node :
>>>>
>>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>>> total 1.2G
>>>>
>>>> on one of the old nodes :
>>>>
>>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>>> total 52G
>>>>
>>>>
>>>> I am monitoring the amount of data on each node, and they grow at the
>>>> same rate. So I suspect that my new data are replicated on the 3 nodes
>>>> but the old data stored on the first 2 nodes are not replicated on the
>>>> new node.
>>>>
>>>> I ran nodetool repair (on each node, one at a time), but the new node
>>>> still does not have a copy of the old data.
>>>>
>>>> Could you please help me understand why the old data is not replicated
>>>> to the new node ? Please let me know if you need further information.
>>>>
>>>> Thank you,
>>>>
>>>> Cheers,
>>>>
>>>> Bertrand
>>>>
>>>>
>>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf of
> the addressee you must not use, copy, disclose or take action based on this
> message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
>
>
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.
Re: data not replicated on new node
Posted by Bertrand Brelier <be...@gmail.com>.
Hello Shalom, Vladimir,
Thanks for your help.
I had initially 3 nodes, had a hardware failure and reinstalled
Cassandra on the node (I took that opportunity to upgrade from 3.1.1 to
3.0.9). I ran nodetool upgradesstables and nodetool repair on each node
once I updated Cassandra.
The 3 nodes are in the same private network, I am using the private IPs
for the seeds and the listen_address and the public IPs for rpc_address
I am using ssl to encrypt the communication between the nodes, so I am
using the port 7001 :
telnet PRIVATEIP 7001
Trying PRIVATEIP...
Connected to PRIVATEIP.
Each node can connect with any other node.
I selected some old data from the new node :
CONSISTENCY;
Current consistency level is ONE.
select count(*) from ....;
count
-------
0
CONSISTENCY ALL;
Consistency level set to ALL.
count(*) from ....;
count
-------
64
When I switched to ALL I could get the data while the initial level ONE
did not have any data. I did not expect to get any data with ALL, am I
missing something ?
I do not know if this is related, but while I was inquiring the
database, I had the following messages in the debug.log :
DEBUG [ReadRepairStage:15292] 2016-11-21 18:15:59,719
ReadCallback.java:234 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key
DecoratedKey(2288259866140251828, 00040000002a0000040000421500)
(d41d8cd98f00b204e9800998ecf8427e vs ce211ac5533e1a146d9fee734fd8de26)
at
org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85)
~[apache-cassandra-3.0.10.jar:3.0.10]
at
org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:225)
~[apache-cassandra-3.0.10.jar:3.0.10]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Thanks for your help,
Cheers,
Bertrand
On 16-11-21 01:28 AM, Shalom Sagges wrote:
> I believe the logs should show you what the issue is.
> Also, can the node "talk" with the others? (i.e. telnet to the other
> nodes on port 7000).
>
> Shalom Sagges
> DBA
> T: +972-74-700-4035
> <http://www.linkedin.com/company/164748>
> <http://twitter.com/liveperson> <http://www.facebook.com/LivePersonInc>
>
> We Create Meaningful Connections
>
> <https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
>
>
>
> On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier
> <bertrand.brelier@gmail.com <ma...@gmail.com>> wrote:
>
> Hello Jonathan,
>
> No, the new node is not a seed in my cluster.
>
> When I ran nodetool bootstrap resume
> Node is already bootstrapped.
>
> Cheers,
>
> Bertrand
>
> On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad
> <jon@jonhaddad.com <ma...@jonhaddad.com>> wrote:
>
> Did you add the new node as a seed? If you did, it wouldn't
> bootstrap, and you should run repair.
> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier
> <bertrand.brelier@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello everybody,
>
> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>
> I recently added a new node (to make it a 3-node cluster).
>
> I am using a replication factor of 3 , so I expected to
> have a copy of
> the same data on each node :
>
> CREATE KEYSPACE mydata WITH replication = {'class':
> 'SimpleStrategy',
> 'replication_factor': '3'} AND durable_writes = true;
>
> But the new node has less data that the other 2 :
>
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns
> (effective) Host
> ID Rack
> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>
>
> On the new node :
>
> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
> total 1.2G
>
> on one of the old nodes :
>
> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
> total 52G
>
>
> I am monitoring the amount of data on each node, and they
> grow at the
> same rate. So I suspect that my new data are replicated on
> the 3 nodes
> but the old data stored on the first 2 nodes are not
> replicated on the
> new node.
>
> I ran nodetool repair (on each node, one at a time), but
> the new node
> still does not have a copy of the old data.
>
> Could you please help me understand why the old data is
> not replicated
> to the new node ? Please let me know if you need further
> information.
>
> Thank you,
>
> Cheers,
>
> Bertrand
>
>
>
>
> This message may contain confidential and/or privileged information.
> If you are not the addressee or authorized to receive this on behalf
> of the addressee you must not use, copy, disclose or take action based
> on this message or any information herein.
> If you have received this message in error, please advise the sender
> immediately by reply email and delete this message. Thank you.
Re: data not replicated on new node
Posted by Vladimir Yudovin <vl...@winguzone.com>.
>try SELECT for some undoubtedly old data with consistency ALL
It's worth to turn trace on for this query.
Best regards, Vladimir Yudovin,
Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.
---- On Mon, 21 Nov 2016 02:05:35 -0500Vladimir Yudovin <vladyu@winguzone.com> wrote ----
>can the node "talk" with the others? (i.e. telnet to the other nodes on port 7000).
According to nodetool status all nodes are joined and UP, so they seems to be able talk one with other (though there still can be connectivity issues). Are all nodes on internal network (e.g. 10.x.x.x) in the same locations?
>I suspect that ... the old data stored on the first 2 nodes are not replicated on the new node.
Can you try SELECT for some undoubtedly old data with consistency ALL (just one raw is enough)?
Also the whole event sequence are not clear: you had two nodes with data, then added third. When did you change replication factor for existing keyspace? Or it was created with factor three?
Best regards, Vladimir Yudovin,
Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.
---- On Mon, 21 Nov 2016 01:28:54 -0500Shalom Sagges <shaloms@liveperson.com> wrote ----
I believe the logs should show you what the issue is.
Also, can the node "talk" with the others? (i.e. telnet to the other nodes on port 7000).
Shalom Sagges
DBA
T: +972-74-700-4035
We Create Meaningful Connections
On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier <bertrand.brelier@gmail.com> wrote:
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein.
If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
Hello Jonathan,
No, the new node is not a seed in my cluster.
When I ran nodetool bootstrap resume
Node is already bootstrapped.
Cheers,
Bertrand
On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jon@jonhaddad.com> wrote:
Did you add the new node as a seed? If you did, it wouldn't bootstrap, and you should run repair.
On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <bertrand.brelier@gmail.com> wrote:
Hello everybody,
I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
I recently added a new node (to make it a 3-node cluster).
I am using a replication factor of 3 , so I expected to have a copy of
the same data on each node :
CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
But the new node has less data that the other 2 :
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
On the new node :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 1.2G
on one of the old nodes :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 52G
I am monitoring the amount of data on each node, and they grow at the
same rate. So I suspect that my new data are replicated on the 3 nodes
but the old data stored on the first 2 nodes are not replicated on the
new node.
I ran nodetool repair (on each node, one at a time), but the new node
still does not have a copy of the old data.
Could you please help me understand why the old data is not replicated
to the new node ? Please let me know if you need further information.
Thank you,
Cheers,
Bertrand
Re: data not replicated on new node
Posted by Vladimir Yudovin <vl...@winguzone.com>.
>can the node "talk" with the others? (i.e. telnet to the other nodes on port 7000).
According to nodetool status all nodes are joined and UP, so they seems to be able talk one with other (though there still can be connectivity issues). Are all nodes on internal network (e.g. 10.x.x.x) in the same locations?
>I suspect that ... the old data stored on the first 2 nodes are not replicated on the new node.
Can you try SELECT for some undoubtedly old data with consistency ALL (just one raw is enough)?
Also the whole event sequence are not clear: you had two nodes with data, then added third. When did you change replication factor for existing keyspace? Or it was created with factor three?
Best regards, Vladimir Yudovin,
Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.
---- On Mon, 21 Nov 2016 01:28:54 -0500Shalom Sagges <shaloms@liveperson.com> wrote ----
I believe the logs should show you what the issue is.
Also, can the node "talk" with the others? (i.e. telnet to the other nodes on port 7000).
Shalom Sagges
DBA
T: +972-74-700-4035
We Create Meaningful Connections
On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier <bertrand.brelier@gmail.com> wrote:
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein.
If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.
Hello Jonathan,
No, the new node is not a seed in my cluster.
When I ran nodetool bootstrap resume
Node is already bootstrapped.
Cheers,
Bertrand
On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jon@jonhaddad.com> wrote:
Did you add the new node as a seed? If you did, it wouldn't bootstrap, and you should run repair.
On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <bertrand.brelier@gmail.com> wrote:
Hello everybody,
I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
I recently added a new node (to make it a 3-node cluster).
I am using a replication factor of 3 , so I expected to have a copy of
the same data on each node :
CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
But the new node has less data that the other 2 :
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host
ID Rack
UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
On the new node :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 1.2G
on one of the old nodes :
/XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
total 52G
I am monitoring the amount of data on each node, and they grow at the
same rate. So I suspect that my new data are replicated on the 3 nodes
but the old data stored on the first 2 nodes are not replicated on the
new node.
I ran nodetool repair (on each node, one at a time), but the new node
still does not have a copy of the old data.
Could you please help me understand why the old data is not replicated
to the new node ? Please let me know if you need further information.
Thank you,
Cheers,
Bertrand
Re: data not replicated on new node
Posted by Shalom Sagges <sh...@liveperson.com>.
I believe the logs should show you what the issue is.
Also, can the node "talk" with the others? (i.e. telnet to the other nodes
on port 7000).
Shalom Sagges
DBA
T: +972-74-700-4035
<http://www.linkedin.com/company/164748> <http://twitter.com/liveperson>
<http://www.facebook.com/LivePersonInc> We Create Meaningful Connections
<https://engage.liveperson.com/idc-mobile-first-consumer/?utm_medium=email&utm_source=mkto&utm_campaign=idcsig>
On Sun, Nov 20, 2016 at 8:50 PM, Bertrand Brelier <
bertrand.brelier@gmail.com> wrote:
> Hello Jonathan,
>
> No, the new node is not a seed in my cluster.
>
> When I ran nodetool bootstrap resume
> Node is already bootstrapped.
>
> Cheers,
>
> Bertrand
>
> On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
>> Did you add the new node as a seed? If you did, it wouldn't bootstrap,
>> and you should run repair.
>> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <
>> bertrand.brelier@gmail.com> wrote:
>>
>>> Hello everybody,
>>>
>>> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>>>
>>> I recently added a new node (to make it a 3-node cluster).
>>>
>>> I am using a replication factor of 3 , so I expected to have a copy of
>>> the same data on each node :
>>>
>>> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
>>> 'replication_factor': '3'} AND durable_writes = true;
>>>
>>> But the new node has less data that the other 2 :
>>>
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host
>>> ID Rack
>>> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
>>> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
>>> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>>>
>>>
>>> On the new node :
>>>
>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>> total 1.2G
>>>
>>> on one of the old nodes :
>>>
>>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>>> total 52G
>>>
>>>
>>> I am monitoring the amount of data on each node, and they grow at the
>>> same rate. So I suspect that my new data are replicated on the 3 nodes
>>> but the old data stored on the first 2 nodes are not replicated on the
>>> new node.
>>>
>>> I ran nodetool repair (on each node, one at a time), but the new node
>>> still does not have a copy of the old data.
>>>
>>> Could you please help me understand why the old data is not replicated
>>> to the new node ? Please let me know if you need further information.
>>>
>>> Thank you,
>>>
>>> Cheers,
>>>
>>> Bertrand
>>>
>>>
>
--
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this on behalf of the
addressee you must not use, copy, disclose or take action based on this
message or any information herein.
If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Thank you.
Re: data not replicated on new node
Posted by Bertrand Brelier <be...@gmail.com>.
Hello Jonathan,
No, the new node is not a seed in my cluster.
When I ran nodetool bootstrap resume
Node is already bootstrapped.
Cheers,
Bertrand
On Sun, Nov 20, 2016 at 1:43 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> Did you add the new node as a seed? If you did, it wouldn't bootstrap, and
> you should run repair.
> On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <
> bertrand.brelier@gmail.com> wrote:
>
>> Hello everybody,
>>
>> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>>
>> I recently added a new node (to make it a 3-node cluster).
>>
>> I am using a replication factor of 3 , so I expected to have a copy of
>> the same data on each node :
>>
>> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '3'} AND durable_writes = true;
>>
>> But the new node has less data that the other 2 :
>>
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host
>> ID Rack
>> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
>> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
>> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>>
>>
>> On the new node :
>>
>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>> total 1.2G
>>
>> on one of the old nodes :
>>
>> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
>> total 52G
>>
>>
>> I am monitoring the amount of data on each node, and they grow at the
>> same rate. So I suspect that my new data are replicated on the 3 nodes
>> but the old data stored on the first 2 nodes are not replicated on the
>> new node.
>>
>> I ran nodetool repair (on each node, one at a time), but the new node
>> still does not have a copy of the old data.
>>
>> Could you please help me understand why the old data is not replicated
>> to the new node ? Please let me know if you need further information.
>>
>> Thank you,
>>
>> Cheers,
>>
>> Bertrand
>>
>>
Re: data not replicated on new node
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
Did you add the new node as a seed? If you did, it wouldn't bootstrap, and
you should run repair.
On Sun, Nov 20, 2016 at 10:36 AM Bertrand Brelier <
bertrand.brelier@gmail.com> wrote:
> Hello everybody,
>
> I am using a 3-node Cassandra cluster with Cassandra 3.0.10.
>
> I recently added a new node (to make it a 3-node cluster).
>
> I am using a replication factor of 3 , so I expected to have a copy of
> the same data on each node :
>
> CREATE KEYSPACE mydata WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'} AND durable_writes = true;
>
> But the new node has less data that the other 2 :
>
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host
> ID Rack
> UN XXX.XXX.XXX.XXX 53.28 GB 256 100.0% xxxxxx rack1
> UN XXX.XXX.XXX.XXX 64.7 GB 256 100.0% xxxxxx rack1
> UN XXX.XXX.XXX.XXX 1.28 GB 256 100.0% xxxxxx rack1
>
>
> On the new node :
>
> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
> total 1.2G
>
> on one of the old nodes :
>
> /XXXXXX/data-6d674a40efab11e5b67e6d75503d5d02/:
> total 52G
>
>
> I am monitoring the amount of data on each node, and they grow at the
> same rate. So I suspect that my new data are replicated on the 3 nodes
> but the old data stored on the first 2 nodes are not replicated on the
> new node.
>
> I ran nodetool repair (on each node, one at a time), but the new node
> still does not have a copy of the old data.
>
> Could you please help me understand why the old data is not replicated
> to the new node ? Please let me know if you need further information.
>
> Thank you,
>
> Cheers,
>
> Bertrand
>
>