You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saijal Chauhan <sa...@goevive.com> on 2020/08/17 12:00:08 UTC
Load discrepancy between old vs new nodes of Cassandra
Hi,
We are using Cassandra 3.0.13
We have the following datacenters:
- DC1 with 7 Cassandra nodes with RF:3 (2 years old)
- DC2 with 1 Cassandra node with RF:1 (4 years old)
- DC3 with 2 Cassandra nodes with RF:2 (one-month-old)
On DC2 and DC3, each node has 100% data.
Seed nodes while setting up new-datacenter DC3 were:
- 1 DC2 node
- 1 DC1 node
We are planning to remove the 4-year-old data center (DC2).
We are seeing a discrepancy (around ~250GB) in the load of the nodes in DC2
and DC3 via the "nodetool status" command.
Datacenter: DC2
> ================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 2.2.2.2 747.57 GB 256 100.0% efgh
> RAC1
>
> Datacenter: DC3
> ================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 3.3.3.3 504.57 GB 256 100.0% ijkl
> RAC1
> UN 4.4.4.4 502.17 GB 256 100.0% mnop
> RAC1
>
>
What could be the possible reasons for the 250gb data discrepancy?
Also, we run a repair on every weekend.
Thank you!
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Also, we are observing a decrease in some Gb's load in our Cassandra
cluster, every time we restart a particular node in the cluster.
Could it be because of the stale data being removed?
If not what could be other possible reasons for that.
On Wed, Aug 19, 2020 at 6:28 PM Saijal Chauhan <sa...@goevive.com>
wrote:
> Thanks, Johny
> But with "nodetool status keyspace" also we are getting the same result as
> "nodetool status"
>
> On Tue, Aug 18, 2020 at 4:33 PM Johnny Miller <jo...@digitalis.io> wrote:
>
>> Saijal,
>>
>> Try running it for specific keyspaces as from what I understand if your
>> cluster has keyspaces with differing replication factors and you run
>> nodetool status without providing a keyspace, the information it provides
>> is not going to be accurate.
>>
>> Regards,
>>
>> Johnny
>>
>> On Tue, 18 Aug 2020 at 12:24, Saijal Chauhan <sa...@goevive.com>
>> wrote:
>>
>>> Hi,
>>>
>>> These are the following replication settings that we have in our cluster.
>>> All the application related keyspaces have been replicated to the new dc
>>> DC3.
>>> Also, the weekend repair should eliminate all the possibilities of
>>> missing data, if any.
>>> Not sure what the issue could be here.
>>> [image: image.png]
>>>
>>>
>>> DFW3 = DC1 (2 years old)
>>> IAD2 = DC2 (4 years old)
>>> IAD3 = DC3 (one-month-old)
>>>
>>> Thanks for your reply!
>>>
>>> On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <
>>> erick.ramirez@datastax.com> wrote:
>>>
>>>> I would start by checking the replication settings on all your
>>>> keyspaces. There's a chance that you have keyspaces not replicated to DC3.
>>>> FWIW it would have to be an application keyspace (vs system keyspaces)
>>>> because of the size. Cheers!
>>>>
>>>
>>>
>>> --
>>>
>>> Saijal Chauhan
>>>
>>> Infrastructure Systems Engineer
>>>
>>> Evive
>>>
>>> goevive.com
>>>
>>>
>>
>>
>>
>>
>> --
>>
>> The information contained in this electronic message and any attachments
>> to this message are intended for the exclusive use of the addressee(s) and
>> may contain proprietary, confidential or privileged information. If you are
>> not the intended recipient, you should not disseminate, distribute or copy
>> this e-mail. Please notify the sender immediately and destroy all copies of
>> this message and any attachments. WARNING: Computer viruses can be
>> transmitted via email. The recipient should check this email and any
>> attachments for the presence of viruses. The company accepts no liability
>> for any damage caused by any virus transmitted by this email.
>> www.digitalis.io
>>
>
>
>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Thanks, Johny
But with "nodetool status keyspace" also we are getting the same result as
"nodetool status"
On Tue, Aug 18, 2020 at 4:33 PM Johnny Miller <jo...@digitalis.io> wrote:
> Saijal,
>
> Try running it for specific keyspaces as from what I understand if your
> cluster has keyspaces with differing replication factors and you run
> nodetool status without providing a keyspace, the information it provides
> is not going to be accurate.
>
> Regards,
>
> Johnny
>
> On Tue, 18 Aug 2020 at 12:24, Saijal Chauhan <sa...@goevive.com>
> wrote:
>
>> Hi,
>>
>> These are the following replication settings that we have in our cluster.
>> All the application related keyspaces have been replicated to the new dc
>> DC3.
>> Also, the weekend repair should eliminate all the possibilities of
>> missing data, if any.
>> Not sure what the issue could be here.
>> [image: image.png]
>>
>>
>> DFW3 = DC1 (2 years old)
>> IAD2 = DC2 (4 years old)
>> IAD3 = DC3 (one-month-old)
>>
>> Thanks for your reply!
>>
>> On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
>> wrote:
>>
>>> I would start by checking the replication settings on all your
>>> keyspaces. There's a chance that you have keyspaces not replicated to DC3.
>>> FWIW it would have to be an application keyspace (vs system keyspaces)
>>> because of the size. Cheers!
>>>
>>
>>
>> --
>>
>> Saijal Chauhan
>>
>> Infrastructure Systems Engineer
>>
>> Evive
>>
>> goevive.com
>>
>>
>
>
>
>
> --
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.digitalis.io
>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Johnny Miller <jo...@digitalis.io>.
Saijal,
Try running it for specific keyspaces as from what I understand if your
cluster has keyspaces with differing replication factors and you run
nodetool status without providing a keyspace, the information it provides
is not going to be accurate.
Regards,
Johnny
On Tue, 18 Aug 2020 at 12:24, Saijal Chauhan <sa...@goevive.com>
wrote:
> Hi,
>
> These are the following replication settings that we have in our cluster.
> All the application related keyspaces have been replicated to the new dc
> DC3.
> Also, the weekend repair should eliminate all the possibilities of
> missing data, if any.
> Not sure what the issue could be here.
> [image: image.png]
>
>
> DFW3 = DC1 (2 years old)
> IAD2 = DC2 (4 years old)
> IAD3 = DC3 (one-month-old)
>
> Thanks for your reply!
>
> On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
> wrote:
>
>> I would start by checking the replication settings on all your keyspaces.
>> There's a chance that you have keyspaces not replicated to DC3. FWIW it
>> would have to be an application keyspace (vs system keyspaces) because of
>> the size. Cheers!
>>
>
>
> --
>
> Saijal Chauhan
>
> Infrastructure Systems Engineer
>
> Evive
>
> goevive.com
>
>
--
--
The information contained in this electronic message and any
attachments to this message are intended for the exclusive use of the
addressee(s) and may contain proprietary, confidential or privileged
information. If you are not the intended recipient, you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email. www.digitalis.io <http://www.digitalis.io>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Hi,
These are the following replication settings that we have in our cluster.
All the application related keyspaces have been replicated to the new dc
DC3.
Also, the weekend repair should eliminate all the possibilities of
missing data, if any.
Not sure what the issue could be here.
[image: image.png]
DFW3 = DC1 (2 years old)
IAD2 = DC2 (4 years old)
IAD3 = DC3 (one-month-old)
Thanks for your reply!
On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
wrote:
> I would start by checking the replication settings on all your keyspaces.
> There's a chance that you have keyspaces not replicated to DC3. FWIW it
> would have to be an application keyspace (vs system keyspaces) because of
> the size. Cheers!
>
--
Saijal Chauhan
Infrastructure Systems Engineer
Evive
goevive.com
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
Saijal
My bad, it is read repair which does not propagate expired tombstones.
Regards
Manish
On Tue, Aug 18, 2020 at 4:45 PM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> Saijal
>
> Probably you have lots of expired tombstones, expired tombstones are not
> replicated during repair.
>
> Regards
> Manish
>
> On Tue, Aug 18, 2020 at 4:34 PM Saijal Chauhan <sa...@goevive.com>
> wrote:
>
>> Hi,
>>
>> These are the following replication settings that we have in our cluster.
>> All the application related keyspaces have been replicated to the new dc
>> DC3.
>> Also, the weekend repair should eliminate all the possibilities of
>> missing data, if any.
>> Not sure what the issue could be here.
>> [image: image.png]
>> DFW3 = DC1 (2 years old)
>> IAD2 = DC2 (4 years old)
>> IAD3 = DC3 (one-month-old)
>>
>> Thanks for your reply!
>>
>> On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
>> wrote:
>>
>>> I would start by checking the replication settings on all your
>>> keyspaces. There's a chance that you have keyspaces not replicated to DC3.
>>> FWIW it would have to be an application keyspace (vs system keyspaces)
>>> because of the size. Cheers!
>>>
>>
>>
>>
>>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
Saijal
Probably you have lots of expired tombstones, expired tombstones are not
replicated during repair.
Regards
Manish
On Tue, Aug 18, 2020 at 4:34 PM Saijal Chauhan <sa...@goevive.com>
wrote:
> Hi,
>
> These are the following replication settings that we have in our cluster.
> All the application related keyspaces have been replicated to the new dc
> DC3.
> Also, the weekend repair should eliminate all the possibilities of
> missing data, if any.
> Not sure what the issue could be here.
> [image: image.png]
> DFW3 = DC1 (2 years old)
> IAD2 = DC2 (4 years old)
> IAD3 = DC3 (one-month-old)
>
> Thanks for your reply!
>
> On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
> wrote:
>
>> I would start by checking the replication settings on all your keyspaces.
>> There's a chance that you have keyspaces not replicated to DC3. FWIW it
>> would have to be an application keyspace (vs system keyspaces) because of
>> the size. Cheers!
>>
>
>
>
>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Hi,
These are the following replication settings that we have in our cluster.
All the application related keyspaces have been replicated to the new dc
DC3.
Also, the weekend repair should eliminate all the possibilities of
missing data, if any.
Not sure what the issue could be here.
[image: image.png]
DFW3 = DC1 (2 years old)
IAD2 = DC2 (4 years old)
IAD3 = DC3 (one-month-old)
Thanks for your reply!
On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
wrote:
> I would start by checking the replication settings on all your keyspaces.
> There's a chance that you have keyspaces not replicated to DC3. FWIW it
> would have to be an application keyspace (vs system keyspaces) because of
> the size. Cheers!
>
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Also, we are observing a decrease in some Gb's load in our Cassandra
cluster, every time we restart a particular node in the cluster.
Could it be because of the stale data being removed?
If not what could be other possible reasons for that.
On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
wrote:
> I would start by checking the replication settings on all your keyspaces.
> There's a chance that you have keyspaces not replicated to DC3. FWIW it
> would have to be an application keyspace (vs system keyspaces) because of
> the size. Cheers!
>
--
Saijal Chauhan
Infrastructure Systems Engineer
Evive
goevive.com
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
Also, recently we have been observing a lot of repair logs like these.
>
> INFO [RepairJobTask:3] 2020-08-12 12:07:46,325 SyncTask.java:73 - [repair #aaaaaa-bbb-ccccc-dddddd-eeeeeeee] Endpoints /2.2.2.2 and /3.3.3.3 have 9146 range(s) out of sync for table_a
>
>
Could this be somehow related?
On Tue, Aug 18, 2020 at 7:26 AM Erick Ramirez <er...@datastax.com>
wrote:
> I would start by checking the replication settings on all your keyspaces.
> There's a chance that you have keyspaces not replicated to DC3. FWIW it
> would have to be an application keyspace (vs system keyspaces) because of
> the size. Cheers!
>
--
Saijal Chauhan
Infrastructure Systems Engineer
Evive
goevive.com
Re: Load discrepancy between old vs new nodes of Cassandra
Posted by Erick Ramirez <er...@datastax.com>.
I would start by checking the replication settings on all your keyspaces.
There's a chance that you have keyspaces not replicated to DC3. FWIW it
would have to be an application keyspace (vs system keyspaces) because of
the size. Cheers!