You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by raman gugnani <ra...@gmail.com> on 2024/01/10 06:20:51 UTC
unbalanced data in second datacenter
We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
We have now added a new data centre *mumbai* consisting of 10 nodes
m5.4xlarge instances and version 3.11.4 and RF = 3.
Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
Load distribution is very uneven. *What could be the reason for an uneven
Load ?*
[image: Screenshot 2024-01-10 at 11.24.53 AM.png]
--
Raman Gugnani
Re: unbalanced data in second datacenter
Posted by Arvydas Jonusonis <ar...@gmail.com>.
Another thing that could cause a node to bootstrap empty is if it is in the
seed list at the time of bootstrap (maybe a seed node was removed and
subsequently re-bootsrapped without removing from seed list?)
Arvydas
On Thu, Jan 11, 2024 at 02:23 Bowen Song via user <us...@cassandra.apache.org>
wrote:
> The effective ownership information suggest the rack/RF misconfiguration
> was *not* the cause of the data size imbalance between nodes. You may be
> looking at incorrect (e.g. from the wrong DC), incomplete or forgotten
> bootstrapping if "auto_bootstrap" was set to "false". Without knowing what
> were the steps taken to initially setup the DC and the parameters for
> keyspaces, nodes, etc., it's hard to guess what's causing the imbalance.
>
>
> On 11/01/2024 06:54, raman gugnani wrote:
>
> HI Bowen,
>
> Understood the same , is the above reason could cause load difference max
> 1.11Tib and min 51.56 Gib in new region (*mumbai*) while previous region
> have 2.72Tib and 2.19Tib (*ap-south-1*) both have 2 racks.
>
> On Thu, 11 Jan 2024 at 08:26, manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>> Totally agree with Bowen there. Well explained.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 10, 2024 at 10:24 PM Bowen Song via user <
>> user@cassandra.apache.org> wrote:
>>
>>> I don't think the following statement is correct:
>>>
>>> *> When using racks one should always have a number of racks equal to or
>>> more than RF for equal distribution*
>>>
>>> For example, 3 racks with RF=6 is fine, as long as there's 2 or more
>>> nodes in each rack. In this case, each rack will have exactly two copies of
>>> the same data, the data is evenly distributed across the racks, and the
>>> system can continue to serve QUORUM (EACH/LOCAL_QUORUM where appropriate)
>>> queries with a single rack failure.
>>>
>>> I believe the correct statement is:
>>>
>>> *The replication factor for a DC should be either a multiple of or less
>>> than the number of racks in the corresponding DC.*
>>>
>>> This should allow an even distribution of data across the racks, and the
>>> DC can still meet the QUORUM CL with a single rack failure, as long as the
>>> number of racks and RF both are no less than 3. Keep in mind that quorum of
>>> 2 is 2 (i.e. all).
>>>
>>> That means, for a specific DC, if it has
>>>
>>> - 1 rack, RF can be any positive integer: 1, 2, 3, 4, 5, ...
>>> - 2 racks, RF should be 1 or a positive even number: 1, 2, 4, 6, 8,
>>> ...
>>> - 3 racks, RF should be 1, 2 or a multiple of 3: 1, 2, 3, 6, 9, ...
>>> - 4 racks, RF should be less than 4 or a multiple of 4: 1, 2, 3, 4,
>>> 8, ..
>>>
>>> or, if it has:
>>>
>>> - RF=1 or RF=2, number of racks can be any positive integer: 1, 2,
>>> 3, 4, 5, ...
>>> - RF=3, number of racks can be 1 or any integer >= 3: 1, 3, 4, 5, 6,
>>> ...
>>> - RF=4, number of racks can be 1, 2 or any integer >= 4: 1, 2, 4, 5,
>>> 6, ...
>>>
>>> However, in practice, the recommendation is to set the number of racks
>>> to either 1 or equal to the RF. It is *not* recommended to have the
>>> Cassandra (logical) racks accurately represents the physical configuration
>>> of servers in a server room, because Cassandra is likely going to prefer
>>> more nodes in a (logical) rack than the number of servers can fit in a
>>> (physical) rack, and also because adding or removing a (physical) rack of
>>> servers to/from a server room is easy, but adding or removing a (logical)
>>> rack to/from an existing Cassandra DC is not.
>>>
>>>
>>> On 10/01/2024 06:45, manish khandelwal wrote:
>>>
>>> I see you have two racks with RF=3. For better distribution, you should
>>> have three racks . When using racks one should always have a number of
>>> racks equal to or more than RF for equal distribution.
>>>
>>> Slight differences in load distribution (with rack/ without rack) is
>>> expected as it is a random distribution of data based on vnodes.
>>>
>>> Regards
>>> Manish
>>>
>>> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <
>>> ramangugnani.007@gmail.com> wrote:
>>>
>>>> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
>>>> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
>>>> We have now added a new data centre *mumbai* consisting of 10 nodes
>>>> m5.4xlarge instances and version 3.11.4 and RF = 3.
>>>>
>>>>
>>>> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
>>>> Load distribution is very uneven. *What could be the reason for an
>>>> uneven Load ?*
>>>>
>>>>
>>>> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raman Gugnani
>>>>
>>>
>
> --
> Raman Gugnani
>
>
Re: unbalanced data in second datacenter
Posted by Bowen Song via user <us...@cassandra.apache.org>.
The effective ownership information suggest the rack/RF misconfiguration
was *not* the cause of the data size imbalance between nodes. You may be
looking at incorrect (e.g. from the wrong DC), incomplete or forgotten
bootstrapping if "auto_bootstrap" was set to "false". Without knowing
what were the steps taken to initially setup the DC and the parameters
for keyspaces, nodes, etc., it's hard to guess what's causing the imbalance.
On 11/01/2024 06:54, raman gugnani wrote:
> HI Bowen,
>
> Understood the same , is the above reason could cause load difference
> max 1.11Tib and min 51.56 Gib in new region (*mumbai*) while previous
> region have 2.72Tib and 2.19Tib (*ap-south-1*) both have 2 racks.
>
> On Thu, 11 Jan 2024 at 08:26, manish khandelwal
> <ma...@gmail.com> wrote:
>
> Totally agree with Bowen there. Well explained.
>
> Regards
> Manish
>
> On Wed, Jan 10, 2024 at 10:24 PM Bowen Song via user
> <us...@cassandra.apache.org> wrote:
>
> I don't think the following statement is correct:
>
> /> When using racks one should *always* have a number of racks
> equal to or more than RF for equal distribution/
>
> For example, 3 racks with RF=6 is fine, as long as there's 2
> or more nodes in each rack. In this case, each rack will have
> exactly two copies of the same data, the data is evenly
> distributed across the racks, and the system can continue to
> serve QUORUM (EACH/LOCAL_QUORUM where appropriate) queries
> with a single rack failure.
>
> I believe the correct statement is:
>
> /The replication factor for a DC should be either a multiple
> of or less than the number of racks in the corresponding DC./
>
> This should allow an even distribution of data across the
> racks, and the DC can still meet the QUORUM CL with a single
> rack failure, as long as the number of racks and RF both are
> no less than 3. Keep in mind that quorum of 2 is 2 (i.e. all).
>
> That means, for a specific DC, if it has
>
> * 1 rack, RF can be any positive integer: 1, 2, 3, 4, 5, ...
> * 2 racks, RF should be 1 or a positive even number: 1, 2,
> 4, 6, 8, ...
> * 3 racks, RF should be 1, 2 or a multiple of 3: 1, 2, 3, 6,
> 9, ...
> * 4 racks, RF should be less than 4 or a multiple of 4: 1,
> 2, 3, 4, 8, ..
>
> or, if it has:
>
> * RF=1 or RF=2, number of racks can be any positive integer:
> 1, 2, 3, 4, 5, ...
> * RF=3, number of racks can be 1 or any integer >= 3: 1, 3,
> 4, 5, 6, ...
> * RF=4, number of racks can be 1, 2 or any integer >= 4: 1,
> 2, 4, 5, 6, ...
>
> However, in practice, the recommendation is to set the number
> of racks to either 1 or equal to the RF. It is *not*
> recommended to have the Cassandra (logical) racks accurately
> represents the physical configuration of servers in a server
> room, because Cassandra is likely going to prefer more nodes
> in a (logical) rack than the number of servers can fit in a
> (physical) rack, and also because adding or removing a
> (physical) rack of servers to/from a server room is easy, but
> adding or removing a (logical) rack to/from an existing
> Cassandra DC is not.
>
>
> On 10/01/2024 06:45, manish khandelwal wrote:
>> I see you have two racks with RF=3. For better distribution,
>> you should have three racks . When using racks one should
>> always have a number of racks equal to or more than RF for
>> equal distribution.
>>
>> Slight differences in load distribution (with rack/ without
>> rack) is expected as it is a random distribution of data
>> based on vnodes.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani
>> <ra...@gmail.com> wrote:
>>
>> We had one datacenter *ap-south-1* consisting of 5 nodes
>> m5.4xlarge instances with average size of 2.2~2.3 Tib and
>> version 3.11.2 and RF = 3.
>> We have now added a new data centre *mumbai* consisting
>> of 10 nodes m5.4xlarge instances and version 3.11.4 and
>> RF = 3.
>>
>>
>> Ownership in ap-south-1 is 60% per node and in mumbai 30%
>> per node, but Load distribution is very uneven. *What
>> could be the reason for an uneven Load ?*
>>
>>
>> Screenshot 2024-01-10 at 11.24.53 AM.png
>>
>>
>>
>>
>> --
>> Raman Gugnani
>>
>
>
> --
> Raman Gugnani
Re: unbalanced data in second datacenter
Posted by raman gugnani <ra...@gmail.com>.
HI Bowen,
Understood the same , is the above reason could cause load difference max
1.11Tib and min 51.56 Gib in new region (*mumbai*) while previous region
have 2.72Tib and 2.19Tib (*ap-south-1*) both have 2 racks.
On Thu, 11 Jan 2024 at 08:26, manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> Totally agree with Bowen there. Well explained.
>
> Regards
> Manish
>
> On Wed, Jan 10, 2024 at 10:24 PM Bowen Song via user <
> user@cassandra.apache.org> wrote:
>
>> I don't think the following statement is correct:
>>
>> *> When using racks one should always have a number of racks equal to or
>> more than RF for equal distribution*
>>
>> For example, 3 racks with RF=6 is fine, as long as there's 2 or more
>> nodes in each rack. In this case, each rack will have exactly two copies of
>> the same data, the data is evenly distributed across the racks, and the
>> system can continue to serve QUORUM (EACH/LOCAL_QUORUM where appropriate)
>> queries with a single rack failure.
>>
>> I believe the correct statement is:
>>
>> *The replication factor for a DC should be either a multiple of or less
>> than the number of racks in the corresponding DC.*
>>
>> This should allow an even distribution of data across the racks, and the
>> DC can still meet the QUORUM CL with a single rack failure, as long as the
>> number of racks and RF both are no less than 3. Keep in mind that quorum of
>> 2 is 2 (i.e. all).
>>
>> That means, for a specific DC, if it has
>>
>> - 1 rack, RF can be any positive integer: 1, 2, 3, 4, 5, ...
>> - 2 racks, RF should be 1 or a positive even number: 1, 2, 4, 6, 8,
>> ...
>> - 3 racks, RF should be 1, 2 or a multiple of 3: 1, 2, 3, 6, 9, ...
>> - 4 racks, RF should be less than 4 or a multiple of 4: 1, 2, 3, 4,
>> 8, ..
>>
>> or, if it has:
>>
>> - RF=1 or RF=2, number of racks can be any positive integer: 1, 2, 3,
>> 4, 5, ...
>> - RF=3, number of racks can be 1 or any integer >= 3: 1, 3, 4, 5, 6,
>> ...
>> - RF=4, number of racks can be 1, 2 or any integer >= 4: 1, 2, 4, 5,
>> 6, ...
>>
>> However, in practice, the recommendation is to set the number of racks to
>> either 1 or equal to the RF. It is *not* recommended to have the
>> Cassandra (logical) racks accurately represents the physical configuration
>> of servers in a server room, because Cassandra is likely going to prefer
>> more nodes in a (logical) rack than the number of servers can fit in a
>> (physical) rack, and also because adding or removing a (physical) rack of
>> servers to/from a server room is easy, but adding or removing a (logical)
>> rack to/from an existing Cassandra DC is not.
>>
>>
>> On 10/01/2024 06:45, manish khandelwal wrote:
>>
>> I see you have two racks with RF=3. For better distribution, you should
>> have three racks . When using racks one should always have a number of
>> racks equal to or more than RF for equal distribution.
>>
>> Slight differences in load distribution (with rack/ without rack) is
>> expected as it is a random distribution of data based on vnodes.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <
>> ramangugnani.007@gmail.com> wrote:
>>
>>> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
>>> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
>>> We have now added a new data centre *mumbai* consisting of 10 nodes
>>> m5.4xlarge instances and version 3.11.4 and RF = 3.
>>>
>>>
>>> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
>>> Load distribution is very uneven. *What could be the reason for an
>>> uneven Load ?*
>>>
>>>
>>> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>>>
>>>
>>>
>>>
>>> --
>>> Raman Gugnani
>>>
>>
--
Raman Gugnani
Re: unbalanced data in second datacenter
Posted by manish khandelwal <ma...@gmail.com>.
Totally agree with Bowen there. Well explained.
Regards
Manish
On Wed, Jan 10, 2024 at 10:24 PM Bowen Song via user <
user@cassandra.apache.org> wrote:
> I don't think the following statement is correct:
>
> *> When using racks one should always have a number of racks equal to or
> more than RF for equal distribution*
>
> For example, 3 racks with RF=6 is fine, as long as there's 2 or more nodes
> in each rack. In this case, each rack will have exactly two copies of the
> same data, the data is evenly distributed across the racks, and the system
> can continue to serve QUORUM (EACH/LOCAL_QUORUM where appropriate) queries
> with a single rack failure.
>
> I believe the correct statement is:
>
> *The replication factor for a DC should be either a multiple of or less
> than the number of racks in the corresponding DC.*
>
> This should allow an even distribution of data across the racks, and the
> DC can still meet the QUORUM CL with a single rack failure, as long as the
> number of racks and RF both are no less than 3. Keep in mind that quorum of
> 2 is 2 (i.e. all).
>
> That means, for a specific DC, if it has
>
> - 1 rack, RF can be any positive integer: 1, 2, 3, 4, 5, ...
> - 2 racks, RF should be 1 or a positive even number: 1, 2, 4, 6, 8, ...
> - 3 racks, RF should be 1, 2 or a multiple of 3: 1, 2, 3, 6, 9, ...
> - 4 racks, RF should be less than 4 or a multiple of 4: 1, 2, 3, 4, 8,
> ..
>
> or, if it has:
>
> - RF=1 or RF=2, number of racks can be any positive integer: 1, 2, 3,
> 4, 5, ...
> - RF=3, number of racks can be 1 or any integer >= 3: 1, 3, 4, 5, 6,
> ...
> - RF=4, number of racks can be 1, 2 or any integer >= 4: 1, 2, 4, 5,
> 6, ...
>
> However, in practice, the recommendation is to set the number of racks to
> either 1 or equal to the RF. It is *not* recommended to have the
> Cassandra (logical) racks accurately represents the physical configuration
> of servers in a server room, because Cassandra is likely going to prefer
> more nodes in a (logical) rack than the number of servers can fit in a
> (physical) rack, and also because adding or removing a (physical) rack of
> servers to/from a server room is easy, but adding or removing a (logical)
> rack to/from an existing Cassandra DC is not.
>
>
> On 10/01/2024 06:45, manish khandelwal wrote:
>
> I see you have two racks with RF=3. For better distribution, you should
> have three racks . When using racks one should always have a number of
> racks equal to or more than RF for equal distribution.
>
> Slight differences in load distribution (with rack/ without rack) is
> expected as it is a random distribution of data based on vnodes.
>
> Regards
> Manish
>
> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <ra...@gmail.com>
> wrote:
>
>> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
>> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
>> We have now added a new data centre *mumbai* consisting of 10 nodes
>> m5.4xlarge instances and version 3.11.4 and RF = 3.
>>
>>
>> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
>> Load distribution is very uneven. *What could be the reason for an
>> uneven Load ?*
>>
>>
>> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>>
>>
>>
>>
>> --
>> Raman Gugnani
>>
>
Re: unbalanced data in second datacenter
Posted by Bowen Song via user <us...@cassandra.apache.org>.
I don't think the following statement is correct:
/> When using racks one should *always* have a number of racks equal to
or more than RF for equal distribution/
For example, 3 racks with RF=6 is fine, as long as there's 2 or more
nodes in each rack. In this case, each rack will have exactly two copies
of the same data, the data is evenly distributed across the racks, and
the system can continue to serve QUORUM (EACH/LOCAL_QUORUM where
appropriate) queries with a single rack failure.
I believe the correct statement is:
/The replication factor for a DC should be either a multiple of or less
than the number of racks in the corresponding DC./
This should allow an even distribution of data across the racks, and the
DC can still meet the QUORUM CL with a single rack failure, as long as
the number of racks and RF both are no less than 3. Keep in mind that
quorum of 2 is 2 (i.e. all).
That means, for a specific DC, if it has
* 1 rack, RF can be any positive integer: 1, 2, 3, 4, 5, ...
* 2 racks, RF should be 1 or a positive even number: 1, 2, 4, 6, 8, ...
* 3 racks, RF should be 1, 2 or a multiple of 3: 1, 2, 3, 6, 9, ...
* 4 racks, RF should be less than 4 or a multiple of 4: 1, 2, 3, 4, 8, ..
or, if it has:
* RF=1 or RF=2, number of racks can be any positive integer: 1, 2, 3,
4, 5, ...
* RF=3, number of racks can be 1 or any integer >= 3: 1, 3, 4, 5, 6, ...
* RF=4, number of racks can be 1, 2 or any integer >= 4: 1, 2, 4, 5,
6, ...
However, in practice, the recommendation is to set the number of racks
to either 1 or equal to the RF. It is *not* recommended to have the
Cassandra (logical) racks accurately represents the physical
configuration of servers in a server room, because Cassandra is likely
going to prefer more nodes in a (logical) rack than the number of
servers can fit in a (physical) rack, and also because adding or
removing a (physical) rack of servers to/from a server room is easy, but
adding or removing a (logical) rack to/from an existing Cassandra DC is not.
On 10/01/2024 06:45, manish khandelwal wrote:
> I see you have two racks with RF=3. For better distribution, you
> should have three racks . When using racks one should always have a
> number of racks equal to or more than RF for equal distribution.
>
> Slight differences in load distribution (with rack/ without rack) is
> expected as it is a random distribution of data based on vnodes.
>
> Regards
> Manish
>
> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani
> <ra...@gmail.com> wrote:
>
> We had one datacenter *ap-south-1* consisting of 5 nodes
> m5.4xlarge instances with average size of 2.2~2.3 Tib and version
> 3.11.2 and RF = 3.
> We have now added a new data centre *mumbai* consisting of 10
> nodes m5.4xlarge instances and version 3.11.4 and RF = 3.
>
>
> Ownership in ap-south-1 is 60% per node and in mumbai 30% per
> node, but Load distribution is very uneven. *What could be the
> reason for an uneven Load ?*
>
>
> Screenshot 2024-01-10 at 11.24.53 AM.png
>
>
>
>
> --
> Raman Gugnani
>
Re: unbalanced data in second datacenter
Posted by manish khandelwal <ma...@gmail.com>.
That's what I mentioned "*number of racks in your case is problematic*'',
If that was not the case then load difference wouldn't be that high. Since
you have only two racks with RF=3, you are bound to create such imbalances.
On Wed, Jan 10, 2024 at 1:43 PM raman gugnani <ra...@gmail.com>
wrote:
> A slight difference is acceptable. Here the difference is 1TB vs 51GiB.
>
> On Wed, 10 Jan 2024 at 12:16, manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>> I see you have two racks with RF=3. For better distribution, you should
>> have three racks . When using racks one should always have a number of
>> racks equal to or more than RF for equal distribution.
>>
>> Slight differences in load distribution (with rack/ without rack) is
>> expected as it is a random distribution of data based on vnodes.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <
>> ramangugnani.007@gmail.com> wrote:
>>
>>> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
>>> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
>>> We have now added a new data centre *mumbai* consisting of 10 nodes
>>> m5.4xlarge instances and version 3.11.4 and RF = 3.
>>>
>>>
>>> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
>>> Load distribution is very uneven. *What could be the reason for an
>>> uneven Load ?*
>>>
>>>
>>> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>>>
>>>
>>>
>>>
>>> --
>>> Raman Gugnani
>>>
>>
>
> --
> Raman Gugnani
>
Re: unbalanced data in second datacenter
Posted by raman gugnani <ra...@gmail.com>.
A slight difference is acceptable. Here the difference is 1TB vs 51GiB.
On Wed, 10 Jan 2024 at 12:16, manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> I see you have two racks with RF=3. For better distribution, you should
> have three racks . When using racks one should always have a number of
> racks equal to or more than RF for equal distribution.
>
> Slight differences in load distribution (with rack/ without rack) is
> expected as it is a random distribution of data based on vnodes.
>
> Regards
> Manish
>
> On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <ra...@gmail.com>
> wrote:
>
>> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
>> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
>> We have now added a new data centre *mumbai* consisting of 10 nodes
>> m5.4xlarge instances and version 3.11.4 and RF = 3.
>>
>>
>> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
>> Load distribution is very uneven. *What could be the reason for an
>> uneven Load ?*
>>
>>
>> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>>
>>
>>
>>
>> --
>> Raman Gugnani
>>
>
--
Raman Gugnani
Re: unbalanced data in second datacenter
Posted by manish khandelwal <ma...@gmail.com>.
I see you have two racks with RF=3. For better distribution, you should
have three racks . When using racks one should always have a number of
racks equal to or more than RF for equal distribution.
Slight differences in load distribution (with rack/ without rack) is
expected as it is a random distribution of data based on vnodes.
Regards
Manish
On Wed, Jan 10, 2024 at 11:51 AM raman gugnani <ra...@gmail.com>
wrote:
> We had one datacenter *ap-south-1* consisting of 5 nodes m5.4xlarge
> instances with average size of 2.2~2.3 Tib and version 3.11.2 and RF = 3.
> We have now added a new data centre *mumbai* consisting of 10 nodes
> m5.4xlarge instances and version 3.11.4 and RF = 3.
>
>
> Ownership in ap-south-1 is 60% per node and in mumbai 30% per node, but
> Load distribution is very uneven. *What could be the reason for an uneven
> Load ?*
>
>
> [image: Screenshot 2024-01-10 at 11.24.53 AM.png]
>
>
>
>
> --
> Raman Gugnani
>