You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2021/10/28 19:03:53 UTC
4.0.1 - adding a node
I recently added a node to a cluster. Immediately after adding the
node, the cluster status (nyx is the new node):
UJ nyx.querymasters.com 181.25 KiB 250 ?
07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
UN enceladus.querymasters.com 569.53 GiB 200 35.1%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN calypso.querymasters.com 578.79 GiB 200 34.8%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN fortuna.querymasters.com 593.79 GiB 200 34.6%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN charon.querymasters.com 603.3 GiB 200 35.0%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN eros.querymasters.com 589.04 GiB 200 34.2%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN hercules.querymasters.com 12.31 GiB 4 0.7%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN ursula.querymasters.com 611.65 GiB 200 35.0%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN gaia.querymasters.com 480.62 GiB 200 34.7%
b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
UN chaos.querymasters.com 358.07 GiB 120 20.5%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN pallas.querymasters.com 537.88 GiB 200 35.3%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
If I add up the Load column, I get 4,632.67GiB. After overnight:
[joeo@calypso ~]$ nodetool status -r
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective)
Host ID Rack
UN nyx.querymasters.com 535.16 GiB 250 38.0%
07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
UN enceladus.querymasters.com 568.87 GiB 200 30.4%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN calypso.querymasters.com 578.81 GiB 200 30.4%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN fortuna.querymasters.com 593.82 GiB 200 30.4%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN charon.querymasters.com 602.38 GiB 200 30.4%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN eros.querymasters.com 588.3 GiB 200 30.4%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN hercules.querymasters.com 12.31 GiB 4 0.6%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN ursula.querymasters.com 610.54 GiB 200 30.3%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN gaia.querymasters.com 480.53 GiB 200 30.5%
b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
UN chaos.querymasters.com 358.44 GiB 120 18.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN pallas.querymasters.com 537.94 GiB 200 30.4%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
If I add up the Load, I get 5,466.79GiB. I have added no new data to
the cluster, yet the Load has increased by 834. Is this expected behavior?
Thank you!
-Joe
Re: 4.0.1 - adding a node
Posted by Joe Obernberger <jo...@gmail.com>.
Hi Erick - yes I do. There is a good chance that I'll remove hercules
from the cluster (4 tokens), it was the first system that I put
cassandra on. Chaos has less drives than the other systems.
-joe
On 10/29/2021 7:57 PM, Erick Ramirez wrote:
> Out of curiosity, what's up with hercules and chaos? Do you have
> different hardware deployed in your cluster? Cheers!
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: 4.0.1 - adding a node
Posted by Erick Ramirez <er...@datastax.com>.
Out of curiosity, what's up with hercules and chaos? Do you have different
hardware deployed in your cluster? Cheers!
>
Re: 4.0.1 - adding a node
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Jeff - after cleanup:
UN nyx.querymasters.com 535.15 GiB 250 38.0%
07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
UN enceladus.querymasters.com 468.97 GiB 200 30.4%
660f476c-a124-4ca0-b55f-75efe56370da rack1
UN calypso.querymasters.com 470.21 GiB 200 30.4%
e83aa851-69b4-478f-88f6-60e657ea6539 rack1
UN fortuna.querymasters.com 593.82 GiB 200 30.4%
49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
UN charon.querymasters.com 475.65 GiB 200 30.4%
d9702f96-256e-45ae-8e12-69a42712be50 rack1
UN eros.querymasters.com 476.46 GiB 200 30.4%
93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
UN hercules.querymasters.com 12.31 GiB 4 0.6%
a1a16910-9167-4174-b34b-eb859d36347e rack1
UN ursula.querymasters.com 481.43 GiB 200 30.3%
4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
UN gaia.querymasters.com 436.18 GiB 200 30.5%
b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
UN chaos.querymasters.com 289.83 GiB 120 18.2%
08a19658-40be-4e55-8709-812b3d4ac750 rack1
UN pallas.querymasters.com 447.83 GiB 200 30.4%
b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
-Joe
On 10/28/2021 4:05 PM, Jeff Jirsa wrote:
> I think you started at 4930 and ended at 5461, difference of 530
> (which is the new host)
>
> If you run `nodetool cleanup` on every other node in the cluster, you
> likely drop back down close to 4931 again.
>
>
>
> On Thu, Oct 28, 2021 at 12:04 PM Joe Obernberger
> <jo...@gmail.com> wrote:
>
> I recently added a node to a cluster. Immediately after adding the
> node, the cluster status (nyx is the new node):
>
> UJ nyx.querymasters.com <http://nyx.querymasters.com> 181.25 KiB
> 250 ?
> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
> UN enceladus.querymasters.com <http://enceladus.querymasters.com>
> 569.53 GiB 200 35.1%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN calypso.querymasters.com <http://calypso.querymasters.com>
> 578.79 GiB 200 34.8%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN fortuna.querymasters.com <http://fortuna.querymasters.com>
> 593.79 GiB 200 34.6%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN charon.querymasters.com <http://charon.querymasters.com> 603.3
> GiB 200 35.0%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN eros.querymasters.com <http://eros.querymasters.com> 589.04
> GiB 200 34.2%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN hercules.querymasters.com <http://hercules.querymasters.com>
> 12.31 GiB 4 0.7%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN ursula.querymasters.com <http://ursula.querymasters.com> 611.65
> GiB 200 35.0%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN gaia.querymasters.com <http://gaia.querymasters.com> 480.62
> GiB 200 34.7%
> b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
> UN chaos.querymasters.com <http://chaos.querymasters.com> 358.07
> GiB 120 20.5%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN pallas.querymasters.com <http://pallas.querymasters.com> 537.88
> GiB 200 35.3%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> If I add up the Load column, I get 4,632.67GiB. After overnight:
>
> [joeo@calypso ~]$ nodetool status -r
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective)
> Host ID Rack
> UN nyx.querymasters.com <http://nyx.querymasters.com> 535.16 GiB
> 250 38.0%
> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
> UN enceladus.querymasters.com <http://enceladus.querymasters.com>
> 568.87 GiB 200 30.4%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN calypso.querymasters.com <http://calypso.querymasters.com>
> 578.81 GiB 200 30.4%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN fortuna.querymasters.com <http://fortuna.querymasters.com>
> 593.82 GiB 200 30.4%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN charon.querymasters.com <http://charon.querymasters.com> 602.38
> GiB 200 30.4%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN eros.querymasters.com <http://eros.querymasters.com> 588.3
> GiB 200 30.4%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN hercules.querymasters.com <http://hercules.querymasters.com>
> 12.31 GiB 4 0.6%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN ursula.querymasters.com <http://ursula.querymasters.com> 610.54
> GiB 200 30.3%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN gaia.querymasters.com <http://gaia.querymasters.com> 480.53
> GiB 200 30.5%
> b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
> UN chaos.querymasters.com <http://chaos.querymasters.com> 358.44
> GiB 120 18.2%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN pallas.querymasters.com <http://pallas.querymasters.com> 537.94
> GiB 200 30.4%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> If I add up the Load, I get 5,466.79GiB. I have added no new data to
> the cluster, yet the Load has increased by 834. Is this expected
> behavior?
> Thank you!
>
> -Joe
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: 4.0.1 - adding a node
Posted by Jeff Jirsa <jj...@gmail.com>.
I think you started at 4930 and ended at 5461, difference of 530 (which is
the new host)
If you run `nodetool cleanup` on every other node in the cluster, you
likely drop back down close to 4931 again.
On Thu, Oct 28, 2021 at 12:04 PM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> I recently added a node to a cluster. Immediately after adding the
> node, the cluster status (nyx is the new node):
>
> UJ nyx.querymasters.com 181.25 KiB 250 ?
> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
> UN enceladus.querymasters.com 569.53 GiB 200 35.1%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN calypso.querymasters.com 578.79 GiB 200 34.8%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN fortuna.querymasters.com 593.79 GiB 200 34.6%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN charon.querymasters.com 603.3 GiB 200 35.0%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN eros.querymasters.com 589.04 GiB 200 34.2%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN hercules.querymasters.com 12.31 GiB 4 0.7%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN ursula.querymasters.com 611.65 GiB 200 35.0%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN gaia.querymasters.com 480.62 GiB 200 34.7%
> b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
> UN chaos.querymasters.com 358.07 GiB 120 20.5%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN pallas.querymasters.com 537.88 GiB 200 35.3%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> If I add up the Load column, I get 4,632.67GiB. After overnight:
>
> [joeo@calypso ~]$ nodetool status -r
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective)
> Host ID Rack
> UN nyx.querymasters.com 535.16 GiB 250 38.0%
> 07bccfce-45f1-41a3-a5c4-ee748a7a9b98 rack1
> UN enceladus.querymasters.com 568.87 GiB 200 30.4%
> 660f476c-a124-4ca0-b55f-75efe56370da rack1
> UN calypso.querymasters.com 578.81 GiB 200 30.4%
> e83aa851-69b4-478f-88f6-60e657ea6539 rack1
> UN fortuna.querymasters.com 593.82 GiB 200 30.4%
> 49e4f571-7d1c-4e1e-aca7-5bbe076596f7 rack1
> UN charon.querymasters.com 602.38 GiB 200 30.4%
> d9702f96-256e-45ae-8e12-69a42712be50 rack1
> UN eros.querymasters.com 588.3 GiB 200 30.4%
> 93f9cb0f-ea71-4e3d-b62a-f0ea0e888c47 rack1
> UN hercules.querymasters.com 12.31 GiB 4 0.6%
> a1a16910-9167-4174-b34b-eb859d36347e rack1
> UN ursula.querymasters.com 610.54 GiB 200 30.3%
> 4bbbe57c-6219-41e5-bbac-de92a9594d53 rack1
> UN gaia.querymasters.com 480.53 GiB 200 30.5%
> b2e5366e-8386-40ec-a641-27944a5a7cfa rack1
> UN chaos.querymasters.com 358.44 GiB 120 18.2%
> 08a19658-40be-4e55-8709-812b3d4ac750 rack1
> UN pallas.querymasters.com 537.94 GiB 200 30.4%
> b74b6e65-af63-486a-b07f-9e304ec30a39 rack1
>
> If I add up the Load, I get 5,466.79GiB. I have added no new data to
> the cluster, yet the Load has increased by 834. Is this expected behavior?
> Thank you!
>
> -Joe
>
>