You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Daniel Klopp (JIRA)" <ji...@apache.org> on 2016/08/26 16:45:21 UTC
[jira] [Updated] (CASSANDRA-12557) Cassandra 3.0.6 New Node
Perpetually in UJ State and Streams More Data Than Any Node
[ https://issues.apache.org/jira/browse/CASSANDRA-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Klopp updated CASSANDRA-12557:
-------------------------------------
Description:
Hello,
We are using Cassandra 3.0.6, we've added a fifth Cassandra node to our four node cluster. Earlier on the streams kept failing. I tweaked some cassandra.yaml settings and got them to not fail. However, we have noticed strange behavior in the sync. Please see the output of nodetool:
ubuntu@ip-172-28-4-238:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UJ 172.28.4.238 1.48 TB 256 ? a797ed18-1d50-4b19-924a-f6b37b8859af rack1
UN 172.28.4.79 988.83 GB 256 ? 9eec70ec-5d7a-4ba8-bba8-f7d229d00358 rack1
UN 172.28.4.69 891.9 GB 256 ? 1d429d87-ec4a-4e14-92d7-df2aa129041e rack1
UN 172.28.4.129 985.48 GB 256 ? 677c7585-ed31-4afc-b17c-288a3a1e3666 rack1
UN 172.28.4.146 760.38 GB 256 ? 13ab7037-ec9b-4031-8d6c-4db95b91fa21 rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
ubuntu@ip-172-28-4-238:~$
The fifth node is 172.28.4.238. Why is its load 1.48 TB, when all of the original four nodes are less than 1 TB? I can also see this on disk usage. The original four nodes are utilizing 900 GB to 1100 GB on data volume. The fifth node, however, has ballooned to 2380 GB. I had to stop the sync and add a second disk to support it.
I've attached our cassandra.yaml file. What could be causing this?
was:
Hello,
We are using Cassandra 3.0.6, we've added a fifth Cassandra node to our four node cluster. Earlier on the streams kept failing. I tweaked some cassandra.yaml settings and got them to not fail. However, we have noticed strange behavior in the sync. Please see the output of nodetool:
ubuntu@ip-172-28-4-238:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UJ 172.28.4.238 1.48 TB 256 ? a797ed18-1d50-4b19-924a-f6b37b8859af rack1
UN 172.28.4.79 988.83 GB 256 ? 9eec70ec-5d7a-4ba8-bba8-f7d229d00358 rack1
UN 172.28.4.69 891.9 GB 256 ? 1d429d87-ec4a-4e14-92d7-df2aa129041e rack1
UN 172.28.4.129 985.48 GB 256 ? 677c7585-ed31-4afc-b17c-288a3a1e3666 rack1
UN 172.28.4.146 760.38 GB 256 ? 13ab7037-ec9b-4031-8d6c-4db95b91fa21 rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
ubuntu@ip-172-28-4-238:~$
The fifth node is 172.28.4.238. Why is its load 1.48 TB, when all of the original four nodes are less than 1 TB? I can also see this on disk usage. The original four nodes are utilizing 900 GB to 1100 GB on data volume. The fifth node, however, has ballooned to 2380 GB. I had to stop the sync and add a second disk to support it.
I've attached our cassandra.yaml file and some of our log outputs. What could be causing this?
> Cassandra 3.0.6 New Node Perpetually in UJ State and Streams More Data Than Any Node
> ------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12557
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12557
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Ubuntu 14.04, AWS EC2, m4.2xlarge, 2TB dedicated data disks per node (except node 5, with 2x2TB dedicated data disks), Cassandra 3.0.6
> Reporter: Daniel Klopp
> Fix For: 3.x
>
> Attachments: cassandra.yaml
>
>
> Hello,
> We are using Cassandra 3.0.6, we've added a fifth Cassandra node to our four node cluster. Earlier on the streams kept failing. I tweaked some cassandra.yaml settings and got them to not fail. However, we have noticed strange behavior in the sync. Please see the output of nodetool:
> ubuntu@ip-172-28-4-238:~$ nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UJ 172.28.4.238 1.48 TB 256 ? a797ed18-1d50-4b19-924a-f6b37b8859af rack1
> UN 172.28.4.79 988.83 GB 256 ? 9eec70ec-5d7a-4ba8-bba8-f7d229d00358 rack1
> UN 172.28.4.69 891.9 GB 256 ? 1d429d87-ec4a-4e14-92d7-df2aa129041e rack1
> UN 172.28.4.129 985.48 GB 256 ? 677c7585-ed31-4afc-b17c-288a3a1e3666 rack1
> UN 172.28.4.146 760.38 GB 256 ? 13ab7037-ec9b-4031-8d6c-4db95b91fa21 rack1
> Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
> ubuntu@ip-172-28-4-238:~$
> The fifth node is 172.28.4.238. Why is its load 1.48 TB, when all of the original four nodes are less than 1 TB? I can also see this on disk usage. The original four nodes are utilizing 900 GB to 1100 GB on data volume. The fifth node, however, has ballooned to 2380 GB. I had to stop the sync and add a second disk to support it.
> I've attached our cassandra.yaml file. What could be causing this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)