You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2014/08/29 19:23:54 UTC
[jira] [Comment Edited] (CASSANDRA-7239) Nodetool Status Reports
Negative Load With VNodes Disabled
[ https://issues.apache.org/jira/browse/CASSANDRA-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115520#comment-14115520 ]
Brandon Williams edited comment on CASSANDRA-7239 at 8/29/14 5:23 PM:
----------------------------------------------------------------------
It appears the root of the problem here is that we have a serious bookkeeping problem. We only increment totalDiskSpaceUsed in DataTracker.replaceFlushed, which is to say that we only count flushed memtables. This means if you do a compaction, even if the result isn't negative, your reported load is completely wrong:
{noformat}
root@bw-2:/srv/cassandra# du -sh /var/lib/cassandra/data/Keyspace1/
163M /var/lib/cassandra/data/Keyspace1/
root@bw-2:/srv/cassandra# bin/nodetool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.208.35.225 33.16 MB 256 100.0% ac9b6dd7-233e-4cd5-a7e0-bc9564871704 rack1
{noformat}
(33MB because that's the size this machine likes to flush at, and there was one flush after the compaction.) It looks like we need to increment somewhere in compaction as well, probably when CompactionTask completes. This area isn't my forte, though.
was (Author: brandon.williams):
It appears the root of the problem here is that we have a serious bookkeeping problem. We only increment totalDiskSpaceUsed in DataTracker.replaceFlushed, which is to say that we only count flushed memtables. This means if you do a compaction, even if the result isn't negative, your reported load is completely wrong:
{noformat}
root@bw-2:/srv/cassandra# du -sh /var/lib/cassandra/data/Keyspace1/
163M /var/lib/cassandra/data/Keyspace1/
root@bw-2:/srv/cassandra# bin/nodetool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 10.208.35.225 33.16 MB 256 100.0% ac9b6dd7-233e-4cd5-a7e0-bc9564871704 rack1
{noformat}
(33MB because that's the size this machine likes to flush at, and there was one flush after the compaction.) It looks like we need to increment somewhere in compaction as well, probably CompactionTask completes. This area isn't my forte, though.
> Nodetool Status Reports Negative Load With VNodes Disabled
> ----------------------------------------------------------
>
> Key: CASSANDRA-7239
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7239
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: 1000 Nodes EC2 m1.large ubuntu 12.04
> Reporter: Russell Alexander Spitzer
> Priority: Critical
> Fix For: 2.1.0
>
> Attachments: nodetool.png, opscenter.png
>
>
> When I run stress on a large cluster without vnodes (num_token =1 initial token set) The loads reported by nodetool status are negative, or become negative after stress is run.
> {code}
> UN 10.97.155.31 -447426217 bytes 1 0.2% 8d40568c-044c-4753-be26-4ab62710beba rack1
> UN 10.9.132.53 -447342449 bytes 1 0.2% 58e7f255-803d-493b-a19e-58137466fb78 rack1
> UN 10.37.151.202 -447298672 bytes 1 0.2% ba29b1f1-186f-45d0-9e59-6a528db8df5d rack1
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)