You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Benjamin Coverston (Created) (JIRA)" <ji...@apache.org> on 2011/11/15 21:18:51 UTC
[jira] [Created] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Load from `nodetool ring` does not update after cleanup.
--------------------------------------------------------
Key: CASSANDRA-3496
URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.2
Reporter: Benjamin Coverston
Assignee: Sylvain Lebresne
Repro:
Bring up a node.
Insert 1M rows:
127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
(Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
Bootstrap a second node into the cluster:
162877269496252595336256012556853953561
127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
Cleanup
162877269496252595336256012556853953561
127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3496.
---------------------------------------
Resolution: Fixed
Assignee: Jackson Chung (was: Jonathan Ellis)
Committed.
(Tried to catch the bug w/ a check on live size in RecoveryManagerTruncateTest, but I couldn't reproduce the race without adding sleeps inside the deletion task. So I just committed the patch as above.)
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Benjamin Coverston
> Assignee: Jackson Chung
> Fix For: 1.0.4
>
> Attachments: 3496.txt
>
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Jackson Chung (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150750#comment-13150750 ]
Jackson Chung commented on CASSANDRA-3496:
------------------------------------------
my test case (which seems difference from Ben's?):
1) put a break between the releaseReference and releaseReference
2) insert to a CF, flush, insert again, flush, now you have 2 sstables
3) force major compaction
with the following break/sleep in the test:
{code}
sstable.releaseReference();
try
{
logger.debug("xxxxxxxxxxxx zz for 10s");
Thread.sleep(7000);
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
long bytesOnDisk = sstable.bytesOnDisk();
logger.debug("xxxxxxxxxx wake up -------- size to remove from livesize" + bytesOnDisk);
liveSize.addAndGet(-bytesOnDisk);
{code}
the following is observed in the log:
{noformat}
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:43,050 DataTracker.java (line 345) removing /var/lib/cassandra/data/testsize/testcfsize-hb-1 from list of files tracked for testsize.testcfsize
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:43,050 SSTableReader.java (line 742) Marking /var/lib/cassandra/data/testsize/testcfsize-hb-1-Data.db compacted
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:43,050 MmappedSegmentedFile.java (line 139) All segments have been unmapped successfully
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:43,051 MmappedSegmentedFile.java (line 139) All segments have been unmapped successfully
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:43,051 DataTracker.java (line 353) xxxxxxxxxxxx zz for 10s
DEBUG [NonPeriodicTasks:1] 2011-11-15 12:29:43,051 FileUtils.java (line 51) Deleting testcfsize-hb-1-Index.db
DEBUG [NonPeriodicTasks:1] 2011-11-15 12:29:43,051 FileUtils.java (line 51) Deleting testcfsize-hb-1-Filter.db
DEBUG [NonPeriodicTasks:1] 2011-11-15 12:29:43,052 FileUtils.java (line 51) Deleting testcfsize-hb-1-Digest.sha1
DEBUG [NonPeriodicTasks:1] 2011-11-15 12:29:43,052 FileUtils.java (line 51) Deleting testcfsize-hb-1-Statistics.db
DEBUG [NonPeriodicTasks:1] 2011-11-15 12:29:43,052 SSTable.java (line 144) Deleted /var/lib/cassandra/data/testsize/testcfsize-hb-1
DEBUG [CompactionExecutor:6] 2011-11-15 12:29:50,051 DataTracker.java (line 363) xxxxxxxxxx wake up -------- size to remove from livesize0
{noformat}
cfstats:
Column Family: testcfsize
SSTable count: 1
Space used (live): 14395
Space used (total): 5491
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Benjamin Coverston
> Assignee: Sylvain Lebresne
> Attachments: 3496.txt
>
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3496:
--------------------------------------
Attachment: 3496.txt
patch to reduce size before releasing reference
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Benjamin Coverston
> Assignee: Sylvain Lebresne
> Attachments: 3496.txt
>
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150744#comment-13150744 ]
Jonathan Ellis commented on CASSANDRA-3496:
-------------------------------------------
Jackson pointed out that doing it in the current order, the data will be removed by the time we call length() on it:
{code}
. sstable.markCompacted();
sstable.releaseReference();
liveSize.addAndGet(-sstable.bytesOnDisk());
{code}
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Benjamin Coverston
> Assignee: Sylvain Lebresne
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3496:
----------------------------------------
Reviewer: slebresne
Affects Version/s: (was: 1.0.2)
1.0.0
Fix Version/s: 1.0.4
Assignee: Jonathan Ellis (was: Sylvain Lebresne)
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Benjamin Coverston
> Assignee: Jonathan Ellis
> Fix For: 1.0.4
>
> Attachments: 3496.txt
>
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3496) Load from `nodetool ring` does
not update after cleanup.
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151127#comment-13151127 ]
Sylvain Lebresne commented on CASSANDRA-3496:
---------------------------------------------
patch lgtm, +1
> Load from `nodetool ring` does not update after cleanup.
> --------------------------------------------------------
>
> Key: CASSANDRA-3496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3496
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Benjamin Coverston
> Assignee: Jonathan Ellis
> Fix For: 1.0.4
>
> Attachments: 3496.txt
>
>
> Repro:
> Bring up a node.
> Insert 1M rows:
> 127.0.0.1 datacenter1 rack1 Up Normal 406.92 MB 100.00% 77747037169725419723056812679314618801
> (Already looks wrong, 406.92 is higher than I'm used to seeing from a single run of stress)
> Bootstrap a second node into the cluster:
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 407.03 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Cleanup
> 162877269496252595336256012556853953561
> 127.0.0.1 datacenter1 rack1 Up Normal 551.2 MB 49.96% 77747037169725419723056812679314618801
> 127.0.0.2 datacenter1 rack1 Up Normal 157.91 MB 50.04% 162877269496252595336256012556853953561
> Looks like each operation that adds and removes SSTables only adds to the total and doesn't remove the old sstables from the total size count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira