You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Created) (JIRA)" <ji...@apache.org> on 2012/03/08 18:47:58 UTC
[jira] [Created] (CASSANDRA-4022) Compaction of hints can get stuck
in a loop
Compaction of hints can get stuck in a loop
-------------------------------------------
Key: CASSANDRA-4022
URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Assignee: Brandon Williams
Priority: Critical
Fix For: 1.1.0
Not exactly sure how I caused this as I was working on something else in trunk, but:
{noformat}
INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 5.912220MB/s. Time: 748ms.
INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 5.479976MB/s. Time: 807ms.
INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 4.786083MB/s. Time: 924ms.
INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 4.485132MB/s. Time: 986ms.
INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 4.223821MB/s. Time: 1,047ms.
INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 4.526449MB/s. Time: 977ms.
INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
for 1 keys at 6.263938MB/s. Time: 706ms.
INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
{noformat}
I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225342#comment-13225342 ]
Brandon Williams commented on CASSANDRA-4022:
---------------------------------------------
It seems part of the problem is it doesn't know about one of the sstables it needs to compact:
{noformat}
cassandra-1:/srv/cassandra# ls -l /var/lib/cassandra/data/system/HintsColumnFamily/
total 66804
-rw-r--r-- 1 root root 63642821 Mar 8 17:36 system-HintsColumnFamily-hd-13-Data.db
-rw-r--r-- 1 root root 80 Mar 8 17:36 system-HintsColumnFamily-hd-13-Digest.sha1
-rw-r--r-- 1 root root 976 Mar 8 17:36 system-HintsColumnFamily-hd-13-Filter.db
-rw-r--r-- 1 root root 26 Mar 8 17:36 system-HintsColumnFamily-hd-13-Index.db
-rw-r--r-- 1 root root 4344 Mar 8 17:36 system-HintsColumnFamily-hd-13-Statistics.db
-rw-r--r-- 1 root root 4637160 Mar 8 17:59 system-HintsColumnFamily-hd-639-Data.db
-rw-r--r-- 1 root root 81 Mar 8 17:59 system-HintsColumnFamily-hd-639-Digest.sha1
-rw-r--r-- 1 root root 496 Mar 8 17:59 system-HintsColumnFamily-hd-639-Filter.db
-rw-r--r-- 1 root root 26 Mar 8 17:59 system-HintsColumnFamily-hd-639-Index.db
-rw-r--r-- 1 root root 5944 Mar 8 17:59 system-HintsColumnFamily-hd-639-Statistics.db
-rw-r--r-- 1 root root 0 Mar 8 17:59 system-HintsColumnFamily-tmp-hd-640-Data.db
-rw-r--r-- 1 root root 0 Mar 8 17:59 system-HintsColumnFamily-tmp-hd-640-Index.db
{noformat}
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Critical
> Fix For: 1.1.0
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225680#comment-13225680 ]
Yuki Morishita commented on CASSANDRA-4022:
-------------------------------------------
The basic idea behind CASSANDRA-3442 is to perform single sstable compaction if its droppable tombstone ratio is above threshold.
This works because single sstable compaction drops tombstones and lowers droppable tombstone ratio, which prevents recursive compaction on that sstable.
But there is a situation which compaction does not drop tombstones. That happens when the key in compacting sstable appears in another sstables.
Let me find the way to handle above case...
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4022) Compaction of hints can get stuck
in a loop
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4022:
--------------------------------------
Attachment: 4022.txt
One possible solution is to add check for overwrap of key range stored in sstable.
Patch attached with minor fix for test.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225463#comment-13225463 ]
Brandon Williams commented on CASSANDRA-4022:
---------------------------------------------
What is happening very reproducibly now is that I started the node, and 5 minutes later the forced compaction check in ACS kicks off, and then I have looping compaction on the hints but it's only compacting the last sstable over and over.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Critical
> Fix For: 1.1.0
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4022) Compaction of hints can get stuck
in a loop
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4022:
--------------------------------------
Affects Version/s: 1.2
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227274#comment-13227274 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
I think we should check for overlaps *and* timestamp is old enough to have data we care about suppressing. The former alone will be common in size-tiered compaction.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4022) Compaction of hints can get stuck
in a loop
Posted by "Yuki Morishita (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4022:
--------------------------------------
Attachment: 4022-v2.txt
Dropping tombstone is only done when the key tombstone belongs to does not appear in other sstables that are not compacting(ConpactionController#shouldPurge).
We may be able to tweak the above to look through timestamp of columns, but it will cost too much.
Instead, I come up with "guessing" how many tombstones there are outside of overlapping keys among sstables. When sstable has droppable tombstone ratio > threshold but overlaps keys with others, then calculate:
{code}
(# of keys outside of overlap) remainingKeys = sstable.estimatedKeys - sstable.estimatedKeysForRanges(overlapped range)
(# of columns outside of overlap) remainingColumns = sstable.estimatedColumnCount.percentile(remaingingKeys / total keys) * remainingKeys
{code}
and if (remainingColumns / total columns) * (droppable tombstone ratio) is greater than threshold, compact that sstable itself.
I think in this way, the chance of single sstable compaction increases, while avoiding recursive sstable compaction.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022-v2.txt, 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230570#comment-13230570 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
Suppose for example we have two sstables:
SSTable A has a tombstone for row K, column foo, at time=100.
SSTable B has data for row K, column bar, at time=200.
We would like to allow A to be compacted by itself to get rid of tombstones, since even though B has overlapping data it is new enough that removing the TS is safe.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229548#comment-13229548 ]
Yuki Morishita commented on CASSANDRA-4022:
-------------------------------------------
I tried to use timestamp to determine whether sstable should be compacted, but it does not guarantee to suppress tombstones. Tombstones only get dropped when those keys don't appear in other sstables besides compacting ones. Currently I think the only way to stop compaction loop is to make sure interested sstable does not have overlap so its tombstones actually drop.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Yuki Morishita (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230238#comment-13230238 ]
Yuki Morishita commented on CASSANDRA-4022:
-------------------------------------------
I understand the situation, but isn't it covered by just checking key overlap?
If there is no overlap, then tombstones in target sstable are guaranteed to be the only and the newest ones?
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225350#comment-13225350 ]
Brandon Williams commented on CASSANDRA-4022:
---------------------------------------------
Yuki mentions that it may be caused by CASSANDRA-3442 too.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Critical
> Fix For: 1.1.0
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230284#comment-13230284 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
Right, if there's no overlap we're free to compact -- but I'm worried that with SizeTiered compaction we'll have overlap in a lot of cases where we could still compact if we looked closer.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225690#comment-13225690 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
bq. But there is a situation which compaction does not drop tombstones. That happens when the key in compacting sstable appears in another sstables.
What if we checked the most-recent-timestamp of the other sstables, and avoid compaction only if there is potentially data old enough that we'd need the tombstones to suppress it?
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229556#comment-13229556 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
We don't need to suppress tombstones -- we just need to make sure that any data that the tombstones we're compacting, is new enough that we don't need the tombstones to suppress them. In other words, that throwing away our tombstones won't make deleted data start showing up again.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4022) Compaction of hints can get stuck
in a loop
Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4022:
----------------------------------------
Fix Version/s: (was: 1.1.0)
1.2
Assignee: Yuki Morishita (was: Brandon Williams)
Confirmed that this only happens with CASSANDRA-3442 applied.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225352#comment-13225352 ]
Brandon Williams commented on CASSANDRA-4022:
---------------------------------------------
I should note that the machine does not hand anything off, so everything in these sstables must be tombstones.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Critical
> Fix For: 1.1.0
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236885#comment-13236885 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
v2 lgtm but I'm going to rebase on top of CASSANDRA-4080 before committing.
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022-v2.txt, 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4022) Compaction of hints can get
stuck in a loop
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236831#comment-13236831 ]
Jonathan Ellis commented on CASSANDRA-4022:
-------------------------------------------
bq. Dropping tombstone is only done when the key tombstone belongs to does not appear in other sstables that are not compacting(ConpactionController#shouldPurge).
Right. I'm saying we can make that more sophisticated, e.g. by changing shouldPurge signature to {{(key, maxTombstoneTimestamp)}} which we could then compare to the min timestamp from the overlapping-but-not-compaction-participant sstables.
But, thinking about that more, it's unlikely to help much since both STCS and LCS mix data of different ages together routinely. So I'll let that drop now. :)
> Compaction of hints can get stuck in a loop
> -------------------------------------------
>
> Key: CASSANDRA-4022
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4022
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2
> Reporter: Brandon Williams
> Assignee: Yuki Morishita
> Priority: Critical
> Fix For: 1.2
>
> Attachments: 4022-v2.txt, 4022.txt
>
>
> Not exactly sure how I caused this as I was working on something else in trunk, but:
> {noformat}
> INFO 17:41:35,682 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-339-Data.db')]
> INFO 17:41:36,430 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.912220MB/s. Time: 748ms.
> INFO 17:41:36,431 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-340-Data.db')]
> INFO 17:41:37,238 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 5.479976MB/s. Time: 807ms.
> INFO 17:41:37,239 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-341-Data.db')]
> INFO 17:41:38,163 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.786083MB/s. Time: 924ms.
> INFO 17:41:38,164 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-342-Data.db')]
> INFO 17:41:39,014 GC for ParNew: 274 ms for 1 collections, 541261288 used; max is 1024458752
> INFO 17:41:39,151 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.485132MB/s. Time: 986ms.
> INFO 17:41:39,151 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-343-Data.db')]
> INFO 17:41:40,016 GC for ParNew: 308 ms for 1 collections, 585582200 used; max is 1024458752
> INFO 17:41:40,200 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.223821MB/s. Time: 1,047ms.
> INFO 17:41:40,201 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-344-Data.db')]
> INFO 17:41:41,017 GC for ParNew: 252 ms for 1 collections, 617877904 used; max is 1024458752
> INFO 17:41:41,178 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 4.526449MB/s. Time: 977ms.
> INFO 17:41:41,179 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-345-Data.db')]
> INFO 17:41:41,885 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes
> for 1 keys at 6.263938MB/s. Time: 706ms.
> INFO 17:41:41,887 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-346-Data.db')]
> INFO 17:41:42,617 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 6.066311MB/s. Time: 729ms.
> INFO 17:41:42,618 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-347-Data.db')]
> INFO 17:41:43,376 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 5.834222MB/s. Time: 758ms.
> INFO 17:41:43,377 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-348-Data.db')]
> INFO 17:41:44,307 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.760323MB/s. Time: 929ms.
> INFO 17:41:44,308 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-349-Data.db')]
> INFO 17:41:45,021 GC for ParNew: 245 ms for 1 collections, 731287832 used; max is 1024458752
> INFO 17:41:45,316 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db,]. 4,637,160 to 4,637,160 (~100% of original) bytes for 1 keys at 4.395965MB/s. Time: 1,006ms.
> INFO 17:41:45,316 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-350-Data.db')]
> INFO 17:41:46,022 GC for ParNew: 353 ms for 1 collections, 757476872 used; max is 1024458752
> INFO 17:41:46,451 Compacted to [/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hd-351-Data.db,]. 4,637,160 to 4,637
> {noformat}
> I suspect we broke something subtle in CASSANDRA-3955.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira