You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/09/26 20:49:16 UTC
[jira] Created: (CASSANDRA-459) Commitlog segments don't get
deleted
Commitlog segments don't get deleted
------------------------------------
Key: CASSANDRA-459
URL: https://issues.apache.org/jira/browse/CASSANDRA-459
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 0.4, 0.5
Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Teodor Sigaev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760045#action_12760045 ]
Teodor Sigaev commented on CASSANDRA-459:
-----------------------------------------
Yes, sure.
Another run (r819316 + 0001 patch), just for rechek:
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1 /spool/cassandra/data/system
/spool/cassandra/commitlog:
total 44144
-rw-r--r-- 1 teodor wheel 45154304 Sep 27 18:32 CommitLog-1254061669607.log
/spool/cassandra/data/Keyspace1:
total 35130
-rw-r--r-- 1 teodor wheel 10475393 Sep 27 18:29 Standard1-1-Data.db
-rw-r--r-- 1 teodor wheel 30365 Sep 27 18:29 Standard1-1-Filter.db
-rw-r--r-- 1 teodor wheel 1443066 Sep 27 18:29 Standard1-1-Index.db
-rw-r--r-- 1 teodor wheel 10472280 Sep 27 18:30 Standard1-2-Data.db
-rw-r--r-- 1 teodor wheel 30189 Sep 27 18:30 Standard1-2-Filter.db
-rw-r--r-- 1 teodor wheel 1434966 Sep 27 18:30 Standard1-2-Index.db
-rw-r--r-- 1 teodor wheel 10455997 Sep 27 18:32 Standard1-3-Data.db
-rw-r--r-- 1 teodor wheel 30293 Sep 27 18:32 Standard1-3-Filter.db
-rw-r--r-- 1 teodor wheel 1439401 Sep 27 18:32 Standard1-3-Index.db
/spool/cassandra/data/system:
total 6
-rw-r--r-- 1 teodor wheel 255 Sep 27 18:32 LocationInfo-1-Data.db
-rw-r--r-- 1 teodor wheel 85 Sep 27 18:32 LocationInfo-1-Filter.db
-rw-r--r-- 1 teodor wheel 50 Sep 27 18:32 LocationInfo-1-Index.db
After hour and a half:
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1 /spool/cassandra/data/system
/spool/cassandra/commitlog:
total 813840
-rw-r--r-- 1 teodor wheel 134217982 Sep 27 19:57 CommitLog-1254061669607.log
-rw-r--r-- 1 teodor wheel 134217875 Sep 27 19:57 CommitLog-1254062560798.log
-rw-r--r-- 1 teodor wheel 134217761 Sep 27 19:57 CommitLog-1254063429524.log
-rw-r--r-- 1 teodor wheel 134217809 Sep 27 19:57 CommitLog-1254064318004.log
-rw-r--r-- 1 teodor wheel 134217785 Sep 27 19:57 CommitLog-1254065174349.log
-rw-r--r-- 1 teodor wheel 134217849 Sep 27 19:57 CommitLog-1254066038411.log
-rw-r--r-- 1 teodor wheel 27459584 Sep 27 19:58 CommitLog-1254066927001.log
/spool/cassandra/data/Keyspace1:
total 612880
-rw-r--r-- 1 teodor wheel 203500680 Sep 27 19:07 Standard1-38-Data.db
-rw-r--r-- 1 teodor wheel 806965 Sep 27 19:07 Standard1-38-Filter.db
-rw-r--r-- 1 teodor wheel 32636516 Sep 27 19:07 Standard1-38-Index.db
-rw-r--r-- 1 teodor wheel 202494151 Sep 27 19:42 Standard1-75-Data.db
-rw-r--r-- 1 teodor wheel 801685 Sep 27 19:42 Standard1-75-Filter.db
-rw-r--r-- 1 teodor wheel 32441650 Sep 27 19:42 Standard1-75-Index.db
-rw-r--r-- 1 teodor wheel 67331570 Sep 27 19:49 Standard1-83-Data.db
-rw-r--r-- 1 teodor wheel 209845 Sep 27 19:49 Standard1-83-Filter.db
-rw-r--r-- 1 teodor wheel 9598920 Sep 27 19:49 Standard1-83-Index.db
-rw-r--r-- 1 teodor wheel 67375520 Sep 27 19:57 Standard1-92-Data.db
-rw-r--r-- 1 teodor wheel 209605 Sep 27 19:57 Standard1-92-Filter.db
-rw-r--r-- 1 teodor wheel 9579322 Sep 27 19:57 Standard1-92-Index.db
/spool/cassandra/data/system:
total 6
-rw-r--r-- 1 teodor wheel 255 Sep 27 18:32 LocationInfo-1-Data.db
-rw-r--r-- 1 teodor wheel 85 Sep 27 18:32 LocationInfo-1-Filter.db
-rw-r--r-- 1 teodor wheel 50 Sep 27 18:32 LocationInfo-1-Index.db
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-459:
-------------------------------------
Attachment: 0002-logging.txt
0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-459:
-------------------------------------
Attachment: (was: 0002-logging.txt)
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759963#action_12759963 ]
Jonathan Ellis commented on CASSANDRA-459:
------------------------------------------
Yes, 02 doesn't need to apply to 0.4
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760371#action_12760371 ]
Jonathan Ellis commented on CASSANDRA-459:
------------------------------------------
New patches attached.
What was happening is that when a ColumnFamily got marked dirty in the commitlog, all subsequent commitlog segments would also have that CF marked dirty, even after it was flushed. When flush happened it could clear out old segments, but if the CF never had any more updates (like the LocationInfo CF in the system keyspace) then log segments would pile up.
This makes log segments only treat a CF as dirty if it actually has data written to it for that CF.
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760046#action_12760046 ]
Jonathan Ellis commented on CASSANDRA-459:
------------------------------------------
Could you try trunk + 0001 ?
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-459:
-------------------------------------
Attachment: 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt
0001-logging.txt
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760024#action_12760024 ]
Jonathan Ellis commented on CASSANDRA-459:
------------------------------------------
Are you on the latest 0.4 branch before applying the patch? It needs both r819004 and this one to work.
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760417#action_12760417 ]
Jun Rao commented on CASSANDRA-459:
-----------------------------------
Patch looks good to me. +1
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Teodor Sigaev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760065#action_12760065 ]
Teodor Sigaev commented on CASSANDRA-459:
-----------------------------------------
Sure, results:
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1 /spool/cassandra/data/system
/spool/cassandra/commitlog:
total 89344
-rw-r--r-- 1 teodor wheel 91422720 Sep 27 20:36 CommitLog-1254068803560.log
/spool/cassandra/data/Keyspace1:
total 78400
-rw-r--r-- 1 teodor wheel 70339417 Sep 27 20:36 Standard1-9-Data.db
-rw-r--r-- 1 teodor wheel 210085 Sep 27 20:36 Standard1-9-Filter.db
-rw-r--r-- 1 teodor wheel 9601776 Sep 27 20:36 Standard1-9-Index.db
/spool/cassandra/data/system:
total 6
-rw-r--r-- 1 teodor wheel 255 Sep 27 20:31 LocationInfo-1-Data.db
-rw-r--r-- 1 teodor wheel 85 Sep 27 20:31 LocationInfo-1-Filter.db
-rw-r--r-- 1 teodor wheel 50 Sep 27 20:31 LocationInfo-1-Index.db
After hour:
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1 /spool/cassandra/data/system
/spool/cassandra/commitlog:
total 564560
-rw-r--r-- 1 teodor wheel 134218167 Sep 27 21:27 CommitLog-1254068803560.log
-rw-r--r-- 1 teodor wheel 134218321 Sep 27 21:27 CommitLog-1254069672194.log
-rw-r--r-- 1 teodor wheel 134218547 Sep 27 21:27 CommitLog-1254070532837.log
-rw-r--r-- 1 teodor wheel 134217848 Sep 27 21:27 CommitLog-1254071417379.log
-rw-r--r-- 1 teodor wheel 40793246 Sep 27 21:28 CommitLog-1254072254525.log
/spool/cassandra/data/Keyspace1:
total 487238
-rw-r--r-- 1 teodor wheel 245938744 Sep 27 21:05 Standard1-37-Data.db
-rw-r--r-- 1 teodor wheel 807205 Sep 27 21:05 Standard1-37-Filter.db
-rw-r--r-- 1 teodor wheel 32638599 Sep 27 21:05 Standard1-37-Index.db
-rw-r--r-- 1 teodor wheel 70395458 Sep 27 21:13 Standard1-46-Data.db
-rw-r--r-- 1 teodor wheel 210325 Sep 27 21:13 Standard1-46-Filter.db
-rw-r--r-- 1 teodor wheel 9612077 Sep 27 21:13 Standard1-46-Index.db
-rw-r--r-- 1 teodor wheel 41011329 Sep 27 21:18 Standard1-51-Data.db
-rw-r--r-- 1 teodor wheel 122245 Sep 27 21:18 Standard1-51-Filter.db
-rw-r--r-- 1 teodor wheel 5619283 Sep 27 21:18 Standard1-51-Index.db
-rw-r--r-- 1 teodor wheel 70296583 Sep 27 21:27 Standard1-60-Data.db
-rw-r--r-- 1 teodor wheel 210085 Sep 27 21:27 Standard1-60-Filter.db
-rw-r--r-- 1 teodor wheel 9592964 Sep 27 21:27 Standard1-60-Index.db
-rw-r--r-- 1 teodor wheel 10444198 Sep 27 21:27 Standard1-61-Data.db
-rw-r--r-- 1 teodor wheel 30253 Sep 27 21:27 Standard1-61-Filter.db
-rw-r--r-- 1 teodor wheel 1437846 Sep 27 21:27 Standard1-61-Index.db
/spool/cassandra/data/system:
total 6
-rw-r--r-- 1 teodor wheel 255 Sep 27 20:31 LocationInfo-1-Data.db
-rw-r--r-- 1 teodor wheel 85 Sep 27 20:31 LocationInfo-1-Filter.db
-rw-r--r-- 1 teodor wheel 50 Sep 27 20:31 LocationInfo-1-Index.db
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Teodor Sigaev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759961#action_12759961 ]
Teodor Sigaev commented on CASSANDRA-459:
-----------------------------------------
I'm very sorry, but patch could not be applied cleanly for cassandra-0.4:
% cat /spool/home/teodor/tmp/0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt /spool/home/teodor/tmp/0002-logging.txt | patch -p1 -C
....
Patching file src/java/org/apache/cassandra/db/CommitLog.java using Plan A...
Hunk #1 succeeded at 97 (offset 1 line).
Hunk #2 succeeded at 463 (offset -1 lines).
Hunk #3 succeeded at 514 (offset 1 line).
Hunk #4 failed at 532.
1 out of 4 hunks failed--saving rejects to src/java/org/apache/cassandra/db/CommitLog.java.rej
....
% cat src/java/org/apache/cassandra/db/CommitLog.java.rej
***************
*** 516,521 ****
}
else
{
BufferedRandomAccessFile logWriter = CommitLog.createWriter(oldFile);
writeCommitLogHeader(logWriter, oldCommitLogHeader.toByteArray());
logWriter.close();
--- 532,539 ----
}
else
{
+ if (logger_.isDebugEnabled())
+ logger_.debug("Not safe to delete commit log " + oldFile + "; dirty is " + oldCommitLogHeader.dirtyString());
BufferedRandomAccessFile logWriter = CommitLog.createWriter(oldFile);
writeCommitLogHeader(logWriter, oldCommitLogHeader.toByteArray());
logWriter.close();
It seems to me that is not a stopper to test.
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759934#action_12759934 ]
Jonathan Ellis commented on CASSANDRA-459:
------------------------------------------
01 fixes the bug
02 adds a bunch of debug logging to commitlog purging so you can see which CFs are blocking the discard
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Teodor Sigaev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759956#action_12759956 ]
Teodor Sigaev commented on CASSANDRA-459:
-----------------------------------------
I started a test to check that for whole night.
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-459:
-------------------------------------
Attachment: (was: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt)
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Teodor Sigaev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759989#action_12759989 ]
Teodor Sigaev commented on CASSANDRA-459:
-----------------------------------------
Patch doesn't change anything :( I use the same test script as for CASSANDRA-458.
Immediately after test start:
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1
/spool/cassandra/commitlog:
total 784
-rw-r--r-- 1 teodor wheel 786432 Sep 27 02:42 CommitLog-1254004919295.log
/spool/cassandra/data/Keyspace1:
total 0
After night
% ls -l /spool/cassandra/commitlog /spool/cassandra/data/Keyspace1
/spool/cassandra/commitlog:
total 3856720
-rw-r--r-- 1 teodor wheel 134217971 Sep 27 09:52 CommitLog-1254004919295.log
-rw-r--r-- 1 teodor wheel 134217839 Sep 27 09:52 CommitLog-1254005826443.log
-rw-r--r-- 1 teodor wheel 134218376 Sep 27 09:52 CommitLog-1254006732843.log
-rw-r--r-- 1 teodor wheel 134217734 Sep 27 09:52 CommitLog-1254007676088.log
-rw-r--r-- 1 teodor wheel 134218324 Sep 27 09:52 CommitLog-1254008589671.log
-rw-r--r-- 1 teodor wheel 134218501 Sep 27 09:52 CommitLog-1254009457777.log
-rw-r--r-- 1 teodor wheel 134217841 Sep 27 09:52 CommitLog-1254010329142.log
-rw-r--r-- 1 teodor wheel 134218312 Sep 27 09:52 CommitLog-1254011183824.log
-rw-r--r-- 1 teodor wheel 134217795 Sep 27 09:52 CommitLog-1254012066380.log
-rw-r--r-- 1 teodor wheel 134218031 Sep 27 09:52 CommitLog-1254012920007.log
-rw-r--r-- 1 teodor wheel 134217764 Sep 27 09:52 CommitLog-1254013812856.log
-rw-r--r-- 1 teodor wheel 134217888 Sep 27 09:52 CommitLog-1254014728895.log
-rw-r--r-- 1 teodor wheel 134217842 Sep 27 09:52 CommitLog-1254015574082.log
-rw-r--r-- 1 teodor wheel 134217861 Sep 27 09:52 CommitLog-1254016456859.log
-rw-r--r-- 1 teodor wheel 134218257 Sep 27 09:52 CommitLog-1254017313996.log
-rw-r--r-- 1 teodor wheel 134217874 Sep 27 09:52 CommitLog-1254018190196.log
-rw-r--r-- 1 teodor wheel 134218228 Sep 27 09:52 CommitLog-1254019039153.log
-rw-r--r-- 1 teodor wheel 134217768 Sep 27 09:52 CommitLog-1254019901588.log
-rw-r--r-- 1 teodor wheel 134218072 Sep 27 09:52 CommitLog-1254020850481.log
-rw-r--r-- 1 teodor wheel 134218463 Sep 27 09:52 CommitLog-1254021705210.log
-rw-r--r-- 1 teodor wheel 134217787 Sep 27 09:52 CommitLog-1254022584617.log
-rw-r--r-- 1 teodor wheel 134217817 Sep 27 09:52 CommitLog-1254023445022.log
-rw-r--r-- 1 teodor wheel 134218548 Sep 27 09:52 CommitLog-1254024325493.log
-rw-r--r-- 1 teodor wheel 134217870 Sep 27 09:52 CommitLog-1254025178129.log
-rw-r--r-- 1 teodor wheel 134217763 Sep 27 09:52 CommitLog-1254026031303.log
-rw-r--r-- 1 teodor wheel 134217782 Sep 27 09:52 CommitLog-1254026990171.log
-rw-r--r-- 1 teodor wheel 134217781 Sep 27 09:52 CommitLog-1254027844609.log
-rw-r--r-- 1 teodor wheel 134217742 Sep 27 09:52 CommitLog-1254028740042.log
-rw-r--r-- 1 teodor wheel 134217810 Sep 27 09:52 CommitLog-1254029593664.log
-rw-r--r-- 1 teodor wheel 54067200 Sep 27 09:53 CommitLog-1254030447553.log
/spool/cassandra/data/Keyspace1:
total 785468
-rw-r--r-- 1 teodor wheel 215470950 Sep 27 08:48 Standard1-372-Data.db
-rw-r--r-- 1 teodor wheel 3833605 Sep 27 08:48 Standard1-372-Filter.db
-rw-r--r-- 1 teodor wheel 88353365 Sep 27 08:48 Standard1-372-Index.db
-rw-r--r-- 1 teodor wheel 204226037 Sep 27 09:18 Standard1-408-Data.db
-rw-r--r-- 1 teodor wheel 812005 Sep 27 09:18 Standard1-408-Filter.db
-rw-r--r-- 1 teodor wheel 32812386 Sep 27 09:18 Standard1-408-Index.db
-rw-r--r-- 1 teodor wheel 10467605 Sep 27 09:51 Standard1-444-Data.db
-rw-r--r-- 1 teodor wheel 30349 Sep 27 09:51 Standard1-444-Filter.db
-rw-r--r-- 1 teodor wheel 1442137 Sep 27 09:51 Standard1-444-Index.db
-rw-r--r-- 1 teodor wheel 201238111 Sep 27 09:53 Standard1-445-Data.db
-rw-r--r-- 1 teodor wheel 795205 Sep 27 09:53 Standard1-445-Filter.db
-rw-r--r-- 1 teodor wheel 32237630 Sep 27 09:53 Standard1-445-Index.db
-rw-r--r-- 1 teodor wheel 10458480 Sep 27 09:52 Standard1-446-Data.db
-rw-r--r-- 1 teodor wheel 30261 Sep 27 09:52 Standard1-446-Filter.db
-rw-r--r-- 1 teodor wheel 1438054 Sep 27 09:52 Standard1-446-Index.db
The single problem in logs is:
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 09:41:59,422 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 03:41:59,303 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 04:41:59,234 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 05:41:59,355 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 06:41:59,386 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 07:41:59,392 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 08:48:58,819 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
Log fragment of one of that warnings:
INFO [MINOR-COMPACTION-POOL:1] 2009-09-27 09:41:53,827 ColumnFamilyStore.java (line 1155) Compacted to /spool/cassandra/data/Keyspace1/Standard1-434-Data.db. 0/67462071 bytes for 111274/107527 keys read/written. Time: 31884ms.
INFO [PERIODIC-FLUSHER-POOL:1] 2009-09-27 09:41:59,033 ColumnFamilyStore.java (line 367) Standard1 has reached its threshold; switching in a fresh Memtable
INFO [PERIODIC-FLUSHER-POOL:1] 2009-09-27 09:41:59,033 ColumnFamilyStore.java (line 1178) Enqueuing flush of Memtable(Standard1)@8993320
INFO [MEMTABLE-FLUSHER-POOL:1] 2009-09-27 09:41:59,033 Memtable.java (line 186) Flushing Memtable(Standard1)@8993320
INFO [MINOR-COMPACTION-POOL:1] 2009-09-27 09:41:59,422 ColumnFamilyStore.java (line 1013) Compacting []
WARN [MINOR-COMPACTION-POOL:1] 2009-09-27 09:41:59,422 ColumnFamilyStore.java (line 1034) Nothing to compact (all files empty or corrupt)
INFO [MEMTABLE-FLUSHER-POOL:1] 2009-09-27 09:42:06,148 Memtable.java (line 220) Completed flushing Memtable(Standard1)@8993320
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-CASSANDRA-459-flush-system-table-every-five-minutes.txt, 0002-logging.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Resolved: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by Teodor Sigaev <te...@sigaev.ru>.
Thank you a lot!
Jonathan Ellis (JIRA) wrote:
> Resolution: Fixed
>
> committed w/ one more bug fix (r819823) to 0.4 and trunk
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
[jira] Resolved: (CASSANDRA-459) Commitlog segments don't get
deleted
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-459.
--------------------------------------
Resolution: Fixed
committed w/ one more bug fix (r819823) to 0.4 and trunk
> Commitlog segments don't get deleted
> ------------------------------------
>
> Key: CASSANDRA-459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-459
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 0001-logging.txt, 0002-CASSANDRA-459-flush-locationinfo-every-minute-and-hint.txt, 0003-cleanup-don-t-preserve-dirty-bits-from-older-replay-s.txt
>
>
> System table is not created with a periodic flush, so any update there (such as storing token info) can prevent commitlog segments from being deleted.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.