You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/09/29 06:01:15 UTC
[jira] Created: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Make FlushPeriodInMinutes default to 60
---------------------------------------
Key: CASSANDRA-463
URL: https://issues.apache.org/jira/browse/CASSANDRA-463
Project: Cassandra
Issue Type: Task
Components: Core
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 0.5
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761626#action_12761626 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
... if there is a heisenbug there, it's not related to this patch, since neither the old nor new timed-flush code would kick in during that test.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761637#action_12761637 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
> why after recovery, no key is supposed to be found?
because the flush marks the commitlog as "i've turned everything into sstables, so you can skip replay up to this point" and then clearUnsafe wipes out the sstables.
like the comment says, i'm not sure this is a very useful test... but other things being equal i'd lean towards keeping it for now :)
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt, RM2T.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760629#action_12760629 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
Thinking about it more: instead of having a FlushPeriodInMinutes per-CF that could possibly force extra flushes if the CF is busy (filling up and flushing from normal updates), I suggest adding a global MemtablesFlushAfter setting (along with SizeInMB and ObjectCountInMillions) which flushes any memtable when it has been dirty for longer than the given number of minutes. This way memtables that are constantly being flushed from high activity don't get "extra" flushes from a misconfigured FlushPeriodInMinutes, and you'll never have commitlog segments pile up because a CF you thought would always be busy, didn't fill up for a few days at one point.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) add MemtableFlushAfterMinutes, a
global replacement for FlushPeriodInMinutes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761866#action_12761866 ]
Hudson commented on CASSANDRA-463:
----------------------------------
Integrated in Cassandra #216 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/216/])
fix heisenbug in RecoveryManager2Test where a compaction could add back a new sstable to the "cleared" ColumnFamilyStore
patch by jbellis; reviewed by junrao for
add MemtableFlushAfterMinutes, a global replacement for the old per-CF FlushPeriodInMinutes setting
patch by jbellis; reviewed by junrao for
change instance variables to final where appropriate
patch by jbellis; reviewed by junrao for
> add MemtableFlushAfterMinutes, a global replacement for FlushPeriodInMinutes
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt, RM2T.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-463) add MemtableFlushAfterMinutes, a
global replacement for FlushPeriodInMinutes
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-463:
-------------------------------------
Summary: add MemtableFlushAfterMinutes, a global replacement for FlushPeriodInMinutes (was: Make FlushPeriodInMinutes default to 60)
> add MemtableFlushAfterMinutes, a global replacement for FlushPeriodInMinutes
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt, RM2T.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761650#action_12761650 ]
Jun Rao commented on CASSANDRA-463:
-----------------------------------
With the new patch, RM2T test always passes. +1.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt, RM2T.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761611#action_12761611 ]
Jun Rao commented on CASSANDRA-463:
-----------------------------------
The following unit test fails with the patch.
[junit] Testcase: testWithFlush(org.apache.cassandra.db.RecoveryManager2Test): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.cassandra.db.RecoveryManager2Test.testWithFlush(RecoveryManager2Test.java:56)
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-463:
-------------------------------------
Attachment: RM2T.patch
fix heisenbug in RecoveryManager2Test where a compaction could add back a new sstable to the "cleared" ColumnFamilyStore
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt, RM2T.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761633#action_12761633 ]
Jun Rao commented on CASSANDRA-463:
-----------------------------------
RM2T fails before the patch too and the patch looks good to me. The failure doesn't happen on every run though. BTW, in this test, why after recovery, no key is supposed to be found?
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761363#action_12761363 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
After more thought I decided we don't want to reset the timer (as it were) after each update: this allows a CF that grows large very slowly, or reaches a steady state (the same keys are modified over and over) to block commitlog purge indefinitely. Instead, set a large enough lifetime that most CFs will flush out normally during that time and you only have to worry about catching the stragglers.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761364#action_12761364 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
01 just makes some fields "final." 02 makes the change.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-463:
-------------------------------------
Attachment: 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
0001-CASSANDRA-463-cleanup.txt
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-463) Make FlushPeriodInMinutes default
to 60
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761624#action_12761624 ]
Jonathan Ellis commented on CASSANDRA-463:
------------------------------------------
Hmm, is that reproducible for you? I ran RM2T 10+ times w/ no failures.
> Make FlushPeriodInMinutes default to 60
> ---------------------------------------
>
> Key: CASSANDRA-463
> URL: https://issues.apache.org/jira/browse/CASSANDRA-463
> Project: Cassandra
> Issue Type: Task
> Components: Core
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-463-cleanup.txt, 0002-add-MemtableFlushAfterMinutes-a-global-replacement-fo.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.