You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Phillips (JIRA)" <ji...@apache.org> on 2011/07/27 19:28:09 UTC
[jira] [Created] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Flush memtables on shutdown when durable writes are disabled
------------------------------------------------------------
Key: CASSANDRA-2958
URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
Project: Cassandra
Issue Type: Bug
Reporter: David Phillips
Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2958.
---------------------------------------
Resolution: Fixed
thanks -- fixed in r1152891
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "David Phillips (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Phillips reopened CASSANDRA-2958:
---------------------------------------
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Coli updated CASSANDRA-2958:
-----------------------------------
Description: Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log. (was: )
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Coli updated CASSANDRA-2958:
-----------------------------------
Description: (was: Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
)
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073671#comment-13073671 ]
Hudson commented on CASSANDRA-2958:
-----------------------------------
Integrated in Cassandra-0.8 #250 (See [https://builds.apache.org/job/Cassandra-0.8/250/])
avoid NPE when flushing in shutdown hook
patch by jbellis for CASSANDRA-2958
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152891
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073118#comment-13073118 ]
Hudson commented on CASSANDRA-2958:
-----------------------------------
Integrated in Cassandra-0.8 #245 (See [https://builds.apache.org/job/Cassandra-0.8/245/])
Flush memtables on shutdown when durable writes are disabled
patch by jbellis; reviewed by tjake for CASSANDRA-2958
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152419
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/LazilyCompactedRow.java
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/StorageService.java
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2958:
--------------------------------------
Attachment: 2958.txt
You're right. Here's a patch to add flushing of non-durable CFs to the shutdown hook.
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Reporter: David Phillips
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071987#comment-13071987 ]
T Jake Luciani commented on CASSANDRA-2958:
-------------------------------------------
+1
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447980#comment-13447980 ]
Robert Coli commented on CASSANDRA-2958:
----------------------------------------
> "a memtable holds changes until it hits a flush condition like being full or graceful shutdown, or you explicitly flush it."
This is not an unambiguous summary of the current behavior. It is true only if durable_writes are off. If durable_writes are on, graceful shutdown does not flush.
"a memtable holds changes until it hits a flush condition like being full (or, if durable_writes are disabled, on graceful shutdown), or you explicitly flush it"
Seems to unambiguously describe the current behavior and doesn't read very well.
I gather from CASSANDRA-3564 that there is interest in extending this flush-on-graceful shutdown behavior to occur in all cases of graceful shutdown. If that happens, your sentence will be a correct summation of Cassandra's newly predictable behavior. It will also answer my primary objection here, which is that the flush only occurs in *some* graceful shutdown cases. :)
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Robert Coli
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "David Phillips (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073624#comment-13073624 ]
David Phillips commented on CASSANDRA-2958:
-------------------------------------------
It looks like waitOnFutures() can return a null pointer exception if forceFlush() returns null?
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
> Memtables need to be flushed on shutdown when durable_writes is set to false, otherwise data loss occurs as the data is not available to be replayed from the commit log.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446416#comment-13446416 ]
Robert Coli commented on CASSANDRA-2958:
----------------------------------------
What I "want" is to be able to explain Cassandra to customers in simple sentences like "a memtable holds changes until it is full enough to hit a flush condition or you explicitly flush it" without having to pepper these sentences with caveats like "except if you restart your node or if you stop it with durable_writes off, triggering an unexpected flush." Patches such as this one, where vaguely defined ends appear to justify whatever ad-hoc inconsistent means, do not appear to further this goal.
Let me phrase my objection to this patch in another way...
"What does this patch gain us, and at what cost?"
Your stated goal is to not lose more than "necessary" when stopping a node. It seems your goal can be achieved without patching, by simply advising "durable_writes off" operators to run "nodetool drain" when stopping a node. They are, after all, the ones stopping their node and are perfectly capable of draining it if they do not want to lose the explicitly non-durable non-durable_writes contents of memtables.
>From what I can tell, the only thing this patch gains us is "people who are running with durable_writes off don't have to run 'nodetool drain' before stopping nodes."
What we trade for that is the until-now universal expectation that stopping a Cassandra node never triggers a flush.
Is "the very small group of operators who run with non-durable writes don't have to run 'nodetool drain'" such a compelling win that we should change a fundamental behavior of Cassandra, making it less predictable, in order to obtain it? My answer is no.
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2958:
-----------------------------------------
Assignee: Jonathan Ellis (was: Robert Coli)
Robert, we use assign-to to track who fixed an issue. Assigning to yourself at this point is not appropriate.
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446425#comment-13446425 ]
Brandon Williams commented on CASSANDRA-2958:
---------------------------------------------
bq. be able to explain Cassandra to customers in simple sentences like "a memtable holds changes until it is full enough to hit a flush condition or you explicitly flush it"
"a memtable holds changes until it hits a flush condition like being full or graceful shutdown, or you explicitly flush it."
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2958) Flush memtables on shutdown when
durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Coli reassigned CASSANDRA-2958:
--------------------------------------
Assignee: Robert Coli (was: Jonathan Ellis)
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Robert Coli
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2958) Flush memtables on shutdown
when durable writes are disabled
Posted by "Robert Coli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467949#comment-13467949 ]
Robert Coli commented on CASSANDRA-2958:
----------------------------------------
Jonathan, I had no intent of assigning this ticket to myself or wiping the description body, both of which I apparently accidentally did in the process of clicking on this ticket. I will restore the description body from the history, sorry for the confusion.
> Flush memtables on shutdown when durable writes are disabled
> ------------------------------------------------------------
>
> Key: CASSANDRA-2958
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2958
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: David Phillips
> Assignee: Jonathan Ellis
> Fix For: 0.8.3
>
> Attachments: 2958.txt
>
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira