You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aaron Morton (JIRA)" <ji...@apache.org> on 2011/03/30 12:34:06 UTC
[jira] [Created] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
if out of disk space reclaim compacted SSTables during memtable flush
---------------------------------------------------------------------
Key: CASSANDRA-2404
URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7.4
Reporter: Aaron Morton
Assignee: Aaron Morton
Priority: Minor
Fix For: 0.7.5
During compaction if there is not enough disk space we invoke GC to reclaim unused space.
During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024836#comment-13024836 ]
Sylvain Lebresne commented on CASSANDRA-2404:
---------------------------------------------
Minor comments: in the estimation, the index size could be slightly improved by adding 10 * nb of keys (2 for the written size of the key + 8 for the recorded position). It's an estimation though so not a huge deal.
But otherwise, +1.
Made me wonder (but not directly related to the ticket), wouldn't it be "safer" when we're out of space to hold on the memtable (and thus block writes fairly quickly) instead of dropping the memtable "fairly silently" (we throw a RuntimeException that will be logged, but it still could be some time during which the node seems to accept write fine but will throw them out).
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6
>
> Attachments: 2404-0.7.txt, 2404-0.8.txt
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013908#comment-13013908 ]
Stu Hood commented on CASSANDRA-2404:
-------------------------------------
Actually, I would be all for _moving_ the logic from getDataFileLocation to createFlushWriter... not having room to flush is much more urgent than not having room to compact, and as a node gets closer to full, large bucket compactions will begin to spam GC for compactions that aren't going to be possible anyway.
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Aaron Morton
> Assignee: Aaron Morton
> Priority: Minor
> Fix For: 0.7.5
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2404:
--------------------------------------
Attachment: 2404-0.8.txt
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6
>
> Attachments: 2404-0.7.txt, 2404-0.8.txt
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2404:
--------------------------------------
Affects Version/s: (was: 0.7.4)
Assignee: Jonathan Ellis (was: Aaron Morton)
The reason we didn't do this historically is we didn't know how much free space should be the Panic Threshold on flush, whereas compaction has an easy-to-compute upper bound. Now we we can compute the flush size very easily (basically the throughput value + 2xkey sizes) so we can run the same "if not enough room then attempt a GC" logic.
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024878#comment-13024878 ]
Hudson commented on CASSANDRA-2404:
-----------------------------------
Integrated in Cassandra-0.7 #457 (See [https://builds.apache.org/hudson/job/Cassandra-0.7/457/])
force GC to reclaim disk space on flush, if necessary
patch by jbellis; reviewed by slebresne for CASSANDRA-2404
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6
>
> Attachments: 2404-0.7.txt, 2404-0.8.txt
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2404:
--------------------------------------
Fix Version/s: 0.8.0 beta 2
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6, 0.8.0 beta 2
>
> Attachments: 2404-0.7.txt, 2404-0.8.txt
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2404) if out of disk space reclaim
compacted SSTables during memtable flush
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2404:
--------------------------------------
Attachment: 2404-0.7.txt
Patch to route getFlushPath through table.getDataFileLocation so it shares the gc-to-clean-up-if-necessary logic.
> if out of disk space reclaim compacted SSTables during memtable flush
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-2404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2404
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Aaron Morton
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.6
>
> Attachments: 2404-0.7.txt, 2404-0.8.txt
>
>
> During compaction if there is not enough disk space we invoke GC to reclaim unused space.
> During memtable and binary memtable flush we just error out if there is not enough disk space to flush the table.
> Can we make cfs.createFlushWriter() use the same logic as Table.getDataFileLocation() to reclaim space if needed?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira