You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ian Soboroff (JIRA)" <ji...@apache.org> on 2010/06/04 18:54:54 UTC
[jira] Created: (CASSANDRA-1161) Data file quotas
Data file quotas
----------------
Key: CASSANDRA-1161
URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
Project: Cassandra
Issue Type: New Feature
Components: Core
Affects Versions: 0.6.2
Reporter: Ian Soboroff
Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1161) Data file quotas
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1161.
---------------------------------------
Resolution: Won't Fix
if you have files X and Y then preventing Cassandra from combining them into Z will not make things better, and may make things worse (since compaction saves you from wasting space on old versions)
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1161) Data file quotas
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875682#action_12875682 ]
Stu Hood edited comment on CASSANDRA-1161 at 6/4/10 1:58 PM:
-------------------------------------------------------------
In the case of cleanup compactions, we could utilize the compaction range and patch 0005 from CASSANDRA-579 to determine a more accurate post-compaction size for sstables, which should allow us to complete cleanup the majority of the time.
EDIT: 0005 isn't quite ready, but you get the idea.
was (Author: stuhood):
In the case of cleanup compactions, we could utilize the compaction range and patch 0005 from CASSANDRA-579 to determine a more accurate post-compaction size for sstables, which should allow us to complete cleanup the majority of the time.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Ian Soboroff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875670#action_12875670 ]
Ian Soboroff commented on CASSANDRA-1161:
-----------------------------------------
I agree this isn't the best solution, btw, because to actually work the limit has to be to have enough free space to anticompact the largest db file.
So let me rephrase my tradeoff question - what's the tradeoff between fewer larger db files, and more smaller ones? The latter would be quicker to anticompact (and compact), and easier to recover from in case of corruption. Linear scans of all dbs on a node become somewhat more expensive since each db costs an open(). Beyond this, I don't know the internal architecture of the storage layer to hypothesize.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Ian Soboroff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875669#action_12875669 ]
Ian Soboroff commented on CASSANDRA-1161:
-----------------------------------------
Well, it can always create a new file, too, right?
What's the tradeoff between appending an existing file and creating a new one?
The future problem - a db file growing past the point where it can be anticompacted - is currently hard to keep track of and hard to recover from.
Anyway, it was just a suggestion... if it doesn't make sense, mark it closed.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875682#action_12875682 ]
Stu Hood commented on CASSANDRA-1161:
-------------------------------------
In the case of cleanup compactions, we could utilize the compaction range and patch 0005 from CASSANDRA-579 to determine a more accurate post-compaction size for sstables, which should allow us to complete cleanup the majority of the time.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875674#action_12875674 ]
Jeremy Hanna commented on CASSANDRA-1161:
-----------------------------------------
Ian - not sure if you are running into this problem as a result of trying to add nodes, but CASSANDRA-579 might address that specific problem and is nearing completion, slated for 0.7.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875661#action_12875661 ]
Jonathan Ellis commented on CASSANDRA-1161:
-------------------------------------------
so, you propose to cause some kind of badness now (by not allowing it to create the file it wants to) in order to prevent badness in the future?
this doesn't sound like a good solution to me.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1161) Data file quotas
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875676#action_12875676 ]
Jeremy Hanna commented on CASSANDRA-1161:
-----------------------------------------
I'm not trying to detract from what you're saying though - it would be nice to have something to help out with making sure there is enough space for anti-compaction.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1161) Data file quotas
Posted by "Ian Soboroff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875670#action_12875670 ]
Ian Soboroff edited comment on CASSANDRA-1161 at 6/4/10 1:28 PM:
-----------------------------------------------------------------
I agree this isn't the best solution, btw, because to actually work the limit has to be to have enough free space to anticompact the largest db file.
So let me rephrase my tradeoff question - what's the tradeoff between fewer larger db files, and more smaller ones? The latter would be quicker to anticompact (and compact), and easier to recover from in case of corruption. Linear scans of all dbs on a node become somewhat more expensive since each db costs an open(). Beyond this, I don't know the internal architecture of the storage layer to hypothesize.
If one can place a quota on the size of a single db file, then it's easy to make sure enough space is reserved for an anticompaction.
was (Author: isoboroff):
I agree this isn't the best solution, btw, because to actually work the limit has to be to have enough free space to anticompact the largest db file.
So let me rephrase my tradeoff question - what's the tradeoff between fewer larger db files, and more smaller ones? The latter would be quicker to anticompact (and compact), and easier to recover from in case of corruption. Linear scans of all dbs on a node become somewhat more expensive since each db costs an open(). Beyond this, I don't know the internal architecture of the storage layer to hypothesize.
> Data file quotas
> ----------------
>
> Key: CASSANDRA-1161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1161
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
> Affects Versions: 0.6.2
> Reporter: Ian Soboroff
>
> Since anticompaction may require significant disk space, it would be nice to be able to specify a disk quota.
> A simple quota option supporting literal and percentage quantities would be nice, but I think a better approach would be an option to not allow a Data.db file to grow larger in size than the existing free space on the volume.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.