You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2011/04/11 21:08:05 UTC
[jira] [Created] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Deprecate or modify per-cf memtable sizes in favor of the global threshold
--------------------------------------------------------------------------
Key: CASSANDRA-2449
URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Stu Hood
Fix For: 0.8
The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2449:
-----------------------------------------
Assignee: Brandon Williams
By all appearances memtable auto-tuning is working well. Let's nuke the old settings. (I.e., leave them in the Thrift definitions so old clients don't freak out, but make them optional and ignore them if they are set.)
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Brandon Williams
> Fix For: 1.0
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2449:
-----------------------------------------
Assignee: Jonathan Ellis (was: Pavel Yaskevich)
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2449:
--------------------------------------
Attachment: (was: 2449.txt)
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099765#comment-13099765 ]
Hudson commented on CASSANDRA-2449:
-----------------------------------
Integrated in Cassandra #1087 (See [https://builds.apache.org/job/Cassandra/1087/])
remove explicit per-CF memtable thresholds
patch by jbellis; reviewed by brandonwilliams for CASSANDRA-2449
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166520
Files :
* /cassandra/trunk/NEWS.txt
* /cassandra/trunk/conf/cassandra.yaml
* /cassandra/trunk/interface/cassandra.thrift
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/CfDef.java
* /cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Constants.java
* /cassandra/trunk/src/avro/internode.genavro
* /cassandra/trunk/src/java/org/apache/cassandra/config/CFMetaData.java
* /cassandra/trunk/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
* /cassandra/trunk/src/java/org/apache/cassandra/cql/CreateColumnFamilyStatement.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStoreMBean.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/Memtable.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/Table.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
* /cassandra/trunk/src/java/org/apache/cassandra/db/index/keys/KeysIndex.java
* /cassandra/trunk/src/java/org/apache/cassandra/service/StorageService.java
* /cassandra/trunk/test/unit/org/apache/cassandra/db/DefsTest.java
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
> Attachments: 2449.txt
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2449:
-----------------------------------------
Assignee: Pavel Yaskevich (was: Brandon Williams)
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Pavel Yaskevich
> Fix For: 1.0
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099731#comment-13099731 ]
Brandon Williams commented on CASSANDRA-2449:
---------------------------------------------
+1
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
> Attachments: 2449.txt
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2449:
--------------------------------------
Attachment: 2449.txt
Removing optional fields is backwards-compatible in both Thrift and Avro so I went ahead and removed them in the new patch.
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
> Attachments: 2449.txt
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2449:
--------------------------------------
Attachment: 2449.txt
the memtable threshold fields were already optional in both Thrift and Avro, so I've just taken them out of our internal CFMetadata and anything that still touched that.
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Jonathan Ellis
> Fix For: 1.0
>
> Attachments: 2449.txt
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2449) Deprecate or modify per-cf
memtable sizes in favor of the global threshold
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018528#comment-13018528 ]
Jonathan Ellis commented on CASSANDRA-2449:
-------------------------------------------
bq. replace them with a preference value, which controls the relative memory usage of one CF versus another
I'm not a fan of adding additional complexity here. At best, you'll get substantially the current behavior; at worst (when you tell it to prefer keeping the largest CF in memory), you'll create a flush storm of smaller CFs.
> Deprecate or modify per-cf memtable sizes in favor of the global threshold
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2449
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2449
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Fix For: 0.8
>
>
> The new memtable_total_space_in_mb setting is an excellent way to cap memory usage for memtables, and one could argue that it should replace the per-cf memtable sizes entirely. On the other hand, people may still want a knob to tune to flush certain cfs less frequently.
> I think a best of both worlds approach might be to deprecate the memtable_(throughput|operations) settings, and replace them with a preference value, which controls the relative memory usage of one CF versus another (all CFs at 1 would mean equal preference). For backwards compatibility, we could continue to read from the _throughput value and treat it as the preference value, while logging a warning.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira