You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Johan Oskarsson (JIRA)" <ji...@apache.org> on 2009/09/08 19:57:57 UTC
[jira] Created: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Remove item flush limit in BinaryMemtable
-----------------------------------------
Key: CASSANDRA-433
URL: https://issues.apache.org/jira/browse/CASSANDRA-433
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.5
Reporter: Johan Oskarsson
Assignee: Johan Oskarsson
Fix For: 0.5
The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Goffinet closed CASSANDRA-433.
------------------------------------
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758654#action_12758654 ]
Johan Oskarsson commented on CASSANDRA-433:
-------------------------------------------
I still see the storage-conf.xml and DatabaseDescriptor entries for the binary memtable that are removed in Chris' patch in trunk. Is it just waiting to be merged?
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758674#action_12758674 ]
Jonathan Ellis commented on CASSANDRA-433:
------------------------------------------
I think I screwed up a conflict resolution. Will fix.
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753728#action_12753728 ]
Jonathan Ellis commented on CASSANDRA-433:
------------------------------------------
50k is ridiculously small, but the reason you need 2 checks is, the size estimate isn't even close to correct for small rows.
should we just make BMt use the memtable size and object count settings instead of making Binary versions of each in the config file?
I can't think of a reason to want to have separate settings for these, since if you are using both normal Mt and BMt at the same time you are Doing It Wrong. :)
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Johan Oskarsson
> Fix For: 0.5
>
> Attachments: CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-433:
-------------------------------------
Attachment: 433.patch
from irc:
jbellis: if you really use different sizes for BMt and normal Mt then i think we should have different settings. i'm just not sure why you'd want to do that
goffinet: we wouldnt want it to run that high [a heap] at all times; we'd rather use the ram for FS cache
jbellis: so if you do want different sizes for Mt/BMt let's just leave the separate BMt setting in and not make ops have to mess with things when bulk load is done
goffinet: ok
here is patch.
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, 433.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757799#action_12757799 ]
Chris Goffinet commented on CASSANDRA-433:
------------------------------------------
As per discussion with jbellis, I agree this is the right way going forward. You will want to increase the Memtable threshold and object sizes when doing Binary Memtable import, which also would most likely yield increasing the heap size. But once done, you will not want to have these thresholds that high at all times. So since you are decreasing the heap, this makes sense. This might cause a little difficulty with op situations but we will be providing a way to change these thresholds/sizes using either nodeprobe or another utility.
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Goffinet reassigned CASSANDRA-433:
----------------------------------------
Assignee: Chris Goffinet (was: Johan Oskarsson)
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758824#action_12758824 ]
Jonathan Ellis commented on CASSANDRA-433:
------------------------------------------
committed
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, 433.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated CASSANDRA-433:
--------------------------------------
Attachment: CASSANDRA-433.patch
This patch removes the 50000 item limit, making the BMT only rely on data size
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Johan Oskarsson
> Fix For: 0.5
>
> Attachments: CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Goffinet updated CASSANDRA-433:
-------------------------------------
Attachment: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch
Consolidated BMT threshold and object count sizes into getMemtableSize() and getMemtableObjectCount()
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Chris Goffinet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758819#action_12758819 ]
Chris Goffinet commented on CASSANDRA-433:
------------------------------------------
+1
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, 433.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-433) Remove item flush limit in
BinaryMemtable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758259#action_12758259 ]
Hudson commented on CASSANDRA-433:
----------------------------------
Integrated in Cassandra #205 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/205/])
Consolidate BMT threshold and object count sizes into getMemtableSize() and getMemtableObjectCount(). patch by Chris Goffinet; reviewed by jbellis for
> Remove item flush limit in BinaryMemtable
> -----------------------------------------
>
> Key: CASSANDRA-433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-433
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.5
> Reporter: Johan Oskarsson
> Assignee: Chris Goffinet
> Fix For: 0.5
>
> Attachments: 0001-CASSANDRA-433.-Consolidated-BMT-threshold-and-object.patch, CASSANDRA-433.patch
>
>
> The BinaryMemtable flushes in memory data to disk when the size of the data reaches a certain limit. There is also a hard coded limit that initiates the flush when more then 50000 items have been inserted. That causes issues if a lot of small items are inserted, we should remove or make the limit configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.