You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ching-cheng (JIRA)" <ji...@apache.org> on 2011/02/17 17:52:24 UTC
[jira] Created: (CASSANDRA-2183) memtable_flush_after_mins setting
not working
memtable_flush_after_mins setting not working
---------------------------------------------
Key: CASSANDRA-2183
URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.1, 0.7.0
Reporter: Ching-cheng
Priority: Minor
We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
------------------------------------------------------------------------------------------------------------------
int minCheckMs = Integer.MAX_VALUE;
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
}
Runnable runnable = new Runnable()
{
public void run()
{
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
cfs.forceFlushIfExpired();
}
}
};
flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
------------------------------------------------------------------------------------------------------------------------------
Now for our application, we will create a keyspacewithout any columnfamily first. And only add needed columnfamily later depends on request.
However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
since there is no columnfamily yet.
Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2183) memtable_flush_after_mins
setting not working
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998409#comment-12998409 ]
Hudson commented on CASSANDRA-2183:
-----------------------------------
Integrated in Cassandra-0.7 #309 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/309/])
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Ching-cheng
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.3
>
> Attachments: 2183.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2183) memtable_flush_after_mins setting
not working
Posted by "Ching-cheng (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ching-cheng updated CASSANDRA-2183:
-----------------------------------
Description:
We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
------------------------------------------------------------------------------------------------------------------
int minCheckMs = Integer.MAX_VALUE;
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
}
Runnable runnable = new Runnable()
{
public void run()
{
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
cfs.forceFlushIfExpired();
}
}
};
flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
------------------------------------------------------------------------------------------------------------------------------
Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
since there is no columnfamily yet.
Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
was:
We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
------------------------------------------------------------------------------------------------------------------
int minCheckMs = Integer.MAX_VALUE;
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
}
Runnable runnable = new Runnable()
{
public void run()
{
for (ColumnFamilyStore cfs : columnFamilyStores.values())
{
cfs.forceFlushIfExpired();
}
}
};
flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
------------------------------------------------------------------------------------------------------------------------------
Now for our application, we will create a keyspacewithout any columnfamily first. And only add needed columnfamily later depends on request.
However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
since there is no columnfamily yet.
Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Ching-cheng
> Priority: Minor
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2183) memtable_flush_after_mins setting
not working
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2183:
--------------------------------------
Attachment: 2183.txt
The original approach of checking barely more often than we think is necessary is overengineering the problem; even with 1000s of CFs, checking every 10s would have no affect whatsoever on performance. Patch attached.
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Ching-cheng
> Priority: Minor
> Fix For: 0.7.3
>
> Attachments: 2183.txt
>
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2183) memtable_flush_after_mins setting
not working
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2183:
--------------------------------------
Remaining Estimate: 1h
Original Estimate: 1h
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Ching-cheng
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.3
>
> Attachments: 2183.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2183) memtable_flush_after_mins setting
not working
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2183:
--------------------------------------
Reviewer: chenyy
Affects Version/s: (was: 0.7.1)
Fix Version/s: 0.7.3
Assignee: Jonathan Ellis
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Ching-cheng
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.3
>
> Attachments: 2183.txt
>
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2183) memtable_flush_after_mins
setting not working
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997591#comment-12997591 ]
Peter Schuller commented on CASSANDRA-2183:
-------------------------------------------
(was asked to review) I agree on the overengineering. Patch looks good to me. The obvious caveat of being limited to 10 second resolution should be utterly irrelevant for the purposes for which the time based flushing is intended.
> memtable_flush_after_mins setting not working
> ---------------------------------------------
>
> Key: CASSANDRA-2183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2183
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Reporter: Ching-cheng
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.3
>
> Attachments: 2183.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We have observed the behavior that memtable_flush_after_mins setting not working occasionally. After some testing and code digging, we finally figured out what going on.
> The memtable_flush_after_mins won't work on certain condition with current implementation in Cassandra.
> In org.apache.cassandra.db.Table, the scheduled flush task is setup by the following code during construction.
> ------------------------------------------------------------------------------------------------------------------
> int minCheckMs = Integer.MAX_VALUE;
>
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> minCheckMs = Math.min(minCheckMs, cfs.getMemtableFlushAfterMins() * 60 * 1000);
> }
> Runnable runnable = new Runnable()
> {
> public void run()
> {
> for (ColumnFamilyStore cfs : columnFamilyStores.values())
> {
> cfs.forceFlushIfExpired();
> }
> }
> };
> flushTask = StorageService.scheduledTasks.scheduleWithFixedDelay(runnable, minCheckMs, minCheckMs, TimeUnit.MILLISECONDS);
> ------------------------------------------------------------------------------------------------------------------------------
> Now for our application, we will create a keyspacewithout without any columnfamily first. And only add needed columnfamily later depends on request.
> However, when keyspacegot created (without any columnfamily ), the above code will actually schedule a fixed delay flush check task with Integer.MAX_VALUE ms
> since there is no columnfamily yet.
> Later when you add columnfamily to this empty keyspace, the initCf() method in Table.java doesn't check whether the scheduled flush check task interval need
> to be updated or not. To fix this, we'd need to restart the Cassandra after columnfamily added into the keyspace.
> I would suggest that add additional logic in initCf() method to recreate a scheduled flush check task if needed.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira