You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Zhu Han (JIRA)" <ji...@apache.org> on 2012/08/13 08:42:37 UTC
[jira] [Created] (CASSANDRA-4533) Cache saving does not work
Zhu Han created CASSANDRA-4533:
----------------------------------
Summary: Cache saving does not work
Key: CASSANDRA-4533
URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.3, 1.0.11
Reporter: Zhu Han
Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
Due to the effect of above atomic flag, only a cache can be written out to disk. Other writer are cancelled when the flag is true.
I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439199#comment-13439199 ]
Yuki Morishita commented on CASSANDRA-4533:
-------------------------------------------
bq. Hmm, I don't think this quite works because it still means we can skip saving cache for CF X when CF Y is being flushed.
In my understanding, since 1.1, C* stores key and row caches globally, those are saved at once for every CF for each cache type.
AutoSavingCache$Writer writes all CF for certain CacheType in one execution.
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
> Attachments: 4533-1.1.txt
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435116#comment-13435116 ]
Jonathan Ellis commented on CASSANDRA-4533:
-------------------------------------------
No, it's not worth risking regressions over in 1.0.x.
The good news is that 1.1.4+ look pretty stable.
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Zhu Han (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435198#comment-13435198 ]
Zhu Han commented on CASSANDRA-4533:
------------------------------------
Looks like we have to maintain our own fork, as we can not keep up with the upgrade cycle...
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4533) Multithreaded cache saving
can skip caches
Posted by "Zhu Han (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434830#comment-13434830 ]
Zhu Han edited comment on CASSANDRA-4533 at 8/15/12 5:15 PM:
-------------------------------------------------------------
Is it possible to fix it also in 1.0?
was (Author: hanzhu):
Is it possible that it is fixed also in 1.0?
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4533:
--------------------------------------
Reviewer: jbellis
Description:
Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
was:
Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
Due to the effect of above atomic flag, only a cache can be written out to disk. Other writer are cancelled when the flag is true.
I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
Priority: Trivial (was: Major)
Affects Version/s: (was: 1.1.3)
(was: 1.0.11)
0.8.0
Fix Version/s: 1.1.5
Assignee: Yuki Morishita
Summary: Multithreaded cache saving can skip caches (was: Cache saving does not work)
Looks like we should switch to a ConcurrentSet like we did in Memtable.meteringInProgress.
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Zhu Han (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434830#comment-13434830 ]
Zhu Han commented on CASSANDRA-4533:
------------------------------------
Is it possible that it is fixed also in 1.0?
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439231#comment-13439231 ]
Jonathan Ellis commented on CASSANDRA-4533:
-------------------------------------------
You're right, my mistake. +1
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
> Attachments: 4533-1.1.txt
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438996#comment-13438996 ]
Jonathan Ellis commented on CASSANDRA-4533:
-------------------------------------------
Hmm, I don't think this quite works because it still means we can skip saving cache for CF X when CF Y is being flushed.
I think the problem this code is trying to solve, over a basic executor + queue, is multiple tasks for X getting queued up while (say) compaction is sucking a lot of i/o, then firing off those cache-save tasks for X faster than the defined saving period when it speeds up.
I guess we could make it a Pair<CF, CacheType>?
TBH this is probably premature optimization, if your cache period is so frequent that multiple queued tasks is a problem, then you should just fix that. I'd be okay with just ripping this out. Alternatively, we could have the task check to see if the last-saved cache is older than M minutes before overwriting it, similar to how normal background compaction submissions are a no-op if it turns out there's nothing to do by the time we execute the task.
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
> Attachments: 4533-1.1.txt
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4533) Multithreaded cache saving can
skip caches
Posted by "Yuki Morishita (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuki Morishita updated CASSANDRA-4533:
--------------------------------------
Attachment: 4533-1.1.txt
Attaching patch against 1.1 branch.
Caches are grobal since 1.1, so I used CacheType as key for flushInProgerss concurrent set.
> Multithreaded cache saving can skip caches
> ------------------------------------------
>
> Key: CASSANDRA-4533
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4533
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Reporter: Zhu Han
> Assignee: Yuki Morishita
> Priority: Trivial
> Fix For: 1.1.5
>
> Attachments: 4533-1.1.txt
>
>
> Cassandra flushes the key and row cache to disk periodically. It also uses a atomic flag in flushInProgress to enforce single cache writer at any time.
> However, the cache saving task could be submitted to CompactionManager concurrently, as long as the number of worker thread in CompactionManager is larger than 1.
> Due to the effect of above atomic flag, only one cache will be written out to disk. Other writer are cancelled when the flag is true.
> I observe the situation in Cassandra 1.0. If nothing is changed, the problem should remain in Cassandra 1.1, either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira