You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2011/01/08 04:35:45 UTC
[jira] Created: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Double-check or replace RRW memtable lock
-----------------------------------------
Key: CASSANDRA-1954
URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Stu Hood
Priority: Minor
Fix For: 0.7.1, 0.8
{quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-1954:
---------------------------------------
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010732#comment-13010732 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
bq. Can't we just have some volatile boolean that we check before writing (and wait on some simpleCondition if the boolean is set). We could set that flag (and the condition) whenever we detect that we're over capacity, and release the flag and condition when a flush thread gets available
Now we are talking about:
volatile boolean
condition variable
writer atomic counter
Table.lock for switching
Is this really simpler/better than the old approach?
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010750#comment-13010750 ]
Sylvain Lebresne commented on CASSANDRA-1954:
---------------------------------------------
The main advantages in my opinion is that write don't have to acquire the flush read lock anymore. This means that you avoid the stop-the-world behavior each time a memtable is switched. The idea of the volatile boolean for writes is that it will be set only if while scheduling a flush we detect a over capacity problem. This may not be less code that we had before, but I do believe this will help getting much uniform latencies for writes.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996734#comment-12996734 ]
Stu Hood commented on CASSANDRA-1954:
-------------------------------------
I like this.
Nitpick: rather than nesting most of the body of {{maybeSwitchMemtable}} inside {{if (oldMemtable.markPendingFlush())}}, you could negate and return early.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007320#comment-13007320 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
are you saying that the assert is obviously correct and hence redundant, or the assert is incorrect and will throw exceptions under some conditions?
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014855#comment-13014855 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
Reverted until we can fix this (r1087919).
I also think we should measure the benefits of the volatile + writer count approach in smoothing latency. If it is not reproducibly better let's stick with the approach that's been debugged longer. :)
I'm fine with leaving this until post-0.8 fwiw.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995078#comment-12995078 ]
Stu Hood commented on CASSANDRA-1954:
-------------------------------------
Also, it looks like we increment the memtableswitch count outside the writeLock, so it is being incremented far too many times.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987358#action_12987358 ]
Stu Hood commented on CASSANDRA-1954:
-------------------------------------
Yea, that should work.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1954.
---------------------------------------
Resolution: Later
Fix Version/s: (was: 0.7.1)
(was: 0.8)
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1954:
----------------------------------------
Attachment: 1954_trunk.patch
Rebased (replaced 1954_trunk.patch)
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1954:
----------------------------------------
Attachment: 0001-Remove-flusherLock-readLock.patch
Attached patch implements what's hinted in my previous comment. Mainly it keeps the flusher lock to unsure we discard commit log segment in the order we read them, but remove the readLock on writes, so flush don't block writes anymore.
It passes unit test but that doesn't say much.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995091#comment-12995091 ]
Stu Hood commented on CASSANDRA-1954:
-------------------------------------
An idea to replace this lock was proposed in IRC yesterday: if the list of memtables and sstables was stored in a single immutable, cas'able structure, you could atomically swap a memtable from active to flushing, and then from flushing into an sstable. Example structure:
{code:java}class View {
final List<Memtable> active;
final List<Memtable> flushing;
final List<SSTableReader> sstables;
}{code}
So a writing thread noticing an active Memtable past its threshold would attempt to CAS in a new Memtable while moving the current memtable to flushing. A thread finishing a flush would attempt to CAS the memtable it flushed out of flushing and into the sstable list.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010687#comment-13010687 ]
Sylvain Lebresne commented on CASSANDRA-1954:
---------------------------------------------
bq. Ideas?
Can't we just have some volatile boolean that we check before writing (and wait on some simpleCondition if the boolean is set). We could set that flag (and the condition) whenever we detect that we're over capacity, and release the flag and condition when a flush thread gets available.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1954:
----------------------------------------
Attachment: (was: 1954_trunk.patch)
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986392#action_12986392 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
The benefit is that we can have multiple writers acquire the readlock (yes, that's confusing :), but they will all be blocked on flush while the writelock is acquired.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028374#comment-13028374 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
I think the approach here also has a subtle problem keeping index CFs in sync with the data CFs: if I mark a given replay position in the data CF, then I need to guarantee that the index updates for all mutations before that position are flushed along with the the data updates, since index updates are made during data mutation and not recorded separately in the CL.
Since we do not block other writing threads while switching memtables, I think it's possible that we violate that requirement here.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.0
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000727#comment-13000727 ]
Hudson commented on CASSANDRA-1954:
-----------------------------------
Integrated in Cassandra #751 (See [https://hudson.apache.org/hudson/job/Cassandra/751/])
avoid aquiring (and contending with flush for) flusherlock on each write
patch by slebresne; reviewed by jbellis and stuhood for CASSANDRA-1954
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010731#comment-13010731 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
Sylvain points out, "[the 0.7 patch] just avoids too much contention on the write lock. There will still be one thread that will acquire it in write mode; all other writes will be blocked on the read lock." So reverting that is not necessary.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1954:
--------------------------------------
Attachment: 1954-0.7-v2.txt
double-checked locking patch for 0.7
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-1954:
----------------------------------------
Attachment: 1954_trunk.patch
Attaching the v2 patch rebased against trunk.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Vijay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007310#comment-13007310 ]
Vijay commented on CASSANDRA-1954:
----------------------------------
+1 but a invalid check which was not created by this patch though... "assert memtable == oldMemtable;" if it is really needed...
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986831#action_12986831 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
if we made Memtable.isFrozen volatile, I think we wouldn't even need to acquire the readlock for the first check
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979085#action_12979085 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
flusherlock is part of how we make sure we update commitlog headers post-flush in the correct order, it's not just to keep writes out of being-flushed memtables. so replacing w/ a counter isn't as simple as it might look at first.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Fix For: 0.7.1, 0.8
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood reopened CASSANDRA-1954:
---------------------------------
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated CASSANDRA-1954:
--------------------------------
Attachment: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
Apparently I was confused about what "reentrant" means, oi. Nonetheless, we can still perform the double checking to cut back on writeLock acquisitions. Attached.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006990#comment-13006990 ]
Sylvain Lebresne commented on CASSANDRA-1954:
---------------------------------------------
+1 on the double-checkd locking patch for 0.7.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1954:
--------------------------------------
Remaining Estimate: 8h
Original Estimate: 8h
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-1954) Double-check or
replace RRW memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995091#comment-12995091 ]
Stu Hood edited comment on CASSANDRA-1954 at 2/16/11 12:39 AM:
---------------------------------------------------------------
An idea to replace this lock was proposed in IRC yesterday: if the list of memtables and sstables was stored in a single immutable, cas'able structure, you could atomically swap a memtable from active to flushing, and then from flushing into an sstable. Example structure:
{code:java}class View {
final List<Memtable> active;
final List<Memtable> flushing;
final List<SSTableReader> sstables;
}{code}
So a writing thread noticing an active Memtable past its threshold would attempt to CAS in a new Memtable while moving the current memtable to flushing. A thread finishing a flush would attempt to CAS the memtable it flushed out of flushing and into the sstable list.
EDIT: Bah... this doesn't actually remove the necessity of the write lock, since we need to ensure that writers are not looking at a memtable that has been moved to flushing. We'd need another solution to that problem.
was (Author: stuhood):
An idea to replace this lock was proposed in IRC yesterday: if the list of memtables and sstables was stored in a single immutable, cas'able structure, you could atomically swap a memtable from active to flushing, and then from flushing into an sstable. Example structure:
{code:java}class View {
final List<Memtable> active;
final List<Memtable> flushing;
final List<SSTableReader> sstables;
}{code}
So a writing thread noticing an active Memtable past its threshold would attempt to CAS in a new Memtable while moving the current memtable to flushing. A thread finishing a flush would attempt to CAS the memtable it flushed out of flushing and into the sstable list.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reopened CASSANDRA-1954:
---------------------------------------
reopening to address this. will revert the 0.7 change.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1954.
---------------------------------------
Resolution: Fixed
committed
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008314#comment-13008314 ]
Hudson commented on CASSANDRA-1954:
-----------------------------------
Integrated in Cassandra-0.7 #391 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/391/])
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1954:
--------------------------------------
Attachment: 1954-v2.txt
I like it. I renamed Frozen in v2 to PendingFlush (since with the new design we accept writes to the old memtable even while maybeSwitch is running, so the old implication of "once it's frozen nothing new gets added" is no longer true).
I do think we should move this to trunk though.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997369#comment-12997369 ]
Sylvain Lebresne commented on CASSANDRA-1954:
---------------------------------------------
I agree with the change of frozen to pendingFlush, as well as that this should probably go to trunk (given it's in a fairly critical path).
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986252#action_12986252 ]
Stu Hood commented on CASSANDRA-1954:
-------------------------------------
We're really not gaining all of the benefit of a RRW lock if N threads are going to acquire it in write mode, but suit yourself.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995277#comment-12995277 ]
Sylvain Lebresne commented on CASSANDRA-1954:
---------------------------------------------
Correct me if I am wrong but I think there is a 2 things:
* We need to ensure that we call the discard of the commmit context post-flush in the same order that we got the context pre-flush.
A lock is fine, especially if we diminish contention on it with double-checking or with an atomic boolean. That this block all writes however seems unnecessary. Why lock the read-lock during write (for that purpose I mean)? As long as we get the commit log context and schedule the post-flush task before changing the active memtable, we should be right. We may have to replay a tiny bit more, since a few write will go into the being flushed memtable after we got the context, but we won't lose any.
On a related note, why I understand why we want to preserve this 'pre-flush/post-flush' order er column family, I'm not sure I understand why it must be global since the commit log header distinguishes between the different CFs ?
* We need to keep writes out of being-flushed memtables.
For that, we can use per-memtable counters and make flush start by waiting on the counter to reach 0.
Am I missing something obvious here ?
As for the View structure proposed by Stu in the previous comment, this could actually be a quite reasonable solution for CASSANDRA-2105 so I'll probably give that a shot too in this context.
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010549#comment-13010549 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
I think we forgot a 3rd goal of the Big Lock: make sure that when a memtable is full, all the flush threads are busy, and the flush queue is full, we _want to_ block writes so we don't OOM from shoving more data into the heap before we can finish freeing what is being flushed.
Ideas?
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8
>
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1954.
---------------------------------------
Resolution: Later
Fix Version/s: (was: 1.0)
Assignee: (was: Sylvain Lebresne)
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-0.7-v2.txt, 1954-v2.txt, 1954_trunk.patch
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-1954) Double-check or replace RRW
memtable lock
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999716#comment-12999716 ]
Jonathan Ellis commented on CASSANDRA-1954:
-------------------------------------------
Sorry, I was too slow -- already needs rebase
> Double-check or replace RRW memtable lock
> -----------------------------------------
>
> Key: CASSANDRA-1954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1954
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Stu Hood
> Priority: Minor
> Attachments: 0001-Double-check-in-maybeSwitchMemtable-to-minimize-writeL.txt, 0001-Remove-flusherLock-readLock.patch, 1954-v2.txt, 1954_trunk.patch
>
>
> {quote}...when a Memtable reaches its threshold, up to (all) N write threads will often notice, and race to acquire the writeLock in order to freeze the memtable. This means that we do way more writeLock acquisitions than we need to...{quote}
> See CASSANDRA-1930 for backstory, but adding double checking inside a read lock before trying to re-entrantly acquire the writelock would eliminate most of these excess writelock acquisitions.
> Alternatively, we should explore removing locking from these structures entirely, and replacing the writeLock acquisition with a per-memtable counter of active threads.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira