You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Oleg Kibirev (JIRA)" <ji...@apache.org> on 2012/09/18 22:51:08 UTC
[jira] [Created] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Oleg Kibirev created CASSANDRA-4681:
---------------------------------------
Summary: SlabAllocator spends a lot of time in Thread.yield
Key: CASSANDRA-4681
URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.5
Environment: OEL Linux
Reporter: Oleg Kibirev
When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
if (oldOffset == UNINITIALIZED)
{
// The region doesn't have its data allocated yet.
// Since we found this in currentRegion, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
}
do:
if (oldOffset == UNINITIALIZED)
return ByteBuffer.allocate(size);
I achieved 4x speed up in my (admittedly specialized) benchmark.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458737#comment-13458737 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
I have also prototyped the newly attached version that keeps a list of extra slabs in the event of a race condition, however it's throughput and GC performance are inferior to the simpler fix, at least in my benchmark.
I don't think allocating separate blocks only in the event of a race condition "defeats the purpose" of SlabAllocator. Either race condition is rare and impact on GC is minimum - after all there are many other objects of different size, including an instance of ByteBuffer per call, that are allocated in addition to slabs. Or the node is so congested that race condition is the default case, in which case no solution, but especially not "while (!done) Thread.yield()" is likely to produce good results.
It is of course entirely possible that the .list version performs better in a different benchmark.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458811#comment-13458811 ]
Jonathan Ellis commented on CASSANDRA-4681:
-------------------------------------------
If you're seeing 4x performance, that doesn't sound very rare to me.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458928#comment-13458928 ]
Jonathan Ellis commented on CASSANDRA-4681:
-------------------------------------------
Suppose current region A is exhausted. A.next = B. All threads race to CAS B into A, then continue. Now you have current = B and B.next = null. Nobody will try to allocate B.next until B is exhausted, in which case they will all create the new Region and CAS again.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458180#comment-13458180 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
The benchmark exercised two column families. I think the key is that I am using a large number of very small column names/values and insert concurrently from dozens of threads (due to characteristics of the real world use case). The system used has 24 CPU cores and an SSD raid. So basically there is a very high rate of allocations and large number of concurrent threads hitting the same SlabAllocator.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kibirev updated CASSANDRA-4681:
------------------------------------
Attachment: SlabAllocator.java
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocated unneeded ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kibirev updated CASSANDRA-4681:
------------------------------------
Attachment: slab-list.patch
patch for list-based slab allocator against 1.1.5
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459087#comment-13459087 ]
Jonathan Ellis commented on CASSANDRA-4681:
-------------------------------------------
What we want is to create a Next with a single thread while everyone else gets to allocate from old-next and get back to what they were doing.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4681:
--------------------------------------
Attachment: 4681-v3.txt
This doesn't quite do what you want; it just postpones the race until next == null. (And the cost of a race is much higher since each Region construction now includes the full byte[].)
v3 attached to allow just one thread* to continue and allocate next when the original is exhausted.
*other threads that fail allocation, but check for next != null after a successful CAS will still race, which is why we need to make .next an AtomicReference as well.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kibirev updated CASSANDRA-4681:
------------------------------------
Description:
When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
if (oldOffset == UNINITIALIZED)
{
// The region doesn't have its data allocated yet.
// Since we found this in currentRegion, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
}
do:
if (oldOffset == UNINITIALIZED)
return ByteBuffer.allocate(size);
I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocated unneeded ByteBuffer instances has a measurable effect on performance
was:
When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
if (oldOffset == UNINITIALIZED)
{
// The region doesn't have its data allocated yet.
// Since we found this in currentRegion, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
}
do:
if (oldOffset == UNINITIALIZED)
return ByteBuffer.allocate(size);
I achieved 4x speed up in my (admittedly specialized) benchmark.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocated unneeded ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4681) SlabAllocator spends a lot
of time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458928#comment-13458928 ]
Jonathan Ellis edited comment on CASSANDRA-4681 at 9/20/12 5:27 AM:
--------------------------------------------------------------------
Suppose current region A is exhausted. A.next = B. All threads CAS B into A, then continue. Now you have current = B and B.next = null. Nobody will try to allocate B.next until B is exhausted, in which case they will all create the new Region and CAS again.
was (Author: jbellis):
Suppose current region A is exhausted. A.next = B. All threads race to CAS B into A, then continue. Now you have current = B and B.next = null. Nobody will try to allocate B.next until B is exhausted, in which case they will all create the new Region and CAS again.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kibirev updated CASSANDRA-4681:
------------------------------------
Attachment: SlabAllocator.java.list
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458920#comment-13458920 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
Jonathan - why is there a race when next is not modified once a region is inserted into the list? I am either inserting or removing a node into the linked list and use compareAndSet to make sure the head of the list still has an expected value.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458362#comment-13458362 ]
Jonathan Ellis commented on CASSANDRA-4681:
-------------------------------------------
The "fix" here defeats the purpose of the SlabAllocator, and you will pay a penalty in STW GC for non-uniform value sizes. So I'm unenthusiastic about adopting this for general use.
Instead you might try preallocating a queue of buffers or regions to reduce init time.
(Note that we never allocate unneeded bytebuffers, that is the point of the UNINITIALIZED dance instead of just each thread creating the BB and CASing to see who wins.)
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458834#comment-13458834 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
This was a complete file for general review and discussion. I have just uploaded a patch for the same against 1.1.5.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458828#comment-13458828 ]
Jonathan Ellis commented on CASSANDRA-4681:
-------------------------------------------
Not sure:
{noformat}
$ patch -p1 < SlabAllocator.java.list
patch: **** Only garbage was found in the patch input.
{noformat}
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4681) SlabAllocator spends a lot
of time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458821#comment-13458821 ]
Oleg Kibirev edited comment on CASSANDRA-4681 at 9/20/12 3:33 AM:
------------------------------------------------------------------
So in this case the list version should be good? It triggered more full GCs than simple version in my benchmark, perhaps because the threads were delayed by more complex code/allocating a bigger block from completing activity that would have otherwise released some memory. But things could certainly be different in another benchmark
was (Author: okibirev):
So in this case the list version should be good? It triggered more full GCs than simple version in my benchmark, perhaps because the threads were delayed by more complex code/allocating a bigger block from completing activity that would have otherwise released some memory.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458173#comment-13458173 ]
Brandon Williams commented on CASSANDRA-4681:
---------------------------------------------
bq. I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached.
How many CFs were you using when you saw this? Any other details would be appreciated.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458821#comment-13458821 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
So in this case the list version should be good? It triggered more full GCs than simple version in my benchmark, perhaps because the threads were delayed by more complex code/allocating a bigger block from completing activity that would have otherwise released some memory.
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java, SlabAllocator.java.list
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleg Kibirev updated CASSANDRA-4681:
------------------------------------
Description:
When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
if (oldOffset == UNINITIALIZED)
{
// The region doesn't have its data allocated yet.
// Since we found this in currentRegion, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
}
do:
if (oldOffset == UNINITIALIZED)
return ByteBuffer.allocate(size);
I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
was:
When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
if (oldOffset == UNINITIALIZED)
{
// The region doesn't have its data allocated yet.
// Since we found this in currentRegion, we know that whoever
// CAS-ed it there is allocating it right now. So spin-loop
// shouldn't spin long!
Thread.yield();
continue;
}
do:
if (oldOffset == UNINITIALIZED)
return ByteBuffer.allocate(size);
I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocated unneeded ByteBuffer instances has a measurable effect on performance
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: SlabAllocator.java
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4681) SlabAllocator spends a lot of
time in Thread.yield
Posted by "Oleg Kibirev (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458935#comment-13458935 ]
Oleg Kibirev commented on CASSANDRA-4681:
-----------------------------------------
So they will all create new regions and push them onto the stack. No regions will be leaked. By definition, there are only 3 alternatives 1. make slab allocation single-threaded (current approach) 2. allocate non slabbed buffers if a slab is not ready 3. Allocate multiple slabs in racing threads and make them available for subsequent use. How does "next is atomicreference" approach differ from the original one?
> SlabAllocator spends a lot of time in Thread.yield
> --------------------------------------------------
>
> Key: CASSANDRA-4681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4681
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.5
> Environment: OEL Linux
> Reporter: Oleg Kibirev
> Attachments: 4681-v3.txt, SlabAllocator.java, SlabAllocator.java.list, slab-list.patch
>
>
> When profiling high volume inserts into Cassandra running on a host with fast SSD and CPU, Thread.yield() invoked by SlabAllocator appeared as the top item in CPU samples. The fix is to return a regular byte buffer if current slab is being initialized by another thread. So instead of:
> if (oldOffset == UNINITIALIZED)
> {
> // The region doesn't have its data allocated yet.
> // Since we found this in currentRegion, we know that whoever
> // CAS-ed it there is allocating it right now. So spin-loop
> // shouldn't spin long!
> Thread.yield();
> continue;
> }
> do:
> if (oldOffset == UNINITIALIZED)
> return ByteBuffer.allocate(size);
> I achieved 4x speed up in my (admittedly specialized) benchmark by using an optimized version of SlabAllocator attached. Since this code is in the critical path, even doing excessive atomic instructions or allocating unneeded extra ByteBuffer instances has a measurable effect on performance
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira